Related data from nested objects

Feature(s) impacted

Creating related data for a collection which contains nested objects.

Observed behavior

  1. I have collection of lawyers. Each lawyer contains an array of objects called PA. PA looks like this:
{
value : { 
     inside: {
        field1: string
        fields2: string
    },
    inside2: string
}
}

If I just create a model for PA, under forest/models, then the Related data becomes uneditable. Theoretically, the data should be there. The model declared under forest/models is identical with the data I have stored in the databse.

However, I tried implementing Flattened nested fields, which I can see its available from version 8+. I am currently using version7.
I observed that implementing this will have the following effect:

  1. PA Related data will dissapear

If I specify the level for flattening, it will look like this:

For both variants there will be no Practice Area field under Related Data. Ignore those null values.

Expected behavior

I would like to have a Related data field called Practice Areas that will allow me see the list of practice areas a lawyer has added in their profile. Also, I would like to have the ability to edit this list, modify existing data or add/delete practice areas.

I would prefer not to upgrade to version 8, so I am looking for solutions on how to use what I already have.

Failure Logs

Context

  • Project name: Jurata
  • Team name: Jurata
  • Environment name: Staging
  • Agent (forest package) name & version: …
"forest-express": "^7.4.0",
    "forest-express-mongoose": "^7.7.1",
  • Database type: …MongoDB
  • Recent changes made on your end if any: … none

I take in consideration to just upgrate to version 8. If that will be the case, could you describe me few keypoints I should pay attention to while upgrading ?

Let me consult with my colleagues.

I’m not sure that upgrading to forest-express-mongoose@8 will solve your issue, let me check and come back to you

1 Like

Sure ! Thanks for the prompt answer. I will be waiting for your suggestion.

So!

Here I am after asking around.

As you well know, the forest admin UI is built around tables and flat forms, which don’t play well with nested data: it’s much easier to design an admin panel for SQL than mongo!

Three consecutive attempts were made in the product to deal with nested data.

First iteration: when using agents < forest-express-mongoose@8.1.0

Agents before that version did not have any support for nested data.

The UI however does the following:

When a model contains fields that are nested one level

{
  firstName: 'john',
  lastName: 'doe',
  address: {
    // like this
    streetNumber: 1,
    streetName: '5th avenue'
  }
}

Those are displayed with an indentation in the forms
image

When a model contains a list of flat objects

{
  firstName: 'john',
  bills: [
    // like this
    { title: 'sdf', amount: 234 },
  }
}

Those get displayed in the related data section with the “white folder” logo.

Note that

  • All of this is happening in the UI.
  • The virtual relation is not visible in the explorer section

image

When a model contains anything else

{
  firstName: 'john',
  bills: [
    // like this, but can be anything else that does not fit into the
    // two previous cases
    { title: 'sdf', amount: 234, deep: { otherInfo: 2 } },
  }
}

In that case, the UI can no longer display the table component, so it falls back to using the JSON editor

image

Second iteration: when using agents >= forest-express-mongoose@8.1.0

The flattener was added to the forest-express-mongoose at that version.

It is backend code (in the agent) that transforms the models before sending them to the UI so that the
UI sees the data as flat, and then performs the inverse transformation on updates / creations.

It has a strong limitation, which is that it stops exploring the schema when you reach the first array on any document

It basically knows how to transform this:

{
  firstName: 'john',
  lastName: 'doe',
  address: {
    streetNumber: 1,
    streetName: '5th avenue',
    country: { id: 'fr', name: 'France' }
    // can nest more levels
  },
  nestedInfo: {
    deep: [
      { id: 1, name: 'whatever', deepDeep: { mode: 1 }
    ]
  }
}

into this:

{
  firstName: 'john',
  lastName: 'doe',
  'address->streetNumber': 1,
  'address->streetName':  '5th avenue',
  'address->streetName':  '5th avenue',
  'address->country->id':  'fr',
  'address->country->name':  'France',
  'nestedInfo->deep': [
     // Note that it is not flattening the submodels in the array
     { id: 1, name: 'whatever', deepDeep: { mode: 1 }
  ]
}

We can see here, that it would not solve your issue, because the relations are still being handled

Third iteration: when using @forestadmin/agent

This was released around one year ago.

The mongoose driver is a complete rewrite.

It can do anything that the previous flattener did, but also get rid of the one-to-one mapping between mongoose collections and forest admin collections.

To take your case, you can have one collection on mongoose (lawyers, which contains PAs), but two collections on forest admin (lawyers and PAs).

With this new system, the UI is not doing any work: it thinks that the agent is exposing two flat tables and just behaves as if the underlying storage was all flat tables.

This allows to do charts on submodels, or display them all a once in a table view, without taking in consideration where they come from

So it goes from this:

users
{
  firstName: 'john',
  bills: [
    { title: 'sdf', amount: 234, deep: { otherInfo: 2 } },
    { title: 'sdf', amount: 234, deep: { otherInfo: 2 } },
  }
}

to this:

users
{ _id: 1234, firstName: 'john' }

bill
{ _id: '1234.bills.0', billId: 1234, title: 'sdf', amount: 234, deep: { otherInfo: 2 } }
{ _id: '1234.bills.1', billId: 1234, title: 'sdf', amount: 234, deep: { otherInfo: 2 } }

(let me submit this reply, I’ll write the solutions for your issue on the next one as this is getting longer than I wanted)

1 Like

Now about solutions for the issue that you are having.

Migrating @forestadmin/agent

That would fix the issue.

However:

  • The documentation to do that will only be released next week
  • It wasn’t released earlier because we waited for feedback from early adopters
  • The migration is not trivial (we basically changed the whole API for the better, it was long due), and you may not want to invest time for such a small issue

Migrating to v8

It won’t help, as we’ve seen before, the implementation that is provided stops exploring models on arrays.

Staying on v7

If you want to stay on v7, what you would need to do is to trick the UI back into the mode you were in before adding the field that is causing issues.

This looks achievable with a smartfield flatPracticeAreas which would convert between (on both ways!) the nested and flat version of PA

{ 
  value: { 
    practiceArea: { _id: null, link: null, display: null },
    preference: null
  }
}

to

{ _id: null, link: null, display: null, preference: null }

So something like this

const { collection } = require('forest-express-sequelize');

collection('lawyers', {
  fields: [{
    field: 'flat PA',
    type: [{ _id: 'String', field1: 'String', fields2: 'String', inside2: 'String' }],
    get: (record) => {
      return record.PA.map(pa => ({
         // flatten the inner records
          _id: pa.value.inside._id,
          ...
        }))
    },
    set: (record, flatPa) => {
      record.PA = flatPa.map(item => ({
       // unflatten them
       value: { _id: flatPa._id, ...},
       inside2: flatPa.inside2,
      });

      return record;
    }
  }]
});

This should work, however, as I’m reading on the source code, you may have an issue for updates (I did not test the code).

If that happens, you will need to override the PUT /forest/lawyers/relationships/flatPA/:index route and implement the same logic than in the set handler there as well

2 Likes

Thanks for the help. Your explanation it is very clear. I will give it a try and then come with an update.
Update:

I have structured the flat object like this:
image
but according to your first comment, the result should not look like this

It depends on the widget that you choose

If your choose the JSON viewer / editor, you will be in the same situation than me when I tested

Unfortunately, I do not have the same options as you do.

Am I missing something ?

If you are using a mongo database, use forest-express-mongoose, if you are using a SQL database, then forest-express-sequelize.

So in your case it should be forest-express-mongoose everywhere.

About the missing options, I never saw that before.
Can you send me

  • a screenshot of chrome dev tools when you are in configuration page
  • the code of your smart field, so that I can check


The object in terminal having null values is alright, thats how its stored.
The 403 error should be ignored, its not related to Forest Admin. Thanks.

I’m going to have to create a sample project to try to replicate your issue.

Keep posted

1 Like

@anon39940173 Thanks for the help. At this moment I decided I will try to do it using smart actions instead of relationships.