Loosing data with properties not in the database schema

[Aymeric]

Hello,

We’ve been testing ForestAdmin with a MongoDB database since last week. We have some problems with data that isn’t in the mongoose model of the database.

Observed behavior

If I add a new property to an object of my database, I should add it manually in the models of my forest admin backend.
It is not very practical and if I don’t add this property to the model, I can’t see it in ForestAdmin. Moreover, when I update my data with the ForestAdmin panel, the property even gets removed from the database (which can delete some data from the database !).

Example: if I add a property testProp in the profile object of my user, without putting it in the model file users.js, it gets deleted from the database when I update the profile object in the ForestAdmin panel.

I tried to set strict mode to false in my database schema, but the data of my smart fields get added in my database, at the root of the document and with the wrong key (see the ‘Viewer token duration (hours)’ property added by Forest) :

Problems

If the database model isn’t up-to-date, why does ForestAdmin delete the data that isn’t in the model? And without any warning?
A solution is using “strict: false” in the schema of the mongodb collection, but then why does ForestAdmin add unwanted fields (the smart field values) to my documents when I update them using the admin panel?

Currently, to update our database schema we found two solutions:

• Update it manually each time we modify the database structure, and push it into production
• Delete the old model files (existing files can’t be updated), use forest schema:update to scan the whole database and regenerate all files, rewrite our modifications, then push it into production

Both solutions can be time-wasting for our team and we wonder if there is an easier way to do that?

Context

• Project name: -
• Team name: -
• Environment name: all environments
• Agent type & version: forest-cli/2.6.3 with express/mongoose v8

[Sliman_Medini]

Hi @Aymeric,

Thank you for this valued feedback.

First of all, I’ve created a bug ticket in order to fix the data deletion inside the embedded document having properties not defined in the schema. Best regards for pointing this out. I am not able to give an ETA about this, but it should be studied in a short amount of time.

Secondly, I’ve also created a bug ticket in order to fix the problem of writing smart fields into the database when using strict mode.

Waiting for the first bug to be fixed, I am afraid you already point out the workarounds like you said to update schema rigorously.

Best regards

[Aymeric]

Hi @Sliman_Medini

Thanks for your quick answer.
Is-it possible to notify us when these problems are fixed ? We will wait for it.

Best regards

[Aymeric]

Hello @Sliman_Medini

I would like to know if it is possible to have an update on the status of these fixes ?

Best regards,
Aymeric

[Sliman_Medini]

Hello @Aymeric,

All my apologies for the lack of update.
The bug was returned as non-reproductible since the sub-document properties are updated one by one.
Do you have some route override on your setup?
What is your express/mongoose version? Last is 8.6.7

Best regards

[Aymeric]

Hello @Sliman_Medini ,

Thanks for your answer.
I don’t override any of the default routes created by ForestAdmin.
I was using express/mongoose version 8.6.5, I’ve updated to 8.6.7, but I still have these bugs. I will illustrate them:

• For the first bug, data deletion:

1. I add two properties to my mongodb document, that are not in the mongoose model used by ForestAdmin.
   Mongoose model of my document:
   
   

Data in my database:

The two properties testProp and display.otherTest are not in the model.

2. I open this profile on the ForestAdmin panel:
   The profile field is displayed like this, as it’s an object, and it’s editable.
   
   

You can note that the additional properties testProp and display.otherTest are not shown (and this is not a problem for us since they are not in the database model).

3. The bug happens when I edit the “profile” object: if I update display.someDisplaySetting, the other properties (testProp and display.otherTest) get deleted, without any warning or error message. Here is the result in the database when I do this modification:
   

Finally, as our database model may not be up-to-date every time (we choose MongoDB for its flexibility, without having to define a model), it may be a big issue to accidentally lose data like these.

• And then the second bug, only when I set “strict: false” in my database schema:

1. I have a smart field “Viewer token duration hours” linked to profile.viewerTokenDuration field:
   

   

2. If I update it, a new property is created in my database document, for each SmartField:
   

I hope my explanations will help you reproduce and solve these bugs. Feel free to email me if you want more details.
Thanks for your help.

Best regards

[Sliman_Medini]

Thank you,

We are looking at this.

Regards

[Aymeric]

Hello @Sliman_Medini

I would like to know if you have any news on these bugs ?

Best regards,
Aymeric

[Steve_Bunlon]

Hello @Aymeric

After some more investigation, it is the intended behaviours with it comes to JSON fields.

The best thing to do here, if you don’t want data that forest is not aware of to be overwritten, is to actually define them in your model. If having unwanted fields displayed in the UI is not something you can afford, you can flatten your field profile. Flattening this parent field will make any inner fields as regular fields, and interact with them in a more user friendly fashion, meaning that you will be able to hide them.

Let me past you the documentation around the flattener → Flatten nested fields (MongoDB) - Developer guide

Please, if you have any question, feel free to ask

Steve.