Collections shouldn't be created

Feature(s) impacted

Collections creation.

image

Observed behavior

Collections created on the left, shouldn’t exist… because they are not collections, but references ! It seems it creates a collection for each reference.

Expected behavior

Only collections should appear on the left panel.

Failure Logs

No logs.

Context

  • Project name: lereacteur-cfa
  • Database type: mongoose

Packages :

 "dependencies": {
    "@forestadmin/agent": "^1.9.0",
    "@forestadmin/datasource-mongoose": "^1.4.0",
    "axios": "^1.4.0",
    "dotenv": "^16.0.3",
    "express": "^4.18.2",
    "module-alias": "^2.2.2",
    "mongoose": "^7.2.0"
  }

Models :

image

Model Enterprise :

const mongoose = require("mongoose");

const schema = new mongoose.Schema(
  {
    name: String,
    description: String,
    logo: String,
    wttj: String,
    website: String,
    city: String,
    status: String,
    employees: [{ type: mongoose.Schema.Types.ObjectId, ref: "Employee" }],
    comments: [
      { type: mongoose.Schema.Types.ObjectId, ref: "CommentEnterprise" },
    ],
    networks: [{ name: String, url: String }],
  },
  {
    timestamps: true,
  }
);

module.exports = function (connection) {
  return connection.model("Enterprise", schema);
};

/models/index.js contains :

const fs = require("fs");
const path = require("path");

module.exports = function (connection) {
  const models = {};

  const array = fs
    .readdirSync(__dirname)
    .filter((file) => file !== "index.js" && file.endsWith(".js"));

  for (let index = 0; index < array.length; index++) {
    const file = array[index];
    const model = require(path.join(__dirname, file))(connection);
    const modelName = path.basename(file, ".js");
    models[modelName] = model;
  }

  return models;
};

Hi @XavierColombel,

As you are using our new nodejs-agent, this is the new default behavior of the mongoose-datasource.

You can easily switch back from the previous forest-express-mongoose behavior by passing flattenMode: 'legacy' to the mongoose datasource, as visible in this documentation.

Let me know if that helps.

1 Like

In fact, putting flattenMode: 'legacy' doesn’t switch back to the previous behavior. I mean, there was no “collection” created on the left side in the previous behavior (but references in related data were still visible), or I miss something?

And flattenMode: 'none' removes references in related data of the collection.

I think its a non-sense, look at the left side, now you have to identical collections created (the reference and the collection created though flattenMode: 'none') :

So choices are :

  • flattenMode: 'none' : you won’t see references in related data of the collection
  • flattenMode: 'legacy' : you will see references in related data of the collection (but not the original one you renamed) as new collections added on the left panel.

Did I misunderstand something?

Hi @XavierColombel

Jeff reached out to me so that I take over this thread.
I wrote most of the datasource-mongoose driver, so hopefully I can explain how it is supposed to work.

flattenMode: 'legacy' does not refer to how thing were with forest-express-mongoose, but to an earlier version of @forestadmin/datasource-mongoose (before this PR ).

It is there to ensure that we don’t introduce a breaking change for customers that started using agent-nodejs with mongo earlier than January (if that is not your case, you should not use it :slight_smile: )

Virtual collections are there by design, you can hide them in the UI.

The reason is that the Forest Admin UI is built around table views and chart builders that don’t play well with deeply nested data.

To work around that, in the new agent-nodejs we completely changed how we map mongoose collections to forest admin collections: depending on its structure, a single collection in mongo can be mapped to an arbitrary number of collections in forest admin.

This is the case when you choose flattenMode: 'auto' where we’ll map mongo collections to as many forestadmin collections as needed so that none of them contain nested fields or arrays.

Try using flattenMode: auto, and you will see that Enterprise.networks is no longer a field of Enterprise but a collection.

This would allow you for instance to create charts on the Enterprise Network collection, irrespective of which enterprise they belong to (and many other things).

That is because, in your model, you are using lists of ids (Enterprise.employees and Enterprise.comments).

If you had added an enterpriseId field to the Employee collection, the @forestadmin/datasource-mongoose would have mapped that relation to a one to many relationship and no virtual collection would have been created.

But with that structure, we have to suppose that this is a many to many relationships “just in case”: an employee can work for many enterprises, and enterprises have many employees.

Many to many relationships use an intermediary table: that’s the virtual collection that you are seeing. You can hide it on the GUI if you don’t need it.

You would have gotten the same result if you had additional metadata on the relationship.

// imaginary schema
const schema = new mongoose.Schema(
  {
    name: String,
    [...],
    employees: [
      {
        recruitedAt: Date,
        firedAt: Date,
        employeeId: type: mongoose.Schema.Types.ObjectId, ref: "Employee" },
      }
    ],
  }
);

When using flattenMode: 'none' you are telling us that you want none of that virtual collection thing.

This prevents all many to many relationships in mongo from working, and thus, removes the related data section.

Your probable next question should be: But then why did it work in forest-express without the virtual collection?

The answer is quite simple actually: forest-express-mongoose contained custom code that manually handles the array of ids use case.

Because the core package has gotten better at handling generic relationships, we did not port that code to the new system, and use the same code to handle many to many relationships in SQL databases and Mongo (or any other system really, we have drafts of connectors that work with elasticsearch, google drive, … that also use that code)

1 Like

In you case, I recommend that you use the following settings, and hide the extra collections in the left panel:

const dataSource = createMongooseDataSource(mongoose.connection, {
  flattenMode: 'manual',
  flattenOptions: {
    Enterprise: { asModels: ['employees', 'comments'] }
  }
});
1 Like

It’s still a bit confusing, honestly. I have to dig into configuration and do some more tests.

An another problem is, when you rename collections (references), the new name is not kept.

Look a it :

I’ve made tests, adding test before references names.

Relationships and collections can have different names.
You need to rename the relationships from the Fields panel in the settings of the collection that have them.