Mongo: Custom schema type support on _id field

Hello,

Feature(s) impacted

‘Get a record’ route, using the recordId, and possibly other routes using recordIds (not tested).

Observed behavior

We are using Forest Admin with a MongoDB database, and in some collections the ‘_id’ field identifying the documents are of two types : ‘String’ or ‘Mongo.Schema.Types.ObjectId’.

Initially, the mongoose model used by Forest admin was generated with a ‘_id’ field of type String, it works well to display all records with a String id, but if it’s an ‘ObjectId’, we have the following error:

I tried to use a custom schema type (mongo doc) on my _id field to take both String and ObjectId types into account (I based my code on this stackoverflow topic), but the result is still a “cannot be found” message.

Here is the code I’m using:

class StringOrObjectId extends Mongoose.SchemaType {
  constructor(key, options) {
    super(key, options, 'StringOrObjectId');
  }

  convertToObjectId(v) {...}

  convertToString(v) {...}

  cast(val) {
    console.log("Testing for schema ",val)
    const objectIdVal = this.convertToObjectId(val);
    if (objectIdVal) return objectIdVal;

    const stringVal = this.convertToString(val)
    if (stringVal) return stringVal;

    throw new Error('StringOrObjectId: ' + val +
        ' Nor string nor ObjectId');
  }
}

Mongoose.Schema.Types.StringOrObjectId = StringOrObjectId;

To debug the problem, I use a custom route to get a record, using a RecordGetter (forest admin doc).
Before calling recordGetter.get(recordId), I cast ‘recordId’ to its type, String or ObjectId.
I enabled them Mongoose debug so we can see the requests done by Forest:

  • With a String id:
Mongoose: myCollection.aggregate([ { '$match': { '$and': [ { _id: 'yoG9PXegwDgBGnjWc' } ] } }, { '$group': { _id: null, count: { '$sum': 1 } } }], {})
Mongoose: myCollection.findOne({ _id: 'yoG9PXegwDgBGnjWc' }, { projection: {} })
GET /forest/myCollection/yoG9PXegwDgBGnjWc?timezone=Europe%2FParis 200 144 - 523.165 ms
  • With an Object id:
Mongoose: myCollection.aggregate([ { '$match': { '$and': [ { _id: '627cc11fddf7e1c05344d70b' } ] } }, { '$group': { _id: null, count: { '$sum': 1 } } }], {})
GET /forest/myCollection/627cc11fddf7e1c05344d70b?timezone=Europe%2FParis 404 113 - 106.366 ms

I conclude that the aggregate call fails to find the right record, because the ObjectId that I give to recordGetter.get(recordId) is cast back to a String, so mongoose doesn’t find it. If I manually execute the aggregate request with an ObjectId, the requested record is found.
Note that if I use an ObjectId in my Schema definition ('_id': Mongoose.Schema.Types.ObjectId), the aggregate request is correct (the ObjectId isn’t casted to String) and the record is correctly loaded in ForestAdmin (but all records with a String ‘_id’ fail to load):

Mongoose: myCollection.aggregate([ { '$match': { '$and': [ { _id: 627cc11fddf7e1c05344d70b } ] } }, { '$group': { _id: null, count: { '$sum': 1 } } }], {})
Mongoose: myCollection.findOne({ _id: ObjectId("627cc11fddf7e1c05344d70b") }, { projection: {} })
GET /forest/myCollection/627cc11fddf7e1c05344d70b?timezone=Europe%2FParis 304 - - 95.351 ms

Expected behavior

Being able to use a custom schema type for the ‘_id’ field, and prevent RecordGetter.get(recordId) from casting the ObjectId given in parameter to a String object.

Context

  • Project name: -
  • Team name: -
  • Environment name: all environments
  • Agent type & version: forest-cli/2.6.3 with express/mongoose v8.6.7

Thanks for your help

Hello @Aymeric

I’ll set up a test project and put a couple breakpoints to see where the id is being casted and come back to you

Hello @romaing

I come back to you to know if you have any news of these tests ?

Regards,
Aymeric

Yes.

Sorry for the delay

I’ve been investigated and custom types are definitively not supported: the code that converts forest-admin filters to mongo $match has a switch case which list native types, and nothing else.

I wrote a simple patch that looks to fix the issue which you are having (at least on my test project), but I can’t ensure that you won’t have other issues on other parts of the app.

Here is the patch: forest-express-mongoose+8.6.9.patch - Google Drive

diff --git a/node_modules/forest-express-mongoose/dist/services/filters-parser.js b/node_modules/forest-express-mongoose/dist/services/filters-parser.js
index cd7a2d1..76bccab 100644
--- a/node_modules/forest-express-mongoose/dist/services/filters-parser.js
+++ b/node_modules/forest-express-mongoose/dist/services/filters-parser.js
@@ -189,6 +189,7 @@ function FiltersParser(model, timezone, options) {
         return parseObjectId;
 
       default:
+        if (type.prototype.cast) return value => type.prototype.cast(value);
         return parseOther;
     }
   };

To apply it automatically on your project your can use patch-package - npm

We’ll be releasing a new version of NodeJS agents shortly, which should address this issue, among many others

Hello @romaing

Thanks for your answer. I applied the patch and it seems to have fixed the problems, I’m now able to open all of my records no matter the type of their _id.

But it created another minor issue :
If I use my custom StringOrObjectId type on my collection that have two SmartActions based on two segments (Active and Inactive users), when I open any record of that collection, an error is print in the console:

GET /forest/users/MwEM2fAQow8Nn.....?timezone=Europe%2FParis 304 - - 501.698 ms
[forest] 🌳🌳🌳  Unexpected error: $or/$and/$nor entries need to be full objects
{
  "ok": 0,
  "code": 2,
  "codeName": "BadValue",
  "$clusterTime": {
    "clusterTime": "7099741782518267905",
    "signature": {
      "hash": "PNFVX68MUnL13xxpsMXzkDBP5t8=",
      "keyId": "7051609406881923077"
    }
  },
  "operationTime": "7099741782518267905",
  "name": "MongoError",
  "stack": "MongoError: $or/$and/$nor entries need to be full objects\n    at MessageStream.messageHandler (----/TestBasePreProd/node_modules/mongodb/lib/cmap/connection.js:299:20)\n    at MessageStream.emit (node:events:526:28)\n    at processIncomingData (----/TestBasePreProd/node_modules/mongodb/lib/cmap/message_stream.js:144:12)\n    at MessageStream._write (----/TestBasePreProd/node_modules/mongodb/lib/cmap/message_stream.js:42:5)\n    at writeOrBuffer (node:internal/streams/writable:389:12)\n    at _write (node:internal/streams/writable:330:10)\n    at MessageStream.Writable.write (node:internal/streams/writable:334:10)\n    at TLSSocket.ondata (node:internal/streams/readable:754:22)\n    at TLSSocket.emit (node:events:526:28)\n    at addChunk (node:internal/streams/readable:315:12)\n    at readableAddChunk (node:internal/streams/readable:289:9)\n    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)\n    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)"
}

And the two SmartActions become unavailable:

You can note that the two other SmartActions (Generate …), not based on the segments, are still available. Also, if I try to use “Activate user” or “Disable user” from the page listing all records of the collections, they are available and working without any error in the console.

I have these problem only if I use a custom type on the _id field of my records, if I use the String type, the SmartActions work fine. For the others collections, without such SmartActions everything if working fine too, without any error.

Regards,
Aymeric

Hello @romaing

I allow myself to relaunch you on this bug, which is blocking for us, do you have any news ?

Regards,
Aymeric

As said before, mongoose custom types are not supported by the agent.

The error is due to the frontend breaking because there is an unknown “ObjectIdOrString” type in the schema.

Here is a new version of the patch that will fix that issue

However, keep in mind that other issues will happen for sure at some point and you would be better off with not having a mixed type primary key in your models (for forest admin usage at least).

diff --git a/node_modules/forest-express-mongoose/dist/adapters/mongoose.js b/node_modules/forest-express-mongoose/dist/adapters/mongoose.js
index f1048a0..56061a0 100644
--- a/node_modules/forest-express-mongoose/dist/adapters/mongoose.js
+++ b/node_modules/forest-express-mongoose/dist/adapters/mongoose.js
@@ -203,7 +203,7 @@ module.exports = /*#__PURE__*/function () {
                 return null;
               } // Deal with primitive type
 
-
+              if (fieldInfo.instance === 'StringOrObjectId') return 'String'
               return fieldInfo.instance || fieldInfo.options && getTypeFromNative(fieldInfo.options.type) || null;
             };
 
diff --git a/node_modules/forest-express-mongoose/dist/services/filters-parser.js b/node_modules/forest-express-mongoose/dist/services/filters-parser.js
index cd7a2d1..76bccab 100644
--- a/node_modules/forest-express-mongoose/dist/services/filters-parser.js
+++ b/node_modules/forest-express-mongoose/dist/services/filters-parser.js
@@ -189,6 +189,7 @@ function FiltersParser(model, timezone, options) {
         return parseObjectId;
 
       default:
+        if (type.prototype.cast) return value => type.prototype.cast(value);
         return parseOther;
     }
   };