Upgrading to v7, MongoDB/Typescript documentation might be a bit obsolete

Hello,

I freshly updated my project from v4 to v7 (recreated a fresh project and reimported my code as it got way to hard xD)

However, as I prefer typescript, my models/routes are in typescript and… I have a bit of a problem…

Based on this documentation, regarding the models part: https://docs.forestadmin.com/woodshop/how-tos/translate-your-app-into-typescript#how-it-works

As we can see, you give an example that seems to be v5/v6:

const mongoose = require('mongoose');

const schema = mongoose.Schema({
  'firstName': String,
  'lastName': Date,
}, {
  timestamps: false,
});

module.exports = mongoose.model('users', schema, 'users');

This is easely translatable into typescript with the following:

import { Schema, Document, model} from 'mongoose';

interface IUser extends Document {
    firstName: string;
    lastName: string;
}

const schema = new Schema({
    'firstName': String,
    'lastName': Boolean,
}, {
    timestamps: false,
});

export default model<IUser>('users', schema, 'users');

However, in v7 (example reproducible by creating a fresh new project), the base model in js will look like so:

// This model was generated by Lumber. However, you remain in control of your models.
// Learn how here: https://docs.forestadmin.com/documentation/v/v6/reference-guide/models/enrich-your-models

module.exports = (mongoose, Mongoose) => {
  // This section contains the properties of your model, mapped to your collection's properties.
  // Learn more here: https://docs.forestadmin.com/documentation/v/v6/reference-guide/models/enrich-your-models#declaring-a-new-field-in-a-model
  const schema = Mongoose.Schema({
      'firstName': String,
      'lastName': Boolean,
  }, {
      timestamps: false,
  });

  return mongoose.model('users', schema, 'users');
};

And… if you translate it into typescript, from my point of view, it should give something like this:

// This model was generated by Lumber. However, you remain in control of your models.
// Learn how here: https://docs.forestadmin.com/documentation/v/v6/reference-guide/models/enrich-your-models
import {Document, Schema} from "mongoose";

export interface IUser extends Document {
    firstName: string,
    lastName: boolean,
}

// This section contains the properties of your model, mapped to your collection's properties.
// Learn more here: https://docs.forestadmin.com/documentation/v/v6/reference-guide/models/enrich-your-models#declaring-a-new-field-in-a-model
const schema = new Schema({
    'firstName': String,
    'lastName': Boolean,
}, {
    timestamps: false,
});

//// Not sure about this
// export const collection = mongoose.model<IUser>('users', schema, 'Users');

module.exports = (mongoose, Mongoose) => {
    return mongoose.model('users', schema, 'users');
};

And here the problème starts… if I want to get a record by ID, then new RecordGetter(...) will do the job. However, when I want to import/findOne it doesn’t work… Either the default exports doesn’t have .findOne(...) method, either I try to export my collection/model manually:

export const collection = mongoose.model<IUser>('users', schema, 'Users');

and then import it like import {collection as usersCollection} from '../models/users.ts' but then, I get:

(node:252) UnhandledPromiseRejectionWarning: TypeError: Cannot read property ‘findOne’ of undefined

I really have no idea how to deal with that situation… I think first that I am missing something, of course, but I do feel as a second point that the documentation is obosolete and doesn’t show a croncret working example for the v7 when trying to make advanced queries from routes like shown at this link: https://docs.forestadmin.com/documentation/reference-guide/routes/override-a-route#changing-forest-admins-behavior

Of course, assessors as RecordGetter, RecordUpdater, etc are good, but sometimes, updateMany, findOne, etc can do advanced stuff you know, so it’s good to keep their way of uses explained through your documentation :slight_smile:

Expected behavior

From my routes/users.ts, I want to (for some reason) query my collection (... from '../models/users.ts') using operation such as:

  • findOne(…)
  • updateMany(…)
  • etc
    instead of using RecordGetter, RecordRemover, etc

Actual behavior

I can’t find the way to access such operation from the default function exported or the collection itself using a Typescript approach

Failure Logs

  • When exporting the collection as above:
import {collection as usersCollection} from "../models/users";
// ...
function foo() : IUser {
    // ...
    usersCollection.findOne({...})...
}

(node:252) UnhandledPromiseRejectionWarning: TypeError: Cannot read property ‘findOne’ of undefined


  • When trying to access the default export’s findOne() method:
import * as users from "../models/users";
// ...
users.findOne(...)

TS2339: Property ‘findOne’ does not exist on type ‘typeof import(“C:/Workspace/…/models/users”)’.


const { users} = require('../models');
// ...
const recordGetter = new RecordGetter(users);

TS2305: Module ‘“…/models”’ has no exported member ‘users’.

Context

Please provide any relevant information about your setup.

package.json

{
  "name": "project-backoffice",
  "version": "0.0.1",
  "private": true,
  "scripts": {
    "start": "node ./dist/server.js",
    "legacy-start": "node server.js",
    "build": "tsc",
    "start-dev": "tsc-watch --project ./tsconfig.json --noClear --onSuccess \"nodemon --watch ./dist ./dist/server.js\""
  },
  "dependencies": {
    "@types/express": "^4.17.11",
    "@types/forest-express-mongoose": "^7.5.0",
    "@types/mongoose": "^5.10.5",
    "@types/node": "^15.3.0",
    "axios": "^0.21.1",
    "body-parser": "1.19.0",
    "chalk": "~4.1.1",
    "cookie-parser": "1.4.5",
    "cors": "2.8.5",
    "debug": "~4.3.1",
    "dotenv": "~9.0.2",
    "express": "~4.17.1",
    "express-jwt": "6.0.0",
    "forest-express-mongoose": "^7.6.0",
    "mongoose": "~5.12.9",
    "morgan": "1.10.0",
    "nodemon": "^2.0.7",
    "require-all": "^3.0.0",
    "tsc-watch": "^4.2.9",
    "typescript": "^4.2.4"
  }
}

PS: IMPORTANT NOTES

/models/index.js conversion into typescript has issues

File example: https://docs.forestadmin.com/documentation/how-tos/maintain/upgrade-notes-sql-mongodb/upgrade-to-v7#models-index

  • First, it can’t work as const db = {} doesn’t have any of objectMapping or connections defined, so the two of them must be added manually.
  • Second, theline .filter((file) => file.indexOf('.') !== 0 && file !== 'index.js') doesn’t work properly and loads .js.map files which creates some errors… The way to fix it is simply to replace the line by .filter((file) => !file.includes('.map') && file.indexOf('.') !== 0 && file !== 'index.js'

/models/*.js generation has empty schema

  • I don’t know if it’s the expected behavior, but, using latest NPM lumber version (as of today), when the models files gets created, they all have empty schema, is it normal? Only the timestamp part is present:
// This model was generated by Lumber. However, you remain in control of your models.
// Learn how here: https://docs.forestadmin.com/documentation/v/v6/reference-guide/models/enrich-your-models

module.exports = (mongoose, Mongoose) => {
  // This section contains the properties of your model, mapped to your collection's properties.
  // Learn more here: https://docs.forestadmin.com/documentation/v/v6/reference-guide/models/enrich-your-models#declaring-a-new-field-in-a-model
  const schema = Mongoose.Schema({
  }, {
    timestamps: false,
  });

  return mongoose.model('users', schema, 'users');
};

FOREST_AUTH_SECRET if I loose this variable, do I have a way to retrieve it?

  • I had an issue (fully my mistake) where I removed the project, but not the right one… As the .env file is not versionned… I had a little bit of a hard time retrieving it from another computer. When I recreated it manually, I was missing this auth secret and couldn’t find the way to get it from ForestAdmin dashboard

express-jwt version 6 support instead of v5.3.1 (at project generation)

PLEASE ADD FULL TYPESCRIPT GENERATION

I know, it’s a lot of work and you guys are doing really great, to be honest, I have a lot of respect for your work, your product, your reactivity etc. However, I big point which is missing is that you don’t have a full typescript version of your product and, even when you try to convert everything into typescript, you run into errors with stuff such as Liana, so please, put it on your roadmap, I am sure it would give you a lot of precious points!! :slight_smile:


Thank you so much and sorry for that big post!

Hope to hear from you soon! :slight_smile:

Best,

Max

Hi @Emixam23,

Thank you for reaching out and sharing your issue with us :slight_smile:

Regarding the usage of the RecordsGetter, it is used by us to add logic in queries such as handling properly the parameters, ensuring user rights to access data, etc. Bypassing it and implementing your own get would work but it’s not the preferred way. And unfortunately yes, we do not expose a getOne but it is still possible to do it through the getAll and returning the first element.

Also if you only want to make a get by id, forest-express-mongoose exposes a RecordGetter that has a get(recordId) method.

But in your case, when using the RecordGetter it seems that the models cannot be imported properly from the models folder. Could should share with me the content of the models/index.js please? :pray:

Thank you also for the feedback you provided on the fact that our generated code is not in Typescript. I do understand that some users, as yourself, consider TypeScript as the preferred way of developing in Javascript. But not all users are used to TypeScript and plain JavaScript still seems to us the best way to reach a wider range of developers (and not to mention that it would indeed require a lot of dev time to provide a fully supported TypeScript project generation).

However it’s important for us to keep track of those feedbacks to prioritize our future roadmap. I’ll definitely log it into our ProductBoard :+1:

Hello :slight_smile:

Alright then I am ok with what you said, but for some reason, I sometimes need to “getOne” but not based on the document ID. I actually query another parameter…

Also, for the update, I can also update using record updater maybe… I don’t know I need to check tbh… I just felt like it was easier to use directly mongoose but I can definetly try RecordUpdater! :slight_smile:

/models/index.ts

import * as fs from 'fs';
import * as path from 'path';
import * as Mongoose from 'mongoose';

const databasesConfiguration = require('../config/databases');

const connections = {};
const db = {
    objectMapping: undefined,
    connections: undefined
};

databasesConfiguration.forEach((databaseInfo) => {
    const connection = Mongoose.createConnection(databaseInfo.connection.url, databaseInfo.connection.options);
    connections[databaseInfo.name] = connection;

    const modelsDir = databaseInfo.modelsDir || path.join(__dirname, databaseInfo.name);
    fs
        .readdirSync(modelsDir)
        .filter((file) => !file.includes('.map') && file.indexOf('.') !== 0 && file !== 'index.js')
        .forEach((file) => {
            try {
                const model = require(path.join(modelsDir, file))(connection, Mongoose);
                db[model.modelName] = model;
            } catch (error) {
                console.error(`Model "${file}" creation error: ${error}`);
            }
        });
});

db.objectMapping = Mongoose;
db.connections = connections;

module.exports = db;

There is my index as you asked :slight_smile:

Hope it helps, thanks for your feedback :slight_smile:

Max

Hello @Emixam23 :wave:

First of all, thanks for the very complete and useful feedback :raised_hands: This will really help to improve the Typescript ecosystem at Forest.

I have investigated on this topic, and indeed, the document to translate projects from js into ts is not compatible for v7 projects (especially for mongoose projects).

I’m writing this message just to keep you posted, I will write a brand new documentation for v7 projects to workaround every pains / blocking points you pointed out above. Sadly I can’t provide any information for now, I have just started the investigation.

I will answer on this thread as soon as I have more information on this topic to unlock your situation.

Sorry for the inconvenience,

Looking forward to get back to you with good news.

Steve.

2 Likes

Hey @Steve_Bunlon! Hope you’re doing great :slight_smile:

Thanks for your message. At first, I’d like to apologize, I thought I responded to you, sorry about it.

Second, I’d like to thank you for your answer and I am glad my feedback could lead you to some future updates :slight_smile:

Third, regarding the update of your documentation/product, when do you think you could make it up? (no pressure, but… We are kinda stuck and we don’t want our project to stop to work on June 30th…)

Feel free to keep me updated because if the project will for sure stop working on the 30th of June, I need to prepare myself :slight_smile:

Best!

Max

Hello @Emixam23,

We are currently actively working on fixing all the Typescript related issues on our project. I cannot give you an estimation for now, but I am pretty confident it will be before the 30th of June. :crossed_fingers:

Please stay assured that your project won’t stop working after the 30th of June.

Let me explain to you what could happen after June 30th.

Only the development environment could be impacted and only if you will upgrade your Chrome browser to version 92 (you can find some details here about the changes from Chome side). As those changes will only affect the access to localhost, the production environment won’t be impacted in any way. :tada:

To sum up, even if you don’t upgrade the liana by the 30th of June, your main operations should not suffer from the upgrade of the Chrome version.

In case we are not able to provide you with clear documentation about Typescript and forest-express-mongoose v7, we have an intermediary solution for you. As it’s not the recommended approach I will not publish it right now, let’s keep it for a worst-case scenario.

We’ll keep you posted about our progress in translating a v7 project into typescript. I will let this thread open so we can update you here.

Please let me know if you have any other questions or concerns.

Regards,
Olesya

1 Like

Hello @Emixam23 :wave:

I am glad to announce that the new Types implementation has just been released :tada:

To benefit from the new Types, you just have to upgrade your Liana to version 7.9.0. Yes, Types our now directly integrated in the forest-express-mongoose repository, you don’t need to install @types/forest-express-mongoose anymore :confetti_ball:

To help you with your migration, we also created a documentation to guide you as much as possible, please refer to this link.

Don’t hesitate to post any issue or frustration you encounter regarding Types, we will be glad to help you out and every feedback are appreciated.

I hope you will enjoy the new version !

Steve.

Hello :slight_smile:

Good work guys, keep up!

So, I’ve been working since you updated everything but haven’t got the time to answer…

I found some weird behavior, they even might be bugs… First, let me tell you that, if you create a new project with a MongoDB running locally (docker or background process) or if you use an Atlas Free Tier for your cluster, you can’t create a new project. For the atlas part, the error is clear, it’s related to the free plan. For the local part however, we get something like the following error (no matter if initiated from npm or docker) (sorry, it’s from my memories): connection.close() is not a function

Now, regarding the upgrade to v7, I have some bugs when trying to use functionalities such as RecordGetter(), I so wanted to create a new project to see if I had a wrong version of a certain package. Following this documentation you gave: https://docs.forestadmin.com/woodshop/how-tos/translate-your-project-into-typescript/v7/migrate-mongoose-files , when trying to use a RecordGetter, I have the following situation/error:

const recordCreator = new RecordCreator<IAuthor>(Authors);

TS2554: Expected 3 arguments, but got 1.
image

I have the same implementation as your doc :slight_smile:

// This model was generated by Lumber. However, you remain in control of your models.
// Learn how here: https://docs.forestadmin.com/documentation/v/v6/reference-guide/models/enrich-your-models
import {Document, Schema} from 'mongoose';

export interface IAuthor extends Document {
    created_at: number,
    email: string,
    first_name: string,
    forest_admin_id: string,
    image_url: string,
    last_name: string,
    pseudo: string,
    team: string,
    updated_at: number,
}

// This section contains the properties of your model, mapped to your collection's properties.
// Learn more here: https://docs.forestadmin.com/documentation/v/v6/reference-guide/models/enrich-your-models#declaring-a-new-field-in-a-model
const schema = new Schema<IAuthor>({
    'created_at': {type: Number, required: true},
    'email': {type: String, required: true},
    'first_name': {type: String, required: true},
    'forest_admin_id': {type: String, required: true},
    'image_url': {type: String, required: true},
    'last_name': {type: String, required: true},
    'pseudo': {type: String, required: true},
    'team': {type: String, required: true},
    'updated_at': {type: Number, required: true},
}, {
    timestamps: false,
});

export default schema;

Note: Unlike your doc however, I can’t do export schema, I get the following: TS1128: Declaration or statement expected.

In my index.ts, I do as your documentation but… yeah, I have issues with the RecordGetter (as well have the Updater, Remover, etc)

import * as Mongoose from 'mongoose';
import {model} from 'mongoose';
import databasesConfiguration from 'config/databases';
import authorSchema, {IAuthor} from './authors';

const connections: Record<string, Mongoose.Connection> = {};
const objectMapping = Mongoose;

const connection = Mongoose.createConnection(databasesConfiguration[0].connection.url, databasesConfiguration[0].connection.options);
connections[connection.name] = connection;

let Authors = model<IAuthor>('authors', authorSchema, 'Authors');

export {
    objectMapping,
    connections,
    /// ....
    Authors,
    // ...
};

If you want to see my package.json, I have the following:

{
  "name": "backoffice-forest-admin",
  "version": "0.0.1",
  "private": true,
  "scripts": {
    "start": "node ./dist/server.js",
    "build": "rm -rf ./dist && npm run compile && npm run cp-config",
    "dev": "tsc-watch --onSuccess \"node ./dist/server.js\"",
    "compile": "tsc",
    "lint": "eslint . -c .eslintrc.json --ext .ts",
    "cp-config": "cp .env ./dist/"
  },
  "dependencies": {
    "@types/express": "^4.17.13",
    "@types/mongoose": "^5.11.97",
    "@types/node": "^16.3.3",
    "axios": "^0.21.1",
    "body-parser": "^1.19.0",
    "chalk": "^4.1.1",
    "cookie-parser": "^1.4.5",
    "cors": "^2.8.5",
    "debug": "^4.3.2",
    "dotenv": "^10.0.0",
    "express": "^4.17.1",
    "express-jwt": "^6.0.0",
    "forest-express-mongoose": "^8.0.0",
    "mongoose": "^5.13.3",
    "morgan": "^1.10.0",
    "nodemon": "^2.0.12",
    "require-all": "^3.0.0",
    "tsc-watch": "^4.4.0",
    "typescript": "^4.3.5"
  }
}

Note: You can see that I removed the @types/forest-express-mongoose

I think it’s almost all, just a small note, I think it would be for the users to have a little bit more of information regarding the Migrate Mongoose files - Woodshop for old agent generation part of your documentation. You only show String value implementation which can be fine if only working with primitives such as Number/Boolean, but what if we work with array/document/objectID?

On my end, I have the following but I can’t really confirm that it work yet as my project doesn’t compile (related mostly to RecordGetter/RecordUpdater/etc)

Item

// This model was generated by Lumber. However, you remain in control of your models.
// Learn how here: https://docs.forestadmin.com/documentation/v/v6/reference-guide/models/enrich-your-models
import {Document, Schema, Types} from 'mongoose';
import {INote, schema as note} from './note';

export interface IItem extends Document {
    notes: [INote],
    author_id: Types.ObjectId
    related_post_ids: [Types.ObjectId],
}

// This section contains the properties of your model, mapped to your collection's properties.
// Learn more here: https://docs.forestadmin.com/documentation/v/v6/reference-guide/models/enrich-your-models#declaring-a-new-field-in-a-model
const schema = new Schema<IItem>({
    'notes': [note],
    'author_id': {type: Schema.Types.ObjectId, ref: 'authors', required: true},
    'related_post_ids': [{type: Schema.Types.ObjectId, ref: 'posts', required: true}],
   // ...
}, {
    timestamps: false,
});

export default schema;

I know that it’s tough to do case-to-case doc when trying to make it general, but I think it would be a good point to add arrays (document, number, string, etc) and types such as mongodb objectid :slight_smile:

Well, thanks again for your work and I hope my first feedbacks will help you!

Max

Hi @Emixam23,

Thank you for the update.

The documentation we provided you with is for version 7 of forest-express-mongoose.

I can see in your package.json the you are using the version 8 of the package

You are getting the correct error as we changed the way RecordGetter works. You can find more information about the new version in this migration note.

Let me know if it’s clearer or if you need more help :slight_smile:

Hey!

Oh… My bad on this one, thanks :stuck_out_tongue:

Before I close this issue, could you “confirm” the other issue I brought up? And the suggestions regarding the doc?

Amazing job by the way, you are so reactive, love it!

Max