Upgraded forest-express from 6.1.0 --> 6.1.2, now getting getting the "Cannot reach your database" error on production

[austinrupie]

Expected behavior

Forest to connect to our production service.

Actual behavior

I see requests coming in from forest’s portal, but there is no response from our server running the app. See logs below to view some example partially served requests. No errors show up on the serverside, either. The only errors are on app.forest.

Failure Logs

Here are the logs from the server. Oddly, there are no errors – but you can see that none of the requests are responding.

17:17:32 0|index  | Fetching app secret... prod/rupie/db
17:17:32 0|index  | info Secret prod/rupie/db fetch success!
17:17:35 0|index  | Your application is listening on port 8085.
17:17:36 0|index  | Your admin panel is available here: https://app.forestadmin.com/projects
17:18:38 0|index  | POST /forest/stats/Job - - - - ms
17:18:38 0|index  | POST /forest/stats/Gig - - - - ms
17:18:38 0|index  | POST /forest/stats/Gig - - - - ms
17:18:38 0|index  | POST /forest/stats/User - - - - ms
17:18:38 0|index  | POST /forest/stats/Team - - - - ms
17:18:38 0|index  | POST /forest/stats/User - - - - ms
17:18:38 0|index  | POST /forest/stats/Job - - - - ms
17:18:38 0|index  | POST /forest/stats/Team - - - - ms
17:19:08 0|index  | OPTIONS /forest/Gig?fields%5BGig%5D=id%2Cname%2Cstatus%2COwner%2Cproject%2CshortDescription%2CnegotiableFields%2CminWorkerCount%2CmaxWorkerCount%2Cdescription%2CupdatedAt%2CcreatedAt%2CdeletedAt%2CtermsId&fields%5BOwner%5D=username&fields%5Bproject%5D=id&page%5Bnumber%5D=1&page%5Bsize%5D=15&searchExtended=0&sort=-id&timezone=America%2FLos_Angeles 204 0 - 0.582 ms
17:19:08 0|index  | OPTIONS /forest/Gig/count?fields%5BGig%5D=id%2Cname%2Cstatus%2COwner%2Cproject%2CshortDescription%2CnegotiableFields%2CminWorkerCount%2CmaxWorkerCount%2Cdescription%2CupdatedAt%2CcreatedAt%2CdeletedAt%2CtermsId&fields%5BOwner%5D=username&fields%5Bproject%5D=id&searchExtended=0&timezone=America%2FLos_Angeles 204 0 - 0.179 ms
17:19:11 0|index  | GET /forest/Gig?fields%5BGig%5D=id%2Cname%2Cstatus%2COwner%2Cproject%2CshortDescription%2CnegotiableFields%2CminWorkerCount%2CmaxWorkerCount%2Cdescription%2CupdatedAt%2CcreatedAt%2CdeletedAt%2CtermsId&fields%5BOwner%5D=username&fields%5Bproject%5D=id&page%5Bnumber%5D=1&page%5Bsize%5D=15&searchExtended=0&sort=-id&timezone=America%2FLos_Angeles - - - - ms
17:19:11 0|index  | GET /forest/Gig/count?fields%5BGig%5D=id%2Cname%2Cstatus%2COwner%2Cproject%2CshortDescription%2CnegotiableFields%2CminWorkerCount%2CmaxWorkerCount%2Cdescription%2CupdatedAt%2CcreatedAt%2CdeletedAt%2CtermsId&fields%5BOwner%5D=username&fields%5Bproject%5D=id&searchExtended=0&timezone=America%2FLos_Angeles - - - - ms
17:19:16 0|index  | OPTIONS /forest/SubscriptionPlan?fields%5BSubscriptionPlan%5D=createdAt%2CdeletedAt%2Cid%2Cmeta%2Cname%2CplanId%2CplanValues%2Cstatus%2CtrialDays%2CupdatedAt&page%5Bnumber%5D=1&page%5Bsize%5D=15&searchExtended=0&sort=-id&timezone=America%2FLos_Angeles 204 0 - 0.237 ms
17:19:16 0|index  | OPTIONS /forest/SubscriptionPlan/count?fields%5BSubscriptionPlan%5D=createdAt%2CdeletedAt%2Cid%2Cmeta%2Cname%2CplanId%2CplanValues%2Cstatus%2CtrialDays%2CupdatedAt&searchExtended=0&timezone=America%2FLos_Angeles 204 0 - 0.175 ms
17:19:19 0|index  | GET /forest/SubscriptionPlan?fields%5BSubscriptionPlan%5D=createdAt%2CdeletedAt%2Cid%2Cmeta%2Cname%2CplanId%2CplanValues%2Cstatus%2CtrialDays%2CupdatedAt&page%5Bnumber%5D=1&page%5Bsize%5D=15&searchExtended=0&sort=-id&timezone=America%2FLos_Angeles - - - - ms
17:19:19 0|index  | GET /forest/SubscriptionPlan/count?fields%5BSubscriptionPlan%5D=createdAt%2CdeletedAt%2Cid%2Cmeta%2Cname%2CplanId%2CplanValues%2Cstatus%2CtrialDays%2CupdatedAt&searchExtended=0&timezone=America%2FLos_Angeles - - - - ms
17:19:32 0|index  | POST /forest/stats/Job - - - - ms
17:19:32 0|index  | POST /forest/stats/Team - - - - ms
17:19:32 0|index  | POST /forest/stats/Gig - - - - ms
17:19:32 0|index  | POST /forest/stats/Team - - - - ms
17:19:32 0|index  | POST /forest/stats/User - - - - ms
17:19:32 0|index  | POST /forest/stats/User - - - - ms
17:19:32 0|index  | POST /forest/stats/Gig - - - - ms
17:19:32 0|index  | POST /forest/stats/Job - - - - ms
17:21:02 0|index  | GET /forest/Gig?fields%5BGig%5D=id%2Cname%2Cstatus%2COwner%2Cproject%2CshortDescription%2CnegotiableFields%2CminWorkerCount%2CmaxWorkerCount%2Cdescription%2CupdatedAt%2CcreatedAt%2CdeletedAt%2CtermsId&fields%5BOwner%5D=username&fields%5Bproject%5D=id&page%5Bnumber%5D=1&page%5Bsize%5D=15&searchExtended=0&sort=-id&timezone=America%2FLos_Angeles - - - - ms
17:21:02 0|index  | GET /forest/Gig/count?fields%5BGig%5D=id%2Cname%2Cstatus%2COwner%2Cproject%2CshortDescription%2CnegotiableFields%2CminWorkerCount%2CmaxWorkerCount%2Cdescription%2CupdatedAt%2CcreatedAt%2CdeletedAt%2CtermsId&fields%5BOwner%5D=username&fields%5Bproject%5D=id&searchExtended=0&timezone=America%2FLos_Angeles - - - - ms

{
  "name": "admin",
  "version": "0.0.1-beta.34",
  "private": true,
  "dependencies": {
    "@babel/cli": "^7.8.4",
    "@babel/core": "^7.9.0",
    "@babel/node": "^7.8.4",
    "aws-sdk": "^2.653.0",
    "chalk": "~3.0.0",
    "cookie-parser": "1.4.5",
    "debug": "~4.1.1",
    "dotenv-safe": "^8.2.0",
    "express": "~4.17.1",
    "forest-express-sequelize": "^6.1.2",
    "intercom-client": "^2.11.0",
    "morgan": "^1.10.0",
    "node-json2html": "^1.4.0",
    "node-quill-converter": "^0.3.3",
    "pg": "~7.18.2",
    "pm2": "^4.2.3",
    "require-all": "^3.0.0",
    "sequelize": "^5.21.5"
  }
}

import sequelize from 'models';
import Liana from 'forest-express-sequelize';
import requireAll from 'require-all';
const intercomClient = require('intercom-client');
const chalk = require('chalk');
const path = require('path');
module.exports = async function (app) {
  const liana = await Liana.init({
    modelsDir: path.join(__dirname, '../models'),
    configDir: path.join(__dirname, '../forest'),
    envSecret: process.env.FOREST_ENV_SECRET,
    authSecret: process.env.FOREST_AUTH_SECRET,
    sequelize,
    integrations: {
      intercom: {
        accessToken: process.env.INTERCOM_ACCESS_TOKEN,
        intercom: intercomClient,
        mapping: ['User.email'],
      },
    },
  });
  app.use(liana);
  console.log(chalk.cyan('Your admin panel is available here: https://app.forestadmin.com/projects'));
};

Context

Please provide any relevant information about your setup.

• Package Version: 6.1.2
• Express Version: 4.17.1
• Sequelize Version: ~5.21.5
• Database Dialect: pg
• Database Version: 12

[austinrupie]

Can I get some help, please?

[jeffladiray]

Hi @austinrupie !
I’ve just upgraded one of my project to forest-express-sequelize: 6.1.2 and I’m not able to reproduce your issue.
Could you provide me more infos about the problem you are experiencing ? (Eg. Network tabs of your browser, browser console log?)

According to the forest-express-sequelize changelog 6.1.0 to 6.1.2, there was not a lot of changes between theses two versions.
Two more questions here, so I might be able to reproduce your issue :

• Does a revert to 6.1.0 still works on your end?
• Are you using Enums in your project ?

[ruphy]

@austinrupie

Can I get some help, please?

Hi @austinrupie, while the entire team indeed tries their best to monitor the forum and provide timely support, and there are other community members on this forum, please be patient in regard to time differences: most of the team is living in Central Europe, and your posts happened to line up almost perfectly with Europe’s night.

[austinrupie]

Got it, thanks for explaining the time differential.
Is this also the right place for premium support? We’re a business account

[austinrupie]

Is there migration information around the upgrade?

We do indeed use Enums in our project.

Here is an example:

    type: {
      type: DataTypes.ENUM(
        'AVATAR',
        'BANNER',
        'MEDIA',
      ),
      field: 'Type',
      defaultValue: Sequelize.literal('MEDIA'),
    },

The revert does indeed work.

[austinrupie]

Is there a way to further debug Liana?

[Yoad_Snapir]

We sometimes find the ability to override routes helpful for debugging.
After all - it’s just an Express app.
You can for example override the Gig “List” route and print out what was the request and the response (or error).

Also, use dev tools on the Admin app within the browser - we learn a lot about what’s broken by investigating the network activity and the console logs.

Last, and this is advanced move I would say (Rarely needed) is to add traces to the Liana module itself. It’s open source, you can fork it and add what ever traces you want to see exactly what happens from the moment a request is passed down to the handler and until it should be executed against the DB. You would have to get to know Liana’s code better - this is why I would start with the first two approaches.

To be able to help you, like Forest’s team I would love to get info about what activity you see on the browser side. (Can you get the HAR of the first load? Can you get the console logs?).

G’luck

[austinrupie]

x

[austinrupie]

Yeah, a lot of errors int the browser console

[Yoad_Snapir]

Austin,
I suspect this is all around the CORS errors.
I recall Liana 6 has changed where the CORS middleware is being mounted and who’s responsibility is to do that (it used to be liana express, but I believe now it’s on the developer side).
Another hint that his is the problem is that you said it’s on Production and not your local env.

Can you find in your code something similar to this:

app.use(
  cors({
    origin: /forestadmin\.com$/,
    allowedHeaders: ['Authorization', 'X-Requested-With', 'Content-Type'],
    maxAge: 86400, // NOTICE: 1 day
    credentials: true,
  })
);

This should come before you mount the forest middleware which looks like this:

app.use('/forest', (request, response, next) => {
...

Also not that to use cors you need to require it first:

const cors = require('cors');

[austinrupie]

Hey Yoad, yeah – got past the CORs issue. I think it’s caused by a timeout somewhere.

[austinrupie]

Here is the browser.

Also here is my entire app.js file:

import configureEnv from 'utils/envUtils';

import {
  ensureAuthenticated,
  PUBLIC_ROUTES,
} from 'forest-express-sequelize';

const express = require('express');
const requireAll = require('require-all');
const path = require('path');
const cookieParser = require('cookie-parser');
const bodyParser = require('body-parser');
const cors = require('cors');
const jwt = require('express-jwt');
const morgan = require('morgan');

configureEnv();

// eslint-disable-next-line
let serverResponse;

// IIFE to kickstart
export default (async () => {
  const app = express();

  app.use(morgan('tiny'));
  app.use(bodyParser.json());
  app.use(bodyParser.urlencoded({ extended: false }));
  app.use(cookieParser());
  app.use(express.static(path.join(__dirname, 'public')));

  let allowedOrigins = [
    /\.forestadmin\.com$/, '*'
  ];

  if (process.env.CORS_ORIGINS) {
    allowedOrigins = allowedOrigins.concat(process.env.CORS_ORIGINS.split(','));
  }

  app.use(cors({
    origin: allowedOrigins,
    allowedHeaders: [
      'Authorization', 'X-Requested-With', 'Content-Type'
    ],
    maxAge: 86400, // NOTICE: 1 day
    credentials: true,
  }));

  app.use(jwt({
    secret: process.env.FOREST_AUTH_SECRET,
    credentialsRequired: false,
  }));

  app.use('/forest', (request, response, next) => {
    if (PUBLIC_ROUTES.includes(request.url)) {
      return next();
    }
    return ensureAuthenticated(request, response, next);
  });

  requireAll({
    dirname: path.join(__dirname, 'routes'),
    recursive: true,
    resolve: Module => app.use('/forest', Module),
  });

  requireAll({
    dirname: path.join(__dirname, 'middleware'),
    recursive: true,
    resolve: Module => new Module(app),
  });

  serverResponse = app;
  return app;
})();


export { serverResponse };

[austinrupie]

btw, all of this works perfectly fine in dev. I forgot to mention that. Only in prod is it breaking like this.

[austinrupie]

also getting this warning on dev

[Yoad_Snapir]

I am not sure, But the Promise warnings do give a hint.
I know that Liana moved to get mount the middleware in an async way not long ago to allow Liana.init to return a Promise. have you made that change in your code?

They asked to do a change like this:

-     resolve: Module => new Module.default(app),
+     resolve: Module => Module.default(app),

when requiring the “middleware” and a change like this:

-   app.use(Liana.init({
+   app.use(await Liana.init({

when calling Liana.init.

But this would not have worked in dev as well - that’s the puzzling thing here!
Your app.js clearly shows that you did not make that change - which should be madatory!
See here

Any difference in Node.js versions between dev and Prod? Are you sure all modules are the same between envs?

[austinrupie]

Yes, made those changes and the timeout issue persists. We’re running Node 14.x, is that a problem?

[Yoad_Snapir]

Shouldn’t be!
Can you share your updated app.js?
Those timeouts just feel like the requests and not terminated correcelty in express. The middleware somehow is not working on Production env as it should.
Are we 100% sure the DB connection is working as expected? Could it be that the request is “sitting” on a DB connection which cannot get through?

Also I tried and compare 6.1.0 and 6.1.2 here.
I dont think the “GLUE” character could be the problem - but the new handling of enums might introduce a bug which was not caught.
If that is the case, the models might not be properly setup and thus DB access might be impaired.
If this helps at all.

[ruphy]

@austinrupie

Is this also the right place for premium support? We’re a business account

Hi @austinrupie,
yes, this is also the place for premium support. We are maintaining the same level of premium support we previously had on Slack (i.e. a minimum of one developer dedicated solely to support for every business day), expanding it to all our customers, including the ones on a free plan, and enriching it with more features.

For example, moving support on to a forum allows for clearer conversations, gives you the possibility of saving and accessing your answers later, and adds the ability of searching in already solved problems.

This is the reason why we encourage everyone, even Enterprise users (who have a dedicated slack channel), to post here for help.
We’ll soon be publishing a blog post with more details about this migration.

[austinrupie]

Ok, figured out the issue.

Was using Node 14.2.0-alpine3.10 dockerfile – and for some reason Forest doesn’t like it. No errors or anything. Had to go back to 13.x and it worked;