Unable to access certain forest environments

Request:

curl 'https://woodpecker.dev.fini.city/forest/sessions' \
  -H 'Connection: keep-alive' \
  -H 'sec-ch-ua: ";Not A Brand";v="99", "Chromium";v="94"' \
  -H 'DNT: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'Content-Type: application/json; charset=utf-8' \
  -H 'Accept: */*' \
  -H 'Origin: https://app.forestadmin.com' \
  -H 'Sec-Fetch-Site: cross-site' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Referer: https://app.forestadmin.com/' \
  -H 'Accept-Language: en-US,en;q=0.9' \
  --data-raw '{"email":"andrew******@finicity.com","password":"*********","token":null,"twoFactorRegistration":false,"renderingId":"102811","projectId":"21447"}' \
  --compressed

Response

{"source":"7ea354a7daa8270914ba7f618fd1f283","code":3001,"status":400,"title":"UNKNOWN ERROR","message":"Missing required authentication headers","user_message":"Missing required authentication headers","tags":""}%                                                                                                                      

Please provide any relevant information about your setup.

  • Package Version: “forest-express”: “^9.2.5”,
    “forest-express-sequelize”: “^8.2.2”,
  • Express Version: express@8.3.1
  • Sequelize Version: sequelize@6.6.5
  • Database Dialect: mysql
  • Database Version: 5.7.12
  • Project Name: Woodpecker

Hello @Andrew_Prior,

Do you still encounter any issue with your backends ?

I see that you use /forest/sessions this route doesn’t exist anymore in forest-express-sequelize v8.

This system has been completely redesign using OIDC which brings a lot of security and improve the way we can add services like Single Sign On for example. To be simple it delegates authentication to the Forest Admin server, no more direct email/password exchanges with the agents (backend).

  • I would advice to use /forest/healthcheck to see if you backend runs properlly.

Let me know if it helps with your issue.

Kind regards,
Morgan

The issues connecting to our environments is intermittent. Yesterday they resolved ‘on their own’ after about 2-3 hours of downtime. We did not implement any changes, but suddenly we were just able to connect again.

To be clear, forest/sessions is a route that is being called by the FA client. We are not using that route for anything.

That is really interesting. We followed the migration guide when we updated to v8.*, I wonder if we missed a step or if it was not clear how to migrate authentication. I will look at the migration documentation again to see if there is something we can put in place to address this. If you have any documentation that you can share about this that would be helpful.

2 Likes

Maybe this part of the upgrade to v7 about authentication could be helpful?

1 Like

I wonder if this is what is happening

Without a static client id, authentication will fail whenever a user makes a request to a different instance than the one he logged into.

But we have assigned FOREST_CLIENT_ID for all of our environments, so it should not be happening.

Looking at our CORS config, we are allowing 'Forest-Context-Url' as a header. Could this be causing issues? In the docs, this header is not included with the allowedHeaders

Indeed that problematic. The new authentication route is /forest/authentication and uses authorisation header along with tokens to perform high standard security authentication.

We will find the culprit. We really need to understand what can cause this. Can you share your CORS definition ? (/app.js)

:warning: You only need a FOREST_CLIENT_ID if you use multiple instance (with load balancer and several instances for example) for the same environment. Also, do not use the same FOREST_CLIENT_ID across environment.

I’m looking forward to hearing from you.

Kind regards,
Morgan

It’s a v8 change. Every collection calls (CRUD operations) to your agent will now be performed with a new header called Forest-Context-Url . This header contains the current URL of the user performing requests. This can be handy if you need information on the context this user is working on.

Regards,
Morgan

We are using multiple instances with a load balancer in our stage and production environments. In our Dev, which we call Testing, we do not have multiple instances running. Could the presence of the FOREST_CLIENT_ID be causing issues?

For sure you don’t need it in this specific environment. Can you try again without this environment variable ?

Do the other environments also use /forest/sessions to connect to the agents ?

Regards,
Morgan

I have removed the env from our dev environment.

The other environments (staging and production) do not make a call to /forest/sessions but neither is our Testing/dev environment today.

Ok, perfect.

I let you the time to make sure that resolve the issue.:pray:

Sorry for the inconvenience. We should make the documentation more warning about the FOREST_CLIENT_ID usages.

Kind regards,
Morgan

For now, since we are not experiencing issues, and since I have removed the variable, I am going to consider this issues resolved. If it comes up again, we can refer to this thread for context.

Thanks for all your help.

2 Likes

Your welcome. :pray:

Let us know.

Regards,
Morgan

Update: We have identified the potential root cause, which was that we had a canary deployment that we were unaware of running a different version. We will consider this problem solved and return here if otherwise.

Hello again, @morganperre

I am on @Andrew_Prior’s team also working on this project. This problem has resurfaced for us again.

What is interesting, is that after the dev environment is restarted after we push changes, the problem seems to go away and /forest/sessions is not being called. However within around 30 minutes, the issue comes back and a failed attempt is made to /forest/sessions

One thing to note: our dev environment is named TESTING_V2. This is also the only environment that we are experiencing this with.

Hey @rwinz,

Interesting, how this is done on your side ? You you use custom code in your CD to make this deployment ?

I want to share some informations with you. Here’s the latest forest-admin-schema meta for the remote environment TESTING_V2

  • 09/11/2021 00:31:32 (UTC) liana: “forest-express-sequelize” liana_version: “8.4.7”
  • 09/11/2021 01:12:08 (UTC) liana: “forest-express-sequelize” liana_version: “6.7.11
  • 09/11/2021 19:25:43 (UTC) liana: “forest-express-sequelize” liana_version: “8.4.7”
  • 09/11/2021 19:36:20 (UTC) liana: “forest-express-sequelize” liana_version: “6.7.11
  • 09/11/2021 20:08:34 (UTC) liana: “forest-express-sequelize” liana_version: “8.4.7”

As you can see sometime the liana version is 6.7.11 which indicates that an old version of forest-express-sequelize is running on this deployed environment. (Leading to the issue since v6 doesn’t use the same authentication mechanism)

I’m wondering if you have a routine that can cause automatic downgrade of forest-express-sequelize.
Do you use a package manager as yarn to manage forest-express-sequelize version ?

Last thing, your development environment local-varnish match version 6.7.11. Can it collides with your canary deployment and change what is deployed on TESTING_V2 ?

I hope those informations will help you find any issue on your side.

Kind regards,
Morgan

1 Like