Unexpected rollback schema in production

Help me!
1

We have been witnessing some strange behaviour in our production.
The schema seems to rollback to an old version with no reason some time after deploying to production.

Some more context :
We have 3 remote environments : production, staging and develop ( staging is based on production and develop on staging). Every time we do a deployment on one of those 3, the schema is correctly reflected (we do include the .forestadmin-schema.json file in our source control and deployments). However, after a while, the production environment seems to reflect an old version (from months ago) of our schema; some old collections are reappearing and the new ones are missing.
This happens only on production, few hours only after a deployment.
Also, nothing shows up on our production logs

2

Hi @AugustinBarbe ,

Does the problem appeared recently ?
I will need further information in order to help you.
Can you please fill the information below :

Context

  • Project name:
  • Team name:
  • Environment name:
  • Agent type & version:
  • Recent changes made on your end if any:

Best regards,

Shohan

3

Hello @shohanr ,

It’s hard to tell when this problem occurred (we frequently ship into production).
Here are some more information :

  • Project name: Folde
  • Team name: Operations
  • Environment name: production
  • Agent type & version: forest_liana 7.4.5
4

In our side, the 3 last updates in production are dated from :

  • 15/11/2022 10:41:26
  • 11/11/2022 14:53:19
  • 10/11/2022 14:55:26
  1. Are these are all intended or not ?
  2. Do you know if it happens only on some of them ?
  3. It could happen that an old version is pushed because of your env_secret is the same on production and one of your development’s repository, could you also check that ?

Best regards,

Shohan

5

Regarding of today, I have made only one deployment to production around 10:41:26 so this checks out for today. However you don’t have more events on this day ? The schema has yet rolled back to the bad version (it was ok at least 2 hours after I deployed ).

I have just checked the config for each environment and env_secrets are on their right places.

Would it be possible to have an extract of the ip adresses that are pushing schema updates ?

6

We have another event at 15/11/2022 16:39:44.
For the moment, we do not have found the unintended update, so it might not lead us any further to share the ip list.

When an update appears to rollback your configuration, keep us updated and we will see what are the details of the release and it will be easier to verify the conformity of the update.

7

This one is unexpected !

8

FYI: I just double-checked and every update has a different IP address.

9

It’s normal, our infrastructure relies on a container based solution. However I’m curious to know the one associated to the deployment done on 15/11/2022 16:39:44.

11

The problem was that an old instance run by a 3rd-party contractor on Heroku had the same env_secret as the production one.
Problem solved.

2 Likes