Project not found - URGENT

Feature(s) impacted

All the features

Observed behavior

I have noticed that our project is not loading in FA!!! Everyone trying to connect is getting an error, and when I try to access it (as admin), I see a page to setup the app from scratch,

All the company is blocked, given that we use FA across the whole company.

Expected behavior

Load the app.

Failure Logs

No error logs for the last 24 hours.

Context

  • Project: TymeshiftFA, ID 42335
  • Environment name: us1, staging, eu1, US2, DEVELOPMENT
  • Agent (forest package) name & version:
  • Database type: MySQL
  • Recent changes made on your end if any: None

envs

Screenshot 2023-05-26 at 09.38.18

We have the same problem. ALL environments not connecting.

Hi @Raphael_Neves,

Sorry to read that, we are investigating right now.
We’ll let you know as soon as possible.

Please follow our incident in the status page for update.

@Raphael_Neves, @Emil_Rasmussen,

We’ve identified and rollback a contribution.
Do you still have the issue right now?

Thanks for your help.

The app is running as expected, but considering this has triggered an internal incident and impacted our sales pipeline, I would like a bit more information about what happened than the information added to the status page, please.

We are having same problem. We are not back yet. Can you help us?

Thanks for your response @Raphael_Neves.

As for any declared incident in our status page, we will do a post-mortem with the technical team.

We are not perfectly sure, so far, what exactly happened but I can try to provide some elements

A contribution, at the heart of the platform, that aimed to make the projects schema management simpler and more efficient, was released this morning. This contribution was using a different part of our persistence layer.

Some scheduled jobs that run every night conflicted with this contribution by removing some mandatory data.

The projects impacted seem to be the one that never received a schema update in the last 6 months.

By rollbacking the release made this morning, the system got back to its ancien behavior, serving layouts as expected.

Although very technical, I hope it will help.

2 Likes

Hi @captain

Can you share your project information for investigation?

As much technical as better in this case. Thanks.
I will FUP on the public PM later on.