Hi
I recently made about a thousand layout changes for our project jurata-2. Deployed it to the production environment. Last I checked yesterday, they were all still there.
Today I open FA and quite a few fields have lost their name and display/edit configuration. No deployment on our FA server, nor on the layout pipeline, was made on our end.
Can you please check on your end, if something went wrong? If so, is it possible to revert it?
Thanks.
Hello @David_Roegiers,
Thanks for your message 
We received a new schema version yesterday on your production environment (at 1am).
Can you confirm it was well intended?
Could you give a simple example of what field is affected and what configuration is missing?
Thanks.
Yes, well, yesterday Thursday at 1am, that was intended and it worked afterwards.
Last time I checked, it worked fine at some point yesterday evening, 9pm maybe.
The first time I saw the missing fields, was one hour ago.
Partial selection of broken fields:
clients->address@@@co
clients->address@@@postcode
clients->address@@@streetNr
legalProductOrders->paymentStatus
legalProductOrders->totalPrice
legalProductOrders->data@@@creditBexio
…
FYI
Before you flip a switch, I have 135 changes pending on a branch in the Development environment, which has the staging environment as an origin.
Would be cool if I didn’t end up losing those.
Copy that, I’m not flipping any switch.
I’ve checked the logs, it seems that your production rendering was indeed updated today at 8h38.
From what I can see, it’s a manual edition (but I still need to deep dive to understand it well).
Did you (or another user) do anything on the layout this morning around that time?
I checked with the team, no-one consciously made any changes.
I doubled checked my terminal to see if I didn’t accidentally call forest push since yesterday evening, but it’s not the case.
Strange.
Do your logs show which user triggered the update by any chance?
Judging from the logs, it seems that you’ve made some layout changes this morning around 8h38.
I can’t be sure what you did exactly, I’m seeing some visibility changes (of fields or collections).
But I don’t know if those match the break of some of your fields.
Is that plausible?
What do you mean by “broken” fields exactly?
Are they not configured anymore (back to default)? Or something else?
That’s very strange…
About your layout changes on your development branch, they should not be any issue.
Actually, your staging environment layout (which is based on production) should have those “broken” fields already, and same as your development branch (which is based on staging) but this one should have kept its own changes with no problem.
Can you confirm that all your environments have those broken fields configuration?
And that your development branch still has its own layout changes you made earlier on top of it?
Thanks.
Judging from the logs, it seems that you’ve made some layout changes this morning around 8h38.
It’s possible, that I made an individual layout change around that time, but I didn’t think so. To be honest, I am making so many changes this week, it’s really hard for me to keep track.
I can’t be sure what you did exactly, I’m seeing some visibility changes (of fields or collections). But I don’t know if those match the break of some of your fields. Is that plausible?
I am 100% certain, that many of the changes I am seeing now, I didn’t introduce myself.
What do you mean by “broken” fields exactly? Are they not configured anymore (back to default)? Or something else?
Well, they basically lost their configuration. The name is back to the default and also the edit/display settings.
Actually, your staging environment layout (which is based on production) should have those “broken” fields already, and same as your development branch (which is based on staging) but this one should have kept its own changes with no problem.
Ah, good to know, didn’t realize, that staging builds on production at all times.
Can you confirm that all your environments have those broken fields configuration?
Yes, it appears on all environments.
And that your development branch still has its own layout changes you made earlier on top of it?
Yes, they are still there.
I hope this clarifies some things.
Thanks for those clarifications.
The current situation is pretty clear on my mind now.
However, I don’t see how this could have happen without human intervention (but I surely believe you that you are not the origin of those changes).
I’m going to ask other experts on our Tech team to see if I’m missing something. I’ll let you know.
Now, about the lost configuration, I’m afraid we don’t have any solution to suggest here to revert it (it exists layout backups but they are activated only on demand for some paying projects…).
Okay, thanks for checking. I will start to manually rebuild it, so there is no more urgency from my end.
We’ll check deeper on our end.
But can you think of another instance of your agent with the production FOREST_ENV_SECRET that could be running sometimes and whose schema is not up to date?
That’s our best bet for now.
Hello @David_Roegiers,
After deepest investigation, we confirm that you have a ghost environment that is pushing an old schema from time to time (ip: 3.72.55.25).
Could you double check?
Thanks.
Hello @anon34731316
Thanks for sharing that IP address!
I believe to have figured out the issue. We had a deployment around that time, where a pod seems to have been respawning and falling back to an older revision, thereby pushing an older .forestadmin-schema.json file intermittently. Not sure how the impact works on the ForestAdmin layout, but I guess this seems to be a valid explanation and statically very unlikely, if it wasn’t the cause.
I apologize, to have made you investigate an issue, which our systems caused.
Nonetheless, this was a worthy experience, as now I understand, what can go wrong in the FA layout in case of a server revision fallback on our end.
Marking your comment as the Solution, hope that’s ok.
Hey @David_Roegiers,
Thanks for your answer.
No worries, always happy to help a client when something unexpected is detected!
Anyway, despite there was a “trouble” in your system, the experience could be improved. And this will be planed soon.
In the meantime, please be aware that if a field / collection / … is dropped from your schema, you will lose the UI configuration when add it again (and we won’t be able to retrieve the old UI configuration unless you are on a subscription that enables the layout backup)…
Cheers!