EMERGENCY: Can't Authenticate/Log into Staging or Production Environments

Hi there!

When anyone on our team troes to log into our production or staging environment, they currently get a 403 error on the endpoint /forest/authentication/callback?code=… but I can access my development environment. I wanted to check if there was anything on your end you’re aware of that could have caused this? I looked in my server logs but didn’t see any relevant errors off first glance.

Thanks!

Hi @Matt,

Everything looks fine on our end, according to our monitoring tools.
We are pretty confident that no regressions have been introduced in the recent days.

I see that you did an update of your staging and production environments yesterday with various schema changes. Do you have in mind specific things you could have changed in your infra or codebase that could explain such situation?

Thanks for you help :pray:

Gotcha. The changes have been in staging for a few days with no errors, and I only made adjustments to production after I noticed the error, hoping it might fix it. That was the reason I wanted to check on your end. And again, everything works fine locally.

Thank you giving me the update! If you happen to think of any similar errors you’ve run into, feel free to let me know.

Do you happen to know generally what might be causing the 403 error? Something you guys may have run into before?

Here’s a video showing me getting passed authentication in my production environment and being able to access the data, but after a refresh it shows the authentication error: Error-Video.mov - Google Drive

I’m able to get passed it about 1/10 times, and when the error happens I don’t see any server errors in my logs outside of the 403 error above that can pin point what the issue might be. And I didn’t make any changes to architecture or code when the error first presented itself. Just wanted to double check that there wasn’t anything you guys know about on your end that could help? I’ve just never run into this specific error before. Thanks again!

Hello @Matt,

Are you deploying your environment on multiple servers?

Can you share with us the body of the 403 response? This should contain more details about the error.

@GuillaumeGautreau It appears that the error has gone away. I ended up running across an error in our logs that could have been the reason for it, if so it’s just strange it would have caused an authentication error. It was hard to follow the “bread crumb trail”. But I resolved that error on my end and everything appears to be operational again. Thank you for your help! :slight_smile:

1 Like