πŸ›‘ We observe some latencies and timeouts on our API services. πŸ›‘

Hi all,

Here is a quick report on the situation. :newspaper_roll:

On june 11th, between 9:00am and 10:40am, our API has been really slow and instable.
As a consequence, the Forest Admin users experience has been deteriorated by latencies or service errors.

From a technical point of view, we observed very high response time, memory increases, errors and server web instances crashes.

On wednesday, June 9th, we released a feature that introduced the role information into user session tokens. It appears that this contribution modified a specific SQL query used by our services during agent session creation. This new query added more joins and this appeared to be very very slow for specific projects.

We rollbacked the specific commit and the latencies and errors instantly disappeared.

We’re working internally to prevent similar issues in the future and are deeply sorry for the inconvenience :pray:

Thanks.

3 Likes