Dashboard graphs and charts have stopped rendering

Feature(s) impacted

Dashboard (rendering of charts)

Observed behavior

Charts suddenly stopped rendering yesterday despite no changes being made our side.

Expected behavior

Charts should render.

Failure Logs

See console errors. Looks like CORS issue but doesn’t make sense as other graphs that are rendering are relying on same endpoints with similar data and filters (only the dates differ)

Context

Please provide in this mandatory section, the relevant information about your configuration:
Project: Bridgement.1
Package Version: “forest-express-sequelize”: “^8.4.10”,
Express Version: “express”: “^4.17.1”,
Sequelize Version: “sequelize”: “6.6.2”,
Database Dialect: postgresql

1 Like

Hi @dig :wave: I’m sad to hear that,
In order to try to reproduce and understand what happen, I have some questions.

Have you got any errors logs on your backend?
Did the issue appear on all environment?
Did the issue appear on the dashboard view or on the collection analytics view?

EDIT:
And what kind of chart it is? Simple, Query or API?

Looks like mostly are single value charts, but some time based too.
Only have a single environment.
Issue occurred in Dashboard view (we don’t use collection analytics)
I’ll have to check with engineers on the backend error logs

Hey @dig :wave:

According to the error you have in the browser console, could you:

  • Locate the failing call in your network browser developer console
  • Right click and Copy as curl
  • Run the curl request in a terminal
  • Share the output here or as a PM if it contains sensible informations

It looks super weird if you didn’t touch anything on your end. However, running the failing request as curl could provide more explicit infos.

Thanks in advance :pray:

Hi @jeffladiray,

I’m on the same project as @dig so I’ll give some more info.

The curl request won’t give any useful information as cloudfront manages the errors and so it returns an html page.

However, I have tried loading the graphs locally and this is the error I get:

[forest] 🌳🌳🌳 'statWithParameters' access forbidden on DbCollection

[forest] 🌳🌳🌳 'statWithParameters' access forbidden on DbAccountTransaction

[forest] 🌳🌳🌳 'statWithParameters' access forbidden on DbFacility

Each error is for a different graph, which uses different models. Hope this helps.

Well … the cloudfront HTML response may have helped here :slight_smile:

Do you use some WAF rules on your cloudfront for security purpose? We do have a few clients referencing this issue (Request blocked) using cloudflare.
This is easily checked by:

  • Checking your agent for the stats call, and check if your backend is able to send a response (At least, check if the agent is contacted)
  • Check the CORS request response, and check for headers, just in case the request is blocked by a CDN (X-server, or any other header)

Not sure if it’s related, but looking for cloudflare, it seems like we have a few threads around this topic :slight_smile:

Let me know if that helps

As for the WAF issues, I’ll PM you the returned html.

The stats calls seems to be working fine on different dashboard graphs. (All done using simple graphs) which leads me to believe that it is not an issue with Cloudflare.

I see here that this problem is similar to what we are experiencing, but it seems that the issue was resolved. However I am getting the same errors.

Thanks for the response. I’ll check tomorrow SOB if i’m able to reproduce in a similar context :+1:

In the meantime, I suggest to still check for any cloudflare interference, even though it may not be the issue :pray:

EDIT: The issue has been located. We’ll let you know once the fix is released.

2 Likes

After further discussion, it appear that the fix did not retro-actively fixed the failing charts.

Re-saving the configuration of a chart having this issue should fix the issue as well. This is far from being a perfect solution, however, this will unlock the issue on your end.

This issue has been reported - and everything seems to work as expected by now, so I’m marking this thread as resolved. Don’t hesitate to open a new thread if you are still encountering the issue :slight_smile:

1 Like