Dashboard graphs and charts have stopped rendering

[dig]

Feature(s) impacted

Dashboard (rendering of charts)

Observed behavior

Charts suddenly stopped rendering yesterday despite no changes being made our side.

Expected behavior

Charts should render.

Failure Logs

See console errors. Looks like CORS issue but doesn’t make sense as other graphs that are rendering are relying on same endpoints with similar data and filters (only the dates differ)

Context

Please provide in this mandatory section, the relevant information about your configuration:
Project: Bridgement.1
Package Version: “forest-express-sequelize”: “^8.4.10”,
Express Version: “express”: “^4.17.1”,
Sequelize Version: “sequelize”: “6.6.2”,
Database Dialect: postgresql

[Arnaud_Moncel]

Hi @dig I’m sad to hear that,
In order to try to reproduce and understand what happen, I have some questions.

Have you got any errors logs on your backend?
Did the issue appear on all environment?
Did the issue appear on the dashboard view or on the collection analytics view?

EDIT:
And what kind of chart it is? Simple, Query or API?

[dig]

Looks like mostly are single value charts, but some time based too.
Only have a single environment.
Issue occurred in Dashboard view (we don’t use collection analytics)
I’ll have to check with engineers on the backend error logs

[jeffladiray]

Hey @dig

According to the error you have in the browser console, could you:

• Locate the failing call in your network browser developer console
• Right click and Copy as curl
• Run the curl request in a terminal
• Share the output here or as a PM if it contains sensible informations

It looks super weird if you didn’t touch anything on your end. However, running the failing request as curl could provide more explicit infos.

Thanks in advance

[SoftwareBridgementKe]

Hi @jeffladiray,

I’m on the same project as @dig so I’ll give some more info.

The curl request won’t give any useful information as cloudfront manages the errors and so it returns an html page.

However, I have tried loading the graphs locally and this is the error I get:

[forest] 🌳🌳🌳 'statWithParameters' access forbidden on DbCollection

[forest] 🌳🌳🌳 'statWithParameters' access forbidden on DbAccountTransaction

[forest] 🌳🌳🌳 'statWithParameters' access forbidden on DbFacility

Each error is for a different graph, which uses different models. Hope this helps.

[jeffladiray]

Well … the cloudfront HTML response may have helped here

Do you use some WAF rules on your cloudfront for security purpose? We do have a few clients referencing this issue (Request blocked) using cloudflare.
This is easily checked by:

• Checking your agent for the stats call, and check if your backend is able to send a response (At least, check if the agent is contacted)
• Check the CORS request response, and check for headers, just in case the request is blocked by a CDN (X-server, or any other header)

Not sure if it’s related, but looking for cloudflare, it seems like we have a few threads around this topic

Cloudflare WAF blocking custom SQL

Expected behavior Charts that use a Custom SQL data source should not trigger basic generic security layers such as Cloudflare WAF. Sending the raw SQL as the payload seems to have this effect Actual behavior Cloudflare WAF and similar services (including Sqreen) will automatically block the request with a HTTP 403 Context We imagine Custom SQL is a popular feature and considering how widespread security tools like cloudflare are, we were wondering if there is a recommended approach to this…

Smart action that upload a file return 403

Expected behavior Uploading an xml file, doing some processing, and export csv files. Actual behavior a 403 issue on that particular endpoint when clicking on the submit button to launch the post call. What is the current behavior? a 404 error over get /values (I understand this is normal from a different post here) Failure Logs Access to fetch at ‘api-to-post-the-file’ from origin ‘https://app.forestadmin.com’ has been blocked by CORS policy: No ‘Access-Control-Allow-Origin’ header is pr…

Let me know if that helps

[SoftwareBridgementKe]

As for the WAF issues, I’ll PM you the returned html.

The stats calls seems to be working fine on different dashboard graphs. (All done using simple graphs) which leads me to believe that it is not an issue with Cloudflare.

I see here that this problem is similar to what we are experiencing, but it seems that the issue was resolved. However I am getting the same errors.

'statWithParameters' access forbidden on xxxxx and this chart cannot be rendered

Feature(s) impacted Some of my charts are not work properly since months. Maybe it’s related to nested fields. Observed behavior I set up the chart, at first time it’s seems ok but when a go to another environment and come back it is no more working. Expected behavior My chart always work correctly Failure Logs ‘statWithParameters’ access forbidden on Context I did update from version 5 to version 8 to be up to date and it’s the same. I looked on my backend today and i see the message…

[jeffladiray]

Thanks for the response. I’ll check tomorrow SOB if i’m able to reproduce in a similar context

In the meantime, I suggest to still check for any cloudflare interference, even though it may not be the issue

EDIT: The issue has been located. We’ll let you know once the fix is released.

[jeffladiray]

After further discussion, it appear that the fix did not retro-actively fixed the failing charts.

Re-saving the configuration of a chart having this issue should fix the issue as well. This is far from being a perfect solution, however, this will unlock the issue on your end.

This issue has been reported - and everything seems to work as expected by now, so I’m marking this thread as resolved. Don’t hesitate to open a new thread if you are still encountering the issue