Impossible to extract data when the user base becomes large

Help me!
1

:warning:This is a template you must use to report issues. :warning:
You can also drag images, videos and include Preformatted text.

export 1
export 12553×1387 419 KB

export 3
export 32554×1396 435 KB

export 2
export 22556×1402 429 KB

Feature(s) impacted

Please describe in this mandatory section the feature(s) that will be discussed in this topic.
Impossible to extract data anymore

Observed behavior

Please describe in this mandatory section the current behavior you observe.
Export a data set is impossible even when sorting users

Expected behavior

Please describe in this mandatory section the behavior you are expecting.

Failure Logs

In this optional section, please:

  • include any relevant log snippets if necessary,
  • or remove this section if left empty.

Context

Please provide in this mandatory section, the relevant information about your configuration:

  • Project name: Feels
  • Team name: Feels
  • Environment name: Production
  • Agent type & version: …
  • Recent changes made on your end if any: …
2

Hello @Daniel_Cheaib,

Thanks for your message and welcome to our community :raised_hands:

Could you please provide the error logs that should be displayed in your admin panel server when you try this export?

Thanks.

4

Hi, thanks
Here are the logs

OPTIONS /forest/batch?timezone=Europe%2FParis&fields%5Bbatch%5D=creationTime%2Cid%2Ctype%2CupdateTime%2CvalidUntil%2Cstories%2CallUserSentFeelsIds%2CbatchUserSentFeelsIds%2CuserFilters%2Cuser&fields%5Buser%5D=id&page%5Bnumber%5D=1&page%5Bsize%5D=30&sort=-creationTime 204 0 - 0.181 ms
OPTIONS /forest/batch/count?fields%5Bbatch%5D=creationTime%2Cid%2Ctype%2CupdateTime%2CvalidUntil%2Cstories%2CallUserSentFeelsIds%2CbatchUserSentFeelsIds%2CuserFilters%2Cuser&fields%5Buser%5D=id&timezone=Europe%2FParis 204 0 - 0.136 ms
GET / 200 10026 - 0.666 ms
GET / 200 10026 - 0.810 ms
GET / 200 10026 - 3.193 ms
GET / 200 10026 - 1.290 ms
GET / 200 10026 - 0.705 ms
GET / 200 10026 - 0.664 ms
GET / 200 10026 - 0.611 ms
GET / 200 10026 - 0.754 ms
GET / 200 10026 - 1.211 ms
GET /forest/batch?timezone=Europe%2FParis&fields%5Bbatch%5D=creationTime%2Cid%2Ctype%2CupdateTime%2CvalidUntil%2Cstories%2CallUserSentFeelsIds%2CbatchUserSentFeelsIds%2CuserFilters%2Cuser&fields%5Buser%5D=id&page%5Bnumber%5D=1&page%5Bsize%5D=30&sort=-creationTime - - - - ms
GET /forest/batch/count?fields%5Bbatch%5D=creationTime%2Cid%2Ctype%2CupdateTime%2CvalidUntil%2Cstories%2CallUserSentFeelsIds%2CbatchUserSentFeelsIds%2CuserFilters%2Cuser&fields%5Buser%5D=id&timezone=Europe%2FParis - - - - ms
GET / 200 10026 - 0.717 ms
GET / 200 10026 - 0.888 ms
GET / 200 10026 - 0.650 ms
GET / 200 10026 - 0.659 ms
GET / 200 10026 - 0.669 ms
GET / 200 10026 - 0.600 ms
GET / 200 10026 - 1.232 ms
GET / 200 10026 - 0.636 ms
GET / 200 10026 - 0.679 ms
GET /forest/batch?timezone=Europe%2FParis&fields%5Bbatch%5D=creationTime%2Cid%2Ctype%2CupdateTime%2CvalidUntil%2Cstories%2CallUserSentFeelsIds%2CbatchUserSentFeelsIds%2CuserFilters%2Cuser&fields%5Buser%5D=id&page%5Bnumber%5D=1&page%5Bsize%5D=30&sort=-creationTime - - - - ms
GET /forest/batch/count?fields%5Bbatch%5D=creationTime%2Cid%2Ctype%2CupdateTime%2CvalidUntil%2Cstories%2CallUserSentFeelsIds%2CbatchUserSentFeelsIds%2CuserFilters%2Cuser&fields%5Buser%5D=id&timezone=Europe%2FParis - - - - ms

5

@Daniel_Cheaib,

Thanks for your answer.

I’m not sure these are the logs we are looking for :thinking:
It should be something like forest/<name-of-your-collection>.csv.
Can you see something like this?
And we are expecting an error on that call…

Thanks.

6

Well, it starts with /forest/batch.
It doesn’t end with .csv, since it’s not an export. I just clicked on the Batches collection.
Not being from ForestAdmin, I can clearly see that the response timeout is missing in the logs.
A timeout problem maybe?

7

@Daniel_Cheaib,

I’m really sorry but I am not sure I understood well.

Your issue is when you’re trying to export your data, correct?
So can you try to export and catch the matching logs in your server?

And can you tell me if your problem is occurring on every collections?
Are they all containing a lot of records?

Thanks

8

we have an error message after a while after clicking on the “Batches” data export (Screenshots already shared).
The problem is not on all the collections. It seems to happen only on the largest collections.
The logs already sent show two unanswered calls to the ‘GET:/forest/batch’ and ‘GET:/forest/batch/count’ endpoints.

What else can I provide for your investigation please?
Do you have a timeout problem? Is it possible to solve it?

9

@Daniel_Cheaib,

I don’t believe the two calls you are pointing at are the source of the problem here.
One is for the loading the page records and the other one the total records count - and they seem to work as you can see the 1st page of the table with the records and the total records count.

Can you open your Network (in your browser console) and check what call is sent when trying to export?
It should be something like /forest/batch.csv, I let you confirm (or not).
And then you should be able to look for that call into your logs and found out what’s going on here.

Let me know.
Thanks!