Problem on getIdsFromRequest with "All records" selection

Help me!
1

Hi team,

I have some huge performance problems when using getIdsFromRequest along with “All records” selection.

Example usage:
const userIds = await new RecordsGetter(models.users, req.user, req.query).getIdsFromRequest(req);

I can see that there is a batch that is retrieving all the records, per package of 1000 records, I am ok with that.

The problem is coming from the way in which the IDs are retrieved from database.
I can see that, while this function should retrieve only the Ids of my records, it will select all the attributes (I mean all the columns) of the current collection displayed.
This is generating very high useless CPU and memory usage.

It makes a big difference in the amount of data managed when you have an action that has to work on more than 50,000 records.

Could you please improve this method?
When querying the database, please retrieve only the IDs when only the IDs are requested.
This is the expected behaviour.

Thanks in advance for your help.

My current setup:

  "meta": {
    "liana": "forest-express-sequelize",
    "liana_version": "8.0.2",
    "stack": {
      "database_type": "postgres",
      "engine": "nodejs",
      "engine_version": "14.16.0",
      "orm_version": "5.22.3"
    }
  }
2

Hi @Louis-Marie unfortunately it’s a a real performance issue.
I will open a bug report.
You will be notified when the problem is resolved.
Thank to your feedback :pray:

3

Hi @Louis-Marie

I’m posting a quick update, as this thread has been stale for two weeks.

I just finished the pull request that should fix that issue, it should go live in a couple days (it still needs to be reviewed / deployed)

Have a nice day!

4

Hi @anon39940173

Thanks for this update.
I appreciate.

5

Hi @anon39940173,

Do you have any news on this request please?
Thanks in advance

6

Hi @Louis-Marie,

Sorry for the delayed response, the issue should be fixed by now! :tada:

Can you confirm that the 8.1.1 version of forest-express-sequelize fixes the problem?

Thanks!

7

Hi @anon34731316

I was just testing with the last release of forest-express-sequelize : 8.2.0
It seems to work in general cases, and the impact on performances is enormous…

Excellent!

However, there is still a problem with smart segments.
It seems that it does not take into account the smart segment when “all records” is selected. It is selecting the whole collection.
I was using the same route for my tests.
Could you please check on your side?

Thanks in advance

8

@Louis-Marie thanks for spotting this huge problem!
We’re investigating at the moment and we’ll get back to you asap.

Thanks.

9

@Louis-Marie could you try to update your forest-express-sequelize to version 8.2.2 and tell me if it works better?
Really sorry for the inconvenience!

10

Hi @anon34731316

Yes, it works as expected now.
Thank you very much

Have a nice day.

1 Like