99 / 5.000 When combining search and pagination change, the following error occurs: You server encountered an error

martinshumberto · January 27, 2025, 4:19pm

Feature(s) impacted

Listing records from a given collection, after using the combination of search and pagination, the problem appears after the third page

Observed behavior

Expected behavior

Server does not load the requested page records as expected

Failure Logs

2025-01-23T13:41:27.437Z | error: {"level":"Error","event":"request","status":500,"method":"GET","path":"/editorials","duration":326,"error":{"message":"PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.","stack":"MongoServerError: PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.
    at t.onMessage (/opt/nodejs/index.js:271:538704)
    at _5e.<anonymous> (/opt/nodejs/index.js:271:536655)
    at _5e.emit (node:events:517:28)
    at processIncomingData (/opt/nodejs/index.js:271:533973)
    at _5e._write (/opt/nodejs/index.js:271:532522)
    at writeOrBuffer (node:internal/streams/writable:392:12)
    at _write (node:internal/streams/writable:333:10)
    at Writable.write (node:internal/streams/writable:337:10)
    at TLSSocket.ondata (node:internal/streams/readable:809:22)
    at TLSSocket.emit (node:events:517:28)"}}

Context

Project name: Bombarco
Team name: Bombarco
Environment name: Production
Agent technology: Node
Agent (forest package) name & version: latest
Database type: mongodb
Recent changes made on your end if any: …

dogan.ay · January 27, 2025, 4:26pm

Hello @martinshumberto and welcome to the community !

I’ll investigate the issue and keep you updated with any findings.

dogan.ay · January 27, 2025, 4:51pm

Would you be able to share a few info with us;

what is the size of your collection ?
how many records are there ?
do you have any indexes declared as to optimize the performance, notably on the createdAt column which you’re sorting on ?

Are you also experiencing a similar result in the same case scenario with other tools or is it happening only with Forest Admin ?

martinshumberto · January 27, 2025, 6:47pm

Hello Dogan,

Size: 206.10 MB

Number of docs: 6112

We have indeces for the most used properties for ordering, conditionals. Some: category, createdAt, updatedAt, publishedAt, slug, status, type.

From what we can see, we are only seeing this result in ForestAdmin, we have a blog-like section on our website, and we have not had a similar scenario, and with similar conditionals.

Thank you for your support in advance.

dogan.ay · January 28, 2025, 5:09pm

Sorry for the delay, this is clearly an issue with how the aggregation pipeline is generated, however as the Cloud mongo support is quite recent, we do not currently have a way to enable debug mode on Mongoose which is being used by our agent.

I am currently adding the possibility of enabling Mongoose debug mode as well as specifying the LogLevel via environment variables (when pushing code customization). With this you will be able to set Debug log level, thus making the investigation possible to hopefully resolve it ASAP.

dogan.ay · January 31, 2025, 8:54am

Hello @martinshumberto,

I have deployed a new version of the Cloud agent for it to log requests and their aggregation pipeline, could you reproduce your issue again now that we have the logs enabled ?

martinshumberto · January 31, 2025, 11:24am

Hello @dogan.ay

Tks for support.

I replicated the problem, here are the logs:

2025-01-31T11:21:10.631Z | info: [200] GET /editorials - 2205ms
2025-01-31T11:21:10.900Z | info: [200] GET /editorials/count - 2547ms
2025-01-31T11:21:11.864Z | debug: editorials.aggregate: [{"$addFields":{"image@@@_id":"$image._id","image@@@path":"$image.path","image@@@size":"$image.size","image@@@fileName":"$image.fileName","image@@@mimeType":"$image.mimeType","image@@@createdAt":"$image.createdAt","image@@@updatedAt":"$image.updatedAt","image@@@uploadedAt":"$image.uploadedAt","author@@@_id":"$author._id","author@@@cpf":"$author.cpf","author@@@cnpj":"$author.cnpj","author@@@type":"$author.type","author@@@email":"$author.email","author@@@status":"$author.status","author@@@address@@@city":"$author.address.city","author@@@address@@@state":"$author.address.state","author@@@address@@@number":"$author.address.number","author@@@address@@@address":"$author.address.address","author@@@address@@@country":"$author.address.country","author@@@address@@@zipCode":"$author.address.zipCode","author@@@address@@@district":"$author.address.district","author@@@address@@@complement":"$author.address.complement","author@@@profile@@@_id":"$author.profile._id","author@@@profile@@@path":"$author.profile.path","author@@@profile@@@size":"$author.profile.size","author@@@profile@@@fileName":"$author.profile.fileName","author@@@profile@@@mimeType":"$author.profile.mimeType","author@@@profile@@@uploadedAt":"$author.profile.uploadedAt","author@@@contacts@@@phoneNumber":"$author.contacts.phoneNumber","author@@@contacts@@@phonePrincipal":"$author.contacts.phonePrincipal"}},{"$addFields":{"author@@@lastName":"$author.lastName","author@@@createdAt":"$author.createdAt","author@@@firstName":"$author.firstName","author@@@emailTerms":"$author.emailTerms","author@@@externalId":"$author.externalId","author@@@dateOfBirth":"$author.dateOfBirth","author@@@fantasyName":"$author.fantasyName","author@@@acceptedTerms":"$author.acceptedTerms","author@@@corporateName":"$author.corporateName","author@@@emailPromotion":"$author.emailPromotion","author@@@emailTermsPartner":"$author.emailTermsPartner"}},{"$project":{"image._id":0,"image.path":0,"image.size":0,"image.fileName":0,"image.mimeType":0,"image.createdAt":0,"image.updatedAt":0,"image.uploadedAt":0,"author._id":0,"author.cpf":0,"author.cnpj":0,"author.type":0,"author.email":0,"author.status":0,"author.address.city":0,"author.address.state":0,"author.address.number":0,"author.address.address":0,"author.address.country":0,"author.address.zipCode":0,"author.address.district":0,"author.address.complement":0,"author.profile._id":0,"author.profile.path":0,"author.profile.size":0,"author.profile.fileName":0,"author.profile.mimeType":0,"author.profile.uploadedAt":0,"author.contacts.phoneNumber":0,"author.contacts.phonePrincipal":0,"author.lastName":0,"author.createdAt":0,"author.firstName":0,"author.emailTerms":0,"author.externalId":0,"author.dateOfBirth":0,"author.fantasyName":0,"author.acceptedTerms":0,"author.corporateName":0,"author.emailPromotion":0,"author.emailTermsPartner":0}},{"$addFields":{"string__id":{"$toString":"$_id"},"string_image@@@_id":{"$toString":"$image@@@_id"},"string_author@@@_id":{"$toString":"$author@@@_id"},"string_author@@@profile@@@_id":{"$toString":"$author@@@profile@@@_id"}}},{"$match":{"$or":[{"string__id":{}},{"city":{}},{"type":{}},{"state":{}},{"title":{}},{"video":{}},{"status":{}},{"content":{}},{"category":{}},{"description":{}},{"string_image@@@_id":{}},{"image@@@path":{}},{"image@@@fileName":{}},{"image@@@mimeType":{}},{"string_author@@@_id":{}},{"author@@@cpf":{}},{"author@@@type":{}},{"author@@@email":{}},{"author@@@status":{}},{"author@@@address@@@country":{}},{"string_author@@@profile@@@_id":{}},{"author@@@profile@@@path":{}},{"author@@@profile@@@fileName":{}},{"author@@@profile@@@mimeType":{}},{"author@@@contacts@@@phoneNumber":{}},{"author@@@contacts@@@phonePrincipal":{}},{"author@@@lastName":{}},{"author@@@firstName":{}},{"author@@@fantasyName":{}},{"author@@@corporateName":{}}]}},{"$sort":{"createdAt":-1}},{"$skip":30},{"$limit":10},{"$project":{"_id":true,"FOREST_RECORD_DOES_NOT_EXIST":true,"title":true,"author@@@firstName":true,"author@@@lastName":true,"category":true,"status":true,"type":true,"createdAt":true,"updatedAt":true,"publishedAt":true,"views":true}}]
2025-01-31T11:21:12.014Z | debug: editorials.aggregate: [{"$addFields":{"image@@@_id":"$image._id","image@@@path":"$image.path","image@@@size":"$image.size","image@@@fileName":"$image.fileName","image@@@mimeType":"$image.mimeType","image@@@createdAt":"$image.createdAt","image@@@updatedAt":"$image.updatedAt","image@@@uploadedAt":"$image.uploadedAt","author@@@_id":"$author._id","author@@@cpf":"$author.cpf","author@@@cnpj":"$author.cnpj","author@@@type":"$author.type","author@@@email":"$author.email","author@@@status":"$author.status","author@@@address@@@city":"$author.address.city","author@@@address@@@state":"$author.address.state","author@@@address@@@number":"$author.address.number","author@@@address@@@address":"$author.address.address","author@@@address@@@country":"$author.address.country","author@@@address@@@zipCode":"$author.address.zipCode","author@@@address@@@district":"$author.address.district","author@@@address@@@complement":"$author.address.complement","author@@@profile@@@_id":"$author.profile._id","author@@@profile@@@path":"$author.profile.path","author@@@profile@@@size":"$author.profile.size","author@@@profile@@@fileName":"$author.profile.fileName","author@@@profile@@@mimeType":"$author.profile.mimeType","author@@@profile@@@uploadedAt":"$author.profile.uploadedAt","author@@@contacts@@@phoneNumber":"$author.contacts.phoneNumber","author@@@contacts@@@phonePrincipal":"$author.contacts.phonePrincipal"}},{"$addFields":{"author@@@lastName":"$author.lastName","author@@@createdAt":"$author.createdAt","author@@@firstName":"$author.firstName","author@@@emailTerms":"$author.emailTerms","author@@@externalId":"$author.externalId","author@@@dateOfBirth":"$author.dateOfBirth","author@@@fantasyName":"$author.fantasyName","author@@@acceptedTerms":"$author.acceptedTerms","author@@@corporateName":"$author.corporateName","author@@@emailPromotion":"$author.emailPromotion","author@@@emailTermsPartner":"$author.emailTermsPartner"}},{"$project":{"image._id":0,"image.path":0,"image.size":0,"image.fileName":0,"image.mimeType":0,"image.createdAt":0,"image.updatedAt":0,"image.uploadedAt":0,"author._id":0,"author.cpf":0,"author.cnpj":0,"author.type":0,"author.email":0,"author.status":0,"author.address.city":0,"author.address.state":0,"author.address.number":0,"author.address.address":0,"author.address.country":0,"author.address.zipCode":0,"author.address.district":0,"author.address.complement":0,"author.profile._id":0,"author.profile.path":0,"author.profile.size":0,"author.profile.fileName":0,"author.profile.mimeType":0,"author.profile.uploadedAt":0,"author.contacts.phoneNumber":0,"author.contacts.phonePrincipal":0,"author.lastName":0,"author.createdAt":0,"author.firstName":0,"author.emailTerms":0,"author.externalId":0,"author.dateOfBirth":0,"author.fantasyName":0,"author.acceptedTerms":0,"author.corporateName":0,"author.emailPromotion":0,"author.emailTermsPartner":0}},{"$addFields":{"string__id":{"$toString":"$_id"},"string_image@@@_id":{"$toString":"$image@@@_id"},"string_author@@@_id":{"$toString":"$author@@@_id"},"string_author@@@profile@@@_id":{"$toString":"$author@@@profile@@@_id"}}},{"$match":{"$or":[{"string__id":{}},{"city":{}},{"type":{}},{"state":{}},{"title":{}},{"video":{}},{"status":{}},{"content":{}},{"category":{}},{"description":{}},{"string_image@@@_id":{}},{"image@@@path":{}},{"image@@@fileName":{}},{"image@@@mimeType":{}},{"string_author@@@_id":{}},{"author@@@cpf":{}},{"author@@@type":{}},{"author@@@email":{}},{"author@@@status":{}},{"author@@@address@@@country":{}},{"string_author@@@profile@@@_id":{}},{"author@@@profile@@@path":{}},{"author@@@profile@@@fileName":{}},{"author@@@profile@@@mimeType":{}},{"author@@@contacts@@@phoneNumber":{}},{"author@@@contacts@@@phonePrincipal":{}},{"author@@@lastName":{}},{"author@@@firstName":{}},{"author@@@fantasyName":{}},{"author@@@corporateName":{}}]}},{"$group":{"_id":null,"value":{"$sum":1}}},{"$project":{"_id":0,"value":"$value","group":{"$literal":{}}}},{"$sort":{"value":-1}}]
2025-01-31T11:21:12.606Z | error: {"level":"Error","event":"request","status":500,"method":"GET","path":"/editorials","duration":787,"error":{"message":"PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.","stack":"MongoServerError: PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.
    at t.sendCommand (/opt/nodejs/index.js:279:67726)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async t.command (/opt/nodejs/index.js:279:68372)
    at async t.command (/opt/nodejs/index.js:279:99053)
    at async Vte.executeCommand (/opt/nodejs/index.js:275:71976)
    at async Vte.execute (/opt/nodejs/index.js:275:104791)
    at async tryOperation (/opt/nodejs/index.js:275:69358)
    at async executeOperation (/opt/nodejs/index.js:275:67330)
    at async t._initialize (/opt/nodejs/index.js:279:229500)
    at async t.cursorInit (/opt/nodejs/index.js:279:226219)"}}
✔ Requested 30 logs in the last month
You have received logs from 2025-01-31T11:20:28.728Z to 2025-01-31T11:21:12.606Z

dogan.ay · January 31, 2025, 4:19pm

Does this error happen when you simply navigate via pagination (without search) ?

Could you also show the list request that is being sent via the network tab whenever you do the search ?

If nothing else until then, you could try to set the option allowDiskUse:true in your mongo server as suggested by the error, I am not too familiar with Mongo environments but 32MB of allocated memory does not seem that high either.

martinshumberto · February 12, 2025, 10:31pm

This error does not occur when using paging. To simulate it, you must use the search together. I first search on page one, and then click next.

I recorded a video showing the behavior, along with the set of requests in the network tab.

Here it is: https://drive.google.com/file/d/1PoWkWeaUv7AiAzI15mU1yrPFd_WwUnLP/view?usp=drive_link

Looking a little more about allowDiskUse, it says that it is enabled in each query, so it is not something that I can configure on the server. But it is something that should be used in the query, see the example:

db.collection.aggregate(pipeline, { allowDiskUse: true });

Is this something that you can do?

Thanks for the support and sorry for the delay.

dogan.ay · February 13, 2025, 1:50pm

Hello @martinshumberto,

After investigating once more, because I do not feel warranted to enable allowDiskUse to all users on the @forestadmin/datasource-mongoose, I suspect that the performance issue on your search comes from the fields in your collections.
Without the search the performance is quite good, even when accessing large collections and a high number of pages. And the response times I observe on your collection without the search are quite fast too.

The search operation we use on Forest will create match operators on all string fields of your collection and nested fields, the match being a regex of the search input.
The regex match is quite consuming and may impact you more when you have a large amount of string fields and if the content is long (especially the field content of editorials which is suspicious to me).

Could you restrict the fields of your search to only the relevant fields of your collection ? following this documentation;

Topic		Replies	Views
Sorting/filtering no longer functional in related data Help me!	11	541	September 8, 2021
Some weird error on a browser console Help me!	3	465	May 13, 2022
Pagination issue - 1 of 0 Help me!	4	379	September 23, 2021
Sorting on large mongo collections Help me!	3	317	November 16, 2020
Pagination Problem on forestexpress-sequelize@3.3.0 Help me!	9	249	March 2, 2022