99 / 5.000 When combining search and pagination change, the following error occurs: You server encountered an error

Feature(s) impacted

Listing records from a given collection, after using the combination of search and pagination, the problem appears after the third page

Observed behavior

Expected behavior

Server does not load the requested page records as expected

Failure Logs

2025-01-23T13:41:27.437Z | error: {"level":"Error","event":"request","status":500,"method":"GET","path":"/editorials","duration":326,"error":{"message":"PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.","stack":"MongoServerError: PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.
    at t.onMessage (/opt/nodejs/index.js:271:538704)
    at _5e.<anonymous> (/opt/nodejs/index.js:271:536655)
    at _5e.emit (node:events:517:28)
    at processIncomingData (/opt/nodejs/index.js:271:533973)
    at _5e._write (/opt/nodejs/index.js:271:532522)
    at writeOrBuffer (node:internal/streams/writable:392:12)
    at _write (node:internal/streams/writable:333:10)
    at Writable.write (node:internal/streams/writable:337:10)
    at TLSSocket.ondata (node:internal/streams/readable:809:22)
    at TLSSocket.emit (node:events:517:28)"}}

Context

  • Project name: Bombarco
  • Team name: Bombarco
  • Environment name: Production
  • Agent technology: Node
  • Agent (forest package) name & version: latest
  • Database type: mongodb
  • Recent changes made on your end if any: …

Hello @martinshumberto and welcome to the community !

I’ll investigate the issue and keep you updated with any findings.

1 Like

Would you be able to share a few info with us;

  • what is the size of your collection ?
  • how many records are there ?
  • do you have any indexes declared as to optimize the performance, notably on the createdAt column which you’re sorting on ?

Are you also experiencing a similar result in the same case scenario with other tools or is it happening only with Forest Admin ?

Hello Dogan,

Size: 206.10 MB

Number of docs: 6112

We have indeces for the most used properties for ordering, conditionals. Some: category, createdAt, updatedAt, publishedAt, slug, status, type.

From what we can see, we are only seeing this result in ForestAdmin, we have a blog-like section on our website, and we have not had a similar scenario, and with similar conditionals.

Thank you for your support in advance.

1 Like

Sorry for the delay, this is clearly an issue with how the aggregation pipeline is generated, however as the Cloud mongo support is quite recent, we do not currently have a way to enable debug mode on Mongoose which is being used by our agent.

I am currently adding the possibility of enabling Mongoose debug mode as well as specifying the LogLevel via environment variables (when pushing code customization). With this you will be able to set Debug log level, thus making the investigation possible to hopefully resolve it ASAP.

Hello @martinshumberto,

I have deployed a new version of the Cloud agent for it to log requests and their aggregation pipeline, could you reproduce your issue again now that we have the logs enabled ?

1 Like

Hello @dogan.ay

Tks for support.

I replicated the problem, here are the logs:

2025-01-31T11:21:10.631Z | info: [200] GET /editorials - 2205ms
2025-01-31T11:21:10.900Z | info: [200] GET /editorials/count - 2547ms
2025-01-31T11:21:11.864Z | debug: editorials.aggregate: [{"$addFields":{"image@@@_id":"$image._id","image@@@path":"$image.path","image@@@size":"$image.size","image@@@fileName":"$image.fileName","image@@@mimeType":"$image.mimeType","image@@@createdAt":"$image.createdAt","image@@@updatedAt":"$image.updatedAt","image@@@uploadedAt":"$image.uploadedAt","author@@@_id":"$author._id","author@@@cpf":"$author.cpf","author@@@cnpj":"$author.cnpj","author@@@type":"$author.type","author@@@email":"$author.email","author@@@status":"$author.status","author@@@address@@@city":"$author.address.city","author@@@address@@@state":"$author.address.state","author@@@address@@@number":"$author.address.number","author@@@address@@@address":"$author.address.address","author@@@address@@@country":"$author.address.country","author@@@address@@@zipCode":"$author.address.zipCode","author@@@address@@@district":"$author.address.district","author@@@address@@@complement":"$author.address.complement","author@@@profile@@@_id":"$author.profile._id","author@@@profile@@@path":"$author.profile.path","author@@@profile@@@size":"$author.profile.size","author@@@profile@@@fileName":"$author.profile.fileName","author@@@profile@@@mimeType":"$author.profile.mimeType","author@@@profile@@@uploadedAt":"$author.profile.uploadedAt","author@@@contacts@@@phoneNumber":"$author.contacts.phoneNumber","author@@@contacts@@@phonePrincipal":"$author.contacts.phonePrincipal"}},{"$addFields":{"author@@@lastName":"$author.lastName","author@@@createdAt":"$author.createdAt","author@@@firstName":"$author.firstName","author@@@emailTerms":"$author.emailTerms","author@@@externalId":"$author.externalId","author@@@dateOfBirth":"$author.dateOfBirth","author@@@fantasyName":"$author.fantasyName","author@@@acceptedTerms":"$author.acceptedTerms","author@@@corporateName":"$author.corporateName","author@@@emailPromotion":"$author.emailPromotion","author@@@emailTermsPartner":"$author.emailTermsPartner"}},{"$project":{"image._id":0,"image.path":0,"image.size":0,"image.fileName":0,"image.mimeType":0,"image.createdAt":0,"image.updatedAt":0,"image.uploadedAt":0,"author._id":0,"author.cpf":0,"author.cnpj":0,"author.type":0,"author.email":0,"author.status":0,"author.address.city":0,"author.address.state":0,"author.address.number":0,"author.address.address":0,"author.address.country":0,"author.address.zipCode":0,"author.address.district":0,"author.address.complement":0,"author.profile._id":0,"author.profile.path":0,"author.profile.size":0,"author.profile.fileName":0,"author.profile.mimeType":0,"author.profile.uploadedAt":0,"author.contacts.phoneNumber":0,"author.contacts.phonePrincipal":0,"author.lastName":0,"author.createdAt":0,"author.firstName":0,"author.emailTerms":0,"author.externalId":0,"author.dateOfBirth":0,"author.fantasyName":0,"author.acceptedTerms":0,"author.corporateName":0,"author.emailPromotion":0,"author.emailTermsPartner":0}},{"$addFields":{"string__id":{"$toString":"$_id"},"string_image@@@_id":{"$toString":"$image@@@_id"},"string_author@@@_id":{"$toString":"$author@@@_id"},"string_author@@@profile@@@_id":{"$toString":"$author@@@profile@@@_id"}}},{"$match":{"$or":[{"string__id":{}},{"city":{}},{"type":{}},{"state":{}},{"title":{}},{"video":{}},{"status":{}},{"content":{}},{"category":{}},{"description":{}},{"string_image@@@_id":{}},{"image@@@path":{}},{"image@@@fileName":{}},{"image@@@mimeType":{}},{"string_author@@@_id":{}},{"author@@@cpf":{}},{"author@@@type":{}},{"author@@@email":{}},{"author@@@status":{}},{"author@@@address@@@country":{}},{"string_author@@@profile@@@_id":{}},{"author@@@profile@@@path":{}},{"author@@@profile@@@fileName":{}},{"author@@@profile@@@mimeType":{}},{"author@@@contacts@@@phoneNumber":{}},{"author@@@contacts@@@phonePrincipal":{}},{"author@@@lastName":{}},{"author@@@firstName":{}},{"author@@@fantasyName":{}},{"author@@@corporateName":{}}]}},{"$sort":{"createdAt":-1}},{"$skip":30},{"$limit":10},{"$project":{"_id":true,"FOREST_RECORD_DOES_NOT_EXIST":true,"title":true,"author@@@firstName":true,"author@@@lastName":true,"category":true,"status":true,"type":true,"createdAt":true,"updatedAt":true,"publishedAt":true,"views":true}}]
2025-01-31T11:21:12.014Z | debug: editorials.aggregate: [{"$addFields":{"image@@@_id":"$image._id","image@@@path":"$image.path","image@@@size":"$image.size","image@@@fileName":"$image.fileName","image@@@mimeType":"$image.mimeType","image@@@createdAt":"$image.createdAt","image@@@updatedAt":"$image.updatedAt","image@@@uploadedAt":"$image.uploadedAt","author@@@_id":"$author._id","author@@@cpf":"$author.cpf","author@@@cnpj":"$author.cnpj","author@@@type":"$author.type","author@@@email":"$author.email","author@@@status":"$author.status","author@@@address@@@city":"$author.address.city","author@@@address@@@state":"$author.address.state","author@@@address@@@number":"$author.address.number","author@@@address@@@address":"$author.address.address","author@@@address@@@country":"$author.address.country","author@@@address@@@zipCode":"$author.address.zipCode","author@@@address@@@district":"$author.address.district","author@@@address@@@complement":"$author.address.complement","author@@@profile@@@_id":"$author.profile._id","author@@@profile@@@path":"$author.profile.path","author@@@profile@@@size":"$author.profile.size","author@@@profile@@@fileName":"$author.profile.fileName","author@@@profile@@@mimeType":"$author.profile.mimeType","author@@@profile@@@uploadedAt":"$author.profile.uploadedAt","author@@@contacts@@@phoneNumber":"$author.contacts.phoneNumber","author@@@contacts@@@phonePrincipal":"$author.contacts.phonePrincipal"}},{"$addFields":{"author@@@lastName":"$author.lastName","author@@@createdAt":"$author.createdAt","author@@@firstName":"$author.firstName","author@@@emailTerms":"$author.emailTerms","author@@@externalId":"$author.externalId","author@@@dateOfBirth":"$author.dateOfBirth","author@@@fantasyName":"$author.fantasyName","author@@@acceptedTerms":"$author.acceptedTerms","author@@@corporateName":"$author.corporateName","author@@@emailPromotion":"$author.emailPromotion","author@@@emailTermsPartner":"$author.emailTermsPartner"}},{"$project":{"image._id":0,"image.path":0,"image.size":0,"image.fileName":0,"image.mimeType":0,"image.createdAt":0,"image.updatedAt":0,"image.uploadedAt":0,"author._id":0,"author.cpf":0,"author.cnpj":0,"author.type":0,"author.email":0,"author.status":0,"author.address.city":0,"author.address.state":0,"author.address.number":0,"author.address.address":0,"author.address.country":0,"author.address.zipCode":0,"author.address.district":0,"author.address.complement":0,"author.profile._id":0,"author.profile.path":0,"author.profile.size":0,"author.profile.fileName":0,"author.profile.mimeType":0,"author.profile.uploadedAt":0,"author.contacts.phoneNumber":0,"author.contacts.phonePrincipal":0,"author.lastName":0,"author.createdAt":0,"author.firstName":0,"author.emailTerms":0,"author.externalId":0,"author.dateOfBirth":0,"author.fantasyName":0,"author.acceptedTerms":0,"author.corporateName":0,"author.emailPromotion":0,"author.emailTermsPartner":0}},{"$addFields":{"string__id":{"$toString":"$_id"},"string_image@@@_id":{"$toString":"$image@@@_id"},"string_author@@@_id":{"$toString":"$author@@@_id"},"string_author@@@profile@@@_id":{"$toString":"$author@@@profile@@@_id"}}},{"$match":{"$or":[{"string__id":{}},{"city":{}},{"type":{}},{"state":{}},{"title":{}},{"video":{}},{"status":{}},{"content":{}},{"category":{}},{"description":{}},{"string_image@@@_id":{}},{"image@@@path":{}},{"image@@@fileName":{}},{"image@@@mimeType":{}},{"string_author@@@_id":{}},{"author@@@cpf":{}},{"author@@@type":{}},{"author@@@email":{}},{"author@@@status":{}},{"author@@@address@@@country":{}},{"string_author@@@profile@@@_id":{}},{"author@@@profile@@@path":{}},{"author@@@profile@@@fileName":{}},{"author@@@profile@@@mimeType":{}},{"author@@@contacts@@@phoneNumber":{}},{"author@@@contacts@@@phonePrincipal":{}},{"author@@@lastName":{}},{"author@@@firstName":{}},{"author@@@fantasyName":{}},{"author@@@corporateName":{}}]}},{"$group":{"_id":null,"value":{"$sum":1}}},{"$project":{"_id":0,"value":"$value","group":{"$literal":{}}}},{"$sort":{"value":-1}}]
2025-01-31T11:21:12.606Z | error: {"level":"Error","event":"request","status":500,"method":"GET","path":"/editorials","duration":787,"error":{"message":"PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.","stack":"MongoServerError: PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.
    at t.sendCommand (/opt/nodejs/index.js:279:67726)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async t.command (/opt/nodejs/index.js:279:68372)
    at async t.command (/opt/nodejs/index.js:279:99053)
    at async Vte.executeCommand (/opt/nodejs/index.js:275:71976)
    at async Vte.execute (/opt/nodejs/index.js:275:104791)
    at async tryOperation (/opt/nodejs/index.js:275:69358)
    at async executeOperation (/opt/nodejs/index.js:275:67330)
    at async t._initialize (/opt/nodejs/index.js:279:229500)
    at async t.cursorInit (/opt/nodejs/index.js:279:226219)"}}
✔ Requested 30 logs in the last month
You have received logs from 2025-01-31T11:20:28.728Z to 2025-01-31T11:21:12.606Z

Does this error happen when you simply navigate via pagination (without search) ?

Could you also show the list request that is being sent via the network tab whenever you do the search ?

If nothing else until then, you could try to set the option allowDiskUse:true in your mongo server as suggested by the error, I am not too familiar with Mongo environments but 32MB of allocated memory does not seem that high either.

1 Like

This error does not occur when using paging. To simulate it, you must use the search together. I first search on page one, and then click next.

I recorded a video showing the behavior, along with the set of requests in the network tab.

Here it is: https://drive.google.com/file/d/1PoWkWeaUv7AiAzI15mU1yrPFd_WwUnLP/view?usp=drive_link

Looking a little more about allowDiskUse, it says that it is enabled in each query, so it is not something that I can configure on the server. But it is something that should be used in the query, see the example:

db.collection.aggregate(pipeline, { allowDiskUse: true });

Is this something that you can do?

Thanks for the support and sorry for the delay.

1 Like

Hello @martinshumberto,

After investigating once more, because I do not feel warranted to enable allowDiskUse to all users on the @forestadmin/datasource-mongoose, I suspect that the performance issue on your search comes from the fields in your collections.
Without the search the performance is quite good, even when accessing large collections and a high number of pages. And the response times I observe on your collection without the search are quite fast too.

The search operation we use on Forest will create match operators on all string fields of your collection and nested fields, the match being a regex of the search input.
The regex match is quite consuming and may impact you more when you have a large amount of string fields and if the content is long (especially the field content of editorials which is suspicious to me).

Could you restrict the fields of your search to only the relevant fields of your collection ? following this documentation;