Streaming file uploaded from Forest (with Action)

Help me!
1

Feature(s) impacted

Uploading file from Smart Action.

Observed behavior

If the CSV file sent is too large, my Heroku app will crash because it requires too much memory (NodeJS / Express / MongoDB / Mongoose)

Expected behavior

The smart action calls my API route, and I can stream the CSV file to avoid loading the entire file into memory.

Context

  • Agent technology: NodeJS (Express)
  • Agent (forest package) name & version: forest-express-mongoose (9.3.17)
  • Database type: MongoDB, Mongoose

Message

Bonjour tout le monde !
J’ai cherché sur le forum sans résultat, donc je me permets de créer ce topic. Je voulais savoir s’il était possible de streamer (ou d’unee manière) un fichier volumineux envoyé depuis une smart action ?

J’ai mon app Heroku crash dès lors que j’envoie un fichier trop volumineux, car je charge l’ensemble du CSV en mémoire. Étant donné que je reçois la donnée sous base64 dans son ensemble, il me semble qu’à partir de ce moment-là j’ai déjà tout en mémoire et c’est trop tard.

I defined my Smart Action like this:

{
        name: 'Import Finance Deals',
        endpoint: '/forest/importCSV',
        type: 'single',
        fields: [{
            field: 'CSV file',
            type: 'File'
        }]
    }

My API route

router.post(
  '/forest/importCSV',
  middleware.asyncWrap(async (req: express.Request, res: express.Response) => {
    try {
      const csvDataUri = req.body.data.attributes.values['CSV file']
      const parsed = parseDataUri(csvDataUri)
      const csvString = parsed.data.toString().replace('\ufeff', '')
      const parsedCSV = await parseCSV(csvString, {
        columns: true,
        skip_empty_lines: true,
        delimiter: ','
      })
      //Some stuff here
      res.send({ success: `Found ${deals.length} deals from ${store.name}!` })
    } catch (error) {
      res.status(400).send({ error: error.message })
    }
  })
)

Here is the body of my smart action:

{
  "data": {
    "attributes": {
        "ids": ["65ef715e1e5f59fffdb7cd2b"],
      "values": {
        "CSV file": "//base 64 file"
      }
    }
  }
}

How could I stream the file as it goes?
Thank you very much for your help!

2

Hello,
Unfortunately smart action doesn’t allow to use stream to upload large file on your server.
I push your request to the product board to be processed.
An alternative is to use S3 AWS service
We apologize for the inconvenience,
Best regards,
Alban