Upload file to Google Cloud Storage

[Emixam23]

Hey guys,

I am coming today because I am stuck on an upload feature i am trying to integrate. As usual, i am quite sure you will be able to help me out, so I open this topic for the next people struggling on uploading file to Google Cloud Storage.

So I followed this tutorial of yours (SQL + AWS S3) but i adapted it to MongoDB + GCS

Expected behavior

So what I should be able to is:

• For a given item, upload its picture
• While uploading the file, I want its _id file to be his name (i.e.: x1y2z3.jpg if the item id is x1y2z3)

Actual behavior

I created a smart action
forest/items.js

{
        name: 'Upload item picture',
        endpoint: '/forest/actions/items/upload-item-picture',
        httpMethod: 'POST',
        type: 'single',
        fields: [{
            field: 'Item picture',
            description: 'The picture of the item in format xxxxxxxxx .........',
            type: 'File',
            isRequired: true
        }]
    }

I have my model
models/items.js

// This model was generated by Lumber. However, you remain in control of your models.
// Learn how here: https://docs.forestadmin.com/documentation/v/v6/reference-guide/models/enrich-your-models
const mongoose = require('mongoose');

// This section contains the properties of your model, mapped to your collection's properties.
// Learn more here: https://docs.forestadmin.com/documentation/v/v6/reference-guide/models/enrich-your-models#declaring-a-new-field-in-a-model
const schema = mongoose.Schema({
  // ...  
  'image_url': String,
  // ...
}, {
  timestamps: false,
});

module.exports = mongoose.model('items', schema, 'Items');

I also created the route:
routes/items.js

// Upload picture to gcs
router.post('/actions/upload-item-picture', permissionMiddlewareCreator.smartAction(), (req, res) => {

    // Get the current item id
    console.log("---------------------------------------------------------")
    console.log("id: "+request.params.recordId)

    res.status(204).send();
});

Failure Logs

When (in my smart action) I selected the file and then click on “Upload item picture”, I get an error and the following output:

Unexpected error with Upload item picture action.

POST /forest/actions/items/upload-product-picture 413 63 - 1.130 ms
GET /forest/items/x1y2z3 304 - - 54.423 ms

Context

    "@google-cloud/storage": "^5.7.4",
    "@types/express": "^4.17.11",
    "@types/forest-express-mongoose": "^6.3.3",
    "@types/mongoose": "^5.10.3",
    "@types/node": "^14.14.25",
    "body-parser": "1.19.0",
    "chalk": "~1.1.3",
    "cookie-parser": "1.4.4",
    "cors": "2.8.5",
    "debug": "~4.0.1",
    "dotenv": "~6.1.0",
    "express": "~4.17.1",
    "express-jwt": "5.3.1",
    "forest-express-mongoose": "^6.0.0",
    "mongoose": "~5.8.2",
    "morgan": "1.9.1",
    "nodemon": "^2.0.7",
    "require-all": "^3.0.0",
    "tsc-watch": "^4.2.9",
    "typescript": "^4.1.3"

Note

Once it works, and if you agree, I would like to propose this topic’s answer to be added to the doc as you did for SQL/AWS S3. Of course, the answer will have to be transformed into a proper documentation

Thanks for your help

[vince]

Hi @Emixam23,

Thanks for detailing that much it really simplify everything for people trying to help you

You have a 413 which means your payload is too large.
You need to add/update this to your app.js:

app.use(bodyParser.urlencoded({ extended: true ,limit: '50mb' }));
app.use(bodyParser.json({ limit: '50mb' }));

[Emixam23]

Hey

Thanks for your feedback, I could move forward and everything is almost ok, but one last point.
In fact, the default approache of GCS and its nodejs SDK is bucket.upload(...) [EXAMPLE]

However, as you saw, it takes as input the filepath which isn’t provided (as far as I could investigate) in the data that you get throughout req.body.data.attributes.values[...] in the backend route.

So I tried a different approach but… it doesn’t work, even if I think I am close.
I worked on Google Storage part and found some code in order to directly upload the file content:

async put(input: string, extension: MediaExtension, destinationFilename: string) {
    // load local file
    let bufferStream = new stream.PassThrough();
    bufferStream.end(Buffer.from(input, 'base64'));

    console.log("input: " + input.substring(0, 250))

    // init gcs file object
    let file = this.bucket.file(destinationFilename);
    bufferStream.pipe(file.createWriteStream({
        metadata: {
            contentType: 'image/' + toStr(extension),
            metadata: {
                custom: 'metadata'
            }
        },
        public: false,
    }))
        .on('error', function (err) {
            console.log("error: "+err.message.substring(0, 250))
        })
        .on('finish', function () {
            console.log("finish")
        });
}

The above function works as expected and do upload the file. However, once uploaded, the file seems invalid… So I dug and found something weird.

I am not a NodeJS expert (not NodeJS dev at all actually ahah) but as far as I could debug, it seems that the File Picker actually sends the content of the file with some data that looks like header(s). The thing is, I have no idea about how to debug/parse this format (based on the piece of code above):

input: data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsNFBQUFBQ
UFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBT/wAARCAhwEAADARE......

So as I said, maybe I am not taking the input the right way but as I said, I am not able to find the format of the input file, which makes it hard for me to upload it the right way using GCS DSK

I really want to find out because I do think that some user(s) could be really interested into this solution (using Google Cloud Sotrage)

Thanks

Max

[remi]

Hello Maxime and thanks for this update!

I may be mistaken, but from the documentation about GCS I just read online, your code looks fine to me. However the custom: 'metadata' is a code snippet from this stackoverflow post I guess, that you don’t need at all. The author added it just to say that if you want to add metadata, it needs to be here.

What do you mean with “the file is invalid” ?

The input string is a standard image encoded in base64 (this is why you use Buffer.from(input, 'base64'), so there’s nothing wrong here

[Emixam23]

Hello

For some reason I need metadata, sorry if it confused you, it wasn’t relevant in my example
Soo yes, the method above works fine but… I need to remove a piece of the input file to make it “valiid”.

The thing is, upload a jpg image using the above code will result of having an image on GCS that is not openable, invalid. (you get the image icon but the image is broken)

The solution I found is to remove the begining of the file containing the infos of the file

// ...
let fileContent = removeHeaderFromFile(productImage)

export function removeHeaderFromFile(input: string): string {
    let regex = new RegExp('data:(.+);(.+),');
    let r  = input.match(regex);
    if (r)
        console.log(r[0]); //=> "data:image/jpeg;base64,"
    input = input.replace(r[0], "")
    return input
}