Performance optimization request

Help me!
1

Hi team,

This is not really a feature suggestion, this is why I tagged it as “Help me”.

While continuously looking at improving Forest admin performances, I have noticed that all relationships of a collection are returned even when only some of them are used.

Here is my use case.
I have a users collection which has a lot of relationships.
When displaying the list of users, here is a query example (url decoded):

http://127.0.0.1:3002/forest/users
	?fields[Birth_country]=name_en
	&fields[card]=CHID_and_status
	&fields[kyc]=Onboarding
	&fields[sponsor]=fullname
	&fields[subscription]=Subscription
	&fields[users]=created_at,email,firstname,lastname,card,CardOrdered_Date,kyc,compliance_score,fraud_suspicion_level,nb_accounts_same_device,status,language,last_balance,subscription,default_currency,phone_number,current_address,sponsor,nbSponsored,CurrentCountry,city,Birth_country,CardActivated_Date,birthdate
	&filters={"aggregator":"and","conditions":[{"field":"email","operator":"not_contains","value":"jbtest"},{"field":"deleted_at","operator":"blank","value":null}]}
	&page[number]=1&page[size]=20&search=test test&searchExtended=0&sort=-created_at&timezone=Europe/Paris

As you can see, only 5 relations are used in the main list.

However, when looking at the result returned by my server, I can see that all of my relations are returned.
Here is an excerpt:

{
    "data": [{
            "type": "users",
            "id": "abcd167b-630c-46e2-b67c-0358e923f4e7",
            "attributes": {
                "CardOrdered_Date": null,
                "CardActivated_Date": null,
                "Completed_KYC_Date": null,
                "CurrentCountry": "France",
                "nbSponsored": "0",
                "CreationStatus": "Waiting account creation",
                "LastContact": "(not found)",
                "id": "abcd167b-630c-46e2-b67c-0358e923f4e7",
                "created_at": "2020-12-08T23:57:55.268Z",
                "email": "test2v2@test.com",
                "firstname": "Test",
                "lastname": "Test",
                "birthdate": "1987-12-12",
                "default_currency": "EUR",
                "phone_number": "+33614141411",
                "language": "en",
                "status": 1,
                "compliance_score": 3,
                "fraud_suspicion_level": 2,
                "city": "Paris",
                "current_address": "12 rue de la paix Chez michou 75002 Paris - France",
                "nb_accounts_same_device": "21",
                "last_balance": null
            },
            "relationships": {
                "addresses": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/addresses"
                        }
                    }
                },
                "pfs_check_results": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/pfs_check_results"
                        }
                    }
                },
                "Birth_country": {
                    "data": {
                        "type": "cod_countries_labels",
                        "id": "FR"
                    },
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/Birth_country"
                        }
                    }
                },
                "cardTokens": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/cardTokens"
                        }
                    }
                },
                "contacts": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/contacts"
                        }
                    }
                },
                "recipients": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/recipients"
                        }
                    }
                },
                "transactions": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/transactions"
                        }
                    }
                },
                "scheduled_transfers": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/scheduled_transfers"
                        }
                    }
                },
                "payout_transfers": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/payout_transfers"
                        }
                    }
                },
                "kyc": {
                    "data": {
                        "type": "kyc",
                        "id": "1fd6ecba-eb69-42e7-b3c1-cb2365fe0d7f"
                    },
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/kyc"
                        }
                    }
                },
                "device_details": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/device_details"
                        }
                    }
                },
                "notifications": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/notifications"
                        }
                    }
                },
                "topups": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/topups"
                        }
                    }
                },
                "subscription": {
                    "data": {
                        "type": "subscriptions",
                        "id": "e9364530-278f-4d7d-a469-1cf2d3fa2b52"
                    },
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/subscription"
                        }
                    }
                },
                "card": {
                    "data": null,
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/card"
                        }
                    }
                },
                "crm_user_exchanges": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/crm_user_exchanges"
                        }
                    }
                },
                "sponsored": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/sponsored"
                        }
                    }
                },
                "sponsor": {
                    "data": null,
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/sponsor"
                        }
                    }
                },
                "user_ab_testings": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/user_ab_testings"
                        }
                    }
                },
                "alert_logs": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/alert_logs"
                        }
                    }
                },
                "vstat_transactions_months": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/vstat_transactions_months"
                        }
                    }
                },
                "vouchers": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/vouchers"
                        }
                    }
                },
                "userAdminLogs": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/userAdminLogs"
                        }
                    }
                },
                "user_ip_logs": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/user_ip_logs"
                        }
                    }
                },
                "cgu_agreements": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/cgu_agreements"
                        }
                    }
                },
                "partnerSenderInfos": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/partnerSenderInfos"
                        }
                    }
                },
                "eligible_discounts": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/eligible_discounts"
                        }
                    }
                },
                "discounts_applied": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/discounts_applied"
                        }
                    }
                },
                "devices": {
                    "links": {
                        "related": {
                            "href": "/forest/users/abcd167b-630c-46e2-b67c-0358e923f4e7/relationships/devices"
                        }
                    }
                }
            }
        }, {
            "type": "users",
            "id": "3bb28b15-c7d2-4792-bf8d-9c836eeeb74b",
            ...
			... following lines ...
            ...
       }
    ],
    "included": [{
            "type": "cod_countries_labels",
            "id": "FR",
            "attributes": {
                "country_code": "FR",
                "name_en": "France"
            },
            "relationships": {}
        }, {
            "type": "kyc",
            "id": "1fd6ecba-eb69-42e7-b3c1-cb2365fe0d7f",
            "attributes": {
                "Onboarding": "SDD/Waiting KYC docs",
                "id": "1fd6ecba-eb69-42e7-b3c1-cb2365fe0d7f",
                "kyc_level": 1,
                "validation_status": 2,
                "proof_of_id": 0,
                "proof_of_selfie": 0
            }
        }, {
			...
			... following relations included ...
			...
    ]
}

Fortunately, the included part does contain only useful data.

As you can see, 29 relations are returned.
And each one will be returned for my 20 records.

For 20 users, the size of the returned data is more than 100K Bytes.
I have made a test by removing useless relationships, and the whole size decreased to 36K Bytes.
=> We could earn around 30% of the payload!

Now, we could suspect that the delay between the data returned, and its display in Forest could be easily improved.
I guess that the browser would need far more time to parse 100K of data rather than 36K.

What is your opinion?

Hoping that it will help you.

Best regards.

My setup:

    "database_type": "postgres",
    "liana": "forest-express-sequelize",
    "liana_version": "6.6.3",
    "engine": "nodejs",
    "engine_version": "12.13.1",
    "orm_version": "5.22.3"

PS. Please feel free to tag this ticket differently if needed.

3 Likes
2

Hi @Louis-Marie,

This is a super interesting topic.

What is your opinion?

I guess you should not be having this behavior if the data are not required by the table view.

What I find weird though is that I cannot reproduce the behavior you are having (But my setup might not reflect yours exactly). Hiding a column in the table view does indeed remove it from the relationship related data.

Your feedback has been shared with the team though.

I really would like to have a reproductible setup for this, so if that’s possible:

  • Was the route edited on your side or is it a classic/untouched forest-express-sequelize generated route?
  • Do you use something like this example of our documentation?
  • Are some of theses relation smart relationships ?

Any piece of informations that could help use reproduce this would help :pray:

3

Hi @jeffladiray ,

  • Was the route edited on your side or is it a classic/untouched forest-express-sequelize generated route?
  • Do you use something like this example of our documentation?

That was the case, but I commented all the customized routes to make a new test, and I have the same result.

  • Are some of theses relation smart relationships ?

No, there is no smart relationship from my users collection.

A new note:
It seems that the problem could come from the hasMany associations.
They should not appear in the result, but they do.
Example: db.users.hasMany(db.vouchers);

Hoping this will help.

Thanks

4

A new note :
It seems that the problem could come from the hasMany associations.
They should not appear in the result, but they do.
Example: db.users.hasMany(db.vouchers);

That was totally what I was missing :pray:
I’m able to reproduce on my end as well. I’ll look in the code to check if this is done for a specific reason or if that’s something we missed. I’ll update this thread if I can find anything interesting.

Thanks for your help