Rails - Export CSV is missing entries

Help me!
1

When exporting a collection from the dashboard some entries are missing at the generated csv. Tested across multiple collections and this issue is happening across multiple collections in my environment.

When the CSV is generated no error is thrown and a blank (only headers) is generated or only a few entries were added to the generated CSV.

This error persists if the collection has some filters applied to it.

Expected behavior

When exporting a collection, the generated CSV should have all the entries for that collection.

Actual behavior

When exporting a collection the generated CSV is missing entries.

I’ve tested across multiple environments, and in all of them, most of the entries are missing. No errors were raised at the backend when generating such CSV

Failure Logs

No errors were thrown when exporting the CSV

Context

  • Rails Version: 5.2
  • PG Gem version: 0.20
  • Forest Liana Gem Version: forest_liana 5.4.2
  • PSQL version: 13.2
2

Hi @felipe.cabezudo,

Sorry to read that you observe issues with the CSV export actions.

If I try to sum things up, whatever the environment, whatever the collection, the exports you trigger generate files that contain only te CSV header line and absolutely 0 data raws. Is that exact?

3

No, for some collections in particular we get a few entries. It also depends on the environment. On production, the generated CSV should have 180 entries but it has none, on my local environment it should have 18K and it has 4.

I’ve tested with other collections and the behavior is similar to the one described above.

4

Ok, it will be really hard to investigate without minimalist reproductions steps.

I also tried this feature on our official live demo (available here: https://app.forestadmin.com/livedemo).
It is running on Rails 5.2 with the forest_liana v5.4.4 and the CSV exports are working as expected.

5

I’ve tried removing all the smart fields for the collection we want to export and the problem persists.

I’ll keep you posted

6

@arnaud How many items has the collection that you are exporting? I’ve tested with another collection that had 15 items and it exported all of them

7

@arnaud I’ve noticed that the Collections experiencing this kind of issue always try to fetch nested associations for each entry. While the ones that export the CSV with all the information required aren’t loading nested associations

8

Also, I’ve noticed that the collections that fail to build up the CSV have way more complex queries than the ones that get all the entries.

In my case the generated query for the failing CSV is

SELECT "parents"."id" AS t0_r0, "parents"."user_id" AS t0_r1, "parents"."created_at" AS t0_r2, "parents"."updated_at" AS t0_r3, "parents"."notes_for_caregiver" AS t0_r4, "parents"."payout_increase" AS t0_r5, "parents"."jobs_count" AS t0_r6, "parents"."shortlists_count" AS t0_r7, "parents"."different_sitters_count" AS t0_r8, "parents"."total_paid_on_subscriptions" AS t0_r9, "users"."id" AS t1_r0, "users"."email" AS t1_r1, "users"."encrypted_password" AS t1_r2, "users"."reset_password_token" AS t1_r3, "users"."reset_password_sent_at" AS t1_r4, "users"."remember_created_at" AS t1_r5, "users"."sign_in_count" AS t1_r6, "users"."current_sign_in_at" AS t1_r7, "users"."last_sign_in_at" AS t1_r8, "users"."current_sign_in_ip" AS t1_r9, "users"."last_sign_in_ip" AS t1_r10, "users"."confirmation_token" AS t1_r11, "users"."confirmed_at" AS t1_r12, "users"."confirmation_sent_at" AS t1_r13, "users"."unconfirmed_email" AS t1_r14, "users"."created_at" AS t1_r15, "users"."updated_at" AS t1_r16, "users"."validation_status" AS t1_r17, "users"."first_name" AS t1_r18, "users"."last_name" AS t1_r19, "users"."middle_name" AS t1_r20, "users"."no_middle_name" AS t1_r21, "users"."tos_accepted" AS t1_r22, "users"."avatar_processing" AS t1_r23, "users"."postal_code" AS t1_r24, "users"."admin" AS t1_r25, "users"."is_active" AS t1_r26, "users"."uuid" AS t1_r27, "users"."authentication_token" AS t1_r28, "users"."activated_on" AS t1_r29, "users"."hear_about_us" AS t1_r30, "users"."enterprise_booker" AS t1_r31, "users"."represented_entity_name" AS t1_r32, "users"."auth_provider" AS t1_r33, "users"."auth_uid" AS t1_r34, "users"."access_token" AS t1_r35, "users"."token_id" AS t1_r36, "users"."tos_acceptance_ip_v2" AS t1_r37, "users"."tos_accepted_v2" AS t1_r38, "customers"."id" AS t2_r0, "customers"."stripe_customer_id" AS t2_r1, "customers"."user_id" AS t2_r2, "customers"."created_at" AS t2_r3, "customers"."updated_at" AS t2_r4 FROM "parents" LEFT OUTER JOIN "users" ON "users"."id" = "parents"."user_id" LEFT OUTER JOIN "users" "users_parents_join" ON "users_parents_join"."id" = "parents"."user_id" LEFT OUTER JOIN "customers" ON "customers"."user_id" = "users_parents_join"."id" WHERE (LOWER("parents"."notes_for_caregiver") LIKE '%%') ORDER BY "parents"."id" ASC LIMIT $1;

If you look closely you should be able to see that in the WHERE condition is checking for the notes_for_caregiver param.

I Ran this query manually and I’m only getting the entries exported on the csv.

If I remove this WHERE clause then I get all the expected results

9

The WHERE condition is an interesting point.
I guess you don’t have default scopes in the parents ActiveRecord model, right?

It could be interesting to see if a search parameter is transmitted by the Forest Admin web client to the admin backend, during the export request, and would lead to the unexpected insert of this kind of conditions.

10

We don’t have a default scope for that model

[8] pry(main)> Parent.default_scoped
 Parent Load (12.7ms)  SELECT "parents".* FROM "parents"

This are the parameters I’m getting at the csv generation request

Parameters: {"fields"=>{"Parent"=>"id,user,created_at,updated_at,notes_for_caregiver,payout_increase,primary_address,customer,different_sitters_count,jobs_count,metro_area,plan,shortlists_count,subscription,total_paid_on_subscriptions", "user"=>"id", "primary_address"=>"id", "customer"=>"id", "metro_area"=>"name", "plan"=>"id", "subscription"=>"id"}, "search"=>"", "filters"=>"", "searchExtended"=>"0", "timezone"=>"America/Montevideo", "filename"=>"parents", "header"=>"id,user,created at,updated at,notes for caregiver,payout increase,primary address,customer,different sitters count,jobs count,metro area,plan,shortlists count,subscription,total paid on subscriptions", "collection"=>"Parent"}

11

Also, We found out that when creating a custom chart for the Parent collection the notes_for_caregiver is the only field listed on the drop-down

image
image606×955 51.1 KB
.

I’ve tried removing the column from my local database and then when generating CSV it had all the expected entries. I couldn’t find any hint of a scope using notes_for_caregiver in our codebase

12

Hi @felipe.cabezudo

It is hard to investigate without a reproductible example. If possible, could you share an empty/fake database DDL (Privately) so I can try to reproduce this issue on my end?
Using a custom project I’m still not able to reproduce this, but since removing this specific field from your database fixes the issue, I think it could be related to the field definition.

Thanks again for your help

13

Hey @felipe.cabezudo,

I’m still not able to reproduce the issue with the provided dump, I’m still getting the complete csv (Around 300 records in my case) :frowning:
Also, I’m not getting the WHERE (LOWER("parents"."notes_for_caregiver"), so, to me, the issue is not related to the database structure.

I might ask much, but is there any chances you can share a reproductible example of this issue?
Looking at the forest-rails in relation with your database structure, I can’t find any reason for this where condition to appear.

Thanks again for your help