forestadmin.datasource_toolkit.exceptions.DatasourceToolkitException: 🌳🌳🌳Column not found

Feature(s) impacted

DataSource

Observed behavior

I have been working on migrating our forestadmin app from a legacy django-forestadmin agent to the newer forestadmin-agent-django. During the migration process, I have encountered this datasourcetooklitexception which seems to be a newer part of the agent.

Expected behavior

I am expecting the data to be loaded without such exceptions.

Failure Logs

Context

  • Project name: GumboUI
  • Team name: DepMap
  • Environment name: DEVELOPMENT | NAYEEMAQUIB (TEST-FEATURE-1)
  • Agent (forest package) name & version:
    forestadmin-agent-django 1.4.0b2
    forestadmin-agent-toolkit 1.4.0b2
    forestadmin-datasource-django 1.4.0b2
    forestadmin-datasource-toolkit 1.4.0b2
  • Database type: PostgreSQL
  • Recent changes made on your end if any: Followed the migration steps provided here: Migration steps - Python Developer Guide

Question: Is “id” hardcoded somewhere in the agent’s codebase as the default primary key?

Hi @naquib,

Thanks for your message and welcome to our community :raised_hands:

An hardcoded id could make this error, but I just checked and I’m able to create a model with a primary key not named id.

Can you share with us the HTTP request (and it’s parameters) that generate this error ?
Do you have any customizations on the agent side related to your patient collection ?

Regards.

Hi Julien,

Thanks for your reply!

Following is the http request:

[07/Feb/2024 19:44:56] “GET /forest/model?timezone=America%2FNew_York&fields%5Bage_category%5D=term&fields%5Bcell_line_received%5D=id&fields%5Bconsent_2015%5D=id&fields%5Bdbgap%5D=id&fields%5Bdepmap_model_type%5D=id&fields%5Bengineered_model%5D=id&fields%5Bgrowth_pattern%5D=id&fields%5Bmodel%5D=model_id%2Cpatient_id%2Ccell_line_name%2Cstripped_cell_line_name%2Ccell_line_in_stock%2Ccell_line_ordered_date%2Ccomments%2Cage%2Cage_category%2Csex%2Cproposed_release_date%2Cproposed_deliverable%2Cprimary_or_metastasis%2Csample_collection_site%2Csource_type%2Csource_detail%2Corspid%2Cconsent_2015%2Cconverge_id%2Crrid%2Cdepmap_model_type%2Cmolecular_subtype%2Cncit_code%2Cncit_subtype%2Cnew_molecular_subtype%2Cnew_histological_subtype%2Ctreatment_status%2Cgrowth_pattern%2Conboarded_media%2Cplate_coating%2Conboarded_doubling_time%2Cengineered_model%2Ccell_line_received%2Cdate_cell_line_received%2Ccell_line_aliases%2Ctissue_origin%2Ccosmic_id%2Cwtsi_master_cell_id%2Cpart_of_prism%2Cccle_name%2Cccle_line%2Cstated_race%2Cvendor_link%2Ccatalog_number%2Cderived_outside_us%2Cpeddep_line%2Cpeddep_subgroup%2Csj_compbio_id%2Cmodel_transfer%2Cmodel_transfer_comments%2Cmodel_data_sharing%2Cmodel_data_sharing_comments%2Cdate_model_derived%2Cdate_first_publication%2Cfirst_publication_link%2Cdbgap%2Cdate_shared_in_dbgap%2Cpublic_comments%2Cpermission_to_release%2Csub_subtype%2Cpatient_subtype_features%2Cmodel_subtype_features%2Cmodel_derivation_material%2Ctreatment_details%2Cstage%2Ctumor_regression_score&fields%5Bmodel_data_sharing%5D=id&fields%5Bmodel_derivation_material%5D=id&fields%5Bmodel_transfer%5D=id&fields%5Bmolecular_subtype%5D=id&fields%5Bonboarded_media%5D=id&fields%5Bpart_of_prism%5D=id&fields%5Bpatient_id%5D=id&fields%5Bpeddep_subgroup%5D=id&fields%5Bplate_coating%5D=id&fields%5Bprimary_or_metastasis%5D=id&fields%5Bsample_collection_site%5D=id&fields%5Bsex%5D=id&fields%5Bsource_type%5D=id&fields%5Bstated_race%5D=id&fields%5Bsub_subtype%5D=id&fields%5Btissue_origin%5D=id&fields%5Btreatment_status%5D=id&page%5Bnumber%5D=1&page%5Bsize%5D=15 HTTP/1.1” 500 145

No customization regarding the patient collection.

Hi,
Your HTTP request is asking the field id that doesn’t exist in you collection patient; that is the source of the issue.

Could you:

  • Copy as curl the request you just share with us and modify &fields%5Bpatient_id%5D=id by &fields%5Bpatient_id%5D=patient_id and share with us the result or stacktrace this new request generate?
  • Could you try to onboard your django project on a new empty forest project without existing customization and tell us if you face the same issue?

Regards.

Hi Julien,
Thanks for your reply.
Sure. With that particular change the curl request returns None.
Stacktrace:

“GET /forest/model?timezone=America%2fNew_York&fields[age_category]=term&fields[cell_line_received]=id&fields[consent_2015]=id&fields[dbgap]=id&fields[depmap_model_type]=id&fields[engineered_model]=id&fields[growth_pattern]=id&fields[model]=model_id%2cpatient_id%2ccell_line_name%2cstripped_cell_line_name%2ccell_line_in_stock%2ccell_line_ordered_date%2ccomments%2cage%2cage_category%2csex%2cproposed_release_date%2cproposed_deliverable%2cprimary_or_metastasis%2csample_collection_site%2csource_type%2csource_detail%2corspid%2cconsent_2015%2cconverge_id%2crrid%2cdepmap_model_type%2cmolecular_subtype%2cncit_code%2cncit_subtype%2cnew_molecular_subtype%2cnew_histological_subtype%2ctreatment_status%2cgrowth_pattern%2conboarded_media%2cplate_coating%2conboarded_doubling_time%2cengineered_model%2ccell_line_received%2cdate_cell_line_received%2ccell_line_aliases%2ctissue_origin%2ccosmic_id%2cwtsi_master_cell_id%2cpart_of_prism%2cccle_name%2cccle_line%2cstated_race%2cvendor_link%2ccatalog_number%2cderived_outside_us%2cpeddep_line%2cpeddep_subgroup%2csj_compbio_id%2cmodel_transfer%2cmodel_transfer_comments%2cmodel_data_sharing%2cmodel_data_sharing_comments%2cdate_model_derived%2cdate_first_publication%2cfirst_publication_link%2cdbgap%2cdate_shared_in_dbgap%2cpublic_comments%2cpermission_to_release%2csub_subtype%2cpatient_subtype_features%2cmodel_subtype_features%2cmodel_derivation_material%2ctreatment_details%2cstage%2ctumor_regression_score&fields[model_data_sharing]=id&fields[model_derivation_material]=id&fields[model_transfer]=id&fields[molecular_subtype]=id&fields[onboarded_media]=id&fields[part_of_prism]=id&fields[patient_id]=patient_id&fields[peddep_subgroup]=id&fields[plate_coating]=id&fields[primary_or_metastasis]=id&fields[sample_collection_site]=id&fields[sex]=id&fields[source_type]=id&fields[stated_race]=id&fields[sub_subtype]=id&fields[tissue_origin]=id&fields[treatment_status]=id&page[number]=1&page[size]=15 HTTP/1.1” 401 4

Which makes me think if there’s any issue with authentication. I can log in to forestadmin normally; then in the UI I can see that the request failed for some tables. For some tables, the request does go through and I get a 200.

With that being said, we used to use the get_token function from django_forest.utils(old agent) to request middleware. Is there an equivalent function to that in the new agent? Or did anything change there(e.g. encoding algorithm)?

I will be out of office for the next couple of weeks. So I will circle back after that.

Best,
Nayeem

Hi,
I expected an error, but not an authentication error.

For your requests that are not working, is it the same error that occurred in your project (column not found collection.unknown_field)?

The get_token function doesn’t exist anymore, instead in all custom function called by the agent, there is a context parameter which contain a caller attribute which is the user decoded as the same way of get_token. Here is an example when using hooks, context.caller is an User object.

We’re still interesting to know if this error appear on a new and empty forest project.

Regards,
Julien.

1 Like

Hi @naquib
I hope you’re doing well. I was wondering if you were back at work and had time to check out the above solution?

Thanks

Hi @julien.barreau Sorry for the late reply. I got pulled away to other projects. Unfortunately, I haven’t had the time to implement the solution you mentioned. I will try to do that next week. When I do, I will let you know.

Thank you for following up! I appreciate it.

2 Likes

Hello @naquib, hope you’re well! Do you have some time next week to adress this issue with @julien.barreau during a tech session? We’ll close this thread in the meantime.

Talk soon,
Julien