Smart relationship, various questions about the API

Hi! Hope you’re having a good day.

There are two parts to my topic: General understanding of the Forest records API and a specific code example.

General understanding

  1. Why do we use RecordsGetter and others instead of the ORM (Sequelize in our case)? Is it for authentication, a simpler API, possibly other purposes? We think we get the general idea but want to make sure =)
  2. Following on the first question, are there instances when it would be more appropriate to use the ORM directly? Our use of FA is mostly editorial, any heavy lifting / business critical logic is handled by our backend via jobs.
  3. There is a fairly old GitHub issue about whether you are interested in any commits / improvements over the TypeScript definitions. This is something which would greatly enhance the developer experience, as we regularly have to dive into code examples or the internals to even understand which parameters to pass to an API method. Are you willing to work on this topic or take contributions? If so, in which form?
  4. What care do issues and PR receive on forest-express-sequelize and other repos? Some PRs have been opened and inactive for years.

Code

Now let’s take the following simplified example, in which a user can participate in events. There is a one-to-many relationship between users and participants. We’ve built a smart relationship to get participants based on the user email.

This is a trivial example to demonstrate our use of the API.

Note that some of the questions below may be based on wrong premises or a poor understanding on our part, comments and criticisms are very welcome.

router.get('/users/:user_id/relationships/participants', async (request, response, next) => {
  try {
    const userId = request.params.user_id
    const usersGetter = new RecordGetter(Users, request.user, request.query)
    const { email } = await usersGetter.get(userId)

    const participantsGetter = new RecordsGetter(Participants, request.user, request.query)
    const query = {
      filters: JSON.stringify({ // (1)
        aggregator: 'and',
        conditions: [{
          field: 'email',
          operator: 'equal',
          value: email
        }, {
          field: 'accountId',
          operator: 'equal',
          value: '0'
        }]
      }),
      page: {
        number: request.query.page.number,
        size: request.query.page.size
      }
    }
    const participants = await participantsGetter.getAll(query)

    const participantsCounter = new RecordsCounter(Participants, request.user, {
      ...request.query,
      filters: query.filters,
      searchExtended: '0'
    })

    response.send(await participantsGetter.serialize(participants, { // (2)
      count: await participantsCounter.count(), // (3)
      page: {
        number: request.query.page.number,
        size: request.query.page.size
      }
    }))
  } catch (e) {
    next(e)
  }
})

We have several questions about the above code:

  1. As you can see in the code at (1), we have to pass a string of filters instead of the actual object. This has to be done this way because the parameter is JSON.parsed if it exists, in base-filters-parser.js. It feels very strange to pass filters this way, particularly considering this bonks all type information.
    a. Speaking of types, they specify an object and not a string (filters?: Filter|AggregatedFilters;). Is the behaviour of the filters parser a bug?
  2. At (2), we pass a second object to our serializer to carry page and count information. This is done both against the documentation and the types specified by forest-express-sequelize.
    It is the only way to show page and count information on the front-end, otherwise we see no count below the table and a 1 of NAN display at the bottom.
  3. At (3), we were somewhat expecting to use count like counter.count(myFilters) instead of having to rebuild a query object, but this may be a wrong expectation on our part.

Context

  • Package Version: 8.0.9
  • Express Version: 4.17.1
  • Sequelize Version: 6.6.5
  • Database Dialect: MSSQL
  • Database Version: -
  • Project Name: -

Thanks for your help =)

2 Likes

Hi @kll :wave:

A lot of questions here :smiley:
I’ll answer the first part of your message, and I’ll sync with the team for the ts related questions.

General understanding

  1. Why do we use RecordsGetter and others instead of the ORM (Sequelize in our case)? Is it for authentication, a simpler API, possibly other purposes? We think we get the general idea but want to make sure =)

You can, of course, use both. However, RecordsGetter (And all the Record* you may encounter) implement some useful feature you’ll not have to code by yourself.
Eg. Using the v8, RecordsGetter automatically validate scope condition (That you may have setup on your forestadmin UI) where using the ORM directly will force you to reimplement it.
Eg 2. RecordsGetter also have a method getIdsFromRequest and serialize that are especially handy when you deal with smart actions/…

  1. Following on the first question, are there instances when it would be more appropriate to use the ORM directly? Our use of FA is mostly editorial, any heavy lifting / business critical logic is handled by our backend via jobs.

The only instance that came to my mind while reading your question is when you have to deal with raw SQL queries. There might be other cases where you may need to use Sequelize directly, but as a general rule, I would suggest to stick as much as possible to Record(s)* classes, and fallback to Sequelize functions when our implementation does not suit your needs.

  1. There is a fairly old GitHub issue about whether you are interested in any commits / improvements over the TypeScript definitions. This is something which would greatly enhance the developer experience, as we regularly have to dive into code examples or the internals to even understand which parameters to pass to an API method. Are you willing to work on this topic or take contributions? If so, in which form?

Actually, this is a pretty hot topic right now! Our v7 and v8 packages for forest-express-sequelize & forest-express-mongoose now declare types directly within our package (We where using definitely-typed for our previous version).
The issue you mentionned is really old and was never cleaned, but I can assure you that types should now be up-to-date.
We do accept contribution over the typescript types. However, due to recent changes and since you seems to have deep dive into our code, you’ll see that our base package forest-express does not currently have types. We are still working on this point but contributions on types should be done on both forest-express-mongoose and forest-express-sequelize packages.
If you were to open a PR about types on forest-express-sequelize, a PR on forest-express-mongoose is always appreciated :slight_smile:

  1. What care do issues and PR receive on forest-express-sequelize and other repos? Some PRs have been opened and inactive for years.

Issues and PR on Github depends a lot of their content. We tend to redirect all theses to here when we can. If you encounter any issues while working with forest-express-sequelize, I would suggest to post it here, and you should get a response pretty quickly.

Code

  1. As you can see in the code at (1) , we have to pass a string of filters instead of the actual object. This has to be done this way because the parameter is JSON.parsed if it exists, in base-filters-parser.js . It feels very strange to pass filters this way, particularly considering this bonks all type information.
    a. Speaking of types, they specify an object and not a string ( filters?: Filter|AggregatedFilters; ). Is the behaviour of the filters parser a bug?

I don’t think you need to pass JSON.stringify(yourQuery) in the getAll() function. Normally, RecordsGetter should already have the filters given - Still, I’ll double check if that’s the case or not.

At (2), we pass a second object to our serializer to carry page and count information. This is done both against the documentation and the types specified by forest-express-sequelize.
It is the only way to show page and count information on the front-end, otherwise we see no count below the table and a 1 of NAN display at the bottom.

Nice catch. Indeed, our serializer needs an update to support smart relationships (optionnal params)

At (3), we were somewhat expecting to use count like counter.count(myFilters) instead of having to rebuild a query object, but this may be a wrong expectation on our part.

Indeed, this might feel strange because for the records counter, filters are automatically picked up from the params passed in the counter class instanciation.

Our typescript integration without definitely typed is actually pretty recent, so feedback like theses are really precious.
I’ll open a ticket on our end for these to get fixed.

Let me know if you still have any questions :pray:

2 Likes

Hey @jeffladiray! Thanks for your reply, lightning-fast as usual =)

General understanding

Got it! Thanks for your replies, they were extremely clear and helpful. The explanation about Record(s)* may deserve its own page on the docs, to explain what are all these helpers and why they should be used against the ORM that developers might be more familiar with.

There may be some work done on forest-express-* on our end, we’ll make sure to upstream it.

Code

(1) Normally, RecordsGetter should already have the filters given

Do you mean they should exist because they were passed as the HTTP query? These filters are actually static ones we need in our business logic, they do not appear on the front-end. They define what the smart relationship actually is, in this case.

Also, what about the issue of having to JSON.stringify the filters despite the types saying otherwise? This was pretty surprising when coding.

(3) Indeed, this might feel strange because for the records counter, filters are automatically picked up from the params passed in the counter class instanciation.

Makes sense for queries coming from the frontend, but it’s sort of awkward when used from the backend. Maybe allow passing a filter to count directly? Not that it’s really a big deal tbh, it just made the discovery slightly harder…

For any further feedback on TypeScript, should we direct them here or on GitHub? Or both =)

Code

At first sight, (1) & (3) looks pretty weird indeed (Especially the JSON.stringify & Filter|AggregatedFilters one). But, as stated, our TS integration is pretty recent so…

I’ll need to deep dive a little bit more in our code to provide a more detailed explanation of what’s going on. Depending on the outcome, I’ll update my answer :slight_smile:

For any further feedback on TypeScript, should we direct them here or on GitHub? Or both =)

If you have some code to share, I would suggest to open a github PR. If you have general feedback (On TS or not) I would suggest to open a thread here. As a general rule, threads open here are most likely to have a “fast” answer.

Let me know if that helps :pray:

1 Like

Hey @kll,

After some more in-depth look in our code base:

For (1) you are totally right, and I created a ticket on our end. The best workaround for now would be to JSON parse request.query.filters, add you custom filters, then instanciate RecordsGetter and use getAll with no parameters at all. Not an ideal solution, but everything should work as expected using this trick.

For (3), as stated, the types are actually very recent on our end. They shown us a lot a issues like that here and there, and hopefully these kind of issues will be fixed overtime.

Again, thanks a lot for you valuable feedback :pray:

1 Like

Hiya @jeffladiray,

Thanks for your replies. We’re currently moving our entire Forest project to TypeScript, we’ll compile all our feedback and let you know how it went at the end =)

There’s already some ts-ignore we had to spread here and there, and clear improvement paths.

More to come soon =D

1 Like

Good to know. If you don’t have any others questions related to the original topic, I’m closing this thread.

However, don’t hesitate to open a new one if you have anything to share :raised_hands:

Thanks again for your great feedback.

1 Like

It is very sad to see this : fix(types): fix typing errors and limitations by romain-gilliotte · Pull Request #799 · ForestAdmin/forest-express-sequelize · GitHub

Particularly the following change:
diff screenshot

This is barely better than any, and doesn’t carry any useful type information…

Wouldn’t it have been way better to actually fix the broken implementation that expects a string? From an external point of view, it seems that a typeof check would be enough to avoid very stupid call site implementations, as a reminder the first point in OP was how eerie having to stringify a filter object was…

I mean, okay at least it’s one less ts-ignore, but that seemed like the very easy and very weird solution to this problem.

Why not just fix parseFiltersString to return the object if it is one? It wouldn’t change anything to string or falsey value calls and would allow having sane call sites.

The end code could look something like:

const parseFiltersString = (filters) => {
  if (!filters) return null

  if (typeof filters === 'string') {
    try {
      return JSON.parse(filtersString);
    } catch (error) {
      throw new InvalidFiltersFormat('Invalid filters JSON format');
    }
  }

  return filters
};

Depending on your call scenarios (which are impossible to know without very deep introspection since it’s not typed), this could be retroactively compatible.

1 Like

This is barely better than any , and doesn’t carry any useful type information…

I agree, we made this to at least resolve the ts-ignore issue.

Also, we though about your suggestion while discussing a potential solution for this issue (When I filed the bug in our bug tracker). Your initial message revealed an issue with the getAll implementation.

The fix for the issue is actually a breaking change for users using RecordsGetter.getAll, so the PR you are mentioning is actually the “best” solution we are able to provide for the v8.

It is planned to be fixed in the v9 however (No ETA for now), with a few others issues that our new typescript types showed us.

Sorry for this inconvenience, but be sure we took into account everything pointed out in this thread :pray:

1 Like