ElasticSearch Smart Collection Search

Help me!
1

Hi,

I have followed the documentation below to create a smart collection through ElasticSearch:

I was able to create the 2 basic get routes but I am stuck on how to implement the search capability. I have set isSearchable to true in my forest file. How do I modify this route to enable the search function:

router.get('/logs', async (request, response, next) => {
try {
  const pageSize = Number(request?.query?.page?.size) || 20;
  const page = Number(request?.query?.page?.number) || 1;
  const search = request.query?.search;
  const options = { timezone: request.query?.timezone };

  let filters;
  try {
    filters = request.query?.filters && JSON.parse(request.query.filters);
  } catch (e) {
    filters = undefined;
  }

  const result = await elasticsearchHelper.functionSearch({
    page,
    pageSize,
    filters: filters || undefined,
    options,
  });

  const serializer = new RecordSerializer({ name: 'logs' });

  response.send({
    ...await serializer.serialize(result.results),
    meta: {
      count: result.count,
    }
  });
} catch (e) {
  next(e);
}

});

Dependecies:

"axios": "^0.21.1",
 "body-parser": "1.19.0",
"chalk": "~1.1.3",
"cookie-parser": "1.4.4",
"cors": "2.8.5",
"debug": "~4.0.1",
"dotenv": "~6.1.0",    
"@elastic/elasticsearch": "^7.13.0",
"elasticsearch-helper": "^1.15.2",
"express": "~4.17.1",
"express-jwt": "5.3.1",
"forest-express-sequelize": "^7.0.0",
"morgan": "1.9.1",
"mysql2": "~2.2.5",
"require-all": "^3.0.0",
"sequelize": "~5.15.1"
2

Hello @Arda_Yurdakul :wave:

:warning: search is different from filters

The search functionality is the most painful to code. You have to hard code your own way to search your data. In this example you have a simple use case.

Here’s an example that handle search

  ...

  const search = request.query?.search;

  // Declare the filters query retrieval first
  try {
    filters = request.query?.filters && JSON.parse(request.query.filters);
  } catch (e) {
    filters = undefined;
  }

  if (search) {
    // In case of search you need to handle it

    // NOTICE: Create a custom boolean query for Elasticsearch
    const booleanQuery = {
      should: [{
        terms: {
          fieldYouNeedToSearchOn: search,
        },
      }],
      minimum_should_match: 1,
    };
    
    // NOTICE: Use the elasticsearchHelper to query Elasticsearch
    const [results, count] = await Promise.all([
      elasticsearchHelper.esSearch(
        { page, pageSize },
        booleanQuery,
      ),
      elasticsearchHelper.esCount(booleanQuery),
    ]);
    
    response.send({
      ...await serializer.serialize(results),
      meta: {
        count: count,
      }
    });
  } else {
   // The already existing code that retrieve records using filters
   const result = await elasticsearchHelper.functionSearch({
      page,
      pageSize,
      filters: filters || undefined,
      options,
   });

   const serializer = new RecordSerializer({ name: 'logs' });

   response.send({
     ...await serializer.serialize(result.results),
     meta: {
       count: result.count,
     }
   });
 }

} catch (e) {
  next(e);
}

In this example you cannot search and filter at the same time. To do so you need to change the search part

// import esTranslateFilter
const { esTranslateFilter } = require('../utils/filter-translator');

...

// AFTER - NOTICE: Create a custom boolean query for Elasticsearch
booleanQuery = { ...booleanQuery, filter: esTranslateFilter(elasticsearchHelper.filterDefinition, filters, options)};

...

Finally you can find some documentation about boolean queries !

Let me know if it helps.
Have a nice day. :slight_smile:

Morgan

3

Hi Morgan,

Thank you for your in-depth explanation! It made sense to me but I ran into a nasty error whilst trying to implement the search functionality.

image
image1214×697 17.2 KB

I’m not quite sure what the problem is. Here are my field definitions and my boolQuery:

image
image779×344 11.2 KB

image
image853×303 7.12 KB

Thank you once again!

4

Hey again @Arda_Yurdakul,

This error is specific to Elastocsearch. It’s due to the type of the charge_point_identity in your ES shard. I think you should use a must query with match in this case ?
I let you check the matching you want to implement here: Boolean query | Elasticsearch Guide [7.13] | Elastic

Some example of booleanQuery

// strict matching example
must: {
    match: {
      charge_point_identity: search,
    }
}


// start with example
filter: {
 wildcard: {
      charge_point_identity: {
        value: `${search}*`,
        case_insensitive: true,
      },
    },
}

Have a nice day,
Morgan

2 Likes
5

Yes, it works now! Very helpful, thank you!

1 Like