Feature(s) impacted
Smart Field of type Date will not process filtering properly in all cases.
Observed behavior
Creating a Date smart field, relying on the “where” argument will provide wrong “String” dates when the operator is not an exclusive “Date” operator.
The expected “where” structure is an object where the value is a Date object combined into a valid formatted query.
For a mongodb - that would be a valid mongoose query expression.
Expected behavior
Date smart fields should properly support filtering using the “where” argument.
Failure Logs
Here is a details explanation of the problem.
Let’s assume a basic mongoose schema:
{ modified_at: Date }
A Smart Field is defined of type “Date” for example
{
field: 'funny_modified_at',
type: 'Date',
isFilterable: true,
get: (record) => {
return record.modified_at;
},
filter({ condition, where }) {
return {
'modified_at': where,
};
},
}
The good
Client sends a filter on the Smart Field the mongoose express backend.
For example:
{"field":"funny_modified_at","operator":"yesterday","value":null}
The backend processes this properly!
The “filter” method on the smart field gets the following “where” argument:
{ '$gte': 2023-11-27T22:00:00.000Z, '$lte': 2023-11-28T21:59:59.999Z }
Note - Those is a valid ISODate() time values. This works since “yesterday” is considered a dedicated “Date” operator as defined here.
The Bad
Now the client send the following filter:
{"field":"funny_modified_at","operator":"before","value":"2023-11-21T22:00:00.000Z"}
The “filter” method on the smart field gets the following “where” argument:
{ '$lt': '2023-11-21T22:00:00.000Z' }
Note - this is a string formatted Date - not a Date object.
The Ugly
Why is this happening?
The “where” input is generated within the forest-express lib in base-filters-parser.js within the parseCondition function here.
const where = await formatCondition(condition, true);
const formattedCondition = await fieldFound
.filter({
where,
condition,
});
formatCondition is injected from the implementation layer - in my case forest-express-mongoose in filter-parse.js and defined here like this:
this.formatCondition = async (condition, isSmartField = false) => {
if (isSmartField) {
return this.formatOperatorValue(
condition.field,
condition.operator,
condition.value,
);
}
...
};
In this scenario, it is called with isSmartField = true so that “if” is entered.
At the begging of formatOperatorValue we check for the special “Date” operators like so:
this.formatOperatorValue = async (field, operator, value) => {
if (this.operatorDateParser.isDateOperator(operator)) {
return this.operatorDateParser.getDateFilter(operator, value);
}
Here something like “yesterday” would enter the if and will “save the day” - prevent the bug - the “value” will be formatted as a Date object inside getDateFilter.
But! If the operator is something like before it is not considered an exclusive Date operator and will skip this if clause.
Then - it does the following:
const parseFct = await this.getParserForField(field);
And within getParserForField there is:
this.getParserForField = async (key) => {
const [fieldName, subfieldName] = key.split(':');
const field = SchemaUtils.getField(modelSchema, fieldName);
if (!field) {
throw new InvalidFiltersFormatError(`Field '${fieldName}' not found on collection '${modelSchema.name}'`);
}
const fieldPath = subfieldName ? `${fieldName}.${subfieldName}` : fieldName;
const fieldType = utils.getNestedFieldType(model.schema, fieldPath);
// fieldType is empty here for Smart Fields!
if (!fieldType) return (val) => val;
Which tries to infer the field type from the utils.getNestedFieldType function.
This does not return Date, instead it tries to infer only from the mongoose schema which of course does not include the Smart Field type.
This is the root cause of the bug.
Looking at this post from 2021 - it seems like “where” was working as expected back then.
Indeed this commit from 2 years ago broke things changing how field types are inferred ignoring the Smart Field case.
Context
- Project name: Any Project
- Team name: Any Team
- Agent (forest package) name & version: mongoose agent 8+9
- Database type: mongodb
