Feature(s) impacted
Filters of Flattened fields which require parsing (like Date) are not working.
Observed behavior
The Date value string sent from the client is NOT converted to a Date by the lib and mongoose compares text to date and the filter is not working.
Non Flattened fields work as expected.
Expected behavior
Filters should work also for flattened fields.
Failure Logs
Consider the following filter sent from the client to a “list” endpoint:
{"aggregator":"and","conditions":[{"field":"monitoring_status","operator":"equal","value":"active"},{"field":"modified_at","operator":"before","value":"2023-11-27T07:29:54.576Z"},{"field":"conversation@@@last_msg_at","operator":"before","value":"2023-11-26T07:29:54.576Z"}]}
The field “modified_at” is not flattened - the field “conversation.last_msg_at” is flattened.
The relevant Mongoose schema looks like this:
{ monitoring_status: String, modified_at: Date, conversation: { last_msg_at: Date } }
The output filter to mongodb looks like this:
{ '$match': { '$and': [ { '$and': [ { monitoring_status: 'active' }, { modified_at: { '$lt': **2023-11-27T07:35:33.662Z** } }, { 'conversation.last_msg_at': { '$lt': **'2023-11-26T07:35:33.662Z'** } } ] } ] } }
Note! one date value is a string and the other is an ISODate
Root cause
See the code here
this.formatCondition = async (condition, isSmartField = false) => {
if (isSmartField) {
return this.formatOperatorValue(
condition.field,
condition.operator,
condition.value,
);
}
const formatedField = this.formatField(condition.field);
return {
//!==>
[Flattener.unflattenFieldName(formatedField)]: await this.formatOperatorValue(
condition.field, //<==!
condition.operator,
condition.value,
),
};
};
The field sent into the formatOperatorValue is not unflattened. Thus it cannot find that field in the schema and falls back to an unknown type and does no parsing on it.
this.formatOperatorValue = async (field, operator, value) => {
if (this.operatorDateParser.isDateOperator(operator)) {
return this.operatorDateParser.getDateFilter(operator, value);
}
//!==>
const parseFct = await this.getParserForField(field);
...
And
this.getParserForField = async (key) => {
const [fieldName, subfieldName] = key.split(':');
const field = SchemaUtils.getField(modelSchema, fieldName);
if (!field) {
throw new InvalidFiltersFormatError(`Field '${fieldName}' not found on collection '${modelSchema.name}'`);
}
const fieldPath = subfieldName ? `${fieldName}.${subfieldName}` : fieldName;
const fieldType = **utils.getNestedFieldType**(model.schema, fieldPath);
//!==>
if (!fieldType) return (val) => val;
...
};
Context
- Project name: Not Project specific
- Environment name: Not Env Specific
- Agent (forest package) name & version: forest-mongoose-express versions 8+9
- Database type: Mongodb
