@forestadmin/datasource-replica fails on startup: missing cache-interface module

Feature(s) impacted

createReplicaDataSource

Observed behavior

On npm run dev, the project crashes with

Error: Cannot find module './cache-interface/datasource'

My code looks like this (I call addDataSource on server startup and pass the agent to it):

export const addDataSource = (agent: Agent<Schema>) => {
  const stripeDataSource = createReplicaDataSource({
    pullDumpOnSchedule: ['@hourly'],
    pullDumpHandler: async () => {
      return {
        more: false,
        entries: [...(await getAllSubscriptions())],
      };
    },
  });

  const getAllSubscriptions = async () => {
    const query = 'total>1';
    let page = undefined;
    let hasMore = true;
    const entries = [];

    while (hasMore) {
      const batchedSubscriptions = await stripe.subscriptions.search({ limit: 50, query, page });
      hasMore = batchedSubscriptions.has_more;
      page = batchedSubscriptions.next_page;
      entries.push(
        ...batchedSubscriptions.data.map((s) => {
          return { record: s, collection: 'stripeSubscriptions' };
        })
      );
    }

    return entries;
  };

  agent.addDataSource(stripeDataSource);
};

Expected behavior

No error and it runs.

Failure Logs

Error: Cannot find module './cache-interface/datasource'
Require stack:
- /Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/@forestadmin/datasource-replica/dist/index.js
- /Users/davidroegiers/Jurata/jurata-next/forest-2/external-datasources/stripe/subscriptions.ts
- /Users/davidroegiers/Jurata/jurata-next/forest-2/external-datasources/index.ts
- /Users/davidroegiers/Jurata/jurata-next/forest-2/index.ts
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:955:15)
    at Function.Module._resolveFilename.sharedData.moduleResolveFilenameHook.installedValue [as _resolveFilename] (/Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/@cspotcode/source-map-support/source-map-support.js:811:30)
    at Function.Module._load (node:internal/modules/cjs/loader:803:27)
    at Module.require (node:internal/modules/cjs/loader:1021:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at Object.<anonymous> (/Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/@forestadmin/datasource-replica/src/index.ts:7:1)
    at Module._compile (node:internal/modules/cjs/loader:1119:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1173:10)
    at Object.require.extensions.<computed> [as .js] (/Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/ts-node/src/index.ts:1608:43)
    at Module.load (node:internal/modules/cjs/loader:997:32) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/@forestadmin/datasource-replica/dist/index.js',
    '/Users/davidroegiers/Jurata/jurata-next/forest-2/external-datasources/stripe/subscriptions.ts',
    '/Users/davidroegiers/Jurata/jurata-next/forest-2/external-datasources/index.ts',
    '/Users/davidroegiers/Jurata/jurata-next/forest-2/index.ts'
  ]
}
Waiting for the debugger to disconnect...
[nodemon] app crashed - waiting for file changes before starting...

Context

  • Project name: jurata-2
  • Team name: jurata
  • Environment name: Development
  • Agent (forest package) name & version:
  "dependencies": {
    "@forestadmin/agent": "^1.13.1",
    "@forestadmin/datasource-mongoose": "^1.4.2",
    "@forestadmin/datasource-replica": "^1.0.0",
    "axios": "^1.4.0",
    "dataloader": "^2.2.2",
    "dotenv": "^16.0.1",
    "jsonwebtoken": "^9.0.0",
    "mongoose": "^6.10.3",
    "stripe": "^12.10.0",
    "typescript": "^4.9.4"
  },
  • Database type: external api
  • Recent changes made on your end if any: it’s new code

Some files seem to be missing in the distribution?

Hi @David_Roegiers ,

We found the issue. A fix should be released soon. Thanks for reaching out.

Best regards,

Shohan

2 Likes

Hello @David_Roegiers,

Really sorry for that and thanks for the head-up. :pray:
We just released a new version of the @forestadmin/datasource-replica package. It should be good this time.

Let us know if it works for you.

Kind regards,
Morgan

Hi thanks,

Mh, now I get the following.

Error: Please install sqlite3 package manually
    at ConnectionManager._loadDialectModule (/Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/sequelize/src/dialects/abstract/connection-manager.js:81:15)
    at new ConnectionManager (/Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/sequelize/src/dialects/sqlite/connection-manager.js:24:21)
    at new SqliteDialect (/Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/sequelize/src/dialects/sqlite/index.js:15:30)
    at new Sequelize (/Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/sequelize/src/sequelize.js:368:20)
    at Function.build (/Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/@forestadmin/datasource-sql/src/connection/sequelize-factory.ts:14:11)
    at connect (/Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/@forestadmin/datasource-sql/src/connection/index.ts:43:34)
    at async buildSequelizeInstance (/Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/@forestadmin/datasource-sql/src/index.ts:39:17)
    at async createSequelize (/Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/@forestadmin/datasource-replica/src/sequelize.ts:92:21)
    at async /Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/@forestadmin/datasource-replica/src/index.ts:23:24
    at async /Users/davidroegiers/Jurata/jurata-next/forest-2/node_modules/@forestadmin/datasource-customizer/src/datasource-customizer.ts:61:24
[nodemon] app crashed - waiting for file changes before starting...

Hello @David_Roegiers,

I will make a feedback to our team to improve the documentation.

But, this is the expected behavior, as SQLite is the default persistence storage strategy. You can choose another strategy or you need to add this package to your project. See more details about this in the link above. :slight_smile:

Regards,
Morgan

Hello @morganperre

Ah, okay, makes sense.

Thanks for the quick fix.

Best,
David

1 Like