NodeJS: loading ES Modules and native addons in the same project

Question

Before the actual questions (see at the end), please let me show the steps that lead to that question through an example:

Creating the project

tests$ mkdir esm && cd esm
tests/esm$ nvm -v
0.37.2
tests/esm$ nvm use v15
Now using node v15.6.0 (npm v7.5.6)
tests/esm$ node -v
v15.6.0
tests/esm$ npm -v
7.5.6
tests/esm$ npm init
package name: (esm) test-esm
entry point: (index.js)

Installing nodehun

tests/esm$ npm install nodehun
added 2 packages, and audited 3 packages in 11s
tests/esm$ npm ls
test-esm@1.0.0 tests/esm
└── nodehun@3.0.2

• dependencies of nodehun here

index.js

import { suggest } from './checker.js'
suggest("misspeling");

checker.js

import Nodehun  from 'nodehun'
import fs from 'fs';

const affix       = fs.readFileSync('dictionaries/en_NZ.aff')
const dictionary  = fs.readFileSync('dictionaries/en_NZ.dic')
const nodehun     = new Nodehun(affix, dictionary)

export const suggest = (word) => hun_suggest(word);

async function hun_suggest(word) {
  let suggestions = await nodehun.suggest(word);
  console.log(suggestions);
}

To obtain the required Hunspell dictionary files (affix and dictionary):

tests/esm$ mkdir dictionaries && cd dictionaries
tests/esm/dictionaries$ curl https://www.softmaker.net/down/hunspell/softmaker-hunspell-english-nz-101.sox > en_NZ.sox
tests/esm/dictionaries$ unzip en_NZ.sox en_NZ.aff en_NZ.dic

Running the project

As per nodejs documentation (Determining Module System) to support the import / export:

Node.js will treat the following as ES modules when passed to node as the initial input, or when referenced by import statements within ES module code: • Files ending in .js when the nearest parent package.json file contains a top-level "type" field with a value of "module".

We add "type": "module" field in the package.json file of the project.

package.json

{
  ...
  "main": "index.js",
  "type": "module",
  ...
}

First Failed Run

tests/esm$ node index.js
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".node" for tests/esm/node_modules/nodehun/build/Release/Nodehun.node
... omitted ...
at async link (node:internal/modules/esm/module_job:64:9) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

Digging a bit on the reason of the above error:

• in the documentation on how to load addons, it refers to the use of require

The filename extension of the compiled addon binary is .node (as opposed to .dll or .so). The require() function is written to look for files with the .node file extension and initialize those as dynamically-linked libraries.

• once you define your node project as a "type": "module", require it ceases to be supported (as specified in Interoperability with CommonJS):

Using require to load an ES module is not supported because ES modules have asynchronous execution. Instead, use import() to load an ES module from a CommonJS module.

Temporary Solution

After some time searching the documentation, I found a temporary solution: Customizing ESM specifier resolution algorithm:

The current specifier resolution does not support all default behavior of the CommonJS loader. One of the behavior differences is automatic resolution of file extensions and the ability to import directories that have an index file. The --experimental-specifier-resolution=[mode] flag can be used to customize the extension resolution algorithm. To enable the automatic extension resolution and importing from directories that include an index file use the node mode.

tests/esm$ node --experimental-specifier-resolution=node index.js
(node:XXXXX) ExperimentalWarning: The Node.js specifier resolution in ESM is experimental.
(Use `node --trace-warnings ...` to show where the warning was created)
[
  'misspelling',
  'misspending',
  'misspeaking',
  'misspell',
  'dispelling',
  'misapplier',
  'respelling'
]

There are a some posts that get to this same resolution (ref 1, ref 2). However, using experimental flags does not seem a proper way to run your application on production.

Failed Alternative with esm package

From that point, several failed attempts have been tried to avoid the use of --experimental-* flags. Doing some search, I found some posts (ref 1, ref 2) recommending the use of the esm package.

• esm gets 1.3M downloads per week.
• According the read-me file in GitHub, it does not require any changes.

However, at this point, when I try this node -r esm index.js, a new error appears:

tests/esm$ npm install esm
added 1 package, and audited 4 packages in 709ms
tests/esm$ npm ls
test-esm@0.1.0 tests/esm
├── esm@3.2.25
└── nodehun@3.0.2
tests/esm$ node -r esm index.js
tests/esm/index.js:1
Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: tests/esm/index.js
    at new NodeError (node:internal/errors:329:5)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1125:13) {
  code: 'ERR_REQUIRE_ESM'
}

The above could be due to a reported issue (Error [ERR_REQUIRE_ESM]: Must use import to load ES Module / require() of ES modules is not supported).

• There is proposed patch to fix it, although I do not know how to use it myself.

const module = require('module');
module.Module._extensions['.js'] = function(module, filename) {
  const content = fs.readFileSync(filename, 'utf8');
  module._compile(content, filename);
};

Questions

1. Is there a (standard) way to use import / export (ES Modules) without incurring in issues with import addons?
   • Avoiding the use of the --experimental-specifier-resolution=node flag.
2. Perhaps esm could be the solution to the above. Is there anything I am doing wrong with the usage esm package?
   • If it is a correct usage, is there a way to use the proposed patch myself as a working around?

Any hints to help to solve it would be really appreciated.

Note: the final status of the example can be cloned from https://github.com/rellampec/test-esm.git

Answer 1

I ran into a similar problem and fixed it this way: https://nodejs.org/api/module.html#module_module_createrequire_filename

// The project is "type": "module" in package json
// createRequire is native in node version >= 12
import { createRequire } from 'module'; 
import path from 'path'; 
// Absolute path to node modules (or native addons)
const modulesPath = path.resolve(process.cwd(), 'node_modules');
// Create the require method
const localRequire = createRequire(modulesPath);
// require the native add-on
const myNativeAddon = localRequire('my-native-addon'); 

Answer 2

After some ramblings trying to figure this out got to the root cause.

When using node -r esm index.js, the esm package does already all the work for your (as noted in other answers), and therefore (not mentioned in other answers):

• the package.json should be updated by removing "type:" "module" (as it creates unintended interactions between the native node ES Modules feature and the esm package you installed)

Aside note: if you tried to use node ES Modules and then you try to switch to esm package, it is very easy to miss this point.