node-resolve-experiments

Overview

This repo is for experimenting with Node's module resolution logic to evaluate it for usage in programmatically looking up the location of npm packages on disk.

Setup

To play around with this repo and follow along, make sure you have current NodeJS installed (>=16.x). You can check your environment accordingly

% node -v
v16.4.2

% npm -v
7.18.1

After cloning, run npm ci.

nvm is helpful if you want to manage different versions of NodeJS on your local machine.

Commands

You can run either of two commands to see the results documented below:

• npm run serve:cjs - to start the CJS example server
• npm run serve:esm - to start the ESM example server

Open localhost:3000 to view the demo / console output.

Problem Statement

When writing frontend JavaScript, you may want to take advantage of some package from npm, like Lit.

import { LitElement } from 'lit';

If we were using require (CJS) or import (ESM) in Node natively, NodeJS (and / or a bundler) would handle:

1. Finding where on disk the top level of the package resides
2. Determing from its package.json what the correct entry point to resolve to is

However, in browser land there is of course, none of that.

So what are we to do?

The Challenge

While we have importmap for being able to link these bare specifiers to an actual file on disk

ex.

<script type="importmap-shim">
  {
    "imports": {
      "lit-element": "/node_modules/lit-element/lit-element.js",
      "lit-html/lib/shady-render.js": "/node_modules/lit-html/lib/shady-render.js",
      "lit-html/lit-html.js": "/node_modules/lit-html/lit-html.js",
      "lit-html": "/node_modules/lit-html/lit-html.js"
    }
  }
</script>

And setting up a server to help route those requests accordingly from browser to filesystem isn't too hard, there's still the matter of actually knowing where specifically this file on disk actually lives.

We might just assume that everything lives at the root of a project in its node_modules directory, but this is not always the case:

• Package Managers may create nested node_modules within the top level node_modules to handle duplicates or honor semver package needs, or put them all in a shared location (like pnpm)
• The point above may come even more into play when dealing with workspaces and monorepos.
• Where does npx put all those dependencies when it runs?

Basically, it all boils down to at best, you can guess and maybe use some heuristics but since Node has all this logic built in already for CommonJS and ESM, maybe we should find a way to tap into that same logic, just like if we were to use require or import.

Note: as alluded to above, just knowing where the package resides is only part of the challenge. Second challenge is knowing if you need CJS vs ESM and what entry point, etc etc. We'll see if we can try and solve that too!

Solution(s) (WIP)

1. Initial suggestion when reaching out in NodeJS Slack was to try using require.resolve to see if that would provide the information we are looking for. Will try this first.

Conclusions

require.resolve

This seems promising so far, with the following code (see it in server.js)

console.debug('require.resolve(lit) =>', require.resolve('lit'));
console.debug('require.resolve.paths(lit) =>', require.resolve.paths('lit'));

Yielding the following output:

require.resolve(lit) => /Users/owenbuckley/Workspace/github/repos/node-resolve-expirements/node_modules/lit/index.js
require.resolve.paths(lit) => [
  '/Users/owenbuckley/Workspace/github/repos/node-resolve-expirements/node_modules',
  '/Users/owenbuckley/Workspace/github/repos/node_modules',
  '/Users/owenbuckley/Workspace/github/node_modules',
  '/Users/owenbuckley/Workspace/node_modules',
  '/Users/owenbuckley/node_modules',
  '/Users/node_modules',
  '/node_modules',
  '/Users/owenbuckley/.node_modules',
  '/Users/owenbuckley/.node_libraries',
  '/Users/owenbuckley/.nvm/versions/node/v14.16.0/lib/node'
]

I think though since we are using CJS in server.js, we are getting Lit's CJS entry point as defined in package.json#main

"main": "index.js",
"module": "index.js",
"type": "module",
"exports": {
  ".": {
    "default": "./index.js"
  },
  "./decorators.js": {
    "default": "./decorators.js"
  },
  "./decorators/": {
    "default": "./decorators/"
  },
  "./directive-helpers.js": {
    "default": "./directive-helpers.js"
  },
  "./directive.js": {
    "default": "./directive.js"
  },
  "./directives/": {
    "default": "./directives/"
  },
  "./async-directive.js": {
    "default": "./async-directive.js"
  },
  "./html.js": {
    "default": "./html.js"
  },
  "./experimental-hydrate-support.js": {
    "default": "./experimental-hydrate-support.js"
  },
  "./experimental-hydrate.js": {
    "default": "./experimental-hydrate.js"
  },
  "./polyfill-support.js": {
    "default": "./polyfill-support.js"
  },
  "./static-html.js": {
    "default": "./static-html.js"
  }
},

Ideally we want to (make sure) we are getting something from exports (map) or module in package.json.

So either we need to:

• only be using ESM for our NodeJS code (something we will try next!)
• force NodeJS to lookup ESM explicitely. assuming that if already using ESM, this would be the default, right? (what would a fallback looklike though?)
• or otherwise just use require.resolve as the starting point of our adventure, and use our own logic to decide what file we want to return to the browser, of which we want to favor ESM exclusively, or eles detect + convert CJS -> ESM on the fly?

import.meta

In server.mjs, we can start using ESM, and although we no longer have access to require, when using the --experimental-import-meta-resolve flag, we can now use import.meta.resolve instead.

And indeed, for the following code

console.debug('import.meta.resolve(lit) =>', await import.meta.resolve('lit'));

we can see the following output

import.meta.resolve(lit) => file:///Users/owenbuckley/Workspace/github/repos/node-resolve-experiments/node_modules/lit/index.js

Since index.js is the same for both CJS and ESM entry points for Lit, I might add another test package wherein the difference between the two is perhaps not as ambiguous, just to make sure we are indeed getting the expected results, e.g. a guaranteed ESM first entry point? (assuming it exists)

Differing Entry Points (CJS vs ESM)

So after adding a couple more packages, redux and lodash, I was a little surprised by the findings. It looks like the results are the same for both CJS and ESM versions? (Aside from the obvious difference that ESM uses file:// protocol)

Specifically for redux, which ships a main and a module entry in its package.json, I would have expected the following:

• CJS - path/to/node_modules/redux/lib/redux.js
• ESM - path/to/node_modules/redux/es/redux.mjs

CommonJS (require.resolve)

{
  "lit": "/Users/owenbuckley/Workspace/github/repos/node-resolve-experiments/node_modules/lit/index.js",
  "lodash": "/Users/owenbuckley/Workspace/github/repos/node-resolve-experiments/node_modules/lodash/lodash.js",
  "redux": "/Users/owenbuckley/Workspace/github/repos/node-resolve-experiments/node_modules/redux/lib/redux.js"
}

ESM (import.meta.resolve)

{
  "lit": "file:///Users/owenbuckley/Workspace/github/repos/node-resolve-experiments/node_modules/lit/index.js",
  "lodash": "file:///Users/owenbuckley/Workspace/github/repos/node-resolve-experiments/node_modules/lodash/lodash.js",
  "redux": "file:///Users/owenbuckley/Workspace/github/repos/node-resolve-experiments/node_modules/redux/lib/redux.js"
}

However, both are returning redux/lib/redux.js? Is this the expected result? 🤔

Update: Yes, it looks like the above is indeed the expected behavior. Appears module was just a community convention so as far as NodeJS is concerned, it really comes down to just main or an exports map.

So for a userland tool perhaps in the absence of an exports map supporting module could be an option, but ideally always favor an exports map so as to spec compliant?

require.resolve.paths

One thing I observed was that if a project has no main entry in its package.json (or it is empty; main: ""), which happened for me in @babel/runtime and @types/trusted-types, then require.resolve will actually fail. However, an alternative is to use require.resolve.paths which when passed a module (package) name, will return all the paths that NodeJS used to look for it.

From there, you can find out where it is programmatically.

const packageName = 'lit';
let nodeModulesUrl;
  
try {
  const packageEntryLocation = require.resolve(packageName);
  const packageRootPath = packageEntryLocation.split(packageName)[0];

  nodeModulesUrl = `${packageRootPath}${packageName}`;
} catch (e) {
  const locations = require.resolve.paths(packageName);

  for (const location in locations) {
    const nodeModulesPackageRoot = `${locations[location]}/${packageName}`;
    const packageJsonLocation = `${nodeModulesPackageRoot}/package.json`;

    if (fs.existsSync(packageJsonLocation)) {
      nodeModulesUrl = nodeModulesPackageRoot;
    }
  }
}