Skip to content

Commit

Permalink
feat(npm): add support for npm5
Browse files Browse the repository at this point in the history 
BREAKING CHANGE: node_shrinkwrap is now archived-packages
BREAKING CHANGE: directory option removed and instead uses process.cwd
BREAKING CHANGE: node.js API removed, CLI only
BREAKING CHANGE: --keep-optional and --compress options removed

closes #83, #93
  • Loading branch information
@JamieMason
JamieMason committed Apr 2, 2018
1 parent c50981e commit 730e2c8
Show file tree
Hide file tree
Showing 37 changed files with 6,724 additions and 1,025 deletions.
4 changes: 4 additions & 0 deletions .prettierrc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"printWidth": 120,
"singleQuote": true
}
16 changes: 3 additions & 13 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
@@ -1,21 +1,11 @@
language: node_js
node_js:
- '8'
- '7'
- '6'
- '5'
- '4'
env:
- NPM_VERSION=4.1.1
- NPM_VERSION=4.1.0
- NPM_VERSION=4.0.5
- NPM_VERSION=4.0.3
- NPM_VERSION=4.0.2
- NPM_VERSION=4.0.1
- NPM_VERSION=4.0.0
- NPM_VERSION=3.10.10
- NPM_VERSION=3.10.9
- NPM_VERSION=3.10.8
- NPM_VERSION=3.10.7
- NPM_VERSION=3.10.6
- NPM_VERSION=3.10.5
- NPM_VERSION=5.8.0
script:
- NPM_VERSION=$NPM_VERSION npm test
139 changes: 30 additions & 109 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,16 @@
[![Build Status](http://img.shields.io/travis/JamieMason/shrinkpack/master.svg?style=flat-square)](https://travis-ci.org/JamieMason/shrinkpack)
[![Gitter Chat for shrinkpack](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/JamieMason/shrinkpack)
[![Donate via PayPal](https://img.shields.io/badge/donate-paypal-blue.svg)](https://www.paypal.me/foldleft)
[![Donate via Gratipay](https://img.shields.io/gratipay/user/JamieMason.svg)](https://gratipay.com/~JamieMason/)
[![Analytics](https://ga-beacon.appspot.com/UA-45466560-5/shrinkpack?flat&useReferer)](https://github.com/igrigorik/ga-beacon)
[![Follow JamieMason on GitHub](https://img.shields.io/github/followers/JamieMason.svg?style=social&label=Follow)](https://github.com/JamieMason)
[![Follow fold_left on Twitter](https://img.shields.io/twitter/follow/fold_left.svg?style=social&label=Follow)](https://twitter.com/fold_left)

Shrinkpack complements the [npm shrinkwrap](https://docs.npmjs.com/cli/shrinkwrap) command by
maintaining a `node_shrinkwrap` directory in your project, containing the exact same tarballs that
maintaining a `archived-packages` directory in your project, containing the exact same tarballs that
`npm install` downloads from [https://registry.npmjs.org](https://registry.npmjs.org).

The rest of the `npm install` process is exactly the same. The only difference is that no network
activity is necessary when installing and building your project. The `node_shrinkwrap` directory can
activity is necessary when installing and building your project. The `archived-packages` directory can
be ignored in your editor (much like is done with the `node_modules` directory) but is instead
checked into source control.

Expand All @@ -28,7 +27,6 @@ checked into source control.
* [Installation](#installation)
* [Usage](#usage)
* [Command Line](#command-line)
* [Node.js](#nodejs)
* [Target Problem](#target-problem)
* [Justification](#justification)
* [npm shrinkwrap](#npm-shrinkwrap)
Expand All @@ -39,78 +37,43 @@ checked into source control.
* [Set some sensible npm defaults](#set-some-sensible-npm-defaults)
* [Install dependencies](#install-dependencies)
* [Shrinkwrap dependencies](#shrinkwrap-dependencies)
* [Create a project-specific cache (optional)](#create-a-project-specific-cache-optional)
* [Shrinkpack dependencies](#shrinkpack-dependencies)
* [Check into Git](#check-into-git)
* [Clean install](#clean-install)
* [Update Dependencies](#update-dependencies)
* [Toggle Compression](#toggle-compression)
* [Remove Optional Dependencies](#remove-optional-dependencies)

## Installation

```
npm install --global shrinkpack
```

> **Note:** npm had a [regression affecting shrinkwrap](https://github.com/npm/npm/pull/13214) in
> versions 3.8.8 to 3.10.3.<br>
> Please ensure your version of `npm` is 3.10.4 or newer, or 3.8.7 or older.
> Please ensure your version of `npm` is 5.8.0 or newer.
## Usage

### Command Line

```
Usage: shrinkpack [options] [directory]
Usage: shrinkpack [options]
Options:
-h, --help output usage information
-V, --version output the version number
-c, --compress use compressed .tgz tarballs instead of .tar
-o, --keep-optional do not exclude optional dependencies
Icons:
+ Added
↓ Downloaded
→ Imported from Cache
i Information
- Removed
✓ Resolved
12:34 Time Taken
Compression:
Although compressed .tgz files have lower filesizes, storing binary files in
Git can result in a gradual increase in the time it takes to push to your
repository. Shrinkpack uses uncompressed, plain text .tar files by default,
which are handled optimally by Git in the same way that .md, .js, and .css
files are for example.
```

### Node.js

Shrinkpack works in 2 phases;

1. Analyse the project and gather all the diffing information between the file system and the shrinkwrap.
2. Use the diffing information to bring the file system in sync with the shrinkwrap.

```js
var shrinkpack = require('shrinkpack');

shrinkpack.analyse({ compress: false, directory: process.cwd(), keepOptional: false })
.then(data => shrinkpack.update(data));
```

Or to run `shrinkpack` in full, including all the additional logging that you see when using the CLI.

```js
var shrinkpack = require('shrinkpack');

shrinkpack.cli({ compress: false, directory: process.cwd(), keepOptional: false })
.then(() => {});
repository. Shrinkpack uses uncompressed, plain text .tar files, which are
handled optimally by Git in the same way that .md, .js, and .css files are.
```

## Target Problem
Expand All @@ -127,8 +90,8 @@ We were all very happy with this process and the convenience of npm in particula
our builds where `npm install` listed a huge amount of network traffic would always raise the same
concerns;

+ This seems slow, wasteful, and inefficient.
+ We _really_ depend on registry.npmjs.org, what do we do if it goes down?
* This seems slow, wasteful, and inefficient.
* We _really_ depend on registry.npmjs.org, what do we do if it goes down?

The first suggestion was always to check in our dependencies, but the idea of some large and chatty
commits whenever we chose to upgrade or change them would put us off.
Expand All @@ -139,7 +102,7 @@ to find that some packages such as [phantomjs](https://www.npmjs.com/package/pha
depending on what system you're running.

This meant that if Chris added `phantomjs` or `node-sass` to the project on his Mac and checked it
into the repository, Helen wouldn't be able to use it on her Windows Machine.
into the repository, Helen wouldn't be able to use it on her Windows Machine.

The remaining alternatives were proxies, mirrors, and caches-of-sorts. None of which appealed to us
and, grudgingly, we continued as we were (<abbr title="Your Mileage May Vary">YMMV</abbr>).
Expand All @@ -149,10 +112,10 @@ and, grudgingly, we continued as we were (<abbr title="Your Mileage May Vary">YM
Whenever we add, remove, or update an npm dependency — we should test our application for
regressions before locking down our dependencies to avoid them mutating over time.

+ You can't be sure of this without `npm shrinkwrap`.
+ Checking in `node_modules` is horrible (and doesn't work in many cases anyway).
+ You can be reasonably sure of this with `npm shrinkwrap`.
+ You can be completely sure of this with `npm shrinkwrap` and `shrinkpack`.
* You can't be sure of this without `npm shrinkwrap`.
* Checking in `node_modules` is horrible (and doesn't work in many cases anyway).
* You can be reasonably sure of this with `npm shrinkwrap`.
* You can be completely sure of this with `npm shrinkwrap` and `shrinkpack`.

### npm shrinkwrap

Expand Down Expand Up @@ -182,17 +145,16 @@ contains a regression.
With you hopefully convinced of the merits of `npm shrinkwrap`, `shrinkpack` will hopefully be seen
as a small and complementary addition.

`shrinkpack` takes the .tgz tarballs of that specific, shrinkwrapped dependency graph saved by `npm
shrinkwrap` and stores them within your project.
`shrinkpack` takes the .tgz tarballs of that specific, shrinkwrapped dependency graph saved by `npm shrinkwrap` and stores them within your project.

This means;

+ No need for repeated requests to registry.npmjs.org.
+ Each package/version pair can be checked in as a single tarball, avoiding commits with all kinds
* No need for repeated requests to registry.npmjs.org.
* Each package/version pair can be checked in as a single tarball, avoiding commits with all kinds
of noisy diffs.
+ Packages can be checked in, while still installed by members of the team on different operating
* Packages can be checked in, while still installed by members of the team on different operating
systems.
+ Complements the typical `npm shrinkwrap` workflow.
* Complements the typical `npm shrinkwrap` workflow.

## Suitability to your project

Expand All @@ -202,7 +164,7 @@ a dependency of another project using `npm install`, let those downstream projec
decisions on bundling.

That said, if you're developing an npm package and want to use `shrinkpack` to speed up and
harden your development and CI environments, adding `npm-shrinkwrap.json` and `node_shrinkwrap` to
harden your development and CI environments, adding `npm-shrinkwrap.json` and `archived-packages` to
your `.npmignore` file will allow you to do that, without publishing your shrinkpacked dependencies
to the registry.

Expand Down Expand Up @@ -274,51 +236,35 @@ npm shrinkwrap --dev

<a href="https://asciinema.org/a/83795" target="_blank"><img src="https://asciinema.org/a/83795.png" alt="asciicast"></a>

### Create a project-specific cache (optional)

When using `shrinkpack`, the local file path to dependencies will be added to the `npm` client's
[local cache](https://docs.npmjs.com/cli/cache). This can be problematic when working on several
projects on a single machine ([#31](https://github.com/JamieMason/shrinkpack/issues/31)).

This step prevents npm from using this project as a registry should you install the same
package/version pair on another project on your machine.

```
echo cache=node_cache >> .npmrc
echo /node_cache >> .gitignore
```

### Shrinkpack dependencies

Whenever you run `npm install`, npm downloads a .tgz file from http://registry.npmjs.org containing
the installation for each package. Shrinkpack saves these files in a `node_shrinkwrap` directory in
your project, before updating each record in `npm-shrinkwrap.json` to point at those instead of the
public registry.
the installation for each package. Shrinkpack saves these files in an `archived-packages` directory
in your project, before updating each record in `npm-shrinkwrap.json` or `package-lock.json` to
point at those instead of the public registry.

```
shrinkpack .
shrinkpack
```

Each entry will look something like this

```json
"lodash": {
"version": "4.0.0",
"from": "lodash@4.0.0",
"resolved": "./node_shrinkwrap/lodash-4.0.0.tgz"
"version": "4.17.5",
"resolved": "file:archived-packages/lodash-4.17.5-b2f2f7ba2.tar",
"integrity": "sha512-Hjp8iABK5ONegfU0eVs0mjEmu+TM7XzsThYtxQ+DbCc6Rg+AtVlE/z4e0LAtCjA/kGE807VdPO4hBWMZSJ4PQw== sha512-svL3uiZf1RwhH+cWrfZn3A4+U58wbP0tGVTLQPbjplZxZ8ROD9VLuNgsRniTlLe7OlSqR79RUehXgpBW/s0IQw=="
}
```

<a href="https://asciinema.org/a/83796" target="_blank"><img src="https://asciinema.org/a/83796.png" alt="asciicast"></a>

### Check into Git

By this point, `git status` should list the following untracked files;

```
.gitignore
.npmrc
node_shrinkwrap/
archived-packages/
npm-shrinkwrap.json
package.json
```
Expand All @@ -330,8 +276,6 @@ git add .
git commit -m 'chore(project): initial commit'
```

<a href="https://asciinema.org/a/83797" target="_blank"><img src="https://asciinema.org/a/83797.png" alt="asciicast"></a>

### Clean install

We check this code into Git and tell a co-worker that it's ready for them to contribute to.
Expand All @@ -343,8 +287,8 @@ npm install --loglevel http
```

This is new behaviour, npm didn't hit the network at all. Instead it read the packages from the
`node_shrinkwrap` directory directly and installed them straight away. Shrinkpack has allowed us to
install our project without any network activity whatsoever – and in a fraction of the time.
`archived-packages` directory directly and installed them straight away. Shrinkpack has allowed us
to install our project without any network activity whatsoever – and in a fraction of the time.

If everything went to plan, the only output will be these expected warnings because we didn't choose
to add a `description` or `repository` to our `package.json`.
Expand All @@ -354,8 +298,6 @@ to add a `description` or `repository` to our `package.json`.
> npm WARN [email protected] No repository field.
> ```
<a href="https://asciinema.org/a/83799" target="_blank"><img src="https://asciinema.org/a/83799.png" alt="asciicast"></a>
### Update Dependencies
Later, we may choose to add, update, or remove some dependencies;
Expand All @@ -367,30 +309,9 @@ npm uninstall express
```
With our local `node_modules` now up to date, we now need to update our `npm-shrinkwrap.json` file
and get our `node_shrinkwrap` directory back in sync with the new changes.
and get our `archived-packages` directory back in sync with the new changes.
```
npm shrinkwrap --dev
shrinkpack
```
<a href="https://asciinema.org/a/83806" target="_blank"><img src="https://asciinema.org/a/83806.png" alt="asciicast"></a>
### Toggle Compression
The tarballs in the npm registry are gzipped for optimal network performance, but storing binary
files in Git repositories is not optimal. Git is decentralized, so every developer has the full
change history on their computer. Changes in large binary files cause Git repositories to grow by
the size of the file in question every time the file is changed and committed, this growth directly
affects the amount of data end users need to retrieve when they need to clone the repository.
You can toggle between compressed and uncompressed tarballs with `shrinkpack --compress`.
<a href="https://asciinema.org/a/83810" target="_blank"><img src="https://asciinema.org/a/83810.png" alt="asciicast"></a>
### Remove Optional Dependencies
`optionalDependencies` are removed by default, to avoid issues when trying to `npm shrinkwrap` your
project on platforms where that optional dependency was not installed. More detail is available in
this [issue comment](https://github.com/JamieMason/shrinkpack/issues/17#issuecomment-202340196) and
this behaviour can be overriden by using `shrinkpack --keep-optional`.
39 changes: 7 additions & 32 deletions cli.js
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
#!/usr/bin/env node

// node modules
var path = require('path');

// 3rd party modules
var chalk = require('chalk');
var program = require('commander');
Expand All @@ -12,46 +9,24 @@ var cli = require('./src/cli');
var version = require('./package.json').version;

// implementation
var directoryValue = process.cwd();

program
.version(version)
.option('-c, --compress', 'use compressed .tgz tarballs instead of .tar')
.option('-o, --keep-optional', 'do not exclude optional dependencies')
.arguments('[directory]')
.action(function (directory) {
directoryValue = path.resolve(directory);
});
program.version(version);

program.on('--help', onHelp);
program.parse(process.argv);

cli.run({
compress: program.compress === true,
directory: directoryValue,
keepOptional: program.keepOptional === true
});
cli.run();

function onHelp() {
console.log(' Icons:');
console.log('');
logIcon(chalk.green, '+', 'Added');
logIcon(chalk.yellow, '↓', 'Downloaded');
logIcon(chalk.magenta, '→', 'Imported from Cache');
logIcon(chalk.blue, 'i', 'Information');
logIcon(chalk.red, '-', 'Removed');
logIcon(chalk.green, '✓', 'Resolved');
logIcon(chalk.grey, '12:34', 'Time Taken');
console.log(' ' + chalk.green('+') + ' Added');
console.log(' ' + chalk.red('-') + ' Removed');
console.log(' ' + chalk.grey('12:34') + ' Time Taken');
console.log('');
console.log(' Compression:');
console.log('');
console.log(' Although compressed .tgz files have lower filesizes, storing binary files in');
console.log(' Git can result in a gradual increase in the time it takes to push to your');
console.log(' repository. Shrinkpack uses uncompressed, plain text .tar files by default,');
console.log(' which are handled optimally by Git in the same way that .md, .js, and .css');
console.log(' files are for example.');

function logIcon(colour, icon, label) {
console.log(' ' + colour(icon) + ' ' + label);
}
console.log(' repository. Shrinkpack uses uncompressed, plain text .tar files, which are');
console.log(' handled optimally by Git in the same way that .md, .js, and .css files are.');
}
Loading

0 comments on commit 730e2c8

Please sign in to comment.