Repository navigation
[BUG] tarball data currupted when installing from 'chunked encoding' server. #3884
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
While using npm v7/8 to install packages from our private npm registry we'll regularly see tar extract and integrity errors
npm WARN tar TAR_ENTRY_INVALID checksum failure
npm WARN tar zlib: incorrect data check
npm WARN tarball tarball data for lodash@http://localhost:4000/lodash/-/lodash-4.17.19.tgz (sha512-JNvd8XER9GQX0v2qJgsaN/mzFCNA5BRe/j8JN9d+tWyGLSodKQHKFicdwNYzWwI3wjRnaKPsGj1XkBjx/F96DQ==) seems to be corrupted. Trying again.
npm ERR! code Z_DATA_ERROR
npm ERR! errno -3
npm ERR! zlib: incorrect data check
I can reproduce these errors using a chunked-encoding proxy in front of registry.npmjs.org. I can resolve the issue by changing make-fetch-happen's integrity check from a MinipassPipeline to a basic stream. I don't know if that really fixes the issue or just changes the timing enough to hide it.
Expected Behavior
npm should be able to install with any valid http server.
Steps To Reproduce
-
In this environment...
I've reproduced this on Ubuntu with node 16, npm v7 and npm v8. I can't reproduce with v6 but one user has reported the issue with npm v6. -
With this config...
# my example proxy, or any other http registry that uses chunked-encoding and small chunks.
registry=http://localhost:4000/
node proxy.js
- Run '...'
$ mkdir example
$ cd example
$ # create package.json so node_modules will be installed here
$ echo {} > package.json
$ # remove cache. cannot reproduce with cache disabled or a cache hit.
$ rm ~/.npm/_cacache node_modules -rf
$ install any package
$ npm install react
- See error...
npm WARN tarball tarball data for react@http://localhost:4000/react/-/react-17.0.2.tgz (sha512-gnhPt75i/dq/z3/6q/0asP78D0u592D5L1pd7M8P+dck6Fu/jJeL6iVVK23fptSUZj8Vjf++7wXA8UNclGQcbA==) seems to be corrupted. Trying again.
npm ERR! code Z_DATA_ERROR
npm ERR! errno -3
npm ERR! zlib: incorrect data check
Environment
- OS: Ubuntu
- Node: 16.11.0, 16.8.0
- npm: v8.0.0, v7.21.0
Reacted by Justin "J.R." Hill and London Brown
Activity
everett1992 commentedon Oct 12, 2021
I think that my proxy implementation is correct because files downloaded with curl have the correct checksum and extract cleanly.
Our actual private registry is written in java, so I don't think it's a bug in common.
everett1992 commentedon Oct 12, 2021
I've traced thru pacote, minipass-fetch, and make-fetch-happen and the stream's integrity is correct thru here
https://github.com/everett1992/make-fetch-happen/blob/chunked-encoding/lib/remote.js#L53
But the stream here is wrong
https://github.com/everett1992/make-fetch-happen/blob/chunked-encoding/lib/cache/entry.js#L288
replace minipass pipeline with simple pipe
booniepepper commentedon Nov 29, 2021
For some context, Caleb (and I) are build engineers at Amazon. This is blocking us from moving our company onto NPM 7 because of the way we handle internal repository mirrors.
demonstrate fetch tar extract errors with chunked encoding.
test writing chunk wile flowing with buffer
test writing chunk wile flowing with buffer
buffer chunk after flushing buffer in write if paused
buffer chunk after flushing buffer in write if paused
wraithgar commentedon Jan 31, 2022
npm is now on the version of minipass that had this fixed. isaacs/minipass#28.
Closing as fixed, please reopen if this is still an issue.
Jiasm commentedon Mar 21, 2023
Our private Registry service had same problem after the migration of
express.js->koa.js, and finally concluded thatkoawill override theContent-Typeof the return value of thestreamtype asContent-Type: application/octet-stream, butcharsetis not set. It is suspected that this is whynpm v7cannot read the tarball normally. When we manually overrideContent-Typetoapplication/octet- stream; charset=utf-8works fine.Reference: https://github.com/koajs/koa/blob/65f9c939e173ff2fab82ee9229a2213690959426/docs/api/response.md#buffer