panic: runtime error: slice bounds out of range [8290305:212989] in compress/flate/fast_encoder.go:61

[zachad]

Hello,

We've just run into a error similar to #630. It seems different enough that to open a new issue. I apologize for the formatting or typos. It needed to be copied out of a screenshot and sanitized a little.

This is seen in klauspost/compress v1.17.7.

Operation: Creating backup
Progress: 432.26 MiB / 121.82 GiB [>----] 0.35% (33.15 MiB/s, ETA: 1h00m49s)

panic: runtime error: slice bounds out of range [8290305:212989]

goroutine 2005 [running]:
github.com/klauspost/compress/flate.load3232(...)
        /src/vendor/github.com/klauspost/compress/flate/fast_encoder.go:61
github.com/klauspost/compress/flate.(*fastEncL5).Encode(0xc0008be000, 0xc0002a088, {0xc0002160002, 0xffff, 0xffff})
       /src/vendor/github.com/klauspost/compress/flate/level5.go:143 +0x115f
github.com/klauspost/compress/flate.(*compressor).storeFast(0xc0002aa000)
        /src/vendor/github.com/klauspost/compress/flate/deflate.go:758 +0x16a
github.com/klauspost/compress/flate.(*compressor).write(0xc0002aa000, {0xc0025280002, 0x100000, 0xc00069e0002})
        /src/vendor/github.com/klauspost/compress/flate/deflate.go:783 +0x88
github.com/klauspost/compress/flate.(*Writer).Write(...)
        /src/vendor/github.com/klauspost/compress/flate/deflate.go:971
github.com/klauspost/pgzip.(*Writer).compressBlock(0x000288000, {0x002528000, 0x100000, 0x110000}, {0xc00069e000, 0x4000, 0x110000}, {0xc000ab61202, 0xc000961802}, 0x0)
        /src/vendor/github.com/klauspost/pgzip/gzip.go:430 +0x1ff
created by github.com/klauspost/pgzip.(*Writer).compressCurrent in goroutine 1
        /src/vendor/github.com/klauspost/pgzip/gzip.go:270

We're working on a reproduction, but it appears to only happen at a customer site.

Is there additional information I can try to provide to help debug this?

[klauspost]

Thanks! I will also see what I can figure out on my side.

[klauspost]

@zachad Do you have the integration code with pgzip available? AFAICT shouldn't be relevant.

(fwiw, it is unrelated to #630)

[klauspost]

This is a tricky one. Still haven't been able to locate any way this can happen.

[zachad]

I really appreciate you digging into it - we too have been unable to reproduce it on any other machines. We're starting to wonder if it's an underlying hardware problem, in which case, all bets of reproducing are off.

[klauspost]

There are some rare conditions that only come up every 2GB (an offset reset) - for pgzip it will be even rarer.

This is my main investigation thread. But nothing provably wrong yet.

A bitflip could trigger it, but I want to rule out everything else before that.

[klauspost]

For reference which platform/arch is this?

I can't seem to find any "gap", where this could fall through like this.

[zachad]

This was x86-64 Rocky Linux. We've since experienced other issues with the system, where it will stop responding in the middle of heavy data transfers. I believe this is a hardware issue. Sorry for the wild goose chase.

[klauspost]

Hey - I found #1043 while sifting through the code, so something good came from it :)

Thanks for the feedback. Of course, you can always re-open if you should encounter it elsewhere.