Compression Challenge: Truly Lossless?

[c5huracan]

@YassineYousfi I know I missed the compression challenge but poking at it anyway for educational purposes. Are you looking for truly lossless performance or is there some level of loss you might accept if the compression ratio is > 3.4?

[YassineYousfi]

The goal was truly lossless.

But if you have an interesting lossy solution, with a reasonable measure of loss, feel free to explore it and tell us about it!

[c5huracan]

@YassineYousfi Please take a look at the gist of this notebook: https://gist.github.com/c5huracan/f13725ed896754ee2896c1522cf80469 It is still a bit janky but I wanted to get you something to review. I am getting an average compression rate of ~3x (+/-0.20) Not quite as good yet as my preliminary investigation was suggesting I might expect. Let me know what questions, feedback, suggestions for improvement, etc. you have for me. Thank you!

[YassineYousfi]

This is interesting, but doesn't say much about the incurred loss.
How do the videos look like after the lossy process? Can you show examples in your notebook?

[c5huracan]

I have since determined that this approach might have a reconstruction problem and the metadata for the Huffman tree isn't being accounted for in my math but I will take another look at this during the coming week.

[c5huracan]

@YassineYousfi Please have at look at this one: https://gist.github.com/c5huracan/dd24af6389497f0c1cf3dc6c340b4a87 This is preliminary but I might have found a 3.2x lossless compression solution. Any feedback is appreciated! Thank you!

[YassineYousfi]

I don't see any decompression function. Are you sure you can decompress to original data...?

[c5huracan]

You are spot on @YassineYousfi This run also has a reconstruction problem and as you know, adding indices to keep track of locations adds overhead back into the equation. So for now I am looking at alternatives that achieve a better balance between compression and overhead. I am currently testing optimized lzma settings and trying to pack the resulting metadata into a smaller space. Something like this:

def pack_dimensions(frames: int, height: int, width: int) -> bytes:
    """
    Pack dimensions into 3 bytes (24 bits):
    - frames: 11 bits (up to 2048)
    - height: 4 bits (up to 16)
    - width: 5 bits (up to 32)
    """
    packed = (frames << 9) | (height << 5) | width
    return packed.to_bytes(3, byteorder='big')

def unpack_dimensions(metadata: bytes) -> tuple:
    """
    Unpack 3 bytes back into dimensions
    """
    packed = int.from_bytes(metadata[:3], byteorder='big')
    width = packed & 0x1F          # last 5 bits
    height = (packed >> 5) & 0x0F  # next 4 bits
    frames = (packed >> 9) & 0x7FF # next 11 bits
    return (frames, height, width)

# Update compress_tokens to use this:
def compress_tokens(tokens: np.ndarray) -> bytes:
    metadata = pack_dimensions(tokens.shape[0], tokens.shape[1], tokens.shape[2])
    arranged_data = tokens.astype(np.int16).reshape(-1, 128).T.ravel()
    
    compressed = lzma.compress(
        arranged_data.tobytes(),
        format=lzma.FORMAT_RAW,
        filters=[{
            "id": lzma.FILTER_LZMA2,
            "dict_size": 1 << 17,
            "lc": 4,
            "lp": 0,
            "pb": 2,
            "mode": lzma.MODE_NORMAL
        }]
    )
    
    return metadata + compressed

[c5huracan]

@YassineYousfi Quick update: I haven't forgotten about this one. This week I mainly ran into and resolved some lingering .npy formatting issues while trying to get my decompressed tokens running through the VQ-VAE decoder properly. This is working now in the "enhanced" lzma pipeline but the compression rate went the wrong way: 1.4x.

https://gist.github.com/c5huracan/44605dda659c59121703e0b210a3ba87

At this point, I am a bit less naive about how solvable this is than I was last month ;-)

I will keep working on this and let you see what the video quality is for any lossy output worth sharing and / or any lossless output that truly moves the needle toward better compression rates. Good luck with Hack this weekend! Perhaps someone up there will crack the code on better compression in the token space.

BTW, how is the video quality of the Arithmetic Coding with GPT and the Self-Compressing Neural Network solutions posted on the Leaderboard?