Why there is no review articles about dwarfs on the web, if it looks like better SquashFS?

[vitaly-zdanevich]

For me it looks strange. SquashFS somehow better?

Do you know some notable projects that use DwarFS? I know only Conty.

SquashFS is more popular because it integrated into Linux Kernel? Would you like to integrate DwarFS?

[mhx]

For me it looks strange. SquashFS somehow better?

SquashFS is almost 20 years older than DwarFS, so it is much more established. Better? I guess that really depends on the use case.

Do you know some notable projects that use DwarFS? I know only Conty.

There's (at least) also Tebako.

Then again, my primary use case for DwarFS is by itself, not through any other projects.

SquashFS is more popular because it integrated into Linux Kernel?

Possibly. But then again, even if it was in the Linux kernel, what about macOS and Windows?

Would you like to integrate DwarFS?

I personally would not like to integrate it. That's not because I wouldn't like it to be part of the kernel, but because I would rather spend my time improving DwarFS.

But about your original question...

Why there is no review articles about dwarfs on the web, if it looks like better SquashFS?

I have no idea! :)

It's discussed on HN about once every two years (which is by far what has been driving most people to this project) and the occasional article on other sites with rather insignificant reach.

I guess I just suck at marketing.

[vitaly-zdanevich]

I just suck at marketing.

Best marketing - is a good product.

[vitaly-zdanevich]

I added link to your repo at https://en.wikipedia.org/wiki/SquashFS#See_also

[mhx]

I guess here are some of the actual issues that might prevent DwarFS from being more popular:

• It's not (yet) available as a Debian/Ubuntu package.
• Using it on Windows is not for the faint of heart unless you're coming from a Unix / command-line background.
• Probably the dominant use case for SquashFS is live images. I honestly haven't seen SquashFS being used for anything else. That use case is significantly simplified by SquashFS being in the kernel for 20+ years. It seems that it should be possible to use FUSE-based file systems for live images (at least there's an option to disable FUSE in the live-boot docs). Then again, I'm not sure that DwarFS would be significantly better at that particular use-case. It would certainly compress better, but likely not by more than 20-30% (that's the numbers I got from a raspbian image). It might be worth trying to get it to work though! I can think of certain cases that would definitely benefit from it, e.g. a live image with hundreds of different version of a certain toolchain.

There are a few things that I could imaging could help the popularity of DwarFS. For example, using it as a storage driver for docker seems like a really interesting idea.

[mhx]

Regarding read performance, this is probably dumb but it's the simplest I could think of right now: just creating a tarball and piping that through dd using a freshly mounted file system.

First for the original SquashFS:

$ time tar cf - gentoo-live-filesystem | dd status=progress >/dev/null
12741458432 bytes (13 GB, 12 GiB) copied, 348 s, 36.6 MB/s 
24891100+0 records in
24891100+0 records out
12744243200 bytes (13 GB, 12 GiB) copied, 348.888 s, 36.5 MB/s

user	3.481
system	9.482
total	5:48.89

Then the "optimized" DwarFS ("optimized" means default compression, plus bumping max-cluster-size for 256k for similarity ordering, plus use a smaller window shift for segmenting — so it's really just trying to optimize for size):

$ time tar cf - gentoo-live-filesystem | dd status=progress >/dev/null
12737628672 bytes (13 GB, 12 GiB) copied, 91 s, 140 MB/s 
24891100+0 records in
24891100+0 records out
12744243200 bytes (13 GB, 12 GiB) copied, 91.1617 s, 140 MB/s

user	3.109
system	8.671
total	1:31.16

That's with 2 GiB cache for the FUSE driver. Cache size makes a massive difference for this particular test, e.g. using an 8 GiB cache:

12739318272 bytes (13 GB, 12 GiB) copied, 33 s, 386 MB/s 

And using a 512 MiB cache (the default):

12741642752 bytes (13 GB, 12 GiB) copied, 528 s, 24.1 MB/s

In a real-world scenario, things may be quite different and you're probably not even going to notice a difference in cache size as it's highly unlikely that you're going to read every single file. In one of my use cases, having an image with 1000+ versions of Perl and running a module build and test suite for every single version, I couldn't tell the difference between a SquashFS or DwarFS image.

[vitaly-zdanevich]

What is gentoo-live-l9.dwarfs vs gentoo-live-opt.dwarfs?

[mhx]

What is gentoo-live-l9.dwarfs vs gentoo-live-opt.dwarfs?

The -l9 images use the level 9 compression defaults. Compared to -opt, this switches from zstd to lzma block compression and from 16 MiB blocks to 64 MiB blocks. While this gives another significant reduction is image size, I wouldn't recommend using it for a live image. Smaller blocks work better with smaller caches and require less data to be unpacked without being used. And zstd decompresses about an order of magnitude faster than lzma. These options are better for archiving data that you only need to access infrequently.

[vitaly-zdanevich]

Thanks, can I download your repacks, to test with real USB?

[vitaly-zdanevich]