increase segment size?

[ThomasWaldmann]

Q: "Could it be beneficial to raise the repo segment size? Maybe to a multiple of the new default chunk size?"

A: There is not really a "default chunk size" though, it's only some statistically medium value (and thus, there is no precise multiple practically, compression and encryption also come into play).

But roughly, it used to target 64kiB (chunk) and 5MiB (segment).
Now (1.0) it targets 2MiB (chunk) and still 5MiB (segment) yet.

Considerations:

• memory needs for buffer (usually no problem)
• better efficiency? iirc it does an fsync at segment end, which makes borg sit there idle until the write has completed.
• compaction: if a chunk gets deleted from a segment, all remaining chunks need to be copied to a new segment. the bigger the segment size is, the more stuff has to be copied.
• if chunk size is relatively close to the segment size, the final segment size can be quite larger than the target segment size (up to 5MiB + 2MiB - 1).

[fsironman]

While playing with borg I noticed theres a setting for it in the config file.
After "borg init" I changed it and then did normal backup to it.

Will anything terrible happen because I did this ?

[ThomasWaldmann]

I never tried, but IIRC it just reads and follows that value when it opens a repo (there is just no init option to set that value). Expected outcome would be a repo with some 5MB segments (before change) and some segments of other size (after the change).

[pszxzsd]

I did a small speed test for comparison, backing up the same data to two repos, one with 5 MiB default segments, one modified to save 10 MiB segments. After working around caching issues (making the second backup always faster) I found no significant difference in backup times.

I used borg 0.30.0 with --verbose --stats --progress --chunker-params 19,23,21,4095 --compression lz4. The sizes refer to the compressed and deduped output, times are mm:ss.

	5 MiB	10 MiB
Backup 1 (6.11 GB)	37:59	38:15
Backup 2 (+354 MB)	03:09	03:08

I noticed that the number of chunks is slightly different, after the second backup the 5 MiB repo has 69922 unique chunks, the other 69935. Bug or harmless side effect?

[ThomasWaldmann]

OK, I am closing this as it currently seems pointless to increate segment size.

@pszxzsd thanks for your experiments / measurements. Yes, the chunk difference is strange. Maybe open another issue if you can reproduce.