Configure Timeout of Lock acquire

[marcohald]

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

QUESTION

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.2.7

Operating system (distribution) and version.

Hardware / network configuration, and filesystems used.

How much data is handled by borg?

Full borg commandline that lead to the problem (leave away excludes and passwords)

Describe the problem you're observing.

Sometimes some hosts fail with the error borg.locking.LockTimeout: Failed to create/acquire the lock [REPO]/lock.exclusive (timeout).
I don't know if the Server which is hosting the Backuprepos is the Problem or not.
How long is this Timeout and is it configurable?

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Not really

Include any warning/errors/backtraces from the system logs

using builtin fallback logging configuration
33 self tests completed in 0.14 seconds
SSH command line: ['ssh', 'backupuser@', 'borg', 'serve', '--debug']
Remote: using builtin fallback logging configuration
Remote: 33 self tests completed in 0.09 seconds
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/[REPO]'
Remote: Resolved repository path to '/[REPO]'
Remote: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve
res = f(**args)
File "/usr/lib/python3/dist-packages/borg/remote.py", line 368, in open
self.repository.enter() # clean exit handled by serve() method
File "/usr/lib/python3/dist-packages/borg/repository.py", line 203, in enter
self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock)
File "/usr/lib/python3/dist-packages/borg/repository.py", line 435, in open
self.lock = Lock(os.path.join(path, 'lock'), exclusive, timeout=lock_wait).acquire()
File "/usr/lib/python3/dist-packages/borg/locking.py", line 389, in acquire
self._wait_for_readers_finishing(remove, sleep)
File "/usr/lib/python3/dist-packages/borg/locking.py", line 402, in _wait_for_readers_finishing
self._lock.acquire()
File "/usr/lib/python3/dist-packages/borg/locking.py", line 148, in acquire
raise LockTimeout(self.path) from None
borg.locking.LockTimeout: Failed to create/acquire the lock /[REPO]/lock.exclusive (timeout).
Failed to create/acquire the lock /[REPO]/lock.exclusive (timeout).
Borg server: Platform: Linux [BACKUPSERVER] 5.10.0-26-amd64 #1 SMP Debian 5.10.197-1 (2023-09-29) x86_64
Borg server: Linux: Unknown Linux
Borg server: Borg: 1.2.7 Python: CPython 3.9.2 msgpack: 1.0.0 fuse: pyfuse3 3.2.0 [pyfuse3,llfuse]
Borg server: PID: 2477827 CWD: /home/backupuser
Borg server: sys.argv: ['/usr/bin/borg', 'serve', '--restrict-to-path', '/[REPO]']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --debug'
Platform: Linux 5.10.0-26-amd64 #1 SMP Debian 5.10.197-1 (2023-09-29) x86_64
Linux: Unknown Linux
Borg: 1.2.7 Python: CPython 3.9.18 msgpack: 1.0.7 fuse: llfuse 1.5.0 [pyfuse3,llfuse]
PID: 1659216 CWD: /root
sys.argv: ['borg', 'create', '--exclude-caches', '--stats', '--debug', '--show-rc', 'ssh://backupuser@<Backupserver.internal>/[REPO]::{hostname}-{now:%Y-%m-%d-%H%M%S}', '/etc', '/root/.borgmatic', '/srv/mysqldumps/nightly', '/srv/www']
SSH_ORIGINAL_COMMAND: None

[ThomasWaldmann]

Check the docs about --lock-wait:

https://borgbackup.readthedocs.io/en/stable/usage/general.html

The default wait time is rather short (1s).

If you run multiple borg client against the same repo, a much longer wait time can make sense to serialise the access to the repo.

If it is only one borg client accessing that repo and you always get locking errors when accessing it, it might be also that you have a leftover lock in that repo and you might need to carefully use borg break-lock (read the docs!).

https://borgbackup.readthedocs.io/en/stable/usage/lock.html#borg-break-lock

[marcohald]

Thank you, I don't know how I missed the config option.

If you run multiple borg client against the same repo, a much longer wait time can make sense to serialise the access to the repo.

Each Client has its own Repository on the Backup Server, they are scheduled via cronjobs with a random wait before the borg backup.
But it could be entire possible that it is to much load on the backup Server to respond fast enough.

I will close the issue as I think this will solve my Problem.

[marcohald]

I just found one Host where even a 10 second timeout seems not to be enough.
Is there a specific Debug Topic I should specifiy to get more Logs?
I normally use borgmatic as wrapper, but this command is executed directly without borgmatic as wrapper but with the same arguments.

root@client:~# borg create --exclude-from /etc/borgmatic/excludes --exclude-caches --lock-wait 10 --debug --show-rc ssh://[email protected]//srv/borg/client.sub.example.com::{hostname}-{now:%Y-%m-%d-%H%M%S} /etc /root/.borgmatic /srv/mysqldumps/nightly /srv/www --debug-topic repository
using builtin fallback logging configuration
Enabling debug topic borg.debug.repository
33 self tests completed in 0.09 seconds
SSH command line: ['ssh', '[email protected]', 'borg', 'serve', '--debug', '--debug-topic=borg.debug.repository']
Remote: using builtin fallback logging configuration
Remote: Enabling debug topic borg.debug.repository
Remote: 33 self tests completed in 0.08 seconds
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'//srv/borg/client.sub.example.com'
Remote: Resolved repository path to '/srv/borg/client.sub.example.com'
Remote: Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve
    res = f(**args)

  File "/usr/lib/python3/dist-packages/borg/remote.py", line 368, in open
    self.repository.__enter__()  # clean exit handled by serve() method

  File "/usr/lib/python3/dist-packages/borg/repository.py", line 203, in __enter__
    self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock)

  File "/usr/lib/python3/dist-packages/borg/repository.py", line 435, in open
    self.lock = Lock(os.path.join(path, 'lock'), exclusive, timeout=lock_wait).acquire()

  File "/usr/lib/python3/dist-packages/borg/locking.py", line 389, in acquire
    self._wait_for_readers_finishing(remove, sleep)

  File "/usr/lib/python3/dist-packages/borg/locking.py", line 402, in _wait_for_readers_finishing
    self._lock.acquire()

  File "/usr/lib/python3/dist-packages/borg/locking.py", line 148, in acquire
    raise LockTimeout(self.path) from None

borg.locking.LockTimeout: Failed to create/acquire the lock /srv/borg/client.sub.example.com/lock.exclusive (timeout).

Failed to create/acquire the lock /srv/borg/client.sub.example.com/lock.exclusive (timeout).
Borg server: Platform: Linux server 5.10.0-26-amd64 #1 SMP Debian 5.10.197-1 (2023-09-29) x86_64
Borg server: Linux: Unknown Linux
Borg server: Borg: 1.2.7  Python: CPython 3.9.2 msgpack: 1.0.0 fuse: pyfuse3 3.2.0 [pyfuse3,llfuse]
Borg server: PID: 2564845  CWD: /home/backupuser
Borg server: sys.argv: ['/usr/bin/borg', 'serve', '--restrict-to-path', '/srv/borg/client.sub.example.com']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --debug --debug-topic=borg.debug.repository'
Platform: Linux client.sub.example.com 5.10.0-26-amd64 #1 SMP Debian 5.10.197-1 (2023-09-29) x86_64
Linux: Unknown Linux
Borg: 1.2.7  Python: CPython 3.9.18 msgpack: 1.0.7 fuse: llfuse 1.5.0 [pyfuse3,llfuse]
PID: 428389  CWD: /root
sys.argv: ['borg', 'create', '--exclude-from', '/etc/borgmatic/excludes', '--exclude-caches', '--lock-wait', '10', '--debug', '--show-rc', 'ssh://[email protected]//srv/borg/client.sub.example.com::{hostname}-{now:%Y-%m-%d-%H%M%S}', '/etc', '/root/.borgmatic', '/srv/mysqldumps/nightly', '/srv/www', '--debug-topic', 'repository']
SSH_ORIGINAL_COMMAND: None

terminating with error status, rc 2

[ThomasWaldmann]

See my first reply, second half.

[ThomasWaldmann]

Guess this is solved?