gitea dump takes a very long time

[SorenHolstHansen]

Description

Doing gitea dump inside docker takes a very long time. I have waited more than 30 minutes on an almost empty instance. What is taking so long?

The logs stop here

2024/01/26 10:20:52 .../setting/security.go:168:loadSecurityFrom() [W] Enabling Query API Auth tokens is not recommended. DISABLE_QUERY_AUTH_TOKEN will default to true in gitea 1.23 and will be removed in gitea 1.24.
2024/01/26 10:20:52 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled
2024/01/26 10:20:52 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled
2024/01/26 10:20:52 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2024/01/26 10:20:52 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: local
2024/01/26 10:20:52 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/attachments
2024/01/26 10:20:52 ...s/storage/storage.go:166:initAvatars() [I] Initialising Avatar storage with type: local
2024/01/26 10:20:52 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/avatars
2024/01/26 10:20:52 ...s/storage/storage.go:192:initRepoAvatars() [I] Initialising Repository Avatar storage with type: local
2024/01/26 10:20:52 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/repo-avatars
2024/01/26 10:20:52 ...s/storage/storage.go:186:initLFS() [I] Initialising LFS storage with type: local
2024/01/26 10:20:52 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/git/lfs
2024/01/26 10:20:52 ...s/storage/storage.go:198:initRepoArchives() [I] Initialising Repository Archive storage with type: local
2024/01/26 10:20:52 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/repo-archive
2024/01/26 10:20:52 ...s/storage/storage.go:208:initPackages() [I] Initialising Packages storage with type: local
2024/01/26 10:20:52 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/packages
2024/01/26 10:20:52 ...s/storage/storage.go:219:initActions() [I] Initialising Actions storage with type: local
2024/01/26 10:20:52 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/actions_log
2024/01/26 10:20:52 ...s/storage/storage.go:223:initActions() [I] Initialising ActionsArtifacts storage with type: local
2024/01/26 10:20:52 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/actions_artifacts
2024/01/26 10:20:52 cmd/dump.go:265:runDump() [I] Dumping local repositories... /var/lib/gitea/git/repositories
2024/01/26 10:20:53 cmd/dump.go:306:runDump() [I] Dumping database...
2024/01/26 10:20:53 cmd/dump.go:318:runDump() [I] Adding custom configuration file from /etc/gitea/app.ini
2024/01/26 10:20:53 cmd/dump.go:334:runDump() [I] Custom dir /var/lib/gitea/custom is inside data dir /var/lib/gitea, skipped
2024/01/26 10:20:53 cmd/dump.go:346:runDump() [I] Packing data directory.../var/lib/gitea

Gitea Version

1.21.3

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

With docker-rootless

Database

SQLite

[eeyrjmr]

Do you stop gitea before starting a dump,.there could be locks held on the database.

My backup process is

1. Stop gitea
2. Gitea dump
3. Move archive
4. Gitea start

[SorenHolstHansen]

Yes, at least if i understand it correctly. I closed the browser instance, though guess that is not enough. However running gitea manager shutdown inside docker just shots down the container too, so can't really do a dump after that

[eeyrjmr]

So rather than shutting down the docker, can you stop the gitea service within the docker?

If you are able to run "gitea dump" you can run "gitea manager stop" (or rc-service gitea stop ... Assuming alpine/Gentoo based docker)

There is also a "gitea manager flush-queues"

There is either some git process or database query/task that is postponing the dump command from being executed in a timely fashion and thus these activities need to be completed to permit gitea exclusive access. This is why I prefer the stop method to ensure the dB is consistent when a dump is performed

[SorenHolstHansen]

Hmmm, doesn't seem to work. gitea manager stop isn't a valid command though.

[maindishes]

Did you resolve the issue?
In my case, I'm using k3s-gitea, but I'm facing a similar situation when I run the ./gitea dump command.

2024/03/12 06:56:33 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled
2024/03/12 06:56:33 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled
2024/03/12 06:56:33 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2024/03/12 06:56:33 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: local
2024/03/12 06:56:33 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/attachments
2024/03/12 06:56:33 ...s/storage/storage.go:166:initAvatars() [I] Initialising Avatar storage with type: local
2024/03/12 06:56:33 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/avatars
2024/03/12 06:56:33 ...s/storage/storage.go:192:initRepoAvatars() [I] Initialising Repository Avatar storage with type: local
2024/03/12 06:56:33 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/repo-avatars
2024/03/12 06:56:33 ...s/storage/storage.go:198:initRepoArchives() [I] Initialising Repository Archive storage with type: local
2024/03/12 06:56:33 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/repo-archive
2024/03/12 06:56:33 ...s/storage/storage.go:208:initPackages() [I] Initialising Packages storage with type: local
2024/03/12 06:56:33 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/packages
2024/03/12 06:56:33 ...s/storage/storage.go:219:initActions() [I] Initialising Actions storage with type: local
2024/03/12 06:56:33 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/actions_log
2024/03/12 06:56:33 ...s/storage/storage.go:223:initActions() [I] Initialising ActionsArtifacts storage with type: local
2024/03/12 06:56:33 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/actions_artifacts
2024/03/12 06:56:33 cmd/dump.go:265:runDump() [I] Dumping local repositories... /data/git/gitea-repositories
2024/03/12 06:56:33 cmd/dump.go:273:runDump() [I] LFS isn't enabled. Skip dumping LFS data
2024/03/12 06:56:33 cmd/dump.go:306:runDump() [I] Dumping database...
2024/03/12 06:56:33 cmd/dump.go:318:runDump() [I] Adding custom configuration file from /data/gitea/conf/app.ini
2024/03/12 06:56:33 cmd/dump.go:334:runDump() [I] Custom dir /data/gitea is inside data dir /data, skipped
2024/03/12 06:56:33 cmd/dump.go:346:runDump() [I] Packing data directory.../data

there is another way to backup gitea ? in kubernetes

[wxiaoguang]

It's likely the dump output file is in the Gitea's data directory, then it packs itself again and again.

Some cases could be fixed by #30240 , while there might be some still simliar cases.

So you could use --file option to set the output file to a non-Gitea directory

[GiteaBot]

We close issues that need feedback from the author if there were no new comments for a month. 🍵