Some branches have disappeared from the GUI.

[raptyk]

Description

I am in the process of migrating from the old gitolite3 repository to gitea.

I have migrated 8 repositories correctly however with 3 I have a problem. The problem is that in the Gitea GUI and in the postgress database in the "branch" table I see only a few of the dozen branches. These branches are present after downloading the code from the new gitea repository however I can't see them in the GUI.

Running the "Sync missed branches from git data to databases" option doesn't help, and I only see a few errors in the logs:

2023/12/12 17:32:54 ...repository/branch.go:384:handlerBranchSync() [E] syncRepoBranches [32] failed: pq: invalid byte sequence for encoding "UTF8": 0xea 0x6b 0x73
2023/12/12 17:32:54 ...repository/branch.go:384:handlerBranchSync() [E] syncRepoBranches [24] failed: pq: invalid byte sequence for encoding "UTF8": 0xbf

In a nutshell what I did:

1. I downloaded the old repository to a local drive along with all the branches.
2. I added a second origin pointing to the gitea server in the same directory.
3. I uploaded all the branches to the new gitea repository.
4. in the GUI I see only a few branches, after downloading from the new repo I see all the branches.

I suspect this has to do with coding some commit or other special character topics. How to solve this problem?

Gitea Version

1.21.1 (in docker & postgresql)

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

Git Version

No response

Operating System

official docker image, postgresql

How are you running Gitea?

docker

Database

PostgreSQL

[lunny]

Maybe related #28162

[raptyk]

Many hours later... I've found that some commit messages contained special characters outside of Unicode (Polish letters), and Gitea was unable to insert those messages into the database (a silent error was noted in the log).

I've found some solutions/workarounds for this type of migration problem, but I still think it should be fixed in Gitea because users may not even be aware that some branches have disappeared.

1. Download sources from the original Git repository.

2. Replace specia characters and change the encoding in every commit message from the original (Windows-1250) to UTF-8. Some letters might be distorted.
   replace all Ą->A, Ń->N, Ł->L etc:
   git filter-repo --replace-refs delete-no-add --replace-message polskie.txt --force
   change encoding:
   git filter-branch --msg-filter 'iconv -f windows-1250 -t utf-8 || cat' -- --all

3. Push the converted repository to Gitea.

polskie.txt

[wxiaoguang]

Maybe it has been fixed by #28356 and has been backported to 1.21.2

For the old missing branches, you can sync them from your admin panel.