ssh_dispatch_run_fatal when git.subprocess=true

[marcinjahn]

Description

After updating jj to 0.27, I started having issues with fetching updates from my git remote:

:> jj git fetch
Error: Git process failed: External git program failed:
ssh_dispatch_run_fatal: Connection to 192.168.1.109 port 7999: error in libcrypto
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Setting git.subprocess = false in my config resolves the issue, I can fetch without issues.

Expected Behavior

jj git fetch should continue to work with default jj settings.

Actual Behavior

fetch fails with git.subprocess = true

Specifications

• Platform: Fedora 41 x64
• Version: 0.27

[emilazy]

Thanks for the report!

I see SSH is connecting to a local network address on an unusual port; would you be able to share anything relevant from your repository remote configuration, your Git configuration, and your SSH configuration?

It’d also be useful to know your Git version and if a direct git fetch works.

[bsdinis]

Hi! Thanks for opening this issue.
Does git work for you?

Also, could you paste the output of jj git fetch --debug?

[marcinjahn]

The IP address and port

it's a remote that is hosted by the company I work for, and it so happens that they chose this IP and port

My git config

core.editor=nvim
core.pager=delta
core.excludesfile=/home/mnj/.gitignore
init.defaultbranch=main
rebase.autosquash=true
interactive.difffilter=delta --color-only
delta.navigate=true
delta.side-by-side=true
merge.conflictstyle=diff3
diff.colormoved=default
diff.tool=difftastic
diff.external=difft
difftool.prompt=false
difftool.difftastic.cmd=difft $LOCAL $REMOTE
pager.difftool=true
alias.dft=difftool
alias.dlog=!f() { GIT_EXTERNAL_DIFF=difft git log -p --ext-diff $@; }; f
core.repositoryformatversion=0
core.filemode=true
core.logallrefupdates=false
core.symlinks=true
core.ignorecase=false
core.precomposeunicode=false
remote.origin.url=ssh://[email protected]:7999/~m.jahn/service-nugets.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*

My SSH config

Host stash.workspace.com
    HostKeyAlgorithms=+ssh-rsa
    PubkeyAcceptedKeyTypes +ssh-rsa

Git version

git version 2.48.1

Does git fetch work?

Surprisingly (at least for me), no, it doesn't. Only jj with git.subprocess = false does work.

Debug output

>: jj git fetch --debug
2025-03-10T17:00:21.532317Z  INFO jj_cli::cli_util: debug logging enabled
2025-03-10T17:00:21.534079Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:workspace_helper:workspace_helper_with_stats:maybe_snapshot_impl:snapshot_working_copy:base_ignores: globset: glob converted to regex: Glob { glob: "**/.aider*", re: "(?-u)^(?:/?|.*/)\\.aider[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Literal('.'), Literal('a'), Literal('i'), Literal('d'), Literal('e'), Literal('r'), ZeroOrMore]) }
2025-03-10T17:00:21.534093Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:workspace_helper:workspace_helper_with_stats:maybe_snapshot_impl:snapshot_working_copy:base_ignores: globset: built glob set; 0 literals, 0 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 1 regexes
2025-03-10T17:00:21.535471Z DEBUG globset: glob converted to regex: Glob { glob: "**/[Dd]ebug", re: "(?-u)^(?:/?|.*/)[Dd]ebug$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Class { negated: false, ranges: [('D', 'D'), ('d', 'd')] }, Literal('e'), Literal('b'), Literal('u'), Literal('g')]) }
2025-03-10T17:00:21.535484Z DEBUG globset: glob converted to regex: Glob { glob: "**/[Dd]ebugPublic", re: "(?-u)^(?:/?|.*/)[Dd]ebugPublic$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Class { negated: false, ranges: [('D', 'D'), ('d', 'd')] }, Literal('e'), Literal('b'), Literal('u'), Literal('g'), Literal('P'), Literal('u'), Literal('b'), Literal('l'), Literal('i'), Literal('c')]) }
2025-03-10T17:00:21.535494Z DEBUG globset: glob converted to regex: Glob { glob: "**/[Rr]elease", re: "(?-u)^(?:/?|.*/)[Rr]elease$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Class { negated: false, ranges: [('R', 'R'), ('r', 'r')] }, Literal('e'), Literal('l'), Literal('e'), Literal('a'), Literal('s'), Literal('e')]) }
2025-03-10T17:00:21.535503Z DEBUG globset: glob converted to regex: Glob { glob: "**/[Rr]eleases", re: "(?-u)^(?:/?|.*/)[Rr]eleases$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Class { negated: false, ranges: [('R', 'R'), ('r', 'r')] }, Literal('e'), Literal('l'), Literal('e'), Literal('a'), Literal('s'), Literal('e'), Literal('s')]) }
2025-03-10T17:00:21.535512Z DEBUG globset: glob converted to regex: Glob { glob: "**/[Bb]in", re: "(?-u)^(?:/?|.*/)[Bb]in$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Class { negated: false, ranges: [('B', 'B'), ('b', 'b')] }, Literal('i'), Literal('n')]) }
2025-03-10T17:00:21.535521Z DEBUG globset: glob converted to regex: Glob { glob: "**/[Oo]bj", re: "(?-u)^(?:/?|.*/)[Oo]bj$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Class { negated: false, ranges: [('O', 'O'), ('o', 'o')] }, Literal('b'), Literal('j')]) }
2025-03-10T17:00:21.535529Z DEBUG globset: glob converted to regex: Glob { glob: "**/[Ll]og", re: "(?-u)^(?:/?|.*/)[Ll]og$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Class { negated: false, ranges: [('L', 'L'), ('l', 'l')] }, Literal('o'), Literal('g'
2025-03-10T17:00:21.535537Z DEBUG globset: glob converted to regex: Glob { glob: "**/[Tt]est[Rr]esult*", re: "(?-u)^(?:/?|.*/)[Tt]est[Rr]esult[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Class { negated: false, ranges: [('T', 'T'), ('t', 't')] }, Literal('e'), Literal('s'), Literal('t'), Class { negated: false, ranges: [('R', 'R'), ('r', 'r')] }, Literal('e'), Literal('s'), Literal('u'), Literal('l'), Literal('t'), ZeroOrMore]) }
2025-03-10T17:00:21.535546Z DEBUG globset: glob converted to regex: Glob { glob: "**/[Bb]uild[Ll]og.*", re: "(?-u)^(?:/?|.*/)[Bb]uild[Ll]og\\.[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Class { negated: false, ranges: [('B', 'B'), ('b', 'b')] }, Literal('u'), Literal('i'), Literal('l'), Literal('d'), Class { negated: false, ranges: [('L', 'L'), ('l', 'l')] }, Literal('o'), Literal('g'), Literal('.'), ZeroOrMore]) }
2025-03-10T17:00:21.535565Z DEBUG globset: glob converted to regex: Glob { glob: "**/[Dd]ebugPS", re: "(?-u)^(?:/?|.*/)[Dd]ebugPS$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Class { negated: false, ranges: [('D', 'D'), ('d', 'd')] }, Literal('e'), Literal('b'), Literal('u'), Literal('g'), Literal('P'), Literal('S')]) }
2025-03-10T17:00:21.535574Z DEBUG globset: glob converted to regex: Glob { glob: "**/[Rr]eleasePS", re: "(?-u)^(?:/?|.*/)[Rr]eleasePS$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Class { negated: false, ranges: [('R', 'R'), ('r', 'r')] }, Literal('e'), Literal('l'), Literal('e'), Literal('a'), Literal('s'), Literal('e'), Literal('P'), Literal('S')]) }
2025-03-10T17:00:21.535597Z DEBUG globset: glob converted to regex: Glob { glob: "**/_Chutzpah*", re: "(?-u)^(?:/?|.*/)_Chutzpah[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Literal('_'), Literal('C'), Literal('h'), Literal('u'), Literal('t'), Literal('z'), Literal('p'), Literal('a'), Literal('h'), ZeroOrMore]) }
2025-03-10T17:00:21.535618Z DEBUG globset: glob converted to regex: Glob { glob: "**/_ReSharper*", re: "(?-u)^(?:/?|.*/)_ReSharper[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Literal('_'), Literal('R'), Literal('e'), Literal('S'), Literal('h'), Literal('a'), Literal('r'), Literal('p'), Literal('e'), Literal('r'), ZeroOrMore]) }
2025-03-10T17:00:21.535627Z DEBUG globset: glob converted to regex: Glob { glob: "**/*.[Rr]e[Ss]harper", re: "(?-u)^(?:/?|.*/)[^/]*\\.[Rr]e[Ss]harper$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, ZeroOrMore, Literal('.'), Class { negated: false, ranges: [('R', 'R'), ('r', 'r')] }, Literal('e'), Class { negated: false, ranges: [('S', 'S'), ('s', 's')] }, Literal('h'), Literal('a'), Literal('r'), Literal('p'), Literal('e'), Literal('r')]) }
2025-03-10T17:00:21.535638Z DEBUG globset: glob converted to regex: Glob { glob: "**/_TeamCity*", re: "(?-u)^(?:/?|.*/)_TeamCity[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Literal('_'), Literal('T'), Literal('e'), Literal('a'), Literal('m'), Literal('C'), Literal('i'), Literal('t'), Literal('y'), ZeroOrMore]) }
2025-03-10T17:00:21.535648Z DEBUG globset: glob converted to regex: Glob { glob: "**/_NCrunch_*", re: "(?-u)^(?:/?|.*/)_NCrunch_[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Literal('_'), Literal('N'), Literal('C'), Literal('r'), Literal('u'), Literal('n'), Literal('c'), Literal('h'), Literal('_'), ZeroOrMore]) }
2025-03-10T17:00:21.535657Z DEBUG globset: glob converted to regex: Glob { glob: "**/nCrunchTemp_*", re: "(?-u)^(?:/?|.*/)nCrunchTemp_[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Literal('n'), Literal('C'), Literal('r'), Literal('u'), Literal('n'), Literal('c'), Literal('h'), Literal('T'), Literal('e'), Literal('m'), Literal('p'), Literal('_'), ZeroOrMore]) }
2025-03-10T17:00:21.535672Z DEBUG globset: glob converted to regex: Glob { glob: "**/*.mm.*", re: "(?-u)^(?:/?|.*/)[^/]*\\.mm\\.[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, ZeroOrMore, Literal('.'), Literal('m'), Literal('m'), Literal('.'), ZeroOrMore]) }
2025-03-10T17:00:21.535678Z DEBUG globset: glob converted to regex: Glob { glob: "**/[Ee]xpress", re: "(?-u)^(?:/?|.*/)[Ee]xpress$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Class { negated: false, ranges: [('E', 'E'), ('e', 'e')] }, Literal('x'), Literal('p'), Literal('r'), Literal('e'), Literal('s'), Literal('s')]) }
2025-03-10T17:00:21.535687Z DEBUG globset: glob converted to regex: Glob { glob: "packages/*", re: "(?-u)^packages/[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([Literal('p'), Literal('a'), Literal('c'), Literal('k'), Literal('a'), Literal('g'), Literal('e'), Literal('s'), Literal('/'), ZeroOrMore]) }
2025-03-10T17:00:21.535692Z DEBUG globset: glob converted to regex: Glob { glob: "**/packages/*", re: "(?-u)^(?:/?|.*/)packages/[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Literal('p'), Literal('a'), Literal('c'), Literal('k'), Literal('a'), Literal('g'), Literal('e'), Literal('s'), Literal('/'), ZeroOrMore]) }
2025-03-10T17:00:21.535727Z DEBUG globset: glob converted to regex: Glob { glob: "**/*.[Cc]ache", re: "(?-u)^(?:/?|.*/)[^/]*\\.[Cc]ache$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, ZeroOrMore, Literal('.'), Class { negated: false, ranges: [('C', 'C'), ('c', 'c')] }, Literal('a'), Literal('c'), Literal('h'), Literal('e')]) }
2025-03-10T17:00:21.535736Z DEBUG globset: glob converted to regex: Glob { glob: "**/*.[Cc]ache", re: "(?-u)^(?:/?|.*/)[^/]*\\.[Cc]ache$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, ZeroOrMore, Literal('.'), Class { negated: false, ranges: [('C', 'C'), ('c', 'c')] }, Literal('a'), Literal('c'), Literal('h'), Literal('e')]) }
2025-03-10T17:00:21.535745Z DEBUG globset: glob converted to regex: Glob { glob: "**/~$*", re: "(?-u)^(?:/?|.*/)\\~\\$[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Literal('~'), Literal('$'), ZeroOrMore]) }
2025-03-10T17:00:21.535753Z DEBUG globset: glob converted to regex: Glob { glob: "**/*~", re: "(?-u)^(?:/?|.*/)[^/]*\\~$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, ZeroOrMore, Literal('~')]) }
2025-03-10T17:00:21.535764Z DEBUG globset: glob converted to regex: Glob { glob: "**/Backup*", re: "(?-u)^(?:/?|.*/)Backup[^/]*$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, Literal('B'), Literal('a'), Literal('c'), Literal('k'), Literal('u'), Literal('p'), ZeroOrMore]) }
2025-03-10T17:00:21.535777Z DEBUG globset: glob converted to regex: Glob { glob: "**/*.HTMLClient/GeneratedArtifacts", re: "(?-u)^(?:/?|.*/)[^/]*\\.HTMLClient/GeneratedArtifacts$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, ZeroOrMore, Literal('.'), Literal('H'), Literal('T'), Literal('M'), Literal('L'), Literal('C'), Literal('l'), Literal('i'), Literal('e'), Literal('n'), Literal('t'), Literal('/'), Literal('G'), Literal('e'), Literal('n'), Literal('e'), Literal('r'), Literal('a'), Literal('t'), Literal('e'), Literal('d'), Literal('A'), Literal('r'), Literal('t'), Literal('i'), Literal('f'), Literal('a'), Literal('c'), Literal('t'), Literal('s')]) }
2025-03-10T17:00:21.535789Z DEBUG globset: glob converted to regex: Glob { glob: "**/*.DesktopClient/GeneratedArtifacts", re: "(?-u)^(?:/?|.*/)[^/]*\\.DesktopClient/GeneratedArtifacts$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, ZeroOrMore, Literal('.'), Literal('D'), Literal('e'), Literal('s'), Literal('k'), Literal('t'), Literal('o'), Literal('p'), Literal('C'), Literal('l'), Literal('i'), Literal('e'), Literal('n'), Literal('t'), Literal('/'), Literal('G'), Literal('e'), Literal('n'), Literal('e'), Literal('r'), Literal('a'), Literal('t'), Literal('e'), Literal('d'), Literal('A'), Literal('r'), Literal('t'), Literal('i'), Literal('f'), Literal('a'), Literal('c'), Literal('t'), Literal('s')]) }
2025-03-10T17:00:21.535801Z DEBUG globset: glob converted to regex: Glob { glob: "**/*.Server/GeneratedArtifacts", re: "(?-u)^(?:/?|.*/)[^/]*\\.Server/GeneratedArtifacts$", opts: GlobOptions { case_insensitive: false, literal_separator: true, backslash_escape: true, empty_alternates: false }, tokens: Tokens([RecursivePrefix, ZeroOrMore, Literal('.'), Literal('S'), Literal('e'), Literal('r'), Literal('v'), Literal('e'), Literal('r'), Literal('/'), Literal('G'), Literal('e'), Literal('n'), Literal('e'), Literal('r'), Literal('a'), Literal('t'), Literal('e'), Literal('d'), Literal('A'), Literal('r'), Literal('t'), Literal('i'), Literal('f'), Literal('a'), Literal('c'), Literal('t'), Literal('s')]) }
2025-03-10T17:00:21.535823Z DEBUG globset: built glob set; 8 literals, 49 basenames, 52 extensions, 0 prefixes, 1 suffixes, 23 required extensions, 29 regexes
2025-03-10T17:00:21.543417Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] depth=None}: jj_lib::git_subprocess: spawning a git subprocess cmd=LC_ALL="C" "git" "--git-dir" "/home/mnj/code/service-nugets/.git" "fetch" "--prune" "--no-write-fetch-head" "--progress" "--" "origin" "+refs/heads/*:refs/remotes/origin/*"
Error: Git process failed: External git program failed:
ssh_dispatch_run_fatal: Connection to 192.168.1.109 port 7999: error in libcrypto
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[emilazy]

Could you show the output of these commands?

$ ssh -V
$ ssh -Q key-sig
$ ssh -vv [email protected] -p 7999

The ssh-rsa stuff in your SSH config has me wondering. OpenSSH has been deprecating the old SHA‐1 RSA signatures in favour of ones that use SHA‐2. It’s possible that one of the following is true:

1. the git.subprocess = false path was allowing ssh-rsa but your OpenSSH distributor has deliberately disabled support for it;
2. the git.subprocess = false path was ignoring your SSH config adding ssh-rsa to the negotiation parameters, and your Git server no longer requires it, but adding it to the negotiation is breaking things anyway.

It seems like libssh2 does still support ssh-rsa but also added support for SHA‐2 RSA signatures in 2023. It may be worth trying to remove that section of your SSH config and seeing if things work.

[emilazy]

It looks like Fedora 41 deliberately disabled support for insecure SHA‐1 signatures:

• https://fedoraproject.org/wiki/Changes/OpenSSLDistrustSHA1SigVer
• https://fedoraproject.org/wiki/SHA1SignaturesGuidance

That is why SSH to the Git server doesn’t work. If your server still requires that deprecated algorithm, it looks like the cryptography policies will need adjusting with update-crypto-policies or such, possibly by your system administrator if you do not have root. I would say this is a distributor choice to ensure you’re not connecting to servers using insecure signatures; git.subprocess = false is accidentally subverting it by not respecting the system cryptography policies. Still, it may be that your server has been upgraded and you can just drop that part of your SSH configuration.

[emilazy]

Also, it looks like your SSH configuration syntax is wrong; I believe HostKeyAlgorithms=+ssh-rsa should be HostKeyAlgorithms +ssh-rsa. Maybe that will help fix it if you still require ssh-rsa and the system cryptography policy has already been adjusted.