[bug]: Unknown Unconfirmed Balance #7104

WalterSmuts · 2022-11-02T08:07:09Z

Background

I folks, am running LND as a router. I have unconfirmed balance and not really sure what's happening. Haven't made any channel opens or blockchain payments since way before this issue came up. So I think it must be related to lnd closing a channel.

Here's the relevant data:

$ lncli walletbalance | grep uncon
    "unconfirmed_balance": "1051070",
            "unconfirmed_balance": "1051070"
$ lncli listunspent --unconfirmed_only
{
	"utxos": [
		{
			"address_type": 4,
			"address": "bc1pgptzpw0jr635psy7z4wzm5e74sqlmf96wfqe0uzd9jvpwjjw0adqnek4jc",
			"amount_sat": 1051070,
			"pk_script": "5120405620b9f21ea340c09e155c2dd33eac01fda4ba724197f04d2c98174a4e7f5a",
			"outpoint": "3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97:0",
			"confirmations": 0
		}
	]
}
$ lncli pendingchannels
{
    "total_limbo_balance": "0",
    "pending_open_channels": [
    ],
    "pending_closing_channels": [
    ],
    "pending_force_closing_channels": [
    ],
    "waiting_close_channels": [
    ]
}

The relevant transaction doesn't appear on public trackers. Any idea what happened and how to fix?

Your environment

version of lnd

$ lnd --version
lnd version 0.15.2-beta commit=v0.15.2-beta

which operating system (uname -a on *Nix)

$ uname -a
Linux ip-172-31-22-39 5.18.12-1-ec2 #1 SMP PREEMPT_DYNAMIC Sat, 16 Jul 2022 21:17:41 +0000 x86_64 GNU/Linux

(Modified arch linux)

version of btcd, bitcoind, or other backend
(neutrino backend)
any other relevant environment details
Happy to grep some logs if you can give pointers. The ERR logs are a bit noisy. Here's the tail end:

Nov 02 07:22:59 [REDACTED_IP] lnd[265812]: 2022-11-02 07:22:59.278 [ERR] HSWC: Unhandled error while reforwarding htlc settle/fail over htlcswitch: UnknownNextPeer
Nov 02 07:23:04 [REDACTED_IP] lnd[265812]: 2022-11-02 07:23:04.825 [ERR] HSWC: UnknownNextPeer
Nov 02 07:23:04 [REDACTED_IP] lnd[265812]: 2022-11-02 07:23:04.825 [ERR] HSWC: Unhandled error while reforwarding htlc settle/fail over htlcswitch: UnknownNextPeer
Nov 02 07:46:13 [REDACTED_IP] lnd[265812]: 2022-11-02 07:46:13.491 [ERR] HSWC: insufficient bandwidth to route htlc
Nov 02 07:46:13 [REDACTED_IP] lnd[265812]: 2022-11-02 07:46:13.491 [ERR] HSWC: Unhandled error while reforwarding htlc settle/fail over htlcswitch: insufficient bandwidth to route htlc
Nov 02 07:46:18 [REDACTED_IP] lnd[265812]: 2022-11-02 07:46:18.812 [ERR] HSWC: insufficient bandwidth to route htlc
Nov 02 07:46:18 [REDACTED_IP] lnd[265812]: 2022-11-02 07:46:18.812 [ERR] HSWC: Unhandled error while reforwarding htlc settle/fail over htlcswitch: insufficient bandwidth to route htlc
Nov 02 07:52:31 [REDACTED_IP] lnd[265812]: 2022-11-02 07:52:31.461 [ERR] HSWC: insufficient bandwidth to route htlc
Nov 02 07:52:31 [REDACTED_IP] lnd[265812]: 2022-11-02 07:52:31.461 [ERR] HSWC: Unhandled error while reforwarding htlc settle/fail over htlcswitch: insufficient bandwidth to route htlc
Nov 02 07:58:37 [REDACTED_IP] lnd[265812]: 2022-11-02 07:58:37.418 [ERR] LTND: Unable to lookup witness: no witnesses

The text was updated successfully, but these errors were encountered:

ellemouton · 2022-11-02T08:11:34Z

Hi @WalterSmuts! Thanks for opening the issue :)

On first glance, this looks very similar to #6367, #6231, #6498, #6434

Could you perhaps grep your logs for the transaction ID? (3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97)

WalterSmuts · 2022-11-02T08:14:43Z

$ journalctl -u lnd | grep 3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97
Oct 24 10:36:20 [REDACTED_IP] lnd[265812]: 2022-10-24 10:36:20.526 [INF] SWPR: Creating sweep transaction 3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97 for 2 inputs (1223529f41409200453b3452a5db8ff4de0145ac8be222100566d3c209a2c3f4:0 (CommitmentAnchor), 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d:0 (TaprootPubKeySpend)) using 1803 sat/kw, tx_weight=725, tx_fee=0.00002501 BTC, parents_count=1, parents_fee=0.00001128 BTC, parents_weight=1288
Oct 24 10:36:26 [REDACTED_IP] lnd[265812]: 2022-10-24 10:36:26.733 [INF] LNWL: Inserting unconfirmed transaction 3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97

ellemouton · 2022-11-02T08:20:28Z

interesting. Is your channel with outpoint 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d:0 still listed as open in your listchannels output?

Do you recall trying to close this channel? Could you maybe also grep a few logs from above the ones you just posted for this channels outpoint (and perhaps for the channel ID)

WalterSmuts · 2022-11-02T08:38:43Z

interesting. Is your channel with outpoint 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d:0 still listed as open in your listchannels output?

Nope, don't think so.

lncli listchannels  | grep 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d | wc
      0       0       0

$ lncli closedchannels  | grep 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d | wc
      1       2     101

Do you recall trying to close this channel?

Definitely not personally. Could be peer-initiated or a force-close by lnd. Atleast that's what I think.

Could you maybe also grep a few logs from above the ones you just posted for this channels outpoint (and perhaps for the channel ID).

Not sure which channel you're referring to.

ellemouton · 2022-11-02T08:41:28Z

Alright - chatted to the OP offline & we found that the associated channel has been closed but the sweep tx did not broadcast correctly it seems and that is what is showing up in LND.

@WalterSmuts, you are running with a neutrino back-end correct? Would be useful to see some more logs -> my suspicion is that LND gave the tx to neutrino and then neutrino didnt broadcast it properly...

Perhaps would help to turn on debug logs & restart so that the tx is given to neutrino again for broadcast

ellemouton · 2022-11-02T08:56:38Z

The OP sent me some logs after turning on debug level. Can confirm that:

LND re-sends the tx to neutrino LNWL: Successfully rebroadcast unconfirmed transaction
Neutrino sends an INV with the tx to a bunch of peers BTCN: Sending inv
The peers to ask the OP's node for the tx BTCN: Received getdata
The OP's node sends the tx BTCN: Sending tx

However, the TX is still not appearing in the mempool.

Perhaps the peers are getting the tx and then rejecting it? I have asked the OP to send me the raw hex of the tx so I can try submit it so bitcoind myself to see what error is returned

ellemouton · 2022-11-02T09:08:38Z

Alrighty! getting closer to the issue:

Tried to submit the tx to the mempool and got:

error code: -25
error message:
bad-txns-inputs-missingorspent

The output from listchaintxns shows the following:

  "label": "0:sweep",
            "previous_outpoints": [
                {
                    "outpoint": "1223529f41409200453b3452a5db8ff4de0145ac8be222100566d3c209a2c3f4:0",
                    "is_our_output": false
                },
                {
                    "outpoint": "67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d:0",
                    "is_our_output": true
                }
            ]

The first tx is not a valid tx

ellemouton · 2022-11-02T09:19:02Z

miiiight have to do with the fact that someone else swept the anchor output which then invalidated our own sweep of the anchor: https://blockstream.info/tx/078c3357606d7d959e9e5d3b70409971880c384886511c0b5d333b6604ea521d?input:12

ellemouton · 2022-11-02T10:04:46Z

related: #6241 & #6274

WalterSmuts · 2022-11-09T15:19:35Z

Issue still persisting...

miiiight have to do with the fact that someone else swept the anchor output which then invalidated our own sweep of the anchor: https://blockstream.info/tx/078c3357606d7d959e9e5d3b70409971880c384886511c0b5d333b6604ea521d?input:12

@Roasbeef I believe #6274 is supposed to solve this issue. Any idea why it's not working?

robertclarkson · 2022-11-15T10:05:19Z

I have this issue as well

lncli walletbalance | grep uncon
    "unconfirmed_balance": "6536184",
            "unconfirmed_balance": "6536184"

{
	"utxos": [
		{
			"address_type": 4,
			"address": "bc1pg68tner0yukcra78gtrsj7remuf0vnsdd462gd7fg0u3v2rtnspsmnwsgj",
			"amount_sat": 1471233,
			"pk_script": "5120468eb9e46f272d81f7c742c7097879df12f64e0d6d74a437c943f916286b9c03",
			"outpoint": "f061053dc0dbbc04fc89dd38dd5ea794992e7cab0eb8cd9d27c06add5e5b3429:0",
			"confirmations": 0
		},
		{
			"address_type": 4,
			"address": "bc1pj3jl7jzg0drqwpqkflemp9cc582vrjkfvd2xepwz7vs2a7tzm40qn85854",
			"amount_sat": 1974652,
			"pk_script": "51209465ff48487b460704164ff3b09718a1d4c1cac963546c85c2f320aef962dd5e",
			"outpoint": "e9e76d85e2eccc466fe70565544de5c4cccbbe1da1400c0b06d4eddbe0d99bde:0",
			"confirmations": 0
		},
		{
			"address_type": 4,
			"address": "bc1pjtlzgmxj7gx7ncjhd9d945xez6g2nn5v6cfcl6df7zkm9jj6awqsnyp458",
			"amount_sat": 1974505,
			"pk_script": "512092fe246cd2f20de9e257695a5ad0d91690a9ce8cd6138fe9a9f0adb2ca5aeb81",
			"outpoint": "703a0b1dd2a7904edcf84b10b48215ae6895e7c66d21c209028836f1aef72aab:0",
			"confirmations": 0
		},
		{
			"address_type": 4,
			"address": "bc1pa3jkv94fjg653mjm8hnarzvpca4anveyddey279hjrqumju63jqqz7zf5c",
			"amount_sat": 465929,
			"pk_script": "5120ec656616a9923548ee5b3de7d18981c76bd9b3246b724578b790c1cdcb9a8c80",
			"outpoint": "1443d232358b56c87bcab6a37c2cda6ab5740d1ff50207c2ea10009536a7f648:0",
			"confirmations": 0
		},
		{
			"address_type": 4,
			"address": "bc1p83v0adgwq3uq2jhwysv6hes47ytzkv7vxqzz0m7agtqplc50m8jqmqsmzg",
			"amount_sat": 649865,
			"pk_script": "51203c58feb50e0478054aee2419abe615f1162b33cc300427efdd42c01fe28fd9e4",
			"outpoint": "261f965ce0c767a11e27f7c224777ae32bffc8879ecf31a7299a1c7ed835de5a:0",
			"confirmations": 0
		}
	]
}

Getting this error in the logs

2022-11-15 09:55:43.596 [ERR] BTCN: utxo scan failed: couldn't retrieve block 00000000000000000005c735989c24181d50ab693933ccf614c04f5ce0fc6538 from network
2022-11-15 09:55:33.594 [ERR] BTCN: utxo scan failed: couldn't retrieve block 00000000000000000005c735989c24181d50ab693933ccf614c04f5ce0fc6538 from network

robertclarkson · 2022-11-16T07:41:34Z

using the command

lncli sendcoins --sweepall --min_confs 0 --addr bc1q....xxx

seems to have collected all of the zero conf transactions into one new transaction

fingers crossed

WalterSmuts · 2022-11-16T07:44:47Z

Thanks for posting Rob, please let us know what the result is.

robertclarkson · 2022-11-16T07:52:45Z

Same problem, except now i have 1 zero confirmation transaction stuck and it doesn't seem to have been broadcast by nutrino either!

robertclarkson · 2022-11-16T07:57:33Z

well the created transaction seems to have a negative amount. that cant be right

{
            "tx_hash": "2ce2b8d0c014f133eb0cbb5c051c40e5459fabf2b5aae3516d23265d12b13c06",
            "amount": "-6308",
            "num_confirmations": 0,
            "block_hash": "",
            "block_height": 0,
            "time_stamp": "1668584398",
            "total_fees": "6308",
            "dest_addresses": [
                "bc1qte75zsmmdvn4rhppuxqxtaxqh7dlmk886lkcwz"
            ],
            "output_details": [
                {
                    "output_type": "SCRIPT_TYPE_WITNESS_V0_PUBKEY_HASH",
                    "address": "bc1qte75zsmmdvn4rhppuxqxtaxqh7dlmk886lkcwz",
                    "pk_script": "00145e7d41437b6b2751dc21e18065f4c0bf9bfdd8e7",
                    "output_index": "0",
                    "amount": "7208118",
                    "is_our_address": true
                }
            ],
            "raw_tx_hex": "0200000000010729345b5edd6ac0279dcdb80eab7c2e9994a75edd38dd89fc04bcdbc03d0561f0000000000000000000de9bd9e0dbedd4060b0c40a11dbecbccc4e54d546505e76f46ccece2856de7e9000000000000000000ab2af7aef136880209c2216dc6e79568ae1582b4104bf8dc4e90a7d21d0b3a7000000000000000000048f6a736950010eac20702f51f0d74b56ada2c7ca3b6ca7bc8568b3532d243140000000000000000005ade35d87e1c9a29a731cf9e87c8ff2be37a7724c2f7271ea167c7e05c961f26000000000000000000fdb1250ec5b1fed711cb3f2727ca29682f90e6e4811d34aef590fc91765b973e000000000000000000f2f9124b32aeae3709c8f098c74cf2da9294d2ba7af2fef22e25d0a9ee42594100000000000000000001b6fc6d00000000001600145e7d41437b6b2751dc21e18065f4c0bf9bfdd8e7014009822a3c929d74b26a3cd9721d0df093ed3d9196174c40e2cac11410c1a253fb0cc706ae3930980690dca37759e6c800155fbb12e3f6fe2245542d557eda2fd401401cc0411531836c0f99f4e6d0311238582a35b31df4819fec5e45d87f3474f1183a6e2ee54fcdf736556976fb516ad10c6213e24fac9500ed1c578ff867c3fe41014082e65b25aeb916039629ec4856a53a18031e2d8b9a2121695d339530d1cb28efa129dfb3f14b5ac6638c64182af119d4808e3f7c94d1a1e3858334df919a18390140877307ada1a76f413260cd37fbb1f3a00c0698fa9f9fe1e08235f1afc1ceb575cbcee5bd68112825d7d66514541c385eee8d0c2b5034386c0b524dc63b03061801400e556a8ef19fb8dffad81980962b5a27722678521137f4cb6771f9b642f4d89b8d3cccaa80033fc33564a119e9e2a657f2df8a7e9c75560e99841289ba4cdeb10140347dc410e448177c88e339b7de00871042f7440999586676be9b189530177be24f2a111dde9854b80438b1ea4fc6f430dd255eb99b17187eec55952fc8798e990140ec27297f201857112184a88c0644dfd3a29046e747f08dacf4f21b3015a8a29c4251f182ea96cb87ef5bfbc178243d7285b5e7ea6a475d1c4c2707e1cbc1cf4203a60b00",
            "label": "external"
        }

robertclarkson · 2022-11-16T08:05:32Z

I've tried broadcasting one of the raw transactions from my original message

02000000000102fd1d8d2fb559d6436a45d12bcdae0b17863188b0bd1a2df046b139e41149a07801000000000000000098383cb259e631f6c19b9ad79ab5d60aeea25bb8d00ef0f823165a99cf4c924f00000000000000000001091c070000000000225120ec656616a9923548ee5b3de7d18981c76bd9b3246b724578b790c1cdcb9a8c8002473044022010b57190fca53a8d89ffd15b1e8beff7302cde78f948aecc34aec50295d96ee902205d8263441ebcd5bb81d0d7a91a556111d31e908daa78de0fb0aed380e905e9c4012821024c821848f3b80302c5625d09ef5b59c1951725cab223d948584f48c70e825f96ac736460b2680140bbffbbabe064fb012f98bed23d7b96b79d728fc59b4df8bba89df08b27b3d3f113ebe932646c1ffafc03d9e545afa4b40d887847ab42488cd1170c62a90f5d10209c0b00

Error is "bad-txns-inputs-missingorspent"

WalterSmuts · 2022-11-29T08:50:33Z

Got two more stuck transactions:

$ lncli listunspent --unconfirmed_only
{
	"utxos": [
		{
			"address_type": 4,
			"address": "bc1p3g7zgu2teq5shu89jflnrzadl4869ft5pl4wl0ppyu43ry07cseqndfkjm",
			"amount_sat": 1113186,
			"pk_script": "51208a3c24714bc8290bf0e5927f318badfd4fa2a5740feaefbc21272b1191fec432",
			"outpoint": "6cb4874ec6bc654fe7f10346d0edb7fee17acbb1fb4df964c8e03d9a4ba97d2b:0",
			"confirmations": 0
		},
		{
			"address_type": 4,
			"address": "bc1ph0c0c9v3en8k9d003e2rtqlpnpqnuk020rcyep96035we5kfwsuqne96em",
			"amount_sat": 486858,
			"pk_script": "5120bbf0fc1591cccf62b5ef8e543583e198413e59ea78f04c84ba7c68ecd2c97438",
			"outpoint": "f3a7d850701d46ee23affb31d8d9831453323b9be710514e9c28ed5ad20a0d52:0",
			"confirmations": 0
		},
		{
			"address_type": 4,
			"address": "bc1pgptzpw0jr635psy7z4wzm5e74sqlmf96wfqe0uzd9jvpwjjw0adqnek4jc",
			"amount_sat": 1051070,
			"pk_script": "5120405620b9f21ea340c09e155c2dd33eac01fda4ba724197f04d2c98174a4e7f5a",
			"outpoint": "3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97:0",
			"confirmations": 0
		}
	]
}```

WalterSmuts · 2022-12-16T11:24:09Z

Added explicit peers about a month ago:

[neutrino]
neutrino.addpeer=btcd-mainnet.lightning.computer
neutrino.addpeer=mainnet1-btcd.zaphq.io
neutrino.addpeer=mainnet2-btcd.zaphq.io
neutrino.addpeer=mainnet3-btcd.zaphq.io
neutrino.addpeer=mainnet4-btcd.zaphq.io

This didn't fix anything but since then I haven't accrued any new unconfirmed tx's:

$ lncli listunspent --unconfirmed_only
{
	"utxos": [
		{
			"address_type": 4,
			"address": "bc1p3g7zgu2teq5shu89jflnrzadl4869ft5pl4wl0ppyu43ry07cseqndfkjm",
			"amount_sat": 1113186,
			"pk_script": "51208a3c24714bc8290bf0e5927f318badfd4fa2a5740feaefbc21272b1191fec432",
			"outpoint": "6cb4874ec6bc654fe7f10346d0edb7fee17acbb1fb4df964c8e03d9a4ba97d2b:0",
			"confirmations": 0
		},
		{
			"address_type": 4,
			"address": "bc1ph0c0c9v3en8k9d003e2rtqlpnpqnuk020rcyep96035we5kfwsuqne96em",
			"amount_sat": 486858,
			"pk_script": "5120bbf0fc1591cccf62b5ef8e543583e198413e59ea78f04c84ba7c68ecd2c97438",
			"outpoint": "f3a7d850701d46ee23affb31d8d9831453323b9be710514e9c28ed5ad20a0d52:0",
			"confirmations": 0
		},
		{
			"address_type": 4,
			"address": "bc1pgptzpw0jr635psy7z4wzm5e74sqlmf96wfqe0uzd9jvpwjjw0adqnek4jc",
			"amount_sat": 1051070,
			"pk_script": "5120405620b9f21ea340c09e155c2dd33eac01fda4ba724197f04d2c98174a4e7f5a",
			"outpoint": "3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97:0",
			"confirmations": 0
		}
	]
}

robertclarkson · 2022-12-18T20:20:41Z

I should add my issue was solved by running btcd in rescanning mode.

WalterSmuts · 2022-12-19T05:45:20Z

I ran lnd with --reset-wallet-transactions, waited for LNWL: Finished rescan for. Removed the second two unconfirmed tx, first one still there:

$ lncli listunspent --unconfirmed_only
{
	"utxos": [
		{
			"address_type": 4,
			"address": "bc1p3g7zgu2teq5shu89jflnrzadl4869ft5pl4wl0ppyu43ry07cseqndfkjm",
			"amount_sat": 1113186,
			"pk_script": "51208a3c24714bc8290bf0e5927f318badfd4fa2a5740feaefbc21272b1191fec432",
			"outpoint": "6cb4874ec6bc654fe7f10346d0edb7fee17acbb1fb4df964c8e03d9a4ba97d2b:0",
			"confirmations": 0
		}
	]
}

Notice new outpoint.

Horndev · 2023-02-08T16:25:33Z

I seem to be having the same issue. sweeper went nuts with a bunch of txns which never go into mempool until 100% of UTXO was showing as unconfirmed. I did a rescan, now 2 remain.

"utxos": [
                {
                        "address_type": 4,
                        "address": "bc1pfec57mkv3n90xpa3c379rds72ru9ldydkmjanz37kkg9crjxgxpqrvtzpg",
                        "amount_sat": 2076264,
                        "pk_script": "51204e714f6ecc8ccaf307b1c47c51b61e50f85fb48db6e5d98a3eb5905c0e464182",
                        "outpoint": "53085820a65121739be6ffb067aedbfff10ddf7f6883b61e4631b8dc2c5adfad:0",
                        "confirmations": 0
                },
                {
                        "address_type": 4,
                        "address": "bc1py45gpxlexcdvwgne7cmfh8zngd6llqpede02zjwmkdu6fmd8x3vqjd0p72",
                        "amount_sat": 2052362,
                        "pk_script": "51202568809bf9361ac72279f6369b9c534375ff80396e5ea149dbb379a4eda73458",
                        "outpoint": "1e345ec40d3a02ddef1112f9425599e51e72269255f3185effcc05a91261f33a:0",
                        "confirmations": 0
                }
        ]

When I query the transaction from listchaintxns I get the raw tx. When I broadcast, I get no error from lnd, but on external site I get this error:

"Code: -26, Error: insufficient fee, rejecting replacement 53085820a65121739be6ffb067aedbfff10ddf7f6883b61e4631b8dc2c5adfad, less fees than conflicting txs; 0.00000183 < 0.00000194"

Digging deeper, I find that the first input is a lightning anchor, and has been swept already. https://mempool.space/address/bc1qrh433dns65em58nrxc3jmqz80zfjphccawn74xr60feuna8m8wvsdw2j60

It seems that this address is collecting and sweeping lots of lightning anchors. https://mempool.space/address/bc1ptztt50y5xk2pngsf8kfjd5qvek7qg485ks6nj8aq9p2k25mgqrwsy8rsar

So maybe this is not detected by LND, and it's broadcasting a transaction with invalid input? I mean, these are all 330 sat utxos, but this address has collected a ton of these. What's going on?

I am running LND 0.15.4-beta

Horndev · 2023-02-08T16:26:47Z

likely related to #6241

Horndev · 2023-02-08T16:34:32Z

Ok, the node keeps trying to sweep these already spent utxos, and it's locking up my actual utxos in transactions that will never be confirmed.

M1ch43lV · 2023-03-18T12:08:04Z

Also likley related to
#7420

WalterSmuts · 2024-11-17T18:24:46Z

So I recently decided to see if it's related to me running lnd in neutrino mode. I've setup a bitcoind node and connected to it with the --reset-wallet-transactions arg. After the sync we have:

$ lncli walletbalance
{
    "total_balance":  "6389436",
    "confirmed_balance":  "6389436",
    "unconfirmed_balance":  "0",
    "locked_balance":  "0",
    "reserved_balance_anchor_chan":  "0",
    "account_balance":  {
        "default":  {
            "confirmed_balance":  "6389436",
            "unconfirmed_balance":  "0"
        }
    }
}

I should have around 14000000 so we're missing around 8000000. So looking at my closed channels I noticed there are 3 that's not claimed:

$ lncli closedchannels | jq .[].[].resolutions[].outcome | uniq -c
      3 "UNCLAIMED"
     12 "CLAIMED"

Perhaps the unclaimed channels are the ones I still have funds in? Here's the info from the unclaimed channels:

$ lncli closedchannels | jq '[.[][] | select(.resolutions[].outcome == "UNCLAIMED")]'
[
  {
    "channel_point": "a725b1c0d8d352798209f4d835c888992f0ebafef7bb13bce35e183938dc0797:1",
    "chan_id": "810590758345900033",
    "chain_hash": "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f",
    "closing_tx_hash": "110ce39c72b251eef89205f1f45d50aa57a1925dc0b4f0ca02f34fe7ee1067c5",
    "remote_pubkey": "03c436af41160a355fc1ed230a64f6a64bcbd2ae50f12171d1318f9782602be601",
    "capacity": "500000",
    "close_height": 799924,
    "settled_balance": "11342",
    "time_locked_balance": "0",
    "close_type": "REMOTE_FORCE_CLOSE",
    "open_initiator": "INITIATOR_REMOTE",
    "close_initiator": "INITIATOR_REMOTE",
    "resolutions": [
      {
        "resolution_type": "COMMIT",
        "outcome": "UNCLAIMED",
        "outpoint": {
          "txid_bytes": "c56710eee74ff302caf0b4c05d92a157aa505df4f10592f8ee51b2729ce30c11",
          "txid_str": "110ce39c72b251eef89205f1f45d50aa57a1925dc0b4f0ca02f34fe7ee1067c5",
          "output_index": 2
        },
        "amount_sat": "11342",
        "sweep_txid": "6568674c1b86fd0481075c3bf2f47801acd378ffdeca915299f8d275976ed452"
      }
    ],
    "alias_scids": [],
    "zero_conf_confirmed_scid": "0"
  },
  {
    "channel_point": "d7e9191a3d01e796be85da28cfa2962207518e10a9af79bbdf42608eeaf549c2:1",
    "chan_id": "810587459901456385",
    "chain_hash": "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f",
    "closing_tx_hash": "222b5b369258d0c5e98de02a894021fe1048fc14dbc19ff8023255fdd05b58fd",
    "remote_pubkey": "03c436af41160a355fc1ed230a64f6a64bcbd2ae50f12171d1318f9782602be601",
    "capacity": "1500000",
    "close_height": 799924,
    "settled_balance": "56484",
    "time_locked_balance": "0",
    "close_type": "REMOTE_FORCE_CLOSE",
    "open_initiator": "INITIATOR_LOCAL",
    "close_initiator": "INITIATOR_REMOTE",
    "resolutions": [
      {
        "resolution_type": "COMMIT",
        "outcome": "UNCLAIMED",
        "outpoint": {
          "txid_bytes": "fd585bd0fd553202f89fc1db14fc4810fe2140892ae08de9c5d05892365b2b22",
          "txid_str": "222b5b369258d0c5e98de02a894021fe1048fc14dbc19ff8023255fdd05b58fd",
          "output_index": 2
        },
        "amount_sat": "56484",
        "sweep_txid": "6568674c1b86fd0481075c3bf2f47801acd378ffdeca915299f8d275976ed452"
      }
    ],
    "alias_scids": [],
    "zero_conf_confirmed_scid": "0"
  },
  {
    "channel_point": "7963009b05c981f8176bf4a58bf24096d9a5ded3f7a88a0246a797756bfb9a0e:1",
    "chan_id": "830501814529359873",
    "chain_hash": "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f",
    "closing_tx_hash": "1dea1cd6ad6e44a15bf3e13ccd1939cf4f4dac9f66a3c523ded7a466fc3d1d37",
    "remote_pubkey": "0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f",
    "capacity": "5000000",
    "close_height": 811573,
    "settled_balance": "0",
    "time_locked_balance": "0",
    "close_type": "LOCAL_FORCE_CLOSE",
    "open_initiator": "INITIATOR_REMOTE",
    "close_initiator": "INITIATOR_LOCAL",
    "resolutions": [
      {
        "resolution_type": "COMMIT",
        "outcome": "UNCLAIMED",
        "outpoint": {
          "txid_bytes": "371d3dfc66a4d7de23c5a3669fac4d4fcf3919cd3ce1f35ba1446eadd61cea1d",
          "txid_str": "1dea1cd6ad6e44a15bf3e13ccd1939cf4f4dac9f66a3c523ded7a466fc3d1d37",
          "output_index": 4
        },
        "amount_sat": "3155497",
        "sweep_txid": "4125205d70d6cecbb2d934c86b5aaf75bc2a7155e3413809e2ded6b4b3a3853b"
      }
    ],
    "alias_scids": [],
    "zero_conf_confirmed_scid": "0"
  }
]

The sum of amount_sat only comes to 11342 + 56484 + 3155497 = 3223323 so definitely not all of it but it's something?

Other info:

$ lncli --version
lncli version 0.18.3-beta commit=v0.18.3-beta
$ lnd --version
lnd version 0.18.3-beta commit=v0.18.3-beta

LMK if you need any other info.

ellemouton · 2024-11-17T18:27:08Z

Thanks for the extra info @WalterSmuts 🙏 Lemme check if the sweeper-god has some insights here cc @yyforyongyu

ellemouton · 2024-11-17T18:29:13Z

also want to cc @saubyk - feels like we should keep an eye on this as it didnt resolve itself & im sure by now all the timeouts have resolved

yyforyongyu · 2024-11-17T23:39:29Z

The current state,

For channel 810590758345900033, the to_local output has been swept in this tx.
For channel 810587459901456385, the to_local output has been swept in this tx.
For channel 830501814529359873, however, the to_local output was never swept, as shown here. This channel appears to be a non-anchor channel, as indicated by one of the outputs using V0_P2WPKH.

The unexpected behaviors are,

for case 1 and 2, these swept utxos should be shown under the balance.
for case 3, the output is not swept.

This might also be related to #8001 and #8786.

To further investigate this issue,

lncli wallet listsweeps --verbose to give us the pending sweeps
lncli listunspent to give us the wallet UTXOs.
restart the node with debug flag, and without the flag --reset-wallet-transactions, and share the debug logs.

WalterSmuts · 2024-11-18T07:38:07Z

So I got the outputs of the above commands and uploaded them. I'm also running my LND node with debug logging per default, so I just did a stop start (without --reset-wallet-transactions) unlocked my wallet and zipped up my entire log directory after waiting a minute or two for the dust to settle.
lnd-logs.tar.gz
list-sweeps.txt
unspent.txt

yyforyongyu · 2024-11-18T10:39:51Z

More findings from the cli results,

For channel 810590758345900033 and 810587459901456385, the outputs were aggregated and swept in this tx, and is properly shown under listunspent as outpoint: 6568674c1b86fd0481075c3bf2f47801acd378ffdeca915299f8d275976ed452:0`.
For channel 830501814529359873, the output was swept in this tx and is properly shown under listunspent as outpoint: 4125205d70d6cecbb2d934c86b5aaf75bc2a7155e3413809e2ded6b4b3a3853b:0

So I think the closechannels has a bug that shows resolved outputs as unclaimed.

I should have around 14000000 so we're missing around 8000000.

@WalterSmuts Do you recall when was the last time you saw this balance? Meanwhile, could you do the followings,

restart the node with the flag --reset-wallet-transactions, and share the debug logs.
run lncli wallet pendingsweeps

WalterSmuts · 2024-11-18T18:22:26Z

@WalterSmuts Do you recall when was the last time you saw this balance? Meanwhile, could you do the followings,

A couple of months before this issue was created. I'd guess around June/July 2022.

Meanwhile, could you do the followings,

restart the node with the flag --reset-wallet-transactions, and share the debug logs.

The previous logs I've uploaded should have the same. Will do this again although AFAIK the sync takes a while.

run lncli wallet pendingsweeps

Right now (before restarting with --reset-wallet-transactions again) I get:

$ lncli wallet pendingsweeps
{
	"pending_sweeps": []
}

yyforyongyu · 2024-11-20T04:46:52Z

Did a quick debugging session with @Roasbeef yesterday, and think the closechannels was giving misleading info due to a previous version of the sweeper. Since the sweeping tx was published first then saved to db, it could be the case that it was published, then the node shut down, and the tx was not stored. When we came back up, the node would republish again, causing the sweeper to think the already published tx wasn't ours, hence saying it's unclaimed. This is no longer an issue now as the new sweeper will always fetch the mempool before attempting another broadcast.

As for the missing balance, we are still investigating, and would be helpful to have some debug logs🙏

WalterSmuts · 2024-11-20T07:06:53Z

Rescan finished:

$ zgrep "LNWL: Finished rescan for" .lnd/logs/bitcoin/mainnet/*
.lnd/logs/bitcoin/mainnet/lnd.log.6787.gz:2024-11-19 12:27:26.452 [INF] LNWL: Finished rescan for 301 addresses (synced to block 00000000000000000001446bb53eba6ebddb6b9784a5fd7c15b3b83e6f4c2372, height 871031)

Uploaded the entire log directory
lnd-logs.tar.gz

WalterSmuts · 2024-11-25T20:43:33Z

@yyforyongyu Any luck with the logs?

yyforyongyu · 2024-12-06T10:00:32Z

@WalterSmuts too bad nope :( But I did find a reproducible failure from our recent itest PRs, will investigate!

WalterSmuts added bug Unintended code behaviour needs triage labels Nov 2, 2022

saubyk added utxo sweeping and removed needs triage labels Nov 2, 2022

saubyk added this to the v0.17.0 milestone Nov 2, 2022

ellemouton mentioned this issue Apr 17, 2023

sweep: remove conflicted sweep txns from the rebroadcaster #7599

Merged

7 tasks

saubyk added the P3 might get fixed, nice to have label Aug 8, 2023

saubyk removed this from the Low Priority milestone Aug 8, 2023

yyforyongyu mentioned this issue Feb 5, 2025

[bug]: wallet gives inaccurate results about UTXOs #9481

Open

[bug]: Unknown Unconfirmed Balance #7104

[bug]: Unknown Unconfirmed Balance #7104

Comments

WalterSmuts commented Nov 2, 2022

Background

Your environment

ellemouton commented Nov 2, 2022

WalterSmuts commented Nov 2, 2022

ellemouton commented Nov 2, 2022

WalterSmuts commented Nov 2, 2022

ellemouton commented Nov 2, 2022

ellemouton commented Nov 2, 2022 • edited Loading

ellemouton commented Nov 2, 2022

ellemouton commented Nov 2, 2022

ellemouton commented Nov 2, 2022

WalterSmuts commented Nov 9, 2022 • edited Loading

robertclarkson commented Nov 15, 2022 • edited Loading

robertclarkson commented Nov 16, 2022

WalterSmuts commented Nov 16, 2022

robertclarkson commented Nov 16, 2022

robertclarkson commented Nov 16, 2022

robertclarkson commented Nov 16, 2022

WalterSmuts commented Nov 29, 2022

WalterSmuts commented Dec 16, 2022

robertclarkson commented Dec 18, 2022

WalterSmuts commented Dec 19, 2022 • edited Loading

Horndev commented Feb 8, 2023 • edited Loading

Horndev commented Feb 8, 2023

Horndev commented Feb 8, 2023

M1ch43lV commented Mar 18, 2023

WalterSmuts commented Nov 17, 2024

ellemouton commented Nov 17, 2024

ellemouton commented Nov 17, 2024

yyforyongyu commented Nov 17, 2024

WalterSmuts commented Nov 18, 2024

yyforyongyu commented Nov 18, 2024 • edited Loading

WalterSmuts commented Nov 18, 2024

yyforyongyu commented Nov 20, 2024

WalterSmuts commented Nov 20, 2024 • edited Loading

WalterSmuts commented Nov 25, 2024

yyforyongyu commented Dec 6, 2024

ellemouton commented Nov 2, 2022 •

edited

Loading

WalterSmuts commented Nov 9, 2022 •

edited

Loading

robertclarkson commented Nov 15, 2022 •

edited

Loading

WalterSmuts commented Dec 19, 2022 •

edited

Loading

Horndev commented Feb 8, 2023 •

edited

Loading

yyforyongyu commented Nov 18, 2024 •

edited

Loading

WalterSmuts commented Nov 20, 2024 •

edited

Loading