[bug]: getting FAILURE_REASON_INSUFFICIENT_BALANCE randomly when I do have sufficient balance

[ZZiigguurraatt]

I'm randomly getting FAILURE_REASON_INSUFFICIENT_BALANCE when trying to send taproot assets over a channel when I do in fact have sufficient balance. As it is random, it is not repeatable. I'm still trying to debug, but wanted to open an issue to start tracking the debugging process. I am getting this issue when using a price oracle server (but with fixed rates) instead of the mock oracle.

[Roasbeef]

In your set up, do the nodes have > 1 channel, with some being asset chans, and some being normal BTC chans?

is the sending serial, or do you have many clients/goroutines/threads all sending at the same time?

Do you wait for the payment to succeed before trying the next, or is it all async?

[ZZiigguurraatt]

In your set up, do the nodes have > 1 channel, with some being asset chans, and some being normal BTC chans?

Some nodes do and some don't. Still trying to figure out if it is related to that or not.

is the sending serial, or do you have many clients/goroutines/threads all sending at the same time?

Do you wait for the payment to succeed before trying the next, or is it all async?

I am not doing anything async right now. There should only be a single HTLC in flight at a time. I'm assuming SendPayment waits to return the gRPC until the HTLC is settled though.

[ZZiigguurraatt]

I may have had some inconsistencies in my previous test runs as well as some ill posed test cases not accounting properly for the spread between the buy and sell rates. I'm still investigating, but this may just be a case of user error. I will report back as soon as I have more clarity.

[ZZiigguurraatt]

Seems to be related: lightningnetwork/lnd#9467

[ZZiigguurraatt]

In lightningnetwork/lnd#9467 you can see I was trying to figure out how to wait for channel balances for the sender and receiver to update before proceeding with testing, but what RPC call would I use in a multi-hop payment to also check if the intermediate nodes are ready and updated as well (assuming those RPC calls also aren't impacted by lightningnetwork/lnd#9467)?

[ZZiigguurraatt]

This issue was a byproduct of a poorly posed test case that was disguised by some other bugs that have been reported that made it difficult to detect.