routing: cancelable payment loop #8734

hieblmi · 2024-05-07T12:07:34Z

While interrupting lncli estimateroutefee ends the client program, the route server is left behind with an abandoned payment loop.

To allow users to cancel payment requests properly, the payment loop in resumePayment has to be abandoned as well as soon as the user request it.

lnd/routing/payment_lifecycle.go

Line 170 in f523f52

func (p *paymentLifecycle) resumePayment() ([32]byte, *route.Route, error) {

This PR makes the SendPaymentV2 streaming context accessible from the payment loop and allows it to safely end if the context was cancelled.

routing/payment_lifecycle.go

lnrpc/routerrpc/router_server.go

ellemouton

Looking good!

Only one blocking comment around removing the context.Context nil check.

The other comment is just a suggestion

routing/payment_lifecycle.go

ellemouton

LGTM! 🚀

yyforyongyu

Don't we have timeout already? Why do we need this? I think once the payment is sent, there's no easy way to cancel it, maybe related #5461

hieblmi · 2024-05-11T09:22:00Z

That's also my understanding, we can't cancel the payment once the htlc is sent. Here, additionally to the existent timeout check that interrupts the payment loop on expiry, we pass in a context (from the rpc server client stream) that is checked for cancellation and interrupts the payment loop if the user cancelled the stream context(Ctrl+C'd the sendpayment from cli). This can only happen if a p.sendAttempt(attempt) completed.

bitromortac

Code wise this looks good 🔥! Wanted to note that this code may break usage patterns, if users expect to be able to dispatch payments in the background with lncli sendpayment or somehow if SendPaymentV2 is not waited on for responses. This would not be possible anymore with the current setting as it stops the payment loop in the next occasion if the context is cancelled. If the cmd is interrupted, this will lead to cancellation of the payment loop, as expected. Should we retain backward compatibility by adding a bool cancellable to the payment request (we could then decide whether we'd like to forward the context or to initialize a new background one)?

routing/payment_lifecycle.go

routing/payment_lifecycle_test.go

routing/router.go

routing/payment_lifecycle.go

yyforyongyu · 2024-05-15T11:48:45Z

routing/payment_lifecycle.go

@@ -319,27 +322,42 @@ lifecycle:
 }

 // checkTimeout checks whether the payment has reached its timeout.


need to update the docs too

yyforyongyu · 2024-05-15T11:50:35Z

routing/payment_lifecycle.go

+	select {
+	case <-p.timeoutChan:
+		log.Warnf("payment attempt not completed before timeout")
+		err := failPayment(channeldb.FailureReasonTimeout)


can instead do return failPayment(...)

yyforyongyu · 2024-05-15T11:51:37Z

routing/payment_lifecycle.go

 	case <-p.router.quit:
 		return fmt.Errorf("check payment timeout got: %w",
 			ErrRouterShuttingDown)

-	// Fall through if we haven't hit our time limit.
+		// Fall through if we haven't hit our time limit.


think gofmt failed here

routing/payment_lifecycle.go

yyforyongyu · 2024-05-15T11:57:45Z

routing/payment_lifecycle.go

+		}
+
+	case <-ctx.Done():
+		log.Warnf("payment attempt context canceled")


Need cap the log and add a p.identifier.

yyforyongyu · 2024-05-15T11:58:32Z

routing/payment_lifecycle_test.go

@@ -179,6 +180,34 @@ func sendPaymentAndAssertFailed(t *testing.T,
 	}
 }

+// sendPaymentAndAssertFailed calls `resumePayment` and asserts that an error


Suggested change

// sendPaymentAndAssertFailed calls `resumePayment` and asserts that an error

// sendPaymentAndAssertContextCancelled calls `resumePayment` and asserts that an error

I've called it sendPaymentAndAssertError.

yyforyongyu · 2024-05-15T11:59:45Z

routing/payment_lifecycle_test.go

@@ -179,6 +180,34 @@ func sendPaymentAndAssertFailed(t *testing.T,
 	}
 }

+// sendPaymentAndAssertFailed calls `resumePayment` and asserts that an error
+// is returned.
+func sendPaymentAndAssertContextCancelled(t *testing.T,


looks like it's the same as sendPaymentAndAssertFailed?

I've consolidated the two.

yyforyongyu · 2024-05-15T12:30:50Z

routing/payment_lifecycle.go

+			return err
+		}
+
+		return ctx.Err()


Unfortunately we cannot return an error here, as it'd cause the payment loop to exit. For the timeout case, we never exit the loop. Consider this case, say a payment is made of two HTLC attempts,

HTLC1 ---> hop1 ---> remote HTLC2 ---> hop2 ---> remote

Then remote settles the invoice, but hop2 is offline, we'd end up in this case,

HTLC1 ---> hop1 ---> remote ---> settled HTLC2 ---> hop2 ---> remote ---> pending

After 60s, we'd time out this payment, but we won't quit the loop as we still think the payment is inflight,

lnd/channeldb/payment_status.go

Line 148 in 9d358bc

// | true | true | false | true | StatusInFlight |

Then hop2 comes online and settles it, we'd consider the payment successful.

lnd/channeldb/payment_status.go

Line 156 in 9d358bc

// | false | true | false | true | StatusSucceeded |

In other words, once the HTLC is sent, there's no easy way to cancel it unless you get a clear response from your peer saying it's failed/settled. If we exit here because the context is canceled, the payment would still have the status StatusInFlight.

Thank you for the detailed explanation. I see why we can't interrupt the entire payment loop but instead behave exactly how we do it the timeout case.

hieblmi · 2024-05-17T08:14:41Z

Thanks to the reviewers so far. I think I addressed all your concerns, namely:

Add a cacelable flag to the SendPaymentRequest that preserves the current behavior of a long running payment loop even if the stream context is canceled, but cancels the payment loop if set to true.
Wrap the send payment context into a deadline context in case the a timeout is set in the SendPaymentRequest.
Remove the timeout channel in payment_lifecycle.go in favor of the new timeout context.

I tested different scenarios locally and didn't encounter an error, additionally to the added test.

Looking forward to another round of feedback.

lnrpc/routerrpc/router.proto

bitromortac · 2024-05-17T13:15:25Z

lnrpc/routerrpc/router_server.go

+
+	// Cancel the timeout context. If the context already timed out or if
+	// there was no timeout provided, this will be a no-op.
+	cancel()


this doesn't cancel the payment directly because trackPayment is blocking?

This call is to avoid a linter issue that would arise if I wrote above ctx, _ = context.WithDeadline(ctx, timeout). And if I assign the cancel I have to call it. The right place seems to be after trackPayment because then the context can be canceled.

an alternative could be to place it after sendPayment in SendPaymentAsync, but not sure that's a good pattern

could also silence the linter here with a comment as to why we don't need the context in this case.

Another opinion here would be appreciated (if the context cancelation may be done within the SendPaymentAsync goroutine as a defer statement).

If we call cancel here I think it will immediately invoke <-ctx.Done in checkContext since it's async sending. I think we can instead decide the right context to use here (stream.Context or background ctx), pass it to SendPaymentAsync -> sendPayment, and inside sendPayment we create the timeout ctx, and call the defer cancel() there.

This timeout thing may also be moved to the payment session, but that's a bigger refactor so future PRs.

If we call cancel here I think it will immediately invoke <-ctx.Done in checkContext since it's async sending.

Before cancel() is called there is the s.trackPayment call which is blocking. So I think cancel() is only called once the track payment call ends which implies that also the async payment loop ended. I think the cancel would then be correct here. We could add clarifying docs here.

Could also create the timeoutCtx in sendPayment. instead.

Before cancel() is called there is the s.trackPayment call which is blocking.

Gotya - yea to me it feels "safer" to do this in sendPayment, so any future changes in s.trackPayment won't affect the behavior of sending payments.

Okay I will change this.

bitromortac · 2024-05-17T13:27:14Z

routing/payment_lifecycle.go

+// checkContext checks whether the payment context has been canceled.
+// Cancellation occurs manually or if the context times out.
+func (p *paymentLifecycle) checkContext(ctx context.Context) error {
+	failPayment := func(reason channeldb.FailureReason) error {


I think this can be inlined again

failPayment is still present :)

routing/payment_lifecycle.go

bitromortac

tACK LGTM ⚡. Perhaps the error could be changed to a timeout error and there's an open question concerning the cancel call.

bitromortac · 2024-05-21T11:51:53Z

routing/payment_lifecycle.go

+// checkContext checks whether the payment context has been canceled.
+// Cancellation occurs manually or if the context times out.
+func (p *paymentLifecycle) checkContext(ctx context.Context) error {
+	failPayment := func(reason channeldb.FailureReason) error {


failPayment is still present :)

bitromortac · 2024-05-21T13:52:42Z

lnrpc/routerrpc/router_server.go

+
+	// Cancel the timeout context. If the context already timed out or if
+	// there was no timeout provided, this will be a no-op.
+	cancel()


Another opinion here would be appreciated (if the context cancelation may be done within the SendPaymentAsync goroutine as a defer statement).

routing/payment_lifecycle.go

yyforyongyu · 2024-05-27T13:14:29Z

lnrpc/routerrpc/router_server.go

+
+	// Cancel the timeout context. If the context already timed out or if
+	// there was no timeout provided, this will be a no-op.
+	cancel()


If we call cancel here I think it will immediately invoke <-ctx.Done in checkContext since it's async sending. I think we can instead decide the right context to use here (stream.Context or background ctx), pass it to SendPaymentAsync -> sendPayment, and inside sendPayment we create the timeout ctx, and call the defer cancel() there.

yyforyongyu · 2024-05-27T13:15:45Z

lnrpc/routerrpc/router_server.go

+
+	// Cancel the timeout context. If the context already timed out or if
+	// there was no timeout provided, this will be a no-op.
+	cancel()


This timeout thing may also be moved to the payment session, but that's a bigger refactor so future PRs.

yyforyongyu · 2024-05-27T13:16:46Z

routing/payment_lifecycle.go

-		err := p.router.cfg.Control.FailPayment(
-			p.identifier, channeldb.FailureReasonTimeout,
-		)
+		reason := channeldb.FailureReasonTimeout


yyforyongyu · 2024-05-27T13:17:19Z

routing/payment_lifecycle.go

 		if err != nil {
 			return fmt.Errorf("FailPayment got %w", err)
 		}

+		return nil


nit: don't think we need to return here

yyforyongyu · 2024-05-27T13:17:50Z

routing/payment_lifecycle.go

@@ -318,23 +312,36 @@ lifecycle:
 	return [32]byte{}, nil, *failure
 }

-// checkTimeout checks whether the payment has reached its timeout.
-func (p *paymentLifecycle) checkTimeout() error {
+// checkContext checks whether the payment context has been canceled.


or timed out...

The second sentence clarifies that.

routing/router.go

yyforyongyu

Left a few comments, and needs a rebase!

routing/router.go

yyforyongyu · 2024-06-12T10:34:04Z

docs/release-notes/release-notes-0.18.1.md

@@ -19,14 +19,23 @@

 # Bug Fixes

+* [SendPaymentV2]() now cancels the background payment loop if the user cancels


missing the PR link

yyforyongyu · 2024-06-12T10:38:31Z

routing/payment_lifecycle.go

+				"timeout, id=%s", p.identifier.String())
+		} else {
+			log.Warnf("Payment attempt context canceled, id=%s",
+				p.identifier.String())


Let's add a new failure FailureReasonCanceled to record the failure reason explicitly - think it will save us dome debugging timeout in the future. Also prefer doing it in a commit, or better a new PR since this PR is already very complete.

Good call, I addressed this in this follow-up PR: #8836. There are unit and itests failing but I think they are unrelated to this PR.

yyforyongyu · 2024-06-13T11:03:02Z

Just merged the PR that fixed the itest, could you do a rebase and let it run again, thanks!

hieblmi · 2024-06-13T16:18:47Z

There still seems to be an unrelated error left: https://github.com/lightningnetwork/lnd/actions/runs/9501738226/job/26188011967?pr=8734#step:6:468

EDIT: Seems to be a flake which is not appearing in the latest.

yyforyongyu

Yea that test is a flake, and thanks for the PR! LGTM☄️

In this commit we set up the payment loop context according to user-provided parameters. The `cancelable` parameter indicates whether the user is able to interrupt the payment loop by cancelling the server stream context. We'll additionally wrap the context in a deadline if the user provided a payment timeout. We remove the timeout channel of the payment_lifecycle.go and in favor of the deadline context.

hieblmi self-assigned this May 7, 2024

hieblmi added this to the v0.18.1 milestone May 7, 2024

hieblmi added the routing label May 7, 2024

hieblmi force-pushed the cancel-estimateroutefee branch 2 times, most recently from 48cfd5b to 1d6e008 Compare May 7, 2024 12:15

hieblmi commented May 7, 2024

View reviewed changes

routing/payment_lifecycle.go Outdated Show resolved Hide resolved

hieblmi force-pushed the cancel-estimateroutefee branch 2 times, most recently from dcdd913 to 02072fa Compare May 7, 2024 12:50

hieblmi changed the title ~~routing: Cancelable payment loop~~ routing: cancelable payment loop May 7, 2024

hieblmi added the payments Related to invoices/payments label May 7, 2024

hieblmi force-pushed the cancel-estimateroutefee branch from 02072fa to e2cfd30 Compare May 8, 2024 09:53

hieblmi commented May 8, 2024

View reviewed changes

lnrpc/routerrpc/router_server.go Show resolved Hide resolved

hieblmi force-pushed the cancel-estimateroutefee branch from e2cfd30 to 4fa701d Compare May 8, 2024 12:28

hieblmi requested a review from bitromortac May 8, 2024 15:02

ellemouton reviewed May 9, 2024

View reviewed changes

routing/payment_lifecycle.go Outdated Show resolved Hide resolved

routing/payment_lifecycle.go Outdated Show resolved Hide resolved

hieblmi force-pushed the cancel-estimateroutefee branch 2 times, most recently from 0bfb3db to 2ab2bd9 Compare May 10, 2024 09:15

ellemouton approved these changes May 10, 2024

View reviewed changes

hieblmi force-pushed the cancel-estimateroutefee branch from 2ab2bd9 to 4f18995 Compare May 10, 2024 09:51

lightningnetwork deleted a comment from coderabbitai bot May 10, 2024

yyforyongyu reviewed May 11, 2024

View reviewed changes

hieblmi mentioned this pull request May 13, 2024

loopout: configurable payment timeout for off-chain payments lightninglabs/loop#743

Merged

1 task

bitromortac reviewed May 13, 2024

View reviewed changes

bhandras reviewed May 14, 2024

View reviewed changes

routing/payment_lifecycle.go Show resolved Hide resolved

yyforyongyu reviewed May 15, 2024

View reviewed changes

hieblmi mentioned this pull request May 16, 2024

routing: log edge when skipping it #8765

Merged

hieblmi force-pushed the cancel-estimateroutefee branch from 4f18995 to 215b4c9 Compare May 16, 2024 14:44

lightningnetwork deleted a comment from coderabbitai bot May 16, 2024

hieblmi force-pushed the cancel-estimateroutefee branch from 215b4c9 to 9a3fa63 Compare May 16, 2024 15:25

bitromortac reviewed May 17, 2024

View reviewed changes

hieblmi force-pushed the cancel-estimateroutefee branch from 83eb67a to 09592bf Compare May 17, 2024 15:38

hieblmi requested a review from bitromortac May 17, 2024 15:39

bitromortac approved these changes May 21, 2024

View reviewed changes

hieblmi force-pushed the cancel-estimateroutefee branch 2 times, most recently from 8acd646 to 6bcac70 Compare May 23, 2024 07:41

yyforyongyu reviewed May 27, 2024

View reviewed changes

hieblmi force-pushed the cancel-estimateroutefee branch from 6bcac70 to e63375a Compare May 28, 2024 09:04

hieblmi requested a review from yyforyongyu May 28, 2024 10:42

hieblmi force-pushed the cancel-estimateroutefee branch 2 times, most recently from 13a9343 to 21b64f5 Compare June 10, 2024 09:16

hieblmi requested review from yyforyongyu and removed request for yyforyongyu June 10, 2024 09:18

yyforyongyu reviewed Jun 12, 2024

View reviewed changes

hieblmi force-pushed the cancel-estimateroutefee branch from 21b64f5 to 1e6e323 Compare June 13, 2024 07:28

hieblmi mentioned this pull request Jun 13, 2024

routing: add payment failure reason FailureReasonCancel #8836

Merged

hieblmi requested a review from yyforyongyu June 13, 2024 08:39

hieblmi force-pushed the cancel-estimateroutefee branch from 1e6e323 to da0b724 Compare June 13, 2024 14:41

yyforyongyu approved these changes Jun 14, 2024

View reviewed changes

hieblmi force-pushed the cancel-estimateroutefee branch from da0b724 to 4568dfc Compare June 17, 2024 07:32

hieblmi added 5 commits June 17, 2024 09:32

routing: fix typos and wrap errors

06bff6f

lnrpc: cancelable sendpayment request

e729084

lncli: cancelable flag for SendPaymentRequest

7bfa616

docs: update release notes

4568dfc

yyforyongyu merged commit e6f7a2d into lightningnetwork:master Jun 17, 2024
31 of 34 checks passed

hieblmi deleted the cancel-estimateroutefee branch June 17, 2024 10:54

		@@ -319,27 +322,42 @@ lifecycle:
		}

		// checkTimeout checks whether the payment has reached its timeout.

	// sendPaymentAndAssertFailed calls `resumePayment` and asserts that an error
	// sendPaymentAndAssertContextCancelled calls `resumePayment` and asserts that an error

		@@ -19,14 +19,23 @@

		# Bug Fixes

		* [SendPaymentV2]() now cancels the background payment loop if the user cancels

routing: cancelable payment loop #8734

routing: cancelable payment loop #8734

Conversation

hieblmi commented May 7, 2024 • edited Loading

ellemouton left a comment

Choose a reason for hiding this comment

ellemouton left a comment

Choose a reason for hiding this comment

yyforyongyu left a comment

Choose a reason for hiding this comment

hieblmi commented May 11, 2024

bitromortac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hieblmi commented May 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitromortac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yyforyongyu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hieblmi Jun 13, 2024 • edited Loading

Choose a reason for hiding this comment

yyforyongyu commented Jun 13, 2024

hieblmi commented Jun 13, 2024 • edited Loading

yyforyongyu left a comment

Choose a reason for hiding this comment

hieblmi commented May 7, 2024 •

edited

Loading

hieblmi commented May 17, 2024 •

edited

Loading

hieblmi Jun 13, 2024 •

edited

Loading

hieblmi commented Jun 13, 2024 •

edited

Loading