routing: fix route fee calculation and channel capacity check #1382

joostjager · 2018-06-13T09:32:50Z

This PR fixes two problems in the routing package:

Fee calculation for a hop does not include the fee that needs to be paid to the next hop
The incoming channel capacity "sanity" check does not include the fee to be paid to the current hop

It also updates the multi hop payment integration test to cover the fee miscalculation.

joostjager · 2018-06-13T09:46:53Z

routing/pathfind_test.go

+			createHop(10000, 1000),
+		},
+		expectedTotalAmount: 102010,
+	} }


This test case fails on master. It returns a total amount of 102000

Can you add to the commit message why this fails on master, and what you did to fix it?

joostjager · 2018-06-13T09:47:30Z

routing/pathfind_test.go

+			createHop(1000, 100), 
+			createHop(1000, 1000),
+		},
+		expectFailure: true,


This test case fails on master. It does not generate a failure even though the first channel has insufficient capacity.

Can you add to the commit message why this fails on master, and what you did to fix it?

halseth

Another very welcome bugfix! 🙌

halseth · 2018-06-13T12:06:08Z

routing/pathfind.go

 		fee := lnwire.MilliSatoshi(0)

 		// If this isn't the last hop, to add enough funds to pay for
 		// transit over the next link.
 		if i != len(pathEdges)-1 {
 			// We'll grab the edge policy and per-hop payload of
 			// the prior hop so we can calculate fees properly.
-			prevEdge := pathEdges[i+1]
 			prevHop := route.Hops[i+1]


should really be named nextHop?

Yes. Renamed.

halseth · 2018-06-13T12:18:44Z

routing/pathfind_test.go

+			createHop(1000, 100), 
+			createHop(1000, 1000),
+		},
+		expectFailure: true,


Maybe we should make sure that the error we get is ErrInsufficientCapacity? :)

Good addition. Also because in another PR I will check for a different error code. I chose to use two fields: expectFailure and expectedErrorCode. Any suggestion to do this more go'ish is welcome.

halseth · 2018-06-13T12:19:40Z

routing/pathfind.go

@@ -395,7 +384,7 @@ func newRoute(amtToSend, feeLimit lnwire.MilliSatoshi, sourceVertex Vertex,

 	// The total amount required for this route will be the value the
 	// source extends to the first hop in the route.
-	route.TotalAmount = runningAmt
+	route.TotalAmount = route.Hops[0].AmtToForward + route.Hops[0].Fee


Comment should be updated to explain the calculation.

Nice. I would also add a note that the source node (the sender of the payment) is not part of the hops here.

halseth · 2018-06-13T12:20:25Z

routing/pathfind_test.go

+			createHop(1000, 100), 
+			createHop(1000, 1000),
+		},
+		expectFailure: true,


Can you add to the commit message why this fails on master, and what you did to fix it?

halseth · 2018-06-13T12:20:42Z

routing/pathfind_test.go

+			createHop(10000, 1000),
+		},
+		expectedTotalAmount: 102010,
+	} }


Can you add to the commit message why this fails on master, and what you did to fix it?

joostjager · 2018-06-13T13:44:36Z

@halseth Changes made.

Roasbeef · 2018-06-13T22:49:46Z

routing/pathfind.go

+		// enough capacity to carry the required amount which
+		// includes the fee dictated at each hop. Make the comparison
+		// in msat to prevent rounding errors.
+		if nextHop.AmtToForward + fee > lnwire.NewMSatFromSatoshis(


This is actually the outgoing channel though. Also at this point w/ the old code, the amtToForward already factors in the fee added at the prior hop. The fee is the difference between the incoming and outgoing amount to a target node.

I thiiink this is correct now.

Roasbeef · 2018-06-13T22:52:08Z

routing/pathfind.go

-			//
-			//     * amt_in - fee >= amt_to_forward
-			amtToForward = runningAmt - fee
+			nextHop := route.Hops[i+1]


This is actually the previous hop. We're traversing backwards right now along the route.

It gets confusing with different definitions of prev and next. Is it next loop-wise or next in the payment route. It would be good to adhere to one definition. Discussed with @halseth and opted for "next = next in route".

Maybe I should rename it to that. nextHopInRoute.

Ah, yeah I was thinking "next = next in route". It is definitely confusing using prev when we are traversing backwards. Looking at the comments, lastHop is also being used, so I think sticking to "first = first in route, last = last in route, next = next in route" makes sense. Comments will need to be updated to reflect this though.

Roasbeef · 2018-06-13T22:52:38Z

routing/pathfind.go

@@ -75,7 +75,7 @@ type ChannelHop struct {
 // included within the Sphinx packet.
 type Hop struct {
 	// Channel is the active payment channel edge that this hop will travel
-	// along.
+	// along. This is the _incoming_ channel to this hop.


This is the outgoing channel. The HTLC will flow on this path.

As I understood it, a hop is an intermediate or final node (from rfc).

The result of findPath is a list of channels that the htlc should flow along. First element in this list contains the first channel. From source to first intermediate hop. So if the first element in the list is the first hop, than this Channel property must be the incoming channel to the first hop?

As I understood it Hop is analogous to an "edge" here, so saying it is incoming/outgoing doesn't make that much sense. However, the ChannelHop contains a ChannelEdgePolicy that houses the NodeID of the node this edge leads to. So something about that can be added to the comment.

Yes, it is in the language. According to the lightning rfc, a hop is a node. The Hop type is really what it should be. But ChannelHop might be better renamed to something else. ChannelEdge then probably. Not too far off, it's a policy, a target and a bandwidth estimate.

Roasbeef · 2018-06-13T22:53:37Z

routing/pathfind.go

+			// The amount that this hop needs to forward is based
+			// on how much the prior hop carried plus the fee
+			// that needs to be paid to the prior hop
+			amtToForward = nextHop.AmtToForward + nextHop.Fee


The fee has already been accumulated into what the next hop pays. The fee is the difference between hop_{i+1}.amount and hop_{i}.amount.

I think what is confusing here is how the Hop.Fee relates to the Hop.AmountToForward. From the godoc on those fields, it looks like the end node of this Hop (=edge) will receive AmountToFoward, meaning the start node subtracted Fee. But here we set the "last hop" fee to zero, which doesn't work with that definition. So I think we should make the documentation match the usage here (by either modifying docs or use).

A hop is a node according to lightning-rfc. I think we should stick to that definition in the code?

I think it makes sense to name hop=node with the current comments :)

halseth · 2018-06-14T12:48:10Z

routing/pathfind.go

-			// following inequality most hold true:
-			//
-			//     * amt_in - fee >= amt_to_forward
-			amtToForward = runningAmt - fee


This is where the current implementation is incorrect in my understanding. The second to last hop will here always have amtToForward equal to the last hop.

If this is correct, how did this ever work? 🤔

That's correct, it should forward this exact amount to the last hop. Note that this is was goes into the hop payload for the node.

halseth · 2018-06-14T13:07:45Z

routing/pathfind_test.go

+			createHop(10, 1000), 
+			createHop(10, 1000),
+		},
+		expectedTotalAmount: 100002,


maybe add a per hop expected fees as well. Will also make it easier to see how the total is calculated :)

Indeed, maybe better to assert the full result. But I'd rather postpone that until #1321 is merged. Then I can properly separate the newRoute impl+test from the findPath impl+test. Different responsibilties, separate files.

I don't see why we would have to postpone that. Looks like that could be added to these testcases without any dependency on #1321.

joostjager · 2018-06-16T14:22:58Z

Integration test added. It fails on master with "unable to route payment to destination: FeeInsufficient(fee=1101000 mSAT, update=(lnwire.ChannelUpdate)"

halseth

This PR is starting to get into a rally good shape! The code makes more sense now, and it is IMO much easier to follow after your recent comment updates 👍

I also emailed Travis to unflag your account such that we can get the tests running.

halseth · 2018-06-18T13:43:48Z

lnd_test.go

+
+	updateFeeReq := &lnrpc.PolicyUpdateRequest{
+		BaseFeeMsat:   baseFee,
+		FeeRate:       float64(feeRate) / feeBase,


Is this a global feeBase?

That was the idea. But I see that I did not reference the global one defined in rpcserver everywhere. And on a second thought, it does not look like such a good idea. Someone could accidentally change feeBase in rpcserver, breaking the external interface, but not failing the test. Replaced all occurences in lnd_test with a different value testFeeBase.

halseth · 2018-06-18T13:50:50Z

lnd_test.go

+		chanPoint,
+	)
+
+	time.Sleep(time.Millisecond * 50)


not needed?

No, removed.

halseth · 2018-06-18T13:53:59Z

lnd_test.go

@@ -2953,19 +3005,26 @@ func testMultiHopPayments(net *lntest.NetworkHarness, t *harnessTest) {
 	// increasing of time is needed to embed the HTLC in commitment
 	// transaction, in channel Carol->David->Alice->Bob, order is Bob,
 	// Alice, David, Carol.
-	const amountPaid = int64(5000)
+	amountPaid := int64(5000)
+	const expectedFeeAlice = 505


These calculations should be explained.

Comments added

halseth · 2018-06-18T13:54:15Z

lnd_test.go

 	assertAmountPaid(t, ctxb, "Alice(local) => Bob(remote)", net.Bob,
 		aliceFundPoint, int64(0), amountPaid)
 	assertAmountPaid(t, ctxb, "Alice(local) => Bob(remote)", net.Alice,
 		aliceFundPoint, amountPaid, int64(0))
+
+	amountPaid += expectedFeeAlice


Same, should be explained.

Comments added and variables clarified.

halseth · 2018-06-18T13:55:14Z

lnd_test.go

-	const exectedFees = 5
-	if feeReport.DayFeeSum != exectedFees {
+
+	if feeReport.DayFeeSum != uint64(expectedFeeDave) {


Comments above should be updated.

Fee amount updated

halseth · 2018-06-18T14:21:49Z

routing/pathfind_test.go

+
+		return &ChannelHop {
+			ChannelEdgePolicy: &channeldb.ChannelEdgePolicy {
+				Node: &channeldb.LightningNode{},


I think we should set the baseFee here for clarity. Either as a parameter, or just BaseFee: 0. I think it makes sense as a parameter, as that lets us also test hops with non-zero base fee.

halseth · 2018-06-18T14:23:20Z

routing/pathfind_test.go

+		// A three hop payment where the first and second hop
+		// will both charge 1 msat. The fee for the first hop
+		// is actually slightly higher than 1, because the amount 
+		// t0 fwd also includes the fee for the second hop. This


"to forward"

halseth · 2018-06-18T14:24:18Z

routing/pathfind_test.go

+		// will both charge 1 msat. The fee for the first hop
+		// is actually slightly higher than 1, because the amount 
+		// t0 fwd also includes the fee for the second hop. This
+		// gets rounded down to 1.


Nice test case! Also add a similar test with the feerate exactly high enough to not get rounded down?

halseth · 2018-06-18T14:25:58Z

routing/pathfind_test.go

+			createHop(10, 1000), 
+			createHop(10, 1000),
+		},
+		expectedTotalAmount: 100002,


I don't see why we would have to postpone that. Looks like that could be added to these testcases without any dependency on #1321.

halseth · 2018-06-18T14:27:07Z

routing/pathfind_test.go

+				expectedCode := testCase.expectedErrorCode
+				if err == nil || !IsError(err, expectedCode) {
+					t.Fatalf("expected newRoute to fail " + 
+						 "with error code %v", 


add "instead got error %v"

joostjager · 2018-06-19T09:46:45Z

@halseth Your comments have been addressed.

Roasbeef

Excellent work! One worthy follow up I may add as an issue is to take advantage of the testing.Quick package to allow us to generate a set of randomized "well formed" routes and assert certain properties against them based on path finding attempts. If we had this type of testing in place, we may have caught this bug much earlier on.

I left one minor comment w.r.t adding a comment within some fields on the newly added testing fixture just to elaborate on the meaning of a particular attribute. Other than that, LGTM ⚡️

Once the fixup commits have been rebased in, we can get this one merged, then proceed with the other related PR's.

Roasbeef · 2018-06-28T02:16:29Z

routing/pathfind_test.go

+		hops 	    	      []*ChannelHop
+		paymentAmount 	      lnwire.MilliSatoshi
+		expectedFees	      []lnwire.MilliSatoshi
+		expectedTimeLocks     []uint32


What exactly is the expected time lock here? I'm having a hard time following this usage, but everything else is pristine. It may help to add a set of comments to all the fields in this struct.

Comments added

This change makes the test more sensitive to bugs than a route with nodes that all enforce the same fee policy. In addition to that, the fee has been increased to a level at which potential problems with improper fee calculation become detectable (and not disappear in rounding).

This commit fixes the logic inside the newRoute function to address the following problems: - Fee calculation for a hop does not include the fee that needs to be paid to the next hop. - The incoming channel capacity "sanity" check does not include the fee to be paid to the current hop.

This comment extends the unit tests for NewRoute with checks on the total time lock for a route as well as the expected time lock values for every hop along the route.

Roasbeef

LGTM 🚀

joostjager force-pushed the newroute-bug branch from 79ef30a to aa0b439 Compare June 13, 2018 09:45

joostjager commented Jun 13, 2018

View reviewed changes

halseth suggested changes Jun 13, 2018

View reviewed changes

joostjager force-pushed the newroute-bug branch from aa0b439 to 36cf58d Compare June 13, 2018 13:40

Roasbeef reviewed Jun 13, 2018

View reviewed changes

joostjager force-pushed the newroute-bug branch from 36cf58d to aa7d37d Compare June 14, 2018 07:41

halseth reviewed Jun 14, 2018

View reviewed changes

meshcollider added routing channels fees Related to the fees paid for transactions (both LN and funding/commitment transactions) labels Jun 15, 2018

joostjager mentioned this pull request Jun 15, 2018

FeeInsufficient error paying invoice to routed (not directly connected) c-lightning node #922

Closed

joostjager force-pushed the newroute-bug branch 3 times, most recently from 813689f to 13366b2 Compare June 16, 2018 14:17

joostjager force-pushed the newroute-bug branch 3 times, most recently from 1e3d558 to 97bdf61 Compare June 16, 2018 19:33

halseth suggested changes Jun 18, 2018

View reviewed changes

joostjager force-pushed the newroute-bug branch from 4574fbd to 9059219 Compare June 19, 2018 09:40

joostjager force-pushed the newroute-bug branch from 9059219 to fa388d5 Compare June 27, 2018 08:46

Roasbeef previously approved these changes Jun 28, 2018

View reviewed changes

Roasbeef added this to the 0.5 milestone Jun 28, 2018

joostjager added 3 commits June 28, 2018 13:53

routing: add time lock asserts to NewRoute test

416f368

This comment extends the unit tests for NewRoute with checks on the total time lock for a route as well as the expected time lock values for every hop along the route.

joostjager dismissed Roasbeef’s stale review via 416f368 June 28, 2018 12:05

joostjager force-pushed the newroute-bug branch from cf484a6 to 416f368 Compare June 28, 2018 12:05

Roasbeef approved these changes Jun 29, 2018

View reviewed changes

Roasbeef merged commit 64a0734 into lightningnetwork:master Jun 29, 2018

halseth mentioned this pull request Jul 12, 2018

FeeInsufficient #1549

Closed

joostjager deleted the newroute-bug branch December 4, 2018 11:49

routing: fix route fee calculation and channel capacity check #1382

routing: fix route fee calculation and channel capacity check #1382

Conversation

joostjager commented Jun 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joostjager Jun 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joostjager commented Jun 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halseth Jun 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joostjager commented Jun 16, 2018

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joostjager commented Jun 19, 2018

Roasbeef left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef left a comment

Choose a reason for hiding this comment

joostjager commented Jun 13, 2018 •

edited

Loading

joostjager Jun 13, 2018 •

edited

Loading

halseth Jun 14, 2018 •

edited

Loading