collectors: collect payment and attempt counts

[calvinrzachman]

Add support for real time tracking of counters for payments and attempts. These can be used to compute and create visualizations for things like average payment throughput and average # of attempts per payment.

rate(lnd_total_payments[5m])
rate(lnd_total_htlc_attempts[30m]) / rate(lnd_total_payments[30m])

• The information is obtained in real time from lnd using the TrackPayments RPC.

After making payments between two directly connected nodes:

alice-64db66cb64-g4sh2:/# curl -s http://localhost:8990/metrics | grep lnd_total
# HELP lnd_total_htlc_attempts Total number of HTLC attempts across all payments
# TYPE lnd_total_htlc_attempts counter
lnd_total_htlc_attempts 5
# HELP lnd_total_payments Total number of payments initiated
# TYPE lnd_total_payments counter
lnd_total_payments 5

[calvinrzachman]

I may be able to consolidate the # attempts with the existing HTLC collector. Checking...

[Roasbeef]

Perhaps let's expand the bucket size here?

One suggestion would be to go with powers of 2, so:

prometheus.LinearBuckets(1, 2, 10)

Which would create buckets of:

[1, 2, 4, 8, 16, 32, 64, 128, 256, 512]

[calvinrzachman]

Sounds good to me. Updated ✅ We'd be really hustling at 500 payment attempts lol.

[Roasbeef]

Awkward line wrapping, what about:

ctx, err := lnd.WithMacaroonAuthForService(
   ctx, lndclient.RouterServiceMac
)

[calvinrzachman]

Updated ✅

[Roasbeef]

Rewrap:

	stream, err := p.client.TrackPayments(
		ctx, &routerrpc.TrackPaymentsRequest{
			// NOTE: We only need to know the final result of the
			// payment and all attempts.
			NoInflightUpdates: true,
		},
)

[calvinrzachman]

Updated ✅

[Roasbeef]

We should also verify that this is using: https://lightning.engineering/api-docs/api/lnd/router/track-payment-v2/

[djkazic]

I think it's not: https://github.com/lightningnetwork/lnd/blob/4ccf4fc24c750d098cf24566ef4bbc0311c7d476/lnrpc/routerrpc/router_grpc.pb.go#L39

[djkazic]

That said, it might not be comparable as TrackPaymentV2 is for a single payment whereas TrackPayments is for payment events in general

[calvinrzachman]

I think we're good to go. Check this out: https://lightning.engineering/api-docs/api/lnd/router/track-payments/. Originally I was looking at using the lndclient library like the rest of the code base, but it doesn't yet support TrackPayments so decided to use raw routerrpc client.

[Roasbeef]

Hmm, I don't think this can quite be used as a proxy for attempts. For that we'd need to watch a payment over time, and increment this counter with each attempt.

We may also want to introspect into the payment state itself: https://lightning.engineering/api-docs/api/lnd/router/track-payment-v2/#lnrpcpaymentpaymentstatus

[Roasbeef]

Double checked, and I think I'm actually wrong about this. We get a new element here for each new attempt, as it isn't just the set of final HTLCs that were settled.

[calvinrzachman]

Prior to learning about this TrackPayments RPC, I had created a draft PR to add these metrics as real time counters in lnd directly. Implemented there, the ChannelRouter can increment counters as each attempt is registered. My hope here was that supplying the NoInflightUpdates directive to the TrackPayments call will give us only the final payment update (settle or fail) from which we can make an accurate determination of how many total attempts were made.

[Roasbeef]

We should look at the payment state and only increment this if it's IN_FLIGHT. That way we won't count a new attempt as a new payment.

[calvinrzachman]

I think we might be safe here so long as my comment about NoInflightUpdates delivering only the terminal payment update (settle/fail) is correct. That way, we'll only process the payment a single time and capture all of the attempts during its life-cycle.

I could add an explicit check that the payment status is terminal to be extra sure though.