Expose amount forwarded in `PaymentForwarded` event #2136

marctyndel · 2023-03-28T16:43:37Z

This PR adds the amount forwarded to PaymentForwarded events.

We'd like this value for analytics. At the moment I've been computing it using fee_earned_msat and the outbound channel's forwarding policy, but this introduces rounding error (and I think could occasionally be incorrect if the policy was recently updated but not yet propagated).

TheBlueMatt · 2023-03-28T17:00:54Z

lightning/src/ln/channelmanager.rs

@@ -4151,6 +4151,7 @@ where
 								claim_from_onchain_tx: from_onchain,
 								prev_channel_id,
 								next_channel_id,
+								amount_forwarded_msat: forwarded_htlc_value_msat,


Ha, that was easy

TheBlueMatt · 2023-03-28T17:01:45Z

lightning/src/events/mod.rs

@@ -603,6 +603,9 @@ pub enum Event {
 		/// If this is `true`, the forwarded HTLC was claimed by our counterparty via an on-chain
 		/// transaction.
 		claim_from_onchain_tx: bool,
+		/// The final amount forwarded, in milli-satoshis, after the fee is deducted.
+		/// The caveat described above the `fee_earned_msat` field applies here as well.
+		amount_forwarded_msat: Option<u64>,


Is there a name that more clearly indicates this is the outbound amount we forwarded downstream, rather than the amount we received? I think this is fine just want to make sure we don't see anything obviously better, I don't have any ideas, though.

Something like amount_outbound_forwarded_msat?

maybe outgoing_amount_forwarded_msat? 🤷

Up to you @marctyndel.

Going with a blend of the above: outbound_amount_forwarded_msat (because HTLCIntercepted, also uses the word "outbound"...I do see "outgoing" is used elsewhere but at a glance not in anything exposed)

lightning/src/events/mod.rs

alecchendev · 2023-03-28T19:16:27Z

lightning/src/events/mod.rs

@@ -603,6 +603,9 @@ pub enum Event {
 		/// If this is `true`, the forwarded HTLC was claimed by our counterparty via an on-chain
 		/// transaction.
 		claim_from_onchain_tx: bool,
+		/// The final amount forwarded, in milli-satoshis, after the fee is deducted.
+		/// The caveat described above the `fee_earned_msat` field applies here as well.
+		amount_forwarded_msat: Option<u64>,


maybe outgoing_amount_forwarded_msat? 🤷

TheBlueMatt · 2023-03-28T19:20:21Z

Feel free to squash the fixup commit(s) down when you next push.

codecov-commenter · 2023-03-29T16:59:11Z

Codecov Report

Patch coverage: 80.00% and project coverage change: -0.04 ⚠️

Comparison is base (723c1a6) 91.40% compared to head (6b3b0c9) 91.37%.

❗ Current head 6b3b0c9 differs from pull request most recent head ee2cb8e. Consider uploading reports for the commit ee2cb8e to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2136      +/-   ##
==========================================
- Coverage   91.40%   91.37%   -0.04%     
==========================================
  Files         102      102              
  Lines       49736    49568     -168     
  Branches    49736    49568     -168     
==========================================
- Hits        45461    45291     -170     
- Misses       4275     4277       +2

Impacted Files	Coverage Δ
lightning/src/ln/functional_test_utils.rs	`92.59% <ø> (ø)`
lightning/src/events/mod.rs	`32.31% <33.33%> (+0.18%)`	⬆️
lightning/src/ln/channelmanager.rs	`89.15% <100.00%> (+0.02%)`	⬆️
lightning/src/ln/functional_tests.rs	`98.24% <100.00%> (+0.01%)`	⬆️

... and 7 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

marctyndel · 2023-03-29T17:03:13Z

lightning/src/ln/functional_test_utils.rs

@@ -1780,7 +1780,7 @@ macro_rules! expect_payment_forwarded {
 		let events = $node.node.get_and_clear_pending_events();
 		assert_eq!(events.len(), 1);
 		match events[0] {
-			Event::PaymentForwarded { fee_earned_msat, prev_channel_id, claim_from_onchain_tx, next_channel_id } => {
+			Event::PaymentForwarded { fee_earned_msat, prev_channel_id, claim_from_onchain_tx, next_channel_id, .. } => {


Question: for this test macro, if I add the new field, I get stuck trying to update the mid_update_fulfill_dance macro, because I can't see how to get the amount. I notice that fee is obtained there in a perhaps indirect way, by getting the forwarding_fee_base_msat from the channel config (meaning the test is run with 0 proportional fees), which suggests getting the amount from the HTLC might not straightforward here? Or I might be missing something.

Hmm, yea, you could do a few different things - you could add a new macro variant (ie the top level ($node,,, thing you can just add another one with a new parameter), you could indeed try to deduce the correct amount based on the fee fetched from the channel (though a few tests will fail that), or you could just not worry about it. The code here is simple enough and there are at least a few tests which exercise it, so I'm admittedly not super worried about ensuring we have Maximal Test Coverage of it.

Then if everyone's okay with it I'll go with the "just not worry about it" option

tnull

Generally looks good to me, just some minor remarks/suggestions.

lightning/src/events/mod.rs

lightning/src/ln/functional_test_utils.rs

lightning/src/events/mod.rs

tnull

LGTM

dunxen

Nice one! :)

0.0.115 - Apr 24, 2023 - "Rebroadcast the Bugfixes" API Updates =========== * The MSRV of the main LDK crates has been increased to 1.48 (lightningdevkit#2107). * Attempting to claim an un-expired payment on a channel which has closed no longer fails. The expiry time of payments is exposed via `PaymentClaimable::claim_deadline` (lightningdevkit#2148). * `payment_metadata` is now supported in `Invoice` deserialization, sending, and receiving (via a new `RecipientOnionFields` struct) (lightningdevkit#2139, lightningdevkit#2127). * `Event::PaymentFailed` now exposes a failure reason (lightningdevkit#2142). * BOLT12 messages now support stateless generation and validation (lightningdevkit#1989). * The `NetworkGraph` is now pruned of stale data after RGS processing (lightningdevkit#2161). * Max inbound HTLCs in-flight can be changed in the handshake config (lightningdevkit#2138). * `lightning-transaction-sync` feature `esplora-async-https` was added (lightningdevkit#2085). * A `ChannelPending` event is now emitted after the initial handshake (lightningdevkit#2098). * `PaymentForwarded::outbound_amount_forwarded_msat` was added (lightningdevkit#2136). * `ChannelManager::list_channels_by_counterparty` was added (lightningdevkit#2079). * `ChannelDetails::feerate_sat_per_1000_weight` was added (lightningdevkit#2094). * `Invoice::fallback_addresses` was added to fetch `bitcoin` types (lightningdevkit#2023). * The offer/refund description is now exposed in `Invoice{,Request}` (lightningdevkit#2206). Backwards Compatibility ======================= * Payments sent with the legacy `*_with_route` methods on LDK 0.0.115+ will no longer be retryable via the LDK 0.0.114- `retry_payment` method (lightningdevkit#2139). * `Event::PaymentPathFailed::retry` was removed and will always be `None` for payments initiated on 0.0.115 which fail on an earlier version (lightningdevkit#2063). * `Route`s and `PaymentParameters` with blinded path information will not be readable on prior versions of LDK. Such objects are not currently constructed by LDK, but may be when processing BOLT12 data in a coming release (lightningdevkit#2146). * Providing `ChannelMonitorUpdate`s generated by LDK 0.0.115 to a `ChannelMonitor` on 0.0.114 or before may panic (lightningdevkit#2059). Note that this is in general unsupported, and included here only for completeness. Bug Fixes ========= * Fixed a case where `process_events_async` may `poll` a `Future` which has already completed (lightningdevkit#2081). * Fixed deserialization of `u16` arrays. This bug may have previously corrupted the historical buckets in a `ProbabilisticScorer`. Users relying on the historical buckets may wish to wipe their scorer on upgrade to remove corrupt data rather than waiting on it to decay (lightningdevkit#2191). * The `process_events_async` task is now `Send` and can thus be polled on a multi-threaded runtime (lightningdevkit#2199). * Fixed a missing macro export causing `impl_writeable_tlv_based_enum{,_upgradable}` calls to not compile (lightningdevkit#2091). * Fixed compilation of `lightning-invoice` with both `no-std` and serde (lightningdevkit#2187) * Fix an issue where the `background-processor` would not wake when a `ChannelMonitorUpdate` completed asynchronously, causing delays (lightningdevkit#2090). * Fix an issue where `process_events_async` would exit immediately (lightningdevkit#2145). * `Router` calls from the `ChannelManager` now call `find_route_with_id` rather than `find_route`, as was intended and described in the API (lightningdevkit#2092). * Ensure `process_events_async` always exits if any sleep future returns true, not just if all sleep futures repeatedly return true (lightningdevkit#2145). * `channel_update` messages no longer set the disable bit unless the peer has been disconnected for some time. This should resolve cases where channels are disabled for extended periods of time (lightningdevkit#2198). * We no longer remove CLN nodes from the network graph for violating the BOLT spec in some cases after failing to pay through them (lightningdevkit#2220). * Fixed a debug assertion which may panic under heavy load (lightningdevkit#2172). * `CounterpartyForceClosed::peer_msg` is now wrapped in UntrustedString (lightningdevkit#2114) * Fixed a potential deadlock in `funding_transaction_generated` (lightningdevkit#2158). Security ======== * Transaction re-broadcasting is now substantially more aggressive, including a new regular rebroadcast feature called on a timer from the `background-processor` or from `ChainMonitor::rebroadcast_pending_claims`. This should substantially increase transaction confirmation reliability without relying on downstream `TransactionBroadcaster` implementations for rebroadcasting (lightningdevkit#2203, lightningdevkit#2205, lightningdevkit#2208). * Implemented the changes from BOLT PRs lightningdevkit#1031, lightningdevkit#1032, and lightningdevkit#1040 which resolve a privacy vulnerability which allows an intermediate node on the path to discover the final destination for a payment (lightningdevkit#2062). In total, this release features 110 files changed, 11928 insertions, 6368 deletions in 215 commits from 21 authors, in alphabetical order: * Advait * Alan Cohen * Alec Chen * Allan Douglas R. de Oliveira * Arik Sosman * Elias Rohrer * Evan Feenstra * Jeffrey Czyz * John Cantrell * Lucas Soriano del Pino * Marc Tyndel * Matt Corallo * Paul Miller * Steven * Steven Williamson * Steven Zhao * Tony Giorgio * Valentine Wallace * Wilmer Paulino * benthecarman * munjesi

marctyndel force-pushed the 2023-03-paymentforwarded-expose-amount-forwarded branch from 6823642 to 3ad0f09 Compare March 28, 2023 16:47

TheBlueMatt previously approved these changes Mar 28, 2023

View reviewed changes

TheBlueMatt reviewed Mar 28, 2023

View reviewed changes

lightning/src/events/mod.rs Outdated Show resolved Hide resolved

marctyndel dismissed TheBlueMatt’s stale review via 6b3b0c9 March 28, 2023 17:08

TheBlueMatt added the Seeking Code Review label Mar 28, 2023

alecchendev reviewed Mar 28, 2023

View reviewed changes

marctyndel force-pushed the 2023-03-paymentforwarded-expose-amount-forwarded branch from 6b3b0c9 to cf3c211 Compare March 29, 2023 16:58

marctyndel commented Mar 29, 2023

View reviewed changes

TheBlueMatt previously approved these changes Mar 29, 2023

View reviewed changes

tnull reviewed Mar 29, 2023

View reviewed changes

lightning/src/events/mod.rs Show resolved Hide resolved

lightning/src/ln/functional_test_utils.rs Outdated Show resolved Hide resolved

lightning/src/events/mod.rs Outdated Show resolved Hide resolved

add outbound_amount_forwarded_msat field to PaymentForwarded event

ee2cb8e

marctyndel dismissed TheBlueMatt’s stale review via ee2cb8e March 29, 2023 18:42

marctyndel force-pushed the 2023-03-paymentforwarded-expose-amount-forwarded branch from cf3c211 to ee2cb8e Compare March 29, 2023 18:42

TheBlueMatt removed the Seeking Code Review label Mar 29, 2023

TheBlueMatt assigned tnull and TheBlueMatt Mar 29, 2023

tnull approved these changes Mar 30, 2023

View reviewed changes

dunxen approved these changes Mar 30, 2023

View reviewed changes

TheBlueMatt merged commit 783e818 into lightningdevkit:main Mar 30, 2023

marctyndel deleted the 2023-03-paymentforwarded-expose-amount-forwarded branch March 30, 2023 20:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose amount forwarded in `PaymentForwarded` event #2136

Expose amount forwarded in `PaymentForwarded` event #2136

marctyndel commented Mar 28, 2023

TheBlueMatt Mar 28, 2023

TheBlueMatt Mar 28, 2023

marctyndel Mar 28, 2023

alecchendev Mar 28, 2023

TheBlueMatt Mar 28, 2023

marctyndel Mar 29, 2023

alecchendev Mar 28, 2023

TheBlueMatt commented Mar 28, 2023

codecov-commenter commented Mar 29, 2023 •

edited

Loading

marctyndel Mar 29, 2023 •

edited

Loading

TheBlueMatt Mar 29, 2023

marctyndel Mar 29, 2023

tnull left a comment

tnull left a comment

dunxen left a comment

Expose amount forwarded in PaymentForwarded event #2136

Expose amount forwarded in PaymentForwarded event #2136

Conversation

marctyndel commented Mar 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt commented Mar 28, 2023

codecov-commenter commented Mar 29, 2023 • edited Loading

Codecov Report

marctyndel Mar 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnull left a comment

Choose a reason for hiding this comment

tnull left a comment

Choose a reason for hiding this comment

dunxen left a comment

Choose a reason for hiding this comment

Expose amount forwarded in `PaymentForwarded` event #2136

Expose amount forwarded in `PaymentForwarded` event #2136

codecov-commenter commented Mar 29, 2023 •

edited

Loading

marctyndel Mar 29, 2023 •

edited

Loading