multi: share the same bitcoind connection between multiple rescan clients #1566

wpaulino · 2018-07-17T00:05:27Z

In this PR, we introduce a nice optimization with regards to lnd's interaction with a bitcoind backend. Within lnd, we currently have three different subsystems responsible for watching the chain: chainntnfs, lnwallet, and routing/chainview. Each of these subsystems has an active RPC and ZMQ connection to the underlying bitcoind node. This would incur a toll on the underlying bitcoind node and would cause us to miss ZMQ events, which are crucial to lnd. We remedy this issue by sharing the same connection to a bitcoind node between the different clients within lnd.

We now also require the backing bitcoind node to use different hosts to provide its ZMQ raw block and raw transaction notifications (zmqpubrawblock and zmqpubrawtx). This was needed as the notification queue maintained by the bitcoind node would sometimes overflow with transactions and would cause block notifications to be dropped/missed.

Depends on btcsuite/btcwallet#511.
Fixes #1174.

halseth

utACK, pending the btcwallet dependency 👍

halseth · 2018-07-17T07:19:21Z

config.go

+	RPCUser      string `long:"rpcuser" description:"Username for RPC connections"`
+	RPCPass      string `long:"rpcpass" default-mask:"-" description:"Password for RPC connections"`
+	ZMQBlockHost string `long:"zmqblockhost" description:"The ZMQ host providing raw block notifications"`
+	ZMQTxHost    string `long:"zmqtxhost" description:"The ZMQ host providing raw transaction notifications"`


Does it make sense to call these zmqpubrawtx and zmqpubrawblock?

halseth · 2018-07-17T07:23:18Z

routing/chainview/interface_test.go

 	"github.com/btcsuite/btcwallet/walletdb"
 	_ "github.com/btcsuite/btcwallet/walletdb/bdb" // Required to register the boltdb walletdb implementation.
+
+	"github.com/lightninglabs/neutrino"
+	"github.com/ltcsuite/ltcd/btcjson"


guess this should be btcd, really

wpaulino · 2018-07-21T02:28:53Z

Note that this won't build until we're caught up with btcsuite master.

…etwork/lnd#1566)

Roasbeef · 2018-08-01T02:55:01Z

The dependent btcwallet PR has been merged now, so this needs a rebase! You may want to wait until after #1579 is in however.

Roasbeef · 2018-08-01T03:02:01Z

lnwallet/interface_test.go

-			if err != nil {
-				t.Fatalf("couldn't start alice client: %v", err)
-			}
+			time.Sleep(time.Second)


Why's this sleep necessary now?

To ensure bitcoind has started before we attempt to connect to it.

Has that changed with the new dependency, or is it something that could fail also earlier?

Something that could have happened earlier. It's also done within the ChainNotifier and FilteredChainView tests.

Roasbeef · 2018-08-09T03:05:20Z

Needs rebase.

Due to recent changes to the BitcoindClient interface, we now require the backing bitcoind to use different hosts for its ZMQ raw block and raw transaction notifications. This was needed as the notification queue maintained by the bitcoind node would sometimes overflow with transactions and cause block notifications to be dropped/missed. In this commit, we expand extractBitcoindRPCParams to account for this.

In this commit, we introduce a nice optimization with regards to lnd's interaction with a bitcoind backend. Within lnd, we currently have three different subsystems responsible for watching the chain: chainntnfs, lnwallet, and routing/chainview. Each of these subsystems has an active RPC and ZMQ connection to the underlying bitcoind node. This would incur a toll on the underlying bitcoind node and would cause us to miss ZMQ events, which are crucial to lnd. We remedy this issue by sharing the same connection to a bitcoind node between the different clients within lnd.

halseth · 2018-08-09T11:14:59Z

config.go

 		}
+
 		// If only one or two of the parameters are set, we assume the


nit: ...or three 😛

halseth · 2018-08-09T11:19:12Z

lnwallet/interface_test.go

-			if err != nil {
-				t.Fatalf("couldn't start alice client: %v", err)
-			}
+			time.Sleep(time.Second)


Has that changed with the new dependency, or is it something that could fail also earlier?

Roasbeef

LGTM 💧

githorray · 2018-08-10T03:33:59Z

Typo in the docs?

lnd --bitcoin.active --bitcoin.testnet --debuglevel=debug --bitcoin.node=bitcoind --bitcoind.rpcuser=REPLACEME --bitcoind.rpcpass=REPLACEME --bitcoind.zmqpubrawblock=tcp://127.0.0.1:28332 --bitcoind.zmqpubrawblock=tcp://127.0.0.1:28333 --externalip=X.X.X.X

Results in:

unable to load RPC credentials for bitcoind: please set all or none of bitcoind.rpcuser, bitcoind.rpcpass, bitcoind.zmqpubrawblock, bitcoind.zmqpubrawtx

wpaulino · 2018-08-10T03:35:29Z

Whoops, yeah the second should be --bitcoind.zmqpubrawtx=tcp://127.0.0.1:28333 .

githorray · 2018-08-10T05:43:23Z

Also, [INF] LNWL: Started listening for blocks via ZMQ on never appears in the log after the update and LND keeps losing chain sync. I am still trying to track that one down.

bretton · 2018-08-10T06:53:05Z

So now bitcoin.conf must set different ports for zmq as follows

zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28333

lnd.conf used to have

bitcoind.zmqpath=tcp://127.0.0.1:28332

however now it must have

bitcoind.zmqpubrawblock=tcp://127.0.0.1:28332
bitcoind.zmqpubrawtx=tcp://127.0.0.1:28333

???

but the sample lnd.conf file is incorrect?

https://github.com/lightningnetwork/lnd/blob/master/sample-lnd.conf

; bitcoind.zmqblockhost=tcp://127.0.0.1:28332
; bitcoind.zmqtxhost=tcp://127.0.0.1:28333

can submit PR if needed?

wpaulino · 2018-08-10T06:56:46Z

See #1717.

Roasbeef requested a review from halseth July 17, 2018 06:25

Roasbeef added this to the 0.5 milestone Jul 17, 2018

halseth reviewed Jul 17, 2018

View reviewed changes

Roasbeef mentioned this pull request Jul 21, 2018

lnd stopped receiving updates from bitcoind zmq #1597

Closed

Roasbeef mentioned this pull request Jul 30, 2018

[Bug, REST]: no graph edges exist only on REST Proxy #1637

Closed

NicolasDorier added a commit to btcpayserver/btcpayserver-docker that referenced this pull request Jul 30, 2018

Use different host for zmqpubrawblock and zmqpubrawtx (See lightningn…

cd11198

…etwork/lnd#1566)

Roasbeef reviewed Aug 1, 2018

View reviewed changes

Roasbeef added ready to merge LGTM'd, may need rebase to address conflicts, anyone can merge and removed needs review PR needs review by regular contributors labels Aug 1, 2018

wpaulino added 4 commits August 8, 2018 20:26

build: update to btcwallet with bitcoind conn changes

533a455

docs: note that different ZMQ sockets must be used with bitcoind

e58486b

halseth approved these changes Aug 9, 2018

View reviewed changes

Roasbeef approved these changes Aug 10, 2018

View reviewed changes

Roasbeef merged commit 30b156c into lightningnetwork:master Aug 10, 2018

wpaulino deleted the bitcoind-rescan-client branch August 10, 2018 00:20

ghost mentioned this pull request Aug 10, 2018

test fails: lnd --bitcoind.zmqpath is now obsolete cdecker/lightning-integration#33

Closed

halseth mentioned this pull request Aug 14, 2018

config.go: check for RPCUser AND RPCPass (AND ZMQPath for bitcoind) #631

Merged

enorrmann mentioned this pull request Feb 18, 2019

Multiple lnd instances sharing bitcoind: update docs #2655

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multi: share the same bitcoind connection between multiple rescan clients #1566

multi: share the same bitcoind connection between multiple rescan clients #1566

wpaulino commented Jul 17, 2018 •

edited

Loading

halseth left a comment

halseth Jul 17, 2018

wpaulino Jul 21, 2018

halseth Jul 17, 2018

wpaulino Jul 21, 2018

wpaulino commented Jul 21, 2018

Roasbeef commented Aug 1, 2018

Roasbeef Aug 1, 2018

wpaulino Aug 1, 2018

halseth Aug 9, 2018

wpaulino Aug 10, 2018

Roasbeef commented Aug 9, 2018

halseth Aug 9, 2018

halseth Aug 9, 2018

Roasbeef left a comment

githorray commented Aug 10, 2018

wpaulino commented Aug 10, 2018

githorray commented Aug 10, 2018

bretton commented Aug 10, 2018

wpaulino commented Aug 10, 2018

		}

		// If only one or two of the parameters are set, we assume the

multi: share the same bitcoind connection between multiple rescan clients #1566

multi: share the same bitcoind connection between multiple rescan clients #1566

Conversation

wpaulino commented Jul 17, 2018 • edited Loading

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wpaulino commented Jul 21, 2018

Roasbeef commented Aug 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef commented Aug 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef left a comment

Choose a reason for hiding this comment

githorray commented Aug 10, 2018

wpaulino commented Aug 10, 2018

githorray commented Aug 10, 2018

bretton commented Aug 10, 2018

wpaulino commented Aug 10, 2018

wpaulino commented Jul 17, 2018 •

edited

Loading