Backport fixes for "stop" RPC behavior #3153
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ueue 11e0151 http: Remove numThreads and ThreadCounter (Wladimir J. van der Laan) f946654 http: Remove WaitExit from WorkQueue (Wladimir J. van der Laan) b1c2370 http: Join worker threads before deleting work queue (Wladimir J. van der Laan) Pull request description: This prevents a potential race condition if control flow ends up in `ShutdownHTTPServer` before the thread gets to `queue->Run()`, deleting the work queue while workers are still going to use it. Meant to fix bitcoin#12362. Tree-SHA512: 8108514aeee5b2067a3736ed028014b580d1cbf8530ac7682b8a23070133dfa1ca21db4358c9158ea57e8811e0551395b6cb769887876b9cfce067ee968d0642
793667a Improve shutdown process (João Barbosa) Pull request description: Improve the shutdown time by not having to wait up to 2 seconds. Here is a comparison running `wallet.py` function tests before this PR: ``` 2017-08-08 03:25:20.881000 TestFramework (INFO): Initializing test directory /var/folders/1v/8_69hby54nj2k3n6fywt44x80000gn/T/testq_ramjjr 2017-08-08 03:25:23.853000 TestFramework (INFO): Mining blocks... 2017-08-08 03:25:24.132000 TestFramework (INFO): test getmemoryinfo 2017-08-08 03:25:24.559000 TestFramework (INFO): test gettxout 2017-08-08 03:25:59.858000 TestFramework (INFO): check -rescan 2017-08-08 03:26:07.735000 TestFramework (INFO): check -reindex 2017-08-08 03:26:15.751000 TestFramework (INFO): check -zapwallettxes=1 2017-08-08 03:26:24.105000 TestFramework (INFO): check -zapwallettxes=2 2017-08-08 03:26:36.694000 TestFramework (INFO): Stopping nodes 2017-08-08 03:26:43.599000 TestFramework (INFO): Cleaning up 2017-08-08 03:26:43.612000 TestFramework (INFO): Tests successful ``` After: ``` 2017-08-08 03:24:04.319000 TestFramework (INFO): Initializing test directory /var/folders/1v/8_69hby54nj2k3n6fywt44x80000gn/T/testoqeyi50_ 2017-08-08 03:24:07.035000 TestFramework (INFO): Mining blocks... 2017-08-08 03:24:07.317000 TestFramework (INFO): test getmemoryinfo 2017-08-08 03:24:07.763000 TestFramework (INFO): test gettxout 2017-08-08 03:24:25.715000 TestFramework (INFO): check -rescan 2017-08-08 03:24:27.792000 TestFramework (INFO): check -reindex 2017-08-08 03:24:29.797000 TestFramework (INFO): check -zapwallettxes=1 2017-08-08 03:24:32.207000 TestFramework (INFO): check -zapwallettxes=2 2017-08-08 03:24:36.812000 TestFramework (INFO): Stopping nodes 2017-08-08 03:24:37.915000 TestFramework (INFO): Cleaning up 2017-08-08 03:24:37.927000 TestFramework (INFO): Tests successful ``` This largely improves the time spent in Travis (under evaluation). Tree-SHA512: 023012fb3f8a380addf5995a4bf865862fed712cdd1a648d82a710e6566bc3bd34b6c49f9f06d6cc6bd81ca859da50d30d7f786c816e702549ab642e3476426f
28479f9 qa: Test bitcond shutdown (João Barbosa) 8d3f46e http: Remove timeout to exit event loop (João Barbosa) e98a9ee http: Remove unnecessary event_base_loopexit call (João Barbosa) 6b13580 http: Unlisten sockets after all workers quit (João Barbosa) 18e9685 http: Send "Connection: close" header if shutdown is requested (João Barbosa) 02e1e4e rpc: Add wait argument to stop (João Barbosa) Pull request description: Fixes bitcoin#11777. Reverts bitcoin#11006. Replaces bitcoin#13501. With this change the HTTP server will exit gracefully, meaning that all requests will finish processing and sending the response, even if this means to wait more than 2 seconds (current time allowed to exit the event loop). Another small change is that connections are accepted even when the server is stopping, but HTTP requests are rejected. This can be improved later, especially if chunked replies are implemented. Briefly, before this PR, this is the order or events when a request arrives (RPC `stop`): 1. `bufferevent_disable(..., EV_READ)` 2. `StartShutdown()` 3. `evhttp_del_accept_socket(...)` 4. `ThreadHTTP` terminates (event loop exits) because there are no active or pending events thanks to 1. and 3. 5. client doesn't get the response thanks to 4. This can be verified by applying ```diff // Event loop will exit after current HTTP requests have been handled, so // this reply will get back to the client. StartShutdown(); + MilliSleep(2000); return "Bitcoin server stopping"; } ``` and checking the log output: ``` Received a POST request for / from 127.0.0.1:62443 ThreadRPCServer method=stop user=__cookie__ Interrupting HTTP server ** Exited http event loop Interrupting HTTP RPC server Interrupting RPC tor: Thread interrupt Shutdown: In progress... torcontrol thread exit Stopping HTTP RPC server addcon thread exit opencon thread exit Unregistering HTTP handler for / (exactmatch 1) Unregistering HTTP handler for /wallet/ (exactmatch 0) Stopping RPC RPC stopped. Stopping HTTP server Waiting for HTTP worker threads to exit msghand thread exit net thread exit ... sleep 2 seconds ... Waiting for HTTP event thread to exit Stopped HTTP server ``` For this reason point 3. is moved right after all HTTP workers quit. In that moment HTTP replies are queued in the event loop which keeps spinning util all connections are closed. In order to trigger the server side close with keep alive connections (implicit in HTTP/1.1) the header `Connection: close` is sent if shutdown was requested. This can be tested by ``` bitcoind -regtest nc localhost 18443 POST / HTTP/1.1 Authorization: Basic ... Content-Type: application/json Content-Length: 44 {"jsonrpc": "2.0","method":"stop","id":123} ``` Summing up, this PR: - removes explicit event loop exit — event loop exits once there are no active or pending events - changes the moment the listening sockets are removed — explained above - sends header `Connection: close` on active requests when shutdown was requested which is relevant when it's a persistent connection (default in HTTP 1.1) — libevent is aware of this header and closes the connection gracefully - removes event loop explicit break after 2 seconds timeout Tree-SHA512: 4dac1e86abe388697c1e2dedbf31fb36a394cfafe5e64eadbf6ed01d829542785a8c3b91d1ab680d3f03f912d14fc87176428041141441d25dcb6c98a1e069d8
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is mostly meant to fix test failures where dashd shuts down too fast and misses to send the reply to the "stop" RPC, causing errors in the test framework.