Rhizome direct operations: push, pull, sync #9

quixotique · 2012-08-27T06:59:04Z

To support Rhizome file sharing and MeshMS via infrastructure, it must be possible to configure Rhizome to connect directly to other Rhizome servers at know locations.

This issue will support a single, optional remote Rhizome instance defined by the configuration items:

rhizome.direct.address a string passed to getaddrinfo(3)
rhizome.direct.port a port number in decimal

Setting either of these without the other will log a warning and ignore it when any of the commands described below is invoked.

The following new servald commands will query servald for local bundles, then connect directly to the configured remote instance, send IHAVE announcements in batches to announce all locally stored bundles, then receive IHAVE and SENDME replies from the remote instance, then perform GET and/or POST operations to exchange bundles:

servald rhizome push will only perform POST operations to satisfy the SENDME replies from the direct remote
servald rhizome pull will only perform GET operations to fetch any bundles discovered from IHAVE announcements from the direct remote
servald rhizome sync will perform a mixture of POST and GET operations, equivalent to performing a push and a pull

These commands will terminate when all bundles identified by the initial IHAVE/SENDME exchange have been transferred. If other bundles become available during the session, these will not be detected.

The text was updated successfully, but these errors were encountered:

See issue #9. These commands are not yet implemented, so of course the new tests fail.

... where they should have been in the first place. Also, clean up the RHIZOME HTTP SERVER START log message. Issue #9.

when starting, so that same server code can be shared for rhizome transfers and rhizome direct. #9

different request parser, but otherwise share code between rhizome transfers and rhizome direct. #9

Doesn't know how to do POST yet, which is needed for all rhizome direct operations. #9

and content-type, and complaining if they aren't in the format we expect. #9

Added call to new (currently stub) function for processing bytes received from a HTTP POST multipart encoded form. #9

feels like a much simpler and cleaner design. Still more to do. #9

need for any parser look-ahead, i.e., it is now LALR(1), and well suited to parsing streaming data. #9

presence is not forgotten. #9

works. #9

partially implemented that recognises the field elements being submitted and complains if they are wrong. #9

everything except actually write manifest and data to files for importing. #9

the boundary string at the end of the form. #9

from being detected. #9

file. #9

for calling rhizome import. #9

add actual call to servald rhizome import bundle. seems to work, though yet to be tested with a genuine bundle. #9

bundles can be imported via HTTP POST multi-part form. #9

gardners · 2012-09-07T13:24:57Z

PULL syncing will use mostly GETs, but will still need to do a POST to do the initial negotiation with the remote end to work out what needs to be PULLed. But once that is known, then the existing GET-based requests can be used to implement that.

for use in Rhizome Direct synchronisation requests. #9

size of associated data in a bundle, so that we can synchronise small things first. Also preliminary work on making a general cursor-type wrapper function for get_bars() so that it is easy for any rhizome direct transport driver to iterate over the known bundles in a rhizome datastore. #9

were recently made to rhizome direct push test. Now to implement the rhizome direct http pull functionality so that they work... #9

still working on remaining issues. #9

manifest in full. #9

data we need to allow the prefix2manifest reception to launch the file body request via normal rhizome http fetch process. #9

Now actually triggers the http request to obtain the file, but the peer address is 0.0.0.0:0. Investigating. #9

But tests don't yet pass ... #9

associated file before checking if it was already in the database. Rhizome Direct can supply a manifest without associated file if the file is already in the database, and so it was breaking. Also removed "assert bundle_received_by"'s from rhizome direct pull and sync tests because they are not needed, and were failing because the same file contents was used for the files being exchanged, and so file storage was not occurring, and thus the log message being looked for was not being produced. Push, pull and sync tests in rhizomeprotocol now pass, leaving only two tests in error. #9

(something broken in merge of AB and PGS changes -- will investigate once merge complete). #9

Must be enabled by using rhizome.api.addfile.* Certainly polishing to be done, including using filename supplied during HTTP POST. Now to fix that, and make it all work with final rhizomeprotocol test case. rhizomeprotocol test cases 8 and 9 currently fail post-merge. #9

supplied in the HTTP POST form. #9

file import. It now rungs in rhizomeprotocol test suite, but fails because the BK= field is missing (but the file is imported). Also removed the assertions about receiving bundles from the push test as it was non-deterministic. The bundle reception is already tested, so there is no loss. Investigating missing BK=. Also test 8 (sync) fails post-merge, although push and pull work intepdendently. Will investigate that post-merge. #9

as hex if not supplying a manifest template. modified rhizomeprotocol test of manifest-less HTTP import to set that option. That test now passes. ALL rhizomeprotocol tests now pass. #9

direct sync enquiry request. Still something else broken post-merge, but we are getting closer. #9

an integer not text column, so that FILESIZE BETWEEN clauses behave numerically rather than lexographically. Added an index for good measure. #9

rhizome direct push/pull/sync tests from working. #9

and made corresponding adjustments. Oddly push and pull work again. Maybe there is some subtle bug in the test framework? Anyway, now to fix the sync test. #9

All rhizomeprotocol tests now PASS with merged changes. #9

buckets. Also removed some debugging. #9

The warning was that 'fd' could be used uninitialised in rhizome_store_file(), and it was true!

Also add checks for the "version", "date" and ".inserttime" columns, that they match the corresponding manifest or are at least well formed. Issue #9.

Rename Push, Pull, Sync to DirectPush, DirectPull, DirectSync. Move HttpAdd test to just after HttpImport test. Tighten up the assertions in Rhizome Direct test cases. Factor the "assert_rhizome_list" test out of assert_received() to avoid having to water down assert_rhizome_list(), which currently takes an exact list of all files that must be present in the list, and is more valuable that way.

It turns out that if the DB is locked, sqlite_prepare_v2() call can return SQLITE_BUSY. The retry logic (implemented for issue #2) only provided for sqlite_step() to return SQLITE_BUSY. It was a fairly straightforward matter to extend the retry logic to cover statement preparation in an equally general fashion. The problem was observed while diagnosing failures in the rhizomeprotocol DirectPush test case: the "servald rhizome list" command was failing due to a locked database. See issue #9.

The "rhizome direct push" command (and also sync) was not waiting for the server's HTTP response, so it was exiting before the server had finished storing the bundle, which led to a race with the subsequent "assert bundle_received_by" test. Fixed by adding the missing code to receive the HTTP response. Refactored the code used for parsing HTTP responses in rhizome_fetch.c, and used it in rhizome_direct_http.c.

quixotique · 2012-10-08T07:04:04Z

The Rhizome Direct code branch has been merged into the master branch (89343c6) and remaining issues with failing tests have been resolved (as of 8b95056).

The Rhizome Direct code still needs a lot of improvement, such as:

convert blocking procedural code to asynchronous code using struct sched_ent
selective push/pull/sync (eg, based on bundle sender, recipient, size, date, service, etc.)
take advantage of an improved BAR format (see issue Refine BAR format #19)
move push/pull into a non-blocking daemon process (see issue Rhizome server process to avoid lock errors and reduce servald latency #1)

All of this work falls under the scope of other issue reports, so closing this issue now as done.

ghost assigned gardners Aug 27, 2012

quixotique pushed a commit that referenced this issue Aug 27, 2012

Add 'rhizomeprotocol' tests for push, pull, sync

f93e32b

See issue #9. These commands are not yet implemented, so of course the new tests fail.

quixotique pushed a commit that referenced this issue Aug 27, 2012

Move functions from testdefs.sh to testdefs_rhizome.sh

41680ff

... where they should have been in the first place. Also, clean up the RHIZOME HTTP SERVER START log message. Issue #9.

quixotique pushed a commit that referenced this issue Aug 27, 2012

Issue #9, add test case for local HTTP add

b3f9e43

gardners added a commit that referenced this issue Aug 31, 2012

Initial work towards rhizome direct syncrhonisation functions #9

2b29893

gardners added a commit that referenced this issue Aug 31, 2012

Rhizome http server now allows specification of client socket callback

bde9d1c

when starting, so that same server code can be shared for rhizome transfers and rhizome direct. #9

gardners added a commit that referenced this issue Aug 31, 2012

Refactoring of Rhizome http server code so that we can supply a

9d2aa61

different request parser, but otherwise share code between rhizome transfers and rhizome direct. #9

gardners added a commit that referenced this issue Aug 31, 2012

added skeleton HTTP request parser for rhizome direct.

389ae8e

Doesn't know how to do POST yet, which is needed for all rhizome direct operations. #9

gardners added a commit that referenced this issue Aug 31, 2012

added str_str() to help parse http POST requests for rhizome direct. #9

fb82043

gardners added a commit that referenced this issue Aug 31, 2012

swapped arg order to what seemed a more sensible order. #9

0a7c633

gardners added a commit that referenced this issue Aug 31, 2012

Added initial parsing of POST requests, extracting content-length

211f75c

and content-type, and complaining if they aren't in the format we expect. #9

gardners added a commit that referenced this issue Sep 5, 2012

Added boundarystring field to rhizome_http_request structure.

ed7edd3

Added call to new (currently stub) function for processing bytes received from a HTTP POST multipart encoded form. #9

gardners added a commit that referenced this issue Sep 5, 2012

more work on multi-part POST form data parser. #9

db79190

gardners added a commit that referenced this issue Sep 5, 2012

revamped multi-part form parser to tokenise into lines. This

35ec3ba

feels like a much simpler and cleaner design. Still more to do. #9

gardners added a commit that referenced this issue Sep 5, 2012

Fixed byte munching bug in multi-part post parser. Also removed

9d7507c

need for any parser look-ahead, i.e., it is now LALR(1), and well suited to parsing streaming data. #9

gardners added a commit that referenced this issue Sep 5, 2012

fixed CRLF handling in multi-part POST parser so that CRLF

f3662c0

presence is not forgotten. #9

gardners added a commit that referenced this issue Sep 5, 2012

Make simple http response clear request type so that it always

dc15ac5

works. #9

gardners added a commit that referenced this issue Sep 5, 2012

Further work on multi-part POST parser. Simple state machine

6aca69a

partially implemented that recognises the field elements being submitted and complains if they are wrong. #9

gardners added a commit that referenced this issue Sep 5, 2012

further work on multipart POST form parser. Now does almost

3dad079

everything except actually write manifest and data to files for importing. #9

gardners added a commit that referenced this issue Sep 5, 2012

more work on parser. Now writes field files, but fails to detect

b51a7b2

the boundary string at the end of the form. #9

gardners added a commit that referenced this issue Sep 5, 2012

fixed bug with boundary line handling that prevented end of form

95f2bc2

from being detected. #9

gardners added a commit that referenced this issue Sep 5, 2012

implemented removal of extraneous CRLF from end of each form field

edf99e9

file. #9

gardners added a commit that referenced this issue Sep 5, 2012

Added multipart form end processing stage. Now has place holder

d342e7b

for calling rhizome import. #9

gardners added a commit that referenced this issue Sep 5, 2012

prevent double-processing of form when fully received.

f9a2964

add actual call to servald rhizome import bundle. seems to work, though yet to be tested with a genuine bundle. #9

gardners added a commit that referenced this issue Sep 6, 2012

Merged rhizome direct httpd back into main rhizome httpd server.

d3f5c6e

bundles can be imported via HTTP POST multi-part form. #9

gardners added a commit that referenced this issue Sep 6, 2012

added rhizome import via http post test. #9

5aa4c8b

gardners added a commit that referenced this issue Sep 7, 2012

Wrote function to get a range of BARs from the Rhizome database

89b0956

for use in Rhizome Direct synchronisation requests. #9

gardners added a commit that referenced this issue Sep 8, 2012

rhizome_direct_get_bars() now sets bid_high correctly. #9

b13c23d

gardners added a commit that referenced this issue Oct 3, 2012

applied similar fixes to rhizome direct pull and sync tests as

7c544b2

were recently made to rhizome direct push test. Now to implement the rhizome direct http pull functionality so that they work... #9

gardners added a commit that referenced this issue Oct 3, 2012

work towards making rhizome direct http pull work. #9

a81dafa

gardners added a commit that referenced this issue Oct 3, 2012

fixed some problems with the rhizome direct http pull setup code.

17de56a

still working on remaining issues. #9

gardners added a commit that referenced this issue Oct 3, 2012

first cut at code to lookup manifest by prefix and deliver the

e9136ec

manifest in full. #9

gardners added a commit that referenced this issue Oct 3, 2012

more work on rhizome direct http pull. Now passing all the right

81cc9dd

data we need to allow the prefix2manifest reception to launch the file body request via normal rhizome http fetch process. #9

gardners added a commit that referenced this issue Oct 3, 2012

fixed some more bugs in rhizome direct http pull.

25a64b5

Now actually triggers the http request to obtain the file, but the peer address is 0.0.0.0:0. Investigating. #9

gardners added a commit that referenced this issue Oct 3, 2012

Looks like rhizome direct http pull works now.

d43db33

But tests don't yet pass ... #9

gardners added a commit that referenced this issue Oct 3, 2012

test 8 (rhizome import via http post using curl) almost works

40dff25

(something broken in merge of AB and PGS changes -- will investigate once merge complete). #9

gardners added a commit that referenced this issue Oct 3, 2012

rhizome direct http manifest-less file import now uses the name

636cba8

supplied in the HTTP POST form. #9

gardners added a commit that referenced this issue Oct 3, 2012

Reduced debug output. #9

33107e7

gardners added a commit that referenced this issue Oct 3, 2012

Reintroduce fix for returning full buffer of responses to rhizome

661f369

direct sync enquiry request. Still something else broken post-merge, but we are getting closer. #9

gardners added a commit that referenced this issue Oct 3, 2012

some rhizome schema cleanups, including importantly making filesize

295f7c6

an integer not text column, so that FILESIZE BETWEEN clauses behave numerically rather than lexographically. Added an index for good measure. #9

gardners added a commit that referenced this issue Oct 3, 2012

progress on tracking down post-merge bugs that are stopping

99f8e9b

rhizome direct push/pull/sync tests from working. #9

gardners added a commit that referenced this issue Oct 3, 2012

... and same fix for sync case, and that passes now as well.

8aa18db

All rhizomeprotocol tests now PASS with merged changes. #9

gardners added a commit that referenced this issue Oct 3, 2012

fixed bug where <1KB manifests would get found in 0-1K and 1K-2K

90fd7e3

buckets. Also removed some debugging. #9

quixotique pushed a commit that referenced this issue Oct 4, 2012

Issue #9, fixed a compiler warning and bug

37d4b4f

The warning was that 'fd' could be used uninitialised in rhizome_store_file(), and it was true!

quixotique pushed a commit that referenced this issue Oct 4, 2012

Re-instate .selfsigned check in assert_rhizome_list()

f18d7ba

Also add checks for the "version", "date" and ".inserttime" columns, that they match the corresponding manifest or are at least well formed. Issue #9.

quixotique pushed a commit that referenced this issue Oct 4, 2012

Issue #9, a little bit of refactoring

9614d4f

quixotique closed this as completed Oct 8, 2012

gardners mentioned this issue Oct 8, 2012

Refine BAR format #19

Closed

quixotique mentioned this issue Nov 6, 2012

Anonymous collection of statistics #38

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rhizome direct operations: push, pull, sync #9

Rhizome direct operations: push, pull, sync #9

quixotique commented Aug 27, 2012

gardners commented Sep 7, 2012

quixotique commented Oct 8, 2012

Rhizome direct operations: push, pull, sync #9

Rhizome direct operations: push, pull, sync #9

Comments

quixotique commented Aug 27, 2012

gardners commented Sep 7, 2012

quixotique commented Oct 8, 2012