Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft: base protocol with merkle trees and new hash algorithms #59

Merged
merged 35 commits into from
May 15, 2017
Merged

Draft: base protocol with merkle trees and new hash algorithms #59

Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
39670fb
First Draft: base protocol with merkle trees and new hash algorithms
the8472 Feb 26, 2017
6c7b87a
dictionary file trees, zero-paddeding when hashing
the8472 Feb 27, 2017
a56c940
reject message, replace digest function field with version indicator
the8472 Feb 28, 2017
e25fa15
add metadata exchange for piece layers
the8472 Mar 1, 2017
8628126
describe legacy/v2 hybrid format
the8472 Mar 1, 2017
57f4f26
omit hashes which can be reconstructed
the8472 Mar 1, 2017
f7ef8ee
v2 and hybrid magnets
the8472 Mar 2, 2017
e7d2f7c
choose hash algo, change piece layers to a dictionary
the8472 Mar 2, 2017
5790658
terrible typo
the8472 Mar 2, 2017
33077e2
add hash request messages to the core protocol
Mar 23, 2017
131b288
updates based on review feedback
Mar 24, 2017
cd1574a
Merge pull request #2 from bittorrent/hash-transfer
the8472 Mar 24, 2017
cfd7cd5
fix missing markup
the8472 Mar 26, 2017
48dfdcb
changes from reviews
the8472 Mar 26, 2017
b0634bc
Document client version field in DHT messages
Apr 28, 2017
369b2bc
add note about possible absence of the version key
May 1, 2017
6034458
Merge pull request #61 from bittorrent/dht-version-string
ssiloti May 2, 2017
940d1c6
regenerate html
May 2, 2017
a160be6
byte[] vs. String vs. path component clarifications
the8472 May 8, 2017
42501e5
minor fixes
the8472 May 9, 2017
5afecdc
relax path name restrictions
the8472 May 9, 2017
4c3f888
typo
the8472 May 9, 2017
4bdc9f6
add v2 torrent creation script
May 12, 2017
6c2e56d
go back to using the file length to filter for 'piece layers'
ssiloti May 13, 2017
655e385
include padding for the last file of a multi-file torrent
ssiloti May 13, 2017
96de012
normalize path before setting name
ssiloti May 13, 2017
055446e
set pad attribute on pad files
ssiloti May 13, 2017
7baf1c9
walk the directory tree in lexicographic order
ssiloti May 13, 2017
96aa178
Merge pull request #3 from bittorrent/new-hash-algos
the8472 May 14, 2017
47ed76d
add infohash output
the8472 May 14, 2017
e14a7a6
link torrent creator script
the8472 May 14, 2017
6a0b817
move v2 spec to new bep
the8472 May 14, 2017
bd77a55
restore bep 0003
the8472 May 14, 2017
9a34522
change status to draft
the8472 May 14, 2017
9049d0c
from review: simplify hex output
the8472 May 15, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 31 additions & 14 deletions beps/bep_0003.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.12: http://docutils.sourceforge.net/" />
<meta name="generator" content="Docutils 0.13.1: http://docutils.sourceforge.net/" />
<title></title>
<meta name="author" content="Bram Cohen &lt;bram&#64;bittorrent.com&gt;" />
<link rel="stylesheet" href="../css/bep.css" type="text/css" />
Expand Down Expand Up @@ -33,26 +33,27 @@ <h1><a href="../index.html">BitTorrent<span>.org</span></a></h1>
<col class="docinfo-name" />
<col class="docinfo-content" />
<tbody valign="top">
<tr class="field"><th class="docinfo-name">BEP:</th><td class="field-body">3</td>
<tr class="bep field"><th class="docinfo-name">BEP:</th><td class="field-body">3</td>
</tr>
<tr class="field"><th class="docinfo-name">Title:</th><td class="field-body">The BitTorrent Protocol Specification</td>
<tr class="title field"><th class="docinfo-name">Title:</th><td class="field-body">The BitTorrent Protocol Specification</td>
</tr>
<tr><th class="docinfo-name">Version:</th>
<td>f61df4010379f5c40090a9b77b73e57db7045dee</td></tr>
<tr class="field"><th class="docinfo-name">Last-Modified:</th><td class="field-body">Fri Oct 11 15:00:20 2013 -0700</td>
<td>0e08ddf84d8d3bf101cdf897fc312f2774588c9e</td></tr>
<tr class="last-modified field"><th class="docinfo-name">Last-Modified:</th><td class="field-body">Sat Feb 4 12:58:40 2017 +0100</td>
</tr>
<tr><th class="docinfo-name">Author:</th>
<td>Bram Cohen &lt;<a class="reference external" href="mailto:bram&#37;&#52;&#48;bittorrent&#46;com">bram<span>&#64;</span>bittorrent<span>&#46;</span>com</a>&gt;</td></tr>
<tr><th class="docinfo-name">Status:</th>
<td>Final</td></tr>
<tr class="field"><th class="docinfo-name">Type:</th><td class="field-body">Standard</td>
<tr class="type field"><th class="docinfo-name">Type:</th><td class="field-body">Standard</td>
</tr>
<tr class="field"><th class="docinfo-name">Created:</th><td class="field-body">10-Jan-2008</td>
<tr class="created field"><th class="docinfo-name">Created:</th><td class="field-body">10-Jan-2008</td>
</tr>
<tr class="field"><th class="docinfo-name">Post-History:</th><td class="field-body">24-Jun-2009 (<a class="reference external" href="mailto:arvid&#37;&#52;&#48;bittorrent&#46;com">arvid<span>&#64;</span>bittorrent<span>&#46;</span>com</a>), clarified the encoding of strings in torrent files.
<tr class="post-history field"><th class="docinfo-name">Post-History:</th><td class="field-body">24-Jun-2009 (<a class="reference external" href="mailto:arvid&#37;&#52;&#48;bittorrent&#46;com">arvid<span>&#64;</span>bittorrent<span>&#46;</span>com</a>), clarified the encoding of strings in torrent files.
20-Oct-2012 (<a class="reference external" href="mailto:arvid&#37;&#52;&#48;bittorrent&#46;com">arvid<span>&#64;</span>bittorrent<span>&#46;</span>com</a>), clarified that info-hash is the digest of en bencoding found in .torrent file.
Introduced some references to new BEPs and cleaned up formatting.
11-Oct-2013 (<a class="reference external" href="mailto:arvid&#37;&#52;&#48;bittorrent&#46;com">arvid<span>&#64;</span>bittorrent<span>&#46;</span>com</a>), correct the accepted and de-facto sizes for request messages</td>
11-Oct-2013 (<a class="reference external" href="mailto:arvid&#37;&#52;&#48;bittorrent&#46;com">arvid<span>&#64;</span>bittorrent<span>&#46;</span>com</a>), correct the accepted and de-facto sizes for request messages
04-Feb-2017 (<a class="reference external" href="mailto:the8472&#46;bep&#37;&#52;&#48;infinite-source&#46;de">the8472<span>&#46;</span>bep<span>&#64;</span>infinite-source<span>&#46;</span>de</a>), further info-hash clarifications, added resources for new implementors</td>
</tr>
</tbody>
</table>
Expand Down Expand Up @@ -168,11 +169,17 @@ <h1>trackers</h1>
<p>Tracker GET requests have the following keys:</p>
<dl class="docutils">
<dt>info_hash</dt>
<dd>The 20 byte sha1 hash of the bencoded form of the info value from the
metainfo file. Note that this is a substring of the metainfo
file. The info-hash must be the hash of the encoded form as found
in the .torrent file, regardless of it being invalid. This value
will almost certainly have to be escaped.</dd>
<dd><p class="first">The 20 byte sha1 hash of the bencoded form of the info value from the
metainfo file. This value will almost certainly have to be escaped.</p>
<p class="last">Note that this is a substring of the metainfo file.
The info-hash must be the hash of the encoded form as found
in the .torrent file, which is identical to bdecoding the metainfo file,
extracting the info dictionary and encoding it <em>if and only if</em> the
bdecoder fully validated the input (e.g. key ordering, absence of leading zeros).
Conversely that means clients must either reject invalid metainfo files
or extract the substring directly.
They must not perform a decode-encode roundtrip on invalid data.</p>
</dd>
<dt>peer_id</dt>
<dd>A string of length 20 which this downloader uses as its id. Each
downloader generates its own id at random at the start of a new
Expand Down Expand Up @@ -360,6 +367,16 @@ <h1>peer messages</h1>
are three times as likely to start as the current optimistic unchoke
as anywhere else in the rotation.</p>
</div>
<div class="section" id="resources">
<h1>Resources</h1>
<ul class="simple">
<li>The <a class="reference external" href="http://bittorrent.org/bittorrentecon.pdf">BitTorrent Economics Paper</a> outlines some request and choking
algorithms clients should implement for optimal performance</li>
<li>When developing a new implementation the Wireshark protocol analyzer and
its <a class="reference external" href="https://wiki.wireshark.org/BitTorrent">dissectors for bittorrent</a> can be useful to debug and compare with
existing ones.</li>
</ul>
</div>
<div class="section" id="copyright">
<h1>Copyright</h1>
<p>This document has been placed in the public domain.</p>
Expand Down
6 changes: 6 additions & 0 deletions beps/bep_0004.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Reserved Bit Allocations
0x02 XBT Peer Exchange
0x04 suggest, haveall, havenone, reject request, and allow fast extensions
0x08 NAT Traversal
0x10 hybrid torrent legacy to v2 upgrade

There are known collisions::

Expand Down Expand Up @@ -84,6 +85,11 @@ Reserved Message IDs
Additional IDs used in deployed clients:
0x14 LTEP Handshake (implemented in libtorrent, uTorrent,...)

Hash Transfer Protocol:
0x15 hash request
0x16 hashes
0x17 hash reject

References
==========

Expand Down
22 changes: 16 additions & 6 deletions beps/bep_0005.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ <h1><a href="../index.html">BitTorrent<span>.org</span></a></h1>
<tr class="title field"><th class="docinfo-name">Title:</th><td class="field-body">DHT Protocol</td>
</tr>
<tr><th class="docinfo-name">Version:</th>
<td>023256c7581a4bed356e47caf8632be2834211bd</td></tr>
<tr class="last-modified field"><th class="docinfo-name">Last-Modified:</th><td class="field-body">Thu Jan 12 12:29:12 2017 -0800</td>
<td>369b2bc90d29397f18f6a51ce517f65780cc0b13</td></tr>
<tr class="last-modified field"><th class="docinfo-name">Last-Modified:</th><td class="field-body">Mon May 1 15:42:08 2017 -0700</td>
</tr>
<tr><th class="docinfo-name">Author:</th>
<td>Andrew Loewenstern &lt;<a class="reference external" href="mailto:drue&#37;&#52;&#48;bittorrent&#46;com">drue<span>&#64;</span>bittorrent<span>&#46;</span>com</a>&gt;, Arvid Norberg &lt;<a class="reference external" href="mailto:arvid&#37;&#52;&#48;bittorrent&#46;com">arvid<span>&#64;</span>bittorrent<span>&#46;</span>com</a>&gt;</td></tr>
Expand Down Expand Up @@ -207,17 +207,20 @@ <h1>KRPC Protocol</h1>
single packet is sent in response. There is no retry. There are three
message types: query, response, and error. For the DHT protocol, there
are four queries: ping, find_node, get_peers, and announce_peer.</p>
<p>A KRPC message is a single dictionary with two keys common to
<p>A KRPC message is a single dictionary with three keys common to
every message and additional keys depending on the type of message.
Every message has a key &quot;t&quot; with a string value representing a transaction
ID. This transaction ID is generated by the querying node and is echoed
in the response, so responses may be correlated with multiple queries
to the same node. The transaction ID should be encoded as a short string
of binary numbers, typically 2 characters are enough as they cover 2^16
outstanding queries. The other key contained in every KRPC message is &quot;y&quot;
with a single character value describing the type of message. The value
outstanding queries. Every message also has a key &quot;y&quot; with a single
character value describing the type of message. The value
of the &quot;y&quot; key is one of &quot;q&quot; for query, &quot;r&quot; for response, or &quot;e&quot; for
error.</p>
error. A key &quot;v&quot; should be included in every message with a client version
string. The string should be a two character client identifier registered
in BEP 20 <a class="footnote-reference" href="#bep-20" id="id3">[3]</a> followed by a two character version identifier. Not all
implementations include a &quot;v&quot; key so clients should not assume its presence.</p>
<div class="section" id="contact-encoding">
<h2>Contact Encoding</h2>
<p>Contact information for peers is encoded as a 6-byte string. Also
Expand Down Expand Up @@ -418,6 +421,13 @@ <h1>References</h1>
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Use SHA1 and plenty of entropy to ensure a unique ID.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="bep-20" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>BEP_0020. Peer ID Conventions.
(<a class="reference external" href="http://www.bittorrent.org/beps/bep_0020.html">http://www.bittorrent.org/beps/bep_0020.html</a>)</td></tr>
</tbody>
</table>
</div>
<div class="section" id="copyright">
<h1>Copyright</h1>
Expand Down
14 changes: 10 additions & 4 deletions beps/bep_0005.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,17 +186,20 @@ single packet is sent in response. There is no retry. There are three
message types: query, response, and error. For the DHT protocol, there
are four queries: ping, find_node, get_peers, and announce_peer.

A KRPC message is a single dictionary with two keys common to
A KRPC message is a single dictionary with three keys common to
every message and additional keys depending on the type of message.
Every message has a key "t" with a string value representing a transaction
ID. This transaction ID is generated by the querying node and is echoed
in the response, so responses may be correlated with multiple queries
to the same node. The transaction ID should be encoded as a short string
of binary numbers, typically 2 characters are enough as they cover 2^16
outstanding queries. The other key contained in every KRPC message is "y"
with a single character value describing the type of message. The value
outstanding queries. Every message also has a key "y" with a single
character value describing the type of message. The value
of the "y" key is one of "q" for query, "r" for response, or "e" for
error.
error. A key "v" should be included in every message with a client version
string. The string should be a two character client identifier registered
in BEP 20 [#BEP-20]_ followed by a two character version identifier. Not all
implementations include a "v" key so clients should not assume its presence.

Contact Encoding
----------------
Expand Down Expand Up @@ -418,6 +421,9 @@ References

.. [#entropy] Use SHA1 and plenty of entropy to ensure a unique ID.

.. [#BEP-20] BEP_0020. Peer ID Conventions.
(http://www.bittorrent.org/beps/bep_0020.html)

Copyright
=========

Expand Down
13 changes: 11 additions & 2 deletions beps/bep_0009.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,18 +114,26 @@ Example::
{'msg_type': 2, 'piece': 0}
d8:msg_typei1e5:piecei0ee


magnet URI format
=================

The magnet URI format is::

magnet:?xt=urn:btih:<info-hash>&dn=<name>&tr=<tracker-url>&x.pe=<peer-address>
v1: magnet:?xt=urn:btih:<info-hash>&dn=<name>&tr=<tracker-url>&x.pe=<peer-address>
v2: magnet:?xt=urn:btmh:<tagged-info-hash>&dn=<name>&tr=<tracker-url>&x.pe=<peer-address>

<info-hash>
Is the info-hash hex encoded, for a total of 40 characters. For
compatability with existing links in the wild, clients should also
support the 32 character `base32`_ encoded info-hash.


<tagged-info-hash>
Is the `multihash`_ formatted, hex encoded full infohash
for torrents in the new metadata format. 'btmh' and 'btih'
exact topics may exist in the same magnet if they describe
the same hybrid torrent.

<peer-address>
A peer address expressed as ``hostname:port``, ``ipv4-literal:port`` or ``[ipv6-literal]:port``.
This parameter can be included to initiate a direct metadata transfer between two clients while reducing the need for external peer sources.
Expand All @@ -148,6 +156,7 @@ References
.. _`base32`: http://www.ietf.org/rfc/rfc3548.txt
.. _`BEP 0010`: http://www.bittorrent.org/beps/bep_0010.html
.. _`BEP 0005`: http://www.bittorrent.org/beps/bep_0005.html
.. _`multihash`: https://github.com/multiformats/multihash


Copyright
Expand Down
Loading