gateway: attempt to resolve hostname to ipfs path #726

kevinwallace · 2015-02-02T07:02:21Z

This allows someone to host a static site by pointing a TXT record at their
content in IPFS, and a CNAME record at an IPFS gateway.

Note that such a setup technically violates RFC1912 (section 2.4; "A CNAME
record is not allowed to coexist with any other data."), but tends to work in
practice.

We may want to consider changing the DNS->IPFS resolution scheme to allow this
scenario to be RFC-compliant (e.g. store the mapping on a well-known subdomain
to allow CNAME records on the domain itself).

Also note that while this has been tested by hand, I don't see a good way to
test it with the existing sharness gateway tests -- how do you mock a DNS entry
from a shell script?

whyrusleeping · 2015-02-02T07:32:17Z

Unless im misunderstanding the change, this was already possible through ipns. but maybe im missing something, the gateway isnt a super familiar part of the codebase for me.

whyrusleeping · 2015-02-02T07:42:08Z

Ah, wait, i got it, if you make (for example) jero.my point to gateway.ipfs.io via CNAME and have a TEXT record QmX it will open up gateway.ipfs.io/ipfs/QmX. Got it. Neat

jbenet · 2015-02-02T08:37:41Z

@kevinwallace thanks for the PR! I'll get to reviewing this in more detail in the next few hrs. I'll add a couple comments now.

jbenet · 2015-02-02T08:42:07Z

core/corehttp/gateway.go

@@ -12,8 +12,7 @@ func GatewayOption(writable bool) ServeOption {
 		if err != nil {
 			return err
 		}
-		mux.Handle("/ipfs/", gateway)
-		mux.Handle("/ipns/", gateway)
+		mux.Handle("/", gateway)


Here, I think we want to keep using the muxer. We're going towards:

mux.Handle("/ipfs/", ipfsHandler) mux.Handle("/ipns/", ipnsHandler) // ipnsHandler is sort of middleware that uses ipfsHandler mux.Handle("/", base)

where baseHandler can be mux between:

base.Handle("/webui", redirectToWebUIHandler) base.Handle("/", rootHandler) // give the user some information about IPFS + links (e.g. to WebUI, ipfs.io, etc) // base should handle 404s with custom error pages.

cc @mappum

Hmm. In that case, this is probably best expressed as a pre-muxer request processing stage, which tries to resolve req.Host as an IPNS name, and if successful, rewrites the request path to /ipfs/<resolved name><path> and allows the muxer/gateway to process the rewritten request. I'd need to extend ServeOption to support registering such a stage.

It may be doable by having two muxers:

one mounted on the server at "/". performs the check, possibly rewriting the path, and then calls the other:

one which handles everything else.

kevinwallace · 2015-02-03T07:19:19Z

PTAL -- I've updated my approach per @jbenet's comments.

A ServeOption now has an additional return value, and can now mediate requests to future options by chaining muxes. A new option, IPNSHostnameOption, uses this to rewrite incoming requests for hostnames that resolve as IPNS names. This option is only installed on the gateway server, rather than any server that installs GatewayOption.

jbenet · 2015-02-03T09:17:02Z

Thanks @kevinwallace -- beautiful, this LGTM.

On tests, maybe the right way is to test it in Go with a mock Namesys the test controls. The full test is hard to do without messing with hosts files/etc. I think the mock covers us 80% of the way.

If we were to go for a well-known subdomain, we could do something like ipfs-gateway.<domain>. (Know examples of similar setups?)

jbenet · 2015-02-03T09:17:36Z

Lets go for that last addition and then merge this?

kevinwallace · 2015-02-03T20:31:38Z

@jbenet Sounds good. Are you imagining something that tests overall gateway functionality end-to-end (including IPNS hostnames), or just the request rewriting option in isolation from the gateway?

About the well-known subdomain: I was thinking about what this would look like if it were done DNS-SD style (which uses names like _http._tcp.example.com along with SRV attributes to point at specific TCP/UDP endpoints and TXT records to specify other key/value attributes). I'd like to read the DNS-SD RFC more closely before making a concrete proposal in this vein, but here's a spitball:

_ipns._ipfs.example.com IN TXT "path=/ipfs/QmX..." points /ipns/example.com to /ipfs/QmX....

jbenet · 2015-02-03T21:23:42Z

I'm imagining testing with http requests to the gateway (in go), with a mocked naming system.

Yeah, i need to read the spec closely too. Wish dns-sd records didnt have the ugly _, but oh well. Maybe use _ipns.example.com directly, as ipns is a thing unto itself. (Also since this is dns-sd, btw, could be the same identifiers to use when advertising link-local services through mdns and so on.)

kevinwallace · 2015-02-04T19:46:09Z

Just as a heads-up, I'm likely going to be preoccupied with other things for the next couple of days, so I'm not sure I'll be able to get to this until this weekend. It's not forgotten, though!

whyrusleeping · 2015-02-04T22:14:01Z

@kevinwallace, no worries! you wont be forgotten

Each option now additionally returns the mux to be used by future options. If every options returns the mux it was passed, the current behavior is unchanged. However, if the option returns an a new mux, it can mediate requests to handlers provided by future options: return func(n *core.IpfsNode, mux *http.ServeMux) (*http.ServeMux, error) { childMux := http.NewServeMux() mux.Handle("/", handlerThatDelegatesToChildMux) return childMux, nil } License: MIT Signed-off-by: Kevin Wallace <[email protected]>

This allows someone to host a static site by pointing a TXT record at their content in IPFS, and a CNAME record at an IPFS gateway. Note that such a setup technically violates RFC1912 (section 2.4; "A CNAME record is not allowed to coexist with any other data."), but tends to work in practice. We may want to consider changing the DNS->IPFS resolution scheme to allow this scenario to be RFC-compliant (e.g. store the mapping on a well-known subdomain to allow CNAME records on the domain itself). License: MIT Signed-off-by: Kevin Wallace <[email protected]>

License: MIT Signed-off-by: Kevin Wallace <[email protected]>

kevinwallace · 2015-02-08T20:57:42Z

OK, finally got around to this. Sorry it took so long!

Added core/corehttp/gateway_test.go, which brings up a gateway server with a mocked namesys, then makes various requests to test its functionality.

jbenet · 2015-02-13T02:49:12Z

core/corehttp/gateway_test.go

+func (m mockNamesys) CanResolve(name string) bool {
+	_, ok := m[name]
+	return ok
+}


this should probably just return true. this is likely a doc failure, but the CanResolve func is supposed to return whether "a name is resolvable by the system" i.e. a DNSResolver would check if the string is a domain.

(can we think of a clearer name for this func?)

LooksResolvable, LooksApplicable? I like the Looks prefix to indicate that we're only examining the name on a surface level, but I don't like the way either of those names read...

Matches, NameMatches?

I like Looks as well... CanResolve implies a little more action than we are actually doing

it will be written ns.<word>(name)

ns.LooksApplicable(name) ns.LooksResolvable(name) ns.Compatible(name) ns.CompatibleWith(name) ns.MayResolve(name) ns.CanResolve(name) ns.IsResolvable(name) ns.Resolvable(name)

jbenet · 2015-02-13T02:52:14Z

This LGTM. thanks again for the PR!

gateway: attempt to resolve hostname to ipfs path

whyrusleeping · 2015-02-13T03:06:25Z

👍

kevinwallace · 2015-02-13T03:34:47Z

Thanks!

jbenet reviewed Feb 2, 2015
View reviewed changes

kevinwallace added 4 commits February 8, 2015 11:27

corehttp: tear off makeHandler, for tests

794b7b7

License: MIT Signed-off-by: Kevin Wallace <[email protected]>

corehttp: add test for gateway with mocked namesys

e5abf07

License: MIT Signed-off-by: Kevin Wallace <[email protected]>

jbenet reviewed Feb 13, 2015
View reviewed changes

jbenet added a commit that referenced this pull request Feb 13, 2015

Merge pull request #726 from kevinwallace/gateway_hostname

2476499

gateway: attempt to resolve hostname to ipfs path

jbenet merged commit 2476499 into ipfs:master Feb 13, 2015

kevinwallace deleted the gateway_hostname branch February 13, 2015 03:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gateway: attempt to resolve hostname to ipfs path #726

gateway: attempt to resolve hostname to ipfs path #726

kevinwallace commented Feb 2, 2015

whyrusleeping commented Feb 2, 2015

whyrusleeping commented Feb 2, 2015

jbenet commented Feb 2, 2015

jbenet Feb 2, 2015

kevinwallace Feb 3, 2015

jbenet Feb 3, 2015

kevinwallace commented Feb 3, 2015

jbenet commented Feb 3, 2015

jbenet commented Feb 3, 2015

kevinwallace commented Feb 3, 2015

jbenet commented Feb 3, 2015

kevinwallace commented Feb 4, 2015

whyrusleeping commented Feb 4, 2015

kevinwallace commented Feb 8, 2015

jbenet Feb 13, 2015

kevinwallace Feb 13, 2015

kevinwallace Feb 13, 2015

whyrusleeping Feb 13, 2015

jbenet Feb 13, 2015

jbenet commented Feb 13, 2015

whyrusleeping commented Feb 13, 2015

kevinwallace commented Feb 13, 2015

gateway: attempt to resolve hostname to ipfs path #726

gateway: attempt to resolve hostname to ipfs path #726

Conversation

kevinwallace commented Feb 2, 2015

whyrusleeping commented Feb 2, 2015

whyrusleeping commented Feb 2, 2015

jbenet commented Feb 2, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinwallace commented Feb 3, 2015

jbenet commented Feb 3, 2015

jbenet commented Feb 3, 2015

kevinwallace commented Feb 3, 2015

jbenet commented Feb 3, 2015

kevinwallace commented Feb 4, 2015

whyrusleeping commented Feb 4, 2015

kevinwallace commented Feb 8, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbenet commented Feb 13, 2015

whyrusleeping commented Feb 13, 2015

kevinwallace commented Feb 13, 2015