Scale ATProto handle resolution

[snarfed]

We need to serve ATProto handle resolution for all users bridged into ATProto (background: #381). They support both DNS and HTTPS methods, but our ATProto handles are multi-level, eg @[email protected] becomes user.mastodon.social.ap.brid.gy, and you can't make multi-level wildcard SSL certs, so HTTPS won't work, so DNS it is.

We're currently using Google Cloud DNS. It serves the entire brid.gy DNS zone, and we create _atproto records for handle resolution manually. The catch is that GCP DNS has a hard limit of 10k records per zone, which we'll likely outgrow. Grr.

We could make a zone per sub-subdomain, eg per fediverse instance, so mastodon.social.ap.brid.gy would become its own zone, but GCP DNS also has a hard limit of 10k zones total.

To do:

• When disabling ATProto for an account, remove its DNS record #1268
• delete handles for disabled accounts
• ask for another quota bump, to 100k or 200k

So, we eventually need to switch to a different programmatic DNS service or run our own DNS server. Whee.

[snarfed]

Provider lists:
https://en.wikipedia.org/wiki/List_of_managed_DNS_providers
https://www.g2.com/categories/managed-dns-providers
https://serverauth.com/posts/comparing-dns-hosting-providers

[snarfed]

Upcoming option: https://dns.kitchen/ , all you can eat zones/records, $5/mo.

[snarfed]

@neatnik mentioned https://desec.io/ too.

[snarfed]

Also https://dns.he.net/, https://www.nova53.net/, https://ns-global.zone/ .

[snarfed]

Maybe obsoleted by #830 🤞

[snarfed]

...nope, turns out #830 probably won't work after all, so this issue definitely still applies.

[snarfed]

The 10k limit in Google Cloud DNS is a quota, rrsetsPerManagedZone, that I can request an increase for. May try that.

• https://cloud.google.com/dns/quotas#quotas
• https://console.cloud.google.com/iam-admin/quotas?service=dns.googleapis.com&metric=dns.googleapis.com%2Fquota%rrsets_per_managed_zone&project=brid-gy

[snarfed]

Tried requesting a quota bump. 🤞

[snarfed]

In unrelated very good news, @bnewbold built us a whole new microservice to solve this! bluesky-social/atproto#1697 (comment) . Extremely generous of him. Thank you Bryan!!!

[snarfed]

More good news, I asked for a GCP DNS quota bump to 50k and got it. Woo!

[snarfed]

We're currently at 6500 DNS records, with the limit still at 50k. We'll (hopefully) still need to figure this out eventually, but it seems like we have plenty of time.

[snarfed]

Count is 28k now! Out of quota of 50k. 😳 I think a decent number of those are disabled, many but not all from Flipboard.

[snarfed]

37k DNS records now! Time to get serious about this.

[louis-lau]

I have no idea what this software is, but stumbled across this issue while researching secondary dns providers. :)

Given the programmatic nature of these records, maybe you'd be better off with something like powerdns with the remote backend?
https://doc.powerdns.com/authoritative/backends/remote.html

Or if you need a managed service, Bunny DNS?
https://bunny.net/dns/

You'd never run into record limitations, as the responses are just created on the fly.

[bnewbold]

Here is a direct link to handlr, the DNS proxying service I wrote:

bluesky-social/indigo#665

it hasn't been reviewed or merged to main branch yet. If I get a chance I can respond to things and try to get it merged, but may or may not be responsive about updating/maintaining it. If bridgy ends up depending on it, might be worth pulling the code out in to a separate repo? I wrote this in spare time and am flexible/amenable to basically any license on the code. Or you could re-write in python for something more familiar/maintainable.

(bumping quota is probably the near-term solution though?)

[shiribailem]

Just chiming in here as it's related to another I just added a comment to... Have you considered just running your own nameserver?

This would both mean no limits and faster updates as records change among other possible gains?

[snarfed]

@shiribailem yes! Definitely considered that, hence "or run our own DNS server" in the original description here. It would mean no limits and faster updates, but it would also mean running and maintaining our own nameserver. One more level of admin cost and risk. Whee.

But yeah we may end up doing that regardless, esp since @bnewbold built us one! #744 (comment)

[shiribailem]

I mean the good news is that nameservers are very very low requirements on resources, performance, and effort, even for security.

The worst risk is the fully custom service vs just using internal APIs/config-files to update zones on an established DNS server.

Given that DNS isn't a big attack surface compared to the entire bridge software itself, I wouldn't stress as much over that.

[snarfed]

#1268 and #1458 have been productive here, we've already reclaimed more than 12k DNS records, down to just 25k now and still shrinking.

[snarfed]

Done! Reclaimed ~13.5k total, currently at 24.5k out of 50k quota.

[snarfed]

Requested a quota bump to 200k. 🤞

[shiribailem]

just randomly thinking... have you considered stripping periods from the converted usernames? That would make it trivial to switch to HTTPS resolution... Sadly that's something that would be easier at the beginning of the project before you had thousands of users, but I could still see it working.

If you did that, I'd probably leverage the code for setting custom domains and just convert all existing listings to custom domains (which would otherwise leave them unchanged). From there you could just use HTTPS verification going forward.

[snarfed]

@shiribailem we have, #744 (comment), #830

[snarfed]

Holy crap, they gave us the quota bump to 200k records! 🎉