Web: don't allow sites without feed or webmentions

[snarfed]

We get a number of sites now that have no RSS or Atom feed or mf2, and are unlikely to post via webmention. It's awkward to bridge those if we'll never bridge any posts for them.

We can easily tell if there's an RSS or Atom feed right now, and we know if a site has already sent us a webmention or not, but we don't distinguish where profile info comes from, ie mf2 vs mf1 vs metaformats.

Maybe the most likely thing to do hwere would be to try to detect mf2 at fetch time and store it in a new Web.has_mf2_hcard property, and either backfill or only block a site if we fetched their homepage after we launch that.

[snarfed]

This is needed to do #1403 for web users.

[snarfed]

I have a start on this in a stash, don't bridge Web users without feed or mf2 h-card, #1458.

[ZipMartini]

Does a site's RSS feed always have to be the site's index? Obviously the fetch can't know if an RSS feed does exist, but in a different page, i.e. blog.xml...

[Tamschi]

I don't know how Bridgy Fed does this, but generally speaking <link rel="alternate" type="application/rss+xml" title="…" href="…"> tags are used for RSS feed auto-discovery. I'd assume that or a similar tag would have to be present on the homepage, and that the RSS feed itself can be anywhere. The system wouldn't work with many sites otherwise.

[snarfed]

Right! More broadly, Bridgy Fed only supports whole web sites right now, ie not specific pages or paths.

[D-Melhede]

Why are direct RSS feed URLs not supported ?
dr.dk doesn't produce any content on Bluesky, but it actually has multiple different RSS feeds available, e.g.
https://www.dr.dk/nyheder/service/feeds/senestenyt

[snarfed]

(^ answered in #1527)

[snarfed]

Once this is done, I'd also like to backfill it, ie go through all existing bridged web sites and disable the ones that don't have a feed or webmentions.

[snarfed]

Currently backfilling this, ie deactivating (and removing DNS! #1268) for matching web sites, with:

ids = []
for w in Web.query(Web.copies.protocol == 'atproto'):
  if w.status and not w.manual_opt_out:
    ids.append(w.key.id())
    print(len(ids), w.key.id())
    try:
      did = w.get_copy(ATProto)
      repo = arroba.server.storage.load_repo(did)
      if not repo.status:
        arroba.server.storage.deactivate_repo(repo)
      ATProto.remove_dns(w.handle_as('atproto'))
    except:
      logging.exception('')

[snarfed]

Done, removed 3362 DNS entries. Now doing ActivityPub, will follow up in #1268