Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use Bluesky firehose instead of polling #978

Closed
6 of 7 tasks
Tracked by #381
snarfed opened this issue Apr 29, 2024 · 4 comments
Closed
6 of 7 tasks
Tracked by #381

Use Bluesky firehose instead of polling #978

snarfed opened this issue Apr 29, 2024 · 4 comments
Labels
now

Comments

@snarfed
Copy link
Owner

snarfed commented Apr 29, 2024 •
edited
Loading

We should revisit using the Bluesky firehose instead of polling notifications. Much heavier, but would get rid of the up to 5m delay, which is bad UX. Thanks @ShreyanJain9 for the nudge!

(This would obsolete #943, #954.)

  • basic firehose consumer
  • detect commits from our bridged Bluesky users
  • detect commits that reference our bridged non-Bluesky users
  • store Objects, enqueue receive tasks
  • store cursor
  • reconnect when disconnected
  • monitoring
This was referenced Apr 29, 2024
Closed
Closed
@snarfed snarfed added the now label May 4, 2024
snarfed added a commit that referenced this issue May 4, 2024
@snarfed
ATProto poll notifs: limit 100 for protocol bot users
832b7a0 
try to make sure we don't miss any follows until we switch to firehose in #978

for #997
This was referenced May 5, 2024
Closed
Closed
snarfed added a commit that referenced this issue May 7, 2024
@snarfed
ATProto: start on firehose subscriber to replace polling
6f4d79a 
can't wait for this! for #978
snarfed added a commit that referenced this issue May 8, 2024
@snarfed
ATProto firehose: correctly load and handle bridged ATProto vs non-AT…
3057dd6 
…Proto DIDs

for #978
snarfed added a commit that referenced this issue May 8, 2024
@snarfed
ATProto firehose: convert to function we can run in a thread
0f35a2e 
for #978
snarfed added a commit that referenced this issue May 8, 2024
@snarfed
ATProto firehose: start tests
7ecf3dc 
for #978
@snarfed
Copy link
Owner Author

snarfed commented May 8, 2024

Making progress on this! Hoping to ship it this week.

This was referenced May 8, 2024
Closed
Closed
snarfed added a commit that referenced this issue May 9, 2024
@snarfed
ATProto firehose: more tests
e214bc0 
for #978
snarfed added a commit that referenced this issue May 9, 2024
@snarfed
ATProto firehose: more tests
44763ca 
for #978
snarfed added a commit that referenced this issue May 9, 2024
@snarfed
ATProto firehose: test replies, mentions
72ae63f 
#978
snarfed added a commit that referenced this issue May 9, 2024
@snarfed
ATProto firehose: implement quote posts
20afb59 
for #978
snarfed added a commit that referenced this issue May 9, 2024
@snarfed
ATProto firehose: bug fix for quote posts with media embeds
5c77f02 
for #978
snarfed added a commit that referenced this issue May 9, 2024
@snarfed
ATProto firehose: implement deletes
7867326 
for #978
snarfed added a commit that referenced this issue May 9, 2024
@snarfed
ATProto firehose: tests for updates
0d24faf 
for #978
snarfed added a commit that referenced this issue May 9, 2024
@snarfed
ATProto firehose: noop refactoring, add Op namedtuple
b4841f3 
for #978
snarfed added a commit that referenced this issue May 9, 2024
@snarfed
atproto_firehose.handle: implement delete
b8e0fec 
for #978
snarfed added a commit that referenced this issue May 9, 2024
@snarfed
ATProto firehose: start implementing cursor
b788105 
for #978
@snarfed snarfed mentioned this issue May 9, 2024
Closed
snarfed added a commit that referenced this issue May 9, 2024
@snarfed
ATProto firehose: subscribe at stored cursor + 1
eea8779 
for #978
@snarfed
Copy link
Owner Author

snarfed commented May 9, 2024

Big progress here, starting to roll this out today.

@snarfed
Copy link
Owner Author

snarfed commented May 9, 2024

Shipped this to production! With training wheels. We're still polling too, but I'm hopeful that we can turn that down pretty soon.

snarfed added a commit that referenced this issue May 10, 2024
@snarfed
ATProto firehose: don't assume required fields, handle error reportin…
940c0e8 
…g errors

for #978
snarfed added a commit that referenced this issue May 10, 2024
@snarfed
ATProto firehose: run in threads in hub
20d23c5 
for #978
snarfed added a commit that referenced this issue May 10, 2024
@snarfed
ATProto firehose: reload our DIDs every 20s
5ef4111 
for #978
snarfed added a commit that referenced this issue May 10, 2024
@snarfed
ATProto firehose: fix race conditions in loading DIDs
d56f405 
for #978
snarfed added a commit that referenced this issue May 11, 2024
@snarfed
ATProto firehose: load bridged dids from AtpRepo instead of users
6bc9269 
hopefully cuts down on datastore load a bit

for #978
snarfed added a commit that referenced this issue May 11, 2024
@snarfed
ATProto firehose: add handle wrapper that catches exceptions
f2b5f79 
also refactor and simplify load_dids, error reporting. #978
@snarfed
Copy link
Owner Author

snarfed commented May 14, 2024

...and new posts for https://bsky.app/profile/rolle.mementomori.social.ap.brid.gy are showing up now too! At least some. So that was either just waiting on getRepo, or hashtag posts weren't working, or both.

Excited that this has (hopefully) landed. Tentatively closing!

@snarfed snarfed closed this as completed May 14, 2024
snarfed added a commit that referenced this issue May 14, 2024
@snarfed
atproto_firehose: make its own GAE service, lots of other refactoring
8eeb588 
optimize loading DIDs, move around threads and exception handling, etc.

for #978
snarfed added a commit that referenced this issue May 14, 2024
@snarfed
ActivityPub: disable polling! 🤞
e7e1f10 
for #978
snarfed added a commit that referenced this issue May 22, 2024
@snarfed
ATProto: delete polling code and config
7f72a62 
for #978. end of a (brief) era!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
now
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@snarfed