refactor(proxy): move to coco pooled storage, cancellable async everywhere #944

kim · 2020-09-21T14:49:43Z

No description provided.

kim · 2020-09-21T14:52:06Z

proxy/coco/src/state.rs

+        Ok(self
+            .api
+            .with_storage(move |storage| storage.has_commit(&urn, oid))
+            .await??)


Before you ask: ?? is pronounced The wat? operator, was stabilised in Rust'98, and is part of the Rust Dawallah Level II certification programme.

I'm looking forward to the interrobang RFC. ?! will be the The WAT?! operator and will indicate to code reviewers that they should move along

kim · 2020-09-21T14:57:17Z

This is unfortunately very invasive, as it changes virtually everything to async.

Note that resetting the on-disk state is removed entirely, as that was done in a way very likely to cause unexplicable bugs down the road -- both on-disk state and network stack must be managed in lockstep.

xla · 2020-09-21T15:17:23Z

This is unfortunately very invasive, as it changes virtually everything to async.

We don't accept any non-invasive PRs.

Note that resetting the on-disk state is removed entirely, as that was done in a way very likely to cause unexplicable bugs down the road -- both on-disk state and network stack must be managed in lockstep.

What do you propose in order to enable the UI integration tests to reset the states inbetween test runs?

kim · 2020-09-21T16:16:29Z

What do you propose in order to enable the UI integration tests to reset the states inbetween test runs?

I don't know, actually. A "safe" API which would take care of polling the network future would probably require `async_closure`. Is it absolutely necessary that the entire state is wiped for those tests? Like, do they depend on the keypair being used? Or could they just randomise the projects being created?

xla · 2020-09-21T16:19:20Z

Is it absolutely necessary that the entire state is wiped for those tests? Like, do they depend on the keypair being used? Or could they just randomise the projects being created?

A lot of the tests depend on the fact that the monorepo is without a userr/owner -- mainly to test onboarding. To take a step back, what we want is to have state isolation between test runs to avoid spurious test failures from residue of other runs, maybe there is a saner way than resetting the entire state carried in the context.

xla · 2020-09-21T16:20:36Z

To expand on this, the granularity between these tests varies but is mostly on the level of user flows closely modeled after figma user flows.

FintanH · 2020-09-21T16:28:49Z

Do the frontend tests need to use the same proxy process or can they change between test runs? What I'm thinking is that a cypress test could consist of proxy --root .tmp/some/dir

xla · 2020-09-21T16:34:03Z

Do the frontend tests need to use the same proxy process or can they change between test runs? What I'm thinking is that a cypress test could consist of proxy --root .tmp/some/dir

AFAIK cypress doesn't control the proxy, maybe @rudolfs can shed some light.

xla · 2020-09-22T05:31:51Z

It looks like Context needs to own the peer handling -- then whenever we reset all the things we start a new peer backed by the new monorepo. Unfortunately JoinHandle is not canceling on drop. To remedy that I can see two strategies:

wrap the Peer::run in an Abortable and the context holds onto the AbortHandle
evaluate exposing cancellation in links Peer/PeerApi with a channel or other construct

@kim Is there any precedence for RunLopp cancellation in link?

rudolfs · 2020-09-22T08:28:23Z

Do the frontend tests need to use the same proxy process or can they change between test runs? What I'm thinking is that a cypress test could consist of proxy --root .tmp/some/dir

AFAIK cypress doesn't control the proxy, maybe @rudolfs can shed some light.

At the moment we start proxy first via a yarn task that also runs the tests. I think technically it would be possible to control the proxy lifecycle from within tests, but I wonder whether it would not slow down the test suite considerably because the proxy takes a while to start up?

warp spawns inside its bowels, so to stop it, we shall implant a poison pill right there

…comes

kim · 2020-09-29T15:06:15Z

Should be good now. Will add some more prose later.

kim · 2020-09-29T15:17:25Z

proxy/coco/tests/gossip.rs

    let alice_events = alice_peer.subscribe();

-    tokio::task::spawn(alice_peer.run(alice_state.clone(), alice_store, RunConfig::default()));
+    let _alice_runs = alice_peer.into_running();


Nb funny semantics: we don't actually need to spawn this, just prevent it from being dropped. It also doesn't hurt to spawn it, though, so whatever people think is preferable in tests

kim · 2020-09-29T15:19:53Z

proxy/api/src/main.rs

-    };
-    let paths = coco::Paths::try_from(paths_config)?;
+            Some(()) = sighup.recv() => {
+                log::info!("SIGHUP received, reloading...");


To the SIG HUP you drop gitit?

Because with --test, everything is in a temp dir, this is equivalent to a reset. Without --test, it's equivalent to, well, restarting.

kim · 2020-09-29T15:20:37Z

proxy/api/src/main.rs

    };

-    if args.test {
-        let state = state.lock().await;
-        // TODO(xla): Given that we have proper ownership and user handling in coco, we should


It did not make any difference whether or not those were set up, so I removed it.

kim · 2020-09-29T15:23:44Z

proxy/coco/src/peer.rs

-            sender.clone().send(TimeoutEvent::SyncPeriod).await.ok();
-        });
-    }
+    announcer_handle.abort();


Not entirely sure about this, actually. Could be we need to wake the tasks when the parent gets aborted.

kim · 2020-09-29T15:26:19Z

proxy/coco/src/state.rs

@@ -328,12 +347,12 @@ impl State {
    /// * If no project for the `urn` was found.
    /// * If the [`git::Browser`] fails.
    /// * If the passed `callback` errors.
-    pub fn with_browser<F, T>(&self, urn: &RadUrn, callback: F) -> Result<T, Error>
+    pub async fn with_browser<F, T>(&self, urn: RadUrn, callback: F) -> Result<T, Error>


Might be worth considering to maintain a pool of Browsers here. Also, spawn_blocking the callback seems advisable.

xla

👏 👏 👏 This is extra dope! Kill all the locks. 👏 👏 👏

🌔 ✋ 💅 🚓

xla · 2020-09-29T19:50:20Z

proxy/coco/src/peer.rs

+                Err(e) => Poll::Ready(Err(Error::Join(e))),
+                Ok(Err(e)) => Poll::Ready(Err(Error::Aborted(e))),
+                Ok(Ok(Err(e))) => Poll::Ready(Err(e)),
+                Ok(Ok(Ok(()))) => Poll::Ready(Ok(())),


Heard you like Ok's, so have some Ok's in ur Ok's.

xla · 2020-09-29T20:01:48Z

proxy/coco/src/state.rs

@@ -328,12 +347,12 @@ impl State {
    /// * If no project for the `urn` was found.
    /// * If the [`git::Browser`] fails.
    /// * If the passed `callback` errors.
-    pub fn with_browser<F, T>(&self, urn: &RadUrn, callback: F) -> Result<T, Error>
+    pub async fn with_browser<F, T>(&self, urn: RadUrn, callback: F) -> Result<T, Error>


xla

👩 📧 🌬 🐣

FintanH

Thank you 🙇 It's good to see your hand over in the proxy ❤️

rudolfs

Looks solid!
We should update the QA docs, but I won't block on it.

rudolfs · 2020-09-30T08:36:16Z

proxy/api/src/main.rs

@@ -20,46 +32,121 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {

    let mut args = pico_args::Arguments::from_env();
    let args = Args {
-        reset_state: args.contains("--reset-state"),


Now that this is removed, we can close: #960.
However we also need to update the QA docs: https://github.com/radicle-dev/radicle-upstream/blob/master/QA.md#prerequisites.

@kim Would you prefer to do that in this issue or should we open another one?

If the PR affects those bits we should address it in here imo

Fixes a regression introduced in #944: while syncing, the other end may not have the URLs we're interested in, or may go away in the middle of us fetching. We should thus just ignore any errors.

As a follow-up to #944, this implements the Peer task orchestration using hand-rolled futures, in order to ensure proper lifecycle management of spawned tasks.

kim added 2 commits September 18, 2020 13:44

pub use spawn_blocking(.await)

8956274

Compiles

599e551

kim commented Sep 21, 2020

View reviewed changes

kim added 2 commits September 21, 2020 17:48

Slow is the new green

0a957e6

Hello clippy my old friend

bcc82e9

rudolfs assigned kim Sep 21, 2020

kim added 15 commits September 22, 2020 18:31

Fix slowness by using memory keystore + lighter scrypt params

98cae02

Bump link

1db2a80

To the SIG HUP don't STOP just DROP

90e8a1a

CLLLIIIPPPIIIYEEE

530d250

fixup! CLLLIIIPPPIIIYEEE

97c0d5d

Aha!

b770901

Wait a bit for reload

2bc2f33

Who said cooperative concurrency is cooperative?

ad0dd29

warp spawns inside its bowels, so to stop it, we shall implant a poison pill right there

It's timing sensitive \o/

b906cd2

Bump link

c68cd20

For the idiotmatic

e8a5c89

Bump link

56fadb9

Merge remote-tracking branch 'origin/master' into pooled-storage

104bcb2

Restore spawning of "subroutines", but ensure the stop when the drop …

525d83b

…comes

Merge remote-tracking branch 'origin/master' into pooled-storage

f6a678a

xla added improvement labels Sep 29, 2020

xla changed the title ~~WIP companion to radicle-link#327~~ refactor(proxy): move to coco pooled storage Sep 29, 2020

Bump link to latest mistress

19954fc

kim changed the title ~~refactor(proxy): move to coco pooled storage~~ refactor(proxy): move to coco pooled storage, cancellable async everywhere Sep 29, 2020

kim commented Sep 29, 2020

View reviewed changes

xla added 2 commits September 29, 2020 21:22

Merge branch 'master' into pooled-storage

7a3e854

Fix specs

7b84445

xla previously approved these changes Sep 29, 2020

View reviewed changes

Jeez what a shitty abstraction

ee8b2d8

kim dismissed xla’s stale review via ee8b2d8 September 29, 2020 20:54

Fix premature drop

037a9f2

xla requested a review from rudolfs September 30, 2020 07:22

xla approved these changes Sep 30, 2020

View reviewed changes

FintanH approved these changes Sep 30, 2020

View reviewed changes

rudolfs approved these changes Sep 30, 2020

View reviewed changes

kim merged commit ebabc6f into master Sep 30, 2020

kim deleted the pooled-storage branch September 30, 2020 08:56

kim mentioned this pull request Oct 1, 2020

chore(proxy): attempt 3 at proper async task management #983

Merged

rudolfs mentioned this pull request Oct 1, 2020

Extend reset commands #960

Closed

kim mentioned this pull request Oct 1, 2020

fix(proxy): ignore fetch errors in when syncing #989

Merged

rudolfs mentioned this pull request Oct 22, 2020

chore(ui): remove footgun leftovers #1088

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(proxy): move to coco pooled storage, cancellable async everywhere #944

refactor(proxy): move to coco pooled storage, cancellable async everywhere #944

kim commented Sep 21, 2020

kim Sep 21, 2020

FintanH Sep 21, 2020

kim commented Sep 21, 2020

xla commented Sep 21, 2020 •

edited

Loading

kim commented Sep 21, 2020 via email

xla commented Sep 21, 2020 •

edited

Loading

xla commented Sep 21, 2020

FintanH commented Sep 21, 2020

xla commented Sep 21, 2020

xla commented Sep 22, 2020

rudolfs commented Sep 22, 2020

kim commented Sep 29, 2020

kim Sep 29, 2020

kim Sep 29, 2020

kim Sep 29, 2020

kim Sep 29, 2020

kim Sep 29, 2020

xla Sep 29, 2020

xla left a comment

xla Sep 29, 2020

xla Sep 29, 2020

xla left a comment

FintanH left a comment

rudolfs left a comment

rudolfs Sep 30, 2020

NunoAlexandre Sep 30, 2020

refactor(proxy): move to coco pooled storage, cancellable async everywhere #944

refactor(proxy): move to coco pooled storage, cancellable async everywhere #944

Conversation

kim commented Sep 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kim commented Sep 21, 2020

xla commented Sep 21, 2020 • edited Loading

kim commented Sep 21, 2020 via email

xla commented Sep 21, 2020 • edited Loading

xla commented Sep 21, 2020

FintanH commented Sep 21, 2020

xla commented Sep 21, 2020

xla commented Sep 22, 2020

rudolfs commented Sep 22, 2020

kim commented Sep 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xla left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xla left a comment

Choose a reason for hiding this comment

FintanH left a comment

Choose a reason for hiding this comment

rudolfs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xla commented Sep 21, 2020 •

edited

Loading

xla commented Sep 21, 2020 •

edited

Loading