Start porting Akka.Remote.TestKit

[smalldave]

Out of time for the day. This is as far as I've got.

From the bits I've gone through looks like we will need some features implemented in Akka.Remoting and Helios.

Still not clear on the bigger picture. i.e what/if we could replace with remoting, how multiple JVMs are initialised.
Going to spend some time getting my head around that.

[Aaronontheweb]

I took some notes yesterday while working on my own implementation:

I've been studying the canonical MultiNode TestKit in Akka and am working on porting it to C#. Here are my observations thus far on how it works and what I think is practical for porting it to C#.

TL;DR;

The original Akka multinode testkit is complex and includes a lot of moving parts that are impractical or impossible to port to C#. For instance, the Conductor depends on its own Netty connection factory pipeline steps and an entire TestConductorProtocol for interprocess communication.

Our goal should be to emulate the ease-of-use of the MultiNodeSpec and MultiNodeConfig without blindly recreating all of the complexity of the original.

#Active Components
The MultiNodeSpec (source) is the developer-usable class that exposes all of the multinode testing capabilities in an easy-to-use fashion, and it allows the developer to specify different roles for each individual actor system participating in the test as well as functionality to execute on each independent system.

Take this example from RemoteNodeRestartDeathWatchSpec, part of the multi-jvm tests for Akka.Remote (source):

"RemoteNodeRestartDeathWatch" must {

"receive Terminated when remote actor system is restarted" taggedAs LongRunningTest in {
  runOn(first) {
    val secondAddress = node(second).address
    enterBarrier("actors-started")

    val subject = identify(second, "subject")
    watch(subject)
    subject ! "hello"
    expectMsg("hello")
    enterBarrier("watch-established")

    // simulate a hard shutdown, nothing sent from the shutdown node
    testConductor.blackhole(second, first, Direction.Send).await
    testConductor.shutdown(second).await

    expectTerminated(subject, 15.seconds)

    within(5.seconds) {
      // retry because the Subject actor might not be started yet
      awaitAssert {
        system.actorSelection(RootActorPath(secondAddress) / "user" / "subject") ! "shutdown"
        expectMsg(1.second, "shutdown-ack")
      }
    }
  }

  runOn(second) {
    val addr = system.asInstanceOf[ExtendedActorSystem].provider.getDefaultAddress
    system.actorOf(Props[Subject], "subject")
    enterBarrier("actors-started")

    enterBarrier("watch-established")

    Await.ready(system.whenTerminated, 30.seconds)

    val freshSystem = ActorSystem(system.name, ConfigFactory.parseString(s"""
                akka.remote.netty.tcp {
                  hostname = ${addr.host.get}
                  port = ${addr.port.get}
                }
                """).withFallback(system.settings.config))
    freshSystem.actorOf(Props[Subject], "subject")

    Await.ready(freshSystem.whenTerminated, 30.seconds)
  }

}

The runOn method makes to easy to encapsulate test behavior on any of the ActorSystem instances participating in this particular multinode test.

What's important to note is that the multi-node tests don't actually run multiple processes - everything happens inside a single test process, but multiple actor systems are created and communicate via the TestConductor and the ServerFSM, which both receive directions from the Controller. The runOn method just ensures that both of these methods are executed concurrently on separate threads and belong to roles that have been previously defined for this test.

[Aaronontheweb]

@smalldave what changes do we need to make to Helios? I took a look at the RemoteConnection piece yesterday and thought it could be done with what we have out of the box - we do need to write a ProtobufSerializer and Deserializer for these wire types, however.

[smalldave]

I'm beginning to doubt this is right but I thought I saw the throttling (also used to black hole messages) being configured against netty. Thinking about it it's more likely that it's done in akka remoting. Will double check tomorrow.

[Aaronontheweb]

@smalldave yes, I think that and the FailureInjection stuff are both done via Akka.Remote adapters.