Slabs design

[thorwhalen]

There's a bit more history before the point I'm showing now, which I might describe another day. Now, I'll start at a point the allows us to see the basic ideas of the design.

class Slabs:
    def _call_on_scope(self, scope):
        '''Calls the components 1 by 1, sourcing inputs and writing outputs in scope'''

    def __next__(self):
        '''Get the next slab by calling _call_on_scope on an new empty scope.
        At least one of the components will have to be argument-less and provide
        some data for other components to get their inputs from, if any are needed.
        '''
        return self._call_on_scope(scope={})

    def __iter__(self):
        '''Iterates over slabs until a handle exception is raised.'''
        # Simplified code:
        with self:  # enter all the contexts that need to be entered
            while True:  # loop until you encounter a handled exception
                try:
                    yield next(self)
                except self.handle_exceptions as exc_val:
                    # use specific exceptions to signal that iteration should stop
                    break

handling exceptions

First, I'd like to note that "exception" doesn't mean "error", though that's is the typical kind of exception we see.

In the simple code above, we see that we can specify a collection of exception types that will lead to breaking out of the loop without a fuss (silently). We can use this to specify, in which conditions should stop the otherwise infinite iteration of the slabs without raising an error or otherwise bugging the user about it: For example, when there's a KeyBoardInterrupt or some other custom exception. This can be used by any of the components to "signal" to the iteration that we should quit.

For example, a component reads the state of a switch, and if the switch is off, it raises a SwitchIsOff exception, which will then have the effect of exiting the loop.

There's the main idea: We want to be able to use exceptions to control the (normal) flow described by the mesh of our components.
There's more to an iteration's life than quitting though.
That's why we need more. Consider the following evolution of the exception block.

    ...
                except tuple(self.handle_exceptions) as exc_val:
                    handler_output = _handle_exception(
                        self, exc_val, self.handle_exceptions
                    )
                    # break, unless the handler tells us not to
                    if handler_output is not do_not_break:
                        self.exit_value = handler_output  # remember, in case useful
                        break
     ...

Note that here, self.handle_exceptions is no longer a tuple, but a dict whose keys are exception types and the values are exception_handler callback functions that should be called if an exception of that type happens. For details on exception type matching (it's more complicated than you think) and what the callback function has visibility on, see the _handle_exception function.

Here's an example of use. Your slabs are there to compute some really smart and energy-consuming stuff, but only if the light switch is on. One way to achieve this is to have all components take as an input the light switch state and condition their execution on that. Compare that too simply having an early component look at the state of the switch, and if off, raise a SwitchIsOff exception with an associated handler that waits a bit (to not get right back in the loop) and returns the do_not_break sentinel to get back in the loop.

Still, we can do more. Note that in the code above:

• There is no reentry, unless you hack it via the exception handler (which has access to the slabs instance) -- but I'm not sure that's clean or not. ** <-- DISCUSS**
• The default is still to break out of the loop. A handler needs to return the do_not_break sentinel to specify otherwise. Is this the right default? Should we add an init argument so that a user can control this ** <-- DISCUSS**

execution

The _call_on_scope is akin to meshed.DAG.call_on_scope (which itself uses FuncNode.call_on_scope): The sequential execution of functions, reading inputs from and writing outputs to the scope (a typing.MutableMapping). Not that there are other choices (which we'll hopefully explore at some point soon) of execution. We can use concurrent execution instead of sequential, or even use the DAG's structure as a declarative, as opposed to an imperative, specification. We could interpret the DAG's structure as the specification of reactive components, for instance.

scope

Note that a lot can be done by specifying a custom scope (which acts as the locals() dictionary of "normal" python execution).
For example, one can have

• self-deleting key-value pairs that offload data when it is no longer needed
• callback values that trigger data pulls when the value of a key is requested
• all kinds of caching/persistence logic
• using a different backend behind the MutableMapping, such as a message-bus or http service
  In DAG, the default scope is a dict, constructed afresh on every call, but you can specify the scope at call time, or specify a scope factory that it will use to make a new scope on every call.

Initially (as seen above) we didn't have that in Slabs, so we changed __next__ to:

    def __next__(self):
        return self._call_on_scope(scope=self.scope_factory())

where scope_factory is given at initiation.

[thorwhalen]

Slabs setup and teardown

Recently, I extended Slabs to include some finally functionality.
We've had exception handling for awhile, which enables us to control the except block in our loop:

     with self:  # enter all the contexts that need to be entered
            while True:  # loop until you encounter a handled exception
                try:
                    yield next(self)
                except ...

But there was no hook to be able to specify the finally block of a try/except clause.

I thought I needed one, so I implemented one.

Removing finally functionality

I was wrong about my need though: What I needed wasn't a finally functionality in my loop, but a finally functionality outside my loop. Essentially a teardown functionality.

I will now remove the finally functionality I've implemented, before it makes babies, but am writing this to remember.

The biggest pro I can see for the finally functionality I've implemented is completeness.
Slabs offers a parametrized declarative way to make contextual computational loops with error handling, but the error handling would only be completely aligned with the normal try/except pythonic way with the finally clause.

That said, the con is extra complexity (before it was really necessary) as well as extra function calls in every loop, which could be detremental to real-time streaming operations.

So we'll leave it at that for now.

Might want a teardown functionality

That said, the functionality I actually needed -- a teardown -- is something that comes up in almost all Slabs use cases.
Usually, it goes like this:

• Get/make the resources you need
• Define the slab's components (functions that, namely, use the resources)
• Make the Slabs instance
• Run the Slabs instance
• clean-up (teardown) if needed

The first and the last of these tasks definitely fit the context manager setup/teardown pattern.
It may be useful to extend the current Slabs so that the user can have slots for these.

Note the current Slabs context management:

    def open(self):
        self._output_of_context_enter = self.context.__enter__()
        return self

    def close(self, exc_type=None, exc_val=None, exc_tb=None) -> None:
        return self._output_of_context_enter.__exit__(exc_type, exc_val, exc_tb)

where

        self.context = ContextFanout(**components)

So the only contributors to the context management are the components.
We may want to add an extra setup and teardown functionality.