Application mode for long running workers

[nbauernfeind]

New Flags:

  -Ddeephaven.console.disable   # prevents any user from accessing any repl session
  -Ddeephaven.application.dir=/path/to/app.d  # instruct the worker where to find configured applications

Due to the restriction that we have no more than one deephaven script session, python and groovy applications cannot be mixed at this time. Note that multiple python (or groovy) scripts share the same query scope which bleed into each other, and into the query scope available to a REPL user. The deephaven.console.disabled flag is handy here, but the user is also welcome to encapsulate any state they wish does not leak to the REPL. The variety of options here yield a ton of flexibility.

Supportive Functionality:

• Fixes gRPC API for fetching various objects by name from a scope #103: ExportFromTicket enables a ticketRouter.resolve(source_ticket) and export into a session managed result_ticket (this is the modern version of fetch)
• Fixes Add TableService getExportedTableCreationResponse gRPC #923: support get ETCR from a tickerRouter.resolve(source_ticket)
• Fixes Provide list of objects in the existing session when a client connects #854: provide list of fields in query scope if not in console disabled mode, provide list of application fields if Application.dir flag supplied
• Fixes Provide a command line argument to a script to seed the application #853: provide command line flags (and other configuration) to support application mode
• The listFields() call subscribes to future updates, too.
• Added ExportNotification.State.RUNNING to distinguish between queued and executing.

Bug Fixes:

• Fixes Support canceling tickets instead of just releasing them #952: release proactively cancels; there is otherwise a race between steps 2 and 3 depending on network latency and out-of-ordering of messages. 1) export e0, 2) export e1 depends on e0, 3) release e0. Instead, the client would need to wait until it sees that all depending exports are PENDING before releasing e0. With a proactive cancel, the client still has to keep track of the same state, but cannot release until all depending exports are EXPORTED. I've introduced a ReleaseRequest in case we want to add-back the "smart-release" functionality --- but better the client gets bit by the proactive cancel and quickly realizes they need to track all of the export state changes.
• wrap new console sessions in a delegating script session that upgrades modified query scope variables to be "new query scope" variables so that the UI may better distinguish multi-window scenarios where a query scope field is overwritten

Sample Groovy Application

File: /data/app.d/01-groovy.app

type=script
scriptType=groovy
enabled=true
id=io.deephaven.db.appmode.ApplicationConfigs.app01
name=My Groovy Application
file_0=01-groovy.groovy
file_1=01-groovy-qs.groovy

File: /data/app.d/01-groovy.groovy
Description: use ApplicationContext and lambdas to encapsulate your application state

import io.deephaven.appmode.ApplicationContext
import io.deephaven.appmode.ApplicationState
import io.deephaven.db.tables.utils.TableTools

def start = { ApplicationState app ->
    size = 42
    app.setField("hello", TableTools.emptyTable(size))
    app.setField("world", TableTools.timeTable("00:00:01"))
}

ApplicationContext.initialize(start)

File: /data/app.d/01-groovy-qs.groovy
Description: you can leak content via the query scope

import io.deephaven.db.tables.utils.TableTools

// Use QueryScope! Careful; this leaks into the REPL state!
size = 42
hello = TableTools.emptyTable(size)
world = TableTools.timeTable("00:00:01")

Sample Static Class Application

File: /data/app.d/00-static.app

type=static
class=io.deephaven.db.appmode.ApplicationConfigs$App00

Class Impl:

    public static class App00 implements Application.Factory {

        @Override
        public final Application create() {
            Field<Table> hello = StandardField.of("hello", TableTools.emptyTable(42).view("I=i"), "A table with one column 'I' and 42 rows, 0-41.");
            Field<Table> world = StandardField.of("world", TableTools.timeTable("00:00:01"));
            return Application.builder()
                    .id(App00.class.getName())
                    .name("My Class Application")
                    .fields(Fields.of(hello, world))
                    .build();
        }
    }

Note that the DynamicApplication can add/remove fields during the runtime of the application, and is just as easy an API to use.

[niloc132]

@mofojed any chance you want an icon field here? easy to add later, but to differentiate different plots, different "custom" things.

[niloc132]

Also, why isn't this a table? wouldnt it make future filtering easier, sorting, eventual access control...

[niloc132]

this should be unnecessary, right? or are we not doing "free" imports any more?

probably should write the kind of code we expect to have users write.

--

also, would it not make sense to have the AppContext import and app = AppContext.get() binding already done for you for these scripts? since any script that does anything interesting will need to do it anyway..

[devinrsmith]

I'd like to get rid of "free" imports here where we have the chance to do something.

Also, I don't like implicit bindings that scripts just "know" about. Often times, I'd see PQs at customers where they would do:

db = (Database)db

just so they would have strong typing. I'd prefer to create a pattern where scripts are explicit.

Alternatively, we can have a pattern where instead of a "script" the scripts provide a function with a known signature:

def script_eval(app):
   ...

But that feels a bit clunkier.

[devinrsmith]

is exact equality desired? or .equals(...)?

[nbauernfeind]

Hmm. I prefer exact equality. We're trying to detect changes that have occurred via black-box checking around the two sets of existing items in the query scope. If you swap from instance A to instance B even if A.equals(B) I think there is value to propagate a modification.

With deep equality it's not too hard to fabricate an example that will chew up lots of CPU time checking for equality. For example, an immutable linked list that gets modified only near the tail would yield terrible performance.

[devinrsmith]

Do we think explicitly passing ScriptSession is necessary? I might imagine that the entity creating the listener will have the listener scoped appropriately.

[nbauernfeind]

I have one listener injected via Dagger. Not one per script session. Let's ignore the single global script session in the context of this API, please.

[devinrsmith]

Do the listeners need to be passed the state?

[nbauernfeind]

The listener is provided to the application state. Yes; I want to be able to dynamically update the application state and have it propagate through the system.

Am I misunderstanding the question?

[devinrsmith]

I'm not convinced we are handling all edge cases w/ this impl without more review time - continuing on for now.