Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Application mode for long running workers #1082

Merged
merged 4 commits into from
Aug 28, 2021

Conversation

nbauernfeind
Copy link
Member

@nbauernfeind nbauernfeind commented Aug 18, 2021

New Flags:

  -Ddeephaven.console.disable   # prevents any user from accessing any repl session
  -Ddeephaven.application.dir=/path/to/app.d  # instruct the worker where to find configured applications

Due to the restriction that we have no more than one deephaven script session, python and groovy applications cannot be mixed at this time. Note that multiple python (or groovy) scripts share the same query scope which bleed into each other, and into the query scope available to a REPL user. The deephaven.console.disabled flag is handy here, but the user is also welcome to encapsulate any state they wish does not leak to the REPL. The variety of options here yield a ton of flexibility.

Supportive Functionality:

Bug Fixes:

  • Fixes Support canceling tickets instead of just releasing them #952: release proactively cancels; there is otherwise a race between steps 2 and 3 depending on network latency and out-of-ordering of messages. 1) export e0, 2) export e1 depends on e0, 3) release e0. Instead, the client would need to wait until it sees that all depending exports are PENDING before releasing e0. With a proactive cancel, the client still has to keep track of the same state, but cannot release until all depending exports are EXPORTED. I've introduced a ReleaseRequest in case we want to add-back the "smart-release" functionality --- but better the client gets bit by the proactive cancel and quickly realizes they need to track all of the export state changes.
  • wrap new console sessions in a delegating script session that upgrades modified query scope variables to be "new query scope" variables so that the UI may better distinguish multi-window scenarios where a query scope field is overwritten

Sample Groovy Application

File: /data/app.d/01-groovy.app

type=script
scriptType=groovy
enabled=true
id=io.deephaven.db.appmode.ApplicationConfigs.app01
name=My Groovy Application
file_0=01-groovy.groovy
file_1=01-groovy-qs.groovy

File: /data/app.d/01-groovy.groovy
Description: use ApplicationContext and lambdas to encapsulate your application state

import io.deephaven.appmode.ApplicationContext
import io.deephaven.appmode.ApplicationState
import io.deephaven.db.tables.utils.TableTools

def start = { ApplicationState app ->
    size = 42
    app.setField("hello", TableTools.emptyTable(size))
    app.setField("world", TableTools.timeTable("00:00:01"))
}

ApplicationContext.initialize(start)

File: /data/app.d/01-groovy-qs.groovy
Description: you can leak content via the query scope

import io.deephaven.db.tables.utils.TableTools

// Use QueryScope! Careful; this leaks into the REPL state!
size = 42
hello = TableTools.emptyTable(size)
world = TableTools.timeTable("00:00:01")

Sample Static Class Application

File: /data/app.d/00-static.app

type=static
class=io.deephaven.db.appmode.ApplicationConfigs$App00

Class Impl:

    public static class App00 implements Application.Factory {

        @Override
        public final Application create() {
            Field<Table> hello = StandardField.of("hello", TableTools.emptyTable(42).view("I=i"), "A table with one column 'I' and 42 rows, 0-41.");
            Field<Table> world = StandardField.of("world", TableTools.timeTable("00:00:01"));
            return Application.builder()
                    .id(App00.class.getName())
                    .name("My Class Application")
                    .fields(Fields.of(hello, world))
                    .build();
        }
    }

Note that the DynamicApplication can add/remove fields during the runtime of the application, and is just as easy an API to use.

@nbauernfeind nbauernfeind force-pushed the nbauernfeind_app_mode branch from f14706b to 0c49b53 Compare August 19, 2021 19:03
/*
* A lightweight object describing the exposed field.
*/
message FieldInfo {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mofojed any chance you want an icon field here? easy to add later, but to differentiate different plots, different "custom" things.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, why isn't this a table? wouldnt it make future filtering easier, sorting, eventual access control...

@@ -0,0 +1,7 @@
import io.deephaven.appmode.ApplicationContext
import io.deephaven.db.tables.utils.TableTools
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be unnecessary, right? or are we not doing "free" imports any more?

probably should write the kind of code we expect to have users write.

--

also, would it not make sense to have the AppContext import and app = AppContext.get() binding already done for you for these scripts? since any script that does anything interesting will need to do it anyway..

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to get rid of "free" imports here where we have the chance to do something.

Also, I don't like implicit bindings that scripts just "know" about. Often times, I'd see PQs at customers where they would do:

db = (Database)db

just so they would have strong typing. I'd prefer to create a pattern where scripts are explicit.

Alternatively, we can have a pattern where instead of a "script" the scripts provide a function with a known signature:

def script_eval(app):
   ...

But that feels a bit clunkier.

@nbauernfeind nbauernfeind force-pushed the nbauernfeind_app_mode branch from 495e61c to ace7de6 Compare August 20, 2021 19:33
@nbauernfeind nbauernfeind changed the title WIP: application mode Application mode for long running workers Aug 20, 2021
@nbauernfeind nbauernfeind marked this pull request as ready for review August 20, 2021 21:58
@nbauernfeind nbauernfeind force-pushed the nbauernfeind_app_mode branch 2 times, most recently from caff216 to 8149f0c Compare August 26, 2021 18:41
diff.created.put(name, toType);
} else if (toValue == null) {
diff.removed.put(name, fromType);
} else if (fromType != toType) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is exact equality desired? or .equals(...)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. I prefer exact equality. We're trying to detect changes that have occurred via black-box checking around the two sets of existing items in the query scope. If you swap from instance A to instance B even if A.equals(B) I think there is value to propagate a modification.

With deep equality it's not too hard to fabricate an example that will chew up lots of CPU time checking for equality. For example, an immutable linked list that gets modified only near the tail would yield terrible performance.

}
}
interface Listener {
void onScopeChanges(ScriptSession scriptSession, Changes changes);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we think explicitly passing ScriptSession is necessary? I might imagine that the entity creating the listener will have the listener scoped appropriately.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have one listener injected via Dagger. Not one per script session. Let's ignore the single global script session in the context of this API, please.

Comment on lines 17 to 19
void onNewField(ApplicationState app, Field<?> field);

void onRemoveField(ApplicationState app, String name);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the listeners need to be passed the state?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The listener is provided to the application state. Yes; I want to be able to dynamically update the application state and have it propagate through the system.

Am I misunderstanding the question?

import java.util.concurrent.locks.ReentrantLock;

@Singleton
public class ApplicationServiceGrpcImpl extends ApplicationServiceGrpc.ApplicationServiceImplBase
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced we are handling all edge cases w/ this impl without more review time - continuing on for now.

@nbauernfeind nbauernfeind force-pushed the nbauernfeind_app_mode branch 6 times, most recently from 4c2c5ba to b783207 Compare August 28, 2021 00:59
@nbauernfeind nbauernfeind force-pushed the nbauernfeind_app_mode branch from 1aaa5a3 to 9c11ec6 Compare August 28, 2021 14:45
@nbauernfeind nbauernfeind force-pushed the nbauernfeind_app_mode branch 2 times, most recently from 9f9ca26 to 634c47a Compare August 28, 2021 15:31
@nbauernfeind nbauernfeind force-pushed the nbauernfeind_app_mode branch from 634c47a to 29e5965 Compare August 28, 2021 15:36
@nbauernfeind nbauernfeind merged commit e0d85c4 into deephaven:main Aug 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment