This repository was archived by the owner on Nov 30, 2022. It is now read-only.
Support for pausing an execution (Access Requests) #521
Assignees
Labels
enhancement
New feature or request
Comments
This was referenced May 23, 2022
seanpreston
pushed a commit
that referenced
this issue
May 26, 2022
* WIP Add structure for pausing/resuming a privacy request. * Allow run_access_request to start from a paused state. * WIP work on privacy request resume endpoint. * Fix downrev. * Restore functions back to main method. * - Fix data type for manual results: Should be a list of Rows. - Fix input into start_function, so it can be used for both identity nodes, and for rebuilding a DAG with previously cached nodes * Add manual api endpoints to postman collection. * Assert customer data in graph too. * Fix downrev. * Close the session later. * Have the postgres payment card collection also depend on upstream manual input. Now the manual node both needs upstream input, as well as being a dependency for downstream nodes. * Add paused status for individual execution logs. * Adds a method that can take in the same dictionary we hand to Dask and build a visualization from it. * Verify that nodes that have already run prior to retrying a paused request are not run again on retry. * Add a second manual node to the graph. * Add validation to manual input to ensure it matches fields configured on the dataset. * Skip running pre policy webhooks if we're resuming from the middle of the graph. * Use box_id as needed user data. * Verify that a privacy request can be resumed from "pause" with an empty list. * Instead of caching the paused node, also cache whether we are paused on the "access" portion of the request or the "erasure" portion of the request, as an erasure will run both. * Run dask delayed single-threaded. * Universally refer to the cached step (access or erasure) with an ActionType instead of a string. Add tests for the new privacy request caching methods. * Remove setting other schedulers. * Pull from the cache by default when running an access request if data exists (should just be on restart). - Close the session immediately after the privacy request is paused. - Rerun the privacy request with a new privacy request id because the test is testing an an unusual situation. - Address a lot of the linting issues. * Fix bug in update_mapping_from_cache. * Add another manual collection to dataset. * Fix sorting. * Fix type. * Cleanup variable names and docstrings. Remove graphviz for its extra installation time. * Add a first pass of docs. * update manual dataset docs * fix typo * Respond to type changes from Paul's CR. * Add a bit about how the dataset config should be added to the connectionconfig. * Add missed items from CR, I don't need these fixture arguments. * Have downrev drop paused execution logs. * manual data docs cleanup * Update privacy request status and update return type. * Fix test with new error message. Co-authored-by: Cole Isaac <[email protected]>
sanders41
pushed a commit
that referenced
this issue
Sep 22, 2022
* WIP Add structure for pausing/resuming a privacy request. * Allow run_access_request to start from a paused state. * WIP work on privacy request resume endpoint. * Fix downrev. * Restore functions back to main method. * - Fix data type for manual results: Should be a list of Rows. - Fix input into start_function, so it can be used for both identity nodes, and for rebuilding a DAG with previously cached nodes * Add manual api endpoints to postman collection. * Assert customer data in graph too. * Fix downrev. * Close the session later. * Have the postgres payment card collection also depend on upstream manual input. Now the manual node both needs upstream input, as well as being a dependency for downstream nodes. * Add paused status for individual execution logs. * Adds a method that can take in the same dictionary we hand to Dask and build a visualization from it. * Verify that nodes that have already run prior to retrying a paused request are not run again on retry. * Add a second manual node to the graph. * Add validation to manual input to ensure it matches fields configured on the dataset. * Skip running pre policy webhooks if we're resuming from the middle of the graph. * Use box_id as needed user data. * Verify that a privacy request can be resumed from "pause" with an empty list. * Instead of caching the paused node, also cache whether we are paused on the "access" portion of the request or the "erasure" portion of the request, as an erasure will run both. * Run dask delayed single-threaded. * Universally refer to the cached step (access or erasure) with an ActionType instead of a string. Add tests for the new privacy request caching methods. * Remove setting other schedulers. * Pull from the cache by default when running an access request if data exists (should just be on restart). - Close the session immediately after the privacy request is paused. - Rerun the privacy request with a new privacy request id because the test is testing an an unusual situation. - Address a lot of the linting issues. * Fix bug in update_mapping_from_cache. * Add another manual collection to dataset. * Fix sorting. * Fix type. * Cleanup variable names and docstrings. Remove graphviz for its extra installation time. * Add a first pass of docs. * update manual dataset docs * fix typo * Respond to type changes from Paul's CR. * Add a bit about how the dataset config should be added to the connectionconfig. * Add missed items from CR, I don't need these fixture arguments. * Have downrev drop paused execution logs. * manual data docs cleanup * Update privacy request status and update return type. * Fix test with new error message. Co-authored-by: Cole Isaac <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
###Description
Our current means of executing DAGs need to be enhanced to support scalability, reliability, and additional use cases when it comes to Subject Request processing (like pausing a request, cancelling a request, etc). The scope of this ticket is focused on adding a node that can pause an ACCESS request, take in manual input for example, and resume the request with control from the user.
We may need to split out the resume endpoint into another ticket or PR, but leaving it in scope for now.
The text was updated successfully, but these errors were encountered: