Purge forms and associated resources #431
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ktuite
force-pushed
the
ktuite/purge-forms
branch
from
777f320 to
6ebeb28
Compare
ktuite
commented
Nov 16, 2021
ktuite
force-pushed
the
ktuite/purge-forms
branch
from
6ebeb28 to
777f61c
Compare
lib/model/migrations/20211027-01-add-purgeddAt-details-to-actees.js
Outdated
Show resolved
Hide resolved
ktuite
force-pushed
the
ktuite/purge-forms
branch
from
777f61c to
bb3258b
Compare
ktuite
commented
Nov 17, 2021
| const up = async (db) => { | ||
| await db.schema.table('form_defs', (t) => { | ||
| t.dropForeign('formId'); | ||
| t.foreign('formId').references('forms.id').onDelete('cascade'); |
There was a problem hiding this comment.
look into doing these with an alter instead of a drop if possible
lib/model/query/forms.js
Outdated
| const del = (form) => ({ run, Assignments }) => | ||
| Promise.all([ run(markDeleted(form)), Assignments.revokeByActeeId(form.acteeId) ]); | ||
| const del = (form) => ({ run, Assignments, Forms }) => | ||
| Promise.all([ run(markDeleted(form)), Assignments.revokeByActeeId(form.acteeId) ]) |
There was a problem hiding this comment.
consider just purging and not having to check it and mark deleted first
There was a problem hiding this comment.
might need to revoke by actee id still even if form purged without being deleted first
| and ${_trashedFilter(force, id)}`) | ||
| .then(() => run(sql` | ||
| insert into audits ("action", "acteeId", "loggedAt") | ||
| select 'form.purge', "acteeId", clock_timestamp() |
There was a problem hiding this comment.
form.purge verb? maybe doesn't need to be in any verb table, but i'll double check
| // 5. Purge unattached blobs | ||
| const purge = (force = false, id = null) => ({ run, all, Blobs }) => | ||
| run(sql` | ||
| update audits set notes = '' |
There was a problem hiding this comment.
take a stab at combining all these into a single quere
| // 3. by a specific form id if deletedAt is also set (again for testing or potential future scenarios) | ||
|
|
||
| const DAY_RANGE = 30; | ||
| const _trashedFilter = (force, id) => { |
There was a problem hiding this comment.
@issa-tseng's going to look up some resource injection stuff
ktuite
commented
Nov 18, 2021
lib/model/migrations/20211027-01-add-purgeddAt-details-to-actees.js
Outdated
Show resolved
Hide resolved
lib/model/query/forms.js
Outdated
| const del = (form) => ({ run, Assignments }) => | ||
| Promise.all([ run(markDeleted(form)), Assignments.revokeByActeeId(form.acteeId) ]); | ||
| const del = (form) => ({ run, Assignments, Forms }) => | ||
| Promise.all([ run(markDeleted(form)), Assignments.revokeByActeeId(form.acteeId) ]) |
There was a problem hiding this comment.
might need to revoke by actee id still even if form purged without being deleted first
ktuite
force-pushed
the
ktuite/purge-forms
branch
from
81eb8d5 to
8b32340
Compare
Rename actees deletedAt to purgedAt
rename opendatakit to getodk in migration
tests cleanup More form purge submission tests more tests: purging client audits tests about purging old/newly deleted forms
ktuite
force-pushed
the
ktuite/purge-forms
branch
from
384fbd5 to
61ef815
Compare
This was referenced Jan 21, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR enables the purging of soft-deleted forms and their associated resources (definitions, versions, submissions, attachments, comments, etc.)
Purging follows these steps:
As a side effect of deleting/purging Forms, we needed to address issue getodk/central#127 and decide what to display in the audit log for deleted resources. To do this, we added
purgedAt,purgedName,detailstoActeesto capture information about a permanently removed resources and then exposed this info via the audit log API.This PR has a number of high-impact migrations:
acteestablePurging is done through a task and command line script that will be called via cron job in the main Central repo. The script will be called daily to purge forms deleted over 30 days prior, but can also be used to purge a specific form by its numeric id.