Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add archiving flow #15

Closed
10 of 12 tasks
Tracked by #27 ...
phwissmann opened this issue Feb 20, 2024 · 1 comment · Fixed by #40
Closed
10 of 12 tasks
Tracked by #27 ...

[Feature] Add archiving flow #15

phwissmann opened this issue Feb 20, 2024 · 1 comment · Fixed by #40
Assignees
Milestone

Comments

@phwissmann
Copy link
Collaborator

phwissmann commented Feb 20, 2024

Goal:
Have a pipeline establish that executes the required jobs that implement the archiving functionality and notifications of job and dataset status updates shall be forwarded reliably (i.e. retries) to SciCat.
Any actual interactions with the dataset shall only be mocked for testing

Tasks:

  • Add sequence of task flow
  • Verify SciCat interactions and request data with Go through Scicat Interactions w. Spencer #41
  • Verify correct error handling and updates of SciCat jobs and dataset lifecycle
    - [ ] Add logging and notifications of task failures
  • Handle multiple datasets in same job
    Smaller tasks:
  • Add worker pool to job system moved to different milestone
  • Add proper config parsing
  • Add table of job and dataset lifecycle: blocked
  • add fetch of origdatablocks
  • handle return of scicat requests properly
  • add retry handling of scicat request
  • add cleanup step for staging at the end
  • split config of backend and worker

Definition of done:
Test suite is passing with all the defined interactions implemented.

@phwissmann phwissmann mentioned this issue Feb 20, 2024
11 tasks
@phwissmann phwissmann self-assigned this Feb 20, 2024
@phwissmann
Copy link
Collaborator Author

@phwissmann phwissmann changed the title Add archive job Add archiving pipeline Apr 4, 2024
@phwissmann phwissmann linked a pull request Apr 19, 2024 that will close this issue
@phwissmann phwissmann changed the title Add archiving pipeline Add archiving flow Apr 30, 2024
@phwissmann phwissmann added this to the MVP milestone May 2, 2024
@phwissmann phwissmann changed the title Add archiving flow [Feature] Add archiving flow May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant