Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge live gupu #77

Merged
merged 136 commits into from
Feb 26, 2025
Merged

Merge live gupu #77

merged 136 commits into from
Feb 26, 2025

Conversation

AmstlerStephan
Copy link
Member

Prepare the first major commit of the pipeline.

  1. Added nf-test testing
  2. Added support for real-time (live) analysis of clusters per sample during sequencing
  3. Added GPU acceleration for medaka polishing
  4. Split the workflow into several subworkflows
  5. Implemented batching of input files for higher parallelization
  6. Implemented a process to glue molecule files before submission to medaka polishing
  • to decrease number of files to process, which can be up to several 100k for large sequencing experiments
  • to make better use of the GPU acceleration
  1. Added support for multi-line bed files to process several targets in one run
  2. Added simple plots and stats to keep track of the number of clusters per sample
  • CAVEAT: GPU acceleration supporting docker container is only available up to v2.0.0, in which medaka smolecule is broken. v2.0.0 uses the new pytorch models, and therefore, CPU medaka and GPU medaka results might differ slightly.

AmstlerStephan and others added 30 commits March 6, 2024 11:42
@AmstlerStephan
Merge pull request #62 from genepi/main
2628ab6 
update dev
@AmstlerStephan
init nf-test
fbf6e1f
@AmstlerStephan
add gitignore
3bb374f
@AmstlerStephan
adapt config files to use nf-test
64b6260
@AmstlerStephan
move test data
025ffa2 
create test for bed file
@AmstlerStephan
adapt test.config
d6d0d43 
add minimal testing data
output files to check processes
@AmstlerStephan
move testing data
bb6b685
@AmstlerStephan
add data and tests for the cluster filtering and reformating
4eabc2f
@AmstlerStephan
add further tests for reformat filter cluster
6a6ebb3 
and the main workflow
@AmstlerStephan
add snapshot to test main workflow
3b62f2b
@AmstlerStephan
no snapshots for workflows
a000151
@AmstlerStephan
move testing data
3bf22c6
@AmstlerStephan
Merge branch 'nf-test' of https://github.com/genepi/umi-pipeline-nf i…
9549bf7 
…nto nf-test
@AmstlerStephan
adapt module structure
8b471b3 
generate test freebayes
@AmstlerStephan
adapt module paths in main workflow
74543c6
@AmstlerStephan
add test data for variant calling
adc32dc
@AmstlerStephan
Add variant calling test data
248918b
@AmstlerStephan
Add test cases
5980ae9 
update to version 0.2.1
@AmstlerStephan
Add freebayes test cases
1bd50f4
@AmstlerStephan
Add tests for all variant callers
21bf952
@AmstlerStephan
a
2fbbb8e
@AmstlerStephan
adapt lofreq parameters to have no minimal coverage
4dbee61
@AmstlerStephan
remove typo
2f9aaf2 
update lofreq snapshot
@AmstlerStephan
Add fastq test files for merge input
170299f
@AmstlerStephan
Add tests and snapshot for merge_fastq
1171cb7
@AmstlerStephan
add test data for mapping
f7064c0
@AmstlerStephan
add tests for consensus and final consensus reads
4ca53b9 
update snapshot
@AmstlerStephan
Create test cases for split_reads
efe9a8e 
add test data
create snapshot
@AmstlerStephan
prepare tests for split reads process
b4c843b
@AmstlerStephan
prepare tests for umi detection
36b8944
AmstlerStephan and others added 28 commits February 20, 2025 16:08
@AmstlerStephan
Include filtering of samples with little reads.
e90692f 
Update README
@AmstlerStephan
Add cluster reports also to
f48c376
@AmstlerStephan
add missing input channels
a32af7a
@AmstlerStephan
only write stats when reads are found
11787bc
@AmstlerStephan
Add vsearch_sequence_identity param
718f871
@AmstlerStephan
Merge branch 'medaka_GPU' into Merge_live_GUPU
6281f39
@AmstlerStephan
add parameter
3b02af6 
readd plugin
@AmstlerStephan
remove custi
190930e
@AmstlerStephan
remove custi
98c500b
@AmstlerStephan
remove log parameter
f5ad04f
@AmstlerStephan
sort input files
999c0c8 
delete deprecated tests
@AmstlerStephan
Medaka v2.0.0 is broken for medaka smolecule
00a90a4 
the specified versiond supports the latest v5.0.0 models and should work with GPU
@AmstlerStephan
update snapshot and sort channel prior tests
cb88560
@AmstlerStephan
Adapt testing and update Snapshots
ebf7084 
Adapt maxForks for medaka
@AmstlerStephan
adapt config for polishing
356c2a8
@AmstlerStephan
include fastq plugin for nf-test
9fd9d6f
@AmstlerStephan
update snapshot
1037549
@AmstlerStephan
asynchronous GPU memory allocation
17437db
@AmstlerStephan
test setting env variables in container
58c3b55
@AmstlerStephan
setting env variables
38b9f11
@AmstlerStephan
add batch_size param to polish cluster
2e39412
@AmstlerStephan
Add multi line bed file support
de30210 
adapt output dirrctories accordingly
adapt cluster stat reports accordingly
@AmstlerStephan
Update tests for multiple line bed files
89aaa8e
@AmstlerStephan
Update tuples for right semantic
eddb3b4
@AmstlerStephan
Update testing for new container
3625d10
@AmstlerStephan
update last snapshot.
57aa942 
Unfortunately medaka gpu and medaka cpu are different versions.
As soon as a gpu supporting docker container with v2.0.1 is released the pipeline will be updated
@AmstlerStephan
Ready for merging
3c3469f
@AmstlerStephan
Merge branch 'main' into Merge_live_GUPU
7dcb578
@AmstlerStephan AmstlerStephan merged commit c3f9bfc into main Feb 26, 2025
0 of 10 checks passed
@AmstlerStephan AmstlerStephan deleted the Merge_live_GUPU branch February 26, 2025 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@AmstlerStephan