Use Seqera containers in pipeline #36

drpatelh · 2024-05-28T18:15:14Z

Added a seqera_containers profile to build and push containers to community registry via Wave
Bumped minimum NF version required by pipeline to 24.04.0
Noticed an important discrepancy when using the conda declaration within a process vs a separate environment.yml (like was updated on nf-core recently for all modules e.g. fastqc). The latter impacts the name of the container being built as NF will use the name entry in the YAML as opposed to the package names in the conda declaration. That makes sense, but it will make it trickier to reuse containers with the same packages across modules. I have always preferred having an explicit conda definition in the process itself to avoid having to open numerous files to figure out software versions. Something worth looking into before rolling this out on nf-core more widely.
Removed --singularity_pull_docker_container for Wave to implicitly handle pulling the correct container

…ty registry

drpatelh · 2024-05-28T18:21:38Z

I ran the command below within the pipeline root to build and push the containers to the community registry:

$ nextflow inspect . -profile test,seqera_containers -concretize

{
    "processes": [
        {
            "name": "NF_AGGREGATE:SEQERA_RUNS_DUMP",
            "container": "community.wave.seqera.io/library/tower-cli:0.9.2--dc544a7e574ab73b"
        },
        {
            "name": "NF_AGGREGATE:MULTIQC",
            "container": "community.wave.seqera.io/library/multiqc:1.21--95b60bb51371925c"
        },
        {
            "name": "NF_AGGREGATE:PLOT_RUN_GANTT",
            "container": "community.wave.seqera.io/library/click_pandas_plotly_express_typing:21adb9e2d1b605a5"
        }
    ]
}

However, I can't pull all of the containers:

✅ docker pull community.wave.seqera.io/library/multiqc:1.21--95b60bb51371925c
❌ docker pull community.wave.seqera.io/library/tower-cli:0.9.2--dc544a7e574ab73b
❌ docker pull community.wave.seqera.io/library/click_pandas_plotly_express_typing:21adb9e2d1b605a5

How can I check if these containers have been built or am I doing something wrong?

…ow.config

Co-authored-by: Esha Joshi <[email protected]>

ewels · 2024-05-28T18:52:06Z

nextflow.config

-// Set default registry for Apptainer, Docker, Podman and Singularity independent of -profile
-// Will not be used unless Apptainer / Docker / Podman / Singularity are enabled
-// Set to your registry if you have a mirror of containers
-apptainer.registry   = 'quay.io'


We could set these to community.wave.seqera.io if we want think. For the same reasons we added quay.io here in the first place.

community.wave.seqera.io/library/tower-cli:0.9.2--dc544a7e574ab73b

What does the library mean in the URI? Will this always be the same for different registries? So do we need:

apptainer.registry = 'community.wave.seqera.io' docker.registry = 'community.wave.seqera.io' podman.registry = 'community.wave.seqera.io' singularity.registry = 'community.wave.seqera.io'

or this:

apptainer.registry = 'community.wave.seqera.io/library' docker.registry = 'community.wave.seqera.io/library' podman.registry = 'community.wave.seqera.io/library' singularity.registry = 'community.wave.seqera.io/library'

or none of these based on @pditommaso comment?

You don't need to use container with Community registry, hence you don't need registry either

The reason for including this is so people can quickly repoint to their own registry if they are using freeze, i.e. set all of these to myprivateecr.registry.io. On the other hand, they can just do this themselves and disregard all the community containers 🤔 . I don't think we ever worked out the developer workflow here.

@drpatelh you should do either 1 of two things here:

docker.registry = 'community.wave.seqera.io/library' and the container directive is container 'click_pandas_plotly_express_typing:21adb9e2d1b605a5'

Do not set docker.registry and use the directivecontainer 'community.wave.seqera.io/library/click_pandas_plotly_express_typing:21adb9e2d1b605a5'

The docker.registry is added as a prefix if it does not exist in the URI of the container already (i.e. it's idempotent). What you have done here is fine but redundant and the registry config option will be ignored.

p.s. not certain about library, this depends on Wave + ECR behaviour.

pditommaso · 2024-05-28T21:11:43Z

modules/local/plot_run_gantt/main.nf

@@ -2,7 +2,7 @@ process PLOT_RUN_GANTT {
    tag "$meta.id"

    conda 'click=8.0.1 pandas=1.1.5 plotly_express=0.4.1 typing=3.10.0.0'
-    container 'seqeralabs/nf-aggregate:click-8.0.1_pandas-1.1.5_plotly_express-0.4.1_typing-3.10.0.0--ccea219dc6c3d6a1'
+    container 'community.wave.seqera.io/library/click_pandas_plotly_express_typing:21adb9e2d1b605a5'


NOOOOOOOOOO

We made this to have containers made via the Conda packages declaration. Why bumping the community container image?

Have we figured out how to run these pipelines in offline environments? Is this going to be inherently taken care of by nf-core download and not handled by NF/Wave anymore @ewels ?

No - if we want to run offline then we cannot use Wave.

The only way that I can see around this is to have the Nextflow Wave plugin create the conda + dockerfile locally and from that figure out what the image name should be. Then this could be checked against the local docker / singularity cache without doing any calls to the internet. But I've spoken to @pditommaso about this already and I think that it was difficult / impossible.

So until we figure that out, I can't see any way to use Wave directly in nf-core pipelines, as it will break all offline usage. So the hardcoded docker + singularity URIs have to stay.

We can of course use Wave to build those images and so my hope is that the developer can specify just the conda requirements and then the automation will pin the image names into a config without human intervention.

If anyone has any better ideas, shout!

We are near to have nextflow inspect to report the HTTP uri of the Singularity image files. Would not that allow handling them in the same manner as it's done for Biocontainers right now?

The only way that I can see around this is to have the Nextflow Wave plugin create the conda + dockerfile locally and from that figure out what the image name should be.

Is it possible to use the hash and check for existing containers before hitting Wave? This doesn't help with offline usage, but does bypass Wave when it isn't needed. Furthermore, it could check for existing Singularity containers for some offline use right now. I'm assuming the hash in the filename is complete enough to do this check which seems like quite a large assumption.

We can of course use Wave to build those images and so my hope is that the developer can specify just the conda requirements and then the automation will pin the image names into a config without human intervention.

I assume this is the end goal but don't see how we'd get there and still include offline support?

Currently the workflow for nf-core would be essentially:

play with 'interactive' Wave + Conda for development

use https://seqera.io/containers/ to build the 'static' container and grab the URI

add this to process container directive(s)

open PR to nf-core/modules

Is it possible to use the hash and check for existing containers before hitting Wave?

Wave generates the hash, so no

OK, we'll have to stick with process.container for now.

drpatelh · 2024-05-29T07:35:11Z

nextflow.config

+        apptainer.ociAutoPull   = true
+        singularity.ociAutoPull = true
+        wave.enabled            = true
+        wave.freeze             = true
+        wave.strategy           = 'conda'


If we are using Wave behind the scenes to get the container URI, do I need to move these out of a profile and into the top-level config so they are always on?

It's keep into a separate profile

drpatelh · 2024-05-29T07:38:29Z

❌ docker pull community.wave.seqera.io/library/tower-cli:0.9.2--dc544a7e574ab73b
❌ docker pull community.wave.seqera.io/library/click_pandas_plotly_express_typing:21adb9e2d1b605a5

I also still don't know why these containers weren't created 😏

pditommaso · 2024-05-29T08:16:45Z

nextflow.config

+        apptainer.ociAutoPull   = true
+        singularity.ociAutoPull = true


Suggested change

apptainer.ociAutoPull = true

singularity.ociAutoPull = true

Not needed. Those forces the use of Docker image what's converted by Singularity

ewels · 2024-05-29T10:08:23Z

❌ docker pull community.wave.seqera.io/library/tower-cli:0.9.2--dc544a7e574ab73b
❌ docker pull community.wave.seqera.io/library/click_pandas_plotly_express_typing:21adb9e2d1b605a5

I also still don't know why these containers weren't created 😏

If you have your tower auth token set locally then you should get emails when you run nextflow inspect, potentially with error messages for these two.

For now that's the only method I know of to be able to figure out the build status.

adamrtalbot · 2024-05-29T14:39:56Z

❌ docker pull community.wave.seqera.io/library/tower-cli:0.9.2--dc544a7e574ab73b
❌ docker pull community.wave.seqera.io/library/click_pandas_plotly_express_typing:21adb9e2d1b605a5

I also still don't know why these containers weren't created 😏

I was able to generate the tower-cli one but I get a different hash: community.wave.seqera.io/library/tower-cli:0.9.2--28258d337ec30808

drpatelh added 2 commits May 28, 2024 19:09

Add seqera_containers profile to build and push containers to communi…

787a382

…ty registry

Use community containers created by Wave in pipeline modules

e992f13

ejseqera and others added 10 commits May 28, 2024 19:25

fix: remove escape from tower env var

1bb1679

Update CHANGELOG

b877955

Add note in README for Gantt plotting with Fusion

2e18ac5

Bump pipeline version to 0.2.0

a77e665

Remove unused params.pipelines_testdata_base_path from nf-test nextfl…

362f4b0

…ow.config

Tidy up filtering syntax before PLOT_RUN_GANTT process

edeffd9

Update README.md

9e58e0f

Co-authored-by: Esha Joshi <[email protected]>

Update CHANGELOG

82f8b87

Merge branch 'dev' into seqera_containers

ada8317

Fix merge conflicts

31560c5

ewels reviewed May 28, 2024

View reviewed changes

pditommaso reviewed May 28, 2024

View reviewed changes

drpatelh commented May 29, 2024

View reviewed changes

pditommaso reviewed May 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Seqera containers in pipeline #36

Use Seqera containers in pipeline #36

drpatelh commented May 28, 2024 •

edited

Loading

drpatelh commented May 28, 2024 •

edited

Loading

ewels May 28, 2024

drpatelh May 29, 2024

pditommaso May 29, 2024

adamrtalbot May 29, 2024 •

edited

Loading

adamrtalbot May 29, 2024

pditommaso May 28, 2024

drpatelh May 28, 2024

pditommaso May 29, 2024

drpatelh May 29, 2024 •

edited

Loading

ewels May 29, 2024

pditommaso May 29, 2024

adamrtalbot May 29, 2024 •

edited

Loading

ewels May 30, 2024

adamrtalbot May 30, 2024

drpatelh May 29, 2024

pditommaso May 29, 2024

drpatelh commented May 29, 2024

pditommaso May 29, 2024

ewels commented May 29, 2024

adamrtalbot commented May 29, 2024

Use Seqera containers in pipeline #36

Are you sure you want to change the base?

Use Seqera containers in pipeline #36

Conversation

drpatelh commented May 28, 2024 • edited Loading

drpatelh commented May 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamrtalbot May 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drpatelh May 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamrtalbot May 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drpatelh commented May 29, 2024

Choose a reason for hiding this comment

ewels commented May 29, 2024

adamrtalbot commented May 29, 2024

drpatelh commented May 28, 2024 •

edited

Loading

drpatelh commented May 28, 2024 •

edited

Loading

adamrtalbot May 29, 2024 •

edited

Loading

drpatelh May 29, 2024 •

edited

Loading

adamrtalbot May 29, 2024 •

edited

Loading