Skip to content

Commit e0b2533

Browse files
adamrtalbotejseqera
adamrtalbot
and
ejseqera
authored
add on_exists param with behaviour fail, ignore and overwrite (#207)
This PR adds a feature where you can customize the behaviour when a resource exists. It has three options: - `fail` (default): Fail and halt seqerakit - `ignore`: Keep going and ignore the error - `overwrite`: Use the existing overwrite behaviour With this, we can repeatedly set up a 'showcase' and it will only create resources where they don't exist and launch pipelines, meaning we can be fairly frugal with resource set up. The downside is status drift may occur, where the platform resources have diverged from the YAML templates which will not be captured and will be ignored by seqerakit. Not ideal, but a problem even for things like [Terraform](https://developer.hashicorp.com/terraform/tutorials/state/resource-drift). --------- Co-authored-by: ejseqera <[email protected]> Co-authored-by: Esha Joshi <[email protected]>
1 parent bf814a9 commit e0b2533

34 files changed

+381
-145
lines changed

‎README.md

+26-4
Original file line numberDiff line numberDiff line change
@@ -301,14 +301,34 @@ params:
301301

302302
**Note**: If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the provided `params-file`.
303303

304-
### 2. `overwrite` Functionality
304+
### 2. `on_exists` Functionality
305305

306-
For every entity defined in your YAML file, you can specify `overwrite: True` to overwrite any existing entities in Seqera Platform of the same name.
306+
For every entity defined in your YAML file, you can specify how to handle cases where the entity already exists using the `on_exists` parameter with one of three options:
307307

308-
`seqerakit` will first check to see if the name of the entity exists, if so, it will invoke a `tw <subcommand> delete` command before attempting to create it based on the options defined in the YAML file.
308+
- `fail` (default): Raise an error if the entity already exists
309+
- `ignore`: Skip creation if the entity already exists
310+
- `overwrite`: Delete the existing entity and create a new one based on the YAML configuration
311+
312+
Example usage in YAML:
313+
314+
```yaml
315+
workspaces:
316+
- name: 'showcase'
317+
organization: 'seqerakit_automation'
318+
on_exists: 'overwrite' # Will delete and recreate if exists
319+
```
320+
321+
```yaml
322+
credentials:
323+
- name: 'github_credentials'
324+
workspace: 'seqerakit_automation/showcase'
325+
on_exists: 'ignore' # Will skip if already exists
326+
```
327+
328+
When using the `overwrite` option, `seqerakit` will first check if the entity exists, and if so, it will invoke a `tw <subcommand> delete` command before attempting to create it based on the options defined in the YAML file:
309329

310330
```console
311-
DEBUG:root: Overwrite is set to 'True' for organizations
331+
DEBUG:root: on_exists is set to 'overwrite' for organizations
312332
313333
DEBUG:root: Running command: tw -o json organizations list
314334
DEBUG:root: The attempted organizations resource already exists. Overwriting.
@@ -317,6 +337,8 @@ DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION
317337
DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization'
318338
```
319339

340+
> **Note**: For backward compatibility, the `overwrite: True|False` parameter is still supported but deprecated. It will be mapped to `on_exists: 'overwrite'|'fail'` respectively.
341+
320342
### 3. Specifying JSON configuration files with `file-path`
321343

322344
The Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file.

‎examples/yaml/e2e/actions.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,4 @@ actions:
99
profile: 'test'
1010
params:
1111
outdir: s3://seqeralabs-showcase/nf-core-rnaseq/action/'
12-
overwrite: True
12+
on_exists: overwrite

‎examples/yaml/e2e/compute-envs.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -14,16 +14,16 @@ compute-envs:
1414
no-ebs-auto-scale: True
1515
max-cpus: 500
1616
wait: AVAILABLE
17-
overwrite: True
17+
on_exists: overwrite
1818
- name: 'seqera_azure_virginia'
1919
workspace: 'seqerakit-e2e/showcase'
2020
credentials: 'azure_credentials'
2121
wait: 'AVAILABLE'
2222
file-path: './examples/yaml/compute-envs/seqera_azure_virginia.json'
23-
overwrite: True
23+
on_exists: overwrite
2424
- name: 'seqera_gcp_finland'
2525
workspace: 'seqerakit-e2e/showcase'
2626
credentials: 'google_credentials'
2727
wait: 'AVAILABLE'
2828
file-path: './examples/yaml/compute-envs/seqera_gcp_finland.json'
29-
overwrite: True
29+
on_exists: overwrite

‎examples/yaml/e2e/credentials.yml

+5-5
Original file line numberDiff line numberDiff line change
@@ -4,31 +4,31 @@ credentials:
44
workspace: 'seqerakit-e2e/showcase'
55
username: 'ejseqera'
66
password: '$TOWER_GITHUB_PASSWORD'
7-
overwrite: True
7+
on_exists: overwrite
88
- type: 'container-reg'
99
name: 'dockerhub_credentials'
1010
workspace: 'seqerakit-e2e/showcase'
1111
username: 'eshajoshi'
1212
password: '$DOCKERHUB_PASSWORD'
1313
registry: 'docker.io'
14-
overwrite: True
14+
on_exists: overwrite
1515
- type: 'aws'
1616
name: 'aws_credentials'
1717
workspace: 'seqerakit-e2e/showcase'
1818
access-key: '$AWS_ACCESS_KEY_ID'
1919
secret-key: '$AWS_SECRET_ACCESS_KEY'
2020
assume-role-arn: '$AWS_ASSUME_ROLE_ARN'
21-
overwrite: True
21+
on_exists: overwrite
2222
- type: 'azure'
2323
name: 'azure_credentials'
2424
workspace: 'seqerakit-e2e/showcase'
2525
batch-key: '$AZURE_BATCH_KEY'
2626
batch-name: 'seqeralabs'
2727
storage-key: '$AZURE_STORAGE_KEY'
2828
storage-name: 'seqeralabs'
29-
overwrite: True
29+
on_exists: overwrite
3030
- type: 'google'
3131
name: 'google_credentials'
3232
workspace: 'seqerakit-e2e/showcase'
3333
key: '$GOOGLE_KEY'
34-
overwrite: True
34+
on_exists: overwrite

‎examples/yaml/e2e/datasets.yml

+5-5
Original file line numberDiff line numberDiff line change
@@ -4,28 +4,28 @@ datasets:
44
header: true
55
workspace: 'seqerakit-e2e/showcase'
66
file-path: './examples/yaml/datasets/rnaseq_samples.csv'
7-
overwrite: True
7+
on_exists: overwrite
88
- name: 'sarek_samples'
99
description: 'Samplesheet to run the nf-core/sarek pipeline from end-to-end'
1010
header: true
1111
workspace: 'seqerakit-e2e/showcase'
1212
file-path: './examples/yaml/datasets/sarek_samples.csv'
13-
overwrite: True
13+
on_exists: overwrite
1414
- name: 'viralrecon_illumina_samples'
1515
description: 'Samplesheet to run the nf-core/viralrecon pipeline from end-to-end with Illumina data'
1616
header: true
1717
workspace: 'seqerakit-e2e/showcase'
1818
file-path: './examples/yaml/datasets/viralrecon_illumina_samples.csv'
19-
overwrite: True
19+
on_exists: overwrite
2020
- name: 'viralrecon_nanopore_samples'
2121
description: 'Samplesheet to run the nf-core/viralrecon pipeline from end-to-end with Nanopore data'
2222
header: true
2323
workspace: 'seqerakit-e2e/showcase'
2424
file-path: './examples/yaml/datasets/viralrecon_nanopore_samples.csv'
25-
overwrite: True
25+
on_exists: overwrite
2626
- name: 'sentieon_samples'
2727
description: 'Samplesheet to run the seqeralabs/nf-sentieon pipeline from end-to-end'
2828
header: true
2929
workspace: 'seqerakit-e2e/showcase'
3030
file-path: './examples/yaml/datasets/sentieon_samples.csv'
31-
overwrite: True
31+
on_exists: overwrite

‎examples/yaml/e2e/labels.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@ labels:
22
- name: 'seqerakit'
33
value: 'test'
44
workspace: 'seqerakit-e2e/showcase'
5-
overwrite: True
5+
on_exists: overwrite

‎examples/yaml/e2e/members.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
members:
22
- user: '[email protected]'
33
organization: 'seqerakit-e2e'
4-
overwrite: False
4+
on_exists: ignore

‎examples/yaml/e2e/organizations.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,4 @@ organizations:
44
description: 'Organization created E2E with seqerakit CLI scripting'
55
location: 'Global'
66
website: 'https://seqera.io/'
7-
overwrite: True
7+
on_exists: overwrite

‎examples/yaml/e2e/participants.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@ participants:
33
type: 'TEAM'
44
workspace: 'seqerakit-e2e/showcase'
55
role: 'ADMIN'
6-
overwrite: True
6+
on_exists: overwrite
77
- name: '[email protected]'
88
type: 'MEMBER'
99
workspace: 'seqerakit-e2e/showcase'
1010
role: 'LAUNCH'
11-
overwrite: True
11+
on_exists: overwrite

‎examples/yaml/e2e/pipelines.yml

+4-4
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,12 @@ pipelines:
1111
outdir: 's3://seqeralabs-showcase/nf-core-rnaseq/results'
1212
config: './examples/yaml/pipelines/nextflow.config'
1313
pre-run: './examples/yaml/pipelines/pre_run.txt'
14-
overwrite: True
14+
on_exists: overwrite
1515
- name: 'nf-core-sarek'
1616
workspace: 'seqerakit-e2e/showcase'
1717
compute-env: 'seqera_azure_virginia'
1818
file-path: './examples/yaml/pipelines/nf-core-sarek_pipeline.json'
19-
overwrite: True
19+
on_exists: overwrite
2020
- name: 'nf-core-viralrecon-illumina'
2121
url: 'https://github.com/nf-core/viralrecon'
2222
workspace: 'seqerakit-e2e/showcase'
@@ -28,9 +28,9 @@ pipelines:
2828
params-file: './examples/yaml/pipelines/nf_core_viralrecon_illumina_params.yml'
2929
config: './examples/yaml/pipelines/nextflow.config'
3030
pre-run: './examples/yaml/pipelines/pre_run.txt'
31-
overwrite: True
31+
on_exists: overwrite
3232
- name: 'nf-sentieon'
3333
workspace: 'seqerakit-e2e/showcase'
3434
compute-env: 'seqera_aws_virginia_fusionv2_nvme'
3535
file-path: './examples/yaml/pipelines/nf_sentieon_pipeline.json'
36-
overwrite: True
36+
on_exists: overwrite

‎examples/yaml/e2e/secrets.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@ secrets:
22
- name: 'SENTIEON_LICENSE_BASE64'
33
workspace: 'seqerakit-e2e/showcase'
44
value: '$SENTIEON_LICENSE_BASE64'
5-
overwrite: True
5+
on_exists: overwrite

‎examples/yaml/e2e/teams.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,4 @@ teams:
66
- '[email protected]'
77
- '[email protected]'
88
- '[email protected]'
9-
overwrite: True
9+
on_exists: overwrite

‎examples/yaml/e2e/workspaces.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,4 @@ workspaces:
44
organization: 'seqerakit-e2e'
55
description: 'Workspace created E2E with seqerakit CLI scripting'
66
visibility: 'PRIVATE'
7-
overwrite: True
7+
on_exists: overwrite

‎examples/yaml/seqerakit-e2e.yml

+23-23
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ organizations:
44
description: 'Organization created E2E with seqerakit CLI scripting'
55
location: 'Global'
66
website: 'https://seqera.io/'
7-
overwrite: True
7+
on_exists: overwrite
88
teams:
99
- name: 'scidev_team'
1010
organization: 'seqerakit-e2e'
@@ -13,64 +13,64 @@ teams:
1313
- '[email protected]'
1414
- '[email protected]'
1515
- '[email protected]'
16-
overwrite: True
16+
on_exists: overwrite
1717
workspaces:
1818
- name: 'showcase'
1919
full-name: 'showcase'
2020
organization: 'seqerakit-e2e'
2121
description: 'Workspace created E2E with seqerakit CLI scripting'
2222
visibility: 'PRIVATE'
23-
overwrite: True
23+
on_exists: overwrite
2424
participants:
2525
- name: 'scidev_team'
2626
type: 'TEAM'
2727
workspace: 'seqerakit-e2e/showcase'
2828
role: 'ADMIN'
29-
overwrite: True
29+
on_exists: overwrite
3030
- name: '[email protected]'
3131
type: 'MEMBER'
3232
workspace: 'seqerakit-e2e/showcase'
3333
role: 'LAUNCH'
34-
overwrite: True
34+
on_exists: overwrite
3535
credentials:
3636
- type: 'github'
3737
name: 'github_credentials'
3838
workspace: 'seqerakit-e2e/showcase'
3939
username: 'ejseqera'
4040
password: '$TOWER_GITHUB_PASSWORD'
41-
overwrite: True
41+
on_exists: overwrite
4242
- type: 'container-reg'
4343
name: 'dockerhub_credentials'
4444
workspace: 'seqerakit-e2e/showcase'
4545
username: 'eshajoshi'
4646
password: '$DOCKERHUB_PASSWORD'
4747
registry: 'docker.io'
48-
overwrite: True
48+
on_exists: overwrite
4949
- type: 'aws'
5050
name: 'aws_credentials'
5151
workspace: 'seqerakit-e2e/showcase'
5252
access-key: '$AWS_ACCESS_KEY_ID'
5353
secret-key: '$AWS_SECRET_ACCESS_KEY'
5454
assume-role-arn: '$AWS_ASSUME_ROLE_ARN'
55-
overwrite: True
55+
on_exists: overwrite
5656
- type: 'azure'
5757
name: 'azure_credentials'
5858
workspace: 'seqerakit-e2e/showcase'
5959
batch-key: '$AZURE_BATCH_KEY'
6060
batch-name: 'seqeralabs'
6161
storage-key: '$AZURE_STORAGE_KEY'
6262
storage-name: 'seqeralabs'
63-
overwrite: True
63+
on_exists: overwrite
6464
- type: 'google'
6565
name: 'google_credentials'
6666
workspace: 'seqerakit-e2e/showcase'
6767
key: '$GOOGLE_KEY'
68-
overwrite: True
68+
on_exists: overwrite
6969
secrets:
7070
- name: 'SENTIEON_LICENSE_BASE64'
7171
workspace: 'seqerakit-e2e/showcase'
7272
value: '$SENTIEON_LICENSE_BASE64'
73-
overwrite: True
73+
on_exists: overwrite
7474
compute-envs:
7575
- type: aws-batch
7676
config-mode: forge
@@ -87,50 +87,50 @@ compute-envs:
8787
no-ebs-auto-scale: True
8888
max-cpus: 500
8989
wait: AVAILABLE
90-
overwrite: True
90+
on_exists: overwrite
9191
- name: 'seqera_azure_virginia'
9292
workspace: 'seqerakit-e2e/showcase'
9393
credentials: 'azure_credentials'
9494
wait: 'AVAILABLE'
9595
file-path: './examples/yaml/compute-envs/seqera_azure_virginia.json'
96-
overwrite: True
96+
on_exists: overwrite
9797
- name: 'seqera_gcp_finland'
9898
workspace: 'seqerakit-e2e/showcase'
9999
credentials: 'google_credentials'
100100
wait: 'AVAILABLE'
101101
file-path: './examples/yaml/compute-envs/seqera_gcp_finland.json'
102-
overwrite: True
102+
on_exists: overwrite
103103
datasets:
104104
- name: 'rnaseq_samples'
105105
description: 'Samplesheet to run the nf-core/rnaseq pipeline from end-to-end'
106106
header: true
107107
workspace: 'seqerakit-e2e/showcase'
108108
file-path: './examples/yaml/datasets/rnaseq_samples.csv'
109-
overwrite: True
109+
on_exists: overwrite
110110
- name: 'sarek_samples'
111111
description: 'Samplesheet to run the nf-core/sarek pipeline from end-to-end'
112112
header: true
113113
workspace: 'seqerakit-e2e/showcase'
114114
file-path: './examples/yaml/datasets/sarek_samples.csv'
115-
overwrite: True
115+
on_exists: overwrite
116116
- name: 'viralrecon_illumina_samples'
117117
description: 'Samplesheet to run the nf-core/viralrecon pipeline from end-to-end with Illumina data'
118118
header: true
119119
workspace: 'seqerakit-e2e/showcase'
120120
file-path: './examples/yaml/datasets/viralrecon_illumina_samples.csv'
121-
overwrite: True
121+
on_exists: overwrite
122122
- name: 'viralrecon_nanopore_samples'
123123
description: 'Samplesheet to run the nf-core/viralrecon pipeline from end-to-end with Nanopore data'
124124
header: true
125125
workspace: 'seqerakit-e2e/showcase'
126126
file-path: './examples/yaml/datasets/viralrecon_nanopore_samples.csv'
127-
overwrite: True
127+
on_exists: overwrite
128128
- name: 'sentieon_samples'
129129
description: 'Samplesheet to run the seqeralabs/nf-sentieon pipeline from end-to-end'
130130
header: true
131131
workspace: 'seqerakit-e2e/showcase'
132132
file-path: './examples/yaml/datasets/sentieon_samples.csv'
133-
overwrite: True
133+
on_exists: overwrite
134134
pipelines:
135135
- name: 'nf-core-rnaseq'
136136
url: 'https://github.com/nf-core/rnaseq'
@@ -144,12 +144,12 @@ pipelines:
144144
outdir: 's3://seqeralabs-showcase/nf-core-rnaseq/results'
145145
config: './examples/yaml/pipelines/nextflow.config'
146146
pre-run: './examples/yaml/pipelines/pre_run.txt'
147-
overwrite: True
147+
on_exists: overwrite
148148
- name: 'nf-core-sarek'
149149
workspace: 'seqerakit-e2e/showcase'
150150
compute-env: 'seqera_azure_virginia'
151151
file-path: './examples/yaml/pipelines/nf-core-sarek_pipeline.json'
152-
overwrite: True
152+
on_exists: overwrite
153153
- name: 'nf-core-viralrecon-illumina'
154154
url: 'https://github.com/nf-core/viralrecon'
155155
workspace: 'seqerakit-e2e/showcase'
@@ -161,12 +161,12 @@ pipelines:
161161
params-file: './examples/yaml/pipelines/nf_core_viralrecon_illumina_params.yml'
162162
config: './examples/yaml/pipelines/nextflow.config'
163163
pre-run: './examples/yaml/pipelines/pre_run.txt'
164-
overwrite: True
164+
on_exists: overwrite
165165
- name: 'nf-sentieon'
166166
workspace: 'seqerakit-e2e/showcase'
167167
compute-env: 'seqera_aws_virginia_fusionv2_nvme'
168168
file-path: './examples/yaml/pipelines/nf_sentieon_pipeline.json'
169-
overwrite: True
169+
on_exists: overwrite
170170
launch:
171171
- name: 'nf-core-rnaseq-launchpad'
172172
workspace: 'seqerakit-e2e/showcase'

0 commit comments

Comments
 (0)