"overwrite" breaks manual CE's

[stevekm]

I am trying out seqerakit and the first problem I am finding is that it does not seem possible to manually add a batch-forge CE to other workspaces without breaking your Compute Environments.

Here is an example

compute-envs:
  # create this CE in the first workspace
  - type: aws-batch
    config-mode: forge
    name: 'EBSAutoScale-test'
    workspace: 'MyOrg/MyWorkspace1'
    credentials: 'AWS_Keys'
    region: 'us-east-1'
    work-dir: 's3://my-bucket/seqera-workdir'
    provisioning-model: 'SPOT'
    fusion-v2: False
    wave: False
    fargate: False
    fast-storage: False
    instance-types: 'optimal'
    no-ebs-auto-scale: False
    max-cpus: 1000
    min-cpus: 0
    head-job-cpus: 8
    head-job-memory: 24000
    subnets: 'subnet-0123456789'
    security-groups: 'sg-0123456789,sg-9876543210'
    allow-buckets: 's3://my-bucket'
    wait: 'AVAILABLE'
    alloc-strategy: 'SPOT_CAPACITY_OPTIMIZED'
    bid-percentage: "100"
    overwrite: False

  # manually add the same CE to the second workspace with modified IAMs
  - type: aws-batch
    config-mode: manual
    name: 'EBSAutoScale-manual-test'
    workspace: 'MyOrg/MyWorkspace2'
    credentials: 'AWS_Keys'
    region: 'us-east-1'
    work-dir: 's3://my-bucket/seqera-workdir'
    head-queue: "TowerForge-abc123efg567-head" # required
    compute-queue: "TowerForge-abc123efg567-work" # required
    head-job-role: "arn:aws:iam::0123456789:role/TowerForge-Special-S3-access"
    compute-job-role: "arn:aws:iam::0123456789:role/TowerForge-Special-S3-access"
    wait: 'AVAILABLE'

In this example, I am creating one AWS Batch Compute Environment with Batch Forge in MyWorkspace1 (EBSAutoScale-test CE) and then manually adding that CE to MyWorkspace2 with modified IAM roles for S3 access control (EBSAutoScale-manual-test).

However, in this current state, you cannot actually use this YAML because the head-queue and compute-queue for EBSAutoScale-manual-test must be hard-coded; beacuse Batch Forge generates random AWS Batch CE names such as TowerForge-abc123efg567-head, there is apparently no way to know ahead of time what the name will be for the AWS Batch CE created for EBSAutoScale-test

So what you are forced to do then, is to create only the first CE for EBSAutoScale-test, then somehow retrieve the AWS Batch CE for it ( tw --output json compute-envs view --name "$CE_NAME" --workspace "$WORKSPACE_ID" | jq -r .config.computeQueue ), then manually update the YAML with the correct head-queue and compute-queue.

But after doing this, you can no longer run the YAML because the first CE EBSAutoScale-test already exists. So the only way to run the YAML is to use overwrite like this

seqerakit file.yml --overwrite

Now you can create both CE's at the same time.

Except that by "overwriting" the first CE EBSAutoScale-test, now you have a new unique label for the head-queue and compute-queue that no longer matches what you put into the YAML. So in effect your second CE EBSAutoScale-manual-test is now broken.

I am not sure how you are meant to handle this type of situation with the YAML syntax shown in the docs here. I could not find any examples of dynamically resolving the attributes of one Seqera Platform resource in order to use it as part of another.

I think a possible solution to this would be to use some sort of variable expansion that seqerakit would use to track the defined resources so that it could resolve resource attributes in the definition of other attributes. Something like this

compute-envs:
  - type: aws-batch
    config-mode: forge
    name: 'EBSAutoScale-test'
    ....

  - type: aws-batch
    config-mode: manual
    name: 'EBSAutoScale-manual-test'
    head-queue: `$EBSAutoScale-test.config.headQueue`
    compute-queue: `$EBSAutoScale-test.config.computeQueue`

Is there a way to accomplish anything like this? Thanks!