My zones input becomes invalid on GCP Batch for Cromwell v88

[yihming]

Hello,

In Cromwell v87, my workflow has the following default values for zones runtime attribute:

zones: "us-central1-a us-central1-b us-central1-c us-central1-f us-east1-b us-east1-c us-east1-d us-west1-a us-west1-b us-west1-c"

And when submitted to my GCP Batch which is deployed in us-west1, it worked smoothly with zones overwritten by the Batch configuration.

However, in Cromwell v88, the same job triggers the following error at server side:

2025-03-11 22:41:49 cromwell-system-akka.dispatchers.engine-dispatcher-29 INFO  - WorkflowManagerActor: Workflow 4257dce0-247f-45c3-9961-5f1db9bfb073 failed (during ExecutingWorkflowState): cromwell.backend.google.batch.api.BatchApiRequestManager$UserBatchApiException: Unable to complete Batch request due to a problem with the request (io.grpc.StatusRuntimeException: INVALID_ARGUMENT: job.allocation_policy field is invalid. all specified locations end up in more than one regions [us-central1 us-east1 us-west1].).

I was able to go around it by explicitly specifying zones: "us-west1-a us-west1-b us-west1-c" in my input JSON file. However, I'm still wondering if there is any way to fix it so that it works as usual.

Or is it true that since Cromwell v88, multi-regon zones value would always be considered invalid?

[jgainerdewar]

This is due to a limitation of GCP Batch - when specifying a location for the job to run in, we can only request a region or a list of zones in a single region.

https://cloud.google.com/batch/docs/reference/rest/v1/projects.locations.jobs#LocationPolicy.FIELDS.allowed_locations