diff --git a/README.md b/README.md index 866b020..13da5d0 100644 --- a/README.md +++ b/README.md @@ -54,14 +54,17 @@ pip install --upgrade --force-reinstall seqerakit ``` ### Local development installation + You can install the development branch of `seqerakit` on your local machine to test feature updates of the tool. Before proceeding, ensure that you have [Python](https://www.python.org/downloads/) and [Git](https://git-scm.com/downloads) installed on your system. 1. To install directly from pip: + ```bash pip install git+https://github.com/seqeralabs/seqera-kit.git@dev ``` 2. Alternatively, you may clone the repository locally and install manually: + ```bash git clone https://github.com/seqeralabs/seqera-kit.git cd seqera-kit @@ -70,6 +73,7 @@ pip install . ``` You can verify your installation with: + ```bash pip show seqerakit ``` @@ -85,11 +89,12 @@ export TOWER_ACCESS_TOKEN= ``` For Enterprise installations of Seqera Platform, you will also need to configure the API endpoint that will be used to connect to the Platform. You can do so by exporting the following environment variable: + ```bash export TOWER_API_ENDPOINT= ``` -By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Platform Cloud. +By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Platform Cloud. ## Usage @@ -100,25 +105,33 @@ seqerakit --info ``` Use the `--help` or `-h` parameter to list the available commands and their associated options: + ```bash seqerakit --help ``` Use `--version` or `-v` to retrieve the current version of your seqerakit installation: + ```bash seqerakit --version ``` + ### Input + `seqerakit` supports input through either file paths to YAMLs or directly from standard input (stdin). #### Using File Path + ```bash seqerakit /path/to/file.yaml ``` + #### Using stdin + ```console $ cat file.yaml | seqerakit - ``` + See the [Defining your YAML file using CLI options](#defining-your-yaml-file-using-cli-options) section for guidance on formatting your input YAML file(s). ### Dryrun @@ -129,6 +142,53 @@ To print the commands that would executed with `tw` when using a YAML file, you seqerakit file.yaml --dryrun ``` +To capture the details of the created resources, you can use the `--json` command-line flag to output the results as JSON to `stdout`. This is equivalent to using the `-o json` flag with the `tw` CLI. + +For example: + +```bash +seqerakit -j examples/yaml/e2e/launch.yml +``` + +This command internally runs the following `tw` command: + +```bash +INFO:root: Running command: tw -o json launch --name hello --workspace $SEQERA_ORGANIZATION_NAME/$SEQERA_WORKSPACE_NAME hello +``` + +The output will look like this: + +```json +{ + "workflowId": "1wfhRp5ioFIyrs", + "workflowUrl": "https://tower.nf/orgs/orgName/workspaces/workspaceName/watch/1wfhRp5ioFIyrs", + "workspaceId": 12345678, + "workspaceRef": "[orgName / workspaceName]" +} +``` + +This JSON output can be piped into other tools for further processing. Note that logs will still be written to `stderr`, allowing you to monitor the tool's progress in real-time. + +If you prefer to suppress the JSON output and focus only on the logs: + +```bash +seqerakit -j examples/yaml/e2e/launch.yml > /dev/null +``` + +This will still log: + +```bash +INFO:root: Running command: tw -o json launch --name hello --workspace $SEQERA_ORGANIZATION_NAME/$SEQERA_WORKSPACE_NAME hello +``` + +Each execution of the `tw` CLI generates a single JSON object. To combine multiple JSON objects into one, you can use a tool like `jq`: + +```bash +seqerakit -j launch/*.yml | jq --slurp > launched-pipelines.json +``` + +This command will merge the individual JSON objects from each `tw` command into a single JSON array and save it to `launched-pipelines.json`. + ### Recursively delete Instead of adding or creating resources, you can recursively delete resources in your YAML file by specifying the `--delete` flag: @@ -170,6 +230,7 @@ seqerakit hello-world-config.yml --cli="-Djavax.net.ssl.trustStore=/absolute/pat Note: Use of `--verbose` option for the `tw` CLI is currently not supported by `seqerakit`. Supplying `--cli="--verbose"` will raise an error. ## Specify targets + When using a YAML file as input that defines multiple resources, you can use the `--targets` flag to specify which resources to create. This flag takes a comma-separated list of resource names. For example, given a YAML file that defines the following resources: @@ -178,18 +239,17 @@ For example, given a YAML file that defines the following resources: workspaces: - name: 'showcase' organization: 'seqerakit_automation' -... +--- compute-envs: - name: 'compute-env' type: 'aws-batch forge' workspace: 'seqerakit/test' -... +--- pipelines: - - name: "hello-world-test-seqerakit" - url: "https://github.com/nextflow-io/hello" + - name: 'hello-world-test-seqerakit' + url: 'https://github.com/nextflow-io/hello' workspace: 'seqerakit/test' - compute-env: "compute-env" -... + compute-env: 'compute-env' ``` You can target the creation of `pipelines` only by running: @@ -197,9 +257,11 @@ You can target the creation of `pipelines` only by running: ```bash seqerakit test.yml --targets pipelines ``` + This will process only the pipelines block from the YAML file and ignore other blocks such as `workspaces` and `compute-envs`. ### Multiple Targets + You can also specify multiple resources to create by separating them with commas. For example, to create both workspaces and pipelines, run: ```bash @@ -240,6 +302,7 @@ params: **Note**: If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the provided `params-file`. ### 2. `overwrite` Functionality + For every entity defined in your YAML file, you can specify `overwrite: True` to overwrite any existing entities in Seqera Platform of the same name. `seqerakit` will first check to see if the name of the entity exists, if so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. @@ -253,22 +316,23 @@ DEBUG:root: The attempted organizations resource already exists. Overwriting. DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization' ``` + ### 3. Specifying JSON configuration files with `file-path` + The Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. An example of the `file-path` option is provided in the [compute-envs.yml](./templates/compute-envs.yml) template: ```yaml compute-envs: - - name: 'my_aws_compute_environment' # required - workspace: 'my_organization/my_workspace' # required - credentials: 'my_aws_credentials' # required - wait: 'AVAILABLE' # optional - file-path: './compute-envs/my_aws_compute_environment.json' # required + - name: 'my_aws_compute_environment' # required + workspace: 'my_organization/my_workspace' # required + credentials: 'my_aws_credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/my_aws_compute_environment.json' # required overwrite: True ``` - ## Quick start You must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform. @@ -281,10 +345,10 @@ You will need to have an account on Seqera Platform (see [Plans and pricing](htt ```yaml # noqa launch: - - name: 'hello-world' # Workflow name - workspace: '' # Workspace name - compute-env: '' # Compute environment - revision: 'master' # Pipeline revision + - name: 'hello-world' # Workflow name + workspace: '' # Workspace name + compute-env: '' # Compute environment + revision: 'master' # Pipeline revision pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL ``` @@ -340,6 +404,7 @@ Options: --revision= A valid repository commit Id, tag or branch name. ... ``` + 2. Define Key-Value Pairs in YAML Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance: @@ -364,12 +429,12 @@ In this example: - The corresponding values are user-defined ### Best Practices: + - Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting. - Use quotes around strings that contain special characters or spaces. - When listing multiple values (`labels`, `instance-types`, `allow-buckets`, etc), separate them with commas as shown above. - For complex configurations, refer to the [Templates](./templates/) provided in this repository. - ## Templates We have provided template YAML files for each of the entities that can be created on Seqera Platform. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/seqera-kit/templates) directory and should form a good starting point for you to add your own customization: diff --git a/examples/python/launch_hello_world.py b/examples/python/launch_hello_world.py index 879a703..5548f47 100755 --- a/examples/python/launch_hello_world.py +++ b/examples/python/launch_hello_world.py @@ -28,5 +28,4 @@ "--wait", "SUBMITTED", "https://github.com/nextflow-io/hello", - to_json=True, ) diff --git a/seqerakit/cli.py b/seqerakit/cli.py index e9bcbdf..893066c 100644 --- a/seqerakit/cli.py +++ b/seqerakit/cli.py @@ -47,6 +47,9 @@ def parse_args(args=None): action="store_true", help="Display Seqera Platform information and exit.", ) + general.add_argument( + "-j", "--json", action="store_true", help="Output JSON format in stdout." + ) general.add_argument( "--dryrun", "-d", @@ -152,9 +155,12 @@ def main(args=None): options = parse_args(args if args is not None else sys.argv[1:]) logging.basicConfig(level=getattr(logging, options.log_level.upper())) - # Parse CLI arguments into a list and create a Seqera Platform instance + # Parse CLI arguments into a list cli_args_list = options.cli_args.split() if options.cli_args else [] - sp = seqeraplatform.SeqeraPlatform(cli_args=cli_args_list, dryrun=options.dryrun) + + sp = seqeraplatform.SeqeraPlatform( + cli_args=cli_args_list, dryrun=options.dryrun, json=options.json + ) # If the info flag is set, run 'tw info' if options.info: diff --git a/seqerakit/computeenvs.py b/seqerakit/computeenvs.py index ab1de1c..9254a54 100644 --- a/seqerakit/computeenvs.py +++ b/seqerakit/computeenvs.py @@ -49,4 +49,4 @@ def export_ce(self, name, *args, **kwargs): ] # Pass the built command to the base class method in SeqeraPlatform - return self._tw_run(command, *args, **kwargs, to_json=True) + return self._tw_run(command, *args, **kwargs) diff --git a/seqerakit/seqeraplatform.py b/seqerakit/seqeraplatform.py index 5268a5f..ca01004 100644 --- a/seqerakit/seqeraplatform.py +++ b/seqerakit/seqeraplatform.py @@ -43,7 +43,7 @@ def __call__(self, *args, **kwargs): return self.tw_instance._tw_run(command, **kwargs) # Constructs a new SeqeraPlatform instance - def __init__(self, cli_args=None, dryrun=False, print_stdout=True): + def __init__(self, cli_args=None, dryrun=False, print_stdout=True, json=False): if cli_args and "--verbose" in cli_args: raise ValueError( "--verbose is not supported as a CLI argument to seqerakit." @@ -51,12 +51,13 @@ def __init__(self, cli_args=None, dryrun=False, print_stdout=True): self.cli_args = cli_args or [] self.dryrun = dryrun self.print_stdout = print_stdout + self.json = json self._suppress_output = False def _construct_command(self, cmd, *args, **kwargs): command = ["tw"] + self.cli_args - if kwargs.get("to_json"): + if self.json: command.extend(["-o", "json"]) command.extend(cmd) @@ -112,13 +113,21 @@ def _execute_command(self, full_cmd, to_json=False, print_stdout=True): print_stdout if print_stdout is not None else self.print_stdout ) and not self._suppress_output - if should_print: + # Do not print output in logging if self.json is enabled + if should_print and not self.json: logging.info(f" Command output: {stdout}") if "ERROR: " in stdout or process.returncode != 0: self._handle_command_errors(stdout) - return json.loads(stdout) if to_json else stdout + if self.json or to_json: + out = json.loads(stdout) + print(json.dumps(out)) + else: + out = stdout + print(stdout) + + return out def _handle_command_errors(self, stdout): # Check for specific tw cli error patterns and raise custom exceptions @@ -139,7 +148,7 @@ def _tw_run(self, cmd, *args, **kwargs): full_cmd = self._construct_command(cmd, *args, **kwargs) if not full_cmd or self.dryrun: logging.info(f"DRYRUN: Running command {full_cmd}") - return + return None return self._execute_command(full_cmd, kwargs.get("to_json"), print_stdout) @contextmanager diff --git a/tests/unit/test_seqeraplatform.py b/tests/unit/test_seqeraplatform.py index ae1d34a..dc1cff0 100644 --- a/tests/unit/test_seqeraplatform.py +++ b/tests/unit/test_seqeraplatform.py @@ -10,7 +10,7 @@ class TestSeqeraPlatform(unittest.TestCase): def setUp(self): - self.sp = seqeraplatform.SeqeraPlatform() + self.sp = seqeraplatform.SeqeraPlatform(json=True) @patch("subprocess.Popen") def test_run_with_jsonout_command(self, mock_subprocess): @@ -38,7 +38,7 @@ def test_run_with_jsonout_command(self, mock_subprocess): command = getattr(self.sp, "pipelines") # Run the command with arguments - result = command("view", "--name", "pipeline_name", to_json=True) + result = command("view", "--name", "pipeline_name") # Check that Popen was called with the right arguments mock_subprocess.assert_called_once_with( @@ -121,7 +121,7 @@ def test_json_parsing(self): command = getattr(self.sp, "pipelines") # Check that the JSON is parsed correctly - self.assertEqual(command("arg1", "arg2", to_json=True), {"key": "value"}) + self.assertEqual(command("arg1", "arg2"), {"key": "value"}) class TestSeqeraPlatformCLIArgs(unittest.TestCase): @@ -139,7 +139,7 @@ def test_cli_args_inclusion(self, mock_subprocess): ) # Call a method - self.sp.pipelines("view", "--name", "pipeline_name", to_json=True) + self.sp.pipelines("view", "--name", "pipeline_name") # Extract the command used to call Popen called_command = mock_subprocess.call_args[0][0] @@ -164,7 +164,7 @@ def test_cli_args_inclusion_ssl_certs(self, mock_subprocess): ) # Call a method - self.sp.pipelines("view", "--name", "pipeline_name", to_json=True) + self.sp.pipelines("view", "--name", "pipeline_name") # Extract the command used to call Popen called_command = mock_subprocess.call_args[0][0] @@ -222,7 +222,7 @@ def setUp(self): @patch("subprocess.Popen") def test_dryrun_call(self, mock_subprocess): # Run a method with dryrun=True - self.dryrun_tw.pipelines("view", "--name", "pipeline_name", to_json=True) + self.dryrun_tw.pipelines("view", "--name", "pipeline_name") # Assert that subprocess.Popen is not called mock_subprocess.assert_not_called() @@ -332,6 +332,13 @@ def test_json_output_handling(self, mock_subprocess): result = self.sp._execute_command("tw pipelines list", to_json=False) self.assertEqual(result, '{"key": "value"}') + _sp = seqeraplatform.SeqeraPlatform(json=True) + result = _sp._execute_command("tw pipelines list", to_json=False) + self.assertEqual(result, {"key": "value"}) + + result = _sp._execute_command("tw pipelines list", to_json=True) + self.assertEqual(result, {"key": "value"}) + @patch("subprocess.Popen") def test_print_stdout_override(self, mock_subprocess): mock_subprocess.return_value = MagicMock(returncode=0)