pipeline health POC #59

edmundmiller · 2024-07-20T23:10:12Z

~~Closes~~ A start on #5

I expect it to go through some iteration before actually using it to manage the pipelines, and even then a slow roll-out.

Just looking for anything that anyone absolutely hates, or settings that we can catch that are wrong.

…dules

1. Importing them all by hand, some code duplication and effort, but probably the least likely to blow up 2. Looping through them all We can also start with 1, and then move to 2 once everything is captured in the Pulumi state with 1(which seems like the sane option)

pulumi/github/repos/README.md

pulumi/github/repos/core_repos.yml

pulumi/github/repos/pipelines.yml

ewels · 2024-07-21T05:32:14Z

Looking very promising!

The old website PHP code for this is here: https://github.com/nf-core/website/blob/old-site/public_html/pipeline_health.php

That contains all of the logic for stuff like the repo website, the set of minimum keywords and the types of things to do (eg. stripping old bad GHA job names). See the functions called fix_* (mostly).

ewels · 2024-07-21T05:35:12Z

Also this is the new pipeline health page in Astro, if it helps: https://github.com/nf-core/website/blob/main/sites/pipelines/src/pages/pipeline_health.astro

That reports on a lot of this stuff but doesn't set anything.

edmundmiller · 2024-07-21T16:28:16Z

Also this is the new pipeline health page in Astro, if it helps: https://github.com/nf-core/website/blob/main/sites/pipelines/src/pages/pipeline_health.astro

That reports on a lot of this stuff but doesn't set anything.

Ah, sweet!

pulumi env run nf-core/github-prod -i pulumi import github:index/repository:Repository nf-core testpipeline

This is the best I'm gonna do. We can iterate in a readable way here.

pulumi import github:index/branchDefault:BranchDefault branch_default_testpipeline testpipeline

pulumi env run nf-core/github-prod -i pulumi import github:index/branch:Branch branch_dev_testpipeline testpipeline:dev

pulumi import github:index/repositoryRuleset:RepositoryRuleset ruleset_branch_default_testpipeline testpipeline:1220601

pulumi import github:index/repositoryRuleset:RepositoryRuleset ruleset_branch_dev_testpipeline testpipeline:1220600

pulumi import github:index/repositoryRuleset:RepositoryRuleset ruleset_branch_TEMPLATE_testpipeline testpipeline:1220597

edmundmiller · 2024-07-22T03:54:39Z

@ewels Ignore the mypy errors will fix that in post. I think this is ready to review and 🚀

pulumi/github/repos/pipelines/testpipeline.py

mashehu

Some thoughts

.github/workflows/repos.yml

docs/1password.md

mashehu · 2024-07-22T07:53:52Z

pulumi/github/repos/Pulumi.dev.yaml

@@ -0,0 +1,5 @@
+config:
+  github:owner: nf-core-tf
+  # https://start.1password.com/open/i?a=O5GICFDKPNABLLVGMKBL5JWDWA&v=rdfcz6oy6qxxrc4clu467a7dmm&i=4ajrv44kc5lcbboa37fr5oydla&h=nf-core.1password.eu


this throws an error for me. is that maybe for your personal account?

Updated the link, you also we're in the Dev vault. Had to make a vault that was specifically accessible to the service accounts, and I didn't want to give them access to everything.

mashehu · 2024-07-22T07:54:36Z

pulumi/github/repos/Pulumi.dev.yaml

+  github:owner: nf-core-tf
+  # https://start.1password.com/open/i?a=O5GICFDKPNABLLVGMKBL5JWDWA&v=rdfcz6oy6qxxrc4clu467a7dmm&i=4ajrv44kc5lcbboa37fr5oydla&h=nf-core.1password.eu
+  github:token:
+    secure: AAABAFMgBNyCNuYsps6YVPV2L7Ji5qBJj0omEQQa9HrdhT2iHo3ex0e9NsDER3Q04itGiY698X/ZQCnTM2zu9op3tcjmzfITdHxGy0FGATuUFamYsSiztHrNAKiIEJ9E0M4Al8/yJeB6X4BXvkLEgik/I+GPvZIXK3tE65Q=


Suggested change

secure: AAABAFMgBNyCNuYsps6YVPV2L7Ji5qBJj0omEQQa9HrdhT2iHo3ex0e9NsDER3Q04itGiY698X/ZQCnTM2zu9op3tcjmzfITdHxGy0FGATuUFamYsSiztHrNAKiIEJ9E0M4Al8/yJeB6X4BXvkLEgik/I+GPvZIXK3tE65Q=

secure: AAABAFMgBNyCNuYsps6YVPV2L7Ji5qBJj0omEQQa9HrdhT2iHo3ex0e9NsDER3Q04itGiY698X/ZQCnTM2zu9op3tcjmzfITdHxGy0FGATuUFamYsSiztHrNAKiIEJ9E0M4Al8/yJeB6X4BXvkLEgik/I+GPvZIXK3tE65Q=

should we set this as a github secret?

We can, the reason this one isn't is because I was struggling with the 1password Pulumi ESC integration, and I didn't realize you have to copy the plain service key into the environment file, and then it encrypts it in place for that specific environment file.

Anyways there's a few options:

GitHub Secret

Pulumi ESC

Encrypting them in place like so(idk if you could run this for example or not)

This one doesn't really matter, because it's just to the nf-core-tf account. I can update it to use Pulumi ESC.

Leaning Pulumi ESC for now as:

That gives us a better access management for the secrets.

It also allows you to develop locally easily, instead of pushing to GitHub anytime you want to preview the changes.

Already have 1Password integration setup with it (So you just pull the secrets in from there instead of copying them, which allows you to roll and update them all in one place)

We could do all of that with GitHub actions, and pass all of these things, but the secret management is already a complicated web, but it's working currently.

TL;DR something to explore, I'll update this one and move it to Pulumi ESC though.

pulumi/github/repos/Pulumi.yaml

pulumi/github/repos/README.md

Co-authored-by: Matthias Hörtenhuber <[email protected]>

Co-authored-by: mashehu <[email protected]>

ewels · 2025-02-03T13:47:01Z

Need to check that we don't clobber certain fields, such as repository descriptions if already written.

Similar for repo keywords: should have a minimal set but not clobber custom ones.

This will vary across fields.

mashehu

needs some cleanup

mashehu · 2025-02-03T14:23:08Z

.github/workflows/repos.yml

+
+      - name: PR previews
+        if: ${{ github.event_name == 'pull_request' }}
+        uses: pulumi/actions@v3


Suggested change

uses: pulumi/actions@v3

uses: pulumi/actions@v6

mashehu · 2025-02-03T14:23:22Z

.github/workflows/repos.yml

+
+      - name: Apply infrastructure update
+        if: ${{ github.event_name == 'push' }}
+        uses: pulumi/actions@v3


Suggested change

uses: pulumi/actions@v3

uses: pulumi/actions@v6

mashehu · 2025-02-03T14:23:55Z

.github/workflows/repos.yml

+      - main
+    paths:
+      - "pulumi/github/repos/**/*"
+  pull_request:


Suggested change

pull_request:

workflow_dispatch:

pull_request:

mashehu · 2025-02-03T14:52:32Z

pulumi/github/repos/core/modules.py

I don't think we need to handle core repos with pulumi

I think not with the "loop" of all of the pipelines. Just as their own individual settings.

I think it can't hurt to import the settings.

mashehu · 2025-02-03T14:53:56Z

pulumi/github/repos/loop_example.py

+    "pairgenomealign",
+    "phaseimpute",
+    "reportho",


Suggested change

"pairgenomealign",

"phaseimpute",

"reportho",

all these are released by now, so better not touch them 🙂

mashehu · 2025-02-03T16:29:05Z

pulumi/github/repos/pipelines/testpipeline.py

+nfcore_testpipeline = github.Repository(
+    NAME,
+    description="A small example pipeline used to test new nf-core infrastructure and common code.",  # 'repo_description' => 'Description must be set',
+    has_downloads=True,


Suggested change

has_downloads=True,

has_downloads=False,

mashehu · 2025-02-03T16:30:13Z

pulumi/github/repos/pipelines/testpipeline.py

+
+CORE_TEAM_ID = 2649377
+MAINTAINERS_TEAM_ID = 4462882
+


Suggested change

repo = github.Repository.get(pipeline, f"nf-core/{pipeline}")

let's first query the old repo, to not overwrite certain fields

I think pulumi has protections for that.

mashehu · 2025-02-03T16:30:32Z

pulumi/github/repos/pipelines/testpipeline.py

+        ),
+    ),
+    visibility="public",
+    topics=TOPICS,  # 'repo_keywords' => 'Minimum keywords set',


Suggested change

topics=TOPICS, # 'repo_keywords' => 'Minimum keywords set',

topics= repo.topics ?? TOPICS, # 'repo_keywords' => 'Minimum keywords set',

could we add missing topics without deleting the existing ones?

yep, that's probably the better approach. the problem is that repo.topics is not just an array, but a pulumi Output object...

mashehu · 2025-02-03T16:31:55Z

pulumi/github/repos/requirements.txt

@@ -0,0 +1,3 @@
+pulumi>=3
+pulumi_github>=5.20.0


Suggested change

pulumi_github>=5.20.0

pulumi_github>=6.6.0

pulumi/github/repos/requirements.txt

mirpedrol · 2025-02-04T09:35:14Z

.github/workflows/repos.yml

+  push:
+    branches:
+      - main
+    paths:
+      - "pulumi/github/repos/**/*"
+  pull_request:
+    branches:
+      - main
+    paths:
+      - "pulumi/github/repos/**/*"


Just to understand. This will run on PRs to ops, right? Is this action updating the pipeline health? And are these PRs on ops related to pipeline health?

mirpedrol · 2025-02-04T09:44:12Z

pulumi/github/repos/pipelines/testpipeline.py

+    description="A small example pipeline used to test new nf-core infrastructure and common code.",  # 'repo_description' => 'Description must be set',
+    has_downloads=True,
+    has_issues=True,  # 'repo_issues' => 'Enable issues',
+    has_projects=True,


I think we don't have this for pipelines?

Suggested change

has_projects=True,

has_projects=False,

I think a lot of them do actually.

mirpedrol · 2025-02-04T09:44:51Z

pulumi/github/repos/pipelines/testpipeline.py

+        ),
+    ),
+    visibility="public",
+    topics=TOPICS,  # 'repo_keywords' => 'Minimum keywords set',


could we add missing topics without deleting the existing ones?

mirpedrol · 2025-02-04T09:46:29Z

pulumi/github/repos/pipelines/testpipeline.py

+# 'branch_master_exists' => 'master branch: branch must exist',
+branch_default_testpipeline = github.BranchDefault(
+    f"branch_default_{NAME}",
+    branch="master",


should we use main already?

Agreed, I think we'll just loop through the pipelines they shouldn't have any special config.

mirpedrol · 2025-02-04T09:46:46Z

pulumi/github/repos/pipelines/testpipeline.py

+        ),
+    ),
+    enforcement="active",
+    name="master",


mirpedrol · 2025-02-04T09:49:08Z

pulumi/github/repos/pipelines/testpipeline.py

+            actor_type="Team",
+            bypass_mode="always",
+        )
+        # TODO 'branch_template_restrict_push' => 'Restrict push to TEMPLATE to @nf-core-bot',


I think we don't need this, everyone can run nf-core sync manually with their github user

Just going for 1 to 1 with pipeline_health.

If I got that wrong, cool, but I vote we give our best attempt to get it the same, then make a PR to update those changes we can get rid of to "self-document" the change.

edmundmiller added 4 commits July 20, 2024 17:57

docs: Get the project started

a985870

chore: Copy over code from teams

c39226a

chore: re-encrypt GitHub token

5a60259

chore: pulumi import github:index/repository:Repository nf-core-tf mo…

77afb52

…dules

edmundmiller self-assigned this Jul 20, 2024

edmundmiller added 5 commits July 20, 2024 18:47

docs: Write up some plans

d43aa80

Add missing repos

abff7e7

chore: Sort lines

20514de

chore: Add pipelines

33c8a50

edmundmiller force-pushed the pipeline_health branch from 28320ba to bd199e9 Compare July 21, 2024 00:26

ewels reviewed Jul 21, 2024

View reviewed changes

pulumi/github/repos/README.md Outdated Show resolved Hide resolved

ewels reviewed Jul 21, 2024

View reviewed changes

pulumi/github/repos/core_repos.yml Show resolved Hide resolved

ewels reviewed Jul 21, 2024

View reviewed changes

pulumi/github/repos/pipelines.yml Show resolved Hide resolved

Add demo and testpipeline

005acec

edmundmiller added 12 commits July 21, 2024 11:32

Add Old Pipeline Health PHP code

1578293

Add docs on using 1password to automatically switch contexts

4b25a4a

feat: Add prod esc environment

f759855

chore: Remove template

feaf7d4

chore: Import nf-core/testpipeline

4141c81

pulumi env run nf-core/github-prod -i pulumi import github:index/repository:Repository nf-core testpipeline

refactor: Move testpipeline into it's own file

5f20c05

Take a stab at some settings

e5f3540

Add Specs from php code

56ac14d

This is the best I'm gonna do. We can iterate in a readable way here.

chore: Import default branch

df0656d

pulumi import github:index/branchDefault:BranchDefault branch_default_testpipeline testpipeline

chore: Import testpipeline template branch

eb4b899

chore: Import testpipeline dev branch

f9aa261

pulumi env run nf-core/github-prod -i pulumi import github:index/branch:Branch branch_dev_testpipeline testpipeline:dev

chore: Check off some TODOs

41db6e0

edmundmiller added 10 commits July 21, 2024 14:29

chore: Import Master Branch protection

bda6686

pulumi import github:index/repositoryRuleset:RepositoryRuleset ruleset_branch_default_testpipeline testpipeline:1220601

chore: Import dev branch ruleset

296ab65

pulumi import github:index/repositoryRuleset:RepositoryRuleset ruleset_branch_dev_testpipeline testpipeline:1220600

chore: Import template branch ruleset for testpipeline

cccbc96

pulumi import github:index/repositoryRuleset:RepositoryRuleset ruleset_branch_TEMPLATE_testpipeline testpipeline:1220597

chore: Remove duplicate TODO

c786f5b

chore: Match up specs with code

4e3785b

feat: Update dev branch requirements

a3611b4

refactor: Move required CI Checks out

6ca606d

feat: Add contributors and core permissions

8dc9365

refactor: Use NAME variable where ever possible

bbb1a42

Remove protect on repo

71e5762

edmundmiller changed the title ~~pipeline health~~ pipeline health POC Jul 22, 2024

edmundmiller marked this pull request as ready for review July 22, 2024 03:16

edmundmiller requested a review from a team as a code owner July 22, 2024 03:16

edmundmiller added 3 commits July 21, 2024 22:27

ci: Add template workflow for repos

78e9aab

ci: Remove python-version

855be9b

style: Run pre-commit

77019ba

ewels reviewed Jul 22, 2024

View reviewed changes

pulumi/github/repos/pipelines/testpipeline.py Show resolved Hide resolved

mashehu reviewed Jul 22, 2024

View reviewed changes

edmundmiller and others added 4 commits July 22, 2024 12:56

Apply suggestions from code review

aa24ea0

Co-authored-by: Matthias Hörtenhuber <[email protected]>

chore: Add 1password link

5fdd914

docs: Clean up README moving planning to GH issue

2e80246

Co-authored-by: mashehu <[email protected]>

refactor: Use Pulumi ESC for dev

d475858

edmundmiller mentioned this pull request Jul 22, 2024

Pulumi ESC or GitHub Secrets #60

Open

docs: Add some nf-core Pulumi starter info

9eebe4b

maxulysse approved these changes Jan 31, 2025

View reviewed changes

mashehu reviewed Feb 3, 2025

View reviewed changes

mirpedrol reviewed Feb 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pipeline health POC #59

pipeline health POC #59

edmundmiller commented Jul 20, 2024 •

edited

Loading

ewels commented Jul 21, 2024

ewels commented Jul 21, 2024

edmundmiller commented Jul 21, 2024

edmundmiller commented Jul 22, 2024

mashehu left a comment

mashehu Jul 22, 2024

edmundmiller Jul 22, 2024

mashehu Jul 22, 2024

edmundmiller Jul 22, 2024

ewels commented Feb 3, 2025

mashehu left a comment

mashehu Feb 3, 2025

mashehu Feb 3, 2025

mashehu Feb 3, 2025

mashehu Feb 3, 2025

edmundmiller Feb 25, 2025

mashehu Feb 3, 2025

mashehu Feb 3, 2025

mashehu Feb 3, 2025

edmundmiller Feb 25, 2025

mashehu Feb 3, 2025

mirpedrol Feb 4, 2025

mashehu Feb 4, 2025

mashehu Feb 3, 2025

mirpedrol Feb 4, 2025

mirpedrol Feb 4, 2025

edmundmiller Feb 25, 2025

mirpedrol Feb 4, 2025

mirpedrol Feb 4, 2025

edmundmiller Feb 25, 2025

mirpedrol Feb 4, 2025

mirpedrol Feb 4, 2025

edmundmiller Feb 25, 2025

	secure: AAABAFMgBNyCNuYsps6YVPV2L7Ji5qBJj0omEQQa9HrdhT2iHo3ex0e9NsDER3Q04itGiY698X/ZQCnTM2zu9op3tcjmzfITdHxGy0FGATuUFamYsSiztHrNAKiIEJ9E0M4Al8/yJeB6X4BXvkLEgik/I+GPvZIXK3tE65Q=
	secure: AAABAFMgBNyCNuYsps6YVPV2L7Ji5qBJj0omEQQa9HrdhT2iHo3ex0e9NsDER3Q04itGiY698X/ZQCnTM2zu9op3tcjmzfITdHxGy0FGATuUFamYsSiztHrNAKiIEJ9E0M4Al8/yJeB6X4BXvkLEgik/I+GPvZIXK3tE65Q=


	repo = github.Repository.get(pipeline, f"nf-core/{pipeline}")

	topics=TOPICS, # 'repo_keywords' => 'Minimum keywords set',
	topics= repo.topics ?? TOPICS, # 'repo_keywords' => 'Minimum keywords set',


		CORE_TEAM_ID = 2649377
		MAINTAINERS_TEAM_ID = 4462882

pipeline health POC #59

Are you sure you want to change the base?

pipeline health POC #59

Conversation

edmundmiller commented Jul 20, 2024 • edited Loading

ewels commented Jul 21, 2024

ewels commented Jul 21, 2024

edmundmiller commented Jul 21, 2024

edmundmiller commented Jul 22, 2024

mashehu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ewels commented Feb 3, 2025

mashehu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edmundmiller commented Jul 20, 2024 •

edited

Loading