Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Move CI infrastructure to GitHub Actions #3321

Draft
wants to merge 68 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
952d3d9
Move to `manylinux_2_28` base image
JCGoran Jan 10, 2025
0b01fd8
Test manylinux_2_28 image
JCGoran Jan 10, 2025
38aa6c4
Modify Dockerfile and build script
JCGoran Jan 13, 2025
c7ae29e
Remove BBP-specific testing
JCGoran Jan 13, 2025
2571c65
Set min version of GCC everywhere
JCGoran Jan 13, 2025
d29bf6f
Change tag of Docker image
JCGoran Jan 13, 2025
de3fcc8
Cleanup Docker image to reduce size
JCGoran Jan 13, 2025
44ef470
MPI headers need some tweaking
JCGoran Jan 13, 2025
4ffe46c
Fix checking of compiler
JCGoran Jan 13, 2025
fae53da
Update docs and CI vars
JCGoran Jan 14, 2025
e95b63f
Add CI for building wheels on MacOS on arm64
JCGoran Nov 15, 2024
1e0293f
Add merging of artifacts to wheel CI job
JCGoran Nov 18, 2024
4bc8e31
Add `pattern: wheels-*` to merge job in CI
JCGoran Nov 18, 2024
1a8e344
Append `matrix.os` to artifact name
JCGoran Nov 19, 2024
c8b9277
Update .github/workflows/wheels.yml
JCGoran Dec 5, 2024
f461d85
Add other configs to CI
JCGoran Jan 28, 2025
cbeb0ce
Remove azure and circleCI
JCGoran Jan 28, 2025
14f925d
Fix CI issue
JCGoran Jan 28, 2025
bb8ac58
Do not use git shenanigans when building wheels
JCGoran Jan 28, 2025
de7d057
Try normal Python on MacOS
JCGoran Jan 28, 2025
dcfb8a1
Minor fixes
JCGoran Jan 28, 2025
3481ccb
Typo
JCGoran Jan 28, 2025
3acfbe5
Maybe quotes?
JCGoran Jan 28, 2025
9f41e67
Maybe dots are the problem
JCGoran Jan 28, 2025
9303ff6
Separate build and test steps
JCGoran Jan 28, 2025
30c69a2
Forgot steps
JCGoran Jan 28, 2025
d1ace17
Revert back to testing after build
JCGoran Jan 28, 2025
3e1817c
Fix missing property
JCGoran Jan 28, 2025
75a30b9
Remove container
JCGoran Jan 28, 2025
5ada0ee
Merge into one big job
JCGoran Jan 28, 2025
f14461f
Merge branch 'master' into jelic/move_infra_to_gha
JCGoran Jan 28, 2025
ecb1fe8
Merge branch 'master' into jelic/move_infra_to_gha
JCGoran Feb 10, 2025
541ec98
Update docs and some CI
JCGoran Feb 10, 2025
d63e0a8
Merge branch 'master' into jelic/move_infra_to_gha
JCGoran Feb 11, 2025
f5f0cd5
Fix typo
JCGoran Feb 11, 2025
afc840b
Schedule does not respect branches
JCGoran Feb 11, 2025
1043444
Better `if` in CI
JCGoran Feb 11, 2025
52f5869
Try out commenting after merging artifacts
JCGoran Feb 11, 2025
87bd9ca
Debugging
JCGoran Feb 11, 2025
eab0aeb
Replace azure with artifacts URL in external.yml
JCGoran Feb 11, 2025
5e5d23f
Revert "Debugging"
JCGoran Feb 11, 2025
8cd011d
Revert this later
JCGoran Feb 11, 2025
114f6d3
Try suggestion
JCGoran Feb 11, 2025
b2a296e
${{ stuff }} or stuff? Who cares, use bash
JCGoran Feb 11, 2025
154055d
Idk, maybe type is reserved?
JCGoran Feb 11, 2025
89bc43d
Maybe it works now
JCGoran Feb 11, 2025
7458984
Fix env var issue
JCGoran Feb 11, 2025
a2351eb
Workaround for gha or setuptools-scm being broken
JCGoran Feb 11, 2025
2c8eda5
Remove NRN_RELEASE_UPLOAD env var
JCGoran Feb 11, 2025
43c7f3a
Add the uploading step
JCGoran Feb 11, 2025
be5eedd
Only make comment if on a PR
JCGoran Feb 11, 2025
7d4e36d
Fix typo
JCGoran Feb 11, 2025
51954ca
Add option to use a custom base image
JCGoran Feb 11, 2025
b2ccf79
Add back MacOS to test it
JCGoran Feb 11, 2025
9f0d828
Use two variables for manylinux images
JCGoran Feb 12, 2025
84b47c5
Maybe `matrix.os`?
JCGoran Feb 12, 2025
fdc7de8
Use `fetch-depth` for custom image as well
JCGoran Feb 12, 2025
841c757
Put back azure pipeline to compare
JCGoran Feb 12, 2025
6e77ff4
Forgot the other one
JCGoran Feb 12, 2025
b5c5417
Checkout self and submodules
JCGoran Feb 12, 2025
6b49704
Once again azure is broken
JCGoran Feb 12, 2025
accacd9
Use the other SHA in workflow
JCGoran Feb 24, 2025
1cc77d6
Pin setuptools-scm
JCGoran Feb 24, 2025
ac27b07
Revert "Pin setuptools-scm"
JCGoran Feb 24, 2025
c5144ef
Merge remote-tracking branch 'origin/master' into jelic/move_infra_to…
JCGoran Feb 24, 2025
d4b2129
[REVERT WHEN READY] try test upload on workflow
JCGoran Feb 25, 2025
7d29ef3
Revert previous commit now that I debugged it
JCGoran Feb 25, 2025
832f5f3
Put back all of the other Python versions
JCGoran Feb 26, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 0 additions & 98 deletions .circleci/config.yml

This file was deleted.

2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/ci.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: NEURON CI bug
about: Bugs related to the NEURON CIs (GitHub Actions, Azure, ReadTheDocs, CodeCov)
about: Bugs related to the NEURON CIs (GitHub Actions, ReadTheDocs, CodeCov)
title: ''
labels: 'bug, CI'
assignees: ''
Expand Down
11 changes: 5 additions & 6 deletions .github/ISSUE_TEMPLATE/release-patch.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,25 +19,24 @@ Pre-release
- [ ] Update changelog below and agree on it with everyone; then commit it to `docs/changelog` in the cherrypicks PR (copy structure as-is)
- [ ] Update `docs/index.rst` accordingly with the new `.pkg` and `.exe` links for `PKG installer` and `Windows Installer`
- [ ] Activate ReadtheDocs for the cherry-pick branch and ensure the documentation builds (when logged in, go to [the versions page](https://readthedocs.org/projects/nrn/versions/) and set the version for your branch to Active and Hidden)
- [ ] Run a test wheel build WITHOUT upload on the cherry-pick branch to ensure all the wheels build ([see details](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-azure))
- [ ] Run a test wheel build WITHOUT upload on the cherry-pick branch to ensure all the wheels build ([see details](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-github-actions))

Sanity checks
---
- [ ] After cherrypicks PR is merged, make sure GitHub, Azure and CircleCI builds pass for `release/x.y` branch
- [ ] Run [nrn-build-ci](https://github.com/neuronsimulator/nrn-build-ci/actions/workflows/build-neuron.yml) for the `release/x.y` branch; see [nrn-build-ci guide](https://github.com/neuronsimulator/nrn-build-ci#azure-wheels-testing---manual-workflow)
- [ ] After cherrypicks PR is merged, make sure GitHub Actions builds pass for `release/x.y` branch
- [ ] Run [nrn-build-ci](https://github.com/neuronsimulator/nrn-build-ci/actions/workflows/build-neuron.yml) for the `release/x.y` branch; see [nrn-build-ci guide](https://github.com/neuronsimulator/nrn-build-ci#wheels-testing---manual-workflow)
- [ ] Activate ReadTheDocs build for `release/x.y` & make it hidden. Check docs are fine after build is done.
- [ ] Run BBP Simulation Stack & other relevant tests


Releasing
---
- [ ] Create new release+tag on GitHub via [release workflow](https://github.com/neuronsimulator/nrn/actions/workflows/release.yml?query=workflow%3A%22NEURON+Release%22). Note that the GitHub release will be marked as pre-release and will contain the full-src-package and the Windows installer at the end of the release workflow.
- [ ] Build release wheels but WITHOUT upload ([see details](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-azure))
- [ ] Build release wheels but WITHOUT upload ([see details](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-github-actions))
- [ ] Create, test and upload manual artifacts
- [ ] MacOS package installer (manual task, ask Michael)
- [ ] arm64 wheels (manual task, check with Erik, Goran or Pramod)
- [ ] aarch64 wheels (use existing `release/x.y-aarch64` branch for this, see [guide](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-circleci))
- [ ] Publish the `x.y.z` wheels on PyPI; see [wheel publishing instructions](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-azure)
- [ ] Publish the `x.y.z` wheels on PyPI; see [wheel publishing instructions](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-github-actions)
- [ ] Once wheels are published, activate the `x.y.z` tag on ReadTheDocs
- [ ] Rename the Windows installer in the GitHub release to match the new version and the supported python versions (i.e. `nrn-8.2.2.w64-mingw-py-39-310-311-312-setup.exe`)
- [ ] Publish release on GitHub (edit https://github.com/neuronsimulator/nrn/releases/tag/x.y.z and un-tick the pre-release checkbox)
Expand Down
12 changes: 5 additions & 7 deletions .github/ISSUE_TEMPLATE/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,10 @@ Pre-release

Sanity checks
---
- [ ] Create `release/x.y` branch and make sure GitHub, Azure and CircleCI builds pass
- [ ] Run [nrn-build-ci](https://github.com/neuronsimulator/nrn-build-ci/actions/workflows/build-neuron.yml) for the respective Azure build; see [Azure drop guide](https://github.com/neuronsimulator/nrn-build-ci#azure-wheels-testing---manual-workflow)
- [ ] Create `release/x.y` branch and make sure GitHub Actions builds pass
- [ ] Run [nrn-build-ci](https://github.com/neuronsimulator/nrn-build-ci/actions/workflows/build-neuron.yml) for the respective GitHub Actions build; see [the guide](https://github.com/neuronsimulator/nrn-build-ci#wheels-testing---manual-workflow)
- [ ] Activate ReadTheDocs build for `release/x.y` & make it hidden. Check docs are fine after build is done.
- [ ] Run a test wheel build WITHOUT upload for `release/x.y` to ensure all the wheels build ([see details](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-azure))
- [ ] Run BBP Simulation Stack & other relevant tests
- [ ] Run a test wheel build WITHOUT upload for `release/x.y` to ensure all the wheels build ([see details](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-github-actions))


Releasing
Expand All @@ -30,12 +29,11 @@ Releasing
- [ ] Update `docs/index.rst` accordingly with the new `.pkg` and `.exe` links for `PKG installer` and `Windows Installer`
- [ ] Run the ReadTheDocs build again for `release-x.y`, make sure the build passes and inspect the Changelog page.
- [ ] Create new release+tag on GitHub via [release workflow](https://github.com/neuronsimulator/nrn/actions/workflows/release.yml?query=workflow%3A%22NEURON+Release%22). Note that the GitHub release will be marked as pre-release and will contain the full-src-package and the Windows installer at the end of the release workflow.
- [ ] Build release wheels but WITHOUT upload ([see details](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-azure))
- [ ] Build release wheels but WITHOUT upload ([see details](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-github-actions))
- [ ] Create, test and upload manual artifacts
- [ ] MacOS package installer (manual task, ask Michael)
- [ ] arm64 wheels (manual task, check with Erik, Goran or Pramod)
- [ ] aarch64 wheels (create a `release/x.y-aarch64` branch for this, see [guide](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-circleci))
- [ ] Publish the `x.y.z` wheels on Pypi; see [wheel publishing instructions](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-azure)
- [ ] Publish the `x.y.z` wheels on Pypi; see [wheel publishing instructions](https://nrn.readthedocs.io/en/latest/install/python_wheels.html#publishing-the-wheels-on-pypi-via-github-actions)
- [ ] Once wheels are published, activate the `x.y.z` tag on ReadTheDocs
- [ ] Rename the Windows installer in the GitHub release to match the new version and the supported python versions (i.e. `nrn-8.2.2.w64-mingw-py-39-310-311-312-setup.exe`
)
Expand Down
40 changes: 20 additions & 20 deletions .github/workflows/external.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,26 +13,26 @@ env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

jobs:
get-last-azure-url:
get-last-artifact-url:
runs-on: ubuntu-latest
if: ${{ github.event.label.name == 'nrn-modeldb-ci-nightly' }}
outputs:
azure_drop_url: ${{ steps.drop.outputs.azure_drop_url }}
pr_azure_sha: ${{ steps.drop.outputs.pr_azure_sha }}
artifact_url: ${{ steps.drop.outputs.artifact_url }}
last_commit_sha: ${{ steps.drop.outputs.last_commit_sha }}
steps:
- id: drop
run: |
# use jq to get the last Azure drop URL from the PR and the SHA1 from the same body
export pr_json=$(gh pr view $PR_URL --json comments -q 'last(.comments[] .body | capture(".*(?<pr_azure_sha>[0-9a-f]{40}).*?(?<azure_drop_url>https://dev.azure.com/neuronsimulator/.*=zip)"))')
# use jq to get the last artifact URL from the PR and the SHA1 from the same body
export pr_json=$(gh pr view $PR_URL --json comments -q 'last(.comments[] .body | capture(".*(?<last_commit_sha>[0-9a-f]{40}).*?(?<artifact_url>https://github.com/neuronsimulator/nrn/actions/runs/.*)"))')
if [ -z "$pr_json" ]; then
echo "Unable to retrieve AZURE drop url and SHA from comments!"
echo "Unable to retrieve artifact URL and SHA from comments!"
exit 1
fi
# use jq to get pr_azure_sha and azure_drop_url from the json
export azure_drop_url=$(echo $pr_json | jq -r .azure_drop_url)
export pr_azure_sha=$(echo $pr_json | jq -r .pr_azure_sha)
echo azure_drop_url=$azure_drop_url >> $GITHUB_OUTPUT
echo pr_azure_sha=$pr_azure_sha >> $GITHUB_OUTPUT
# use jq to get last_commit_sha and artifact_url from the json
export artifact_url=$(echo $pr_json | jq -r .artifact_url)
export last_commit_sha=$(echo $pr_json | jq -r .last_commit_sha)
echo artifact_url=$artifact_url >> $GITHUB_OUTPUT
echo last_commit_sha=$last_commit_sha >> $GITHUB_OUTPUT

- id: remove-label
if: always()
Expand All @@ -41,31 +41,31 @@ jobs:
gh pr edit $PR_URL --remove-label nrn-modeldb-ci-nightly
# if we encounter an error in last github action step, add a comment
if [ ${{ steps.drop.outcome }} == 'failure' ]; then
gh pr comment $PR_URL --body "Unable to retrieve AZURE drop url from comments!"
gh pr comment $PR_URL --body "Unable to retrieve artifact URL from comments!"
else
gh pr comment $PR_URL --body "NEURON ModelDB CI: launching for ${pr_azure_sha} via its [drop url]($azure_drop_url)"
gh pr comment $PR_URL --body "NEURON ModelDB CI: launching for ${last_commit_sha} via its [artifact URL]($artifact_url)"
fi
env:
pr_azure_sha: ${{ steps.drop.outputs.pr_azure_sha }}
azure_drop_url: ${{ steps.drop.outputs.azure_drop_url }}
last_commit_sha: ${{ steps.drop.outputs.last_commit_sha }}
artifact_url: ${{ steps.drop.outputs.artifact_url }}

nrn-modeldb-ci:
needs: get-last-azure-url
needs: get-last-artifact-url
uses: neuronsimulator/nrn-modeldb-ci/.github/workflows/nrn-modeldb-ci.yaml@master
with:
neuron_v1: ${{needs.get-last-azure-url.outputs.azure_drop_url}}
neuron_v1: ${{needs.get-last-artifact-url.outputs.artifact_url}}
neuron_v2: neuron-nightly

pr-update:
needs:
- nrn-modeldb-ci
- get-last-azure-url
- get-last-artifact-url
runs-on: ubuntu-latest
steps:
- run: |
gh pr comment $PR_URL --body "NEURON ModelDB CI: ${pr_azure_sha} -> download reports [from here](${ARTIFACTS_URL})"
gh pr comment $PR_URL --body "NEURON ModelDB CI: ${last_commit_sha} -> download reports [from here](${ARTIFACTS_URL})"
name: Post NEURON ModelDB CI Artifact URL
if: always() || cancelled()
env:
ARTIFACTS_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
pr_azure_sha: ${{ needs.get-last-azure-url.outputs.pr_azure_sha }}
last_commit_sha: ${{ needs.get-last-artifact-url.outputs.last_commit_sha }}
Loading
Loading