Skip to content

Commit 19e32e0

Browse files
LexLuthrLexLuthr
authored
chore: switch to Github Actions (#1933)
* add github actions * fix attempt 1 * fix attempt 2 * fix attempt 3 * fix attempt 4 * fix attempt 5 * fix attempt 6 * fix attempt 7 * fix attempt 8 * fix attempt 9 * fix attempt 10 * fix attempt 11 * fix attempt 12 * fix attempt 13 * fix attempt 14 * fix attempt 15 * fix attempt 16 * fix attempt 17 * fix lint errors * make deps for lint * fix fmt * test self-hosted * self-hosted runner 1 * self-hosted runner 2 * self-hosted runner 3 * self-hosted runner 4 * self-hosted runner 5 * add CI linter, yugabyte cleanup * add setup files * disable shellcheck * fix systemd file * update setup instructions * test without go cache
1 parent 15f5241 commit 19e32e0

17 files changed

+610
-135
lines changed

‎.github/actionlint.yaml

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
self-hosted-runner:
2+
# Labels of self-hosted runner in array of string
3+
labels:
4+
- docker

‎.github/actions/container-builder/action.yaml

-48
This file was deleted.

‎.github/actions/install-deps/action.yml

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: 'Install Dependencies'
2+
description: 'Install common dependencies'
3+
4+
runs:
5+
using: 'composite'
6+
steps:
7+
- name: Install dependencies
8+
run: |
9+
sudo apt-get update
10+
sudo apt-get install -y openssh-client git ocl-icd-opencl-dev libhwloc-dev
11+
shell: bash
12+
13+
- name: Fetch all tags
14+
run: git fetch --all
15+
shell: bash
16+
17+
- name: Sync submodules
18+
run: git submodule sync
19+
shell: bash
20+
21+
- name: Update submodules
22+
run: git submodule update --init
23+
shell: bash
24+

‎.github/actions/setup-go/action.yml

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: 'Setup Go'
2+
description: 'Setup Go environment'
3+
4+
inputs:
5+
go-version:
6+
description: 'Go version to use'
7+
required: true
8+
9+
runs:
10+
using: 'composite'
11+
steps:
12+
- name: Set up Go
13+
uses: actions/setup-go@v5
14+
with:
15+
go-version: ${{ inputs.go-version }}
16+
cache: false

‎.github/image/Dockerfile

+74
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
## This image is used to github action runner for this repo
2+
# Use the official Golang image as the base image
3+
FROM golang:1.22-bullseye AS builder
4+
5+
# Set the working directory inside the container
6+
WORKDIR /app
7+
8+
# Install Git
9+
RUN apt-get update && apt-get install -y git
10+
11+
# Clone the repositories
12+
RUN git clone https://github.com/filecoin-project/boost.git
13+
RUN git clone https://github.com/filecoin-project/curio.git
14+
15+
# Download Go dependencies for Boost
16+
WORKDIR /app/boost
17+
RUN git submodule update --init
18+
RUN go mod download
19+
20+
# Download Go dependencies for Curio
21+
WORKDIR /app/boost
22+
RUN git submodule update --init
23+
RUN go mod download
24+
25+
# Stage 2: Install Lotus binary
26+
FROM ghcr.io/filecoin-shipyard/lotus-containers:lotus-v1.28.1-devnet AS lotus-test
27+
28+
# Stage 3: Build the final image
29+
FROM myoung34/github-runner AS curio-github-runner
30+
31+
# Copy Go dependencies from the builder stage
32+
COPY --from=builder /go/pkg /go/pkg
33+
COPY --from=builder /go/bin /go/bin
34+
COPY --from=lotus-test /usr/local/bin/lotus /usr/local/bin/
35+
36+
RUN apt update && apt install -y \
37+
build-essential \
38+
bzr pkg-config \
39+
clang \
40+
curl \
41+
gcc git \
42+
hwloc \
43+
jq \
44+
libhwloc-dev wget \
45+
mesa-opencl-icd \
46+
ocl-icd-opencl-dev
47+
48+
ENV RUSTUP_HOME=/usr/local/rustup \
49+
CARGO_HOME=/usr/local/cargo \
50+
PATH=/usr/local/cargo/bin:$PATH \
51+
RUST_VERSION=1.76.0
52+
53+
RUN set -eux; \
54+
dpkgArch="$(dpkg --print-architecture)"; \
55+
case "${dpkgArch##*-}" in \
56+
amd64) rustArch='x86_64-unknown-linux-gnu'; rustupSha256='0b2f6c8f85a3d02fde2efc0ced4657869d73fccfce59defb4e8d29233116e6db' ;; \
57+
armhf) rustArch='armv7-unknown-linux-gnueabihf'; rustupSha256='f21c44b01678c645d8fbba1e55e4180a01ac5af2d38bcbd14aa665e0d96ed69a' ;; \
58+
arm64) rustArch='aarch64-unknown-linux-gnu'; rustupSha256='673e336c81c65e6b16dcdede33f4cc9ed0f08bde1dbe7a935f113605292dc800' ;; \
59+
i386) rustArch='i686-unknown-linux-gnu'; rustupSha256='e7b0f47557c1afcd86939b118cbcf7fb95a5d1d917bdd355157b63ca00fc4333' ;; \
60+
*) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; \
61+
esac; \
62+
url="https://static.rust-lang.org/rustup/archive/1.26.0/${rustArch}/rustup-init"; \
63+
wget "$url"; \
64+
echo "${rustupSha256} *rustup-init" | sha256sum -c -; \
65+
chmod +x rustup-init; \
66+
./rustup-init -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION --default-host ${rustArch}; \
67+
rm rustup-init; \
68+
chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \
69+
rustup --version; \
70+
cargo --version; \
71+
rustc --version;
72+
73+
# Allow attaching a volume for the specified path
74+
VOLUME /var/tmp/filecoin-proof-parameters

‎.github/utils/boost-github-actions-runner.env

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
RUNNER_SCOPE=repo
2+
REPO_URL=https://github.com/filecoin-project/boost
3+
LABELS=docker
4+
ACCESS_TOKEN=<ADD TOKEN HERE>
5+
RUNNER_WORKDIR=/tmp/runner/work
6+
DISABLE_AUTO_UPDATE=1
7+
EPHEMERAL=1

‎.github/utils/create-runner-image.sh

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/bin/bash
2+
3+
# Directory where the GitHub Action runner will be set up
4+
RUNNER_DIR="/tmp/github-runner"
5+
6+
# Clone the necessary repositories and set up the runner
7+
setup_github_runner() {
8+
# Remove any existing directory
9+
rm -rf $RUNNER_DIR
10+
11+
# Create the directory
12+
mkdir -p $RUNNER_DIR
13+
14+
# Navigate to the directory
15+
# shellcheck disable=SC2164
16+
cd $RUNNER_DIR
17+
18+
# Clone the required repositories
19+
git clone https://github.com/filecoin-project/curio.git
20+
git clone https://github.com/filecoin-project/boost.git
21+
22+
# Copy necessary files
23+
cp -r boost/.github/utils/* .
24+
25+
# Copy the Dockerfile
26+
cp boost/.github/image/Dockerfile .
27+
28+
# Build the Docker image
29+
docker buildx build -t curio/github-runner:latest .
30+
}
31+
32+
# Execute the function
33+
setup_github_runner

‎.github/utils/curio-github-actions-runner.env

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
RUNNER_SCOPE=repo
2+
REPO_URL=https://github.com/filecoin-project/curio
3+
LABELS=docker
4+
ACCESS_TOKEN=<TOKEN>
5+
RUNNER_WORKDIR=/tmp/runner/work
6+
DISABLE_AUTO_UPDATE=1
7+
EPHEMERAL=1

‎.github/utils/github-actions-runner.service

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
[Unit]
2+
Description=Ephemeral GitHub Actions Runner Container
3+
After=docker.service
4+
Requires=docker.service
5+
[Service]
6+
TimeoutStartSec=0
7+
Restart=always
8+
ExecStartPre=-/usr/bin/docker stop github-runnerNUM
9+
ExecStartPre=-/usr/bin/docker rm github-runnerNUM
10+
#ExecStartPre=-/usr/bin/docker pull curio/github-runner:latest
11+
ExecStart=/usr/bin/docker run --rm \
12+
--env-file /etc/curio-github-actions-runner.env \
13+
-e RUNNER_NAME=docker-runnerNUM \
14+
-v /var/run/docker.sock:/var/run/docker.sock \
15+
-v /opt/filecoin-proof-parameters:/var/tmp/filecoin-proof-parameters \
16+
--name github-runnerNUM \
17+
curio/github-runner:latest
18+
[Install]
19+
WantedBy=multi-user.target

‎.github/utils/setup.md

+77
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
## How to set up Github Action runner
2+
1. Clone this repo and copy out the files in this directory to one level above
3+
```shell
4+
mkdir github-runner
5+
6+
cd github-runner
7+
8+
git clone https://github.com/filecoin-project/curio.git
9+
git clone https://github.com/filecoin-project/boost.git
10+
11+
cp -r boost/.github/utils/* .
12+
```
13+
2. Copy the Dockerfile
14+
15+
```shell
16+
copy boost/.github/image/Dockerfile .
17+
```
18+
19+
3. Create new image
20+
21+
```shell
22+
docker buildx build -t curio/github-runner:latest .
23+
```
24+
25+
4. Create systemd file. Please ensure to equal number files for Boost and Curio. If server can host 10 runner then 5 should be for Boost and 5 for Curio.
26+
27+
```shell
28+
for i in {1..5}; do cat github-actions-runner.service | sed "s/NUM/$i/g" > github-actions-runner$i.service; done
29+
for i in {6..10}; do cat github-actions-runner.service | sed 's/curio-/boost-/g' | sed "s/NUM/$i/g" > github-actions-runner$i.service; done
30+
for i in {1..10}; do install -m 644 github-actions-runner$i.service /etc/systemd/system/ ; done
31+
systemctl daemon-reload
32+
```
33+
5. Add the token to ENV files
34+
6. Copy the ENV files to /etc
35+
36+
```shell
37+
cp boost-github-actions-runner.env
38+
cp curio-github-actions-runner.env
39+
```
40+
41+
7. Start and Enable the services
42+
```shell
43+
for i in {1..10}; do systemctl start github-actions-runner$i.service; done
44+
for i in {1..10}; do systemctl status github-actions-runner$i.service; done
45+
for i in {1..10}; do systemctl enable github-actions-runner$i.service; done
46+
```
47+
48+
8. Verify that new runners are visible in the repo.
49+
50+
## Set up docker image creator
51+
1. Make the script executable
52+
```shell
53+
cd github-runner
54+
chmod +x create-runner-image.sh
55+
```
56+
2. Create a cron job to update the image every day
57+
```shell
58+
crontab -e
59+
```
60+
61+
```shell
62+
0 0 * * * /root/github-runner/create-runner-image.sh
63+
```
64+
65+
## Github Token
66+
Creating GitHub personal access token (PAT) for using by self-hosted runner make sure the following scopes are selected:
67+
68+
```text
69+
repo (all)
70+
admin:public_key - read:public_key
71+
admin:repo_hook - read:repo_hook
72+
admin:org_hook
73+
notifications
74+
workflow
75+
```
76+
77+
## This setup is based in the https://github.com/myoung34/docker-github-actions-runner

0 commit comments

Comments
 (0)