Add GitHub Actions workflow for testing the build on various Linux distributions via docker; macOS with homebrew; and Windows (cygwin)

[mkoeppe]

This is a follow-up on #29053: Add debian/fedora/arch package information to build/pkgs, generate Dockerfiles and installation help

This ticket adds .github/workflows/tox.yml, which installs a number of system packages and then runs ./bootstrap, ./configure, and make.

Most of the interesting parts are in #29053 - everything can be run "locally" (with tox and docker). This ticket only adds a simple additional file to run this automatically with several jobs on GitHub Actions on a GitHub pull request.

For each of the Linux distributions supported, there are two configurations:

• minimal - has enough packages so that the build can succeed;
• standard - has additionally all standard packages for which we have spkg-configure.m4.

GitHub Actions runs these build jobs on 2-core machines with 7 GB of RAM memory and 14 GB of SSD disk space (https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners#supported-runners-and-hardware-resources), and has a time limit of 6h per job. This is just barely enough for a typical minimal build followed by make ptest to succeed; and plenty of time for a typical standard build to succeed.

Example run: https://github.com/mkoeppe/sage/actions/runs/60588138

Context:

• Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages #29060: Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages
• Meta-ticket: spkg-configure: Try to use as many system packages as possible #27330: Meta-ticket: spkg-configure: Try to use as many system packages as possible

Depends on #29367

CC: @dimpase @vbraun @saraedum @isuruf @antonio-rojas @kiwifb

Component: build

Keywords: ContinuousIntegration

Author: Matthias Koeppe

Branch/Commit: cbcd657

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29087

[mkoeppe]

Dependencies: #29053

[saraedum]

comment:2

I am all for more CI, but would it not make more sense to add this to the existing GitLab CI?

[mkoeppe]

Branch: u/mkoeppe/CI-gh-actions-tox-configure-build

[mkoeppe]

Last 10 new commits:

80d2f90	build/pkgs/debian-bootstrap.txt: Add pkg-config
d31441e	remove nonexisting fedora package
e4c2bf8	build/pkgs/arch.txt: Add gcc
c55071f	build/pkgs/debian.txt: add g++
ea04ee7	build/pkgs/debian.txt: python3-minimal->python3
b4e90a0	build/bin/write-dockerfile.sh: Only install a package if it has spkg-configure.m4
0f2b16e	build/bin/sage-spkg: Explain to user what will be done with the system package
ddcac27	Squashed commit of the following:
8c30f07	Arch package sage-data-elliptic_curves corresponds to Sage's elliptic_curves, not to the optional database_cremona_ellcurve
6a2bf3d	Add [GitHub](../wiki/GitHub) Actions workflow for testing spkg-configure / build on various Linux distributions

[mkoeppe]

Commit: 6a2bf3d

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

comment:5

Replying to @saraedum:

I am all for more CI, but would it not make more sense to add this to the existing GitLab CI?

Fine with me too. I have built this on top of tox and docker. It can also run locally. The GitHub-specific part is really minimal - https://github.com/sagemath/sagetrac-mirror/commits/6a2bf3d2140e2d625d4ca96a40485b74fa582e7a

[mkoeppe]

Changed keywords from none to ContinuousIntegration

[saraedum]

comment:7

Personally, I like GitHub. I like GitLab a bit better because it is very easy to provision our own runners (e.g., for obscure platforms.) The problem with GitHub is that some people have quite a few reservations about it (lock-in, Microsoft, …) so I think GitLab is the better choice for such an extensive community.

In any case, glad to see these things happen. Feel free to ping/bug me explicitly if you need reviews on anything docker/CI.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8592e01	build/pkgs/fedora.txt: Add findutils, which
5442c6f	Add [GitHub](../wiki/GitHub) Actions workflow for testing spkg-configure / build on various Linux distributions

[sagetrac-git]

Changed commit from 6a2bf3d to 5442c6f

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

3d9565f	fixup conda
7e59a6f	build/bin/write-dockerfile.sh: Add commands to activate conda env
2ad9a97	Merge tag '9.1.beta2' into t/29053/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time
9fca230	add doc
f6cbb8a	add centos-{7,8}
a9b1ff2	Add [GitHub](../wiki/GitHub) Actions workflow for testing spkg-configure / build on various Linux distributions

[sagetrac-git]

Changed commit from 5442c6f to a9b1ff2

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

c651058	on centos, IGNORE_MISSING_SYSTEM_PACKAGES=yes
98edff6	Add build/pkgs/gfortran/distros/fedora.txt
843ca76	build/pkgs/fedora.txt: Add python3
bd6a545	tox.ini: Add debian stretch, put sid last
23c6334	build/pkgs/openblas/distros/arch.txt: Add lapack, cblas
050d692	Add [GitHub](../wiki/GitHub) Actions workflow for testing spkg-configure / build on various Linux distributions
744b04e	.github/workflows/tox.yml: Also put the tox log in the artifact

[sagetrac-git]

Changed commit from a9b1ff2 to 744b04e

[mkoeppe]

Changed dependencies from #29053 to #29053, #29104, #29100

[sagetrac-git]

Changed commit from 744b04e to 879d1a4

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

fca6654	tox.ini: Add toxenvs local-homebrew-macos-minimal, local-homebrew-macos-standard
e0e8a4c	tox.ini [local-homebrew]: Fixup PATH, PKG_CONFIG_PATH, set LDFLAGS, CPPFLAGS; [local-direct]: New
16ca88f	tox.ini: Use {envlogdir}; use ln -sf
6a1803d	tox.ini [local]: First build the base-toolchain
4b72987	tox.ini [local-homebrew]: Set PATH earlier and set ACLOCAL_PATH for bootstrapping
043c2fa	Merge branch 't/29104/tox-homebrew' into t/29087/CI-gh-actions-tox-configure-build
931cdf8	spkg-configure.m4 for boost_cropped and boost
6f04521	add package names to distros/
998287b	Merge branch 'u/dimpase/packages/boostconfig' of git://trac.sagemath.org/sage into t/29087/CI-gh-actions-tox-configure-build
879d1a4	.github/workflows/tox.yml: Add tox-local-macos

[mkoeppe]

comment:14

Replying to @saraedum:

Personally, I like GitHub. I like GitLab a bit better because it is very easy to provision our own runners (e.g., for obscure platforms.) The problem with GitHub is that some people have quite a few reservations about it (lock-in, Microsoft, …) so I think GitLab is the better choice for such an extensive community.

Yes, I understand these concerns. To avoid lock-in, in this ticket the definition of the tests (for linux) is driven by tox and docker, rather than relying on the specific environments provided by a CI vendor. Likewise for the homebrew tests for macOS on this ticket.

I don't have the time to work on another CI platform but would certainly welcome it if other CI enthusiasts wanted to implement a solution on another platform in a follow-up ticket.

Feel free to ping/bug me explicitly if you need reviews on anything docker/CI.

Thanks. This ticket would be ready for review.

[sagetrac-git]

Changed commit from 879d1a4 to f311a93

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

f311a93

.github/workflows/tox.yml: Add rpy2, ecl, maxima to TARGETS; also run on pull_request

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

2b9eb79	Merge tag '9.1.beta3' into t/29087/CI-gh-actions-tox-configure-build
839d6fb	tox.ini: Add other archs supported by docker
be0bf8d	Merge branch 't/29143/add_to_tox_ini_some_32_bit_linux_environments' into t/29087/CI-gh-actions-tox-configure-build
9b8c45f	tox.ini: Add a few more systems

[dimpase]

comment:70

how does one control the platforms the tests are run on? Often one needs to check on a couple of different OSs at most, and the current setup looks quite wasteful.

[sagetrac-git]

Changed commit from 0f7c9e0 to 4dcb568

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

4dcb568

tox.yml [docker]: Set SAGE_NUM_THREADS to address timeouts in doctests

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

65834f0	Reorder systems so debian-sid comes after other debians
d95a688	tox.ini [ubuntu-focal]: Use IGNORE_MISSING_SYSTEM_PACKAGES=yes because of libpython3.7-dev

[sagetrac-git]

Changed commit from 4dcb568 to d95a688

[sagetrac-git]

Changed commit from d95a688 to 9939051

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

3896794	Merge tag '9.1.beta8' into t/29354/tox_ini__add_slackware_environments
bda9ebd	Add slackware package info
fbb7393	Fixup packages
cfcd8f2	build/bin/sage-guess-package-system, build/bin/sage-print-system-package-command: Handle slackware too
136feeb	build/bin/sage-spkg: Show equivalent system packages for ALL distros
9939051	Merge branch 't/29354/tox_ini__add_slackware_environments' into t/29087/CI-gh-actions-tox-configure-build

[mkoeppe]

comment:74

Replying to @dimpase:

how does one control the platforms the tests are run on? Often one needs to check on a couple of different OSs at most, and the current setup looks quite wasteful.

By editing the section called matrix in the file .github/workflows/tox.yml.

[mkoeppe]

comment:75

https://github.com/mkoeppe/sage/runs/526226655:

----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 3638.0 seconds
    cpu time: 5482.8 seconds
    cumulative wall time: 10347.2 seconds
Error processing tar file(exit status 1): write /sage/src/build/temp.linux-x86_64-3.7/build/cythonized/sage/rings/finite_rings/integer_mod.o: no space left on device
ERROR: InvocationError: '/usr/bin/docker build . -f /home/runner/work/sage/sage/.tox/docker-ubuntu-trusty-minimal/Dockerfile --build-arg EXTRA_CONFIGURE_ARGS=   --build-arg BASE_IMAGE=ubuntu:trusty --build-arg USE_MAKEFLAGS=-k V=0 SAGE_NUM_THREADS=3 --build-arg TARGETS=build ptest'

docker build ends with "no space left on device" · Issue #18144 · moby/moby
moby/moby#18144

Error pulling image (...) no space left on device · Issue #10613 · moby/moby
moby/moby#10613

dockerd | Docker Documentation
https://docs.docker.com/engine/reference/commandline/dockerd/#storage-driver-options

Configure and troubleshoot the Docker daemon | Docker Documentation
https://docs.docker.com/config/daemon/

[dimpase]

Reviewer: Dima Pasechnik

[mkoeppe]

Changed reviewer from Dima Pasechnik to none

[mkoeppe]

comment:77

sysinfo at https://github.com/mkoeppe/sage/runs/526929826

[mkoeppe]

Reviewer: Dima Pasechnik

[dimpase]

comment:80

this needs more documentation, I gather - on a followup ticket, something to be mentioned in README.md too. But this is good to go, I think.

[mkoeppe]

comment:81

Thanks! I've created #29401 - Add documentation

[dimpase]

comment:82

this needs a rebase.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

cbcd657

Merge tag '9.1.beta9' into t/29415/tox_ini__add_environment_local_conda

[sagetrac-git]

Changed commit from 9939051 to cbcd657

[mkoeppe]

comment:84

Merged 9.1.beta9

[mkoeppe]

Changed dependencies from #29104, #29304, #29295, #29301, #29354, #29352, #29367 to #29367

[vbraun]

Changed branch from u/mkoeppe/CI-gh-actions-tox-configure-build to cbcd657