Add Python 3.9 for unit tests in CI (experimental, do not stop other builds if it fails) #3854

kinow · 2020-10-05T22:09:46Z

This is a small change with no associated Issue.

This is how GH actions handles experimental Java versions. I guess it should work with Python versions too…

Requirements check-list

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Already covered by existing tests.
No change log entry required (why? e.g. invisible to users).
No documentation update required.
No dependency changes.

kinow · 2020-10-05T22:15:38Z

(Should fail for now as I think GH hasn't added support to 3.9 yet)

kinow · 2020-10-05T22:36:48Z

The bash container is failing to build due to… virtual memory I think?

2020-10-05T22:17:29.2744656Z    write (2, "virtual memory exhausted\n", 25);
2020-10-05T22:17:29.2744959Z    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-05T22:17:29.4147064Z �[0mgcc -c -g -O2 -Wno-parentheses -Wno-format-security -DHAVE_CONFIG_H  -I. -I../.. -I../.. -I../../lib -I. tparam.c
2020-10-05T22:17:29.4414545Z �[91mtparam.c: In function ‘memory_out’:
2020-10-05T22:17:29.4421362Z �[0m�[91mtparam.c:67:3: warning: implicit declaration of function ‘write’ [-Wimplicit-function-declaration]
2020-10-05T22:17:29.4422068Z    write (2, "virtual memory exhausted\n", 25);

kinow · 2020-10-06T01:22:47Z

Ha! https://docs.docker.com/engine/reference/run/#runtime-constraints-on-resources

Interesting, I never had to limit memory utilization of containers. Today-I-Learned 🎉

kinow · 2020-10-06T01:40:33Z

.github/workflows/bash.yml

-          docker build -t bash:test dockerfiles/bash/
-          docker run --name bash -v $(pwd -P):/root/cylc-flow --rm -t -d bash:test /bin/bash
+          docker build -t bash:test  -m 1G --memory-swap 1G dockerfiles/bash/
+          docker run --name bash -v $(pwd -P):/root/cylc-flow --rm -t -d -m 1G --memory-swap 1G bash:test /bin/bash


Was getting some errors due to low virtual memory when building the container. Not sure if others have seen this, but might be good to constraint resources so our builds don't fail sporadically.

kinow · 2020-10-06T01:41:31Z

(low priority, happy to leave this one waiting until GH has 3.9 support if others prefer 👍 )

hjoliver · 2020-10-06T02:54:29Z

.github/workflows/test.yml

+        experimental: [false]
+        include:
+          - python-version: 3.9
+            experimental: true


Why are python-version and experimental lists above, but scalar values here?

Oh, hadn't noticed it. Both appear to work fine. But let me edit that commit to match the other values 👍

Done, just need to wait for CI to run and confirm it works too.

(I see the GH Actions docs does it the way you originally did it, so maybe it works either way 🤞 )

Bugger:

Error: .github/workflows/test.yml (Line: 25, Col: 27): A sequence was not expected Error: The template is not valid. .github/workflows/test.yml (Line: 25, Col: 27): A sequence was not expected

But good to know! I guess if you have multiple versions, then you will have to do something like

- python-version: 1 experimental: true - python-version: 2 experimental: true ...

What does the "experimental" mean exactly?

It's just a feature in GitHub actions that allows to mark a branch of the matrix-tree as not-ready I guess? It's really common in Java and JavaScript, for new versions of the JVM or of Node (which are experimental in relation to the code; meaning that they may not work, and the developers/CI are "experimenting" with it? 😄 )

OK, fair enough. Just metadata then, no real consequences 👍

Can remove the experimental thing now 3.9 is out.

hjoliver

The 3.9 block fails, as expected. Any opinions on whether or not we should merge like this or wait till GH-A adds 3.9 support? (It might look a bit odd to have expected test fails all the time?)

kinow · 2020-10-06T21:16:01Z

The 3.9 block fails, as expected. Any opinions on whether or not we should merge like this or wait till GH-A adds 3.9 support? (It might look a bit odd to have expected test fails all the time?)

We can review/approve and merge later.

They have some pull requests in GH actions virtual environments repo for Ubuntu 18.04 and 20.04, but the next updates don't seem to include Python 3.9. I think first it needs to make its way to Ubuntu package repositories, then they will create a PR to merge?

https://github.com/actions/virtual-environments/pulls

hjoliver · 2020-10-06T21:20:20Z

I'll label this as "blocked" for now, pending 3.9 on GH-A.

kinow · 2020-10-06T21:25:18Z

Ignore last commits. I added one commit to test something I saw in GH actions docs for setup-python:

python-version: '3.9.0-alpha - 3.9.0' # SemVer's version range syntax

Didn't work 😬 reverted

kinow · 2020-11-04T02:35:27Z

I think the 3.9 unit tests will pass now, actions/setup-python#148

kinow · 2020-11-04T10:23:43Z

CI passed, two approvals. Merging 🎉

kinow force-pushed the py39-experimental branch from 4cc816d to 08df9be Compare October 5, 2020 22:14

kinow self-assigned this Oct 5, 2020

kinow added this to the cylc-8.0a3 milestone Oct 5, 2020

kinow marked this pull request as ready for review October 6, 2020 01:39

kinow changed the title ~~Add Python 3.9 in CI (experimental)~~ Add Python 3.9 for unit tests in CI (experimental, do not stop other builds if it fails) Oct 6, 2020

kinow commented Oct 6, 2020

View reviewed changes

kinow requested review from oliver-sanders and hjoliver October 6, 2020 01:40

hjoliver reviewed Oct 6, 2020

View reviewed changes

kinow mentioned this pull request Oct 6, 2020

Fail the query with a helpful message if the variables defined do not match the expected values #3856

Merged

7 tasks

kinow force-pushed the py39-experimental branch 3 times, most recently from 10048f4 to b8a4821 Compare October 6, 2020 03:13

oliver-sanders approved these changes Oct 6, 2020

View reviewed changes

hjoliver approved these changes Oct 6, 2020

View reviewed changes

hjoliver added the BLOCKED This can't happen until something else does label Oct 6, 2020

kinow force-pushed the py39-experimental branch from 4a62ed6 to b8a4821 Compare October 6, 2020 21:24

kinow added 2 commits November 4, 2020 15:27

Add Python 3.9 in CI (experimental)

50c8a9a

Try restricting memory of Docker containers for bash tests

496c211

kinow force-pushed the py39-experimental branch from b8a4821 to 496c211 Compare November 4, 2020 02:28

kinow removed the BLOCKED This can't happen until something else does label Nov 4, 2020

kinow merged commit f656c97 into cylc:master Nov 4, 2020

kinow deleted the py39-experimental branch November 4, 2020 10:24

hjoliver modified the milestones: cylc-8.0a3, cylc-8.0b0 Feb 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Python 3.9 for unit tests in CI (experimental, do not stop other builds if it fails) #3854

Add Python 3.9 for unit tests in CI (experimental, do not stop other builds if it fails) #3854

kinow commented Oct 5, 2020 •

edited

Loading

kinow commented Oct 5, 2020

kinow commented Oct 5, 2020

kinow commented Oct 6, 2020

kinow Oct 6, 2020

kinow commented Oct 6, 2020

hjoliver Oct 6, 2020

kinow Oct 6, 2020

kinow Oct 6, 2020

hjoliver Oct 6, 2020 •

edited

Loading

hjoliver Oct 6, 2020

kinow Oct 6, 2020

hjoliver Oct 6, 2020

kinow Oct 6, 2020

hjoliver Oct 6, 2020

oliver-sanders Nov 4, 2020

hjoliver left a comment

kinow commented Oct 6, 2020

hjoliver commented Oct 6, 2020

kinow commented Oct 6, 2020 •

edited

Loading

kinow commented Nov 4, 2020 •

edited

Loading

kinow commented Nov 4, 2020

Add Python 3.9 for unit tests in CI (experimental, do not stop other builds if it fails) #3854

Add Python 3.9 for unit tests in CI (experimental, do not stop other builds if it fails) #3854

Conversation

kinow commented Oct 5, 2020 • edited Loading

kinow commented Oct 5, 2020

kinow commented Oct 5, 2020

kinow commented Oct 6, 2020

Choose a reason for hiding this comment

kinow commented Oct 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjoliver Oct 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjoliver left a comment

Choose a reason for hiding this comment

kinow commented Oct 6, 2020

hjoliver commented Oct 6, 2020

kinow commented Oct 6, 2020 • edited Loading

kinow commented Nov 4, 2020 • edited Loading

kinow commented Nov 4, 2020

kinow commented Oct 5, 2020 •

edited

Loading

hjoliver Oct 6, 2020 •

edited

Loading

kinow commented Oct 6, 2020 •

edited

Loading

kinow commented Nov 4, 2020 •

edited

Loading