Build Stages and allow_failures do not work well together #7789

BanzaiMan · 2017-05-19T22:13:58Z

Problem

When Build Stages is defined, as well as allow_failures, queuing order is incorrect.

configuration

language: php
sudo: false
dist: trusty
jobs:
  fast_finish: true
  allow_failures:
    - php: "hhvm"
  include:
    - php: "7.0"
    - php: "7.1"
    - php: "hhvm"
    - stage: next stage
      php: "5.6"

script: if [[ $TRAVIS_PHP_VERSION = "hhvm" ]]; then false; else true; fi

Expected result

The default "Test" stage contains 3 jobs, 7.0, 7.1, and hhvm, and they are queued first. In this particular case, the hhvm job fails with false but is allowed to fail, so upon completion of the "Test" stage, the execution moves on to the "next stage". This, too, will pass, and the build is marked a success.

Actual result

https://travis-ci.org/BanzaiMan/travis_production_test/builds/234179350

The "Test" stage contains 2 jobs ("7.0", and "7.1"), which are queued first. They pass, and the execution moves on to the "next stage". The "5.6" job passes, and the "hhvm" job fails; the latter is allowed to fail, though, so the build is marked a pass.

Preliminary discussion

It appears that the logic to queue allowed-to-fail jobs is taking precedence over the stage queuing logic.

In addition, the allowed-to-fail jobs have lost their visual representation to explicitly mark the job allowed to fail. (This is a separate table in the normal build configuration, but such separation in the Build Stage display might be confusing and undesirable.

The text was updated successfully, but these errors were encountered:

backspace · 2017-05-19T22:36:06Z

We tried to address the visual acknowledgement of allowed failures with these lines:

webknjaz · 2017-05-22T04:35:10Z

@backspace what about highlighting the exact jobs, which jobs exactly are allowed to fail?

svenfuchs · 2017-05-23T10:16:20Z

@BanzaiMan the issue here is that we move allowed failures to the very end of the matrix during matrix expansion ... i.e. before we then assign stages.

I believe the easiest way to resolve this current issue would be to revert https://github.com/travis-ci/travis-gatekeeper/commit/0530ae96c9117c5cab9cb0f70a7aa4bb229d060e ... which probably was based on a user request. The workaround there could be for them to sort allowed failure jobs to the end of the matrix in their .travis.yml file.

In order to keep the existing behaviour AND have it work well with stages we'd have to do some refactoring. This part of the code base is ... rather interesting and could use some love. I'm not 100% sure about the priorities.

svenfuchs · 2017-05-24T10:34:47Z

This should be fixed.

The .travis.yml for the following matrix lists the rvm: 2.3 job first, but it now orders it to the end of the stage, not the end of the entire build matrix. It also displays the allow the failure message (appeared only after the job has failed)

This change is now in production.

webknjaz · 2017-05-24T15:19:04Z

Brilliant!

As an initial step in continuous integration testing for Vincent's cross-platform work. Once we get this passing on OS X we can remove the allow_failures entry. The best docs I could find for using allow_failures with build stages was [1], so that's the pattern I'm following. [1]: travis-ci/travis-ci#7789 (comment) Signed-off-by: W. Trevor King <[email protected]>

As an initial step in continuous integration testing for Vincent's cross-platform work. Once we get this passing on OS X we can remove the allow_failures entry. The best docs I could find for using allow_failures with build stages was [1], so that's the pattern I'm following. Travis' OS X docs are in [2]. [1]: travis-ci/travis-ci#7789 (comment) [2]: https://docs.travis-ci.com/user/reference/osx Signed-off-by: W. Trevor King <[email protected]>

As an initial step in continuous integration testing for Vincent's cross-platform work. Once we get this passing on OS X we can remove the allow_failures entry. The best docs I could find for using allow_failures with build stages was [1], so that's the pattern I'm following. Travis' OS X docs are in [2]. The TRAVIS_OS_NAME approach is recommended in [3]. We're not going to actually use any of the storage drivers yet, so we don't need to install btrfs, etc. And the OS X workers already come with automake [4]. We'll see how things work with the other requirements missing [1]: travis-ci/travis-ci#7789 (comment) [2]: https://docs.travis-ci.com/user/reference/osx [3]: https://docs.travis-ci.com/user/installing-dependencies/#Installing-Dependencies-on-Multiple-Operating-Systems [4]: https://docs.travis-ci.com/user/reference/osx#Compilers-and-Build-toolchain Signed-off-by: W. Trevor King <[email protected]>

blueyed · 2018-08-16T19:26:06Z

I think it would be good to have some indicator before that stage finished already about which jobs are allowed to fail - got me confused, since I've thought allow_failures was not honored anymore at all.
("Your build matrix was set to allow the failure of job 1566.12." only shows up after the stage / this job finished).

krlmlr · 2018-10-03T07:56:23Z

I second @blueyed's comment -- it took me an hour to find out that allowed_failures is honored, just not shown by default.

webknjaz · 2018-10-03T08:43:41Z

@blueyed @krlmlr I believe that it's better to post such suggestions on https://travis-ci.community/c/product to give it more visibility.

@svenfuchs is probably going to miss any messages in the closed issue.

krlmlr · 2018-10-04T11:46:36Z

Done: https://travis-ci.community/t/always-show-allow-failures-allowed-failures-when-build-stages-are-used/217.

Travis now offers stages. Using stages we can: - Run the code style related checks before running any unit tests and stop the build early if any are detected. - Remove the duplicate unit test runs - i.e. previously we had an extra (third) build against PHP 7.2 (now changed to 7.3) which would run the code style related checks, but would also re-run the unit tests. This extra build will now no longer run the unit tests. While this does mean that the unit tests will run with a slight delay (the `Sniff` stage has to finish before they start), it also means that we: * Get code style errors reported earlier as it's been moved to be the first stage and the build will just stop if any are found. * We won't be wasting Travis's resources on builds which wil have to be re-run anyway. Ref: https://docs.travis-ci.com/user/build-stages/ Note that `Allowed failures` is no longer listed as a separate block in the Travis result overview, but is _is_ respected. For more discussion about this: * travis-ci/travis-ci#7789 * https://travis-ci.community/t/always-show-allow-failures-allowed-failures-when-build-stages-are-used/217 * https://travis-ci.community/t/work-out-kinks-in-interactions-between-stages-allow-fail-and-fast-finish/1090 * travis-ci/travis-ci#9677

Travis now offers stages. Using stages we can: - Run the code style related checks before running any unit tests and stop the build early if any are detected. - Remove the duplicate unit test runs - i.e. previously we had an extra (third) build against PHP 7.2 (now changed to 7.3) which would run the code style related checks, but would also re-run the unit tests. This extra build will now no longer run the unit tests. While this does mean that the unit tests will run with a slight delay (the `Sniff` and `Rulesets` stages have to finish before they start), it also means that we: * Get code style errors reported earlier as it's been moved to be the first stage and the build will just stop if any are found. * We won't be wasting Travis's resources on builds which will have to be re-run anyway. Ref: https://docs.travis-ci.com/user/build-stages/ Note that `Allowed failures` is no longer listed as a separate block in the Travis result overview, but is _is_ respected. For more discussion about this: * travis-ci/travis-ci#7789 * https://travis-ci.community/t/always-show-allow-failures-allowed-failures-when-build-stages-are-used/217 * https://travis-ci.community/t/work-out-kinks-in-interactions-between-stages-allow-fail-and-fast-finish/1090 * travis-ci/travis-ci#9677

Travis now offers stages. Using stages we can: - Run the code style related checks before running any unit tests and stop the build early if any are detected. - Only run the (heavy) code coverage builds when we already know that the unit tests for most build pass. While this does mean that the unit tests will run with a slight delay (the `Sniff` stage has to finish before they start), it also means that we: * Get code style errors reported earlier as it's been moved to be the first stage and the build will just stop if any are found. * We won't be wasting Travis's resources on builds which will have to be re-run anyway. * The output of the Travis jobs will be a lot easier to decipher as it only shows the output related to that particular stage. Ref: https://docs.travis-ci.com/user/build-stages/ Note that `Allowed failures` is no longer listed as a separate block in the Travis result overview, but is _is_ respected. For more discussion about this: * travis-ci/travis-ci#7789 * https://travis-ci.community/t/always-show-allow-failures-allowed-failures-when-build-stages-are-used/217 * https://travis-ci.community/t/work-out-kinks-in-interactions-between-stages-allow-fail-and-fast-finish/1090 * travis-ci/travis-ci#9677 In addition to this, this PR: * Removes `sudo: false` as it is no longer supported. This switches the build form container-based builds to VM based. See: https://blog.travis-ci.com/2018-11-19-required-linux-infrastructure-migration * Groups the two separate `cache` directives together. * Simplifies the setting of the `env` variable for the PHPUnit version for those PHP versions which cannot use the Travis image default PHPUnit version. * Removes the global `COVERALLS_VERSION` `env` variable which is no longer needed. * Updates the PHPCS versions in the "mix it up" builds. * Removes `composer self-update` which is no longer needed. Travis does this by default. * Adds `--no-update` to the additional Composer requires so we only invoke `composer install` once, instead of potentially four times. * Attempts to make the script more readable by using multi-line conditions whenever the line would become quite long.

Travis now offers stages. Using stages we can: - Run the code style related checks before running any unit tests and stop the build early if any are detected. - Only run the (heavy) code coverage builds when we already know that the unit tests for most builds pass. While this does mean that the unit tests will run with a slight delay (the `Sniff` stage has to finish before they start), it also means that we: * Get code style errors reported earlier as it's been moved to be the first stage and the build will just stop if any are found. * We won't be wasting Travis's resources on builds which will have to be re-run anyway. * The output of the Travis jobs will be a lot easier to decipher as it only shows the output related to that particular stage. Ref: https://docs.travis-ci.com/user/build-stages/ Note that `Allowed failures` is no longer listed as a separate block in the Travis result overview, but is _is_ respected. For more discussion about this: * travis-ci/travis-ci#7789 * https://travis-ci.community/t/always-show-allow-failures-allowed-failures-when-build-stages-are-used/217 * https://travis-ci.community/t/work-out-kinks-in-interactions-between-stages-allow-fail-and-fast-finish/1090 * travis-ci/travis-ci#9677 In addition to this, this PR: * Removes `sudo: false` as it is no longer supported. This switches the build form container-based builds to VM based. See: https://blog.travis-ci.com/2018-11-19-required-linux-infrastructure-migration * Groups the two separate `cache` directives together. * Simplifies the setting of the `env` variable for the PHPUnit version for those PHP versions which cannot use the Travis image default PHPUnit version. * Removes the global `COVERALLS_VERSION` `env` variable which is no longer needed. * Updates the PHPCS versions in the "mix it up" builds. * Removes `composer self-update` which is no longer needed. Travis does this by default. * Adds `--no-update` to the additional Composer requires so we only invoke `composer install` once, instead of potentially four times. * Updates the PHP `lint` command to have simpler output (no more the long list of files). * Attempts to make the script more readable by using multi-line conditions whenever the line would become quite long.

Travis now offers stages. Using stages we can: - Run the code style related checks before running any unit tests and stop the build early if any are detected. - Only run the (heavy) code coverage builds when we already know that the unit tests for most builds pass. While this does mean that the unit tests will run with a slight delay (the `Sniff` stage has to finish before they start), it also means that we: * Get code style errors reported earlier as it's been moved to be the first stage and the build will just stop if any are found. * We won't be wasting Travis's resources on builds which will have to be re-run anyway. * The output of the Travis jobs will be a lot easier to decipher as it only shows the output related to that particular stage. Ref: https://docs.travis-ci.com/user/build-stages/ Note that `Allowed failures` is no longer listed as a separate block in the Travis result overview, but is _is_ respected. For more discussion about this: * travis-ci/travis-ci#7789 * https://travis-ci.community/t/always-show-allow-failures-allowed-failures-when-build-stages-are-used/217 * https://travis-ci.community/t/work-out-kinks-in-interactions-between-stages-allow-fail-and-fast-finish/1090 * travis-ci/travis-ci#9677 In addition to this, this PR makes the following changes to the Travis script: * Removes `sudo: false` as it is no longer supported. This switches the build form container-based builds to VM based. See: https://blog.travis-ci.com/2018-11-19-required-linux-infrastructure-migration * Groups the two separate `cache` directives together. * Simplifies the setting of the `env` variable for the PHPUnit version for those PHP versions which cannot use the Travis image default PHPUnit version. * Removes the global `COVERALLS_VERSION` `env` variable which is no longer needed. * Updates the PHPCS versions in the "mix it up" builds. * Removes `composer self-update` which is no longer needed. Travis does this by default. * Adds `--no-update` to the additional Composer requires so we only invoke `composer install` once, instead of potentially four times. * Updates the PHP `lint` command to have simpler output (no more the long list of files). * Attempts to make the script more readable by using multi-line conditions whenever the line would become quite long. It also removes the `phpunit-travis.xml` file and moves the additional config into the normal `phpunit.xml.dist` file. To run PHPUnit without code coverage, either run it on a PHP version without Xdebug enabled or run it with the `--no-coverage` command-line flag. This flag has also been added to the relevant commands in the Travis build script.

Travis now offers stages. Using stages we can: - Run the code style related checks before running any unit tests and stop the build early if any are detected. While this does mean that the unit tests will run with a slight delay (the `Sniff` stage has to finish before they start), it also means that we: * Get code style errors reported earlier as it's been moved to be the first stage and the build will just stop if any are found. * We won't be wasting Travis's resources on builds which will have to be re-run anyway. * The output of the Travis jobs will be a lot easier to decipher as it only shows the output related to that particular stage. Ref: https://docs.travis-ci.com/user/build-stages/ Note that `Allowed failures` is no longer listed as a separate block in the Travis result overview, but is _is_ respected. For more discussion about this: * travis-ci/travis-ci#7789 * https://travis-ci.community/t/always-show-allow-failures-allowed-failures-when-build-stages-are-used/217 * https://travis-ci.community/t/work-out-kinks-in-interactions-between-stages-allow-fail-and-fast-finish/1090 * travis-ci/travis-ci#9677

Travis now offers stages. Using stages we can: - Run the code style related checks before running any unit tests and stop the build early if any are detected. - Remove the duplicate unit test runs - i.e. previously we had an extra (third) build against PHP 7.1 (now changed to 7.3) which would run the code style related checks, but would also re-run the unit tests. This extra build will now no longer run the unit tests. - Add a separate "quicktest" stage for non-PR/merge builds. The "quicktest" stage will only run a CS check, ruleset check, linting and the unit tests against low/high PHP/PHPCS/WPCS combinations. This should catch most issues. The more comprehensive complete build against a larger combination of PHP/PHPCS/WPCS combination will now only be run on PRs and merges to `develop`/`master`. While this does mean that the unit tests will run with a slight delay (the `Sniff` stage has to finish before they start), it also means that we: * Get code style errors reported earlier as it's been moved to be the first stage and the build will just stop if any are found. * We won't be wasting Travis's resources on builds which will have to be re-run anyway. Ref: https://docs.travis-ci.com/user/build-stages/ Note that `Allowed failures` is no longer listed as a separate block in the Travis result overview, but is _is_ respected. For more discussion about this: * travis-ci/travis-ci#7789 * https://travis-ci.community/t/always-show-allow-failures-allowed-failures-when-build-stages-are-used/217 * https://travis-ci.community/t/work-out-kinks-in-interactions-between-stages-allow-fail-and-fast-finish/1090 * travis-ci/travis-ci#9677

BanzaiMan added bug build stages scheduler labels May 19, 2017

BanzaiMan assigned svenfuchs and BanzaiMan May 19, 2017

BanzaiMan mentioned this issue May 19, 2017

Build Stages: Flexible and practical Continuous Delivery pipelines travis-ci/beta-features#11

Closed

This was referenced May 22, 2017

Change Travis CI config to use newly presented build stages feature cherrypy/cheroot#29

Merged

[TODO] Change Travis CI config to use newly presented build stages feature cherrypy/cherrypy#1594

Closed

svenfuchs closed this as completed May 24, 2017

wking mentioned this issue Mar 7, 2018

.travis.yml: Add allowed-to-fail OS X compilation cri-o/cri-o#1428

Merged

localheinz mentioned this issue Jul 1, 2018

Use build stages for travis builds infection/infection#399

Merged

nicoddemus mentioned this issue Feb 6, 2019

Add py38-dev job to Travis pytest-dev/pytest#4731

Merged

jrfnl mentioned this issue May 22, 2019

Travis: run the code style related and ruleset checks in separate stages WordPress/WordPress-Coding-Standards#1711

Merged

jrfnl mentioned this issue May 22, 2019

Travis: run the code style related checks in separate stage PHPCompatibility/PHPCompatibility#798

Merged

jrfnl mentioned this issue May 23, 2019

Travis: run the code style related and ruleset checks in separate stages WPTT/WPThemeReview#223

Merged

jrfnl mentioned this issue Jul 31, 2019

Travis: run the code style related checks in separate stages and other improvements Yoast/yoastcs#150

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build Stages and allow_failures do not work well together #7789

Build Stages and allow_failures do not work well together #7789

BanzaiMan commented May 19, 2017

backspace commented May 19, 2017

webknjaz commented May 22, 2017

svenfuchs commented May 23, 2017

svenfuchs commented May 24, 2017

webknjaz commented May 24, 2017

blueyed commented Aug 16, 2018

krlmlr commented Oct 3, 2018

webknjaz commented Oct 3, 2018

krlmlr commented Oct 4, 2018

Build Stages and allow_failures do not work well together #7789

Build Stages and allow_failures do not work well together #7789

Comments

BanzaiMan commented May 19, 2017

Problem

configuration

Expected result

Actual result

Preliminary discussion

backspace commented May 19, 2017

webknjaz commented May 22, 2017

svenfuchs commented May 23, 2017

svenfuchs commented May 24, 2017

webknjaz commented May 24, 2017

blueyed commented Aug 16, 2018

krlmlr commented Oct 3, 2018

webknjaz commented Oct 3, 2018

krlmlr commented Oct 4, 2018