Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add exhaustive/extensive tests #388

Merged
merged 3 commits into from
Jan 6, 2025
Merged

Add exhaustive/extensive tests #388

merged 3 commits into from
Jan 6, 2025

Conversation

tgross35
Copy link
Contributor

@tgross35 tgross35 commented Jan 2, 2025

Add a generator that will test all inputs for input spaces u32::MAX or smaller (e.g. single-argument f32 routines). For anything larger, still run approximately u32::MAX tests, but distribute inputs evenly across the function domain.

Since we often only want to run one of these tests at a time, this implementation parallelizes within each test using rayon. A custom test runner is used so a progress bar is possible.

Specific tests must be enabled by setting the LIBM_EXTENSIVE_TESTS environment variable, e.g.

LIBM_EXTENSIVE_TESTS=all_f16,cos,cosf cargo run --features test-multiprecision --release

Testing on a recent machine, most tests take about two minutes or less. The Bessel functions are quite slow and take closer to 10 minutes, and FMA is increased to run for about the same.

This uses the libtest_mimc harness so a progress bar can be printed. Example:

running 90 tests
test mp_extensive_acos       ... ignored, extensive tests are only run if specified in RUST_LIBM_EXTENSIVE
test mp_extensive_acosf      ... starting extensive tests for `acosf`
[9m  100%] #################### acosf 4,294,967,295/4,294,967,295: 7,672,836.6102/s eta 0s
ok
test mp_extensive_acosh      ... ignored, extensive tests are only run if specified in RUST_LIBM_EXTENSIVE
test mp_extensive_acoshf     ... starting extensive tests for `acoshf`
[5m  73 %] ###############----- acoshf 3,139,620,000/4,294,967,295: 3,544,521.3401/s eta 5m (1.6715916e36,)

Recreated from #364.

@tgross35 tgross35 force-pushed the tgross35/extensive branch 2 times, most recently from 13b009c to 3295bc4 Compare January 2, 2025 23:55
@tgross35 tgross35 force-pushed the tgross35/extensive branch 3 times, most recently from 3295bc4 to 1a91e87 Compare January 3, 2025 09:53
@tgross35 tgross35 force-pushed the tgross35/extensive branch from 1a91e87 to ab5f7bd Compare January 3, 2025 12:54
@tgross35 tgross35 force-pushed the tgross35/extensive branch from ab5f7bd to f0c4aa3 Compare January 4, 2025 00:25
@tgross35 tgross35 force-pushed the tgross35/ntests branch 3 times, most recently from c93054b to ac3ff8c Compare January 6, 2025 01:51
Base automatically changed from tgross35/ntests to master January 6, 2025 02:38
@tgross35 tgross35 force-pushed the tgross35/extensive branch 5 times, most recently from 1e01833 to 66d061b Compare January 6, 2025 07:09
@tgross35 tgross35 force-pushed the tgross35/extensive branch 4 times, most recently from 3f51474 to a521311 Compare January 6, 2025 08:15
Add a generator that will test all inputs for input spaces `u32::MAX` or
smaller (e.g. single-argument `f32` routines).  For anything larger,
still run approximately `u32::MAX` tests, but distribute inputs evenly
across the function domain.

Since we often only want to run one of these tests at a time, this
implementation parallelizes within each test using `rayon`. A custom
test runner is used so a progress bar is possible.

Specific tests must be enabled by setting the `LIBM_EXTENSIVE_TESTS`
environment variable, e.g.

    LIBM_EXTENSIVE_TESTS=all_f16,cos,cosf cargo run ...

Testing on a recent machine, most tests take about two minutes or less.
The Bessel functions are quite slow and take closer to 10 minutes, and
FMA is increased to run for about the same.
@tgross35 tgross35 force-pushed the tgross35/extensive branch from a521311 to 4985826 Compare January 6, 2025 08:22
@tgross35 tgross35 force-pushed the tgross35/extensive branch from 4985826 to 386c2be Compare January 6, 2025 08:29
@tgross35
Copy link
Contributor Author

tgross35 commented Jan 6, 2025

I have been using this for a few weeks and it has worked pretty great. Going to merge this so the exhaustive CI checks become available for all PRs.

@tgross35 tgross35 enabled auto-merge January 6, 2025 08:34
Add a CI job with a dynamically calculated matrix that runs extensive
jobs on changed files. This makes use of the new
`function-definitions.json` file to determine which changed files
require full tests for a routine to run.
@tgross35 tgross35 force-pushed the tgross35/extensive branch from 386c2be to 9b08ee5 Compare January 6, 2025 08:56
@tgross35 tgross35 merged commit 067e97c into master Jan 6, 2025
35 checks passed
@tgross35 tgross35 deleted the tgross35/extensive branch January 6, 2025 09:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant