fix(tester): only regenerate expected test outputs when necessary #699
Conversation
niklasdewally
force-pushed
the
nik/fix-tester-id-churn
branch
from
f1221aa to
e33c076
Compare
For test stages that use a model JSON, only regenerate the expected test outputs if the test stage fails. BACKGROUND In 8636432 (feat!: add parent symbol tables (#680), 2025-02-17), we added an id field to symbol-tables, allowing the reconstruction of pointers during de-serialisation. These ids are set by a global variable; therefore, they tend to change from run to run of the tester. However, they are ignored in the implementation of `Eq` for Model/SubModel, so this does not affect test correctness. After this change, when running the test suite with ACCEPT=true, new expected output files would be created for all test models containing different ids. This caused our git diffs to always contain changes to the ids in every single test case, cluttering them up. As id changes do not tests to fail, this commit means that an id change will not cause a new expected test output file to be generated.
niklasdewally
force-pushed
the
nik/fix-tester-id-churn
branch
from
e33c076 to
6f7c7e3
Compare
Code and Documentation Coverage ReportDocumentation CoverageThis PR: conjure_core: 3% with examples, 52% documented -- 8/143/275 Main: conjure_core: 3% with examples, 52% documented -- 8/143/275 View full documentation coverage for main, this PR Code Coverage SummaryThis PR: Detailed Report Main: Detailed Report Coverage Main & PR Coverage ChangeFunctions coverage changed by 0.00% and covered lines changed by 0
Branches... coverage: No comparison data available |
|
brilliant, thanks! why the new integration test as well? are you testing something related to this PR specifically? |
It was to test this PR: I changed the test and ran ACCEPT=true, and this was the resulting git diff. |
For test stages that use a model JSON, only regenerate the expected test outputs if the test stage fails.
BACKGROUND
In 8636432 (feat!: add parent symbol tables (#680), 2025-02-17), we added an id field to symbol-tables, allowing the reconstruction of pointers during de-serialisation.
These ids are set by a global variable; therefore, they tend to change from run to run of the tester. However, they are ignored in the implementation of
Eqfor Model/SubModel, so this does not affect test correctness.After this change, when running the test suite with ACCEPT=true, new expected output files would be created for all test models containing different ids.
This caused our git diffs to always contain changes to the ids in every single test case, cluttering them up.
As id changes do not cause tests to fail, this commit means that an id change will not cause a new expected test output file to be generated.