|
0.6.0
|
|
|
0.6.0
|
The following guidelines describe how to run the non-regression tests that verify Notus against a database of reference values.
Every change to a large code like Notus carries a risk: while fixing or improving one part of the code, you may unintentionally change the result of another. Such an accidental change is called a regression. The danger is that it is silent — the code still compiles and still runs, it simply produces a slightly different (and now wrong) number.
Non-regression testing removes that blind spot. A set of test cases whose correct results are already known is run again after every significant change, and each result is compared to the value recorded the last time everything was known to be correct. As long as the numbers match, you have evidence that your change did not break anything. The moment one stops matching, you know exactly which case regressed, and you know it now rather than months later.
Non-regression is the automated backbone of the code's verification and validation (V&V) process. Notus is used for research, so its results must be trustworthy and reproducible: a published simulation is only meaningful if the same case, run again, still gives the same answer. Running the test campaign before merging a change is what keeps that guarantee true over time, across many contributors and many years of development.
Concretely, the test cases live under test_cases/, the reference values live in a database (test_cases/database/non-regression.db), and the non-regression mode of notus.py is the tool that runs the cases and confronts them with the database.
This page describes how a case is checked, how to launch and filter the tests, and how to read the results. Building and maintaining the reference database itself is covered in Manipulate the non-regression database.
The whole method rests on one simple idea: compare a fresh result to a trusted one. A trusted result — a reference value — is a number produced by a case at a time when the code was known to be correct, and then stored in the database. Running non-regression means producing the number again and checking it still equals the stored one, within a small tolerance.
The tolerance matters because floating-point computations are never bit-for-bit identical across machines, compilers or run configurations. So the test does not ask "is the new value exactly equal?" but "is it close enough?":
\[ |V_\text{ref} - V_\text{run}| \le \text{tolerance} \]
This is an absolute tolerance, typically very small (of the order of 1e-8 to 1e-14), because the reference cases are chosen so that the physical result is essentially deterministic.
When you launch ./notus.py non-regression, the script goes through the following stages:
@PARAMETER directives to build all parameter combinations. Each combination becomes one job.-Dtype:name=value ...). Locally the jobs run directly; on a cluster they are submitted through a batch template (see Submitting options). Each run writes the quantities listed in its validation block to a .validation file.The status is reported at two levels. Per case:
PASS — every checked value is within tolerance.FAIL — at least one value is outside tolerance (a regression).ERROR — the case could not be evaluated at all: it is absent from the database, or its .validation file is missing or unreadable.Then, for the whole run, an overall status aggregates them: ERROR if any case errored, otherwise FAIL if any case failed, otherwise PASS. This single line tells you at a glance whether your change is safe to merge.
A typical workflow around a code change looks like this: you run the relevant part of the test suite before your change to confirm everything passes, make your change, then run it again. If the same cases still pass, your change is non-regressing. If a case flips to FAIL, you compare the produced value with the reference to understand what moved and whether it is a bug or an intended, legitimate change. In the latter case, once the new result is confirmed correct, the reference in the database is updated (see Manipulate the non-regression database) so that the new value becomes the trusted one from then on. Running the campaign before opening a merge request is part of the standard contribution process.
A non-regression case is a normal NTS file with a validation block that declares which quantities are checked. Only the values named there are compared against the database:
post_processing {
validation {
reference_solution_error velocity;
mean_velocity;
}
}
When the case is run through notus.py, these quantities are written to the .validation file and matched against the database entry for the case.
validation block is missing, the run produces no comparable value and the case is reported as ERROR with the hint that the validation block may be missing.Point the script at one or several NTS files, or at a directory of cases:
./notus.py non-regression test_cases/verification/navier/poiseuille_2D.nts ./notus.py non-regression test_cases/verification/
For each case, the script prints a comparison table and a status. When the case is found in the database:
────────────────────────────────
Name │ Reference value │ Value │ Tolerance │ Difference │ Passed
────────────────────┼─────────────────┼─────────────────┼───────────┼────────────┼───────
mean velocity │ 1.001730e-01 │ 1.001730e-01 │ 1e-13 │ 2.1e-16 │ PASS
Status: PASS
If a case is not yet stored, the values are still printed, but no comparison is possible:
Status: ERROR (not in database)
At the end, an overall group status is printed: PASS if every case passed, FAIL if any case failed, or ERROR if any case could not be evaluated.
Two filters, both regular expressions, let you target a subset of cases without moving files around.
--file-filter REGEX — keep only the cases whose file path matches. The leading test_cases/ prefix is optional. ./notus.py non-regression test_cases/ --file-filter poiseuille
--parameter-filter REGEX — keep only the parameter combinations whose parameters match. The pattern is matched against type:name=value, so all of these are valid:
./notus.py non-regression test_cases/ --parameter-filter "string:solver_name=mumps" ./notus.py non-regression test_cases/ --parameter-filter ".*:.*=mumps" ./notus.py non-regression test_cases/ --parameter-filter "solver_name"
The last form filters on the parameter name only, regardless of its value.
Non-regression covers more than single runs.
@PARAMETER) generate one run per parameter combination; each combination is looked up and validated independently in the database.@CONVERGENCE) are validated too: the observed order of convergence is computed exactly as in the Convergence study with notus.py mode, then compared to the reference order stored in the database. The table then shows the reference order, tolerance, difference and a Passed column.Scalability cases are not handled here; they have their own validation mode, see Scalability non-regression testing with notus.py.
Some verification cases are meant to stop deliberately (for instance to check that an invalid configuration is rejected). These expose a kill status instead of numeric values: the case passes when it stopped as expected, and the table reports the kill status line with a PASS or FAIL accordingly.
On a large test suite you rarely want to re-run everything.
--reload STATUS [STATUS ...] — re-run only the cases currently in a given state, e.g. --reload FAIL ERROR to retry what did not pass, without touching the cases that already passed.--summary [STATUS ...] — print a summary of the current results and exit, optionally restricted to some statuses. Valid statuses are STANDBY, SUBMITTED, PASS, FAIL and ERROR.--database-path PATH — database to compare against (default: test_cases/database/non-regression.db).-o, --directory DIR — output directory for the run and its .validation files (default: non-regression).-n, --mpi-procs N — force the number of MPI processes.-v, --verbose — verbose output (prints the generated jobs and the processed files).Like the other modes, non-regression can be submitted on a cluster in two passes: one to submit the jobs, one to collect and compare the results once they have finished.
With SLURM:
./notus.py non-regression --slurm path/to/template --stage submit ./notus.py non-regression --slurm path/to/template --stage post-process
With the generic dual-pass job manager:
./notus.py non-regression --dual-pass --stage submit --batch-template path/to/template --batch-command "sbatch {BATCH_FILE}"
./notus.py non-regression --dual-pass --stage post-process --batch-template path/to/template --batch-command "sbatch {BATCH_FILE}"
See Submitting options for details on templates and the dual-pass workflow.
The case ran correctly but has no reference values yet. Add it to the database with ./notus.py database --add <filter> (see Manipulate the non-regression database). Once stored, it will be compared on the next run.
Notus did not produce a .validation file. Either the validation block is missing from the NTS file, or, in a dual-pass run, the job has not finished yet (check your job scheduler before launching the post-process stage).
--file-filter selects cases by their file path; --parameter-filter selects, within a case, the parameter combinations to test (by type:name=value). They can be combined to target a single parameter set of a single file.
The observed order of convergence is recomputed and compared to the reference order stored in the database, with its own tolerance. The case fails if the observed order drifts outside that tolerance.
For standard non-regression values the tolerance is absolute. (Relative, percent-based tolerances are used for CPU times in the scalability validation mode, not here.)
1.12.0