|
0.6.0
|
|
|
0.6.0
|
The following guidelines describe how to check that the parallel performance of Notus does not regress, using the non-regression-scalability mode.
Non-regression testing with notus.py protects the correctness of the code: it catches a change that alters a numerical result. But a change can leave every number perfectly correct and still be a serious problem — if it makes the code twice as slow, or if it quietly breaks parallel efficiency so that adding nodes no longer helps.
That kind of damage is invisible to a standard non-regression run. This is what the non-regression-scalability mode is for: it re-runs a scalability study and compares the measured speedup and parallel efficiency to values recorded when the code was known to perform well. A drop beyond tolerance turns the case to FAIL, so a performance regression is caught like any other bug.
Notus offers two complementary tools built on the same measurements:
| Scalability study with notus.py | non-regression-scalability | |
|---|---|---|
| Purpose | Measure performance | Check performance against a reference |
| Study parameters | Given on the command line | Read from @ directives inside the NTS file |
| Result | Speedup / efficiency tables | Same tables, plus reference, difference and PASS/FAIL |
| Database | Not used | Efficiency compared to a per-machine database |
The key practical difference is where the study is described. In scalability mode you pass --arithmetic, --cells-per-direction, and so on at every launch. Here, the case carries its own study definition, so the same test always runs the same way — which is exactly what a regression test requires.
The study is declared with #@ directives at the top of the file (Notus itself ignores these comment lines; they are read by notus.py):
# @SCALABILITY weak # @DIMENSION 2 # @PROGRESSION custom 1 2 4 # @MPI_PROCS_PER_NODE 2 # @THREADS 1 # @CELLS_PER_CORE 16 # @TIME_ITERATIONS 20
| Directive | Role |
|---|---|
@SCALABILITY weak\|strong\|strong-node | Type of study — same meaning as the submodes of Scalability study with notus.py. |
@DIMENSION 2\|3 | Dimension of the case. |
@PROGRESSION TYPE ... | Resource configurations to run (see below). |
@MPI_PROCS_PER_NODE N | MPI processes per node (not used by strong-node). |
@THREADS N | OpenMP threads per MPI process. |
@CELLS_PER_CORE N | Mesh size per core (weak) or total mesh (strong). |
@TIME_ITERATIONS N | Number of time steps performed by each run. |
The @PROGRESSION directive accepts the same three forms as the command-line options of the scalability mode:
# @PROGRESSION custom 1 2 4 → runs on 1, 2 and 4 nodes # @PROGRESSION arithmetic 4 4 4 → 4, 8, 12, 16, 20 # @PROGRESSION geometric 4 4 4 → 4, 16, 64, 256, 1024
As in the scalability mode, the resolution is injected by the script, so the grid block must use the injected variables, and the time-iteration count must come from TIME_ITERATIONS:
grid {
grid_type regular;
number_of_cells (N_CELLS_X, N_CELLS_Y);
number_of_ghost_cells 2;
}
numerical_parameters {
time_iterations TIME_ITERATIONS;
}
The script injects SCALABILITY, NODES, N_PROCS, THREADS and TIME_ITERATIONS into each run, and derives the mesh from @CELLS_PER_CORE and the number of cores of the configuration.
@PARAMETER directives. The study is then repeated for every parameter combination, each one validated independently against the database — exactly like parametric cases in standard non-regression../notus.py non-regression-scalability --file-filter poiseuille_scalab
Since a meaningful study needs several nodes, this mode is normally run on a cluster in two passes:
./notus.py non-regression-scalability --slurm path/to/template --stage submit ./notus.py non-regression-scalability --slurm path/to/template --stage post-process
or with the generic dual-pass job manager:
./notus.py non-regression-scalability --dual-pass --stage submit --batch-template path/to/template --batch-command "sbatch {BATCH_FILE}"
./notus.py non-regression-scalability --dual-pass --stage post-process --batch-template path/to/template --batch-command "sbatch {BATCH_FILE}"
See Submitting options for the dual-pass workflow and Notus usage on clusters for resource allocation.
Each configuration writes a .cputime file; the post-process step then prints one table per timer, with the measured values next to their references:
Total time: 1 threads, 2 cores per node
N cpu │ Nodes │ Mesh │ total time │ Speedup │ Reference speedup │ Efficiency │ Reference efficiency │ Tolerance │ Difference │ Pass
──────┼───────┼────────────┼────────────┼─────────┼───────────────────┼────────────┼──────────────────────┼───────────┼────────────┼──────
2 │ 1 │ (16, 16) │ 1.0000 │ 1.000 │ 1.000 │ 1.000 │ 1.000 │ 10 % │ 0.000% │ PASS
4 │ 2 │ (32, 16) │ 0.5200 │ 1.923 │ 1.900 │ 0.962 │ 0.950 │ 10 % │ 1.263% │ PASS
8 │ 4 │ (32, 32) │ 0.2700 │ 3.704 │ 3.600 │ 0.926 │ 0.900 │ 10 % │ 2.889% │ PASS
The verdict is computed on the efficiency, with a relative tolerance expressed in percent:
\[ \frac{|E_\text{ref} - E_\text{run}|}{E_\text{ref}} \le \frac{\text{tolerance}}{100} \]
A relative tolerance is used here — unlike the absolute tolerance of standard non-regression — precisely because timings fluctuate: the same machine will not reproduce the same time twice to the last digit. The default scalability tolerance is 1e-1, that is 10 %.
Statuses follow the usual convention: PASS when every configuration stays within tolerance, FAIL when at least one drifts too far, and ERROR when the case cannot be evaluated — most often ERROR (not in database) when no reference has been recorded yet on this machine.
Because references are machine-specific, the default database differs from the standard one: --database-path defaults to test_scal.db here, instead of test_cases/database/non-regression.db.
References are added exactly like standard ones, with the database mode — but pointing at the right results directory, since this mode writes to non-regression-scalability rather than non-regression:
./notus.py database --add poiseuille_scalab --directory non-regression-scalability --database-path test_scal.db
The tolerance can be set with --default-scalability-tolerance (default 1e-1). See Manipulate the non-regression database for the full description of the database operations, including --replace to refresh a reference after a legitimate performance change.
--file-filter REGEX — select cases by file path (the test_cases/ prefix is optional).--parameter-filter REGEX — select parameter combinations, matched against type:name=value.--threads N — force the number of OpenMP threads, overriding @THREADS.--cores-per-node N — force the number of cores per node. It requires --threads to be given as well.--database-path PATH — database to compare against (default: test_scal.db).-o, --directory DIR — output directory (default: non-regression-scalability).--reload STATUS [...] — re-run only the cases in a given state, e.g. --reload FAIL ERROR.--summary [STATUS ...] — print a summary of the current results and exit.Because the quantities are of a different nature. A physical result is deterministic and can be compared to 1e-14; a CPU time is not reproducible to that precision — it varies with system load, memory traffic and the machine itself. Only a relative margin (a few percent) is meaningful.
No reference exists yet for this case on this machine. Run it once, check the numbers are sensible, then record them with ./notus.py database --add ... --directory non-regression-scalability.
You can run it to check the workflow, but not to obtain trustworthy references: on a single machine all MPI processes share the same memory bus, so the measured efficiency reflects memory saturation rather than the parallel behaviour of the code. Use a real cluster, with enough cells per core (see Notus usage on clusters).
That is expected: the references belong to the machine that produced them. Re-record the references on the new machine rather than raising the tolerance.
A regression test must be reproducible without the operator remembering a long command. Storing the study in the case guarantees that everyone runs the very same configuration, which is what makes the comparison to a stored reference meaningful.
1.12.0