|
0.6.0
|
|
|
0.6.0
|
The following guidelines describe how to run a weak, strong, or strong-node scalability study with the notus.py script.
A scalability study measures how the computation time of a case evolves when more computing resources are used. Unlike a convergence study, it says nothing about the quality of the solution: it only tells you how efficiently Notus exploits parallelism. The scalability mode of notus.py automates it — it generates one run per resource configuration, collects the CPU times, and reports the speedup and parallel efficiency.
Scalability is run through a submode, given right after scalability:
| Submode | What stays fixed | What varies | Question answered |
|---|---|---|---|
weak | Problem size per core | Number of nodes | Can I solve a bigger problem in the same time if I add resources? |
strong | Total problem size | Number of nodes | Can I solve the same problem faster by adding nodes? |
strong-node | Total problem size | MPI procs on one node | Same as strong, but within a single node. |
In weak mode the mesh given is per core and grows with the number of cores; in strong and strong-node modes the mesh is the total mesh and stays constant.
As with convergence, the resolution is injected by the script, so the NTS file must use the injected variables rather than hard-coded values:
grid {
grid_type regular;
number_of_cells (N_CELLS_X, N_CELLS_Y, N_CELLS_Z);
number_of_ghost_cells 2;
}
The script also injects TIME_ITERATIONS (how many time steps each run performs). The case should run for exactly that many iterations so the timings are comparable.
The case must measure its CPU time, otherwise no .cputime file is produced and there is nothing to post-process. Enable the relevant measurements in the system block:
system {
output_directory PREFIX;
measure_cpu_time;
measure_time_iteration_cpu_time;
}
You can build a per-run output directory from the injected mesh size, which keeps the runs from overwriting each other:
string PREFIX = "case_" + i2s(N_CELLS_X);
A real 2D case header therefore looks like:
system {
output_directory PREFIX;
measure_cpu_time;
measure_time_iteration_cpu_time;
}
domain {
spatial_dimension 2;
corner_1_coordinates (0.0, 0.0);
corner_2_coordinates (channel_length, channel_length);
}
grid {
grid_type regular;
number_of_cells (N_CELLS_X, N_CELLS_Y);
number_of_ghost_cells 2;
}
Each submode needs a progression that defines the list of configurations to run (number of nodes for weak/strong, number of MPI procs for strong-node). Exactly one of these three options is required:
| Option | Meaning | Example → configurations |
|---|---|---|
--arithmetic MIN DIFF MAXTERMS | Arithmetic progression: MIN, MIN+DIFF, MIN+2·DIFF, … | --arithmetic 4 4 4 → 4, 8, 12, 16, 20 |
--geometric MIN RATIO MAXEXP | Geometric progression: MIN·RATIO⁰, MIN·RATIO¹, … (exponent from 0) | --geometric 4 4 4 → 4, 16, 64, 256, 1024 |
--custom N1 N2 N3 … | Explicit list of configurations. | --custom 4 6 9 13 18 → 4, 6, 9, 13, 18 |
In weak mode the mesh is given per core and the total mesh grows with the number of cores, so the problem size per core is constant.
./notus.py scalability weak --arithmetic 4 4 4 --mpi-procs-per-node 16 --threads 2 --cells-per-direction 30 30 30 --time-iterations 300 path/to/case.nts
| Argument | Role | Default |
|---|---|---|
--mpi-procs-per-node N | MPI processes per node. | required |
--cells-per-direction NX NY [NZ] | Mesh per core (2 or 3 values). | required |
--time-iterations N | Number of time steps performed per run. | required |
--threads N | OpenMP threads per MPI process. | 1 |
--dimension {2,3} | Dimension of the case. | 3 |
one of --arithmetic/--geometric/--custom | Node-count progression. | required |
In strong mode the total mesh is fixed and spread over a growing number of nodes:
./notus.py scalability strong --geometric 4 4 4 --mpi-procs-per-node 32 --dimension 2 --cells-per-direction 500 500 --time-iterations 60 path/to/case.nts
The arguments are the same as weak, except --cells-per-direction is now the total mesh, not the per-core mesh.
strong-node keeps the total mesh fixed but varies the number of MPI processes on a single node, which is convenient when you do not have several nodes available. It does not take --mpi-procs-per-node (the progression already gives the number of procs):
./notus.py scalability strong-node --custom 4 6 9 13 18 --dimension 2 --cells-per-direction 100 100 --time-iterations 150 path/to/case.nts
Each run writes a .cputime file; the post-processing step then prints one table per timer (total time, solver, etc.) and writes three result files to the output directory (-o/--directory, default scalability_<submode>):
<case>_<submode>_times — raw CPU times (normalized by the number of time iterations),<case>_<submode>_speedup — speedup relative to one core (not produced in weak mode),<case>_<submode>_efficiency — parallel efficiency,together with a .md summary. By default the tables are written in space-separated format (--output-format ssv); use --output-format csv for comma-separated files.
A printed table looks like this (strong mode, where speedup is shown):
Total time: 2 threads, 32 cores per node
N cpu │ Nodes │ Mesh │ total time │ Speedup │ Efficiency
──────┼───────┼───────────────┼──────────────┼───────────┼───────────
32 │ 1 │ (500, 500, 1) │ 1.0000 │ 1.000000 │ 1.000
64 │ 2 │ (500, 500, 1) │ 0.5200 │ 1.923077 │ 0.962
128 │ 4 │ (500, 500, 1) │ 0.2700 │ 3.703704 │ 0.926
The header line recalls the number of OpenMP threads and the number of cores per node. The columns are the number of CPUs (N cpu), the number of nodes, the mesh, the timer value, the speedup (omitted in weak mode), and the parallel efficiency. One such table is printed per timer (total time, momentum solver, Poisson solver, etc.).
Times are first normalized by the number of time iterations, so a run that performed more steps is comparable to one that performed fewer. From the normalized time on \(N\) cores \(t(N)\):
weak mode efficiency is computed directly as \(t(1)/t(N)\), since the work per core is constant.For a perfectly scaling code the speedup doubles when the resources double and the efficiency stays close to 1.
The numbers are only meaningful if each core has enough work. Following the cluster usage recommendations, aim for about 60,000 cells per core in 3D and 10,000 cells per core in 2D to stay in the range of maximum efficiency of a full-MPI Notus run. Too few cells per core makes communication dominate and the efficiency collapses, which reflects the sizing, not the code. See Notus usage on clusters for resource-allocation guidance.
During post-processing, each run is validated through its .cputime file. A run is flagged as ERROR when the file is missing (the job did not finish, or has not finished yet in a dual-pass run) or empty (the simulation crashed before writing its timings). Failed jobs are ignored so the rest of the study can still be processed, and the group is reported with status ERROR and a warning. If every job failed, the script reports that there is nothing to post-process. At least two valid runs are needed to produce a table.
If a group of jobs from a previous run already exists in the output directory, the script asks for confirmation before overwriting. Answering n skips that case; answering y deletes the old files (including .cputime) and relaunches.
Scalability is almost always run on a cluster through the dual-pass job manager: a first pass submits the jobs (--stage submit), and a second pass collects the results once the jobs have finished (--stage post-process).
./notus.py scalability weak --dual-pass --stage submit --batch-template path/to/template --batch-command "sbatch {BATCH_FILE}" --arithmetic 4 4 4 --mpi-procs-per-node 16 --threads 2 --cells-per-direction 30 30 30 --time-iterations 300 path/to/case.nts
./notus.py scalability weak --dual-pass --stage post-process --batch-template path/to/template --batch-command "sbatch {BATCH_FILE}" --arithmetic 4 4 4 --mpi-procs-per-node 16 --threads 2 --cells-per-direction 30 30 30 --time-iterations 300 path/to/case.nts
See Submitting options for details on the dual-pass job manager and batch templates.
The scalability mode described above only measures performance; it does not check it against anything. To turn a scalability study into a pass/fail test, Notus provides a separate non-regression-scalability mode. The difference in a nutshell:
scalability | non-regression-scalability | |
|---|---|---|
| Parameters | Given on the command line (--arithmetic, --cells-per-direction, …) | Read from @ directives inside the NTS file |
| Output | CPU times, speedup, efficiency tables | Same tables compared to stored references |
| Database | Not used | Efficiency values compared to a per-machine database |
In non-regression-scalability mode, each measured efficiency is compared to the reference stored in the database. The comparison uses a relative tolerance (in percent), because CPU times depend on the machine, and the printed table gains reference and pass/fail columns:
Total time: 2 threads, 32 cores per node
N cpu │ Nodes │ Mesh │ total time │ Speedup │ Reference speedup │ Efficiency │ Reference efficiency │ Tolerance │ Difference │ Pass
──────┼───────┼───────────────┼────────────┼─────────┼───────────────────┼────────────┼──────────────────────┼───────────┼────────────┼──────
32 │ 1 │ (500, 500, 1) │ 1.0000 │ 1.000 │ 1.000 │ 1.000 │ 1.000 │ 10 % │ 0.000% │ PASS
64 │ 2 │ (500, 500, 1) │ 0.5200 │ 1.923 │ 1.900 │ 0.962 │ 0.950 │ 10 % │ 1.263% │ PASS
A run is PASS when, for every configuration, the relative difference between the measured and reference efficiency stays within the tolerance; otherwise it is FAIL. A configuration whose reference is missing from the database is reported as ERROR (not in database).
The reference values are produced and stored with the database mode; see Scalability non-regression testing with notus.py for the full description of this mode, and Manipulate the non-regression database for how to add references (remember --directory non-regression-scalability when adding scalability results).
Use strong to check whether a fixed problem runs faster on more nodes, weak to check whether a proportionally larger problem stays at constant time, and strong-node to study parallel behaviour within a single node when several nodes are not available.
In weak mode --cells-per-direction is the mesh per core (the total mesh grows with the number of cores). In strong and strong-node it is the total mesh.
Speedup compares against a single-core run of the same problem, which does not apply to a weak study (the problem grows with the resources). Weak scalability is assessed through efficiency instead.
The number of values in --cells-per-direction must match --dimension. With --dimension 3 you must give three values; with --dimension 2, give two. If you pass three values in 2D, the third is ignored with a warning. Since --dimension defaults to 3, set it explicitly to 2 for 2D cases.
Each run must write a .cputime file, which only happens if the case enables measure_cpu_time (and measure_time_iteration_cpu_time for per-iteration timers) in its system block. If the file is missing in a dual-pass run, also check that the batch jobs have actually finished before launching the post-process stage.
1.12.0