version 0.6.0
Loading...
Searching...
No Matches
How to run Notus

The following guidelines describe the different ways to run Notus.

Basic way to run Notus

Notus runs only through mpirun command, at least on 1 processor. The description of physical case to simulate is present in an _.nts_ file of one of the test_cases subdirectory that must given on the command-line. For instance, to run Notus on 8 processors and solve the Poiseuille flow, the command is:

mpirun -np 8 notus test_cases/verification/navier/poiseuille_2D.nts

Notus full help:

usage: notus [OPTIONS] [--] CASE_FILE

CASE_FILE               Notus case file (*.nts)

OPTIONS

-D TYPE:IDENT=VALUE     Define an identifier outside the Notus case file
                        TYPE: integer|double|boolean|string
                        IDENT: identifier name
                        VALUE: identifier value corresponding to the type

-h, --help              Display help and exit

-t NUM, --thread NUM    Use NUM OpenMP threads. Has no effect if Notus is compiled without OpenMP.

--no-banner             Do not display Notus banner

--no-color              Disable the use of ANSI color codes

--cpu-time-file FILE    File where the CPU time is output

--validation-file FILE  File where the validation is output

Advanced ways to run Notus with the script notus.py

The script notus.py enables one to run Notus in different modes :

  • Run
  • Non-regression
  • Non-regression-scalability
  • Convergence
  • Scalability

There is also a Database mode which permmit to manipulate the database used for the non-regressions modes. For further infomation, see Manipulate non-regression database

Here is the Notus help, which presents the different modes available:

usage: notus.py [-h] [--no-color] MODE ...

Tool to execute Notus for non-regression, scalability or convergence tests.

positional arguments:
    MODE                sub-command help

    non-regression      run Notus non-regression tests.

    non-regression-scalability
                        run Notus non-regression-scalability tests.

    database            Edit database.

    scalability         run Notus scalability tests.

    convergence         run Notus convergence tests.

    run                 run Notus for parametric analysis.

Each mode use the same submitting options, it is explained how to use them in Submitting options section.

Information about the options of these different modes can be found in the following paragraphs.

Run mode

It is the mode to use if you want to execute Notus. It allows to make parametric analysis.

Here is Notus full help for this mode:

usage: notus.py run [-h] [-n PROCS_NUMBER] [-f] [-m MPI_COMMAND] [-o OUTPUT]
                    [-b {black,white}] [-P PBS] [-S SLURM] [--no-color] [-v]
                    NTS_FILES [NTS_FILES ...]

positional arguments:
    NTS_FILES                                     Notus input files (.nts)

optional arguments:
    -h, --help                                    Show this help message and exit
    -n PROCS_NUMBER, --procs-number PROCS_NUMBER  Number of processors used tu run Notus
    -f, --full                                    Enable full validation
    -m MPI_COMMAND, --mpi-command MPI_COMMAND     MPI command (default: mpirun)
    -o OUTPUT, --output OUTPUT                    Output directory of log files
    -b {black,white}, --background {black,white}  Colors for display are chosen according to the color of the background (default: black)
    -P PBS, --PBS PBS                             Launch validation using a batch system (PBS)
    -S SLURM, --SLURM SLURM                       Launch validation using a batch system (SLURM)
    --no-color                                    Disable the use of ANSI color codes
    -v, --verbose                                 Diplay output

More information about [Submitting] options can be found at Submitting options.

Non-regression mode (validation script)

It is possible to run all or a part of the non regression tests, composed with verification and validation test cases (Verification and Validation) as well as various tests, thanks to the notus.py script. It is useful to check Notus portal on a new computer or to validate new developments.

Notus full help in Non-regression mode:

Usage: notus.py non-regression [-h] [-r REGEXP] [-s] [-f] [-m MPI_COMMAND]
                               [-o OUTPUT] [-b {black,white}] [-P PBS] [-S SLURM]
                               [--no-color] [-v]

Optional arguments:

    -h, --help                                    show this help message and exit
    -r REGEXP, --regexp REGEXP                    filter test cases using a regular expression
    -s, --sequential                              enable sequential non-regression tests
    -f, --full                                    enable full non-regression tests
    -m MPI_COMMAND, --mpi-command MPI_COMMAND     MPI command (default: mpirun)
    -o OUTPUT, --output OUTPUT                    output directory of log files
    -b {black,white}, --background {black,white}  colors for display are chosen according to the color of the background (default: black)
    -P PBS, --PBS PBS                             launch non-regression using a batch system (PBS)
    -S SLURM, --SLURM SLURM                       launch non-regression using a batch system (SLURM)
    --no-color                                    Disable the use of ANSI color codes
    -v, --verbose                                 Diplay output

Since the script detects automatically the test cases to use, if you want to test only some of them, you will have to use the -r argument to filter those cases.

The full non-regression is possible thanks to the syntax introduced in every NTS file that makes it possible to run the same test case several times (to rotate the direction of flow for instance). For further information about this, see Add a test case to Notus test case bank

The SLURM template is yet to be written, which means that the non-regression using SLURM batch system is not available for now.

Results can be compared to some obtained on different hardware and software architecture present in the test_cases/results directory.

Non-regression-scalability mode

It is possible to run all the test related to the non regression of scalability, and compare the result with an existing database. It is usefull to check non regression scalability to check the capacity of Notus after new developments.

Notus full help in Non-regression-scalability mode:

usage: notus.py non-regression-scalability [Filters] [Submitting] [Options] database-path

positional arguments:
  database_path         Path of the scalability database (default: Pas de db par défault).


Options:
  -h, --help            show this help message and exit

  -v, --verbose         Verbose mode.

  --reload {STANDBY,SUBMITTED,PASS,FAIL,ERROR} [{STANDBY,SUBMITTED,PASS,FAIL,ERROR} ...]
                        Reload cases by their status.

  --summary [{STANDBY,SUBMITTED,PASS,FAIL,ERROR} [{STANDBY,SUBMITTED,PASS,FAIL,ERROR} ...]]
                        Print a summary and exit.

  -o DIRECTORY, --directory DIRECTORY
                        Output directory (default: non-regression-scalability).


Filters:
  --file-filter FILE_FILTER
                        Filter file names using a regular expression.

  --parameter-filter PARAMETER_FILTER
                        Filter files by their paramaters using a regular expression.

[tips on how to use it with maybe some exemples]

Run a convergence study with convergence mode

It is possible to run a spatial or temporal convergence study using notus.py. The time step or the spatial step will have a geometrical evolution. It will be needed to put N_CELLS_X, N_CELLS_Y, N_CELLS_Z to make the mesh variable. It will be needed to put TIME_STEP to make the time step variable. The result of the study will be save as a markdown table, or on a css format. To automatically calculate convergence metrics, it is necessary to specify which output values should be tracked (e.g., max, min, average speed) in the post-processing block of the .nts file using the diagonstic quantity and grid_convergence_quantities keywords.

Notus full help in Convergence mode:

usage: notus.py convergence [Options] [Submission Parameters] [Convergence Parameters] NTS_FILE [NTS_FILE ...]

positional arguments:
  nts_files             link to the .nts


Options:
  -h, --help            show this help message and exit

  -v, --verbose         Verbose mode.

  --reload {STANDBY,SUBMITTED,PASS,FAIL,ERROR} [{STANDBY,SUBMITTED,PASS,FAIL,ERROR} ...]
                        Reload cases by their status.

  -o DIRECTORY, --directory DIRECTORY
                        Output directory (default: convergence).


Convergence Parameters:
  --type {space,time}   choose between time and space

  --time-step TIME_STEP
                        give the value of the time step, example 1.0, default value 1.0, if the problem has no dependency on time, the value does not matter

  --final-time FINAL_TIME
                        indicates the time when the calcul ends (optional for spatial convergence)

  --initial-mesh INITIAL_MESH [INITIAL_MESH ...]
                        spacial convergence : give the first mesh, the others will have a lower step-value following a geometric factor time convergence : the mesh used

  --geometric-factor GEOMETRIC_FACTOR
                        geometric factor for the convergence so n step turn into n × geometric factors steps

  --count COUNT         number of simulations of the convergence study


Submission Parameters:
  -n MPI_PROCS, --mpi-procs MPI_PROCS
                        Force the number of MPI processes.

  --mpi-command MPI_COMMAND
                        MPI command (default: mpirun).

  --dual-pass           Submit in dual pass batch system.

  --batch-template BATCH_TEMPLATE
                        Template file for batch jobs.

  --batch-command BATCH_COMMAND
                        Command to launch batch jobs.

  --slurm BATCH_TEMPLATE
                        Submit in dual pass with Slurm.

  --stage {submit,post-process}
                        Set the stage of the dual-pass batch system.


Examples:

  Launch a spacial convergence study:
    ./notus.py convergence --dual-pass --batch-template path/to/template/file --batch-command "sbatch {BATCH_FILE}" --stage submit --type space --initial-mesh 250 100 --geometric-factor 2 path/to/nts_files.nts
    ./notus.py convergence --dual-pass --batch-template path/to/template/file --batch-command "sbatch {BATCH_FILE}" --stage post-process --type space --initial-mesh 250 100 --geometric-factor 2 path/to/nts_files.nts

  Lauch a time convergence study:
    ./notus.py convergence [Submitting] --type time ... --geometric-factor 2 path/to/nts_files.nts
  • An example of the spatial convergence study of a problem called my_nts_2D_space.nts is given below:

    ./notus.py convergence –type space –initial-mesh 16 16 –geometric-factor 2 –count 4 -n 12 -o /tmp/space_cv my_nts_2D_space.nts

Warning: the file (my_nts_2D_space.nts) should not define convergence study variables, e.g. N_CELLS_X, N_CELLS_Y in the case of 2D spatial convergence.

  • An example of the temporal convergence study of a problem called my_nts_2D_time.nts is given below:

    ./notus.py convergence –type time –time-step 1e-2 –initial-mesh 16 16 –geometric-factor 2 –count 4 -n 12 -o /tmp/time_cv my_nts_2D_time.nts

  • An example of the spatial convergence study of a problem called my_nts_2D_space.nts launch on a cluster using SLURM is given below:
    1. First, the submission with slurm

      ./notus.py convergence –slurm path_template_batch –stage submit –type space –initial-mesh 16 16 –geometric-factor 2 –count 4 -n 12 -o /tmp/space_cv my_nts_2D_space.nts

    2. Then, the post-processing

      ./notus.py convergence –slurm path_template_batch –stage post-process –type space –initial-mesh 16 16 –geometric-factor 2 –count 4 -n 12 -o /tmp/space_cv my_nts_2D_space.nts

Warning: please give the path for the slurm template file path_template_batch.

To see a complete working example of spatial and temporal convergence study, using a bash script, please take a look at tools/run_convegence/space and tools/run_convegence/time.

Run a scalability study with scalability mode

It is possible to run different types of scalability study: weak, strong, strong node.

Notus full help in Scalability mode:

usage: notus.py scalability [-h] SUBMODE ...

positional arguments:
  SUBMODE      Scalability type

    weak       Run a weak scalability test on several nodes.

    strong     Run a strong scalability test on several nodes.

    strong-node
               Run a strong scalability test on one node.


options:
  -h, --help   show this help message and exit

Run weak scalability mode

The strong scalability evaluate the efficiency of workload reduction per core by increasing the number of cores while keeping the constant mesh size.

Notus full help in Scalability weak mode:

usage: notus.py scalability weak [Submitting] [Scalability parameters] [Options] nts_files [nts_files ...]

positional arguments:
  nts_files             nts files used to run scalability study


Options:
  -h, --help            show this help message and exit

  -v, --verbose         Verbose mode.

  --reload {STANDBY,SUBMITTED,PASS,FAIL,ERROR} [{STANDBY,SUBMITTED,PASS,FAIL,ERROR} ...]
                        Reload cases by their status.

  --output-format {ssv,csv}
                        Set the output format: Space or Comma Separated Value (default = ssv)

  -o DIRECTORY, --directory DIRECTORY
                        Output directory (default: scalability_weak).


Submitting:
  --mpi-command MPI_COMMAND
                        MPI command (default: mpirun).

  --dual-pass           Submit in dual pass batch system.

  --batch-template BATCH_TEMPLATE
                        Template file for batch jobs.

  --batch-command BATCH_COMMAND
                        Command to launch batch jobs.

  --slurm BATCH_TEMPLATE
                        Submit in dual pass with Slurm.

  --stage {submit,post-process}
                        Set the stage of the dual-pass batch system.


Scalability parameters:
  --time-iterations TIME_ITERATIONS
                        Number of time iterations

  --dimension {2,3}     Dimension of the case (default = 3)

  --arithmetic MIN COMMON-DIFF MAXTERMS
                        Arithmetic progression (N° of nodes):
                                     MIN = Initial term,
                                     COMMON-DIFF = Common difference of the arithmetic sequence,
                                     MAXTERMS = Maximum number of terms to add.

  --geometric MIN COMMON-RATIO MAXEXP
                        Geometric progression (N° of nodes):
                                     MIN = Initial term,
                                     COMMON-RATIO = Common ratio of the geometric sequence,
                                     MAXEXP = Maximum exponent (exponent starts at 0).

  --custom NODES [NODES ...]
                        Custom progression (N° of nodes): N1 N2 N3 ...

  --threads THREADS     Number of OpenMP threads (default: 1).

  --mpi-procs-per-node MPI_PROCS_PER_NODE
                        Number of MPI processes per node.

  --cells-per-direction N_CELLS_X N_CELLS_Y (N_CELLS_Z) [N_CELLS_X N_CELLS_Y (N_CELLS_Z) ...]
                        Number of cells per direction per core.


Examples:

  Launch a weak scalability test on a 3D case using the dual-pass job manager (arithmetic progression (4, 8, 12, 16, 20 nodes), 16 MPI processes per node, 2 OpenMP threads, 300 time iterations , 30^3 cells per core):

    ./notus.py scalability weak --dual-pass --stage submit --batch-template path/to/template/file --batch-command "sbatch {BATCH_FILE}" --arithmetic 4 4 4 --mpi-procs-per-node 16 --threads 2 --cells-per-direction 30 30 30 --time-iterations 300 paths/to/nts_files
    ./notus.py scalability weak --dual-pass --stage post-process --batch-template path/to/template/file --batch-command "sbatch {BATCH_FILE}" --arithmetic 4 4 4 --mpi-procs-per-node 16 --threads 2 --cells-per-direction 30 30 30 --time-iterations 300 paths/to/nts_files
  • An example of the weak scalability run on curta MCIA cluster in given in tools/scalability/run_weak_curta

Run strong scalability mode

The weak scalability evaluate code performance while maintaining a constant workload by increasing both at the same time the problem size and the core count.

Notus full help in Scalability strong mode:

  usage: notus.py scalability strong [Submitting] [Scalability parameters] [Options] nts_files [nts_files ...]

  positional arguments:
    nts_files             nts files used to run scalability study


  Options:
    -h, --help            show this help message and exit

    -v, --verbose         Verbose mode.

    --reload {STANDBY,SUBMITTED,PASS,FAIL,ERROR} [{STANDBY,SUBMITTED,PASS,FAIL,ERROR} ...]
                          Reload cases by their status.

    --output-format {ssv,csv}
                          Set the output format: Space or Comma Separated Value (default = ssv)

    -o DIRECTORY, --directory DIRECTORY
                          Output directory (default: scalability_strong).


  Submitting:
    --mpi-command MPI_COMMAND
                          MPI command (default: mpirun).

    --dual-pass           Submit in dual pass batch system.

    --batch-template BATCH_TEMPLATE
                          Template file for batch jobs.

    --batch-command BATCH_COMMAND
                          Command to launch batch jobs.

    --slurm BATCH_TEMPLATE
                          Submit in dual pass with Slurm.

    --stage {submit,post-process}
                          Set the stage of the dual-pass batch system.


  Scalability parameters:
    --time-iterations TIME_ITERATIONS
                          Number of time iterations

    --dimension {2,3}     Dimension of the case (default = 3)

    --arithmetic MIN COMMON-DIFF MAXTERMS
                          Arithmetic progression (N° of nodes):
                                       MIN = Initial term,
                                       COMMON-DIFF = Common difference of the arithmetic sequence,
                                       MAXTERMS = Maximum number of terms to add.

    --geometric MIN COMMON-RATIO MAXEXP
                          Geometric progression (N° of nodes):
                                       MIN = Initial term,
                                       COMMON-RATIO = Common ratio of the geometric sequence,
                                       MAXEXP = Maximum exponent (exponent starts at 0).

    --custom NODES [NODES ...]
                          Custom progression (N° of nodes): N1 N2 N3 ...

    --threads THREADS     Number of OpenMP threads (default: 1).

    --mpi-procs-per-node MPI_PROCS_PER_NODE
                          Number of MPI processes per node.

    --cells-per-direction N_CELLS_X N_CELLS_Y (N_CELLS_Z) [N_CELLS_X N_CELLS_Y (N_CELLS_Z) ...]
                          Number of cells per direction.


  Examples:

    Launch a strong scalability test on a 2D case using the dual-pass job manager (geometric progression (4, 8, 16, 32, 64 nodes), 32 MPI processes per node, 60 time iterations , 500^2 cells (total)):
      ./notus.py scalability strong --dual-pass --stage submit --batch-template path/to/template/file --batch-command "sbatch {BATCH_FILE}" --geometric 4 4 4 --mpi-procs-per-node 32 --dimension 2 --cells-per-direction 500 500 --time-iterations 60 paths/to/nts_files
      ./notus.py scalability strong --dual-pass --stage post-process --batch-template path/to/template/file --batch-command "sbatch {BATCH_FILE}" --geometric 4 4 4 --mpi-procs-per-node 32 --dimension 2 --cells-per-direction 500 500 500 --time-iterations 60 paths/to/nts_files
  • An example of the strong scalability in given in tools/scalability/run_strong_MPIopenMP for both irene-rome TGCC and curta MCIA cluster.

For both strong and weak modes, it will be needed to put TIME_ITERATIONS in the nts file to control from the oustide of nts file the maximum number of iterations.