version 0.6.0
Loading...
Searching...
No Matches
Submitting options

The following guidelines describe the submitting options when Notus is run with notus.py.

Dual pass options

Notus can be run in dual pass, which means that the computation is done in 2 steps. Firstly, the jobs are submitted and run. Then, there is a phase of post processing where results can be printed in the shell. With the dual pass, Notus can be used on supercomputers, and several calculations can be launch in parallel.

To run Notus in dual pass, use the argument --dual-pass. Other arguments will be required: --batch-template, --batch-command and --stage.

The script will generate a .batch file following the instructions of the batch template. Then it will run the .batch file, using the command line indicated by the --batch-command argument. The command has to be written in quotation marks "" , after the --batch-command. The path to the batch file created from the batch template is accessible with the variable {BATCH_FILE}.

For example, with SLURM, the option is written : ... --batch-command "sbatch {BATCH_FILE}" ...

Stage can be submit or post-process to indicate in which phase of the dual pass you are.

How to write a Batch Template

To write a batch template, you write a general batch script using variables that python will replace for each job. The variables are read in the format {VARIABLE}.

All the variables you have acess to are:

  • N_MPI_PROCS: the number of processes the job uses.
  • N_OMP_THREADS: the number of threads Notus uses.
  • N_NODES: the number of nodes Notus uses.
  • N_CORES_PER_NODE: the number of cores per node Notus uses.
  • NOTUS_ARGS: the arguments of Notus generated by the python script.
  • NTS_FILE: the path to the nts file.
  • LOG_FILE: the path to the log file, where you will output the Notus calcul.

Several essential templates can be found in the folder tools/validation.

For example, the template non-regression-scalability-curta-intel-20.template, which is a batch template for curta, using SLURM, will look like this:

#!/bin/bash
#SBATCH --job-name=notus_scalability
#SBATCH --time=00:15:00
#SBATCH --chdir=.
#SBATCH --output={LOG_FILE}
#SBATCH --error={LOG_FILE}
#SBATCH --constraint=compute
#SBATCH --nodes={N_NODES}
#SBATCH --tasks-per-node={N_MPI_PROCS//N_NODES}
#SBATCH --cpus-per-task={N_OMP_THREADS}

set -e

# Load modules
source /etc/profile.d/modules.sh
module purge
module load notus_third_party/0.6.0/intel/2020.4.304

echo "#############################"
echo "User:" $USER
echo "Date:" $(date)
echo "Host:" $(hostname)
echo "Directory:" $(pwd)
echo "SLURM_JOBID:" $SLURM_JOBID
echo "SLURM_SUBMIT_DIR:" $SLURM_SUBMIT_DIR
echo "SLURM_JOB_NODELIST:" $SLURM_JOB_NODELIST
echo "#############################"

ulimit -s unlimited
export OMP_NUM_THREADS={N_OMP_THREADS}
mpirun -np {N_MPI_PROCS} ./notus --no-banner --no-color {NOTUS_ARGS} -- {NTS_FILE}
sync

exit 0

It works using the python string formatter, so what is in {} brackets is read like a python expression. That is why you can make that kind of division: #SBATCH --tasks-per-node={N_MPI_PROCS//N_NODES}.

Example

For instance, to run a 2D poiseuille flow in dual pass on curta, with intel library, with SLURM, You must run the submitting options:

./notus.py run --dual-pass --batch-template tools/validation/run-curta-intel-20.template 
        --batch-command "sbatch {BATCH_FILE}" --stage submit ../poiseuille_2D.nts

And then you run the post processing phase with:

./notus.py run --dual-pass --batch-template tools/validation/run-curta-intel-20.template 
        --batch-command "sbatch {BATCH_FILE}" --stage post-process ../poiseuille_2D.nts

There is the option --slurm to quickly run a non-regression using SLURM. If you want tu run a non-regression test with SLURM, on surta, you can use the command:

./notus.py non-regression --slurm tools/validation/non-regression-curta-intel-20.template --stage submit
./notus.py non-regression --slurm tools/validation/non-regression-curta-intel-20.template --stage post-process

Other options

It is possible to change the MPI command with the option --mpi-command. By default, it is mpirun.

It is also possible to force some submitting parameters from the command line instead of rewriting the nts file. You can force the following options:

  • The number of processes with --mpi-procs (only in run, non-regression and convergence mode).
  • The number of threads with --threads (only in scalability and non-regression-scalability mode).
  • The number of processors per node with --mpi-proc-per-node (only in scalability mode).
  • The number of cores per node with --cores-per-node (only in non-regression-scalability mode).

For example, if you want to run the 2D poiseuille flow, on 1 processor instead of 9, you can use the command:

./notus.py run --mpi-procs 1 test_cases/verification/navier/poiseuille_2D.nts

Here is Notus full help for the submitting options:

[mettre l'aide quand les shortcut seront fait]