|
version 0.6.0
|
|
|
version 0.6.0
|
The following guidelines describe the submitting options when Notus is run with notus.py.
Notus can be run in dual pass, which means that the computation is done in 2 steps. Firstly, the jobs are submitted and run. Then, there is a phase of post processing where results can be printed in the shell. With the dual pass, Notus can be used on supercomputers, and several calculations can be launch in parallel.
To run Notus in dual pass, use the argument --dual-pass. Other arguments will be required: --batch-template, --batch-command and --stage.
The script will generate a .batch file following the instructions of the batch template. Then it will run the .batch file, using the command line indicated by the --batch-command argument. The command has to be written in quotation marks "" , after the --batch-command. The path to the batch file created from the batch template is accessible with the variable {BATCH_FILE}.
For example, with SLURM, the option is written : ... --batch-command "sbatch {BATCH_FILE}" ...
Stage can be submit or post-process to indicate in which phase of the dual pass you are.
To write a batch template, you write a general batch script using variables that python will replace for each job. The variables are read in the format {VARIABLE}.
All the variables you have acess to are:
N_MPI_PROCS: the number of processes the job uses.N_OMP_THREADS: the number of threads Notus uses.N_NODES: the number of nodes Notus uses.N_CORES_PER_NODE: the number of cores per node Notus uses.NOTUS_ARGS: the arguments of Notus generated by the python script.NTS_FILE: the path to the nts file.LOG_FILE: the path to the log file, where you will output the Notus calcul.Several essential templates can be found in the folder tools/validation.
For example, the template non-regression-scalability-curta-intel-20.template, which is a batch template for curta, using SLURM, will look like this:
#!/bin/bash
#SBATCH --job-name=notus_scalability
#SBATCH --time=00:15:00
#SBATCH --chdir=.
#SBATCH --output={LOG_FILE}
#SBATCH --error={LOG_FILE}
#SBATCH --constraint=compute
#SBATCH --nodes={N_NODES}
#SBATCH --tasks-per-node={N_MPI_PROCS//N_NODES}
#SBATCH --cpus-per-task={N_OMP_THREADS}
set -e
# Load modules
source /etc/profile.d/modules.sh
module purge
module load notus_third_party/0.6.0/intel/2020.4.304
echo "#############################"
echo "User:" $USER
echo "Date:" $(date)
echo "Host:" $(hostname)
echo "Directory:" $(pwd)
echo "SLURM_JOBID:" $SLURM_JOBID
echo "SLURM_SUBMIT_DIR:" $SLURM_SUBMIT_DIR
echo "SLURM_JOB_NODELIST:" $SLURM_JOB_NODELIST
echo "#############################"
ulimit -s unlimited
export OMP_NUM_THREADS={N_OMP_THREADS}
mpirun -np {N_MPI_PROCS} ./notus --no-banner --no-color {NOTUS_ARGS} -- {NTS_FILE}
sync
exit 0
It works using the python string formatter, so what is in {} brackets is read like a python expression. That is why you can make that kind of division: #SBATCH --tasks-per-node={N_MPI_PROCS//N_NODES}.
For instance, to run a 2D poiseuille flow in dual pass on curta, with intel library, with SLURM, You must run the submitting options:
./notus.py run --dual-pass --batch-template tools/validation/run-curta-intel-20.template
--batch-command "sbatch {BATCH_FILE}" --stage submit ../poiseuille_2D.nts
And then you run the post processing phase with:
./notus.py run --dual-pass --batch-template tools/validation/run-curta-intel-20.template
--batch-command "sbatch {BATCH_FILE}" --stage post-process ../poiseuille_2D.nts
There is the option --slurm to quickly run a non-regression using SLURM. If you want tu run a non-regression test with SLURM, on surta, you can use the command:
./notus.py non-regression --slurm tools/validation/non-regression-curta-intel-20.template --stage submit ./notus.py non-regression --slurm tools/validation/non-regression-curta-intel-20.template --stage post-process
It is possible to change the MPI command with the option --mpi-command. By default, it is mpirun.
It is also possible to force some submitting parameters from the command line instead of rewriting the nts file. You can force the following options:
--mpi-procs (only in run, non-regression and convergence mode).--threads (only in scalability and non-regression-scalability mode).--mpi-proc-per-node (only in scalability mode).--cores-per-node (only in non-regression-scalability mode).For example, if you want to run the 2D poiseuille flow, on 1 processor instead of 9, you can use the command:
./notus.py run --mpi-procs 1 test_cases/verification/navier/poiseuille_2D.nts
Here is Notus full help for the submitting options:
[mettre l'aide quand les shortcut seront fait]
1.11.0