Notus usage on clusters

The following guidelines will help you to use Notus on various clusters.

Note

The choice of the allocated ressources for a given simulation (the number of processors asked) must be chosen carefully. Please follow the recommendations of 60,000 cells per core in 3D and 30,000 cells per core in 2D to be in the range of maximum efficiency of a parallel Notus run in full-MPI mode.

The usage of notus is described for the following computers:

1. curta MCIA cluster website
2. irene rome TGCC cluster website

Ssh configuration

SSH (Secure Shell) is critical for secure remote access. A well-configured SSH setup not only enhances security by using key-based authentication but also improves workflow efficiency by simplifying host aliases and file transfers.

Generating an SSH Key

Using a strong key (RSA 4096-bit) is recommended to ensure robust encryption.

ssh-keygen -t rsa -b 4096 -f mykey_clustername

Configuring SSH

Creating a configuration file **~/.ssh/config** reduces the need for long command-line options. Configuration could be:

Host curta
  HostName curta2.mcia.fr
  User username
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/mykey_clustername

Host visu01
  HostName visu01.mcia.fr
  User username
  ProxyJump myproxyserver
  ForwardX11 yes

Host rome
  HostName irene-amd-fr.ccc.cea.fr
  User username
  ProxyJump myproxyserver
  ForwardX11 yes
  ForwardX11Trusted yes

Advanced Tip of TGCC remote access: Consider using ProxyJump keyword if you need to go through a specific host to reach more secure networks.

 Copying Your Public Key

For passwordless authentication, copy your public key to the remote machine:

ssh-copy-id -i mykey username@curta2.mcia.fr

File Transfer Using scp

With the configuration :

scp curta:/scratch/username/notus_run/myfolder /home/username/destination

Without configuration:

scp username@curta2.mcia.fr:/scratch/username/notus_run/myfolder /home/username/destination

Submission scripts

SLURM is a scalable workload manager designed for clusters and HPC environments. It efficiently allocates resources and schedules jobs to maximize cluster utilization, across nodes of a computer.

You need to write a SLURM file to have access to ressources

1. For curta MCIA cluster URL

#!/bin/bash
#SBATCH --job-name=jobname
#SBATCH --time=06:00:00
#SBATCH --chdir=.
#SBATCH --output=%J_nameofoutputfile.out # %J is job id
#SBATCH --error=%J_nameoferrorfile.err # %J is job id
#SBATCH --partition=i2m
#SBATCH --nodes=8 # Nbr of node
#SBATCH --tasks-per-node=32 # tasks per node
# #SBATCH --tasks=32 # Nbr of tasks. It is another way of allocate ressources
set -e
NOTUS_PATH="the/path/of/your/binary/notus/file"
# Load required modules
source /etc/profile.d/modules.sh
module purge
module load notus_third_party/0.6.0/gcc/11.2.0
# Define parameters
myvar=1600
# Display job information
echo "#############################"
echo "User: $USER"
echo "Date: $(date)"
echo "Host: $(hostname)"
echo "Directory: $(pwd)"
echo "SLURM_JOBID: $SLURM_JOBID"
echo "SLURM_SUBMIT_DIR: $SLURM_SUBMIT_DIR"
echo "SLURM_JOB_NODELIST: $SLURM_JOB_NODELIST"
echo "#############################"
ulimit -s unlimited
mpirun -np $SLURM_NTASKS ${NOTUS_PATH} -Ddouble:my_variable=$myvar --no-color -- your_nts_file.nts
sync
exit 0

Submit a Job:

sbatch submission-curta.sh

Check Job Queue :

squeue -u username

Cancel a Job :

scancel JOBID

For more advanced management, consult the SLURM documentation website which provides in-depth guidance on job dependencies, advanced scheduling options, and tuning.

1. For irene rome TGCC cluster URL

Template for job Submission my_submission.template:

#!/bin/bash
#MSUB -r XXXXX                  # Job name
#MSUB -T X                      # Elapsed time (s)
#MSUB -A gen00000               # alloc nbr
#MSUB -o PATH/log.X           # Standard output
#MSUB -e PATH/log.X           # Error output
#MSUB -q rome                   # partition
#MSUB -n X                      # Reservation of MPI processes
#MSUB -m scratch                # location


PATH_NOTUS_BIN="$HOME/notus_path/notus"
PATH_RUN=`pwd`
set -x
ulimit -s unlimited
module purge
module load datadir/gen00000 python3/3.10.6 mkl/default mpi/openmpi
cd $BRIDGE_MSUB_PWD
ccc_mprun $PATH_NOTUS_BIN --no-color $PATH_RUN/mynts.nts
exit 0

Submit a Job: ccc_msub my_submission.template

Check Job Queue : ccc_mstat -u

Cancel a Job : ccc_mdel JOBID