Getting started on a compute cluster
The first time you connect to a cluster, execute the following commands. First, create a work directory, then create a symlink to it and change into it:
ws_allocate workspace 31
ln -s $(ws_find workspace) my_workspace
cd my_workspace
From them on, do all your work in this directory and submit your compute jobs from there.
nano
(recommended for beginners, vim and emacs are also available).
Since the login nodes are shared among all users of the cluster, you should not run any compute intensive programs on the login node and should not use your home directory for data storage. The login node should only be used for compiling code and for setting up and submitting jobs to run on the compute nodes.
Before you can run jobs on the cluster, you have to set up a work directory on the filesystem of the cluster where you can store all your simulation results. To do this, run the following command:
ws_allocate <name> <duration in days>
to create a work directory with a specific name and lifetime on the cluster. The maximum duration is 31 days. After that, the directory will be made read-only and finally deleted. You can extend the duration up to 3 times by running:
ws_extend <name> <days>
After that, you should download the data to your own PC or server for long term storage.
Create a symlink in you home directory for more convenient access of your work directory.
ln -s $(ws_find <name>)
You can then access your work directory by typing:
cd <name>
You can check the status of your work directory by running:
ws_list
Environment Modules
The cluster provides a variety of software packages that can be loaded using the Environment Modules system. To see which modules are available, run:
module avail
For a more refined search, use:
module spider <search term>
If you have found your desired module, copy the entire displayed path of the module by double clicking an typing: ctrl + shift + c
and then paste with
ctrl + shift + v
in the following command:
module load <module name>
You can check which modules are currently loaded by running:
module list
To unload a module, use:
module unload <module name>
or
module purge
to unload all modules.
Your modules will be unloaded automatically when you log out of the cluster. To load a module automatically when you log in, add the module load command to your
.bashrc
or
.profile
file in your home directory.
Submitting compute jobs
The training cluster uses the PBS workload manager to schedule and manage compute jobs. To submit a job, you need to create a job script that specifies the resources required for your job and the commands to execute. The training cluster has two types of nodes; 16 CPU nodes, each with two Intel Skylake processors (20x2 = 40 cores / 80 threads) and 192 GB memory, and 4 GPU nodes, each with one Intel CascadeLake processor (36 cores / 72 threads), 768 GB memory, and 8 NVIDIA V100 GPUs. The workload manager will chose the appropriate nodes for your job based on the resource requests in your job script:#!/bin/bash
# Simple PBS batch script for a CPU job
#PBS -q smp
#PBS -l walltime=00:30:00
### Number of nodes; number of CPUs and MPI processes per node:
#PBS -l select=1:ncpus=10:mpiprocs=10:mem=48GB
### go to directory where your job request was submitted
cd $PBS_O_WORKDIR
### load for example the Intel MPI environment
module load mpi/impi
### run you parallel application on the allocated nodes
mpirun -np 10 ./mpitest arg1 arg2
#!/bin/bash
#!/bin/bash
# Simple PBS batch script for a single-GPU job
#PBS -q smp
#PBS -l walltime=00:05:00
### Number of nodes; CPUs, GPUs, and memory per node:
#PBS -l select=1:ncpus=4:mem=90gb:ngpus=1
### go to directory where your job request was submitted
cd $PBS_O_WORKDIR
### load CUDA environment
module load nvhpc-sdk system/cuda
### run your application
./cudatestprog arg1 arg2
myprogramm.job
and submit it to the workload manager using the command:
qsub myprogramm.job
You can monitor the queue using:
qstat or qstat -u $USER
You can cancel your own jobs using:
qdel <job id>
Alternatively, you can start an interactive job to bring you to the command line on the compute node:
qsub -I -q smp -l select=1:ncpus=10:mem=40gb -l walltime=01:10:00 # CPU
qsub -I -q smp -l select=1:ncpus=5:mem=90gb:ngpus=1 -l walltime=01:10:00 # GPU
On this page you can find more detailed instructions on how to submit jobs on the training cluster: Batch_System_PBSPro_(vulcan).