Hands-On: Modules & Job Submission

1. Start by checking what is already loaded after login:

   module list

   Note which compiler programming environment (PrgEnv-*), MPI implementation, and HLRS software stack are active by default.
2. List all available modules. The output is long, so pipe it through a pager:

   module avail 2>&1 | less

   Press q to quit. Scroll through the categories at the top: programming environments, the HLRS software stack, profiling tools, etc.

   The 2>&1 is needed because module writes to stderr by default; the pager only reads stdin.

3. Get a one-line description of a module without loading it:

   module whatis cray-libsci
   module whatis rocm

   For more detail, use module help <name> or module display <name>. The latter shows exactly which environment variables would be modified.
4. Search for a module by name using module spider:

   module spider python

   This lists everything matching "python", including the Cray Python distribution cray-python. List its available versions, then view the details of one of them:

   module spider cray-python
   module spider cray-python/<version>

   Which versions are available? Does cray-python require any other modules to be loaded first, or does it load on its own?

   For some libraries (HDF5, NetCDF, ...) spider lists prerequisite modules you must load before they become available. cray-python is self-contained and loads from a clean session.

5. Load a module and verify it changed your environment. First check which python3 you get by default:

   which python3

   This is the system Python (usually in /usr/bin). Now load the Cray Python module and look again:

   module load cray-python
   which python3
   echo $PATH | tr ':' '\n' | head

   which python3 now points into /opt/cray/pe/... instead of /usr/bin: the module prepended its own bin directory to the front of your PATH, so it shadows the system Python. The tr ':' '\n' turns the colon-separated PATH into one entry per line so the new directory at the top is easy to spot.

   You already met cray-python yesterday when building your virtual environment. You can also inspect $LD_LIBRARY_PATH the same way; depending on the version it may gain Cray library paths too.

6. Unload the module again:

   module unload cray-python
   which python3

   python3 is not gone: it now points back at the system Python in /usr/bin. Unloading a module that shadows a system tool simply reveals the original again. Confirm with module list that cray-python is no longer loaded.
7. Hunter has both APU and CPU node types. The HLRS software stack comes in matching flavors. Check which is active and how to switch:

   module list HLRS
   echo $HLRS_SOFTWARE_STACK_RELEASE_VERSION
   module avail HLRS

   The default is HLRS/APU. To compile for CPU-only nodes you would module swap HLRS/APU HLRS/CPU. Don't swap now. For the rest of this hands-on we stay on the APU stack.

1. Navigate into the cloned repository and list its contents:

   cd ~/hello-jobstep-hlrs
   ls

   You should see at least a source file, a Makefile, and a jobscript.pbs.

   If ~/hello-jobstep-hlrs doesn't exist, you may need to redo the clone step from yesterday's hands-on.

2. Read through the job script carefully:

   cat jobscript.pbs

   Identify each of the following parts and what it does:

   • The #PBS directives at the top: what resources are requested?
   • The module load commands: which modules are loaded inside the job?
   • The export lines: which environment variables are set, and why?
   • The mpiexec command at the bottom: what does --cpu-bind and --gpu-bind do?

   Compare to the slide "Anatomy of a Job Script". Don't worry if the pinning lines are still mysterious; you will see their effect in the output.

3. Edit the script to add your email address so PBS notifies you when the job starts and ends:

   vim jobscript.pbs

   Find the line #PBS -M <your email>, pressi and replace the placeholder with your real address. Press esc and save with :wq, Enter.

   To skip the email entirely, change #PBS -m bae to #PBS -m n.

For your own work, a good rule of thumb is one workspace per project or campaign, with one subfolder per simulation run inside it and not a separate workspace per run. Workspaces are a limited resource, and runs that belong together usually want to expire together anyway.

1. Move into the cloned repository:

   cd ~/hello-jobstep-hlrs
   ls

2. Before compiling, check which modules are loaded. The build needs ROCm and Cray MPICH; both are loaded automatically in a fresh login session, but it is worth confirming so the binary matches the runtime environment of the job:

   module list

   Look for entries containing rocm and cray-mpich. If either is missing, load it now:

   module load rocm
   module load cray-mpich

3. Compile the program:

   make

   The Makefile creates a build/ subdirectory containing the hello_jobstep binary and a copy of jobscript.pbs. Everything needed for the run is now in that one folder.

   If make complains about ROCM_PATH being undefined, or about missing headers like hip/hip_runtime.h, the required modules are not loaded — go back to the previous step.

4. Stage the run into your workspace. Copy the build folder into the workspace from yesterday and move into it:

   cp -r build ~/work/hello-jobstep-build
   cd ~/work/hello-jobstep-build
   ls -l

   The binary and the jobscript live in the workspace now, and the job's output will be written into the same folder.

   Using the project name as the subfolder is fine for a single run. For repeated runs (and copies of build), give each its own subfolder (hello-jobstep-run1, hello-jobstep-run2, …).

5. Submit the job from the workspace subfolder:

   qsub jobscript.pbs

   PBS prints a job ID like 12345.hunter-batch. Copy it, we will use it to check status and read output.

   The job's .o and .e files land in the directory you submit from.

6. Check the job's status:

   qstat

   The state column shows where your job is:

   • Q: queued, waiting for resources
   • R: running on compute nodes
   • E: exiting, cleaning up

   Run qstat a few times over the next minute. The test queue usually starts jobs within a minute or two.
7. For detailed information about a running job, use:

   qstat -f <jobid>

   This shows the resources requested, the nodes assigned, the submit/start times, and much more. The job runs only for a second, so you will likely miss it. For a finished job, add -x:

   qstat -xf <jobid>

1. Once the job state goes from R to E and then disappears from qstat, it has finished. Look for the log files specified in the job script:

   ls *.o* *.e*

   The .o file contains stdout from the job; the .e file contains stderr.
2. Read the output:

   cat <your_logfile>.o

   Each line represents one MPI rank and reports which node it ran on, which CPU cores it had access to, and which GPU it was bound to.
   
   Look carefully:

   • How many ranks total ran? Does it match select × mpiprocs?
   • How many distinct nodes appear?
   • Which CPU cores did each rank get? Note the ranges 2-23, 26-47, 50-71, 74-95.
   • Which GPU (0, 1, 2, or 3) is paired with each CPU range?

   The CPU pinning skips cores 0, 1, 24, 25, 48, 49, 72, and 73, the first two cores of each APU. These are reserved for system processes such as the operating system, the Lustre client, the ROCm runtime, and the AMD kernel drivers. Each MI300A has 24 CPU cores, so 22 remain available to your job. Keeping system activity off the compute cores also reduces OS noise, which matters increasingly at larger node counts. If you suspect your application would benefit from using these cores as well, profile the effect before including them in your pinning.

3. See what wrong pinning does. Make a copy of the job script, then change the GPU binding so every rank is sent to the same GPU:

   cp jobscript.pbs jobscript-badpin.pbs
   # edit jobscript-badpin.pbs and change the --gpu-bind line to:
   #     --gpu-bind list:0:0:0:0
   qsub jobscript-badpin.pbs

   Compare the GPU_ID column in the new output (a new .o file with the bad-pin jobid) against your first run. All four ranks per node now report GPU_ID 0, while GPUs 1, 2, and 3 sit idle, three quarters of the node's compute wasted on a one-character mistake.

   This is what the warnings mean by "wrong pinning wastes the node". The hello_jobstep output is how you catch it before launching a real, expensive job.

4. Check the error files. For a clean run it should be empty or contain only the warm-up chatter from MPI/PALS:

   cat <your_errorfile>.e

cp jobscript.pbs jobscript-v2.pbs

1. Change the node count. Edit jobscript-v2.pbs, change select=2 to select=1, and update the NODES variable in the script to match. Submit:

   qsub jobscript-v2.pbs

   Compare the output to the 2-node version. Half the ranks? Different node names?
   Change select to a number greater than 16 and try to queue the job.
2. Change the number of MPI ranks per node. Make another copy jobscript-v3.pbs. Change mpiprocs=4 to mpiprocs=2, and adjust --cpu-bind list: and --gpu-bind list: accordingly:

   --cpu-bind list:2-23:26-47
   --gpu-bind list:0:1

   Also update PPN=2 in the script. Submit and inspect the output: how many ranks ran? Which GPUs were used?

   A common mistake is to change mpiprocs but forget to change the -ppn in mpiexec or the bind lists. Re-read your script before submitting.

3. Cancel a job. Submit one more job, then cancel it immediately while it is still queued or running:

   qsub jobscript.pbs
   qstat
   qdel <jobid>
   qstat

   The job should disappear within a few seconds.
4. Inspect a job's resource accounting.

   qsub jobscript.pbs
   qstat -f <jobid> | grep -E "resources_used|exec_host|job_state"

   For finished jobs use qstat -xf <jobid>.

   Note that resources_used.ncpus and ngpus reflect what was reserved via select, not what the application used. Whole-node requests on Hunter leave the per-chunk defaults (ncpus=1, ngpus=0) in place, so these numbers look far too small. Only walltime is a real measurement; for actual usage use node-side tools (rocm-smi, top) from inside the job.

1. Load the helper module and confirm the tools are available:

   module load cae
   type qgen.sh qwtime

2. List the available templates and look at the help for the test template:

   qgen.sh -l
   qgen.sh -h test

   Now generate a job script from it. Note that the template's defaults are tuned for Vulcan, so on Hunter you have to override the node selection explicitly:

   qgen.sh -n 1:node_type=mi300a -t 00:05:00 -o test.pbs test

   Open test.pbs in an editor and skim through it. Note that most of the script is just diagnostic output and not necessary for jobscripts.
3. Submit it and, while it sits in the queue, check the estimated start and remaining walltime:

   qsub test.pbs
   qwtime --help
   qwtime

   With no arguments, qwtime reports on your currently queued/running jobs. The test queue is usually quick; production queues on MI300A nodes can vary substantially depending on the time of day and current cluster load.

1. ⬇  Download pitzDaily-hlrs.zip Download, copy to Hunter, unzip, and review hunter_job.pbs.
2. Submit the simulation and monitor with qstat and tail -f openfoam_pitzDaily.o<jobid>.
3. Once the job completes, verify the results exist in the run.<jobid> directory. Then reconstruct the parallel files on Hunter by submitting reconstruct.pbs.
4. Copy vulcan_vis.pbs from the unzipped folder to Vulcan. Edit the RUN_DIR path with the path to your run.<jobid> on Hunter, then submit.
5. Once visualization finishes, check the output on Vulcan. If you connected with X-forwarding (ssh -X ..., enabled by default in MobaXterm), you can open the images with eog <image file>. Optionally, download the images and create an animated GIF with ffmpeg as shown in the lecture.

1. Cleanup, the right way. A workspace expires automatically, but you should clean up data you no longer need before the deadline so you do not block storage from other users. The proper steps are:

   cd ~/work
   pwd                        # confirm you are in the workspace, not somewhere else
   ls                         # check what you're about to delete
   rm -rf ./*                 # only then delete the contents
   ws_release course-handson  # release the workspace itself

   After ws_release, the workspace enters a grace period during which you can still recover it with ws_restore if you released it by accident.

   
   Removing the symlink does not delete the data. A symlink is just a pointer; rm on a symlink removes only the pointer, not what it points to:

   rm ~/work        # removes the symlink only
   ls $(ws_find course-handson)   # the workspace and its files are still there

   ⚠ Watch the trailing slash. rm -rf ~/work removes the symlink. rm -rf ~/work/ (with trailing slash) follows the symlink and deletes the workspace contents. Same rule applies to rsync and cp -r when symlinks are involved.

• qsub: Job violates queue and/or server resource limits. You requested more nodes or longer walltime than the queue allows. The test queue caps at 25 minutes and 16 nodes. Use qstat -Qf test to confirm.
• command not found inside the job log. The module load in your job script didn't run, or you forgot a module. PBS jobs start with the default environment. The modules from your login session do not carry over.
• Empty .o file and a failing .e file. Check qstat -xf <jobid> for the exit status. A nonzero Exit_status means the program ran but failed; Exit_status equal to 271 means the job was killed for exceeding walltime.
• Job stays in Q for a long time. Either the cluster is busy, or your resource request can't be satisfied at all (e.g. select=500). Use qstat -f <jobid> | grep comment to see why the scheduler is holding it.
• qsub: directive error: -m bae or similar. A typo in a #PBS line. PBS directives are strict. Check the spelling and consult the slide on PBS overview "Anatomy of a Job Script".