Course Agenda

Day 1     Getting Started with Linux

Time Min Topic Format
09:00 15 Welcome & Course Overview Lecture
09:15 65 SSH & Bash Basics Lecture
10:20 30 Break —
10:50 130 SSH & Unix Shell Basics Hands-on
13:00 End

Day 3     Working on the Cluster

Time Min Topic Format
09:00 15 Quiz: Day 2 Quiz
09:15 15 Module & Programming Environment Lecture
09:30 20 Batch System / PBS — Overview Lecture
09:50 5 Energy & Cost Awareness Lecture
09:55 15 Software Available on Hunter Lecture
10:10 10 Profiling (overview) Lecture
10:20 20 Example: OpenFOAM® & ParaView Lecture
10:40 30 Break —
11:10 115 Modules & Job Submission Hands-on
13:00 End

Day 2     Cluster Orientation & Practical Skills

Time Min Topic Format
09:00 10 Quiz: Day 1 Quiz
09:10 20 Security Essentials Lecture
09:30 20 HLRS Systems Overview Lecture
09:50 20 Practical Pitfalls Lecture
10:10 15 SSH Tunneling & Git behind the Firewall Lecture
10:25 15 Software Environments & Containers Lecture
10:40 20 Buffer / Q&A —
11:00 30 Break —
11:30 90 File Transfer & Filesystem Hands-on
13:00 End

HPC Pyramid in Germany

Mission & Vision HLRS

Topics and Technologies

Topics

  • Energy
  • Climate Change & The Environment
  • Health & The Aging Society
  • Mobility in the 21st Century
  • Digital Societies

Technologies

  • AI/Data to Solution (D2S)
  • Cyber Security
  • High Performance Computing
  • Green-IT
  • Customer Base
  • Research
  • Industry
  • Public Agencies

Industrial Customers

Hunter - HLRS’s Flagship Supercomputer

  • HPE Cray EX4000 with the same architecture as the world’s three exascale systems
  • 48.1 PFlop/s peak (FP64), nearly 2× its predecessor Hawk
  • 80% less energy at peak performance compared to Hawk
  • 100% fanless direct liquid cooling
  • Operational since February 2025
  • Transition system toward Herder (exascale, ~2027)
  • #12 on the Green500 (June 2025)
  • Operating System: Red Hat Enterprise Linux
  • Batch system: PBSPro
  • Access via SSH

Vulcan (NEC Cluster)

Vulcan is a heterogeneous computing platform providing specialized resources for pre-/postprocessing, visualization, GPU computing, and data analytics. Access requires a separate allocation - ask your project manager.

  • Operating System: Rocky Linux
  • Batch system: PBSPro
  • Access via SSH

Detailed system overview is given on day 2.

Introduction & Overview

Understanding the foundations of Linux/Unix systems

Terminal vs Shell

  • Terminal (window)
    A text input/output environment (graphical window) that provides access to the shell
  • Shell
    A command-line interpreter that executes commands (e.g., Bash, Zsh, tcsh, etc.)

Why Linux?

  • Nearly all HPC systems run Linux
  • Efficient for remote, text-based workflows
  • Powerful automation through scripting and pipelines

Learning Outcomes

What you will be able to do after this section

Remote Computing
Connect to and use HPC resources efficiently via SSH
File System Navigation
Navigate directories, manipulate files, and understand permissions
Data Processing
Extract, transform, and filter data using command-line tools

Copy & Paste in the Terminal

Ctrl+C does not copy. It cancels the running process.

Platform Action Copy Paste
Linux terminal Keyboard Ctrl+Shift+C Ctrl+Shift+V
Linux terminal Mouse Select text Middle-click
MobaXterm Keyboard Ctrl+Insert Shift+Insert
MobaXterm Mouse Select text Right-click
macOS Keyboard Cmd+C Cmd+V
Windows Terminal Keyboard Ctrl+C Ctrl+V
Windows Terminal Mouse — Right-click

Shell Survival Kit

Two features that make working in a shell easier.

Tab completion

Press Tab to complete a command, file, or path. Press Tab twice to see all matches.

$ cd /et<TAB>            # → /etc/
$ ls /home/<TAB><TAB>    # lists matches
$ ssh hu<TAB>            # → hunter (from SSH config)

Faster than typing, and it prevents typos.

Command history

Key Action
↑ / ↓ Previous / next command
Ctrl+R Search history
history Show full history
!! Repeat last command
!$ Reuse the last argument

Ctrl+R, type a few letters, press Enter to run or → to edit.

Connecting to the Cluster

SSH (Secure Shell)

Log in to a remote system over the network

$ ssh username@hunter-login05.hww.hlrs.de

Copying Files

scp: Secure Copy

$ scp local_file.txt username@hunter-login05.hww.hlrs.de:path/
$ scp local_file.txt username@hunter-login05.hww.hlrs.de:/absolute/path/
$ scp -r username@hunter-login05.hww.hlrs.de:results ./

rsync: Efficient Synchronisation

$ rsync -av local_dir/ username@hunter-login05.hww.hlrs.de:remote_dir/

Only transfers changed files, which is ideal for large datasets. Note: requires a Unix environment on Windows (e.g. MobaXterm, WSL).

Available login nodes:

  • hunter.hww.hlrs.de (selects one login node)
  • hunter-login03.hww.hlrs.de
  • hunter-login04.hww.hlrs.de
  • hunter-login05.hww.hlrs.de
  • vulcan.hww.hlrs.de (selects one login node)
  • cl1fr3.hww.hlrs.de
  • cl1fr4.hww.hlrs.de

Two-factor authentication remains valid for one hour on a specific login node.

SSH Configuration File

Stored in ~/.ssh/config (Linux/Mac/WSL/MobaXterm) or C:\Users\<username>\.ssh\config (Windows OpenSSH)

Linux/Mac/MobaXterm

Host hunter
    HostName       hunter-login05.hww.hlrs.de
    User           <your_HLRS_login>
    IdentityFile   ~/.ssh/HLRS_cs_YYYYMMDD
    IdentitiesOnly yes
    PreferredAuthentications   publickey,keyboard-interactive
    ServerAliveInterval 60
    ServerAliveCountMax 5

Once configured, connect with just:

$ ssh hunter
$ scp local_file.txt hunter:
$ rsync -av local_dir/ hunter:remote_dir/

Windows (OpenSSH)

Host hunter
    HostName       hunter-login05.hww.hlrs.de
    User           <your_HLRS_login>
    IdentityFile   C:\Users\<username>\.ssh\HLRS_cs_YYYYMMDD
    IdentitiesOnly yes
    PreferredAuthentications   publickey,keyboard-interactive
    ServerAliveInterval 60
    ServerAliveCountMax 5

MobaXterm: do this before the Linux steps:

$ mkdir -p ~/.ssh && chmod 700 ~/.ssh
$ cp /drives/c/Users/lucas/.ssh/HLRS_cs_20250728 ~/.ssh/
$ chmod 600 ~/.ssh/HLRS_cs_20250728

ServerAliveInterval 60 - sends a keepalive every 60 seconds
ServerAliveCountMax 5 - drops connection after 5 missed responses (~5 min)
IdentitiesOnly yes - only use the specified key, don’t try others
PreferredAuthentications publickey,keyboard-interactive - try key-based login first, fall back to keyboard-interactive (OTP/challenge-response); skips plain password authentication, which HLRS does not support

Linux/Unix Filesystem, Paths & Navigation

Filesystem Tree

Single-rooted, inverted tree starting at /

/ (root)
├── bin    Commands
├── etc    Configuration
├── home   User home dirs
│   ├── user1
│   └── user2
├── usr    Programs
├── var    Logs, spool
├── tmp    Temporary files
└── opt    Add-on software

Paths

Absolute: start at /, always unambiguous:

/home/user/documents/file.txt

Relative: relative to current directory:

documents/file.txt

Shortcuts

Shortcut Meaning
. Current directory
.. Parent directory
~ Home directory
- Previous directory

Commands

$ pwd
/home/user/documents
$ cd /absolute/path
$ cd ..   # up one level
$ cd ~    # home
$ cd -    # previous
$ ls
$ ls -l   # detailed
$ ls -la  # + hidden
$ ls -lh  # human readable

Viewing File Contents & File Operations

cat: Print Entire File

$ cat file.txt

less: Scrollable Viewer

$ less large_file.log    # Navigate with arrow keys,
                         # q to quit, /pattern to search

head / tail: First or Last Lines

$ head -n 20 file.txt    # First 20 lines
$ tail -n 10 output.log  # Last 10 lines
$ tail -f output.log     # Follow file in real time

touch: Create Empty File (or update timestamp)

$ touch newfile.txt

mkdir: Make Directory

$ mkdir new_directory
$ mkdir -p parent/child/grandchild   # Create nested dirs

cp: Copy

$ cp source.txt destination.txt
$ cp -r source_dir/ dest_dir/   # Copy directory recursively

mv: Move or Rename

$ mv file.txt new_location/
$ mv old_name.txt new_name.txt

rm: Remove

$ rm file.txt
$ rm -r directory/   # Remove directory and contents

Careful: There is no recycle bin on the command line!

Wildcards (Globbing)

The shell expands these characters in paths to match multiple files before running the command.

Pattern Matches Example
* any sequence (except leading .) *.txt matches notes.txt
? any single character file?.log matches file1.log
[abc] any one of the listed characters [0-9]*.csv starts with digit
{a,b} brace expansion {src,test}/*.py
$ ls *.csv                       # all CSV files in current dir
$ rm tmp_*                       # remove all files starting with tmp_
$ cp data/*.txt backup/          # copy all .txt files

$ ls run_?.log                   # single character: run_1.log, run_a.log
$ ls sample_[0-9].dat            # character class: sample_0.dat ... sample_9.dat
$ mkdir {input,output,logs}      # brace expansion: creates three directories

These are not regular expressions. They are filename patterns evaluated by the shell.

Editing Files: nano (Vulcan only)

A beginner-friendly terminal text editor.

$ nano myfile.txt

Key bindings (^ means Ctrl, displayed at the bottom of the screen):

Key Action
Ctrl+O Save (write out)
Ctrl+X Exit
Ctrl+W Search
Ctrl+K Cut a line
Ctrl+U Paste
Ctrl+G Help

Alternatives: vim and emacs are also available but have steeper learning curves.

Editing Files: vim

A powerful terminal text editor available on virtually every Unix system.

$ vim myfile.txt
vim is modal — keys behave differently depending on the current mode:
Mode Enter with Purpose
Normal Esc Navigate, delete, copy
Insert i Type text
Command : Save, quit, search
Key Mode Action
i Normal Enter insert mode
Esc Insert Return to normal
:w Command Save
:q Command Quit
:wq Command Save and quit
:q! Command Quit without saving
dd Normal Delete line
yy / p Normal Copy / paste line
/pattern Normal Search
n / N Normal Next / prev match
v Normal select text

Tip: If you get stuck, press Esc a few times and type :q! to exit without saving.

Finding Files and Getting Help

find: Search for Files

$ find . -name "*.csv"                  # Find by name pattern
$ find /home -type d -name "results"    # Find directories only
$ find . -name "*.log" 2>/dev/null      # Suppress permission errors

which: Locate a Command

$ which python
/usr/bin/python

man and --help: Documentation

$ man grep        # Open the manual for grep
$ man -k search   # Search for commands by keyword
$ ls --help       # Quick built-in help (most commands support it)

Press q to quit man.

Output of man find (Optionality is indicated by square brackets []):

File Permissions

Permission Types

Each file has three permission groups: owner, group, others

Symbol Meaning Value
r Read 4
w Write 2
x Execute 1

Reading ls -l Output

drwxr-xr-x  2  user  group  4096  Mar 27 14:32 my_dir
│└┬┘└┬┘└┬┘
│ │  │  └── others: r-x (5)
│ │  └── group:  r-x (5)
│ └── owner:  rwx (7)
└── file type (- = file, d = directory)

Create an executable script

$ echo '#!/bin/bash' > hello.sh  # shebang: which interpreter to use
$ echo 'echo Hello' >> hello.sh  # the actual command
$ chmod u+x hello.sh             # make it executable
$ ./hello.sh
Hello

Changing Permissions: chmod

Symbolic notation:

Target Meaning
u user (owner)
g group
o others
a all (ugo)
Operator Effect
+ add permission
- remove permission
= set exactly
$ chmod u+x script.sh         # add execute for owner
$ chmod go-w file.txt         # remove write for group & others
$ chmod -R g+rX shared_data/  # +r on all, +x on directories only (capital X)
$ chmod -R g+rx shared_data/  # AVOID on data: marks every file executable

Uppercase X adds execute only to directories and already-executable files.

Numeric notation: each permission group is a sum of r=4, w=2, x=1:

owner group others
r 4 4 4
w 2 0 0
x 1 1 1
= 7 5 5
$ chmod 755 script.sh    # rwxr-xr-x
$ chmod 600 secret.key   # rw-------
$ chmod 750 shared_data/          # group can enter/ls
$ chmod 640 shared_data/file.txt  # group can read

Sharing Folders

Manual: chgrp + chmod

chgrp sets which group has access; chmod sets what kind.

$ id                            # your groups
$ chgrp -R mygroup shared_data/ # assign group
$ chmod g+s shared_data/        # new files inherit the group
$ chmod -R g+rX shared_data/    # group read + traverse
$ namei -l $PWD/shared_data/    # check the whole path

New files get your primary group automatically, so if you and your colleague already share a group, chmod alone is enough.

Every parent directory must be traversable (x) by the group, or they can’t reach the folder.

Manual sharing is not automatic: rerun chmod as new files are added.

The right way to share a workspace with colleagues on HLRS systems. Ask each colleague for their username (whoami).

$ ws_share share my_workspace alice   # grant read access
$ ws_share share my_workspace bob
$ ws_share list my_workspace          # who has access
$ ws_share unshare my_workspace bob   # revoke one
$ ws_share unshare-all my_workspace   # revoke all

On Hunter, ws_share automatically covers newly created files, so no re-running needed.

Environment Variables

Dynamic values that affect processes running on a system

Important Variables

Variable Purpose
PATH Directories searched for executables
HOME User’s home directory
USER Current username
SHELL Path to current shell

Common Operations

$ echo $PATH                          # View a variable
$ export MY_VAR="hello"               # Set a variable
$ export PATH=$PATH:/my/custom/bin    # Extend PATH

Add exports to ~/.bashrc to make them persistent across sessions.

The default $PATH on hunter:

Quoting

Spaces separate command arguments. Quote when arguments contain spaces or special characters.

Quotes Behavior
'single' Literal, no expansion at all
"double" Variable expansion ($VAR), no globbing/word-split
(none) Word-splitting, glob expansion, variable expansion
\char Escape one character (Hello\ World = "Hello World")
$ touch "my file.txt"        # filename with a space
$ echo "$HOME"               # expands variable
$ echo '$HOME'               # prints literal $HOME
$ grep "TODO list" notes.txt # multi-word pattern

Common gotcha: unquoted empty variables disappear

$ FILE=""
$ rm $FILE        # → "rm" with no args (unhelpful error)
$ rm "$FILE"      # → rm: cannot remove '': clearer error

Common gotcha: filenames with spaces break without quotes

$ ls my file.txt          # lists "my" and "file.txt" separately
$ ls "my file.txt"        # correct
$ ls my\ file.txt         # also correct: escape the space
$ cp my file.txt /tmp/    # copies "my" and "file.txt" separately
$ cp "my file.txt" /tmp/  # correct

Standard Streams

Every command has three I/O streams:

ID Name Default
0 stdin (input) keyboard
1 stdout (output) terminal
2 stderr (errors) terminal

stdout and stderr both go to the terminal but travel separately, so we can redirect them independently.

$ ls /etc /nonexistent > out.txt       # only stdout → file
$ ls /etc /nonexistent 2> err.txt      # only stderr → file
$ ls /etc /nonexistent > all.txt 2>&1  # both → same file
$ command < input.txt                  # stdin from file

/dev/null: the discard bin

/dev/null is a special file that discards everything written to it:

$ find / -name "*.conf" 2>/dev/null # discard "Permission denied"

Combine it all:
$ command < input.txt > output.txt 2> error.txt

Pipelines, Redirection & Command Chaining

The Pipe Operator |

Sends stdout of one command as stdin to the next:

$ ls -l | grep ".txt"                    # filter for .txt files
$ cat job.out | grep "ERROR" | wc -l     # count errors in job output
$ ls -lh *.log | sort -k5 -h | tail -n 5 # largest 5 log files

Redirection Operators

Operator Function Example Note
> Redirect stdout to file echo "hi" > file.txt overwrites!
>> Append stdout to file echo "hi" >> file.txt
&> Redirect both streams together cmd &> all.log overwrites!

Process substitution

Feed command output as if it were a file:

$ diff <(sort file1.txt) <(sort file2.txt) # compare sorted versions

Command Chaining

Operator Behaviour Example
; Sequential, regardless of success cd /data; ls -l
&& Second runs only if first succeeds mkdir data && cd data
|| Second runs only if first fails cd dir || mkdir dir

tee: write and watch simultaneously

tee writes output to a file while still printing to the terminal:

$ ./simulation | tee output.log     # see output AND save to file
$ ./simulation | tee -a output.log  # append instead of overwrite

Useful when you want to monitor a long-running job live but also keep a log.

Text Processing: grep & Regular Expressions

grep: Search Text for Patterns

$ grep "error" logfile.txt      # lines containing "error"
$ grep -i "warning" logfile.txt # case-insensitive
$ grep -v "debug" logfile.txt   # lines NOT matching
$ grep -r "TODO" src/           # search recursively
$ grep -c "error" logfile.txt   # count matching lines

Context Flags

$ grep -A 3 "FATAL" logfile.txt    # 3 lines After match
$ grep -B 2 "FATAL" logfile.txt    # 2 lines Before match
$ grep -C 2 "FATAL" logfile.txt    # 2 lines of Context

Regular Expression Basics

Patterns used by grep -E, sed, and awk:

Symbol Meaning Example
. Any single character g.t matches got
* Zero or more ab*c matches ac
+ One or more (ERE) ab+c matches abc
? Zero or one (ERE) colou?r
| or operator (ERE) cat|dog
^ Start of line ^Error
$ End of line done$
[abc] Character class [0-9] matches digit

Use grep -E for +, ?, and |. Regex matches text inside files; unlike globbing, which matches filenames.

$ grep -v "^#" config.cfg            # skip comment lines
$ grep "node[0-9][0-9]" hosts.txt    # character class
$ grep -E "warn|error|fatal" app.log # alternation

Process Management & Archives

Inspecting Processes

$ ps aux                    # List all running processes
$ ps aux | grep python      # Filter for a specific program

Controlling Processes

$ kill 1234                 # Terminate process by PID
$ kill -9 1234              # Force kill
$ long_process &            # Run in background
$ jobs                      # List background jobs
$ fg %1                     # Bring job 1 to foreground

Ctrl+C cancels the current process, Ctrl+Z suspends it.

tar: Create & Extract Archives

$ tar -czf archive.tar.gz my_dir/     # Create compressed archive
$ tar -xzf archive.tar.gz             # Extract archive
$ tar -tzf archive.tar.gz             # List contents without extracting

Flags

Flag Meaning
-c Create archive
-x Extract archive
-z Compress with gzip
-f Specify filename
-t List contents
-v Verbose output

Useful One-Liners for HPC

Disk & File Usage

$ du -sh */                        # size of each subdirectory
$ du -sh * | sort -hr | head -10   # top 10 largest items
$ df -h                            # filesystem disk space usage
$ find . -name "*.log" -size +100M # logs over 100MB

Counting & Sorting

$ wc -l results/*.csv       # count lines in all CSV files
$ sort -nr sizes.txt | head # top 10 largest values

Monitoring & Combining

$ watch -n 5 'ls -lh output/'  # monitor output directory every 5s

watch reruns a command at a fixed interval, useful for monitoring job output as it grows.

$ find . -name "*.csv" \
  | xargs wc -l \
  | sort -nr \
  | head -5

Find all CSV files, count their lines, show the 5 largest.

Hands-On Exercises

Hands-on SSH

Set up your SSH connection

Hands-on Unix Shell Basics

Working in a terminal

  • While logged in to the cluster, go through the following exercises:

High-Performance
Computing Center
Stuttgart

Lucas Jordan

Introduction to the HLRS Systems