===============================================================================
 Finding and Fixing Bugs on Hunter with Linaro DDT  -  Hands-on, Thu 15 Oct
===============================================================================

Everything you need is in this file. Work through it from top to bottom.
Stuck for 10 minutes? Read the hints, or ask in Slack.

  Part 0  Before you start
  Part 1  Set up your own copy                               (10 min)
  Part 2  The saxpy GPU example                              (15 min)
  Part 3  CloverLeaf: what it does, and is my answer right?  (10 min)
  Part 4  CloverLeaf: four bugs, one after the other    (20 min each)
  Part 5  Quick reference

Never run a program on the login node: it has no GPU. Build on the login
node, run everything through "qsub job.sh".


-------------------------------------------------------------------------------
 Part 0  Before you start
-------------------------------------------------------------------------------

You need:
  - Your Hunter account, password and two-factor (one-time code) app.
  - The Linaro Forge 26.0.3 remote client installed on your laptop.
  - In the remote client: Remote Launch -> Configure:
        Host Name:                       <username>@hunter.hww.hlrs.de
        Remote Installation Directory:   /lustre/hpe/ws12/ws12.a/ws/vmprusha-ddt-training/linaro/forge/26.0.3
    Then choose that connection in the client. It asks for your password and
    your one-time code, then waits for a program to connect to it.

How to get a debugging session on Hunter:
  1. You submit a job with qsub.
  2. When the job starts, DDT on Hunter asks your remote client to connect.
     A "Reverse Connect request" pops up in the client: click Accept.
  3. The Run dialog opens. Tick "ROCm" (GPU debugging), then click Run.
  4. Closing DDT ends the job. Please close it when you finish, so the nodes
     are free for others.


-------------------------------------------------------------------------------
 Part 1  Set up your own copy
-------------------------------------------------------------------------------

Log in to Hunter (ssh <username>@hunter.hww.hlrs.de), then:

  # 1. Create your own workspace (30 days) and go there
  ws_allocate ddt-work 30
  cd $(ws_find ddt-work)

  # 2. Copy the course files into it
  cp -r /lustre/hpe/ws12/ws12.a/ws/vmprusha-ddt-training/hunter-handson .
  cd hunter-handson

  # 3. Build both the saxpy and the CloverLeaf examples
  make gpu COMPILER=cray
  cd CloverLeaf_HIP/src && make COMPILER=CRAY DEBUG=1)

Run every qsub from the hunter-handson folder. Never build or run in the
course folder (ddt-training): it is shared by everyone.

Useful job commands:
  qstat -u $USER        see your jobs (Q = waiting, R = running)
  qdel <job id>         cancel a job
  ddt-examples.o<id>    the job's output (.e<id> for errors), in hunter-handson


-------------------------------------------------------------------------------
 Part 2  The saxpy GPU example
-------------------------------------------------------------------------------

What it does: each of 4 MPI processes computes  z = a * x + y  for 1024 values
on the GPU, then checks the answer on the CPU. One value comes out wrong.

Debug it:

  qsub job.sh

  -> Accept the Reverse Connect request, tick ROCm, click Run.
  -> DDT stops every process just after MPI_Init.

Things to try:
  1. Click Play. Each process prints "running on the GPU" and then
     "FAILED CHECK (1 of 1024 values wrong)". Which value is wrong?
  2. Restart (File -> Restart Session). Set a breakpoint on the line inside
     the second GPU kernel (click in the margin next to the line number).
     Click Play: DDT stops within the GPU kernel.
  3. Look at the GPU Thread Selector at the top and the Parallel Stack view
     at the bottom. How many GPU threads are there? Select a different thread.
     (On AMD GPUs, threads run in groups of 64 called wavefronts. DDT steps a
     whole wavefront at once.)
  4. Look at the loop variable i in the Locals window. Which values of i does
     the kernel work on? Compare with the size of the array.
  5. Open the GPU Devices tab: which GPU are you running on?
  6. hightlight and right-click z[i] = a * x[i] + y[i] and add to Evaluations

Hint:   The wrong value is always the same one, on every process.
Hint 2: Set a breakpoint on the Error string.


-------------------------------------------------------------------------------
 Part 3  CloverLeaf: what it does, and is my answer right?
-------------------------------------------------------------------------------

CloverLeaf simulates gas in a closed box (10 x 2). A dense, high-pressure gas
on the left pushes into a light gas on the right. The box is cut into 4
vertical strips, one per MPI process; the maths runs on the GPU.

Each time step the GPU kernels (in CloverLeaf_HIP/src/kernel_files) update
pressure, velocity and density. After the GPU work, each process swaps the
cells along its edges with its neighbours (the "halo exchange", in
clover_exchange_message in clover.f90). The cells to send are copied off the
GPU by "pack" kernels (kernel_files/pack_kernel.cuknl).

Nothing can leave the box, so in the output file clover.out:

  Volume  = 0.2000E+02  at every stezp  (the box is 10 x 2)
  Mass    = 0.1200E+02  at every step  (light gas 0.2 x 10 + dense gas 1.0 x 10)
  The correct run ends after 535 steps, Total Energy 0.2697E+02

To see them, from the hunter-handson folder:

  grep -A1 "step:" CloverLeaf_HIP/src/clover.out | head -4    # start (step 0)
  grep -A1 "step:" CloverLeaf_HIP/src/clover.out | tail -2    # end of the run

Run CloverLeaf under DDT (always 4 processes):

  qsub -v EXAMPLE=cloverleaf job.sh

To just let it run: click Play in DDT and wait. clover.out is written in
CloverLeaf_HIP/src.

After every change to the code, rebuild before you run again:

  cd CloverLeaf_HIP/src
  make clean
  make COMPILER=CRAY DEBUG=1
  cd ../..


-------------------------------------------------------------------------------
 Part 4  CloverLeaf: four bugs, one after the other
-------------------------------------------------------------------------------

The code you copied has 4 bugs. Fixing one reveals the next. You get about
15 minutes for each bug and 5 minutes instructor led to fix.

..........................................................................
 Bug 1   "It never finishes"
..........................................................................
Symptom:  The job starts, clover.out stops after the set-up lines (there is
          no "step: 0" summary), and the program never ends.
Question: What is each process waiting for?
Try:
  - Click Play, wait until nothing happens any more, then click Pause.
  - Look at the Parallel Stack view at the bottom. Where is each process?
  - Then click processes 0, 1, 2 and 3 in turn.
  - In the Locals window compare tag, sender and receiver.
Hint:     A receive only matches a send from the right process with the
          same tag.
Hint 2:   Compare how the tags are worked out for the sends and the receives
          in clover_exchange_message (clover.f90).
Done when: clover.out shows the step 0 summary.
DDT features: Pause, Parallel Stack view, Focus on Process, Locals.

..........................................................................
 Bug 2   "The answer is incorrect"
..........................................................................
Symptom:  At step 0, Volume is 0, Mass is 0 and Density is NaN. The run stops
          almost at once with "small timestep".
Question: Is the GPU doing any work at all?
Try:
  - The volume is just the size of the box. It can't be 0. Follow the 0
    back: open timestep.f90 and put a breakpoint where dt is set from the
    GPU result (dt=dtlp). What value comes back?
  - Set a breakpoint inside a GPU kernel, e.g. in
    kernel_files/ideal_gas_kernel.cuknl. Restart and click Play. Does DDT
    ever stop there?
  - Open the GPU Devices tab: which GPU is in the node?
Hint:     Compare the GPU in the node with the GPU the code was built for.
Hint 2:   Look for --offload-arch in CloverLeaf_HIP/src/Makefile.
Done when: step 0 shows Volume 0.2000E+02 and Mass 0.1200E+02.
DDT features: GPU breakpoints, Current Line(s), GPU Devices tab.

..........................................................................
 Bug 3   "It crashes at step 1"
..........................................................................
Symptom:  Step 0 is correct, then the program stops at step 1 with
          "free(): double free detected" (in DDT's Input/Output tab, or the
          job's .e<id> file).
Question: Which line frees memory, and why is it a problem?
Try:
  - Restart. In the Run dialog, tick "Memory Debugging", then Run and Play.
    DDT stops and tells you what went wrong, and where.
  - Look at the Stack: which function of CloverLeaf were you in?
Hint:     Memory can be freed (deleted) only once.
Done when: the run goes past step 1.
DDT features: Memory debugging, Stack.

..........................................................................
 Finished early?
..........................................................................
  - All 4 processes may be using the same GPU. Check in DDT's GPU Devices
    view. How would you give each process its own GPU?
  - In the step output, the "control" cell position is outside the box
    (x = -5.00E-03). Where does that position come from?


-------------------------------------------------------------------------------
 Part 5  Quick reference
-------------------------------------------------------------------------------

Commands (run from hunter-handson):
  make gpu COMPILER=cray                            build saxpy
  qsub job.sh                                       debug saxpy
  (cd CloverLeaf_HIP/src && make clean && make COMPILER=CRAY DEBUG=1)
                                                    build CloverLeaf
  qsub -v EXAMPLE=cloverleaf job.sh                 debug CloverLeaf
  grep -A1 "step:" CloverLeaf_HIP/src/clover.out    check the results
  qstat -u $USER / qdel <job id>                    check / cancel jobs

DDT:
  Play / Pause / Step over / Step into     toolbar at the top
  Breakpoint                               click in the margin next to a line
  Conditional breakpoint                   Breakpoints tab -> Condition column
  Evaluate window                          right-click -> Add Expression
  Compare across processes                 right-click a variable
  Multi-dimensional Array Viewer           right-click an array -> View Array (MDA)
  Parallel Stack view                      tab at the bottom
  GPU Thread Selector                      bar at the top, when stopped on GPU
  GPU Devices                              tab on the right
  Memory debugging                         Run dialog -> Memory Debugging
  Restart                                  File -> Restart Session

For your own code this afternoon:
  - Compile with -g (and -O0 if you can).
  - GPU code on Hunter: --offload-arch=gfx942.
  - Copy job.sh and change the ddt line to run your own program.
===============================================================================
