Debugging & Dynamic Analysis — Watching a Program in Slow Motion

What you'll learn

By the end of this lesson you will be able to:

• Set up a disposable lab for running code you are studying, and explain why dynamic analysis needs one
• Explain how a debugger works: ptrace, software breakpoints (int3), single-stepping and hardware breakpoints
• Drive GDB: breakpoints, stepping, registers, memory and the process map
• Rebase addresses between a debugger and a disassembler under ASLR
• Read a system-call trace from strace and summarize what a program touched
• Study a bytecode virtual machine by writing a small interpreter with breakpoints

The slow-motion replay 🎬

A football referee who is unsure about a goal does not guess. They watch the replay: pause at the decisive moment, step forward one frame at a time, zoom in on the ball. Nothing about the match changes, but suddenly every detail is visible.

A debugger is that replay for a running program. You pause it at chosen places (breakpoints), advance one instruction at a time (single-stepping), and inspect the "frame": registers, memory, the stack. Dynamic analysis is the wider family of techniques that learn from a program while it runs: debugging, tracing its system calls, watching its files and network.

There is one big difference from static analysis: here the code really runs. So before any command, you need a safe place to run it.

The disposable lab 🧪

Static analysis only reads bytes. Dynamic analysis executes them, with your user's rights, on whatever machine you choose. Even a harmless crackme deserves good habits, because the habits are what protect you on the day a sample is not harmless.

• Run it in a virtual machine, never on your everyday computer. VirtualBox, VMware and QEMU/KVM all support snapshots: take one of a clean system, do your work, then revert. Nothing the program did survives.
• Cut the links to the host: no shared folders, no shared clipboard, no drag and drop. Copy files in once, then disconnect.
• Control the network: disabled, or a host-only network with no route to the internet or to your home or company network.
• Use a separate, unprivileged account inside the VM, and keep your real accounts, keys and browser sessions out of it.
• Write down the file's hash, the snapshot you started from and every command you ran.
• Only run programs that are yours, open source, published for practice, or covered by written permission, exactly as in the first lesson.

Two Linux settings explain behavior you will meet:

• /proc/sys/kernel/randomize_va_space controls ASLR: 2 (the usual default) randomizes the stack, libraries, the heap and PIE executables; 0 turns it off.
• The Yama module's /proc/sys/kernel/yama/ptrace_scope limits who may debug whom: 0 classic rules, 1 only your own descendants (the default on Ubuntu), 2 administrators only, 3 nobody. Starting a program under the debugger (gdb ./prog) makes it a descendant, so it works with 1; attaching to an unrelated running process does not.

Every program run in this lesson was written for the lesson: a small checksum calculator and a program that copies a notes file.

How a debugger really works

On Linux, debuggers and tracers are built on one system call, ptrace(2): it lets a tracer process stop a tracee, read and write its registers and memory, and resume it. GDB, strace and many other tools are ptrace clients.

Software breakpoints. To stop at an address, the debugger overwrites the first byte of the instruction there with 0xCC, the one-byte x86 instruction int3, after saving the original byte. When the CPU executes it, the kernel stops the tracee with SIGTRAP and tells the debugger. At that moment rip already points one byte past the int3, so the debugger:

1. moves rip back by one
2. puts the original byte back
3. shows you the stopped program
4. when you continue, executes one instruction in single-step mode (the trap flag, TF, in RFLAGS makes the CPU stop after every instruction)
5. writes 0xCC again so the breakpoint stays armed, and resumes

This is also why code that checksums itself notices software breakpoints: the bytes in memory really change.

Hardware breakpoints use the CPU's debug registers instead: on x86, DR0 to DR3 hold up to four addresses and DR7 says what to watch (execution, writes, or reads and writes). They do not modify memory, and they power watchpoints (watch counter in GDB stops when a variable changes).

GDB essentials

The commands you will use in almost every session:

set disassembly-flavor intel   # Intel syntax, like the rest of this track
break checksum                 # breakpoint on a function (b for short)
break *0x555555555169          # breakpoint on an exact address
run lab                        # start the program with the argument "lab"
starti                         # or: stop at the very first instruction
x/6i $pc                       # examine 6 instructions at the program counter
stepi / nexti                  # one instruction; nexti steps over calls (si / ni)
info registers rip rsp rbp     # selected registers (i r)
x/2gx $rsp                     # 2 giant (8-byte) words in hex at the stack pointer
x/s $rdi                       # a string at the address in rdi
bt                             # backtrace: who called whom
finish                         # run until the current function returns
watch counter                  # stop when a variable changes
catch syscall write            # stop on a system call
info proc mappings             # the live memory map
continue                       # run until the next stop (c)

The program under the debugger is the lesson's checksum calculator, checksum.c:

#include <stdio.h>
#include <string.h>

/* A tiny program written for the lesson: a checksum of its argument. */
unsigned checksum(const char *s) {
    unsigned sum = 0;
    for (size_t i = 0; i < strlen(s); i++) sum = sum * 31 + (unsigned char)s[i];
    return sum;
}

int main(int argc, char **argv) {
    const char *word = argc > 1 ? argv[1] : "hello";
    unsigned value = checksum(word);
    printf("checksum(%s) = %u\n", word, value);
    return value == 0;
}

A real session on it (built with gcc -O0 -g checksum.c -o checksum, shortened):

(gdb) break *0x555555555169
(gdb) run lab
Breakpoint 1, checksum (s=0x0) at checksum.c:5
(gdb) x/4i $pc
=> 0x555555555169 <checksum>:    endbr64
   0x55555555516d <checksum+4>:  push   rbp
   0x55555555516e <checksum+5>:  mov    rbp,rsp
   0x555555555171 <checksum+8>:  sub    rsp,0x20
(gdb) stepi
(gdb) stepi
(gdb) info registers rip rsp
rip            0x55555555516e      0x55555555516e <checksum+5>
rsp            0x7fffffffc6d0      0x7fffffffc6d0
(gdb) x/2gx $rsp
0x7fffffffc6d0: 0x00007fffffffc700      0x0000555555555202
(gdb) finish
Value returned is $1 = 106893

Read the stack dump with the assembly lesson in mind: after push rbp, the top of the stack holds the caller's saved rbp, and right above it is the return address 0x555555555202 that call pushed, a place inside main. (GDB shows s=0x0 because the prologue has not yet copied the argument from rdi into the stack slot where the debug information says it lives.)

Notice the addresses: 0x5555555.... GDB turns ASLR off for programs it starts (set disable-randomization on is the default), and a 64-bit PIE then lands at 0x555555554000. Outside GDB the base changes on every run.

ASLR and rebasing

You will constantly move between two views of the same code: the disassembler (Ghidra, static addresses) and the debugger (live addresses). They disagree because each one placed the module at a different base:

• the debugger shows runtime addresses: wherever the loader put the module in this run
• Ghidra usually loads a 64-bit PIE at 0x100000, which is why its labels look like FUN_00101169; nm and readelf show plain file offsets from base 0

What never changes is the offset inside the module. So:

offset  = runtime address - runtime base of the module
static  = static base + offset
runtime = runtime base + (static - static base)

Example from the session above: the return address 0x555555555202 minus the base 0x555555554000 is offset 0x1202, which is 0x101202 in Ghidra and 0x1202 in objdump. In the other direction, checksum is 0x101169 in Ghidra, so in this run you break at 0x555555555169.

Find the runtime base in info proc mappings (or /proc/<pid>/maps): it is the start of the module's first mapping, the one with file offset 0. Real map lines from the session:

      Start Addr           End Addr       Size     Offset  Perms  objfile
  0x555555554000     0x555555555000     0x1000        0x0  r--p   .../checksum
  0x555555555000     0x555555556000     0x1000     0x1000  r-xp   .../checksum
  0x555555558000     0x555555559000     0x1000     0x3000  rw-p   .../checksum
  0x7ffff7c00000     0x7ffff7c28000    0x28000        0x0  r--p   .../libc.so.6
  0x7ffff7c28000     0x7ffff7db0000   0x188000    0x28000  r-xp   .../libc.so.6
  0x7ffffffdc000     0x7ffffffff000    0x23000        0x0  rw-p   [stack]

GDB, Ghidra and most disassemblers can also do this for you (in Ghidra, Memory Map → Set Image Base), but you should be able to do it by hand when the tools disagree or when you read a crash log.

Tracing system calls with strace

A program can only touch files, the network or other processes by asking the kernel through system calls. Watching those calls gives you a summary of its behavior without reading a single instruction. On Linux the tool is strace; on Windows, Sysinternals Process Monitor plays a similar role for file, registry and process activity.

strace runs the program, so it belongs in the lab, never on a file you do not trust on your own machine. Real output (strace 6.8) for the lesson's notes program, which copies notes.txt to the screen and appends a line to log.txt (the program's own output goes to /dev/null here, so it does not mix with the trace):

$ strace -e trace=openat,read,write,close ./notes > /dev/null
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\243\2\0\0\0\0\0"..., 832) = 832
close(3)                                = 0
openat(AT_FDCWD, "notes.txt", O_RDONLY) = 3
read(3, "buy milk\n", 64)               = 9
write(1, "buy milk\n", 9)               = 9
close(3)                                = 0
openat(AT_FDCWD, "log.txt", O_WRONLY|O_CREAT|O_APPEND, 0644) = 3
write(3, "read notes\n", 11)            = 11
close(3)                                = 0
+++ exited with 0 +++

How to read a line: name(arguments) = result. openat returns a file descriptor (3), and later calls refer to that number, not to the name. Descriptors are reused: after close(3), the next openat gets 3 again, so you must track which file each number means at that moment. Descriptors 0, 1 and 2 are stdin, stdout and stderr. A failure returns -1 with an error name:

openat(AT_FDCWD, "notes.txt", O_RDONLY) = -1 ENOENT (No such file or directory)

Useful options: -f follows child processes, -e trace=file or -e trace=network selects families, -o trace.txt writes to a file (so the program's own output does not mix in), -s 200 shows longer strings (the default is 32 characters), -p PID attaches to a running process, and -c prints a table of counts. The first lines of any dynamically linked program are the loader at work: opening /etc/ld.so.cache (the index of installed libraries), then libc, reading its ELF header, mapping it. Learn to skip them.

Bytecode virtual machines

Not every program you analyze is machine code. Many run on a virtual machine: a program (the interpreter) that reads instructions for an imaginary CPU, one at a time, and carries them out. You already use several:

• CPython compiles .py files to bytecode (cached in .pyc files) and interprets it
• the JVM runs Java bytecode from .class files, and .NET runs CIL
• Lua, WebAssembly and Linux eBPF programs are bytecode too
• some commercial software protectors translate parts of a program into a custom bytecode with a private instruction set, executed by an interpreter they embed: the original x86 code no longer exists in the file

Every interpreter has the same heart, the dispatch loop:

pc = 0
loop:
    opcode = code[pc]                  # fetch
    handler = table[opcode]            # decode: a switch or a jump table
    run handler, which reads operands  # execute
    and updates pc                     # then go back to "loop"

In a disassembler this shape is easy to spot: one big loop around an indirect jump (jmp [table + rax*8]) or a long switch. A method that works:

1. Find the dispatch loop and the variable that acts as the virtual pc
2. Recover the opcode table: for each handler, what it reads and what it changes
3. Write your own disassembler for the bytecode, so you can read the program
4. Write your own interpreter, with tracing and breakpoints, so you can watch it run, safely and in slow motion, outside the original binary

The standard-library module dis shows CPython's own bytecode. The instructions you get depend on the Python version, which is a good lesson in itself: identify the exact VM version before you trust an opcode table.

Summary and what to practice next

• Dynamic analysis runs the code: always in a disposable VM with snapshots, no shared folders and a controlled network, and only on permitted programs
• Debuggers use ptrace; a software breakpoint is int3 (0xCC) written over an instruction, then rewind, restore, single-step and re-arm; hardware breakpoints use DR0-DR3
• In GDB: break, run, stepi/nexti, x/…, info registers, bt, finish, info proc mappings
• Under ASLR, convert with the module offset: static = static base + (runtime − runtime base)
• strace shows system calls: track descriptors carefully, since they are reused
• Bytecode VMs are studied through their dispatch loop: recover the opcodes, then write your own disassembler and interpreter with breakpoints

Practice next: compile the lesson's checksum.c yourself with -O0 -g, run it under GDB, and find the loop's backward jump in disassemble checksum; set a watch on sum and watch it change. Trace your own small programs with strace -o trace.txt and feed the log to your trace-report tool (leave out -f here: with it, every line starts with a process id that the tool does not expect). Extend your VM with a trace mode that prints every instruction, and a tiny disassembler for the same bytecode. Next: the rules of the road in depth, with law, ethics and responsible disclosure.