Lesson 11.6 — Safety checks: trap vs unwind vs error return¶

Techniques: three ways a failed run-time check (overflow, bounds, division by zero) can end: trap — stop the program at once with a diagnostic, no cleanups (Pebble, Swift, Clang's -fsanitize-trap, Rust with panic=abort); unwind — raise a panic or exception that runs the cleanups of every frame on the way to a handler or to the top (Rust's default panic=unwind, Go's run-time panics, Java's ArithmeticException); error return — the operation returns a value that says it failed (Rust's checked_add returning Option, C's __builtin_add_overflow, Go's (v, err)) · Pebble implements: traps: assert in PIR → a branch to a cold block that calls pebble_trap (E1, E5) · Prerequisites: Lesson 11.2 (branch costs); overflow intrinsics (Ch 9) · Time: 3 hours

Pebble's + traps on overflow (spec §8.3): print(big + 1) prints nothing and exits with status 101 after writing pebble: trap: arithmetic overflow at trap.pbl:4:11. That is one policy. The same check can instead unwind (Rust in a debug build panics, and the panic runs the destructors of every frame it leaves) or turn into a value (Rust's a.checked_add(b) is None on overflow). The check is the same llvm.sadd.with.overflow in all three; what differs is the code on the failure path, what the optimizer may assume, and what the caller must do. The running example is one addition, a + b on 64-bit integers, under each policy (the Rust and Clang boxes in §7 compile exactly this).

1. Problem and motivation¶

Trap¶

A trap is the simplest failure path: call a function that never returns (noreturn), print what failed and where, exit. Nothing after the failed check runs — not even cleanups — so the optimizer knows that on the continuing path the check succeeded, and the failure path can be moved out of the way (cold). Swift traps on overflow, bounds and forced unwrapping of nil (the SIL instruction cond_fail becomes llvm.trap) [SWIFT-CondFail]; Clang's -fsanitize=signed-integer-overflow -fsanitize-trap produces llvm.ubsantrap [CLANG-CGExpr]. Pebble traps through pebble_trap(kind, file, line, column), which flushes standard output, prints the message and exits with 101 (docs/runtime-abi.md §6): the location makes traps debuggable, and exiting with a fixed status makes them testable (the e2e suite's EXPECT-TRAP).

Unwind¶

In a language with destructors (C++, Rust, Swift's defer) stopping at once would skip cleanups: locks stay held, buffers unflushed. Unwinding transfers control from the failure point to the nearest handler, running the cleanup code of every frame it leaves, and can be caught (catch_unwind, recover, catch). Rust's panic=unwind, Go's run-time panics (panicIndex, recoverable with recover) [GO-Bounds] and Java's run-time exceptions work this way. The mechanism is exception handling — Lesson 11.7 covers how the unwinder finds the cleanups.

Error return¶

When failure is an expected outcome rather than a bug, the result type says so: Rust's checked_add returns Option<i64>, C has __builtin_add_overflow(a, b, &r), Go returns (v, err). There is no hidden control flow: the caller must inspect the result, and the compiler checks that it does (Rust's ? operator [RUSTC-Try]). The cost moves to every caller: a branch per call level.

2. Definitions and algorithms¶

Definition 11.6.1 (Checked operation, failure policy)

A checked operation is a pair of a predicate $\mathrm{fail}(\vec{a})$ and a wrapping operation $\mathrm{op}(\vec{a})$ that agrees with the mathematical result whenever the predicate is false (for +: $\mathrm{fail} = $ saddo, $\mathrm{op} = $ add, pir-spec §9.1). A failure policy says what happens when the predicate is true: trap — the program ends immediately with a diagnostic, running no further code of any frame; unwind — control transfers to the nearest enclosing handler, running the cleanups of every frame it leaves; error return — the operation's result is a value of a sum type that distinguishes success from failure, and control continues normally.

Algorithm 11.6.2 (Lowering one check under each policy)

Input: a checked operation with arguments $\vec{a}$, a trap kind and a source location.
Output: LLVM IR.
Precondition: the predicate and the wrapping operation are side-effect free (rvalues, pir-spec §9).
Postcondition: Theorem 11.6.3.
Invariant: on the continuing edge the predicate is false.

function Trap(a, kind, loc):                      # Pebble (E5), Swift, -fsanitize-trap
    (r, f) ← op.with.overflow(a)                  # the flag is the predicate
    br f, Fail, Ok
    Fail: call noreturn cold trap(kind, file, loc.line, loc.col); unreachable
    Ok:   continue with r

function Unwind(a, msg):                          # Rust panic=unwind, Go, Java
    (r, f) ← op.with.overflow(a)
    br f, Fail, Ok
    Fail: call noreturn panic(msg)                # may unwind: in a frame with live cleanups
                                                  # this is an `invoke` with a landing pad
    Ok:   continue with r

function ErrorReturn(a):                          # Rust checked_*, C __builtin_*_overflow
    (r, f) ← op.with.overflow(a)
    return (tag ← not f, value ← r)               # no branch; the caller inspects the tag

Theorem 11.6.3 (Each policy implements its semantics)

Let a program perform a checked operation at location $\ell$. (i) Under trap, the observable behavior of the lowered program is that of the source: all effects before $\ell$, then the diagnostic for $\ell$, then exit status 101 — nothing after $\ell$. (ii) Under unwind, the effects are those before $\ell$, then the cleanups of the frames between $\ell$ and the handler in innermost-first order, then the handler. (iii) Under error return, execution continues with a value whose tag is false exactly when the predicate held, and whose payload is the correct result otherwise.

Proof

(i) The predicate is computed by a side-effect-free rvalue, so computing it changes nothing. If it is false, the Ok edge continues with $r$, which by Definition 11.6.1 equals the mathematical result. If it is true, the only instruction executed is the call to the trap function, which flushes output (so earlier effects are visible), prints the diagnostic with $\ell$'s location and calls exit(101); since it is noreturn, no code after it runs — the unreachable is never reached. (ii) Same up to the call; the panic function raises an exception, and the unwinder (Lesson 11.7, Theorem 11.7.3) runs the landing pads of the frames between in order. (iii) There is no control transfer: the tag is $\neg f$ and the payload $r$, correct when $f$ is false by Definition 11.6.1.

Lemma 11.6.4 (What the optimizer may assume after a trap check)

Under the trap policy, every instruction dominated by the Ok edge of a check may assume the predicate is false. Under unwind the same holds for instructions in the same frame; under error return nothing may be assumed about the payload unless the tag is tested.

Proof

Under trap and unwind the Fail block ends in a noreturn call, so every path to an instruction dominated by the Ok edge passed through that edge, where $f$ is false; $f$ is an SSA value, so it is still false. (Under unwind, execution may continue in a caller's handler — but that code is not dominated by the Ok edge.) Under error return both outcomes continue into the same code, so only a test of the tag establishes anything.

3. Worked example¶

Trap¶

big + 1 with big = $2^{63} - 1$, in Pebble (spec §17 "Traps"):

stage	code for `big + 1` at 4:11
PIR (E1)	`_3 = saddo _0, 1 @4:11` · `assert !_3, overflow @4:11` · `_4 = add _0, 1 @4:11`
LLVM IR (E5)	`%s = call {i64,i1} @llvm.sadd.with.overflow.i64(...)`; `br i1 %f, label %trap.overflow, label %assert.ok`
failure block	`call void @pebble_trap(i32 1, ptr @.str, i32 4, i32 11)` · `unreachable`
run	prints `-9223372036854775808` (the wrapping `&+` before), then the trap line, status 101

The overflow flag of sadd.with.overflow is the predicate, the sum is the wrapping add; LLVM's optimizer (pebblec -O2, or --passes=instcombine after mem2reg) merges PIR's separate saddo and add into one intrinsic call (pir-spec §9.1); pebblec -O1 runs only the course pipeline of registered steps (Lesson 11.9 §2), so at -O1 the two stay separate until a chapter registers a promotion step.

Unwind¶

Rust a + b with overflow checks: the same intrinsic and branch, but the failure block calls core::panicking::panic_const::panic_const_add_overflow, which can unwind. In a caller that holds a value with a destructor (Guard in the box of §7), the call to add becomes invoke … unwind label %cleanup, and the landing pad runs the destructor before resume continues unwinding.

Error return¶

Rust a.checked_add(b) returns Option<i64> as { i64, i64 }: tag = zext (not f), payload = the sum; no branch at all in the callee — the caller branches when it matches on the result.

4. Invariants and correctness¶

Trap¶

Theorem 11.6.3 (i) and Lemma 11.6.4. The trap policy needs the trap function to be noreturn and not to unwind (nounwind): the code generator declares pebble_trap as noreturn nounwind cold (docs/runtime-abi.md §2). If the trap function returned, the wrapping operation would run on the failure path; if it unwound, frames' cleanups would run — both would change observable behavior.

Using llvm.trap for Pebble's traps

llvm.trap becomes an illegal instruction (ud2 on x86-64): the process dies with SIGILL, exit status 132 and no message. The e2e tests expect pebble: trap: … at file:line:col and status 101, which only the runtime's pebble_trap produces (docs/runtime-abi.md §8, rule 3).

Unwind¶

Correctness needs every call that may unwind through a frame with live cleanups to be an invoke with a landing pad for them (Lesson 11.7, Definition 11.7.1); Rust's MIR records an unwind edge on every call terminator for this reason, and panic=abort removes them all (box in §7).

Error return¶

Theorem 11.6.3 (iii); the invariant is that the payload is meaningless when the tag is false. Languages enforce it with types (Option must be matched; Rust warns on an unused Result via #[must_use]); C's __builtin_add_overflow relies on the programmer.

5. Complexity¶

$d$ = frames between a failure and its handler, $c$ = calls between them that carry an error value, $k$ = checks executed.

Policy	Cost when nothing fails	Cost of one failure	Code size	Justification
Trap	per check: 1 flag test + 1 well-predicted branch	$O(1)$ + process exit	one cold call per check (shareable per kind and location)	Algorithm 11.6.2; the Fail block is `cold` and moved out of line
Unwind	per check: the same; per call in a frame with cleanups: nothing executed, but `invoke` constrains code motion	$O(d \cdot t)$: $t$ = table lookup per frame (Lesson 11.7)	landing pads + unwind tables	zero-cost EH moves the work to the throw
Error return	per call level that propagates: 1 test + 1 branch, even when nothing fails	$O(c)$ tests on the way back	one test per propagation site	the tag is checked at each `?`

Pathological family. A loop for i in 0..n { s += xs[i]; } performs $2n$ checks (bounds and overflow) under trap: $2n$ extra compare-and-branch pairs, which is why bounds-check elimination (Ch 18) and range analysis exist. For error return, a recursion of depth $d$ whose base case fails pays $d$ tests on the way back and $d$ tests on every successful return too — the cost that motivated zero-cost unwinding.

Trap¶

Trap with a message vs a bare trap instruction: pebble_trap(kind, file, line, col) vs llvm.trap/llvm.ubsantrap(k) — the message costs a string constant and a call per site; -fsanitize-trap trades the diagnostic for size (UBSan's runtime mode, -fno-sanitize-trap, prints a diagnostic and may continue).
Merging failure blocks: all checks of one kind can branch to a shared trap block (LLVM does not merge calls with different debug locations unless allowed), saving size at the cost of a less precise location.

Unwind¶

panic=abort (Rust) turns every panic into a trap: all invokes become calls and the landing pads disappear (box in §7) — smaller, faster, no cleanups.
nounwind inference (function-attrs, Ch 20) proves that a callee cannot unwind and turns the caller's invoke into a call.

Error return¶

Error in a register (Swift's throws): the callee returns the error in a dedicated register (swifterror, r12 on x86-64) and the caller tests it after every call — error return with a calling-convention assist (Lesson 11.7).
Out-parameters and errno: C's older style; the flag lives in memory, which blocks optimization.

7. In real compilers¶

Trap¶

Clang: CodeGenFunction::EmitTrapCheck in clang/lib/CodeGen/CGExpr.cpp [CLANG-CGExpr]. Swift: IRGenSILFunction::visitCondFailInst in lib/IRGen/IRGenSIL.cpp [SWIFT-CondFail]. Pebble: FunctionLowering::emitAssert and emitTrap in solutions/pebble/lib/CodeGen/PIRToLLVM.cpp; tests/ch11/lit/llvm-traps.pbl checks the shape and the e2e suite's trap-*.pbl the behavior.

Two trap policies: Clang's UBSan trap and Pebble's located trap

Reproduce (clang 23.1.2; pebblec from this repository with -DPEBBLE_USE_SOLUTION=all, LLVM 23.1.2):

cat > ov.c <<'EOF'
int add(int a, int b) { return a + b; }
EOF
clang-23 -O2 -fsanitize=signed-integer-overflow -fsanitize-trap=signed-integer-overflow -S -emit-llvm ov.c -o - | sed -n '/^define/,/^}/p'
cat > trap.pbl <<'EOF'
fn main() -> int {
    let big = 9223372036854775807;
    print(big &+ 1);
    print(big + 1);
    return 0;
}
EOF
pebblec trap.pbl -o trap && ./trap 2>&1; echo "exit status: $?"

Output:

define dso_local noundef i32 @add(i32 noundef %0, i32 noundef %1) local_unnamed_addr #0 {
  %3 = tail call { i32, i1 } @llvm.sadd.with.overflow.i32(i32 %0, i32 %1), !nosanitize !9
  %4 = extractvalue { i32, i1 } %3, 1, !nosanitize !9
  br i1 %4, label %5, label %6, !prof !10, !nosanitize !9

5:                                                ; preds = %2
  tail call void @llvm.ubsantrap(i8 0) #3, !nosanitize !9
  unreachable, !nosanitize !9

6:                                                ; preds = %2
  %7 = extractvalue { i32, i1 } %3, 0, !nosanitize !9
  ret i32 %7
}
-9223372036854775808
pebble: trap: arithmetic overflow at trap.pbl:4:11
exit status: 101

What to notice: Algorithm 11.6.2's trap shape — the overflow flag, a branch weighted as unlikely (!prof), a noreturn call, unreachable. Clang's trap carries only a check kind (i8 0); Pebble's carries the kind and the source location, and flushes the earlier output before the message (Theorem 11.6.3 (i)).

Unwind¶

rustc: codegen_assert_terminator in compiler/rustc_codegen_ssa/src/mir/block.rs turns MIR's Assert into a branch to a panic call, and every MIR call carries an unwind action [RUSTC-Assert]. Go: (*state).boundsCheck in src/cmd/compile/internal/ssagen/ssa.go calls panicIndex (src/runtime/panic.go), a recoverable panic [GO-Bounds].

Rust: a panicking add, and the cleanup it forces on its caller

Reproduce (rustc 1.94.1):

cat > unw.rs <<'EOF'
pub struct Guard;
impl Drop for Guard {
    fn drop(&mut self) { unsafe { cleanup() } }
}
unsafe extern "Rust" { fn cleanup(); }
#[inline(never)]
#[no_mangle]
pub fn add(a: i64, b: i64) -> i64 { a + b }
#[no_mangle]
pub fn with_guard(a: i64, b: i64) -> i64 {
    let _g = Guard;
    add(a, b)
}
EOF
rustc --crate-type=lib -C opt-level=1 -C overflow-checks=on --emit=llvm-ir unw.rs -o unw.ll
sed -n '/^define.*@with_guard/,/^}/p' unw.ll | grep -v '^;'
rustc --crate-type=lib -C opt-level=1 -C overflow-checks=on -C panic=abort --emit=llvm-ir unw.rs -o unwa.ll
sed -n '/^define.*@with_guard/,/^}/p' unwa.ll | grep -v '^;'

Output:

define noundef i64 @with_guard(i64 noundef %a, i64 noundef %b) unnamed_addr #0 personality ptr @rust_eh_personality {
start:
  %_0 = invoke noundef i64 @add(i64 noundef %a, i64 noundef %b)
          to label %bb1 unwind label %cleanup

cleanup:                                          ; preds = %start
  %0 = landingpad { ptr, i32 }
          cleanup
  invoke void @cleanup()
          to label %bb4 unwind label %terminate

bb1:                                              ; preds = %start
  tail call void @cleanup()
  ret i64 %_0

terminate:                                        ; preds = %cleanup
  %1 = landingpad { ptr, i32 }
          filter [0 x ptr] zeroinitializer
  tail call void @_ZN4core9panicking16panic_in_cleanup17h319cecbb01bfa7a4E() #7
  unreachable

bb4:                                              ; preds = %cleanup
  resume { ptr, i32 } %0
}
define noundef i64 @with_guard(i64 noundef %a, i64 noundef %b) unnamed_addr #0 {
start:
  %_0 = tail call noundef i64 @add(i64 noundef %a, i64 noundef %b) #6
  tail call void @cleanup() #4
  ret i64 %_0
}

What to notice: because add may panic and unwind, the caller calls it with invoke and a landing pad that runs Guard's destructor (cleanup) before resume continues unwinding (Theorem 11.6.3 (ii)); a panic during that cleanup aborts (panic_in_cleanup). With -C panic=abort panics become traps and the unwind edges vanish: a plain call (§6).

Error return¶

rustc: ? is desugared in lower_expr_try (compiler/rustc_ast_lowering/src/expr.rs) into a match on the Try trait's result [RUSTC-Try]; checked_add is overflowing_add plus a tag.

Rust: the same add as a trap-free error return

Reproduce (rustc 1.94.1):

cat > ov.rs <<'EOF'
#[no_mangle]
pub fn add(a: i64, b: i64) -> i64 { a + b }
#[no_mangle]
pub fn add_checked(a: i64, b: i64) -> Option<i64> { a.checked_add(b) }
EOF
rustc --crate-type=lib -C opt-level=2 -C overflow-checks=on --emit=llvm-ir ov.rs -o ov-rs.ll
sed -n '/^define.*@add_checked/,/^}/p' ov-rs.ll

Output:

define { i64, i64 } @add_checked(i64 noundef %a, i64 noundef %b) unnamed_addr #1 {
start:
  %0 = tail call { i64, i1 } @llvm.sadd.with.overflow.i64(i64 %a, i64 %b)
  %_5.1 = extractvalue { i64, i1 } %0, 1
  %not._5.1 = xor i1 %_5.1, true
  %. = zext i1 %not._5.1 to i64
  %_5.0 = extractvalue { i64, i1 } %0, 0
  %1 = insertvalue { i64, i64 } poison, i64 %., 0
  %2 = insertvalue { i64, i64 } %1, i64 %_5.0, 1
  ret { i64, i64 } %2
}

What to notice: the same intrinsic, but no branch: the tag is not overflow and the payload the wrapped sum, returned together as Option<i64> ({ tag, value }, by value in two registers — Lesson 11.5). The cost of failure handling moved to the caller's match.

Find where Swift does it. In Swift's lib/IRGen/IRGenSIL.cpp, which IRGenSILFunction method lowers the SIL instruction behind every Swift overflow and bounds trap? (Quiz swift-where-cond-fail.)

8. Comparison¶

Technique	Power / precision	Speed (asymptotic · practical)	Output / error quality	Implementation effort	Typical use
Trap	Stops at the first failure; no recovery, no cleanups	1 test + predictable branch per check · the optimizer may assume success after the check	Precise location (Pebble) or a bare check kind (UBSan trap); fixed exit status	Lowest	Pebble, Swift, hardened C (`-fsanitize-trap`), Rust `panic=abort`, kernels
Unwind	Recoverable; runs cleanups; crosses frames	0 instructions on the normal path (zero-cost EH) · failure costs a table walk per frame	Panic message + backtrace; handlers can report	High (unwind tables, landing pads, personality)	Rust `panic=unwind`, Go run-time panics, Java/C++ exceptions
Error return	Failure is an ordinary value; caller must handle it	a test per propagation step, even on success	Typed: the compiler forces handling (`Option`, `Result`)	Low in the compiler, verbose in code (`?` helps)	Rust `checked_` and `Result`, Go `(v, err)`, C `__builtin__overflow`, Swift `throws` (in a register)

Choose trap when a failure is a bug and the program cannot sensibly continue — Pebble's checks, and any system where a fast, precise stop is better than recovery. Choose unwind when callers need to recover or clean up (servers isolating a failed request, destructors that release resources). Choose error return when failure is an expected outcome the caller should handle explicitly.

9. Assessment¶

Quiz (./course quiz 11): trap-assume, swift-where-cond-fail (tag trap); unwind-invoke, unwind-abort (tag unwind); error-return-cost, error-return-branch (tag error-return).
Drill: none: the three policies differ in the code on one failure edge, which the quiz's questions trace on concrete IR; the counting part (checks per loop iteration) is exercised by Chapter 18's trip-count drill.
Flashcards: tags trap, unwind, error-return.
Exercises: E1 (checks in PIR), E5 (asserts → trap blocks).

References¶

See the chapter references.

Policy	Cost when nothing fails	Cost of one failure	Code size	Justification
Trap	per check: 1 flag test + 1 well-predicted branch	\(O(1)\) + process exit	one cold call per check (shareable per kind and location)	Algorithm 11.6.2; the Fail block is `cold` and moved out of line
Unwind	per check: the same; per call in a frame with cleanups: nothing executed, but `invoke` constrains code motion	\(O(d \cdot t)\): \(t\) = table lookup per frame (Lesson 11.7)	landing pads + unwind tables	zero-cost EH moves the work to the throw
Error return	per call level that propagates: 1 test + 1 branch, even when nothing fails	\(O(c)\) tests on the way back	one test per propagation site	the tag is checked at each `?`

Lesson 11.6 — Safety checks: trap vs unwind vs error return¶

1. Problem and motivation¶

Trap¶

Unwind¶

Error return¶

2. Definitions and algorithms¶

3. Worked example¶

Trap¶

Unwind¶

Error return¶

4. Invariants and correctness¶

Trap¶

Unwind¶

Error return¶

5. Complexity¶

6. Variants and refinements¶

Trap¶

Unwind¶

Error return¶

7. In real compilers¶

Trap¶

Unwind¶

Error return¶

8. Comparison¶

9. Assessment¶

References¶