Skip to content

Chapter review checklist (for the reviewer subagent)

Every chapter is written by an author subagent and then audited by a separate reviewer subagent that did not write it. The reviewer follows this procedure in order, re-derives everything it can independently, fixes what it can fix safely, and produces a findings report. A chapter is flipped to status: available only after a review with no open blocker or major findings.

The reviewer's stance: assume nothing is right until you have checked it. A claim with no way to check it is itself a finding.

0. Setup

uv sync                                   # the pinned Python environment
./course doctor                           # toolchain OK (LLVM 23.1.2, FileCheck, …)
git log --oneline -- chapters/NN-slug labs/chNN-* tests/chNN solutions/ | head   # what changed

Read, in this order: the chapter's entry in PROPOSAL §6, DEPTH_CONTRACT.md, NOTATION.md, REFERENCES.md, LABS.md, then the chapter's README.md.

1. Automated gates (all must pass)

./course validate --strict                # structure, rigor boxes, citations, references in sync, labs, drills, quizzes
./course quiz verify --all
./course refs verify
uv run python -m unittest discover tools/course/tests
./course test NN                          # skeleton: compiles; every failure is a clean TODO(chNN)
./course test NN --solution               # reference solution: everything passes
./course site build                       # strict website build

Record every failure as a finding. Do not continue to "looks good" on a red gate.

2. Width against the syllabus

  • Every technique family and technique named in the chapter's PROPOSAL §6 entry appears in the technique map (W2) and has a lesson section. List any missing technique as a blocker.
  • The comparison table (W5) has one row per technique and matches each lesson's §8 row word for word.
  • "Who uses what" (W4) names ≥ 4 systems (where they exist), each backed by a source pointer in a lesson.
  • Outcomes (W7) are checkable and each is exercised by a quiz question, drill or test.

3. Mathematics and proofs, line by line

For every lesson:

  • The chapter README's ## Notation section defines every symbol the lessons use; notation follows NOTATION.md (no ad-hoc symbols, no custom LaTeX macros, multi-letter names upright).
  • Every Definition is well formed: all symbols defined earlier, no circularity, quantifiers explicit, edge cases (empty sets, unreachable nodes, ε) covered.
  • Every Theorem/Lemma/Proposition is true as stated. Try to break it: look for a counterexample on the running example, on a degenerate input (single node, self-loop, irreducible graph, ε-only grammar), and on the lesson's own pathological family.
  • Every Proof is checked step by step: each step follows from definitions, earlier results or cited facts; case splits are exhaustive; induction has a base case and a correct hypothesis; termination arguments name a well-founded measure. A proof sketch names where the full proof is ([KEY, §n]) — open that source and confirm it proves this statement.
  • Every Algorithm box states input, output, preconditions, postconditions and invariants; the invariant is proved (initialization, maintenance, termination ⇒ postcondition); the pseudo-code defines every helper; it matches the lesson's prose and the course implementation (or the difference is stated).
  • Every complexity claim has variables defined and a justification (a counting argument, a recurrence, or a citation); the pathological family is explicitly constructed and its bound is derived, not asserted.

Report each defect with the exact location (lessons/03-…md, box title) and a corrected statement.

4. Worked examples, recomputed

  • Recompute every trace table and final answer independently: with the course oracles (tools/course/lib/), the drills (./course drill <name> --seed N --solution), or LLVM itself (e.g. opt -passes='print<domtree>'). Never by eye.
  • The example exercises every branch of the algorithm (DEPTH_CONTRACT item 3); traces include the initial state and the final no-change pass.
  • Mermaid diagrams, tables and prose use identical node/symbol names and successor order.
  • Each "Try it" drill command runs and produces a problem of the stated kind.

5. Real-world boxes, re-run

For every !!! real-world box:

  • Run the Reproduce commands exactly as written with the stated tool versions (clang/opt 23.1.2 from the course toolchain; other tools as stated). Paste your output next to the box's Output; any difference other than documented abridgment (…, "abridged") is a finding.
  • The box shows the concept the surrounding theory defines (e.g. the printed dominator tree really is the tree of Definition N.k.m on that input), and the "What to notice" text is correct.
  • Every technique in the lesson's §7 has at least one such box (validate checks presence; you check substance).
  • Commands are copy-pasteable (sh fence, no $ prompt), deterministic, and don't need network access unless stated.

6. Citations and source pointers

  • Open every reference entry: the DOI/URL resolves, authors/title/venue/year are correct, book sections point at the right pages for the claim.
  • Every annotation says why and when to read the work, and is accurate about its content.
  • Each technique's origin paper is present and cited where the technique is introduced; no blog post is cited as an origin.
  • Every citation in the text supports the sentence it is attached to (spot-check at least one per lesson against the source; all of them for claims about performance numbers).
  • Every source pointer (path + symbol) exists at the pinned tag (llvmorg-23.1.2, gcc-15, …): open it on GitHub at that tag or in a local checkout. Missing or renamed symbols are major.
  • Bibliography meets the minimums (≥ 12 entries; paper, book with sections, source, docs) and core entries are cited.

7. Assessment items

  • Take the quiz cold (./course quiz NN). For each question: the chapter teaches the answer; the stored answer is correct (check solutions/quizzes/chNN.yaml, recomputing computational answers with the oracles); the explanation is right and explains why, including why the distractors are wrong.
  • Question mix: 15 to max(30, 2·T + 3) questions (T = techniques), ≥ 40 % computation/tracing, ≥ 3 "find it in LLVM" questions, ≥ 2 per technique.
  • Flashcards: ≥ 30, every technique tag present, backs are correct and self-contained.
  • Drills: registered, self-test passes, worked solutions match lesson formats.

8. Exercises, labs and tests

  • Each lab has a SPEC.md meeting LABS.md §3; requirements are testable and the tests check exactly what the spec says (no hidden requirements, no untested requirements).
  • Over-scaffolding: the learner directory contains only the contract stub(s) and a README; no internal helpers, class skeletons or per-step stubs for the learning objective. Flag every provided function that the learner should have designed (major).
  • Skeleton build: compiles; every chNN test fails with TODO(chNN) (never a crash or wrong answer).
  • Solution build: all tests pass. Then mutation check: introduce 2–3 plausible bugs into the solution (off-by-one in RPO numbering, wrong intersect direction, missing ε case) and confirm the tests catch each one. A surviving mutant is a major finding against the tests.
  • Hints go from "where to start" to "design sketch" without giving code.

9. Website

  • ./course serve, then open every page of the chapter: math renders (no raw $/\( left, no red KaTeX errors), every Mermaid diagram draws, admonition boxes render with the right type, tables fit, code highlighting is right.
  • Literal dollar signs in prose are code (`$`) or escaped (\$); none is swallowed into a formula.
  • Citations link to the right entries on the chapter References page; the global Bibliography lists the chapter's entries.
  • Flashcards and Quiz pages render; the quiz page shows no answers or explanations.
  • No link points to a missing page (the strict build catches most; click the cross-chapter links).

10. Style

  • STYLE.md voice and conventions (no hype words, "you", present tense, American spelling, LLVM 23).
  • No spoilers: lessons contain no exercise solution code; quiz answers are not restated next to their topic.

Findings report

Write the report as the reviewer's final message (and, for the record, as a section appended to the chapter's pull request description — not as a file in the repository). Format:

# Review: Chapter NN · <title>   (reviewer: <agent id>, date, commit <sha>)

Gates: validate --strict ✓/✗ · quiz verify ✓/✗ · refs verify ✓/✗ · unit tests ✓/✗ · test NN ✓/✗ · test NN --solution ✓/✗ · site build ✓/✗

| # | Severity | Where | Finding | Evidence | Fix (done / author must do) |
|---|---|---|---|---|---|
| 1 | blocker | lessons/03-…md, Theorem 15.3.2 | statement false for a self-loop at the entry | counterexample: … | done: added hypothesis "r has no predecessors" |

Rubric (DEPTH_CONTRACT §5): one row per technique, items 1–9 scored 0/1/2.

Verdict: READY / NOT READY (list the open blockers and majors)

Severities:

Severity Meaning Examples
blocker wrong content a learner would learn, or a failing gate false theorem, wrong trace, wrong quiz answer, missing syllabus technique, red CI
major a depth-contract item below minimum, or unverifiable content proof sketch without a source, real-world output that doesn't reproduce, missing source pointer, over-scaffolded lab, surviving mutant
minor correct but unclear or inconsistent notation drift, missing edge case in prose, unclear hint
nit style wording, formatting

Fix policy: the reviewer fixes blockers, majors and minors itself when the fix is unambiguous and local (a wrong number, a missing hypothesis, a dead link, a mis-keyed citation), re-runs the affected gates, and marks them "done". Anything needing the author's design decision (restructuring a lesson, redesigning a lab contract, adding a missing technique) goes to the author with a precise description. The chapter is READY only when no blocker or major is open and every gate is green.