Skip to content

Quiz format

Each chapter's theory test is authored in plaintext and published hashed:

File Who edits Contents
solutions/quizzes/chNN.yaml chapter author questions, answers, explanations (plaintext spoilers)
chapters/NN-slug/quiz.yaml generated by ./course quiz build chNN; never edit prompts and choices in clear text; answers as salted SHA-256 hashes; explanations base64-encoded

CI runs ./course quiz verify --all, which fails when a built quiz is out of date or when a source's own answers don't grade as 100 %. The hashes keep answers from being visible when you open the file. They are not secure: a determined learner can brute-force small answer spaces. That's fine, because the goal is to prevent accidental spoilers.

The engine is tools/course/quiz.py, and the normalization rules live in tools/course/lib/answers.py. If you change either, update this document.

Commands

./course quiz build 15              # compile solutions/quizzes/ch15.yaml -> chapters/15-.../quiz.yaml
./course quiz build --all
./course quiz verify --all          # CI: built files in sync + self-grading = 100 %
./course quiz 15                    # interactive: shows each explanation after you answer
./course quiz template 15 -o answers/ch15.yaml   # answers-file skeleton
./course quiz grade 15 [--answers answers/ch15.yaml] [--explain wrong|all|none]
./course check 15                   # quiz grade + implementation tests

# authoring preview of any file (not tied to a chapter):
./course quiz build --source my-draft.yaml -o /tmp/quiz.yaml
./course quiz grade --quiz /tmp/quiz.yaml --answers my-answers.yaml

Prompts and choices are Markdown with $…$ math: the website's quiz page renders them with KaTeX, and ./course validate checks that every formula renders (NOTATION.md §10).

A complete sample source with one question of every type is at tools/course/tests/fixtures/sample_quiz_source.yaml. Its answer files are sample_answers_good.yaml and sample_answers_partial.yaml.

Source schema (solutions/quizzes/chNN.yaml)

chapter: 15                    # required; must match the file name (ch15)
title: "Dominance & loops — theory test"     # optional
pass_threshold: 0.8            # optional, default 0.8 (the course's "done" bar)
questions:
  - id: idom-of-join           # required; kebab-case [a-z0-9-], unique, STABLE (hashes are salted with it)
    type: mapping              # required; single | multi | number | text | set | mapping | sequence
    prompt: |                  # required; Markdown; include the whole instance (CFG, grammar…)
      CFG: A→B, A→C, B→D, C→D. Give idom(n) for B, C, D.
    answer: {B: A, C: A, D: A} # required; shape depends on type (below)
    explanation: |             # required; shown after answering, TEACHES the why
      D joins B and C; its idom is their nearest common dominator, A.
    points: 1                  # optional, default 1 (positive number)
    tags: [chk, dominators]    # optional; use the technique tags from the lessons/flashcards
    lesson: lessons/01-dominators.md   # optional; where the answer is taught
    hint: "node: idom"         # optional; overrides the default answer-format hint
    partial_credit: true       # optional; see per type
    case_sensitive: false      # optional; text/set/sequence/mapping

Unknown keys are errors, so typos get caught.

Question types

type answer: in the source Learner writes Graded as Partial credit
single a letter: b b (also B, (b), 2) exact no
multi letter list: [a, c] a, c / ac / [a, c] exact set opt-in partial_credit: true → (right − wrong) / |answer|
number 42 or 2.72, plus optional tolerance: 0.01 42, 2.718, 1/3 exact, or within tolerance (below) no
text "DF", plus optional accept: [variants…] free text normalized equality with any variant no
set [B, C] {B, C} / B C / B, C order-free, duplicates ignored opt-in, as for multi
mapping {B: A, C: A} or with set values {B: [D], C: []} B: A lines, or a YAML mapping per key default on: fraction of keys right
sequence [A, C, B, D] A C B D / A -> C -> B -> D exact order opt-in: correct prefix / length

Details

  • single / multi:
  • Give choices: as a list of ≥ 2 strings. They're labelled a, b, c… in order, so reordering them changes the answer letters.
  • Quote a choice that contains : or starts with [, {, *, &, !, |, >, and quote yes/no/on/off/true/false: unquoted, YAML parses - foo: bar as a mapping and - no as a boolean. quiz build/verify reject any choice, answer, accept: entry or mapping key/value that isn't a string (numbers are fine), naming the question id.
  • Write distractors from real misconceptions: post-dominance vs dominance, FIRST vs FOLLOW, preorder vs RPO.
  • number:
  • Without tolerance, the value must be exact. 2 and 2.0 both count.
  • With tolerance: t, the stored answer is round(answer / t) (answer rounded to a multiple of t), and a response is accepted if it lies within t of that rounded value.
  • Choose answers that are multiples of t, e.g. answer: 2.72, tolerance: 0.01.
  • text:
  • Keep answers to a key term (DF, dominance frontier, SemiNCAInfo), and list every reasonable spelling in accept:.
  • For identifiers (dyn_cast, PreservedAnalyses), set case_sensitive: true.
  • Don't use text questions for sentences. Use single instead.
  • set: elements are tokens. Separate them with commas when an element contains spaces (dominance frontier, idom).
  • mapping:
  • The keys asked are the keys of answer:. They're stored in clear text, so the learner sees which keys to answer.
  • Values are single tokens (value_type: token) or sets (value_type: set). This is inferred as set if any value is a list; force it with value_type. Write empty sets as [].
  • The learner sees "wrong or missing: D" feedback.
  • sequence: the feedback reports the length of the correct prefix.

Normalization rules (what counts as "the same answer")

The engine normalizes both the author's answer, at build time, and the learner's answer, at grading time:

  1. Text (text, and every token):
  2. Unicode NFKC.
  3. En and em dashes and minus signs become -, and curly quotes become straight quotes.
  4. Case-folded, unless case_sensitive: true.
  5. Runs of whitespace collapse to one space, and leading and trailing whitespace are trimmed.
  6. One trailing . is dropped.
  7. Tokens (elements of set, sequence and mapping keys and values) also:
  8. have surrounding quotes or backticks stripped, and
  9. map ε, eps, epsilon, '', λ → ε.
  10. List input (set, sequence, multi, set-valued mapping values):
  11. Surrounding {}, [] or () are removed.
  12. If the text contains , or ;, it's split on those; otherwise on whitespace. Sequences also split on ->, →, =>.
  13. {}, ∅, -, none, empty and the empty string mean the empty set.
  14. YAML lists are accepted directly.
  15. Mappings:
  16. Entries are separated by newlines or ;.
  17. Each entry is key: value, key -> value, key → value or key = value.
  18. YAML mappings are accepted directly.
  19. Keys are normalized as tokens. Two source keys that are equal after normalization are an error.
  20. Numbers:
  21. Thousands separators , and _ are ignored, and a/b fractions are accepted.
  22. Non-finite values are rejected.
  23. Choice letters: case-insensitive, parentheses ignored, and a 1-based number is accepted as its letter.

When the learner's answer can't be parsed, grading returns a "could not read your answer" message, and in interactive mode the learner is asked again. Scoring never raises an error.

Built format (chapters/NN-slug/quiz.yaml, format: pebble-quiz/1)

# GENERATED FILE — do not edit.  Source: solutions/quizzes/ch15.yaml
format: pebble-quiz/1
chapter: 15
title: ...
pass_threshold: 0.8
source: solutions/quizzes/ch15.yaml
questions:
- id: idom-of-join
  type: mapping
  points: 1
  prompt: ...
  value_type: token
  partial_credit: true
  keys: [B, C, D]
  answer: {b: <hash>, c: <hash>, d: <hash>}
  salt: 9e57f6d5dcabb84c
  explanation: <base64>
  • salt = sha256("pebble-quiz|chNN|<id>")[:16], so the build is deterministic: the same source gives a byte-identical file and a clean git diff.
  • hash(s) = sha256(salt + "\x1f" + s)[:32], applied to the canonical form:
  • single: the letter
  • multi and set: sorted items joined with ,, plus a per-item hash list for feedback
  • number: round(x / t) or the exact decimal
  • text: every accepted variant
  • sequence: items joined with spaces, plus the per-position hashes i:item
  • mapping: key=value per key, with set values canonicalized as sorted a,b
  • Explanations are base64 so a glance doesn't spoil them.

Answers files (for quiz grade and course check)

This is a YAML mapping from question id to answer. Generate a skeleton with ./course quiz template NN. The default location is answers/chNN.yaml: your own work, which you can commit in your fork.

dom-definition: b
dom-algorithms: [a, c]
chk-passes: 2
df-meaning: dominance frontier
back-edge-heads: "{B}"
idoms: {B: A, C: A, D: A}
df-sets: "B: {D}; C: {D}"
rpo: A -> C -> B -> D

grade prints ✓/✗ per question with feedback and, by default, the explanations for wrong answers. It records your score in .course/progress.json (turn this off with --no-record in CI) and exits non-zero below the pass threshold.

Authoring guidance

  • 15 to max(30, 2·T + 3) questions per chapter, where T is the number of techniques in the chapter. At least 2 per technique (DEPTH_CONTRACT item 9), and ≥ 40 % computation or tracing: mapping, set, sequence or number over a concrete instance in the prompt.
  • ≥ 3 "find it in LLVM" questions that follow the lesson's source-reading tasks, e.g. "Which class in GenericDomTreeConstruction.h implements the tree build? (text, case-sensitive)".
  • Explanations teach. Give the reasoning and the common wrong turn, not "the answer is b".
  • Ids are forever. Renaming an id changes its salt and hashes, and learners' saved answer files stop matching it. Only rename before the chapter is published.
  • Self-contained prompts. Put the whole CFG, grammar or IR in the prompt. Use the same notation as the lessons (STYLE.md §5) and the successor-order convention for CFGs.
  • Check every computed answer with the course oracles (tools/course/lib/cfg.py, grammar.py), not by hand.