Architecture of `pebblec`¶

This page explains how the compiler you build in this course is put together, where each piece lives, and which boundaries are stable contracts. The most important contract is the one between front ends and everything after them. It lets a front end written in another language, such as Rust, plug in without touching any C++.

1. The pipeline¶

                    pebble::Frontend (in-process)       ExternalFrontend (any language)
  foo.pbl ──►  Lexer ─► Parser ─► Sema ─► LowerToPIR      <exe> --emit=pir foo.xyz
               Ch 1     Ch 4      Ch 5-7   Ch 11                  │ PIR text on stdout,
                 │                                                │ JSON diagnostics on stderr
                 └───────────────────────┬────────────────────────┘
                                         ▼
               PIR  (docs/pir/pir-spec.md) ── parse ─► verify ─► pir-run (reference interpreter)
                                         │
                                         ▼  PIRToLLVM (Ch 11, shared by every front end)
               LLVM IR  ◄───────────────────────────── pebblec --from-llvm x.ll
                                         │
                                         ▼  verify ─► -O0 / -O1 (your pipeline) / -O2, --pass-plugin
               LLVM IR ─► TargetMachine ─► object file ─► cc + libpebble_runtime.a ─► executable

Front end: source text in, a verified pir::Module or a list of diagnostics out.
PIR is the stable, language-neutral, versioned boundary. Everything downstream of it is shared.
Back end: PIR → LLVM IR → optimization → machine code → a linked executable.

2. Components¶

Library target	Directory	Contents	Status
`pebble_support`	`pebble/lib/Support`	`SourceManager`, `SourceLocation`, `Diagnostic*`, `Error`	provided
`pebble_ast`	`pebble/lib/AST`	AST node classes, `ASTContext` (arena), `ASTVisitor`, `ASTDumper`	provided
`pebble_lex`	`pebble/lib/Lex`	tokens (provided), `Lexer`	Ch 1 exercise
`pebble_parse`	`pebble/lib/Parse`	`parseModule`, `parseExpression`	Ch 4 exercise
`pebble_sema`	`pebble/lib/Sema`	`resolveNames`, `checkFlow` (Ch 5), `typeCheck` (Ch 6–7)	exercises
`pebble_lower`	`pebble/lib/Lower`	`lowerToPIR`	Ch 11 exercise (switch `lower`)
`pebble_pir`	`pebble/lib/PIR`	PIR data structures, `Builder`, printer, parser, verifier, interpreter	provided
`pebble_codegen`	`pebble/lib/CodeGen`	`lowerPIRToLLVM`	Ch 11 exercise (switch `codegen`)
`pebble_frontend`	`pebble/lib/Frontend`	`Frontend`, `FrontendRegistry`, `ExternalFrontend`, `PebbleFrontend`	provided
`pebble_driver`	`pebble/lib/Driver`	the `pebblec` pipeline, `-O` pipelines, object emission, linking	provided (`-O1` hook: Ch 24)
`pebble_runtime`	`pebble/runtime`	C runtime	provided

Tools: pebblec (the driver), pir-run (the PIR interpreter) and pir-opt (PIR parse/verify/print).

The headers in pebble/include/pebble/ are the interfaces. The tests are written against them, so an exercise is complete when its implementation satisfies the header's documented contract. Reference implementations live under solutions/ at the same relative path and are switched in with -DPEBBLE_USE_SOLUTION=<switch> (docs/dev/FOUNDATION.md §2–3).

3. The `Frontend` interface (in-process)¶

// pebble/include/pebble/Frontend/Frontend.h
struct FrontendInvocation {
  SourceManager &SM;        // diagnostics' locations refer to this
  FileID MainFile;          // already loaded
  FrontendOptions Options;  // ExtraArgs from --frontend-arg=
};
struct FrontendOutput { pir::Module Module; DiagnosticList Warnings; };
using FrontendResult = std::expected<FrontendOutput, DiagnosticList>;

class Frontend {
public:
  virtual std::string_view getName() const = 0;
  virtual FrontendResult compile(const FrontendInvocation &Inv) = 0;
  virtual std::expected<std::string, DiagnosticList>
  dumpStage(FrontendStage Stage, const FrontendInvocation &Inv);  // tokens / AST
};

compile returns a module that has passed pir::verifyModule, or every diagnostic, including at least one error.
FrontendRegistry::instance().add(name, description, factory) makes a front end available to pebblec --frontend=<name>.
The built-in C++ front end is registered as "pebble" by registerBuiltinFrontends(). Registration is explicit, not done by static constructors, because a static library linker drops unreferenced constructors.

resolveFrontend(spec) interprets --frontend=<spec> in this order:

a registered name (pebble);
a path to an executable (the spec contains a /), which runs as an ExternalFrontend;
pebble-frontend-<spec> found on PATH, in the style of git subcommands. Installing a Rust front end as pebble-frontend-rust makes pebblec --frontend=rust foo.rs work with no configuration.

pebblec --list-frontends prints the registered ones.

4. The front-end protocol (out-of-process)¶

Any program in any language can be a Pebble front end. It only has to print PIR text. ExternalFrontend (pebble/lib/Frontend/ExternalFrontend.cpp) runs it like this:

<exe> --emit=<stage> --pir-version=<major>.<minor> [<frontend-arg>...] <input-path>

Argument	Meaning
`--emit=pir`	compile the input and print a PIR module (required)
`--emit=tokens`, `--emit=ast`	print a token or AST dump in any format (optional; exit with 2 if unsupported)
`--pir-version=1.0`	the newest PIR version the consumer reads; emit this version or an older one
frontend args	passed through verbatim from `pebblec --frontend-arg=...`
input path	as given to `pebblec`

stdin is empty.
stdout carries the complete output (a PIR module for --emit=pir). Nothing else may be written there.
stderr carries diagnostics, one JSON object per line (§5). A line that is not a JSON object is shown to the user as warning W0004 with the front end's name, so stray debug output is visible but harmless.
The exit status is one of:

Status	Meaning	pebblec does
0	success	parses and verifies stdout. A parse or verifier failure is E0003 "front end '…' produced invalid PIR", so a buggy front end can never cause a silent miscompile
1	compile errors	shows the diagnostics. If none of them is an error, it adds E0002
2	unsupported `--emit` stage	reports "does not support --emit=…" (E0002)
other	crash	reports E0002 "exit status N (crash?)"

A front end must not report errors and exit with 0 at the same time (E0002).

5. Diagnostics protocol (JSON lines)¶

The same format serves front ends' stderr and pebblec --diagnostics=json:

{"severity":"error","code":"E0301","message":"use of undeclared name 'm'",
 "file":"fib.pbl","line":3,"column":12,"end_line":3,"end_column":13,
 "notes":[{"message":"'n' declared here","file":"fib.pbl","line":1,"column":8}]}

(It is shown wrapped here. On the wire, each object is a single line.)

Key	Required	Meaning
`severity`	yes	`"error"`, `"warning"` or `"note"` (a note attaches to the previous diagnostic)
`message`	yes	text without a trailing period or the `error:` prefix
`code`	no	a stable identifier such as `E0301`; external front ends may use their own prefixes
`file`	no	the path as passed to the front end; defaults to the input file
`line`, `column`	no	1-based; columns count bytes
`end_line`, `end_column`	no	the position one past the last character (exclusive)
`notes`	no	an array of objects with `message` and optional location keys

pebblec maps positions back into its SourceManager, loading the file if needed, and renders diagnostics exactly like its own, with the source line and a ^~~~ marker.

6. Writing a front end in Rust¶

Chapter 24's optional extension is a second front end for Pebble, written in Rust. The protocol means it needs no LLVM bindings and no C++. It is an ordinary Cargo binary that prints text. The sketch below compiles a trivial language (a list of integer literals to print) and shows every protocol obligation. Your real Pebble front end replaces the "compile" part with a lexer, a parser, a type checker and PIR generation that follow docs/language/pebble-spec.md.

// src/main.rs — `cargo build --release`, then install the binary on PATH as
// `pebble-frontend-rust`, and run: pebblec --frontend=rust program.txt
use std::fmt::Write as _;
use std::process::ExitCode;

/// One JSON-lines diagnostic on stderr (docs/architecture.md §5).
fn error(file: &str, line: usize, col: usize, len: usize, msg: &str) {
    // A real front end would use serde_json; this keeps the sketch dependency-free.
    let esc = |s: &str| s.replace('\\', "\\\\").replace('"', "\\\"");
    eprintln!(
        r#"{{"severity":"error","code":"R0001","message":"{}","file":"{}","line":{},"column":{},"end_line":{},"end_column":{}}}"#,
        esc(msg), esc(file), line, col, line, col + len
    );
}

fn main() -> ExitCode {
    let args: Vec<String> = std::env::args().skip(1).collect();
    let emit = args.iter().find_map(|a| a.strip_prefix("--emit=")).unwrap_or("pir");
    let Some(path) = args.last() else { return ExitCode::from(2) };
    if emit != "pir" {
        return ExitCode::from(2); // tokens/ast dumps are optional
    }
    let source = match std::fs::read_to_string(path) {
        Ok(s) => s,
        Err(e) => { error(path, 1, 1, 0, &format!("cannot read file: {e}")); return ExitCode::from(1); }
    };

    // "Compile": every line holds an integer to print.
    let mut body = String::new();
    let mut failed = false;
    for (i, line) in source.lines().enumerate() {
        let text = line.trim();
        if text.is_empty() { continue; }
        let col = line.find(text).unwrap() + 1;
        match text.parse::<i64>() {
            Ok(v) => {
                // Every statement carries its source location: @line:col.
                writeln!(body, "  call @pebble_print_int({v}) @{}:{col}", i + 1).unwrap();
                writeln!(body, "  call @pebble_print_newline() @{}:{col}", i + 1).unwrap();
            }
            Err(_) => { error(path, i + 1, col, text.len(), "expected an integer"); failed = true; }
        }
    }
    if failed {
        return ExitCode::from(1);
    }

    // The module: header, runtime externs, @main. pebblec verifies it.
    print!(
        "pir 1.0\nsource \"{path}\"\n\n\
         extern fn @pebble_print_int(i64)\nextern fn @pebble_print_newline()\n\n\
         fn @main() -> i64 {{\nbb0:\n{body}  return 0\n}}\n"
    );
    ExitCode::SUCCESS
}

Advice for a real Rust front end:

Generate PIR through a small builder that mirrors pebble/include/pebble/PIR/Builder.h: numbered locals, blocks, the current location, and helpers for checked operations. Print it in canonical form (docs/pir/pir-spec.md §14) so that golden tests can compare your output with the C++ front end's. pir-opt canonicalizes anything that is merely valid.
Debug with the oracle. pebblec --frontend=rust --emit=pir x.pbl -o x.pir && pir-run x.pir runs your front end's output under the reference interpreter. The interpreter detects undefined behavior, such as an unchecked out-of-range index or a read of an uninitialized local, that native code would silently exhibit.
The conformance suite (tests/conformance/) is the definition of done: the same end-to-end programs, verifier checks and diagnostic tests that the C++ front end passes.
Emitting LLVM IR instead (for example with inkwell) is possible via pebblec --from-llvm (docs/runtime-abi.md §8), but you give up PIR's verifier, the interpreter oracle and the PIR goldens. Emitting PIR is the recommended path.

7. Driver details¶

pebblec [options] <input>; the implementation is pebble/lib/Driver/Driver.cpp.

Option	Meaning
`--emit=tokens\\|ast\\|pir\\|llvm\\|obj\\|exe`	the stage to stop at (default `exe`)
`--frontend=<name\\|path>`, `--frontend-arg=<a>`	choose and configure the front end (§3)
`--from-pir`, `--from-llvm`	the input is PIR or LLVM IR (also guessed from `.pir`, `.ll` and `.bc`)
`-O0`, `-O1`, `-O2`	LLVM's O0 pipeline; the course pipeline (`getCoursePipeline()` in `pebble/lib/Driver/CoursePipeline.cpp`, the `PEBBLE_COURSE_PIPELINE_STEP`s your chapters register, `default<O1>` while there are none; see `docs/dev/FOUNDATION.md` §7); LLVM's O2
`--passes=<pipeline>`	a textual new-pass-manager pipeline, replacing `-O`
`--pass-plugin=<lib>`	load a pass plugin, e.g. the course's `PebblePasses` (repeatable)
`-o <file>`	the output file (`-` is stdout). By default, text goes to stdout, objects to `<stem>.o` and executables to `<stem>`
`--diagnostics=text\\|json`	the diagnostic format
`--cc=`, `--runtime=`	the linker driver and the runtime archive (§1 of docs/runtime-abi.md)
`--ssa=allocas\\|braun`	how the code generator builds SSA: one `alloca` per local, promoted by `mem2reg` (default), or ★ on the fly (Chapter 11 E6; a code generator without it reports `unsupported:`)

Exit status: 0 on success, 1 on any error, and 99 if an unimplemented exercise is reached (TODO(chNN)).

8. Testing strategy¶

pir-run is the oracle. A correct front end produces PIR that behaves under pir-run as the language specification requires. A correct code generator produces executables that behave exactly like pir-run on the same PIR: same stdout, same stderr, same exit status. tests/pir/unit/CodeGenTest.cpp checks this for every sample program at -O0 and -O2.
The conformance suite (tests/conformance/) holds the contracts every front end and the shared back end must meet: PIR programs with expected behavior, verifier and parser negative tests, and the external front-end protocol. The end-to-end Pebble programs of Chapter 11 (tests/ch11/e2e/, run through pir-run and native executables at -O0/-O1/-O2) hold every front end to the same behavior.
Unit tests for the provided infrastructure are in tests/pir/unit/ (label infra). Exercise tests are in tests/chNN/.

Architecture of pebblec¶