Skip to content

English · Español

Lab 02 — Run the three engineered breaks; diagnose each from the dashboard

Goal: practice diagnosis. Diagnose three engineered failures using the dashboard alone, before peeking at the source.

Estimated time: 2-3 hours (each break runs in ~30 min, plus diagnosis time).

Prereq: Lab 01 (dashboard renderer) committed; experiments/19-healthy/dashboard.html exists as the reference pattern.

What you produce

A directory experiments/19-break-it/ containing three sub-experiments:

experiments/19-break-it/
├── 01/
│   ├── train.py
│   ├── config.yaml
│   ├── inspector.log.jsonl
│   ├── dashboard.html
│   └── manifest.json
├── 02/
│   └── (same as 01)
├── 03/
│   └── (same as 01)
└── borja-diagnoses.md      ← YOUR diagnoses, BEFORE peeking

And, after diagnosis:

  • experiments/19-break-it/diagnosis-score.md — your self-scored 3/3, ⅔, or ⅓ with notes.

TODOs

Block A — write the three break scripts (Claude provides skeletons; Borja confirms)

The three scripts 01/train.py, 02/train.py, 03/train.py each copy the Phase-18 healthy training driver and corrupt it in one known way. Claude provides them as deliberate stubs at phase open (so Borja doesn't peek by reading the source).

Crucially: the directory names 01, 02, 03 are SHUFFLED. Borja does not know which directory contains which break. The order is determined by the just break-it-shuffled recipe.

The three breaks (in random order — Borja sees them as 01/02/03 but won't know the mapping):

  • Bad init. Replace xavier_init(W) with xavier_init(W) * 100.
  • No warmup. Set warmup_steps = 0.
  • Broken mask. Replace tril_mask(L) with np.ones((L, L), dtype=bool).

Block B — run each break

For each of 01, 02, 03:

just run-break 01
just run-break 02
just run-break 03

(The Justfile recipe shells into the appropriate directory and runs train.py with the corrupted config.)

Each run produces inspector.log.jsonl and renders dashboard.html. Open each dashboard in a browser. Compare to the healthy reference.

Block C — write your diagnoses BEFORE peeking

In experiments/19-break-it/borja-diagnoses.md, for each of 01, 02, 03:

## Break 01

**Observed dashboard symptoms:**
- Panel 1 (loss): [your observation]
- Panel 2 (LR): [your observation]
- Panel 3 (grad norm): [your observation]
- Panel 4 (activations): [your observation]
- Panel 5 (spectral): [your observation]
- Panel 6 (dead): [your observation]
- Panel 7 (regular vs irregular): [your observation]

**Most striking abnormality:** [which panel was off first / most]

**Hypothesis:** [your best guess at which engineered failure was applied]

**Rationale:** [why those symptoms imply that failure]

Pay particular attention to Panel 7. The broken-mask failure shows up there with a distinctive signature: the regular and irregular curves both collapse to near-zero in lockstep, with no gap. That's not learning; that's the model copying the future token.

Three sections, one per break. Write all three BEFORE consulting solutions/03-three-failures-ref.md (which reveals the mapping).

Block D — reveal and score

Open solutions/03-three-failures-ref.md (written at phase open by Claude after Borja commits borja-diagnoses.md — this is enforced by the phase-gatekeeper hook).

For each of 01, 02, 03, compare your hypothesis to the actual engineered failure: - ✅ match - 🟡 close (right family, wrong specific cause) - ❌ miss

Write experiments/19-break-it/diagnosis-score.md:

- Break 01: [actual: bad-init | warmup | mask] — your hypothesis: [...] — verdict: [✅/🟡/❌]
- Break 02: ...
- Break 03: ...

Overall: X/3.

Reflection: [what you learned about reading the dashboard. Which panel did you under-weight? Which heuristic worked?]

The diagnoses score and reflection appear in PHASE_19_REPORT.md.

Constraints

  • No source inspection until diagnoses are committed. This is the whole point of the lab. The repo's .pre-commit-config.yaml includes a hook that warns if you stage solutions/03-three-failures-ref.md before borja-diagnoses.md.
  • Each break runs from a fresh clone of the healthy config. Don't accidentally carry over state.
  • Diagnoses are forensic. Cite the panel and the specific abnormality, not "looks bad."

Stop conditions

Done when:

  1. All three dashboard.html files exist and render.
  2. borja-diagnoses.md is committed with three forensic diagnoses, BEFORE consulting solutions.
  3. diagnosis-score.md is committed AFTER consulting solutions.
  4. Score is recorded in PHASE_19_REPORT.md.

Pitfalls

  • Peeking. The hardest constraint. The reward of not peeking is the muscle memory for future debugging; if you peek, you learned much less.
  • Misreading the dashboard. It's easy to miss panel 2 (LR) since it's "boring" — the warmup break specifically hides there. Look at all six panels for every break.
  • Conflating panels. Bad init and broken mask both have "Panel 1 loss looks weird" symptoms, but the direction is opposite (NaN explosion vs suspiciously low). Be precise.
  • Running breaks in parallel. Don't. Each one needs your full attention. Borja's hardware can't run them in parallel anyway (single-process NumPy).

When to consult solutions/

solutions/02-break-it-ref.md and solutions/03-three-failures-ref.md (theory companion) are written by Claude after Borja commits borja-diagnoses.md. This is the order:

  1. Borja commits the three break runs and borja-diagnoses.md.
  2. The phase-gatekeeper subagent verifies the commit.
  3. Claude writes the solution files.
  4. Borja reads them and writes diagnosis-score.md.

This is the strictest lab in the curriculum because diagnosis-without-peeking is a skill that can't be retrofitted.

Next lab: lab/03-overfit-on-purpose.md.