English · Español

Lab — Spike injection (reproduce a controlled loss spike and apply the stability checklist)¶

🇪🇸 Esta práctica te deja causar una spike a propósito en un punto conocido del entrenamiento, y luego camina el árbol de stability-check.md para confirmar que tu diagnóstico coincide con la spike inyectada. Es el ejercicio de "saber leer el dashboard" en su forma más limpia.

Setup¶

Prerequisite: Phase 18's mini-GPT trains cleanly with seed 42 on the §A13 corpus. You have experiments/18-baseline/ with a healthy dashboard.html.

This lab adds an injection knob to src/minitrain/loop.py that lets you deterministically perturb training at a chosen step. The injections are post-hoc fixable — the model recovers if the perturbation stops.

Injection types¶

You will implement and test three injections, in order of subtlety.

Injection A: scaled gradient (loud)¶

At a chosen step \(K\), multiply every gradient by 50× before the optimizer's clip-and-step. Cancel the multiplier after step \(K\).

def maybe_inject(step, grads, cfg):
    if cfg.injection == "scale" and step == cfg.injection_step:
        return {k: 50.0 * g for k, g in grads.items()}
    return grads

Expected: visible spike on the loss panel at \(K\), grad-norm pre-clip ~50× baseline, clipping engages (post-clip ~1.0), loss recovers in 20-40 steps because the moments only saw one bad update.

Injection B: bad-batch (subtle)¶

At step \(K\), replace the batch with a synthetic "bad" batch: 8 copies of the same rare-token sentence (e.g., 8 copies of She has written the letter.). Cancel after step \(K\).

def maybe_inject(step, batch, cfg):
    if cfg.injection == "bad_batch" and step == cfg.injection_step:
        return [BAD_SENTENCE] * batch.size
    return batch

Expected: smaller spike on loss, larger spike on grad-norm (the rare-token concentration is exactly the §A13 long-tail pattern), longer recovery (~50 steps).

Injection C: optimizer-state corruption (sneaky)¶

At step \(K\), multiply v_t by 100× for one parameter (say, the LM-head bias). Cancel after \(K\), but the corruption persists in moment state.

def maybe_inject(step, optimizer_state, cfg):
    if cfg.injection == "v_corrupt" and step == cfg.injection_step:
        optimizer_state["v"]["lm_head.bias"] *= 100.0

Expected: no spike on loss at step \(K\). Instead, a slow elevation over the next 30 steps as the LM-head bias underdates (because \(\hat v\) is too large, so the bias's effective step is too small) until the EMA of \(v\) decays the corruption away. This is the sneakiest injection — it doesn't look like a spike, it looks like a temporary plateau.

Procedure¶

Run baseline: seed=42 injection=none just phase-19-train.
Run Injection A at step 500: seed=42 injection=scale injection_step=500 just phase-19-train.
Run Injection B at step 500: seed=42 injection=bad_batch injection_step=500 just phase-19-train.
Run Injection C at step 500: seed=42 injection=v_corrupt injection_step=500 just phase-19-train.
For each: open the dashboard. Walk stability-check.md from §1 to the first matching leaf. Write your diagnosis in learners/<name>/phase-19/spike-diagnoses.md:
injection ID (which run)
which stability-check.md node you stopped at
your diagnosis (one sentence)
your remediation (one sentence)
Reveal the injections: solutions in solutions/04-spike-injection-ref.md (don't peek before step 5).
Score: number of correct diagnoses / 3.

Stop condition¶

You're done when all three injections are diagnosed correctly and your learners/<name>/phase-19/notes/lessons.md has three new lines describing the three failure modes.

Definition of Done¶

src/minitrain/loop.py has a clean maybe_inject hook (default no-op).
Three runs in experiments/19-injection-{A,B,C}/ with manifests.
spike-diagnoses.md filled in before peeking at solutions.
All three diagnoses match the engineered injection.

Why this lab is different from `02-break-it.md`¶

02-break-it.md provides three pre-broken configurations and asks you to diagnose them once. This lab gives you the injection mechanism and asks you to cause the failure at a chosen step. The difference:

02-break-it.md tests: "can you read the dashboard?"
04-spike-injection.md tests: "can you predict the dashboard's response to a known perturbation?"

The second skill is harder and more useful. When debugging real training runs in Phase 26 or 28, you won't know which break was applied — but you'll need to hypothesize a perturbation that would produce the observed pattern. This lab is the inverse direction of that skill, practiced cleanly.

Cross-refs¶

theory/04-loss-spike-postmortem-template.md — write a post-mortem for each injection.
stability-check.md — the tree you walk for diagnosis.
lab/02-break-it.md — the prerequisite lab (different shape, same domain).