Skip to content

English · Español

Break 00 — Shuffle target labels with the prompts

🇪🇸 Rompemos la asociación entre cada ejemplo y su etiqueta: para cada par (prompt, label), mantenemos prompt igual y barajamos label. La pérdida debería bajar mucho más despacio (o no bajar). Es un test fundamental — si tu pérdida es similar con etiquetas aleatorias y reales, tu modelo no está aprendiendo nada de la entrada.

Anchors: LYNX_CORTEX.md §4 / PHASE 12; theory §04 corpus size and memorization; .claude/commands/break.md.

The break

In src/dataset/grammar_loader.py:

def load_train(seed: int = 0) -> list[tuple[str, int]]:
    pairs = _load_canonical_pairs()
    rng = np.random.default_rng(seed)
    # BUG: shuffle the labels independently of the prompts.
    labels = [p[1] for p in pairs]
    rng.shuffle(labels)
    return [(p[0], labels[i]) for i, p in enumerate(pairs)]

The corpus shape is preserved (same 240 examples, same 5 tenses, same balance). Only the (prompt, label) mapping is randomized. The model sees correct-looking data; the loss landscape is what changed.

Predict, then run

Expected: random-label baseline

With K = 5 tenses, random label prediction achieves loss ≈ log(5) ≈ 1.609. Phase 5 §02 derives this — it's the entropy of the uniform distribution.

A model fed shuffled labels can:

  1. Memorize the shuffled labels if it has capacity. With 50K params and 240 examples × 4 bits each = 960 bits required, the model has plenty of capacity. So memorize-and-overfit is the likely outcome.
  2. But validation labels are also shuffled from a different RNG draw, so val loss stays at log(5).

Predictions

  • Train loss with shuffled labels: starts at ~1.6, drops to near 0 by epoch 100 (memorized).
  • Val loss with shuffled labels: stays at ~1.6 throughout (no real signal to learn).
  • Train loss with correct labels: drops to ~0.05 by epoch 30 (real learning, generalization).
  • Val loss with correct labels: drops to ~0.10 by epoch 30 (real generalization).

The train vs val gap is the diagnostic. With shuffled labels, gap is enormous; with correct labels, the curves track.

Write predictions in learners/borja/phase-12/notes/breaks.md before running.

Observe

just exp 12-train --tag broken-shuffled-labels

Diagnostics:

  1. Overlay train-loss curves: correct-labels vs shuffled-labels.
  2. Overlay val-loss curves: shuffled stays flat at log(5).
  3. Look at the train/val gap. With shuffled labels: huge.

Symptom Borja will see

  • Train loss drops, validation loss stays at the random-baseline entropy log(5) ≈ 1.61.
  • Train accuracy approaches 100%, val accuracy stays at 20%.
  • Phase 19's "train/val gap" instrumentation lights up at maximum.

Hidden cause (one sentence)

load_train shuffles labels independently of prompts, so every (prompt, label) pair is a meaningless association the model is forced to memorize.

Hint cascade

  1. Compare the train and val loss curves. What's special about the val loss's asymptote? (It's log(K) = log(5) ≈ 1.61. What does that mean?)
  2. Print 5 examples from the dataset. Does the label match the prompt? (For the buggy version: no — they're independent.)
  3. Look at load_train. Where does the label come from after the call to rng.shuffle?

Fix diff

def load_train(seed: int = 0) -> list[tuple[str, int]]:
    return _load_canonical_pairs()                     # restored: no shuffling at all

(Or pass seed only to the batch order, not to the pairs.)

Why this teaches the concept

Zhang et al. (2017) ("Understanding deep learning requires rethinking generalization") used exactly this experiment — random labels — to show that modern neural networks can memorize arbitrary labels but only generalize when the labels carry signal. The train/val gap test is the most diagnostic single experiment for distinguishing memorization from generalization. Phase 12's lab gets you fluent with it before Phase 18 (training loop) needs you to interpret it on real runs.

Next: Phase 13's /break on tied-embedding shape mismatch.