Skip to content

English · Español

Phase 03 — Linear Algebra from First Principles

Requires: 02 — Numerical Representation Teaches: tensors · matmul · einsum · svd · rank · norms Jump to any chapter from the phase reference index.

Chapter map

Pre-written per A12; re-anchored to A13 (English verb grammar). Theory + lab statements are stable drafts; solutions land just-in-time at phase open.

🇪🇸 Los tensores no son "arrays con más dimensiones". Son mapas lineales coordinados, y la matmul es la composición de esos mapas. Esta fase te entrena a predecir shapes sin ejecutar código, derivar SVD, y conectar la eficiencia de matmul con la jerarquía de memoria de Fase 1. Los ejemplos viven en el universo del modelo: vectores one-hot que indexan formas verbales y matrices que actúan como tablas de búsqueda sobre el vocabulario §A13.

Goal

Internalize linear algebra deeply enough that Borja can:

  • Read any einsum expression and predict the resulting shape (and number of FLOPs) without running it — including the batched-embedding-lookup expression that anchors Phase 13.
  • Derive ||Ax|| ≤ ||A||·||x|| from SVD on a blank page.
  • Measure a 50× speedup between naive matmul and np.matmul on his i5-8250U and explain it as a roofline fact (Phase 1) compounded by vectorisation absence (Phase 6 preview).
  • Reconstruct a verb-form conjugation-count matrix with k singular values and quantify the reconstruction error as a function of k.
  • See, mechanically, how M @ one_hot(i) is a lookup M[:, i] — the operation every embedding layer does.

Read order

  1. theory/00-motivation.md — why a linear-algebra phase before any autograd.
  2. theory/01-tensors-and-shapes.md — scalar / vector / matrix / tensor; shape arithmetic; einsum as a unifying grammar; one-hot encodings of verb forms.
  3. theory/02-matmul-and-shapes.md — matmul as composition, dot/outer/Hadamard distinctions, batched matmul, embedding lookup as matmul. The most important page in Phase 3.
  4. theory/03-svd-and-rank.md — SVD as rotate-scale-rotate, rank, low-rank approximation, Eckart-Young; applied to a conjugation-count matrix.
  5. theory/04-norms-and-conditioning.md — vector / matrix norms, operator norm, condition number.
  6. lab/00-shapes-by-hand.md — predict shapes from einsum strings (no code at first), including embedding-lookup and batched-matmul patterns.
  7. lab/01-matmul-perf.md — race four matmul implementations on your machine.
  8. lab/02-svd-compression.md — compress a synthetic conjugation-count matrix with rank-k SVD.
  9. lab/03-norms-by-experiment.md — verify ||Ax|| ≤ ||A||·||x|| empirically and find the equality case.

solutions/ is empty at pre-write — populated at phase open.

Definition of Done

See PHASE_03_PLAN.md §6. Briefly:

  • Four experiment directories with manifests + artefacts committed.
  • Matmul performance chart shows ≥ 50× gap between naive Python and np.matmul at N=1024.
  • SVD reconstruction grid + error curve on the conjugation-count matrix committed.
  • Borja can predict einsum shapes on a blank page and derive ||Ax|| ≤ ||A||·||x|| from SVD.

What this phase intentionally does NOT cover

  • Autograd / gradients. Phase 4 (calculus) and Phase 7 (scalar autograd).
  • A src/minigrad/linalg.py module. Phase 3 stays in experiments/; the linalg primitives are promoted in Phase ⅞ when an autograd consumer is built.
  • Probability / random matrices / random projections. Phase 5 + Phase 10 (RMSNorm/init).
  • Eigendecomposition of non-symmetric matrices. Mentioned for vocabulary; the work is SVD.
  • Numerical algorithms for SVD itself. We use np.linalg.svd; deriving Jacobi / QR-based SVD is out of curriculum scope.
  • Sparse / structured matrices. Phase 27 (efficient attention) touches block-diagonal; nothing here.
  • GPU matmul (BLAS / cuBLAS / Triton GEMM). Phase 23+.

Phase 3's scope is understanding tensor shapes, matmul mechanics, and SVD intuition deeply enough to make every later phase's linear-algebra step boring. Nothing more.

Further reading

Optional — enrichment, not required to pass the phase.