Skip to content

English · Español

Lab 04 — Write a manifest-diff helper

Pre-req: read ../theory/04-manifest-anatomy.md. Goal: implement a 30-line CLI that takes two manifest paths, prints only the fields that changed, and ranks them by drift-causing likelihood. CPU-only, < 5 minutes wall-clock.

§1 Setup

  1. Create scripts/manifest_diff.py (no existing implementation — this is greenfield).
  2. Create two synthetic manifests under experiments/diff-demo/:
  3. monday.json (use the example from theory/04-manifest-anatomy.md §1).
  4. tuesday.json (same, but with numpy 2.0.2).
  5. Confirm uv run python scripts/manifest_diff.py experiments/diff-demo/monday.json experiments/diff-demo/tuesday.json runs.

§2 Your task

Implement manifest_diff.py as a CLI:

usage: manifest_diff.py [-h] A B

Prints fields that differ between manifests A and B, grouped by drift category.

Categories (in print order, highest-impact first):
  CODE         git_sha, git_dirty
  VERSIONS     versions.*
  HARDWARE     hardware.*
  ENV          env.*
  TIME         wall_seconds, started_at, finished_at
  OTHER        everything else

Output format (one example):

$ uv run python scripts/manifest_diff.py monday.json tuesday.json
[VERSIONS] numpy: 2.0.1 -> 2.0.2
(no differences in CODE, HARDWARE, ENV, TIME, OTHER)

Constraints: - Must use only stdlib (json, argparse, pathlib, sys). - Must handle the case where a key exists in one manifest but not the other (<missing> on the absent side). - Must mypy --strict. Must ruff check clean. - Must exit non-zero if the manifests differ in CODE or VERSIONS (so it's usable as a CI gate).

§3 Tests

Add tests/test_manifest_diff.py: - test_identical_manifests_have_zero_exit_code — both files equal → exit 0. - test_version_drift_exits_nonzero — numpy version differs → exit code != 0. - test_missing_key_is_reported — manifest A lacks env.OMP_NUM_THREADS; manifest B has it → diff prints <missing> -> 8.

The tests should run in < 1 second total.

§4 Stop conditions

  • uv run python scripts/manifest_diff.py experiments/diff-demo/monday.json experiments/diff-demo/tuesday.json prints exactly one VERSIONS line and exits with code 1.
  • pytest tests/test_manifest_diff.py passes (3/3).
  • mypy --strict scripts/manifest_diff.py passes.
  • ruff check scripts/manifest_diff.py passes.
  • You wrote a journal entry in learners/borja/phase-00/notes/manifest-diff.md covering: which drift class would you most want a louder alarm on (CI breaks build), and which is more of a "FYI"?
  • Commit: lab: phase-00 add manifest-diff CLI for drift triage.

§5 Hints

  1. Flatten nested dicts with a recursive flatten(prefix, obj) -> dict[str, str] helper. Output keys like versions.numpy.
  2. The "categories" are just regex prefixes on the flattened key. A dict of { "CODE": ("git_sha", "git_dirty"), ... } is enough.
  3. Use argparse with two positional arguments; no need for fancy options. Add --quiet to suppress the "(no differences in ...)" line only if you have time.
  4. The "exit non-zero on CODE/VERSIONS drift" makes the script usable as manifest_diff.py prev.json next.json && echo OK || echo DRIFT in a CI step.

§6 What you'll have learned

  • Manifests are not write-only — they are queried.
  • A 30-line diff helper does 80% of the drift-triage work a research-engineer needs.
  • "Severity by category" is a useful design idea (you'll use it again in Phase 19's training-dynamics logger).