Skip to content

English · Español

Lab 00 — Decode floats by hand and by code

Goal: internalize the fp32 bit layout by writing a decoder that produces the same output as your pencil work.

Estimated time: 60–90 minutes.

What you produce

A directory experiments/02-float-anatomy/ containing:

  • decode.py — a Python script implementing decode_fp32(x: float) -> dict and encode_fp32(s, e, m) -> float from raw bit fields.
  • worked.md — your hand-decoded answers for the four target values (see TODOs).
  • bf16_round_trip.py — bf16 emulation via bit-cast.
  • fp16_round_trip.py — fp16 round-trip via numpy.float16.
  • manifest.json — seed, versions, hardware, config.
  • README.md — interpretation.

TODOs

Block A — decode by hand

For each of these four values, decode by hand into (sign, biased_exponent, mantissa_bits, value) and write the result in worked.md. Do not use code yet.

  1. The fp32 bit pattern 0x3F800000.
  2. The fp32 bit pattern 0x40490FDB (you may recognize it).
  3. The fp32 representation of 1/600 (the §A13 vocabulary's uniform probability). To find it, you may use the inverse procedure: write 1/600 in binary, normalize, read off sign / exponent / mantissa. Show your work.
  4. The fp32 representation of 92.0 (an adversarial-magnitude logit from a tense classifier — see theory 02).

🇪🇸 Truco: 1/600 ≈ 0.001\overline{6}. Normaliza al rango [1, 2): 1/600 = 1.70\overline{6} \times 2^{-10} (verifícalo: 1.7066... / 1024 ≈ 0.00166...). La parte después del primer 1. es tu mantisa.

Block B — implement decode_fp32 and encode_fp32

Two functions. Pure Python (struct module is allowed; no NumPy).

decode_fp32(x: float) -> dict returns:

{
  "raw_bits": "0x...",
  "sign": 0 or 1,
  "biased_exponent": int in [0, 255],
  "unbiased_exponent": int (biased_exponent - 127),
  "mantissa_bits": "0b... (23-bit string)",
  "is_normal": bool,
  "is_denormal": bool,
  "is_zero": bool,
  "is_inf": bool,
  "is_nan": bool,
  "reconstructed_value": float
}

encode_fp32(sign, biased_exponent, mantissa) -> float is the inverse.

Constraints:

  • Use struct.pack('<f', x) to get the fp32 bits, struct.unpack('<I', ...) to read as uint32.
  • Mask and shift to extract fields. No numpy.frombuffer shortcuts — the point is to write the bit manipulation explicitly.
  • reconstructed_value should match x to bit identity for any finite input. Test with: ±0, denormals (1e-40), 1.0, 1/600, 92.0, inf, -inf, nan.

Block C — verify hand work matches code

For each of the four values in Block A, run your decode_fp32 and compare to your hand work. They must match exactly. Any mismatch → fix the hand work; if both agree but you're not sure, write down the discrepancy and reason about it in worked.md.

Block D — bf16 emulation

bf16_round_trip.py: implement

def to_bf16(x: float) -> int:
    # returns 16-bit integer (the bf16 bit pattern)
    ...

def from_bf16(bits: int) -> float:
    # returns fp32 value
    ...

def round_trip_bf16(x: float) -> float:
    return from_bf16(to_bf16(x))

Strategy: fp32-to-bf16 is "take the high 16 bits of the fp32 bit pattern" (with appropriate rounding — round-half-to-even, but for this lab simple truncation is fine and you can note the bias). bf16-to-fp32 is "left-shift the 16 bits into a 32-bit slot, zero the bottom 16".

Round-trip these values:

  • 1.0
  • 1/600
  • 92.0
  • 0.1
  • The probability 1e-20

Print: (original_fp32, bf16_bits, recovered_fp32, relative_error).

Block E — fp16 round-trip

fp16_round_trip.py: use np.float16 for the conversion (it's IEEE-754 fp16, bit-correct). Round-trip the same five values. Print the same table.

Observe: 92.0 survives bf16 (range up to ~3.4e38) but overflows to +inf in fp16 (range up to ~6.5e4)? Actually 92.0 is within fp16 range; the overflow threshold for fp16 is for exp(92.0), not for 92.0 itself. Note this distinction in your README.md.

Block F — the killer demonstration

Show, in README.md, with output from your scripts:

0.1 + 0.2 == 0.3            ?  →  False
hex(struct.pack('<d', 0.1+0.2))  →  ...
hex(struct.pack('<d', 0.3))      →  ...

The bit patterns must differ in the lowest bit. This is your evidence — not just an assertion — that fp arithmetic is approximate.

Constraints

  • Hand work first, code second. The point of this lab is not to write a decoder; it's to think like a decoder. If you skip the hand work, you're cheating yourself.
  • No NumPy for decode_fp32/encode_fp32. struct only. Forces you to see the bit fields explicitly.
  • Verify every output. A decode that prints 1.0 is suspicious — verify the bit fields.

Stop conditions

You're done when:

  1. worked.md contains hand decodes for all four Block A targets.
  2. decode.py passes a self-test: decode_fp32(x)['reconstructed_value'] == x for x in [±0, 1.0, 1/600, 92.0, 0.1, 1e-40, np.inf, np.nan] (NaN is is_nan == True, since nan != nan).
  3. bf16_round_trip.py outputs the table.
  4. fp16_round_trip.py outputs the table.
  5. README.md contains the 0.1 + 0.2 evidence and a one-paragraph interpretation of what bf16 vs fp16 lost on each value.
  6. manifest.json is in place.

Pitfalls

  • Endianness. Use '<f' (little-endian) consistently. Mixing with '>f' will silently corrupt your bit-string output.
  • Denormal handling in your decoder. When biased_exponent == 0, the leading 1. becomes 0. and the exponent is 1 - bias, not 0 - bias. Test with 1e-40.
  • NaN. Many NaN bit patterns exist (any with e == 0xFF and m != 0). is_nan shouldn't compare to a specific pattern; check the field condition.
  • bf16 rounding. Simple truncation biases toward zero. The "correct" bf16 rounding is round-half-to-even, but for this lab truncation is acceptable; just note the bias in README.md.

When to consult solutions/

After committing all five files. The solution lives in solutions/00-bit-anatomy-ref.md — written at phase open.

Hint of last resort

If 1/600 is fighting you for two hours, here's the answer to check against:

sign     = 0
exponent = 117  (biased)  →  unbiased -10
mantissa = 0b10110100111010000001110 = 5927950  (with rounding noise)
reconstructed ≈ 0.0016666667... (off by ~3e-11 from the exact 1/600)

Use this only to verify, not to copy. The point is the procedure, not the answer.

Next lab: lab/01-softmax-stability.md.