Eight small games take you from a jar of marbles to Shannon's noisy channel coding theorem. Each level you clear flips one bit in the meter above. Collect all 8 bits, one full byte of intuition.
Built from Pinkard & Waller, "A visual introduction to information theory" and Chris Olah's "Visual Information Theory".
Two marbles are drawn (with replacement) from the urn above. There are 16 possible two-color sequences. Learning a fact about the draw lets you cross some of them off. Rarer facts rule out more, so they carry more information.
Every halving of the remaining probability mass is worth exactly 1 bit.
Entropy is the probability-weighted average information per draw, and it is the shortest possible average encoding of the source. For this urn, H(X) = ½·1 + ¼·2 + ⅛·3 + ⅛·3 = 1.75 bits.
A short codeword is expensive: choosing "0" burns half of all possible codewords, since nothing else may start with 0 (the prefix rule). Spend your codeword budget in proportion to how often each color appears.
| color | p | codeword (0s and 1s) | length |
|---|---|---|---|
| 1/2 | 2 | ||
| 1/4 | 2 | ||
| 1/8 | 2 | ||
| 1/8 | 2 |
A typical 16-draw sequence encoded with your code:
Bob mostly says "dog"; Alice mostly says "cat". Each has a code optimized for their own word frequencies. Cross-entropy Hₓ(p) is the average message length when a speaker with distribution p borrows the code built for q. The extra cost over the speaker's own entropy is the KL divergence.
Take sequences of N draws. Plot each possible sequence by its information per draw, −log₂p / N. In black: how many sequences sit at each value. In magenta: where the probability mass actually lands. As N grows, nearly all the mass piles onto "typical sequences" whose information per draw ≈ H(X).
Now the urn holds colored shapes. You see only a black-and-white photo, so you learn the shape Y but not the color X. Mutual information I(X;Y) measures how much observing shape shrinks your uncertainty about color. Each bit of mutual information halves the remaining possibilities.
A magical urn repeats the previous color with probability s and switches otherwise (a Markov chain). Every single draw still looks uniform on its own, yet the sequence becomes predictable. The entropy rate, your uncertainty about the next draw given the last one, drops below 2 bits. That gap is redundancy.
A channel maps inputs to outputs with noise: here a Z-channel where 0 always arrives as 0, but 1 flips to 0 thirty percent of the time. How much information gets through, I(X;Y), depends on the input distribution you choose. The best achievable value is the channel capacity C.
Two forces compete: use the noisy input more to fill the output space (raise H(Y)), or use it less to avoid noise (lower H(Y|X)). Capacity balances them.
Send 4 marbles (8 source bits) through a binary channel that flips each bit with probability 0.1. Raw bits get corrupted. Repetition coding votes its way to safety but slashes the rate. Block codes like Hamming(7,4) protect several bits together and keep the rate high. Shannon proved that with long enough blocks, any rate below capacity can be made error-free.