Things I built to understand something

Each row is one token deciding where to look; the row sums to 1. Entropy measures how spread out that decision is — 2.58 bits is perfectly uniform over six tokens, and 0 bits is all the weight on one. With scaling on, entropy barely moves as d grows, which is the whole point. Turn it off and push d to 256: entropy collapses and one token takes nearly everything, which in a real model means vanishing gradients through the softmax. The causal mask is the other half of an autoregressive model — token i may not look at anything after it, so the matrix goes triangular. The query and key vectors here are arbitrary; the mechanism around them is not.

Orange is a hit, grey is a miss. The picture is B alone, which is why its rate differs from the total across all three matrices: under i-j-k every single access to B misses, and the grid goes solid grey. Counts are for the full 32×32 run; the animation replays the first passes on a loop, since the pattern repeats. i-j-k walks B down a column: every element sits on a different cache line, so each one drags in eight values and uses exactly one before the line is evicted. i-k-j walks the same matrix along a row and gets all eight. Blocking wins differently — it shrinks the working set until A, B and C all fit at once, which is what buys you reuse across the whole tile rather than just within a line. Nothing about the arithmetic changed.

No locks keeps every thread busy and silently drops writes when two land on the same bucket — the red flashes. One mutex never loses a write and never lets more than one thread work: it is the unsynchronised version plus overhead, which is exactly why it measured ~10% slower than the baseline when I measured it, and 30–40% slower once I pushed it to 4,000 threads. Per bucket only makes threads wait when they actually collide — about 2× on the dual-core machine I wrote it on, and ~8× on an eight-core box. The animation shows the mechanism; those numbers are measured, not modelled.

Read the matrix as nine equations. The top six rows are the identity, which is why the first six shards are just the data itself — that is what "systematic" means, and it is why reading undamaged data costs nothing. The bottom three are Vandermonde rows, so any six of the nine are linearly independent and therefore invertible. Decoding is just that: drop the rows you lost, invert the 6×6 that remains, multiply it by the shards you still have. Pressing reconstruct really runs that — I checked it against all 220 combinations of up to three losses before putting it here. And the reason storage systems bother: tolerating three failures by replication means four whole copies and 300% overhead. This does it for 50%.

The trick is that a uniformly hashed id is a coin-flip sequence: seeing a hash with $k$ leading zeros is evidence you have seen about $2^k$ distinct things. One register would be far too noisy, so the first $p$ bits pick a bucket and the harmonic mean across $m=2^p$ buckets pulls the variance down — the estimate is $E={\alpha }_m{m}^2/{\sum }_j{2}^{-M\left[j\right]}$, and the standard error it leaves you is $1.04/\sqrt{m}$, chosen at design time by choosing how much memory to spend. Drag the slider and watch the band tighten as the register bank grows.

LRU is a stack algorithm: the pages held with n frames are always a subset of those held with n+1, so more memory can only ever help. FIFO has no such property — evicting by arrival order rather than by use means a bigger cache can hold on to exactly the wrong pages for longer. OPT is the offline optimum, which needs to know the future and so exists only as a lower bound to measure real policies against.