The equations behind the plain-language series
Every other page in this project deliberately hid the formulas. This one doesn't. Same tiny AI, same real captured numbers — just the actual equations this time, worked through with real values so you can check the arithmetic yourself.
The simplest step — no formula really needed, just a table lookup.
Think of a phonebook with 65 entries, one per letter. To "look up" a letter, you just flip to its page and copy down what's written there — 384 numbers. No calculating, no thinking. Just fetching a row that was already written down during training.
E is a table with 65 rows (one per letter) and 384 columns. Letter "R" is id 30. Its row, real values, first six of 384:
This one formula, repeated with different numbers, is almost everything a transformer does — every MUL_MAT in this project's traces is this.
Take two lists of numbers of the same length. Pair them up in order, multiply each pair, then add up every result into one single number. Add one more fixed number on top. That's the whole operation — done with different lists, over and over, it's most of what happens inside the AI.
Take the model's real final 384-number output for "ROMEO:", and the real trained row of weights that scores the letter "newline." Multiply matching positions together, add them all up, add one more real number (the bias). That's the entire computation behind one prediction score:
The one formula in this project's ggml traces labeled FLASH_ATTN_EXT.
Picture everyone at a table quietly asking "who here matters to me right now?" Everyone else answers, some strongly, some weakly. Each person then blends a little of what everyone else said into their own notes — more from whoever answered strongly, almost nothing from whoever answered weakly.
In the formula's own terms: every letter asks a question (Q), every letter (including itself) offers an answer to match against (K), and how well they match decides how much of that letter's content (V) gets mixed in. Dividing by √dk (here, √64 = 8) just keeps the numbers from growing too large before the next step — softmax, which is section 4.
The exact formula behind every bar chart in The Next Letter.
Turn a pile of raw scores into percentages that add up to 100%, while keeping the biggest score clearly on top. A slightly bigger raw score turns into a noticeably bigger share of the total — that's the whole trick behind turning "scores" into something you can call odds.
Take the real 65 logits the model produced for what follows "ROMEO:" (including the 14.0107 from section 2). Raise e to each one, add all 65 up, then divide the one you want by that total:
The number behind every bar in Getting Less Wrong.
How surprised should the AI be by the correct answer? If it was confident and right, barely surprised — low score. If it was confident and wrong, extremely surprised — high score. Training's entire goal is to make this "surprise score" smaller, on average, across everything it reads.
Just: take the probability the model assigned to the letter that actually came next, and negative-log it. Confident and right gives a small number; confident and wrong gives a huge one. A model that hasn't learned anything spreads its guesses evenly across all 65 letters, so pure chance predicts a starting score of exactly −log(1/65) = ln(65):
What actually happens 5,000 times in Getting Less Wrong.
For every single number inside the AI, ask: "if I nudge this number up a tiny bit, does the surprise score from section 5 get better or worse?" Then move the number a tiny step in whichever direction helps. Do that for all 10.8 million numbers, thousands of times in a row. There's no bigger idea behind training than this.
θ is any one of the model's 10,788,929 numbers. ∇L(θ) is the slope — which direction, and how strongly, changing that one number would make the wrongness score go up or down. η (our real value: 0.0003) controls how big a step to take. Subtract, and the number gets very slightly less wrong. Repeat for all 10.8 million numbers, 5,000 times.
One honest simplification: this is the classic version of the idea. Our actual training code uses AdamW, a more sophisticated variant that also tracks how each number has been moving recently and adjusts its own step size — same core idea, extra bookkeeping. The formula above is the idea it's built on, not a line-for-line description of the code.
No — and now you've seen exactly why not, in the clearest possible terms. Every equation on this page is completely known, completely public, and just checked twice against our own model's real output. There is nothing hidden in how the numbers are computed.
What's missing was never a formula. It's an interpretation — a mapping from "here is what 384 real numbers equal" to "here is what they mean." These five equations tell you exactly how to compute every one of this project's unlabeled dials. Not one of them tells you what any single dial is for.