A plain-language explainer — same story, real giant-scale numbers

A Much Bigger Machine

The Whole Story walked our tiny AI from a folder of old plays to one sentence. Here's the same six-stage story, side by side with a real reasoning model built the way you asked — a cluster of datacenter GPUs, trillions of words. Same shape. Vastly bigger numbers.

Different kind of page, so a different kind of honesty is needed. Every other page in this series used numbers we captured ourselves, from our own tiny AI, on our own GPU. We don't have a supercomputer — so every giant-scale number below is a real figure published by the company that built that model, with a source link. One correction up front: you said "H100 cluster." The real reasoning model in this family (the same one behind this whole project's earlier reasoning pages) actually trained on H800s — a different, export-restricted chip, not H100s. For a model that genuinely trained on H100s, we're also citing Meta's Llama 3.1 405B alongside it. Sources: DeepSeek-V3 Technical Report · Meta, "Introducing Llama 3.1"
1

A pile of real text — at very different scales

Our tiny AI

1.06 MBone file of Shakespeare, 40,000 lines

DeepSeek-V3 (2024)

14.8 trilliontokens of training text — roughly tens of terabytesSource: DeepSeek-V3 Technical Report

Same idea, wildly different amount of reading. If our file were a single page, a training run like DeepSeek-V3's would be a pile of pages several times taller than Mount Everest.

2

Turning the pile into a working AI

Our tiny AI

10,788,929numbers, nudged 5,000 times, on 1 gaming GPU, in about 17 minutes

DeepSeek-V3

671 billionnumbers (37 billion used per word — more on that below), nudged across 2,048 datacenter GPUsSource: DeepSeek-V3 Technical Report

That's roughly 62,000 times more numbers than our AI has, being nudged on 2,048 times more GPUs at once — and each of those GPUs is itself far more powerful than ours. DeepSeek reports the whole run took 2.788 million GPU-hours. Spread across their 2,048 GPUs, that works out to about 57 days of the entire cluster running nonstop — and cost about $5.6 million in compute, by their own estimate.

Meta's Llama 3.1 405B is the one that really did use H100s: 16,000+ H100 GPUs, 15+ trillion tokens, about 30.84 million GPU-hours total — roughly 80 days across that whole cluster.Source: Meta, "Introducing Llama 3.1"

3

You type something in

Same idea as our AI: real models like this turn your words into numbers using a fixed lookup table, built the same way ours was — by scanning huge amounts of text and cataloging the pieces that show up. Theirs just has a lot more entries than our 65, and the pieces are usually chunks of words, not single letters.

4

The part with no dictionary — now much bigger, and stranger

Our tiny AI

384 numbersper position, per layer — 6 layers deep

DeepSeek-V3

7,168 numbersper position, per layer — 61 layers deepSource: DeepSeek-V3 Technical Report

Same problem as before — nobody has a dictionary for any of these numbers either — just about 18 times as many numbers, spread across 10 times as many layers.

There's a real extra wrinkle at this scale. DeepSeek-V3 doesn't use all 671 billion numbers for every word — it has 256 "expert" sections, and a routing step picks just 8 of them to actually help with any given word, before combining their answers. Imagine a huge office building with 256 specialists, and for every single word that comes in, a receptionist quietly picks 8 of them to handle it — a different 8, potentially, for the very next word. We don't have a directory of what any specialist specializes in. We can only watch which 8 get picked.Source: DeepSeek-V3 Technical Report
5

Squeezed back into an answer

Same shape as our tiny AI's last step: all that mystery math eventually gets squeezed back down into one number for every entry in the dictionary — a real, readable list of odds for what word or word-piece comes next. That list has a key again, for the same reason ours did: it was built to line up with the dictionary from step three, on purpose.

6

The final result

One piece of text out, the same weighted-lottery way ours works — just with a genuinely staggering number of possible next pieces to choose from, and a genuinely staggering number of unlabeled numbers deciding the odds. Then, exactly like ours, it happens again for the next piece. And again.

Does bigger change the answer?

No — and that's the actual point of this page. Every real number above came from a company with essentially unlimited resources to solve this if solving it were just a matter of scale or budget. They didn't write a dictionary for the 7,168 numbers either. It's not on their roadmap because nobody knows how, not because nobody can afford it.

Bigger machine, same missing dictionary — just 18 times more numbers per layer, spread across 10 times more layers, with a routing step on top that we can watch but not explain. Scale made the guessing machine much better at guessing. It didn't make the guessing machine legible.

The rest of the series

The Whole StoryThe same six stages, fully captured from our own tiny AI, start to finish. Getting Less WrongHow our tiny AI's numbers went from random to useful. The Unlabeled DialsThe real 384 numbers with no dictionary, and why. The Next LetterHow those numbers become real, readable odds again.