One prompt, start to finish — tiny-shakespeare

The Journey

Every number below is real, captured from one actual run: the prompt "ROMEO:", through the same 10.8M-parameter model this whole project traced piece by piece — tokenizer, real ggml ops, real GPU-adjacent computation, real sampling. Here it is stitched into one continuous story, ending with what this can and can't explain.

Prompt → tokens

Tokenization

The prompt is six characters. CharTokenizer maps each one to its integer id via a fixed lookup table built from the training corpus — no ambiguity, no model involved yet.

R
30
O
27
M
25
E
17
O
27
:
10

Input to the model: [30, 27, 25, 17, 27, 10]

Tokens → logits

The forward pass

These six ids go through the real transformer — not a diagram of it, the literal ggml op graph captured by running llama.cpp's own eval-callback tool against this exact prompt. 201 real operations fire; here is the spine of it, real shapes included:

1GET_ROWStoken_embd.weight[384,65] → embd[384,6]
2GET_ROWSposition_embd.weight[384,256] → pos_embd[384,6]
3ADDembd + pos_embd → inpL[384,6]
199MULnorm × output_norm.weight
200ADD+ output_norm.bias → result_norm[384,1]
201MUL_MAToutput.weight[384,65] → result_output[65,1]

Op 201 is the end of the line: 65 real numbers, one per character in the vocabulary — the model's raw, unnormalized preferences for what comes next.

Logits → a token

Softmax, then a roll

Those 65 logits pass through softmax, turning them into a real probability distribution. Here are the top 5 for what follows "ROMEO:":

\n
99.996%
Y
0.001%
'
0.001%
S
0.001%
A weighted random draw (seed 1337) landed on '\n' — unsurprising here, since it held 99.996% of the probability mass. That won't always be true.

A note on the number: 99.996% is after scaling the logits by the sampling temperature (0.8) used for this generation run, which sharpens the distribution. The raw, unscaled softmax of the same logits gives 99.9497% — that's the figure The Next Letter and Look at the Maths use. Same forward pass, same logits, one extra division.

Repeat, 39 more times

The output accumulates

Each new character becomes part of the context for the next forward pass — same pipeline, run again. Most steps are as one-sided as step 0. Four were not: the sampler drew a character that wasn't even in the top 5 shown.

ROMEO:
Yet be not desired to repent me: and yo

Marked characters: real long-shot draws — Y at 3.1%, b at 3.6%, d at 2.7%, r at 3.2%. Nothing scripted; same mechanism as step 0, different outcome, because the draw fell in a different slice.

What this explains

Mechanistic why vs. semantic why

The journey above is a complete, gapless causal chain for one specific fact: why '\n' followed "ROMEO:" in this run. That question has a real, fully-cited answer. A different question — why the weights encode a colon-then-newline pattern at all, what the 384-dimensional hidden state at each layer represents — does not.

Answered here

Every number from prompt to output is accounted for: tokenization, all 201 real ops, the exact logits, the exact softmax probability, the exact sampled outcome. No step is inferred or assumed.

Not answered here

What the embedding, attention, or hidden-state vectors mean — why the trained weights produce this preference and not another. That's mechanistic interpretability: a live research problem, distinct from tracing the computation.

The pieces this journey is built from

Live Computation Graph →Module-level PyTorch trace of the same forward pass, via forward hooks. tiny-shakespeare Graph →All 201 real ggml ops for this exact prompt, in full. Sampling Tree →All 40 generation steps as an interactive branching tree, every candidate and its real probability.