One prompt, start to finish — deepseek-r1:8b

The Larger Journey

The same story as The Journey, told for an 8-billion-parameter model we don't own the weights of. Prompt: "The capital of France is". Every number below is real — captured across two separate traces of the same prompt — and one gap is named honestly where the data genuinely runs out.

Prompt → tokens

Tokenization

llama.cpp's own eval-callback, run natively via WSL against Ollama's downloaded weights, printed the real input token ids before doing anything else:

78567223159625374
Honest gap: unlike tiny-shakespeare's tokenizer (65 entries, built by us, a trivial lookup), this is Qwen2's trained BPE vocabulary — 151,936 entries. We don't have that vocab file loaded locally, so we can't decode these 5 ids back to their exact subword strings here. We do know what this kind of token looks like from elsewhere in this project: a space becomes Ġ, a newline becomes Ċ, and whole words like " problem" can be a single token — boundaries a trained vocabulary chooses, not ones a person would draw by hand.

Tokens → logits

The forward pass

Same method as tiny-shakespeare: the real ggml op graph, captured by running eval-callback against this exact prompt on the actual downloaded weights. 1,266 real operations fire across 36 layers — over six times tiny-shakespeare's depth. The spine:

1GET_ROWStoken_embd.weight[4096,151936] → embd[4096,5]
2RMS_NORMembd → norm-0[4096,5]
1265RMS_NORM× output_norm → result_norm[4096,1]
1266MUL_MAToutput.weight[4096,151936] → result_output[151936,1]

Op 1266: 151,936 real numbers — one per vocabulary entry, versus tiny-shakespeare's 65. Same kind of number, same kind of op, 2337× the vocabulary to score.

Logits → a token

Softmax, then a roll

A second, separate trace — this time through Ollama's own API with logprobs requested — captured the real softmax output for this exact prompt. The top candidate for the very first generated token:

<think>
100.000%
Assistant
0.000%
же
0.000%
Even more one-sided than tiny-shakespeare's 99.996%. This model was RL-trained to always open with <think> — a stronger behavioral constraint than tiny-shakespeare's statistical pattern, but mechanically identical: a token in a fixed vocabulary, scored by softmax, drawn by the same sampler.

Repeat, 58 more times

The output accumulates

Same pipeline, run again for each new token. The real text this call produced:

token span Ollama's API labels message.reasoning
<think>Okay, the user is asking for the capital of France. Let me start by recalling the basic facts. I know Paris is the capital, but I should double-check to be sure. Right, it's definitely Paris. Now, I need to think about why the user is asking this.
Honest result, not a cherry-pick: none of these 59 tokens were a long-shot draw — every single one was the sampler's top-ranked candidate. Contrast with tiny-shakespeare's 4 long-shot draws in 40 steps. That doesn't mean this model never takes one — The Sheep Riddle, traced earlier in this project, catches it choosing 'b' at 3.6% mid-sentence. This window just happened not to.

What this explains

Mechanistic why vs. semantic why

The same split as tiny-shakespeare's journey, holding at 15x the scale. We can say exactly why <think> opened this response: a real forward pass produced real logits, softmax turned them into a 100.000% probability, and the sampler drew it. We cannot say why the trained weights represent "this looks like a question I should reason about" in any conceptual sense — and the one honest gap above (unreadable token ids) is a small preview of a much larger one: at 4096 hidden dimensions and 36 layers, there is far more happening per token than at tiny-shakespeare's 384 and 6.

Answered here

Every stage has real, cited numbers: the 5 input token ids, all 1,266 real ops, the exact top-token probability, the exact 59-token output. The <think> tag is not a separate mechanism — it is one entry in the same fixed vocabulary as every other token, sampled the same way.

Not answered here

What any of the 4096-dimensional hidden states represent. Why this model was trained to reach for reasoning tokens at all. Those require interpreting the weights, not tracing the computation — unsolved for this model exactly as for tiny-shakespeare, just at far higher dimension.

The pieces this journey is built from

deepseek-r1:8b Graph →All 1,266 real ggml ops captured from this model's weights, in full. The Sheep Riddle →A different prompt, traced the same way — where a long-shot draw actually happens. GPU Kernel Trace →The same class of ops above, as real CUDA kernels with real hardware timing. The Journey →The small-model original this page mirrors.