Nsight Systems — real CUDA kernel execution

GPU Kernel Trace

One actual deepseek-r1:8b inference call, profiled with NVIDIA Nsight Systems while ollama serve ran wrapped under the profiler. Every row below is a real CUDA kernel that executed on the RTX 2080 Ti, with real hardware timing — not the conceptual ggml op graph, the layer beneath it.

GPU: RTX 2080 Ti
Kernel launches: 2270
Distinct kernels: 27
GPU-busy time: 175.2 ms
Quantization: Q4_K_M

Where the GPU spent its time

mul_mat_q (Q4_K)

41.0%

mul_mat_vec_q (Q4_K)

18.7%

mul_mat_q (Q6_K)

8.2%

mul_mat_vec_q (Q6_K)

5.6%

mul_mat_vec_q (Q6_K)

4.6%

mul_mat_f

3.9%

mul_mat_vec_q (Q4_K)

3.7%

mul_mat_vec_q (Q4_K)

2.9%

mul_mat_vec_f

1.9%

mul_mat_q_stream_k_fixup

1.7%

Bar length is share of total GPU-busy time across all 2270 kernel launches in this call. mul_mat_q — the quantized matrix multiply — alone accounts for the majority: matrix multiplication is where an LLM's compute actually goes.

Every kernel, matched back to its ggml op

Kernel	Meaning	ggml op	Launches	Avg	Total	% time
mul_mat_q (Q4_K)	quantized matrix multiply	MUL_MAT	196	366.6 µs	71.9 ms	41.0%
mul_mat_vec_q (Q4_K)	quantized matrix x vector (single-token decode)	MUL_MAT	58	565.4 µs	32.8 ms	18.7%
mul_mat_q (Q6_K)	quantized matrix multiply	MUL_MAT	17	843.5 µs	14.3 ms	8.2%
mul_mat_vec_q (Q6_K)	quantized matrix x vector (single-token decode)	MUL_MAT	2	4.9 ms	9.8 ms	5.6%
mul_mat_vec_q (Q6_K)	quantized matrix x vector (single-token decode)	MUL_MAT	12	677.5 µs	8.1 ms	4.6%
mul_mat_f	float16 matrix multiply	MUL_MAT	36	191.5 µs	6.9 ms	3.9%
mul_mat_vec_q (Q4_K)	quantized matrix x vector (single-token decode)	MUL_MAT	196	32.8 µs	6.4 ms	3.7%
mul_mat_vec_q (Q4_K)	quantized matrix x vector (single-token decode)	MUL_MAT	45	111.4 µs	5.0 ms	2.9%
mul_mat_vec_f	float16 matrix x vector	MUL_MAT	22	153.2 µs	3.4 ms	1.9%
mul_mat_q_stream_k_fixup	merge step for split matmul	MUL_MAT	196	14.9 µs	2.9 ms	1.7%
flash_attn_ext_f16	fused attention (Q·K, softmax, ·V in one kernel)	FLASH_ATTN_EXT	36	59.3 µs	2.1 ms	1.2%
rms_norm_f32	RMSNorm	NORM	191	10.8 µs	2.1 ms	1.2%
quantize_mmq_q8_1	quantize activations to Q8_1 before matmul	(internal)	196	7.1 µs	1.4 ms	0.8%
mul_mat_vec_q (Q6_K)	quantized matrix x vector (single-token decode)	MUL_MAT	17	76.5 µs	1.3 ms	0.7%
flash_attn_ext_f16	fused attention (Q·K, softmax, ·V in one kernel)	FLASH_ATTN_EXT	58	21.3 µs	1.2 ms	0.7%
quantize_q8_1	quantize activations to Q8_1	(internal)	330	2.8 µs	913.4 µs	0.5%
rms_norm_mul_rope_f32	fused RMSNorm + rotary position embedding	NORM + ROPE	94	8.8 µs	829.3 µs	0.5%
k_bin_bcast (add)	residual / bias add	ADD	142	5.5 µs	781.8 µs	0.4%
rms_norm_mul_rope_f32	fused RMSNorm + rotary position embedding	NORM + ROPE	94	6.8 µs	636.9 µs	0.4%
unary_gated_op_kernel (SiLU)	SwiGLU activation	FFN	70	9.0 µs	630.7 µs	0.4%
mul_mat_vec_f	float16 matrix x vector	MUL_MAT	36	17.2 µs	618.4 µs	0.4%
k_set_rows	write into KV cache	SET_ROWS	94	3.8 µs	359.4 µs	0.2%
flash_attn_stream_k_fixup	merge step for split attention	FLASH_ATTN_EXT	36	7.8 µs	281.3 µs	0.2%
mul_mat_q_stream_k_fixup	merge step for split matmul	MUL_MAT	17	14.9 µs	253.2 µs	0.1%
flash_attn_stream_k_fixup	merge step for split attention	FLASH_ATTN_EXT	58	2.0 µs	116.2 µs	0.1%
quantize_mmq_q8_1	quantize activations to Q8_1 before matmul	(internal)	17	6.7 µs	114.3 µs	0.1%
k_get_rows_float	embedding lookup	GET_ROWS	4	4.1 µs	16.4 µs	0.0%

Memory traffic

Operation	Count	Total	% of memory time
[CUDA memcpy Host-to-Device]	4850	2.05 s	99.8%
[CUDA memset]	4	4.2 ms	0.2%
[CUDA memcpy Device-to-Host]	3	796.8 µs	0.0%

Host-to-Device copies dominate memory time — that is the model's weights moving from system RAM into GPU VRAM once at load time (matches the ~10s load_duration Ollama's own API reported for this call). It is a one-time cost per model load, separate from the per-token compute above.