Exact model tokenization
INTERACTIVE NOTEBOOK
Inside the model
One context. 28 layers. 448 attention heads.
Trace how the prediction at each token changes.
MODEL Qwen3-0.6B28 LAYERS / 16 Q HEADS / 8 KV HEADSCheckpoint · loading…
Probability over the full vocabulary
28 layers × 16 heads
Text→tokens→28 layers→predictions
Every token under the microscope
Choose a template or enter your own context.
Layer predictions and attention maps will appear here after the run.
01 / LAYER EXPLORER
From input to prediction
Input tokens stay at the bottom. Read layers upward: layer 1 → final layer. Click a column to select a token. Use Attention to view its weights across layers.
Cell color and percentage show the next-token probability.
FINAL LAYER ↑ INPUTPrediction probability0%100%Scroll ↕ ↔ · drag the lower corner to resize
Input at the bottom · final layer at the top · positions start at 0
02 / TOKENIZATION
The tokenized context
Control tokenSelected positionAdded token
Appended text ⌄
03 / POSITION INSPECTOR
Position — / layer —
INPUT TOKEN
This position can only attend to itself and preceding tokens.
TOP-5 NEXT TOKENS
Color shows how much attention the selected position assigns to each token. Choose one head or the arithmetic mean of all query heads after softmax.
Intensity0100%▧ Future positions are hidden by the causal mask
Attention matrix
Rows are queries · columns are keys · click a cellQUERY → KEY
Hover over a cell to see its weight.
Click to select the query position.
All 16 heads in this layer
For the selected query position · click to switchMODEL & METHOD
How to read this experiment
The actual model
Qwen3-0.6B has 28 decoder layers, 16 query heads, and 8 KV heads. A full causal forward pass includes attention and MLP blocks. With GQA, query head h uses KV head ⌊h / 2⌋.
One LM head for every layer
After each layer, apply the final RMSNorm and shared language model head, then softmax over all 151,936 tokens. This is a logit lens: early layers were not trained to predict text independently. The final layer gives the standard model prediction.
Reasoning and the template
Chat uses the official apply_chat_template with enable_thinking. Disabling reasoning adds an empty <think>…</think> block. In Raw mode, you supply the control tokens and prefix. <think> is an added token with special=false in the tokenizer configuration.
Attention weights
Each head has a separate map after softmax, with an attention mask. The mean averages these probabilities across all query heads in the selected layer. Each row sums to approximately 100%. Color amplification only affects the display. Attention is not a complete measure of influence: values, output projections, MLPs, and the residual stream also contribute.
hₜ⁽ˡ⁾ → RMSNorm → LM head → softmax → argmax / A⁽ˡ,ʰ⁾ = softmax(QKᵀ / √128 + causal mask)
Float32 computation from the BF16 checkpoint; maps are stored in float16. The inspector limit is 512 tokens, including the continuation; input is never truncated. Checkpoint ↗