SAME 5 STEPS, DIFFERENT IMPLEMENTATION
BERT
Qwen2
ColBERT
1. Norm
2. Q/K/V
3. Position
4. Attention
5. Output
Post-norm (after)
Pre-norm RMSNorm
From base arch
Separate (GQA)
Absolute lookup
RoPE rotation
MHA, all same
GQA (fewer KV)
Bidirectional
1 pooled vector
N vectors (per-token)
All on packed sequences with flash_attn_varlen = zero padding waste