DeepSpeed Ulysses
q0
k0
q1
k1
q2
k2
q3
k3
All2All
Full Flash Attention
All2All
o0
o1
o2
o3
What if we divide by num_heads dim?
AI text/layout recreation from video frame; verify against source image.