Tag: ALiBi

item-image
Jul, 3 2026

RoPE vs ALiBi: How Modern Positional Encodings Power Long-Context LLMs

Explore how RoPE and ALiBi solve positional encoding in LLMs. Compare their math, extrapolation power, and adoption in models like Llama and GPT-NeoX.