← Back to Glossary
Positional Encoding
Numerical vectors added to token embeddings so the transformer knows the exact order of words in a sentence.
Why it exists
Because self-attention processes all tokens simultaneously in parallel, it has no inherent sense of word order without position signals.
How it works
Adds mathematical patterns (sine/cosine waves or Rotary Position Embeddings - RoPE) to input token vectors prior to attention layers.
Related terms
Where you'll use this
roadmap