← Back to Glossary
Tokenization
Breaking raw text into smaller subword units (tokens) that an LLM converts into numbers to process.
Why it exists
LLMs don't read words or characters directly; they operate on subword tokens. Tokenization determines your context window consumption and API billing.
How it works
A tokenizer splits strings into token IDs (e.g. Byte-Pair Encoding). Common words are single tokens, while rare words or code are split into subword fragments.
Related terms
Where you'll use this
roadmap