← Back to Glossary
Pretraining
The initial, large-scale training phase where an LLM learns language patterns, facts, and logic by predicting missing words across billions of web pages.
Why it exists
Pretraining gives the neural network foundational understanding of grammar, world knowledge, and code before domain customization.
How it works
Processes trillions of unlabelled text tokens, predicting next tokens via self-supervised learning across thousands of GPUs over weeks or months.
Related terms
Where you'll use this
roadmap