Transformer from Scratch

What you'll learn

• The inner mechanics of Self-Attention and Multi-Head Attention
• Implementing Positional Encodings
• Building a decoder-only architecture
• Training an autoregressive language model

Architecture

Input Tokens
  → Embedding + Positional Encoding
  → N x Transformer Blocks (Self-Attention + Feed Forward)
  → Linear Projection to Vocabulary
  → Autoregressive Generation

Steps

1. (Highly recommended: Watch Andrej Karpathy's "Let's build GPT" tutorial).
2. Prepare a small text dataset (e.g., the complete works of Shakespeare in a single .txt file).
3. Build a character-level tokenizer (mapping every unique character to an integer).
4. Implement a single SelfAttention head in PyTorch using query, key, and value matrices.
5. Expand it to a MultiHeadAttention module.
6. Build a TransformerBlock containing the attention module, layer normalization, and a feed-forward network.
7. Stack multiple blocks to create the final GPTLanguageModel.
8. Train the model for a few thousand iterations on an Apple Silicon Mac or a Colab GPU.
9. Write a generation function that seeds the model with a starting character and autoregressively generates a paragraph of text.