A language model dismantled into pieces small enough to understand, train, and rebuild.
next-char-model is a pedagogical project that progressively builds a small character prediction model, moving from embeddings toward multi-head attention and transformer-style blocks.
The problem
A complete transformer combines several mechanisms that are difficult to inspect at once. This project introduces them through a sequence of small scripts.
Design & implementation
Numbered scripts introduce the parts of a small autoregressive model: tokenization, embeddings, positions, attention, projections, residual connections, MLP layers, layer normalization, and then multi-head attention.
The project also includes embedding visualizations and a saved checkpoint, making it possible to move from learning code to a small loop of generation and inspection.
What it does
- Pedagogical PyTorch character-level language model
- Explicit progression from embeddings to blocks with multi-head attention
- Embedding visualizations before and after training
- Loading and reuse of a saved checkpoint
