LLMs from Scratch for building a transformer step by step

How it works

The companion repository builds pieces such as tokenization and attention before combining them into training and tuning examples. Each chapter gives readers a narrow part of the model to inspect.

What you need

Choose the chapter corresponding to the component you want to understand. Use the small fixtures before attempting a complete training run or substituting another tokenizer.

What to try first

Work through attention on a tiny input and print the intermediate shapes. Run the accompanying code before moving to the next component.

Test a causal mask on a sequence with an obvious future token. Inspect the attention weights to confirm a position cannot read tokens that follow it.

Where to look for failures

The exercises are designed for learning and reproduction. Scaling up changes memory, data and evaluation requirements substantially.

Related Franklin pages

Use very small inputs when learning the implementation. Inspect the actual token indices, tensor shapes and gradients before relying on a larger training run to show what went wrong.

• Practice implementing tokenization, transformer components and training with a course test suite. Stanford CS336 Assignment 1 for language-model fundamentals
• Read and run a compact implementation of transformer language-model training. nanoGPT for studying a small GPT training loop

For the role of multilayer networks and how they learn, see Franklin’s glossary. Deep learning

Project documentation

• Current project README
• README at the reviewed revision