Stanford CS336 Assignment 1 for language-model fundamentals

How it works

The assignment defines interfaces and tests for parts of a small language model. Students implement the pieces and use the supplied checks to verify behavior.

What you need

Use the dependencies and instructions from the same course revision as the tests. Keep each implementation step small enough to check separately with a fixture.

What to try first

Start with the tokenizer and pass its tests before integrating attention and a training loop. Keep a tiny input that lets you verify tensor shapes and masking by hand.

Check a hand-computed training example before attempting the full exercise. A tiny batch can expose label shifts or masking errors without running a large training job.

Where to look for failures

Match the instructions and dependencies to the repository revision. Local tests check specific behaviors and still leave room for numerical or performance problems at larger scale.

Related Franklin pages

Use very small inputs when learning the implementation. Inspect the actual token indices, tensor shapes and gradients before relying on a larger training run to show what went wrong.

• Read and run a compact implementation of transformer language-model training. nanoGPT for studying a small GPT training loop
• Trace gradients through scalar operations in a deliberately small implementation. micrograd for learning reverse-mode differentiation

For the role of multilayer networks and how they learn, see Franklin’s glossary. Deep learning

Project documentation

• Current project README
• README at the reviewed revision