micrograd for learning reverse-mode differentiation

How it works

A scalar value records the operation and its parents, and a backward pass visits the graph in reverse dependency order. A simple neural network uses the same machinery.

What you need

Read the scalar value class before following the network example. A tiny graph with visible intermediate values is enough to test what the backward pass accumulates.

What to try first

Construct a scalar expression with a reused intermediate and calculate its derivative by hand. Compare that value with the engine's gradient and a finite difference.

Try a function with a branch reused in two places. The hand calculation shows why a backward pass needs to accumulate contributions at shared nodes.

Where to look for failures

Scalar graphs are easy to inspect and expensive for large tensors. Use the examples to understand gradient flow and numerical checks.

Related Franklin pages

Use very small inputs when learning the implementation. Inspect the actual token indices, tensor shapes and gradients before relying on a larger training run to show what went wrong.

• Follow runnable chapter code for language-model components and training. LLMs from Scratch for building a transformer step by step
• Practice implementing tokenization, transformer components and training with a course test suite. Stanford CS336 Assignment 1 for language-model fundamentals

For the role of multilayer networks and how they learn, see Franklin’s glossary. Deep learning

Project documentation

• Current project README
• README at the reviewed revision