The TCSExam paper evaluates what AI agents can produce on theoretical computer science problems. It separates theorem proving, solver engineering, mathematical constructions and attempts on open conjectures, so a model’s aggregate score can be read alongside the kind of work it completed. The preprint was submitted to arXiv on November 5, 2026.
How the tasks are checked
TCSExam-Full contains 100 expert-reviewed tasks. Depending on the track, an output is checked by the Lean kernel or an executable task checker. A solver can be measured across problem instances, while a construction can be checked against the requested mathematical properties. The proof track distinguishes a complete proof from progress on a guided checkpoint.
TCSExam-Lite selects 20 tasks, five per track, and keeps the same budget per attempt. The smaller set reduces the number of attempts required to run the evaluation. Its usefulness depends on how well the selected tasks reflect the differences among agents that the larger set captures.
Which agents led each version
The authors report an index of 71.9 for GPT-6 Astra, which led Full, and 72.3 for Opus 5.5, which led Lite. Across the eight agents in both evaluations, the authors report a Spearman rank correlation of 0.93. On 20 theorem tasks without intermediate lemmas, they report a complete-proof success rate per attempt of 70.0% for GPT-6 Astra and 38.3% for Kimi K3.1 Pro.
What the score leaves for readers to inspect
The combined index summarizes work across tracks with different checking procedures. A reader choosing an agent for proof work can examine the proof attempts separately from solver or construction results. The paper also describes independently verified special cases and bounds from conjecture attempts. These are useful research artifacts with a narrower scope than resolving the original conjecture. Franklin’s Ramanujan Challenge coverage examines how a different benchmark checks mathematical proofs.
Comments