The authors of BabyTheorist test how a learner builds a reusable explanation from observations. The benchmark presents before-and-after scenes and keeps the program that caused each change hidden. A learner has to discover which operations recur across examples and use them in new combinations. The preprint was submitted to arXiv on November 5, 2026.
What the learner sees
An observation is a pair of scenes, one before a change and one after it. The input lacks annotations for the generating program and the groups of examples that share a rule. The designers instead control which short programs recur and how increasingly complex programs are introduced. The curriculum lets an evaluator see where a learner handles new combinations and where performance stalls.
The benchmark distinguishes acquiring a rule from applying it. It can test whether a rule inferred from one scene pair transfers to a different scene that was generated by the same hidden program. That transfer matters because an explanation of one pair can fit the images while failing to generalize to another instance.
What the tested learners managed
The paper evaluates seven baselines, including black-box predictors and models built around latent representations or discrete programs. All seven stalled within the first two of the eight curriculum levels. The result places a limit on the progress those implementations made in this setting. It also gives future evaluations a way to identify which level or type of scene begins to defeat a method.
Who might use the benchmark?
Researchers who build visual reasoning systems can use BabyTheorist to test rule acquisition before asking for transfer. A run can compare learning before and after each level, when the set of observations changes. The tasks come from a controlled generator, so the results describe that particular setup. Understanding an unstructured physical scene is a separate empirical question.
Comments