M4Bench tests whether coding agents can reproduce clinical variables derived from electronic health records. The benchmark contains 28 tasks across fifteen task families on MIMIC-IV and eICU, where definitions depend on item codes, time windows and missing-data rules.
Give the agent an operational definition
The authors compare clinician-reviewed skills with controls for specification completeness, delivery format and schema familiarity. In the reported Codex campaign, skills improve reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving.
That reward change is not a 13.6-percentage-point increase in completed tasks. The paper distinguishes partial agreement with a reference derivation from passing every required check.
Much of the improvement comes from the specification
On an eight-task subset, an operational specification alone raises reward by 0.162. Skills add 0.024 beyond that, an increment the authors describe as indistinguishable from zero.
Other controls find that decoy skills perform like no skill and raw reference SQL nearly matches targeted skills on a ten-task subset. These comparisons argue against interpreting the skill file’s format, or simply adding more context, as the sole explanation for the observed gain.
Strict completion remains uncommon
Only 25 of 280 with-skill Codex runs pass every strict check, compared with 13 of 280 without skills. The authors also report little change in which rows the agents return, with key F1 improving by 0.003.
A Claude Code replication is directionally consistent, while schema changes provide descriptive robustness checks rather than a bound on memorisation. The released tasks, skills and sanitised run evidence support further investigation.
For teams deriving research variables, the useful lesson is to make definitions operational and evaluate row selection as well as calculated values. A better average reward should not replace checks that the full cohort and derivation meet the intended specification.
Comments