OpenAI has described two evaluations for coding in languages with less readily available training material. Its October 19 publication examines both building software and migrating existing programs while preserving their behaviour.
Building an engine across languages
MultiLangBench asks models to build a SQLite-compatible database engine across 20 programming languages. Candidate implementations receive a database and a SQL query, then produce a modified database whose contents are checked against reference SQLite results.
Using database files as the interface lets the evaluation apply comparable tests across languages. OpenAI reports gains from its newer models, but these remain results from the company’s chosen evaluation. They do not establish equal reliability across every legacy application.
A migration has to preserve details
The second evaluation migrates COBOL programs into languages including Java, Python and Rust. Success requires all hidden tests to pass with byte-for-byte agreement on observable outputs.
OpenAI highlights a rounding example: discarding excess decimal digits and rounding them can produce different results even when the rewritten program runs without errors. That illustrates why an apparently clean translation can still change a business rule.
The publication explicitly says production use requires expert review and testing against the system’s rules and dependencies. A benchmark pass cannot substitute for that work.
How to interpret the result for a real project
Franklin’s interpretation is that a migration evaluation should reward preserved behaviour, including awkward edge cases, rather than familiar-looking code. For a real assessment, first identify the outputs and failure conditions that must remain stable. Then compare old and new implementations on the same inputs, including boundary values and missing records.
The remaining question is coverage: passing known comparisons only establishes agreement on the cases exercised. A team still needs a way to identify undocumented dependencies and business rules before replacing an operational system.
Read OpenAI’s publication for the evaluation designs and reported results.