Back to AI Research

AI Research

Estimating confidence in AI-written COBOL summaries

Key Takeaways

  • A preprint combines agreement across language models with a code coverage check to help reviewers triage COBOL summaries without human reference text.
  • The authors propose a confidence score that combines semantic agreement, corroborated contradictions and coverage of critical code paths.
  • A COBOL summarization preprint describes how teams could triage AI-written documentation when they have the source code but no human-written reference summaries.
  • The method compares descriptions produced by several language models and lowers confidence when they omit parts of the code.
  • Its output is a confidence estimate for review rather than a verified description of what the code does.

A COBOL summarization preprint describes how teams could triage AI-written documentation when they have the source code but no human-written reference summaries. The method compares descriptions produced by several language models and lowers confidence when they omit parts of the code. Its output is a confidence estimate for review rather than a verified description of what the code does. The preprint was submitted to arXiv on November 6, 2026.

What the models compare

The authors ask different models to summarize the same code chunk under controlled instructions. Each model gets a fixed set of context and is asked to cover purpose, control flow, inputs, outputs and effects on state. A separate evaluator compares pairs of summaries by meaning and returns structured judgments. That lets the framework distinguish descriptions that use different words for the same behavior from descriptions that make conflicting claims.

How the score handles disagreement

The aggregation rule combines supporting evidence and penalizes only corroborated contradictions. The framework separately checks whether a summary covers critical code paths, something word-overlap scores such as BLEU are not designed to establish. The coverage score reduces the final confidence estimate even when models agree. In the paper's worked example, a base confidence of 8.5 with a coverage score of 4 yields a final confidence of 3.4 on the 0 to 10 scale.

What a maintainer can do with the estimate

The use case is legacy COBOL documentation, where writing a human reference for every fragment would take the work the team is trying to prioritize. A reviewer can look first at summaries that disagree about behavior or skip conditions. The scoring logic is also a reminder that agreement alone cannot establish correctness: several models may omit the same branch or share the same mistaken reading. A team using the method to guide a migration still needs to check important behavior against the code.

Comments