Back to AI Research

AI Research

CompressionBench tests whether coding agents can deliver usable compressed models

Key Takeaways

  • The benchmark evaluates complete compressed artifacts under storage and time limits, including checks that perplexity alone can miss.
  • A smaller checkpoint is useful only if the model still runs correctly and produces acceptable answers.
  • [CompressionBench](https://arxiv.org/abs/2611.06858) turns that problem into a terminal task for coding agents, with a source model, storage limit, inference environment and time budget.
  • ## The agent controls the compression design
  • Agents construct compressed weights and executable inference code.

A smaller checkpoint is useful only if the model still runs correctly and produces acceptable answers. CompressionBench turns that problem into a terminal task for coding agents, with a source model, storage limit, inference environment and time budget.

The agent controls the compression design

Agents construct compressed weights and executable inference code. They can experiment within the task’s limits, using perplexity feedback to guide changes. A separate evaluation checks the submitted artifact after the development window.
That setup examines a complete delivery problem. Choices about numeric representation and the code that executes it interact: saving checkpoint space is not enough if the resulting inference path is incorrect or erases the intended benefit.

What the submissions achieved

Across six agents and six model-size settings, 29 of 36 submissions passed verification. Selecting the lowest-perplexity design for each task produced reported average downstream accuracy of 77.33%, compared with 79.56% for the BF16 reference.
The authors also describe improvements that looked promising under a local measure but changed ranking when checked against the complete model. Generation and inference tests revealed failures that likelihood measurements alone missed.
These results support studying agent-led compression as a bounded engineering task. They do not establish that every generated design is faster, cheaper or suitable for a different model and GPU.

How to use the benchmark’s lesson

For a compression experiment, keep the storage budget, runtime and final artifact fixed before evaluating. Compare quality on actual generated answers as well as likelihood, and verify the execution path independently of the agent’s own progress reports.
The useful contribution is the separation between experimentation and acceptance. An agent may find a better representation, but an independent check still needs to establish that the frozen result satisfies the deployment’s constraints. That makes failures informative rather than silently counting a promising intermediate experiment as a delivered system.

Comments