ICAE-Bench: Evaluating Coding Agents as Interactive...

ICAE-Bench: Evaluating Coding Agents as Interactive... | AI Research

Key Takeaways

ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders The rise of "vibe-coding"—where developers start with a high-level idea rather than a de...
The recent emergence of vibe-coding workflows is changing what coding agents are expected to do.
Yet existing benchmarks have not fully caught up with this shift, evaluating agents on static, fully specified tasks.
In this paper, we introduce ICAE-Bench, a benchmark for evaluating coding agents under interactive project-building settings.
The basic idea is to start from a fuzzy product requirement, simulating the dynamic paradigm with an automated User Agent.

Paper AbstractExpand

The recent emergence of vibe-coding workflows is changing what coding agents are expected to do. Instead of merely completing code under fully specified instructions, agents are increasingly expected to transform incomplete product intent into working software by combining various abilities including planning, requirement clarification, tool use, debugging, and repository-level construction. Yet existing benchmarks have not fully caught up with this shift, evaluating agents on static, fully specified tasks. In this paper, we introduce ICAE-Bench, a benchmark for evaluating coding agents under interactive project-building settings. The basic idea is to start from a fuzzy product requirement, simulating the dynamic paradigm with an automated User Agent. To make this setting both realistic and evaluable, ICAE-Bench introduces three key designs. First, to avoid the ambiguity of unconstrained fuzzy requirements, each task derives ambiguity from a precise real open-source repository with executable behavior. Second, to ensure high-quality and reproducible user simulation, ICAE-Bench grounds interaction through User Agent Data, allowing the User Agent to reveal hidden constraints without inventing new requirements or leaking implementation artifacts. Third, to evaluate open-ended repositories fairly, ICAE-Bench uses standardized black-box tests together with multi-dimensional diagnostics, including functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality.

ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders
The rise of "vibe-coding"—where developers start with a high-level idea rather than a detailed technical plan—has changed what we expect from AI coding agents. Instead of just writing snippets of code, agents are now tasked with building entire software projects from scratch. ICAE-Bench is a new benchmark designed to test how well these agents handle the ambiguity of real-world product requirements, requiring them to plan, clarify, and build functional software through interaction.

Moving Beyond Static Tasks

Traditional coding benchmarks often provide agents with fully specified instructions or existing codebases to edit. However, real-world software development is rarely that straightforward. ICAE-Bench simulates a more realistic workflow by starting with "fuzzy" product requirements. To ensure the benchmark remains fair and evaluable, it uses a grounded User Agent that provides specific, pre-recorded answers to an agent's clarification questions. This prevents the AI from hallucinating new requirements or leaking the final solution, forcing it to actually resolve ambiguity through dialogue.

A Three-Tiered Approach to Ambiguity

To measure how agents handle varying levels of uncertainty, the benchmark organizes tasks into three levels of difficulty. Level 1 provides the least amount of initial information, while Level 3 offers a more detailed starting point. By testing agents across these levels, researchers can observe how well they retrieve and apply information as the initial specification becomes more complete. This structure helps distinguish between an agent's ability to handle vague instructions and its fundamental skill in writing code.

Evaluating Real-World Software

Because there are many ways to build a piece of software, ICAE-Bench does not force agents to copy a specific implementation. Instead, it uses a multi-dimensional evaluation protocol. It combines standardized black-box functional tests—which check if the software actually works as intended—with diagnostics that look at structural fidelity, design quality, and how well the agent interacted with the user. This allows the benchmark to assess diverse, creative solutions while still holding them to a high standard of functional correctness.

Key Findings and Challenges

Experiments with six coding models and two agent frameworks revealed that building projects from ambiguous requirements remains a significant hurdle. While agents are often good at reproducing visible behaviors, they frequently struggle with hidden constraints, edge cases, and the long-term planning required to integrate different parts of a project. The results suggest that simply asking questions is not enough; the real bottleneck is the agent's ability to remember clarified requirements and translate them into a coherent, working repository.