Back to AI Research

AI Research

SWE-Serve: Benchmarking Agentic Engineering For Pro... | AI Research

Key Takeaways

  • SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving The researchers behind SWE-Serve have introduced a new benchmark designed to eva...
  • We introduce SWE-Serve, a benchmark for evaluating agents on production inference engineering tasks.
  • Implementing an inference feature can require coordinating multiple changes across the serving stack, including model support, runtime execution, and public APIs.
  • Dedicated inference benchmarks, meanwhile, focus primarily on isolated kernel generation or performance optimization rather than repository-scale production feature implementation.
  • SWE-Serve provides 53 repository-grounded tasks derived from recent production changes to SGLang, spanning six inference engineering families.
Paper AbstractExpand

We introduce SWE-Serve, a benchmark for evaluating agents on production inference engineering tasks. Implementing an inference feature can require coordinating multiple changes across the serving stack, including model support, runtime execution, and public APIs. Existing benchmarks provide limited coverage of production inference engineering: repository-level software engineering benchmarks do not target inference, while general terminal-agent benchmarks include only a few inference tasks. Dedicated inference benchmarks, meanwhile, focus primarily on isolated kernel generation or performance optimization rather than repository-scale production feature implementation. SWE-Serve provides 53 repository-grounded tasks derived from recent production changes to SGLang, spanning six inference engineering families. Each task executes on either CPU or a single GPU (H100) and is evaluated with hidden functional and regression tests, including, where applicable, end-to-end (E2E) serving tests and calibrated performance gates. Executable no-op and oracle controls, adversarial verifier review, and closed-book execution support task validity and evaluation integrity. Across 11 models and 31 model-effort configurations, the best-performing configuration achieves 75% mean pass@1. SWE-Serve exposes a substantial gap between completing tasks locally and achieving production correctness. On 19 tasks with end-to-end coverage, model-serving E2E tests reject roughly one-third of patches that pass every other test (45.9% under the verifier versus 69.4% with E2E tests excluded from scoring), with pass rate increasing for each model's best-performing configuration. By making the production correctness gap directly measurable, SWE-Serve enables the field to track whether future agents move beyond completing tasks locally to achieving production correctness.

SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
The researchers behind SWE-Serve have introduced a new benchmark designed to evaluate how well AI coding agents can handle the complexities of production-level inference engineering. While many existing benchmarks focus on general software tasks or isolated performance optimizations, SWE-Serve specifically tests whether an agent’s code changes work correctly within a complete, live inference system. By requiring agents to implement features that span model support, runtime execution, and public APIs, the benchmark measures the gap between code that passes local tests and code that actually functions in a production environment. The ai agents story also surfaces in Google’s Gemini AI Accessed Three Outside..., adding another angle.

Bridging the Gap to Production

A central finding of this research is the "production correctness gap." The authors discovered that many AI-generated patches pass standard functional and regression tests but fail when subjected to end-to-end (E2E) serving tests. In these scenarios, the code might look correct in isolation, but it fails to serve models properly through live interfaces or breaks when handling concurrent requests. By including 19 tasks with E2E coverage, the benchmark makes this failure point measurable, showing that removing these rigorous tests from the scoring process artificially inflates an agent's success rate by over 23 percentage points.

How the Benchmark Works

SWE-Serve consists of 53 repository-grounded tasks derived from real-world changes made to SGLang, an open-source inference-serving system. The benchmark is designed to be highly realistic, requiring agents to work within a containerized environment that mimics actual development conditions. To ensure the integrity of the evaluation, the researchers implemented several safeguards:

  • Executable Controls: Each task includes "no-op" tests (which must fail if no changes are made) and "oracle" solutions (which must pass all tests) to ensure the tasks are valid and solvable.

  • Adversarial Probing: The team used agent-assisted testing to identify and fix weaknesses in the task instructions and verification logic.

  • Closed-Book Execution: Agents are prohibited from accessing the public web or upstream repositories, forcing them to rely on their own reasoning rather than copying existing solutions. The ai agents story also surfaces in Google AI Introduces EnvHarness for Adaptive..., adding another angle.

Performance and Resource Trade-offs

When evaluating 11 different models across 31 configurations, the researchers observed a 40-percentage-point spread in performance, with top-performing models achieving a 75% mean pass rate. However, the results highlight that high performance often comes with significant trade-offs. For instance, some models achieve similar success rates while varying wildly in cost, wall-clock time, and token usage. The study suggests that simply chasing a higher pass rate can lead to diminishing returns, where marginal improvements in correctness require disproportionately higher computational and financial costs.

Key Takeaways for Future Development

By providing a standardized way to measure production correctness, SWE-Serve shifts the focus of agent evaluation from simple code completion to system-level reliability. The benchmark demonstrates that for inference engineering, local correctness is not enough. Future progress in the field will depend on developing agents that can navigate the complexities of persistent state, concurrent coordination, and end-to-end system requirements, ensuring that their contributions are truly ready for production deployment. The ai agents story also surfaces in Google launches Gemini 3.8 Flash and..., adding another angle. as detailed in the full paper on Arxiv

More from Franklin

Google launches Gemini 3.8 Flash and...newsGoogle’s Gemini AI Accessed Three Outside...newsGoogle opens early access to AI...newsGoogle AI Introduces EnvHarness for Adaptive...newsPrompt Revision as a Source of...researchVibe Coding is Deadvideo

Comments (0)

No comments yet

Be the first to share your thoughts!