Rethinking Complexity Metrics for LLM-Integrated Ap...

Rethinking Complexity Metrics for LLM-Integrated Ap... | AI Research

Key Takeaways

Rethinking Complexity Metrics for LLM-Integrated Applications: Beyond Source Code Modern software increasingly relies on "LLM-integrated applications," which...
LLM-integrated applications blend natural language prompts with program code, and much of their runtime behavior originates in the prompt layer rather than in the code itself.
Existing complexity metrics, however, operate solely at the code level and therefore overlook this behavioral logic entirely.
We present HECATE, the first tool designed to assess complexity in both the prompt and code layers of such applications.
Central to HECATE is Prompt-as-Specification, a Hoare-logic-inspired formalism that interprets every prompt as a specification of intended behavior.

Paper AbstractExpand

LLM-integrated applications blend natural language prompts with program code, and much of their runtime behavior originates in the prompt layer rather than in the code itself. Existing complexity metrics, however, operate solely at the code level and therefore overlook this behavioral logic entirely. We present HECATE, the first tool designed to assess complexity in both the prompt and code layers of such applications. Central to HECATE is Prompt-as-Specification, a Hoare-logic-inspired formalism that interprets every prompt as a specification of intended behavior. Grounded in 25 complexity dimensions identified across published taxonomies, the tool generates 52 candidate metrics. We assess each metric against 118 components collected from 18 open-source repositories, relying on maintenance activity derived from version history as an empirical proxy for complexity, and discard any metric that loses significance once code size is accounted for. Only ten metrics withstand this test. Seven belong to our newly introduced set; rather than measuring sheer volume, each tallies structurally distinct elements, such as LLM call sites, memory attributes, and prompt templates, an attribute we call structural breadth. Of the three surviving conventional metrics, RFC exhibits a similar breadth-oriented character, while Halstead N and V survive only as a residual effect of size; our top-performing metrics exceed all three. Crucially, the prompt-layer metrics retain significance even when the strongest code-level metric is added as a covariate, establishing prompt complexity as a dimension in its own right. A final validation on 20 components spanning six held-out repositories shows that the two best-performing metrics continue to predict maintenance effort, supporting their generalizability beyond the training set.

Rethinking Complexity Metrics for LLM-Integrated Applications: Beyond Source Code
Modern software increasingly relies on "LLM-integrated applications," which combine traditional code with natural language prompts to handle complex tasks. While developers have long used metrics like cyclomatic complexity to track code quality, these tools only analyze the programming language, leaving the logic hidden within prompts entirely unmeasured. This paper introduces HECATE, a static analysis tool designed to quantify complexity across both the code and the prompt layers, providing a more accurate picture of how difficult these hybrid systems are to maintain.

The Prompt-as-Specification Model

To measure the prompt layer, the researchers developed a framework called "Prompt-as-Specification." Inspired by Hoare logic—a formal method for verifying program correctness—this approach treats prompts not as free-form text, but as structured behavioral specifications. By breaking down prompts into conditional rules, global invariants, and context-dependent predicates, the tool can identify the "logic" embedded in natural language. This allows the system to treat a prompt's instructions with the same analytical rigor usually reserved for source code.

Measuring Structural Breadth

Rather than simply measuring the "volume" or size of an application, HECATE focuses on "structural breadth." The researchers identified 25 complexity dimensions across the code, the prompts, and the interface between them. From these, they derived 52 candidate metrics. Through rigorous testing against 118 components from open-source repositories, they found that most traditional metrics fail to provide meaningful data once the size of the code is accounted for. Only ten metrics remained significant, seven of which were newly proposed by the team. These successful metrics count distinct elements—such as the number of LLM call sites, memory attributes, and prompt templates—rather than just lines of code.

Why Prompt Complexity Matters

A key finding of the research is that prompt complexity is an independent dimension of software health. Even when controlling for the strongest code-level metrics, the prompt-layer metrics continued to predict maintenance effort, such as the number of bug-fix commits. This confirms that the logic inside prompts is not just a byproduct of the code; it is a distinct source of complexity that requires its own monitoring. The researchers validated these findings on a held-out set of 20 components from six unseen repositories, demonstrating that their top-performing metrics are generalizable beyond the initial training data.

Practical Implications for Developers

HECATE operates as a deterministic, dependency-free static analysis tool. It does not require executing the application or calling an LLM to function, making it a practical addition to a developer's toolkit. By identifying which components are truly complex—rather than just large—the tool helps engineering teams pinpoint modules that are most likely to be difficult to debug or maintain. This shift from measuring raw code size to measuring structural breadth offers a more reliable way to manage the risks associated with building AI-integrated software.