Graph-Structured Rubrics: Compiling Rubrics into Ty... | AI Research

Graph-Structured Rubrics (GSR) is a framework designed to improve the reliability of LLM-based evaluation by replacing implicit, prompt-based rubric interpretation with explicit, compiled evaluation graphs. Developed by researchers at Ant Group, the system converts natural-language rubrics into directed acyclic graphs (DAGs) before any candidate responses are observed. This approach ensures that criterion composition, aggregation, and gating are handled by deterministic operators rather than leaving the logic to the LLM during inference.

Compiling Rubrics into Graphs

The GSR framework operates in three distinct stages: compilation, execution, and readout. During compilation, an LLM synthesizes a program from a rubric specification, which is then checked by a deterministic validator. This validator rejects graphs that contain cycles, missing ports, or type-incompatible routes.
Once compiled, the graph is fixed. During the execution stage, criterion nodes elicit judgments from the LLM, which are then processed by deterministic operators—such as transformation, reduction, and gating—in a specific topological order. Finally, a task-specific "Readout" component maps the output of the graph to a final score or preference. Because the graph structure is determined before the model sees the candidate responses, the decision path is fully inspectable and replayable through audit traces.

Pointwise and Pairwise Evaluation

GSR provides a unified interface for both pointwise scoring and pairwise comparison. In pointwise evaluation, the system judges rubric dimensions separately before aggregating them into a final score. In pairwise evaluation, the same graph is reused to judge two candidates, with the final preference determined by the Readout policy.
The researchers evaluated GSR using the GPT-OSS-120B model across four pointwise datasets and two pairwise preference benchmarks. In pointwise tasks, GSR achieved higher exact score agreement compared to baselines like Prometheus, G-Eval, and FLASK, with improvements ranging from 0.62 to 6.75 percentage points. In pairwise settings, GSR achieved the highest end-to-end accuracy on both MT-Bench and RubricBench, while maintaining low rates of invalid outputs and ties.

Procedural Stability and Auditability

The primary motivation for GSR is to address the procedural instability often found in LLM judges, such as sensitivity to candidate order, output length, or inconsistent application of rubric rules. By forcing the evaluation logic into an explicit graph, the framework prevents issues like "safety caps" being ignored or reduction steps being applied out of order.
A key limitation noted in the research is that while the graph execution is deterministic and replayable, the initial criterion-level judgments remain dependent on the underlying LLM. Furthermore, the system assumes an available rubric and does not perform semantic screening to ensure the compiled graph perfectly captures the intent of the original natural-language rules; instead, it relies on downstream agreement metrics to validate the effectiveness of the generated graph.