FLY-EVAL++: An Evidence-Driven Evaluation Protocol... | AI Research

FLY-EVAL++: An Evidence-Driven Evaluation Protocol for Safety-Constrained Flight Prediction with Large Language Models
Evaluating Large Language Models (LLMs) in aviation is challenging because traditional accuracy metrics—which measure how close a prediction is to the ground truth—fail to capture whether a flight path is actually safe or physically possible. A model might produce a trajectory that looks numerically correct but violates critical flight constraints, such as bank limits or stall margins. This paper introduces FLY-EVAL++, a new evaluation protocol designed to move beyond simple accuracy by using deterministic, evidence-based verification to ensure that model outputs are safe, physically consistent, and operationally usable.

A New Standard for Safety

Existing benchmarks for flight prediction often rely on average-case metrics that are phase-agnostic, meaning they do not account for the specific, strict requirements of different flight stages like takeoff, landing, or cruising. FLY-EVAL++ addresses this by implementing a "protocol-first" approach. It uses a library of aviation-standard constraints to check every model output. If a model fails to produce a valid, structured output, it is filtered out immediately. This ensures that only usable, compliant predictions are subjected to deeper safety and physical analysis. The same ai evaluation question is explored in Wrong Prediction, Right Answer, which adds a research perspective.

How the Protocol Works

The methodology relies on "evidence atoms"—typed data points that represent specific checks, such as whether a prediction violates physical laws or exceeds safety thresholds. These atoms are categorized into five dimensions: protocol compliance, field validity, physical consistency, safety constraint satisfaction, and predictive quality.
Crucially, the protocol does not use an LLM to score the results. Instead, it uses deterministic verifiers—fixed, expert-designed rules—to calculate scores. This makes the evaluation process transparent, reproducible, and immune to the biases or inconsistencies that can occur when using an LLM to judge another LLM. The same large language models question is explored in Constrained Entity Selection under Partial Knowledge..., which adds a research perspective.

Key Findings from 66 Models

When testing 66 different LLMs, the researchers discovered that safety compliance is the most significant factor distinguishing high-performing models. While many models achieved similar scores in raw predictive accuracy, their safety scores varied by as much as 28 points. This reveals a "safety-predictive decoupling," where a model can be highly accurate in its numbers but remain operationally dangerous. The study also found that multi-step flight rollouts—where the model must predict a sequence of future states—often expose instability and delayed safety violations that single-step predictions miss.

Implications for Future Research

The authors emphasize that FLY-EVAL++ is a methodological template rather than just a tool for aviation. While the current version is built specifically for Flight Trajectory and Attitude Prediction (FTAP), the framework of using evidence-based, deterministic verifiers can be applied to any safety-critical domain. By shifting the focus from aggregate accuracy to explicit constraint satisfaction, developers can better identify the specific failure modes of their models, leading to more reliable and safer deployments in real-world, physics-governed environments. The same ai evaluation question is explored in Door-in-the-Face Requests and Refusal Behaviour in..., which adds a research perspective. as detailed in the full paper on Arxiv