Back to AI Research

AI Research

ADMITBench: A Safety-Governed Reference Framework f... | AI Research

Key Takeaways

  • ADMITBench is a reference framework designed to evaluate industrial Large Language Model (LLM) advisories by focusing on the admissibility of proposed action...
  • This white paper presents ADMITBench, a reference framework for evaluating industrial LLM advisories at the level of the proposed action.
  • Release 0.1.0 is a public reference implementation for technical and research evaluation, not an authorisation for physical execution.
  • ADMITBench is a reference framework designed to evaluate industrial Large Language Model (LLM) advisories by focusing on the admissibility of proposed actions rather than just the accuracy of a diagnosis.
  • ## Evaluating Actions Instead of Diagnoses
Paper AbstractExpand

This white paper presents ADMITBench, a reference framework for evaluating industrial LLM advisories at the level of the proposed action. The framework implements a versioned, safety-governed evaluation contract that checks whether a recommendation is supported by the available evidence, permitted under the stated authority and procedure, and acceptable under the plant-specific consequence checks encoded in the selected evaluation profile. In this report, \emph{safety-governed} means that eligibility is determined through explicit, non-compensatory checks derived from a versioned plant profile; it does not mean that the evaluator, model, or plant has been safety-certified. Release 0.1.0 is a public reference implementation for technical and research evaluation, not an authorisation for physical execution.

ADMITBench is a reference framework designed to evaluate industrial Large Language Model (LLM) advisories by focusing on the admissibility of proposed actions rather than just the accuracy of a diagnosis. Developed by Yash Misra, Javal Vyas, Siddharth Gutta, and Mehmet Mercangöz, the framework provides a structured method to determine if an AI’s recommendation is safe, authorized, and supported by evidence within a specific industrial plant environment.

Evaluating Actions Instead of Diagnoses

The authors argue that in industrial settings, a correct diagnosis is not a sufficient indicator of safety. An LLM might correctly identify a fault but propose an action that violates operating procedures, exceeds the system's authority, or relies on invalid sensor data. ADMITBench addresses this "diagnosis-to-action gap" by requiring that every LLM response be formatted as a structured "action record." This record includes the proposed action, the diagnosis, cited evidence, and confidence levels, allowing the framework to evaluate the proposal as an operational intervention rather than free-form text.

The Safety-Governed Evaluation Hierarchy

The framework uses a non-compensatory, tiered evaluation system (T0–T4) to determine if an action is eligible for use. Each tier acts as a "hard gate":

  • T0 (Record Integrity): Checks if the action record is structurally complete and compilable.

  • T1 (Evidence and State Validity): Verifies that the decision is based on available, temporally valid, and consistent data.

  • T2 (Hazard Understanding): Ensures the model recognizes the relevant hazard and justifies its action or escalates appropriately.

  • T3 (Action Admissibility): Confirms the action is permitted under the assigned authority level and follows established procedures.

  • T4 (Physical Consequence Verification): Uses rule-based or dynamic checks to ensure the action does not lead to unacceptable physical outcomes.
    If an action fails any of these gates, it is marked as ineligible and excluded from further utility ranking. This process is "safety-governed," meaning it relies on explicit, versioned plant profiles—referred to as "cartridges"—to define the rules for each facility.

Implementation and Scope

Release 0.1.0 includes two plant profiles: cstr_alpha (a continuous stirred-tank reactor) and column_alpha (a distillation column). These profiles allow researchers to test how different models handle specific industrial scenarios, such as when a system must escalate a decision rather than perform a direct intervention. The authors emphasize that this framework is a technical reference for research and engineering teams to inspect failure modes and compare advisory configurations.

Limitations and Important Use Restrictions

The authors explicitly state that ADMITBench is not a functional-safety certification, a site-acceptance method, or an authorization for autonomous deployment. The framework does not guarantee that a plant or model is safe; it only provides an auditable, replayable evaluation of whether a specific action record meets the constraints defined in a versioned profile. Human-authored engineering judgments remain central to the process, and the framework is intended to support, not replace, site-specific safety assessments, operator training, and human oversight.

Comments (0)

No comments yet

Be the first to share your thoughts!