Behavioral Controllability of Agentic Models for In... | AI Research

Behavioral Controllability of Agentic Models for Information Extraction: From Fixed Workflows to Reflective Agents
This research investigates whether adding "agentic" features—such as reflection, memory, and dynamic tool use—actually improves the performance and reliability of AI systems tasked with extracting information from scholarly documents. While many modern AI systems use these advanced features, it is often unclear if they provide a measurable advantage over simpler, fixed workflows. The authors propose a "process-centric" evaluation framework to determine if these agentic mechanisms lead to more controllable, observable, and effective outcomes when identifying datasets in conference papers.

Moving Beyond Fixed Workflows

The study compares a traditional, fixed-sequence workflow against several agentic variants. A fixed workflow follows a rigid, predetermined path to parse PDFs and extract data. While reliable, it lacks the ability to adapt if it encounters missing information or ambiguous terminology. In contrast, the agentic models are designed to "think" and "act" by observing intermediate results, reflecting on potential errors, and retrying tasks when necessary. The researchers aim to see if these behaviors create a more robust system that can recover from mistakes rather than simply failing or producing incomplete records.

Defining Behavioral Controllability

A core contribution of the paper is the concept of "behavioral controllability." The authors argue that for an agent to be useful, its decision-making process must be transparent and manageable. They define this through four key properties:

• Observability: The system must log its internal reasoning, tool calls, and reflection events.

• Configurability: Users should be able to adjust parameters like retry limits and reflection modes.

• Reproducibility: Every run must record the specific settings and environment snapshots used.

• Comparability: Different systems must be tested against the same documents and output requirements to ensure fair assessment.

The Optimized Agent Design

The authors introduce an "S2" optimized agent, which represents the most advanced version of their system. Unlike the simpler agents that follow a fixed high-level plan, the S2 agent uses a dynamic loop to select from twelve specialized tools—including specific functions for table extraction, reference searching, and evidence verification. By allowing the agent to choose the best tool based on the current state of the document, the researchers aim to improve how the system handles complex extraction tasks where evidence is scattered across figures, tables, and supplementary materials.

Measuring Success Through Process

Rather than focusing solely on the final number of records produced, the study emphasizes "process-level" metrics. The researchers track how often an agent reflects, how many times it retries a task, and how it utilizes memory to improve performance. This approach acknowledges that while reflection and memory might not always lead to a massive increase in the quantity of extracted data, they provide critical insights into how the system recovers from errors. The authors conclude that these process signals are essential for building AI agents that are not just capable, but also inspectable and tunable for real-world applications.