Policy Iteration with Human Feedback: Bringing Post... | AI Research

Policy Iteration with Human Feedback (PIHF) is a framework designed to improve the diagnostic accuracy of language models in rare-disease cases by using a human-in-the-loop, iterative refinement process. Instead of fine-tuning model weights, PIHF keeps the underlying language model fixed and instead iteratively updates a versioned, natural-language policy and tool set that guides the model’s reasoning and tool use.

How PIHF Works

The PIHF method functions as a loop of evaluation and improvement. First, a pretrained language model acts as an executor, applying a specific policy and tool set to a panel of clinical cases. Once the model completes these tasks, a critic (an LLM) and a clinical expert review the reasoning and tool-use trajectories to identify recurrent failures.
The expert then authorizes a candidate revision to the policy or tool set. This candidate is "frozen" and tested against the same panel of cases. The system only adopts the new policy if it passes two checks: it must maintain or improve diagnostic performance (measured by Recall@1 and Recall@5) and meet the expert’s qualitative standards for clinical soundness and generalizability.

The Role of Human Expertise

In PIHF, the human expert retains final authority over the development process. While the critic provides an initial analysis of where the model failed, the expert can reinterpret the evidence, reject the critic’s suggestions, or create entirely new revisions. This structure allows the expert to manage the "admission and rollback" of policy changes, ensuring that the model’s behavior remains aligned with clinical requirements. By separating the process of refining reasoning (via expert feedback) from the validation of outcomes (via Recall metrics), the method aims to make credit assignment for model errors more precise.

Performance and Results

The authors, Minh-Ha Nguyen and Cathy Shyr, tested PIHF across ultra-rare-disease benchmarks using both proprietary and open-weight models ranging from 3 billion to 49 billion parameters. The study reported significant improvements in Recall@1, a metric measuring the accuracy of the model's top-ranked diagnosis. Specifically, the PIHF-derived policy improved Recall@1 by 32.7 percentage points for the GPT-5.4 executor and 31.1 percentage points for the Qwen3.6-35B executor.

Franklin Analysis

The results suggest that pretrained language models can serve as effective, fixed-weight execution substrates for specialized tasks when paired with an external, expert-governed policy. The evidence indicates that by focusing on "policy-level reuse"—where the policy and tools are the objects of improvement rather than the model weights—the system can achieve gains in diagnostic accuracy while maintaining a clear, auditable trail of how the model’s behavior evolved across iterations. The reliance on expert-authorized, versioned artifacts provides a mechanism for controlled, iterative development in high-stakes domains like rare-disease diagnosis.