Back to AI Research

AI Research

Why Sample What You Can Enumerate? Exact Policy Opt... | AI Research

Key Takeaways

  • This paper addresses a structural mismatch in how artificial intelligence models are trained to use external scientific tools.
  • Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which external tools to invoke.
  • We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool-subset space is enumerable.
  • Worse, the approximation degrades as training succeeds: as the policy concentrates on preferred subsets it resamples them, sampled rewards collide, and the group-normalized advantage vanishes.
  • On genomic reasoning the fraction of questions yielding no reward signal rises from 0.2% under a uniform reference policy to 20.8% after GRPO training.
Paper AbstractExpand

Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which external tools to invoke. We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool-subset space is enumerable. There, a small set of recurring computational capabilities covers the domain, so the space of tool subsets is combinatorial yet small enough to enumerate, and GRPO still estimates an action expectation from a handful of sampled rollouts. Worse, the approximation degrades as training succeeds: as the policy concentrates on preferred subsets it resamples them, sampled rewards collide, and the group-normalized advantage vanishes. On genomic reasoning the fraction of questions yielding no reward signal rises from 0.2% under a uniform reference policy to 20.8% after GRPO training. As a remedy, we introduce FGPO (Full-Group Policy Optimization), which (1) scores every tool subset and optimizes the exact action expectation, so each update sees the complete action space, and (2) precomputes the reward of each question--subset pair into an exhaustive table, removing frozen-reasoner calls from the training loop entirely. Across five frozen reasoners and three genomic benchmarks, FGPO outperforms GRPO in all 15 settings by 6.75 points on average and up to 14.20, while a standard on-demand GRPO schedule would require 2.4 times as many frozen-reasoner reward evaluations and, on GenomeQA, FGPO cuts invoked tools per question from 2.36 to 1.40.

This paper addresses a structural mismatch in how artificial intelligence models are trained to use external scientific tools. Currently, a common approach is to use reinforcement learning (specifically GRPO) to teach a language model which tools to invoke for a given task. However, in specialized scientific fields like genomics, the number of possible tool combinations is small enough to be fully listed. The authors argue that because these combinations are enumerable, the standard practice of "sampling" a few tool combinations to estimate their value is inefficient and counterproductive.

The Problem with Sampling

In genomic reasoning, researchers often use a small library of tools to help a language model answer complex questions. Because the library is small, there are only a limited number of possible subsets of tools that can be used. The authors found that as a model learns, it tends to focus on a few preferred tool combinations. When the model only samples a handful of these combinations during training, it often encounters "dead groups"—situations where all sampled combinations yield the same reward. When this happens, the model receives no useful feedback to improve its performance, causing the training process to stall or degrade. The ai search story also surfaces in EU Regulators Demand Apple and Google..., adding another angle.

Introducing FGPO

To solve this, the authors introduce Full-Group Policy Optimization (FGPO). Instead of sampling a few random tool combinations, FGPO evaluates every possible subset of tools for every question. By scoring the entire action space, the model receives a complete picture of which tools are effective, rather than relying on a limited, potentially biased sample. Additionally, FGPO precomputes the rewards for every possible tool combination into an exhaustive table. This removes the need to repeatedly call the "frozen reasoner" (the language model being assisted) during the training loop, making the process faster and more stable.

Key Results

The researchers tested FGPO against standard GRPO across three genomic benchmarks and five different language models. The results showed that FGPO outperformed GRPO in every single setting, improving accuracy by an average of 6.75 points and up to 14.20 points. Beyond higher accuracy, FGPO proved to be more efficient; it required significantly fewer tool calls per question, demonstrating that the model learned to select only the most relevant tools rather than relying on excessive, unnecessary computations. The ai search story also surfaces in New AI Architecture Mimics the Human..., adding another angle.

Why This Matters

This research highlights that when a problem space is small and well-defined, "exact" optimization is superior to the approximation methods typically used in reinforcement learning. By moving away from sampling and toward exhaustive evaluation, the authors demonstrate that models can learn more precise, query-dependent strategies. This approach not only boosts performance but also provides a clearer view of how a model makes decisions, as the policy is forced to account for the entire range of available tools rather than just a few randomly selected ones. The ai search story also surfaces in House Intelligence Committee Warns AI Safeguards..., adding another angle. as detailed in the full paper on Arxiv

More from Franklin

Spurious Advantage Hidden in GRPOresearchProgram Learning with Verifiable RewardsresearchNew AI Architecture Mimics the Human...newsC-BRAIN Launches Open-Source AI Tools to...newsEU Regulators Demand Apple and Google...newsHouse Intelligence Committee Warns AI Safeguards...news

Comments (0)

No comments yet

Be the first to share your thoughts!