Contextual Information Policy Optimization for Sear...

Contextual Information Policy Optimization for Sear... | AI Research

Key Takeaways

Contextual Information Policy Optimization (CIPO) is a reinforcement learning framework designed to improve how search agents use external evidence during mu...
Search agents extend large language models beyond static parametric memory by enabling them to acquire and use ex ternal evidence during multi-step reasoning.
For knowledge intensive tasks involving complex or evolving information, their reliability depends not only on retrieving relevant ev idence but also on using it to guide subsequent reasoning.
However, existing methods primarily reward final-answer cor rectness or intermediate progress, without directly assessing whether post-retrieval actions are grounded in the retrieved evidence.
Importantly, CIPO requires neither human process annotations nor an additional reward model.

Paper AbstractExpand

Search agents extend large language models beyond static parametric memory by enabling them to acquire and use ex ternal evidence during multi-step reasoning. For knowledge intensive tasks involving complex or evolving information, their reliability depends not only on retrieving relevant ev idence but also on using it to guide subsequent reasoning. However, existing methods primarily reward final-answer cor rectness or intermediate progress, without directly assessing whether post-retrieval actions are grounded in the retrieved evidence. This misalignment encourages prior-driven reason ing: agents form conclusions based on internal knowledge and use retrieval mainly to confirm them, resulting in confirma tion bias and inefficient this http URL , we propose Contextual Information Policy Optimization (CIPO), an evidence-oriented reinforcement learning framework that explicitly aligns policy optimization with external evidence use. CIPO assigns dense, turn-level credit to reasoning ac tions influenced by retrieved information, while combining this evidence-use signal with a global outcome reward to pre this http URL ,CIPOdiscourages evidence-detached guesses and promotes reasoning trajecto ries in which retrieved facts can guide or revise subsequent reasoning. Importantly, CIPO requires neither human process annotations nor an additional reward model. Extensive exper iments on seven in-domain and out-of-domain benchmarks show that CIPO reduces the prevalence of prior-driven rea soning and achieves excellent performance on most tasks.

Contextual Information Policy Optimization (CIPO) is a reinforcement learning framework designed to improve how search agents use external evidence during multi-step reasoning. By training agents to prioritize information retrieved from external sources, the framework aims to reduce "prior-driven reasoning," where models rely on internal knowledge and use search tools only to confirm pre-existing hypotheses.

Addressing Reward Misalignment

Existing search agents often suffer from reward misalignment. Because many training methods only reward final-answer correctness or general progress, agents can receive high scores even when they ignore retrieved evidence. This leads to confirmation bias, where the agent forms a conclusion based on its internal memory and treats retrieval as a superficial confirmation step. CIPO addresses this by explicitly aligning policy optimization with the actual use of external information, ensuring that reasoning actions are grounded in retrieved facts.

How CIPO Works

CIPO introduces a mechanism called the Evidence-Access Log-Likelihood Ratio (EALR) to provide dense, turn-level feedback. The framework evaluates each reasoning step by comparing the likelihood of an action under two conditions: one where the retrieved evidence is visible to the model, and one where it is masked using an attention mask.
If an action becomes significantly more likely when evidence is visible, the EALR assigns a positive reward to that step. This signal is then combined with a global outcome reward—based on the final answer’s correctness—to guide the agent. This approach does not require human annotations or additional reward models, as it relies on the model’s own likelihood contrasts during training.

Performance and Results

The authors evaluated CIPO across seven benchmarks, including both in-domain tasks like Natural Questions and TriviaQA, and out-of-domain tasks like MuSiQue and PopQA. Using Qwen2.5-3B and Qwen2.5-7B as backbone models, the researchers found that CIPO consistently outperformed existing baselines, including outcome-only RL agents and other process-supervised methods. The results indicate that CIPO reduces the prevalence of prior-driven reasoning and achieves higher F1 scores across the tested datasets.

Key Considerations

CIPO is designed to function without the need for external reward models or human-labeled process data, which simplifies the training pipeline for search agents. However, the framework relies on the assumption that comparing evidence-visible and evidence-masked likelihoods is a sufficient proxy for measuring genuine evidence use. The effectiveness of this method is tied to the agent's ability to integrate these turn-level signals with final-answer rewards to maintain both reasoning quality and factual accuracy.