Back to AI Research

AI Research

Selective Agent Guidance via Entropy: Learning Auto... | AI Research

Key Takeaways

  • Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teachers This paper introduces SAGE, a framework designed to help AI ag...
  • We study how to learn a cheap autonomous policy from an online, expensive, and imperfect but informative VLM teacher.
  • Because VLM advice is not always reliable, SAGE can weight teacher-action distillation using environment-derived advantages rather than treating all suggestions as equally useful.
  • SAGE also reduces VLM usage by prompting the teacher only on a fraction of training steps and requiring no VLM calls at deployment.
  • Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teachers
Paper AbstractExpand

Vision-Language Models (VLMs) provide useful priors for interactive decision-making, but using them directly as policies is expensive and brittle: they must be queried at every step, do not improve from environment interaction, and can repeat systematic errors. We study how to learn a cheap autonomous policy from an online, expensive, and imperfect but informative VLM teacher. We propose SAGE (Selective Agent Guidance via Entropy), a framework that queries a VLM only when the learner is uncertain, executes the suggested action during training, and distills guidance into a lightweight Reinforcement Learning (RL) policy. Because VLM advice is not always reliable, SAGE can weight teacher-action distillation using environment-derived advantages rather than treating all suggestions as equally useful. Across sparse-reward visual reasoning and navigation tasks, SAGE learns policies that act without VLM guidance at evaluation time and improves over unguided RL in several environments, including settings where the learned policy exceeds its VLM teacher. The results show that selective guidance is most beneficial when the VLM can help the agent discover high-reward trajectories, and less useful when unguided exploration already succeeds or teacher actions do not lead to informative experience. SAGE also reduces VLM usage by prompting the teacher only on a fraction of training steps and requiring no VLM calls at deployment. Overall, our results suggest that VLMs don't need to be used as fixed policies to be useful; they can instead act as temporary, imperfect sources of guidance whose value is tested and internalized through interaction.

Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teachers
This paper introduces SAGE, a framework designed to help AI agents learn complex tasks more efficiently by using Vision-Language Models (VLMs) as temporary, selective teachers. While VLMs possess broad knowledge, using them as a permanent policy is often too slow, expensive, and prone to repeating systematic errors. SAGE addresses this by training a lightweight Reinforcement Learning (RL) agent that only consults the VLM when it is uncertain. Once the agent learns from this guidance, it can perform the task autonomously without needing the VLM at all during deployment.

How SAGE Works

The core of SAGE is an "entropy-gated" mechanism. The agent monitors its own decision-making confidence; when its uncertainty (measured by policy entropy) exceeds a specific threshold, it triggers a query to the VLM. The VLM provides an action, which the agent executes in the environment.
Crucially, the framework separates the learning process into two parts. Standard RL updates are used when the agent acts on its own, while teacher-provided actions are distilled into the agent using a technique called Advantage-Weighted Behavioral Cloning. This allows the agent to prioritize teacher advice that leads to successful outcomes while ignoring or down-weighting suggestions that result in poor performance.

Efficiency and Autonomy

A primary goal of SAGE is to move away from constant VLM reliance. Because the agent learns to internalize the teacher's guidance through interaction, it becomes increasingly capable of solving tasks on its own. By the time the agent is deployed, it no longer requires VLM calls, significantly reducing computational costs compared to methods that prompt a model at every single step. This approach allows the agent to improve through trial-and-error, potentially even surpassing the performance of the VLM teacher that guided its initial training.

Performance and Insights

Experiments across various visual reasoning and navigation tasks—such as FrozenLake, MiniGrid, and household interaction benchmarks—demonstrate that SAGE consistently outperforms standard RL agents that lack guidance, particularly in environments where rewards are sparse and difficult to discover.
The results highlight that selective guidance is most effective when the VLM helps the agent find high-reward trajectories that it would otherwise miss. However, the researchers note that the framework's success depends on the quality of the teacher; if the VLM provides uninformative or misleading guidance, the agent's learning can be hindered. Ultimately, the study suggests that VLMs are most valuable as temporary, imperfect mentors that help an agent "bootstrap" its own autonomous capabilities rather than serving as a fixed, permanent controller.

Comments (0)

No comments yet

Be the first to share your thoughts!