Back to AI Research

AI Research

How to Spend Your Oracle Budget: Practical Guidance... | AI Research

Key Takeaways

  • This research provides a practical framework for choosing guidance methods to improve protein structure predictions when using expensive external oracles.
  • Foundation models for protein structure prediction remain unreliable on certain targets.
  • External oracles can flag and correct these failures, but biological oracles are expensive, making oracle budget a critical constraint.
  • Existing guidance methods, such as FK-steering, DPO, and Best K-of-N sampling, differ in how they spend this budget, yet no systematic comparison exists to guide method selection.
  • To bridge this gap, we benchmark these methods alongside the recently proposed Optimisation Over Outputs (O3), which applies off-the-shelf optimisers within a generative model's latent subspace.
Paper AbstractExpand

Foundation models for protein structure prediction remain unreliable on certain targets. External oracles can flag and correct these failures, but biological oracles are expensive, making oracle budget a critical constraint. Existing guidance methods, such as FK-steering, DPO, and Best K-of-N sampling, differ in how they spend this budget, yet no systematic comparison exists to guide method selection. To bridge this gap, we benchmark these methods alongside the recently proposed Optimisation Over Outputs (O3), which applies off-the-shelf optimisers within a generative model's latent subspace. We extend the usage of O3 to protein structure prediction models. Overall, our work provides the first practical reference for oracle budget-aware guidance. Our evaluation on two protein targets, calmodulin (1CLL) and E. coli aspartate transcarbamoylase (9EEH), reveals that no single method consistently dominates across all budgets and oracles. Specifically, O3 proves most effective at low oracle budgets, while FK-steering and DPO demonstrate improved performance as the budget increases. We distil these findings into actionable recommendations for practitioners operating under real-world oracle-budget constraints.

This research provides a practical framework for choosing guidance methods to improve protein structure predictions when using expensive external oracles. Because foundation models like Boltz-2 can produce unreliable structures, researchers use oracles—such as molecular dynamics simulations or wet-lab assays—to score and correct these outputs. However, these oracles are computationally expensive or slow, making it necessary to optimize how a limited "oracle budget" is spent. The authors benchmark four guidance strategies to determine which perform best under different budget constraints.

Guidance Methods for Protein Prediction

The paper evaluates four distinct approaches to steering generative models toward higher-scoring protein structures:

  • Optimisation Over Outputs (O3): This method constructs a low-dimensional subspace within the model’s latent space using a small set of initial samples. It then applies Bayesian optimization to explore this subspace.

  • Feynman-Kac (FK) Steering: An inference-time method that generates multiple interacting particles and resamples them at intermediate steps based on oracle scores.

  • Direct Preference Optimisation (DPO): A fine-tuning approach that updates the model’s parameters by training on pairs of structures, favoring those with higher oracle scores.

  • Best K-of-N Sampling: A baseline method that generates a large batch of samples and simply returns the top K structures based on oracle scores.

Performance Across Oracle Budgets

The study, conducted by researchers from InstaDeep Ltd, the University of Oxford, and Lancaster University, tested these methods on two protein targets: calmodulin (1CLL) and E. coli aspartate transcarbamoylase (9EEH).
The results indicate that no single method is superior in every scenario. O3 is the most effective strategy when the oracle budget is low to moderate (up to 1,000 queries). Conversely, FK-steering and DPO show improved performance as the budget increases, though they struggle to compete with simpler methods at very low budgets. Best K-of-N sampling remains a consistent, albeit lower-performing, baseline across all tested budgets.

Practical Recommendations

The authors conclude that practitioners should select their guidance strategy based on their specific resource constraints:

  • Low-budget scenarios: O3 is recommended because it effectively leverages a low-dimensional subspace to find high-quality structures without requiring extensive oracle evaluations.

  • High-budget scenarios: FK-steering and DPO become more viable as the number of available oracle queries grows, allowing for more complex optimization or model fine-tuning.

Franklin Analysis

The evidence suggests that the effectiveness of these guidance methods is highly dependent on the available oracle budget. The authors’ findings regarding O3 are particularly notable, as it is the only method that meaningfully outperforms the Best K-of-N baseline at low budgets (N ≤ 100). The study also highlights a trade-off in the O3 method: while increasing the subspace dimension (d) can provide a richer search space, it also increases the difficulty of the optimization task, meaning the optimal dimension must be tuned relative to the available budget.

Comments (0)

No comments yet

Be the first to share your thoughts!