Back to AI Research

AI Research

Repo-To-Skill: Distilling GitHub Repositories Into... | AI Research

Key Takeaways

  • Autonomous research agents are becoming increasingly capable of handling machine learning tasks, yet they often struggle with a specific hurdle: the gap betw...
  • Autonomous agents are beginning to carry out machine-learning (ML) research end to end.
  • These agents combine a model backbone with a harness for planning, execution, memory, and verification, but this architecture still leaves domain-specific know-how outside the agent.
  • We call this missing layer operational knowledge, the know-how that separates knowing a method from making it work.
  • That knowledge is not absent from the field.
Paper AbstractExpand

Autonomous agents are beginning to carry out machine-learning (ML) research end to end. These agents combine a model backbone with a harness for planning, execution, memory, and verification, but this architecture still leaves domain-specific know-how outside the agent. We call this missing layer operational knowledge, the know-how that separates knowing a method from making it work. That knowledge is not absent from the field. It appears in repositories and papers, but in forms written for human readers and too large to load during a task. Once distilled into compact, verified skills, this knowledge can be reused across tasks rather than rediscovered during each run. We present DisCo, a skill-powered research agent that creates skills and uses them during research. Its distillation runs in two complementary forms: task-agnostic, condensing the field's widely used repositories into reusable skills, and task-oriented, producing the skills a concrete task calls for. The former, applied across the open ecosystem, yields the AREX-Skill Library, with 5,000+ verified skills distilled from 1,000 widely used ML repositories and organized into 20 areas and 178 capability families. With the GPT-5.5 backbone, research harness, and downstream execution budget held fixed, the skill-equipped research agent scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet than the same agent without skills. These gains come from adding distilled operating context under that fixed setup.

Autonomous research agents are becoming increasingly capable of handling machine learning tasks, yet they often struggle with a specific hurdle: the gap between knowing a method and successfully implementing it. While these agents have strong reasoning backbones and structured planning harnesses, they lack "operational knowledge"—the practical expertise required to configure pipelines, handle API nuances, and avoid common implementation pitfalls. This paper introduces DisCo, a system that distills complex, human-readable documentation from GitHub repositories into compact, verified "skills" that agents can use to perform research more effectively.

The Missing Layer: Operational Knowledge

Current research agents rely on trial and error to figure out how to use tools or libraries, which wastes time and computational budget. The authors argue that this knowledge already exists in the open-source ecosystem, but it is trapped in formats designed for humans, such as long-form documentation and sprawling codebases. DisCo addresses this by converting these sources into a structured "skill" format. Each skill includes a clear interface for the agent to understand when to use it, a substrate of deeper technical documentation, and executable scripts that the agent can invoke directly.

How DisCo Distills Knowledge

DisCo operates in two distinct modes to build and use this knowledge. In "creator mode," the system performs distillation in two ways:

  • Task-agnostic distillation: The system processes widely used machine learning repositories ahead of time, creating a library of reusable skills that can be applied to any future task.

  • Task-oriented distillation: When faced with a specific, novel problem, the system identifies the capabilities it lacks and searches for relevant material to generate custom skills on the fly.
    Regardless of the method, every skill must pass a verification process before it is added to the library, ensuring that the agent is not relying on unproven or broken code.

The AREX-Skill Library

The primary output of the task-agnostic distillation process is the AREX-Skill Library. This collection contains over 5,000 verified skills distilled from 1,000 popular machine learning repositories. These skills are organized into 20 distinct areas and 178 capability families, managed by a router that helps the agent find the exact information it needs for a given task. By using a "progressive disclosure" approach, the agent only loads the specific parts of a skill graph that are necessary for the current step, keeping the system efficient and preventing the agent's context window from becoming overloaded.

Performance Gains

To measure the impact of these skills, the researchers compared an agent equipped with the AREX-Skill Library against an identical agent without it, while keeping the underlying model (GPT-5.5) and the research harness constant. The results showed significant performance improvements across four major benchmarks:

  • MLE-bench: 134.3% higher score.

  • PaperBench: 34.4% higher score.

  • PassNet: 14.0% higher score.

  • FrontierCS: 9.2% higher score.
    These results demonstrate that providing an agent with distilled operational context allows it to bypass the inefficiencies of trial-and-error exploration, leading to more successful and reliable research outcomes.

Comments (0)

No comments yet

Be the first to share your thoughts!