OctoLong: Mid-Training On Cross-Repository Code Con...

OctoLong: Mid-Training On Cross-Repository Code Con... | AI Research

Key Takeaways

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling introduces a pipeline designed to improve how language models handle...
Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows.
Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies.
OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling introduces a pipeline designed to improve how language models handle long-range dependencies in code.
By creating dependency-rich datasets, the researchers aim to solve the scarcity of high-quality, long-distance context in existing training corpora, which are typically limited to isolated books or code repositories.

Paper AbstractExpand

Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies. In this work, we introduce OctoLong, a context engineering pipeline that instruments an AST parser, a language server backend, and a package manager to facilitate the recursive retrieval of code references, enabling the curation of dependency-rich code contexts of millions of tokens in length. We then train OctoLong-Instruct, a suite of capable long-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context-extension mid-training on a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning. Our training ablations and experimental evaluations against 18 state-of-the-art open-weight long-context LMs show that supplanting just 12% of traditional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in short-context coding scenarios.

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling introduces a pipeline designed to improve how language models handle long-range dependencies in code. By creating dependency-rich datasets, the researchers aim to solve the scarcity of high-quality, long-distance context in existing training corpora, which are typically limited to isolated books or code repositories.

The OctoLong Pipeline

The researchers developed a context engineering pipeline that automates the retrieval of related code across different repositories. The system uses three primary components: an Abstract Syntax Tree (AST) parser, a language server backend, and a package manager. By integrating these tools, the pipeline can recursively retrieve code references, allowing for the creation of training contexts that reach millions of tokens in length.

Training OctoLong-Instruct

To test the effectiveness of this data, the authors created OctoLong-Instruct, a suite of open-weight language models ranging from 600M to 14B parameters. The training process involved two stages: 1. Mid-training: The models underwent context-extension training using a 50B-token mixture, of which 6.2B tokens consisted of the new OctoLong dependency-rich code contexts. 2. Instruction tuning: The models were further refined using 10B tokens of instruction-tuning data.

Performance and Results

The authors evaluated OctoLong-Instruct against 18 state-of-the-art open-weight long-context models. The results indicate that replacing just 12% of traditional context-extension training data with OctoLong data leads to improvements in several areas:

Long-range retrieval: The models became more effective at locating specific information within large volumes of code.
State tracking: The models showed better performance in maintaining long-term context.
Repository-level understanding: The models demonstrated a stronger grasp of how different files and dependencies interact within a codebase.
Agentic tasks: The models showed improved performance in downstream agentic workflows and enhanced API usage during short-context coding tasks.

Franklin Analysis

The evidence suggests that the primary value of the OctoLong approach lies in the quality of the training data rather than the quantity. By specifically curating "dependency-rich" contexts, the researchers were able to achieve performance gains by replacing only a small fraction (12%) of the standard training corpus. This indicates that for long-context modeling in software engineering, the structural relationships between code files are more critical for model performance than the sheer volume of generic text or isolated code snippets.