Back to AI Research

AI Research

Agentic coding without the cloud: evaluating open-w... | AI Research

Key Takeaways

  • Researchers often face a significant hurdle when working with sensitive personal data: strict governance requirements that prohibit sending information to th...
  • Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models.
  • Their adoption in research using personal data is constrained by governance requirements that typically prohibit data transmission to external services.
  • Locally deployable open-weight models offer an alternative since sensitive data never leave the local environment.
  • We introduce an open-source framework for evaluating the efficacy of AI agents powered by open-weight LLMs on one of the most persistent bottlenecks in research on longitudinal population studies: data preparation.
Paper AbstractExpand

Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models. Their adoption in research using personal data is constrained by governance requirements that typically prohibit data transmission to external services. Locally deployable open-weight models offer an alternative since sensitive data never leave the local environment. We introduce an open-source framework for evaluating the efficacy of AI agents powered by open-weight LLMs on one of the most persistent bottlenecks in research on longitudinal population studies: data preparation. The framework comprises: a curated ground-truth dataset (cleaning scripts preparing six sweeps of data from a British cohort study), task definitions encompassing tasks such as category harmonization and multi-wave merging, and automated routines for evaluating the LLM-produced R code and outputted data. We benchmark LLMs across the (consumer grade) deployment spectrum to assess their efficacy in 20 data preparation tasks (creation of 102 variables). Current state-of-the-art, 31-35B parameter models almost saturated our benchmark ("average task completion" up to 87.9%). The performance of open-weight LLMs running on consumer-grade hardware shows promise of a viable path toward AI-assisted data preparation in governance-restricted research settings. Our framework is publicly available at: this https URL .

Researchers often face a significant hurdle when working with sensitive personal data: strict governance requirements that prohibit sending information to third-party cloud services. This restriction prevents the use of many popular AI coding assistants, which typically rely on external servers. This paper introduces a new open-source framework designed to enable AI-assisted data preparation using locally deployable, open-weight large language models (LLMs), ensuring that sensitive data never leaves the local environment.

Bridging the Gap in Research

Data preparation is a persistent bottleneck in longitudinal population studies, involving complex tasks like merging data across multiple waves and harmonizing categories. The authors developed a framework to evaluate how well AI agents can handle these specific coding challenges. By using a curated ground-truth dataset—based on cleaning scripts from a British cohort study—the framework provides a standardized way to test whether local AI models can generate accurate R code and reliable data outputs.

Testing on Consumer Hardware

The study benchmarks various open-weight LLMs across the "consumer-grade" deployment spectrum. This is a critical step because it demonstrates that high-performance AI coding assistance does not necessarily require expensive, enterprise-level infrastructure. By running these models on standard hardware, the researchers aim to provide a practical, privacy-compliant solution for researchers who work in environments where data security is the top priority.

Promising Performance Results

The results indicate that current state-of-the-art models are highly capable of handling these research-specific tasks. When testing 31-35B parameter models, the researchers found that these systems almost saturated the benchmark, achieving an average task completion rate of up to 87.9%. These findings suggest that open-weight models are reaching a level of maturity where they can serve as effective, secure tools for automating the labor-intensive aspects of data preparation in restricted research settings.

Comments (0)

No comments yet

Be the first to share your thoughts!