Back to AI Research

AI Research

Do LLM-Generated Skills Make Better AI Data Scienti... | AI Research

Key Takeaways

  • Do LLM-Generated Skills Make Better AI Data Scientists?
  • A Component Ablation Across Data-Science Workflows This paper investigates whether "reusable skills"—...
  • Product data scientists often ask LLM-based agents to help with recurring execution tasks such as cleaning data, writing SQL, choosing statistical tests, and formatting results.
  • Reusable skill files are meant to avoid prompting from scratch by packaging guidance for a task family.
  • Expert-written skills can encode high-quality guidance, but writing and maintaining them across many data-science task families creates a manual bottleneck.
Paper AbstractExpand

Product data scientists often ask LLM-based agents to help with recurring execution tasks such as cleaning data, writing SQL, choosing statistical tests, and formatting results. Reusable skill files are meant to avoid prompting from scratch by packaging guidance for a task family. Expert-written skills can encode high-quality guidance, but writing and maintaining them across many data-science task families creates a manual bottleneck. We ask whether LLM-generated skills offer a useful low-curation alternative: do they improve performance over the task prompt alone? We test this question across four lifecycle stages: data preparation, data extraction, statistical analysis, and reporting, using one generated skill per stage. We find no reliable improvement from full generated skills over No-Skill prompting. We then ask whether any part of the skill is useful by ablating different skill components. The main ablation covers 56 tasks, nine model configurations, and three providers, yielding 7,560 runs. Compared with prompting using the task alone, neither the full generated skill nor any ablated skill variant significantly improves performance; all p-values are at least 0.396, and the total spread across variants is only 1.2 pp. A supplemental token-matched control adds 1,512 runs and finds that Full skills perform similarly to task-irrelevant skill-formatted content. The results caution against using one LLM-generated skill per data-science workflow as a default single-shot prompting strategy.

Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows
This paper investigates whether "reusable skills"—pre-written guidance files designed to help AI agents perform recurring data science tasks—actually improve performance when they are generated by an LLM rather than a human. While expert-written skills can provide high-quality instructions, they are time-consuming to create and maintain. The author tests whether a "low-curation" alternative, where an LLM generates these skill files automatically, provides any measurable benefit over simply prompting the AI with the task alone.

Testing the Low-Curation Approach

The study evaluates four common data science lifecycle stages: data preparation, data extraction (SQL), statistical analysis, and reporting. The researcher generated one "skill" file for each stage using an LLM and tested them across 56 different tasks. To understand if specific parts of these files were helpful, the study used an ablation method, systematically removing sections like "worked examples" or "reference notes" to see if any individual component improved the AI's success rate. The experiment involved 7,560 individual runs across nine different AI model configurations from three major providers.

The Results: No Reliable Improvement

The findings suggest that LLM-generated skills are not an effective default strategy for data science agents. Across all configurations, the study found no statistically significant improvement in performance when using these skills compared to "No-Skill" prompting (where the AI is given only the task instructions). The performance difference between using a full skill file and using no skill at all was negligible, with a total spread of only 1.2 percentage points. Furthermore, the study found that the AI performed similarly when provided with "task-irrelevant" content of the same length, suggesting that the model was not gaining meaningful guidance from the generated data science instructions.

Why Skills May Not Be Helping

The research highlights a significant challenge for current agent-based workflows. Because modern LLMs are already highly capable of handling many standard data science tasks, they may not require the additional, static guidance provided by these skill files. In some cases, the generic instructions within a generated skill might even conflict with the specific requirements of a task. The study concludes that simply prepending a flat, LLM-generated document to a prompt is not a reliable way to boost performance, and it cautions developers against relying on this strategy as a "one-size-fits-all" solution for automating data science workflows.

Key Takeaways

The results serve as a warning for those building AI data science agents. The study demonstrates that adding more context—in the form of generated skill files—does not automatically lead to better outcomes and actually increases the number of tokens processed without a corresponding gain in accuracy. Future efforts in this space may need to move beyond simple, flat prompt injection and toward more sophisticated methods, such as selective loading of information, system-level instruction hierarchies, or task-specific validation, rather than relying on low-curation, reusable skill files.

Comments (0)

No comments yet

Be the first to share your thoughts!