How AI Assistance Affects Human Skill Development:... | AI Research

How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles investigates whether using AI to solve problems hinders a person's ability to develop the skills necessary to solve those same problems independently later. Researchers Shang Wu, Catarina G. Belem, Shuyuan Fu, Mark Steyvers, and Padhraic Smyth conducted a controlled experiment to determine if AI assistance acts as a helpful scaffold or a crutch that replaces the cognitive effort required for learning.

The Three-Phase Experiment

The researchers designed a study where 124 participants solved logic puzzles across three distinct phases. In Phase 1, participants completed puzzles without any AI help to establish a baseline. In Phase 2, participants were randomly assigned to one of three groups: a control group with no AI, a low-cost AI group, and a high-cost AI group. In the AI-enabled groups, participants could request the location of a puzzle object at a specific point cost. Finally, in Phase 3, all participants returned to solving puzzles without AI assistance to measure their skill retention and growth.

How AI Cost Influences Behavior

The study found that the cost of AI access directly dictates how often people rely on it. Participants in the low-cost condition requested AI help significantly more often than those in the high-cost condition. Furthermore, the researchers observed that relying on AI during the middle phase created a false sense of competence. When predicting a participant's future performance based on their AI-assisted work, the model consistently overestimated how well they would perform once the AI was removed. Those who used AI during the middle phase performed worse in the final, unassisted phase compared to those who did not use AI.

The Role of Independent Reasoning

Using a Bayesian latent ability model, the researchers separated initial ability from skill gains. The analysis revealed that the frequency of AI requests was less important to learning than the amount of "solo share"—the fraction of time a participant spent working on a puzzle independently before asking for help. Greater independent problem-solving effort was associated with larger gains in latent ability. This suggests that the negative impact on skill development occurs when AI assistance substitutes for the cognitive work of reasoning, rather than when it is used as a supplementary tool.

Implications for AI-Assisted Learning

The findings indicate that AI assistance can improve short-term task performance while simultaneously weakening long-term skill acquisition. The authors conclude that the primary concern for educational and training environments is not the use of AI itself, but whether the interaction design encourages users to bypass the cognitive processes required to build expertise. To support effective learning, AI systems should be designed to complement, rather than displace, the independent reasoning efforts of the user.