Back to AI Research

AI Research

Sophistication in GenAI Use: Field Evidence from a... | AI Research

Key Takeaways

  • This paper examines how back-office employees at KPMG LLP use generative AI (genAI) in their daily work, specifically focusing on the "sophistication" of the...
  • We study how sophistication in generative AI (genAI) use varies among the back-office workforce of a large firm.
  • Using proprietary data, we observe 713,564 employee prompts and their corresponding large language model responses from nearly 4,000 back-office employees across 15 functional areas over eight months in 2025.
  • First, senior employees exhibit more sophisticated genAI use, consistent with domain expertise complementing genAI capabilities.
  • Third, we observe neither improvements in sophistication over time nor lasting improvements following formal AI training, suggesting that sophisticated use can be difficult to change.
Paper AbstractExpand

We study how sophistication in generative AI (genAI) use varies among the back-office workforce of a large firm. Using proprietary data, we observe 713,564 employee prompts and their corresponding large language model responses from nearly 4,000 back-office employees across 15 functional areas over eight months in 2025. We document three main findings. First, senior employees exhibit more sophisticated genAI use, consistent with domain expertise complementing genAI capabilities. Second, sophistication varies considerably across functions and is highest in Strategy, Digital Innovation, and Project Management, three groups that share a focus on firmwide strategic initiatives and organizational change. Third, we observe neither improvements in sophistication over time nor lasting improvements following formal AI training, suggesting that sophisticated use can be difficult to change. Together, our study provides measures of and insights into sophisticated genAI use that managers can use to improve outcomes and that researchers can use in future research.

This paper examines how back-office employees at KPMG LLP use generative AI (genAI) in their daily work, specifically focusing on the "sophistication" of their interactions. While many studies track whether employees adopt AI, this research investigates how well they use it by analyzing over 700,000 prompts and responses from nearly 4,000 employees across 15 functional areas during 2025.

Defining and Measuring Sophistication

The authors define sophisticated genAI use as a combination of clear, specific instructions, the application of deliberate prompting techniques (such as role assignment or step-by-step reasoning), and the use of the tool across a diverse range of tasks. To measure this, the researchers used an LLM to analyze conversation transcripts, scoring them on clarity, structure, and the use of advanced prompting strategies. They also developed three composite metrics: prompt clarity, deliberate strategy use, and use-case diversity.

Key Findings on Employee Usage

The study identifies three primary trends regarding how employees engage with genAI:

  • Seniority Matters: More senior employees demonstrate higher levels of sophistication than staff-level employees, suggesting that domain expertise helps workers better leverage AI capabilities.

  • Functional Differences: Sophistication varies significantly by department. Employees in Strategy, Digital Innovation, and Project Management—groups focused on organizational change—show the highest levels of sophisticated use. Conversely, the Accounting and Finance functions consistently rank lower on these measures.

  • Stagnation Over Time: Despite widespread adoption, the researchers found no evidence that employees became more sophisticated in their AI use over the eight-month study period. Furthermore, while formal AI training led to a temporary increase in sophistication during the month of completion, this improvement did not persist in subsequent months.

Practical Implications for Management

The authors suggest that managers can gauge AI sophistication without needing to read full conversation transcripts by looking at metadata. The study found that "ambition" (longer initial prompts) and "persistence" (more iteration within a conversation) are reliable indicators of more sophisticated use. This provides a low-cost method for organizations to monitor how effectively their workforce is utilizing AI tools.

Research Limitations

The study is based on data from a single large professional services firm, which may limit the generalizability of the findings to other industries. Additionally, the researchers note that their composite measures serve as proxies for sophisticated behavior rather than direct measurements of output quality or productivity. Finally, the study does not establish that formal training is ineffective; rather, it highlights that such training, in its current form, does not appear to produce lasting changes in employee behavior.

Comments (0)

No comments yet

Be the first to share your thoughts!