IntElicit: Eliciting and Assessing Contextualized C...

IntElicit: Eliciting and Assessing Contextualized C... | AI Research

Key Takeaways

IntElicit: Eliciting and Assessing Contextualized Creativity via Dialogue Policy Optimization This paper introduces IntElicit, an AI-driven framework designe...
To address these issues, this paper proposes IntElicit, a framework for eliciting and assessing contextualized creativity via dialogue policy optimization.
Specifically, to tackle sparse rewards and potential reward hacking (e.g., answer dictation) in open-ended educational dialogue, IntElicit introduces a decomposed process reward mechanism.
This mechanism aligns the policy with pedagogical elicitation, rewarding prompts that draw out participant reasoning rather than producing optimal answers on their behalf.
Extensive experiments, including participant simulation and a human subject study (N=64), show that IntElicit improves elicited creative outcomes over expert-designed baselines.

Paper AbstractExpand

Contextualized assessment offers high ecological validity for evaluating creativity but introduces a critical challenge: observed performance may be confounded with cognitive proficiency (domain knowledge) and agency (willingness to engage). Meanwhile, in the age of generative AI, creative problem solving increasingly occurs in tool-mediated and human--AI interactive environments, making fully static assessment less aligned with contemporary creative practice. To address these issues, this paper proposes IntElicit, a framework for eliciting and assessing contextualized creativity via dialogue policy optimization. IntElicit functions as a constrained adaptive AI Interviewer: it provides non-directive knowledge and agency scaffolds in multi-turn interaction to reduce non-creative confounders, while preserving participants' responsibility for generating the creative content being evaluated. Specifically, to tackle sparse rewards and potential reward hacking (e.g., answer dictation) in open-ended educational dialogue, IntElicit introduces a decomposed process reward mechanism. This mechanism aligns the policy with pedagogical elicitation, rewarding prompts that draw out participant reasoning rather than producing optimal answers on their behalf. Extensive experiments, including participant simulation and a human subject study (N=64), show that IntElicit improves elicited creative outcomes over expert-designed baselines. Together, the results suggest that interactive elicitation can reveal creative potential that static FPSP-style assessment may miss, providing a formative and diagnostic lens for contextualized creativity assessment in AI-mediated learning contexts.

IntElicit: Eliciting and Assessing Contextualized Creativity via Dialogue Policy Optimization
This paper introduces IntElicit, an AI-driven framework designed to assess human creativity in realistic, complex scenarios. Traditional creativity tests often rely on static, simple prompts that fail to capture how people solve problems in the real world. IntElicit addresses this by acting as an "AI Interviewer" that engages in multi-turn conversations with participants. By providing adaptive, non-directive support, the system helps participants express their creative potential without the AI taking over or dictating the answers, ensuring that the final creative output remains the work of the human.

The Challenge of Static Assessment

Current methods for measuring creativity, such as asking someone to list uses for a brick, often lack "ecological validity"—meaning they don't reflect how people actually think in complex, professional, or academic environments. Furthermore, when people struggle in these tests, it is often unclear if they lack creative ability or if they simply lack the necessary background knowledge, confidence, or motivation to perform well. IntElicit aims to bridge this gap by using an interactive, conversational approach that scaffolds the participant, helping them clarify problems and explore ideas while keeping the creative responsibility firmly in their hands.

How IntElicit Works

The framework uses a technique called dialogue policy optimization to train an AI to act as a helpful, yet restrained, interviewer. To prevent the AI from "reward hacking"—a common issue where an AI might simply give the user the best answer to get a high score—the researchers implemented a "decomposed process reward" mechanism. This system rewards the AI for asking questions that encourage the participant to justify their reasoning, identify problems, and reflect on alternatives. The AI is trained using simulated participants with diverse personas, allowing it to learn how to adapt its questioning style to different types of users, such as those who are reticent or those who tend to wander off-topic.

Results and Insights

The researchers tested IntElicit through both simulated interactions and a human subject study involving 64 participants. The results indicate that this interactive approach is more effective at eliciting high-quality creative outcomes than traditional, expert-designed static assessments. By dynamically adjusting to the participant's needs, the AI interviewer can reveal creative potential that might otherwise be missed in a rigid, one-off testing environment.

A New Lens for AI-Mediated Learning

The study suggests that as creative problem-solving increasingly moves into environments where humans and AI collaborate, our methods for assessment must evolve. IntElicit provides a formative and diagnostic tool that treats assessment as a conversation rather than a static test. By focusing on the process of reasoning rather than just the final product, this framework offers a more nuanced way to understand and foster creative potential in modern, AI-mediated educational contexts.