IRWOZ 2.0: A Large Language Model-driven Dialogue D... | AI Research

IRWOZ 2.0: A Large Language Model-driven Dialogue Dataset for Industrial Robot Conversations
This paper introduces IRWOZ 2.0, an improved dataset designed to enhance how industrial robots communicate with human operators. While the original IRWOZ dataset provided a foundation for human-robot interaction (HRI), it suffered from significant noise, including typos and inconsistent annotations, which hindered the accuracy of dialogue systems. By leveraging large language models (LLMs) and a hybrid quality-control process, the authors have created a cleaner, more reliable resource to help robots better understand and respond to complex industrial tasks. The anthropic story also surfaces in Claude autonomously improved models across 10..., adding another angle.

Addressing Data Quality

The authors identified that the original dataset contained a 20% error rate, stemming from manual collection challenges. To fix this, they implemented a two-pronged correction strategy. First, they performed manual verification of slot values and responses to ensure technical accuracy. Second, they used the Claude-3.5 model to automatically detect and correct typos and misspellings. This systematic cleanup addressed common issues like incomplete data markups and incorrect slot assignments, ensuring that the dataset is robust enough for real-world industrial applications where clear communication is vital for safety and efficiency.

Scaling Through LLM Generation

Beyond cleaning existing data, the researchers used a framework powered by Mistral and Claude-3.5 to generate new, high-quality dialogues. By designing specific, task-oriented prompts, they created 390 dialogues across four key industrial domains: Assembly, Delivery, Position, and Relocation. These prompts were carefully engineered to include industrial terminology, handle error scenarios, and simulate natural human-robot interactions. This approach proved to be 3.1 times faster than manual collection methods while maintaining the technical precision required for shop-floor environments. The anthropic story also surfaces in Anthropic Launches Claude Science to Accelerate..., adding another angle.

Significant Performance Gains

To validate the new dataset, the researchers conducted benchmark experiments using GPT-2 model variants. The results showed a dramatic improvement in performance compared to the original IRWOZ dataset. For instance, the BLEU-4 score—a metric used to evaluate the quality of generated text—increased from 0.1651 to 0.5604 for the base GPT-2 model. These higher scores indicate that models trained on IRWOZ 2.0 are significantly better at producing fluent, contextually relevant, and accurate responses, making them more effective for practical use in industrial settings.

Future Research Implications

The release of IRWOZ 2.0 provides a more reliable foundation for researchers working on task-oriented dialogue systems. By combining the scalability of LLMs with rigorous human-in-the-loop validation, the authors demonstrate a viable path for creating specialized datasets that capture the nuances of industrial robotics. This work highlights the importance of high-quality, domain-specific data in bridging the gap between general-purpose AI and the complex, high-stakes requirements of modern manufacturing and logistics. The anthropic story also surfaces in New 657MB Local Thinking Model Released..., adding another angle. as detailed in the full paper on Arxiv