Should We Type or Talk to LLM Agents? A Comprehensi... | AI Research

"Should We Type or Talk to LLM Agents? A Comprehensive Study of Voice and Keyboard Input Perturbations" investigates how common human input errors—such as typos from keyboards or transcription issues from voice—affect the performance of instruction-tuned large language models (LLMs). Researchers Zizhao Hu, Nathan Elijah Segura, Mohammad Rostami, and Jesse Thomason developed the Human Input-Variation Engine (HIVE) to simulate these real-world perturbations and measure their impact on model accuracy across various tasks.

How HIVE Simulates Input Errors

The researchers created HIVE to model two distinct channels of human input. For QWERTY keyboards, the engine simulates motor-based errors, including neighbor-key substitutions, character transpositions, duplicated letters, and missing spaces. For voice, the engine models the transcription process, which often includes disfluencies, homophone errors, and, in the case of AI-backed dictation tools, structural rewrites of the user's original speech. By applying these variations to a set of benchmarks—including arithmetic, code synthesis, and multiple-choice questions—the authors measured how different types of noise degrade model performance.

Key Findings on Input Robustness

The study reports that voice transcription perturbations consistently lower accuracy more than keyboard errors. The researchers found that the structure of the transcription—specifically how much the original wording is changed—is the primary driver of performance loss, rather than the presence of fillers like "um" or "like."
A critical discovery is that the damage caused by these perturbations is linked to the survival of the original tokens in the prompt. When an input error destroys original tokens, accuracy drops; however, adding new tokens alongside the original ones has little effect. Furthermore, the "modality gap"—the performance difference between speaking and typing—is only significant in tasks where the model must construct or deduce an answer, such as math or coding. In multiple-choice tasks, the researchers observed no meaningful performance difference between the two channels.

Limitations and Mitigation

The authors tested several methods to mitigate these issues and found that the harm cannot be easily "trained away" with lightweight adaptation. Additionally, while a "thinking budget" (allowing the model more time to process) can recover performance for keyboard-based errors, it does not improve results for spoken inputs. The study also notes that the performance drop is not solely due to test-set contamination, as the effects persisted across different versions of the benchmarks.

Why This Matters

This research suggests that developers of AI agents should be aware that voice interfaces, particularly those that use AI-based rewriting to "clean up" user speech, may inadvertently introduce structural changes that hinder model reasoning. Because the damage is tied to the destruction of original tokens, the authors indicate that the way a transcription tool reformats a user's request is a critical factor in maintaining system reliability.