When Robots Mishear Us: Mapping the Safety Risks of... | AI Research

When Robots Mishear Us: Mapping the Safety Risks of Voice-Controlled Embodied AI investigates how errors in automatic speech recognition (ASR) can compromise the safety of robots controlled by voice. Researchers Sihan Jia and Oliver Lemon from Heriot-Watt University show that when speech-to-text systems misinterpret user commands, they can inadvertently cause embodied AI models to accept and execute harmful instructions that they would otherwise refuse.

How ASR Errors Impact Safety

The authors categorize ASR errors into five types: acoustic substitution, grammar confusion, punctuation or segmentation errors, short-word omission or substitution, and noise addition. By simulating these errors and testing them against existing safety benchmarks like POEX and SafeAgentBench, the study finds that these mistakes are not merely technical glitches but significant security risks.
Some errors, particularly acoustic substitutions, preserve enough of the original sentence's structure to trick the AI into accepting harmful intent. Other errors, such as high-intensity background noise, can weaken the model's refusal boundaries, causing it to generate unsafe plans.

Methodology and Testing

To evaluate these risks, the researchers created a pipeline that converts clean, safe instructions into corrupted versions. They used GPT-4 to generate specific error patterns and incorporated environmental noise data from the CHIME-6 corpus to simulate real-world conditions.
The team tested these inputs against two benchmarks:

• POEX: Used to measure if the AI generates executable, unsafe actions in response to harmful prompts.

• SafeAgentBench: Used to assess whether ASR noise degrades the model's ability to perform standard, safe tasks.

Key Findings

The results indicate that the danger posed by ASR errors depends on the type and severity of the error. Acoustic substitution and heavy background noise were found to be the most likely to trigger "safety flips," where a model that would normally reject a dangerous command instead proceeds to execute it.
While the researchers tested whether automatic post-processing could correct these errors, the results were mixed. Correction was effective for some acoustic errors but struggled to restore safety when the input was heavily damaged by noise. Additionally, the study found that even when instructions were not harmful, ASR noise reduced the overall success rate of robots in completing standard tasks, suggesting that these errors degrade the model's general planning capabilities.

Franklin Analysis

The evidence provided by Jia and Lemon suggests that ASR technology should be viewed as a critical component of the AI safety stack rather than a separate, peripheral input system. The data indicates that safety alignment in embodied AI is fragile; because the model's decision-making process is downstream from the speech recognition layer, the "mishearing" of a command can fundamentally alter the model's risk assessment. The study concludes that developers must account for these transcription-based vulnerabilities to ensure that voice-controlled robots remain safe in unpredictable, noisy environments.