What Does It Take to Detect an AI Agent? Minimal Fe...

What Does It Take to Detect an AI Agent? Minimal Fe... | AI Research

Key Takeaways

What Does It Take to Detect an AI Agent?
Minimal Feature Sets for Behavioral Detection under Browser Automation addresses the challenge of identifying AI age...
Bot detectors deployed at scale treat traffic as binary: human or bot.
This assumption breaks when AI agents browse the web through browser automation, a traffic class that is neither and that binary classifiers structurally cannot represent.
To measure evasion resistance, we construct a five-level evasion ladder spanning passive observation, GAN-generated trajectories, and replay of real human cursor data ($n = 2299$ evasion sessions).

Paper AbstractExpand

Bot detectors deployed at scale treat traffic as binary: human or bot. This assumption breaks when AI agents browse the web through browser automation, a traffic class that is neither and that binary classifiers structurally cannot represent. We present a three-class detection framework distinguishing humans, bots, and AI agents, and show that the binary-vs-agent confusion is architectural: a binary human-vs-bot detector misroutes agent sessions because its label space lacks an agent class. On our controlled benchmark, an MLP binary classifier misclassifies 39.1% of real AI agents as human and a SAINT binary transformer misclassifies 34.5%; adding an explicit agent class yields per-class agent F1 = 1.000 in all 30 runs (3 model families $\times$ 10 seeds). To measure evasion resistance, we construct a five-level evasion ladder spanning passive observation, GAN-generated trajectories, and replay of real human cursor data ($n = 2299$ evasion sessions). Across 10 seeds and 3 model families we observe zero agent misses in 22990 per-seed predictions. The discriminative signal is a browser-automation artifact, not evidence of agent reasoning: Playwright does not emit the raw pointer-move and wheel-delta streams a physical input device produces, and this absence signature survives trajectory manipulation. Exhaustive search over all feature subsets of size 1-5 (9401 GBMs) shows that two behavioral features (mouse_event_rate, teleport_click_ratio) give 100% observed agent recall at every evasion level with agent precision 0.994; five features lift macro-F1 to 0.991. The signal is redundantly encoded: removing teleport_click_ratio leaves agent detection at 100%. The single-feature regime is degenerate, flagging every agent only by collapsing the classifier to always predict "agent". Two features robustly isolate agents; five separate all three traffic classes at macro-F1 $\geq 0.99$.

What Does It Take to Detect an AI Agent? Minimal Feature Sets for Behavioral Detection under Browser Automation addresses the challenge of identifying AI agents that browse the web using automation tools like Playwright. The authors—Vishisht Choudhary, Lukas Schmidt, Anne Zoë Kenntner, Feras Skhab, Michel Osswald, and Jens Ernstberger—argue that because AI agents are neither human nor traditional bots, existing binary detection systems fail to categorize them correctly.

The Architectural Gap in Detection

Current web security systems rely on a binary classification: human or bot. The researchers show that this structure is fundamentally incapable of identifying AI agents. In their benchmark, a multi-layer perceptron (MLP) binary classifier misclassified 39.1% of AI agents as humans, while a SAINT transformer misclassified 34.5%. The authors conclude that this is an architectural failure because the label space lacks an "agent" category. By introducing a three-class framework (human, bot, agent), the researchers achieved an agent F1 score of 1.000 across all test runs, effectively eliminating the detection gap.

Detecting Automation Artifacts

The paper finds that AI agents are detectable not because of their "reasoning" or intelligence, but because of how browser automation APIs function. Tools like Playwright do not produce the raw pointer-move and wheel-delta streams generated by physical input devices. This "absence signature"—such as the lack of raw-event teleports before clicks or empty mousemove tracks—persists even when agents attempt to mimic human behavior. The researchers tested this by creating an "evasion ladder" that included GAN-generated trajectories and the replay of actual human cursor data. Across 22,990 predictions, the detector maintained 100% agent recall, proving that the automation artifacts are difficult to mask.

Minimal Feature Sets for Deployment

To make detection practical for industry use, the authors performed an exhaustive search over 9,401 gradient-boosted machine (GBM) models to find the smallest number of features required for accurate classification. They determined that just two behavioral features—mouse_event_rate and teleport_click_ratio—are sufficient to achieve 100% agent recall at every level of evasion. Using five features allows the system to distinguish between all three traffic classes with a macro-F1 score of at least 0.99.

Limitations and Considerations

The researchers note that their findings are specific to the current ecosystem of browser automation, where tools like Playwright and Puppeteer are dominant. While their three-class model is highly effective, they emphasize that the "single-feature" regime is degenerate; for example, using only cursor_path_linearity forces the classifier to predict "agent" for every session, resulting in poor overall performance. Additionally, while they hypothesized that specific features related to the "pause-think-act" cycle of LLMs would be useful, these features were ultimately removed during the backward elimination process, suggesting that the most reliable signals are the low-level execution artifacts of the automation software itself.