Back to AI Research

AI Research

Detecting LLM-Generated Tokens in Human--LLM Coauth... | AI Research

Key Takeaways

  • Detecting LLM-Generated Tokens in Human--LLM Coauthored Text As human-AI collaboration becomes common, documents often contain a mix of human-written and mac...
  • The rise of human-AI collaborative writing has created a growing need for fine-grained detection methods that support localizing likely LLM-generated content in mixed-authorship documents.
  • Existing methods for detecting LLM-generated text mainly focus on document-level classification and cannot identify which parts of the text are generated by LLMs.
  • This paper introduces a new method to address this urgent need.
  • Our method operates at the token level, the natural unit of modern language models, and builds on existing token-level detection scores.
Paper AbstractExpand

The rise of human-AI collaborative writing has created a growing need for fine-grained detection methods that support localizing likely LLM-generated content in mixed-authorship documents. Existing methods for detecting LLM-generated text mainly focus on document-level classification and cannot identify which parts of the text are generated by LLMs. This paper introduces a new method to address this urgent need. Our method operates at the token level, the natural unit of modern language models, and builds on existing token-level detection scores. The key idea is to smooth adjacent token scores to reduce their variability, while using an adaptive Lepski-type rule to select the bandwidth according to the local authorship structure. Our method is simple to implement and does not require token-level labeled data for training. Theoretically, we characterize this trade-off and show that the proposed method achieves favorable mean square error performance in estimating the underlying signal. Empirically, we demonstrate strong performance of our method against a wide range of baselines in both synthetic datasets and a realistic dataset. We deploy a publicly accessible website that implements the methods as well.

Detecting LLM-Generated Tokens in Human--LLM Coauthored Text
As human-AI collaboration becomes common, documents often contain a mix of human-written and machine-generated text. Existing detection tools are generally limited to classifying an entire document as "AI" or "human," which fails to identify specific segments or sentences written by an LLM. This paper introduces a new, fine-grained method that operates at the individual token level, allowing for precise localization of AI-generated content without requiring the document to be split into predefined units like sentences or paragraphs.

How the Method Works

The researchers developed a technique that treats each token as a data point and applies a "smoothing" process to reduce the noise inherent in raw token scores. By looking at a token alongside its neighbors, the method creates a more stable estimate of whether that specific piece of text was generated by an LLM.
A key component of this approach is an adaptive "bandwidth" selection rule based on Lepski’s method. This rule automatically determines how many neighboring tokens to consider when evaluating a specific spot in the text. It balances the bias-variance trade-off: smaller windows are better at catching sharp transitions between human and AI authorship, while larger windows provide more stability by reducing random noise.

Key Advantages

This approach offers several practical benefits over current detection strategies:

  • No Training Required: Unlike machine learning-based detectors, this method does not require large, expensive datasets of labeled tokens to "learn" how to identify AI text. It works by repurposing existing token-level detection scores.

  • Flexible Localization: Because it does not rely on fixed segments like sentences or paragraphs, the method can identify authorship changes that happen mid-sentence or in irregular patterns.

  • Accessibility: The authors have deployed a public website where users can test the method on their own text, providing a practical tool for researchers and editors.

Performance and Results

The researchers evaluated their method across four different language models and various human-AI collaboration patterns. Their experiments, conducted on both synthetic and real-world datasets, demonstrate that this approach improves the accuracy of token-level authorship ranking compared to existing baselines. In a case study using an abstract co-written by a human and an LLM, the tool achieved 94% accuracy in predicting the authorship of individual tokens. The authors also provide theoretical guarantees, showing that their adaptive bandwidth selection achieves favorable performance in estimating the underlying authorship signal.

Comments (0)

No comments yet

Be the first to share your thoughts!