Towards an Agent-First Web: Redesigning the Web for...

Towards an Agent-First Web: Redesigning the Web for... | AI Research

Key Takeaways

The World Wide Web was built on the assumption that its primary users are humans.
This design choice influences everything from how websites are built to how...
The World Wide Web was built on an assumption held for three decades: the primary consumer of web content is a human being.
This permeates every layer; its access model presumes human visitors, its economics rest on human attention, and its content targets human perception.
The rapid emergence of AI agents as intermediaries between humans and web content invalidates this assumption.

Paper AbstractExpand

The World Wide Web was built on an assumption held for three decades: the primary consumer of web content is a human being. This permeates every layer; its access model presumes human visitors, its economics rest on human attention, and its content targets human perception. The rapid emergence of AI agents as intermediaries between humans and web content invalidates this assumption. Yet the web resists agents through blanket blocking, CAPTCHA-based exclusion, and economic models that treat agent access as extraction rather than legitimate interaction. This paper proposes a principled redesign across three layers. At the access layer, agents acting for humans should inherit equivalent access rights, governed by rate limiting and agent identification metadata in HTTP requests, analogous to browser headers, alongside a dual-layer architecture serving human-readable and agent-optimized content from the same domain. At the economic layer, we propose an intent-based tier framework grounded in the agent-as-human-proxy principle: an agent's economic obligation mirrors that of the human it represents. A token-based subscription model meters content in tokens rather than pageviews, alongside a commissioned content economy anchoring AI content production in human intentionality. At the content layer, we identify epistemic recursion, the self-referential loop in which AI-generated content is consumed by agents to produce further content, progressively detaching web knowledge from human ground truth. We propose the Agent Text Markup Language (ATML), a four-level human supervision tier model, and a cryptographic provenance chain to counter this threat. Together these constitute ten design principles for an agent-first internet, one in which agents are first-class citizens whose integration requires renegotiating the web's foundational social contract across access, economics, and content.

The World Wide Web was built on the assumption that its primary users are humans. This design choice influences everything from how websites are built to how they make money and protect their content. However, the rise of AI agents—which browse, synthesize, and act on behalf of humans—has rendered this human-centric model outdated. The paper "Towards an Agent-First Web: Redesigning the Web for AI Agents" argues that the web is currently failing to accommodate these agents, leading to a breakdown in the digital social contract. The authors propose a comprehensive redesign across three layers—access, economics, and content—to transform the web into an environment where AI agents are treated as legitimate, first-class citizens.

Redesigning Access and Identity

Currently, the web treats AI agents as intruders, often blocking them with CAPTCHAs or blanket bans. The authors propose moving away from this "block-by-default" approach toward a system based on transparency and intent. By introducing standardized agent identification metadata—similar to how browsers currently identify themselves—agents could declare who they are and which human they represent. This would allow website owners to use rate limiting instead of total exclusion. Additionally, the paper suggests a "dual-layer" architecture where sites provide both human-readable and agent-optimized versions of their content, ensuring that both types of visitors can access information efficiently.

A New Economic Model for Agents

The current economic model of the web relies on human attention and advertising, which does not translate well to AI agents that consume content without viewing ads. The authors propose the "agent-as-human-proxy" principle, which suggests that an agent’s economic obligations should mirror the human it represents. Instead of pageviews, the paper suggests a token-based subscription model that aligns with how AI systems already function. Furthermore, they propose a "commissioned content economy," where human intentionality guides AI content production, ensuring that the value exchange between publishers and AI developers remains fair and sustainable.

Preventing Epistemic Recursion

A major concern identified in the paper is "epistemic recursion"—a cycle where AI agents consume AI-generated content to create even more content, eventually detaching web knowledge from human truth. To combat this, the authors introduce the Agent Text Markup Language (ATML), a semantic format designed for machine consumption. They also propose a four-level human supervision model and a cryptographic provenance chain. These tools would make it clear to both agents and users how much human oversight was involved in creating a piece of content, helping to preserve the integrity of information on the web.

A Framework for the Future

The authors synthesize these ideas into ten design principles aimed at renegotiating the web's social contract. Rather than treating AI as a threat to be managed through reactive patches, the framework encourages a proactive, architectural shift. By addressing access, economics, and content simultaneously, the authors aim to build an internet that supports the next generation of AI-driven interaction while maintaining the foundational values of the web.