Phone-Use: connect AI agents to Android and iPhone

What Phone-Use does

Phone-Use connects an AI agent to an Android phone or iPhone so it can inspect the screen and operate apps. It exposes screen reading, taps, typing, scrolling, app launching and screenshots through an MCP server and a command-line interface. The project README documents connections for Claude Code, Codex, Claude Desktop, Cursor and other clients supporting MCP over standard input and output.

The agent receives numbered on-screen elements with information such as text, role and position. It can act on an element number or use coordinates when appropriate. Actions return an updated screen, which helps the agent inspect what happened before taking another step. Screenshots provide a visual path for content that a structured element list cannot explain well.

Android and iPhone use different control paths

On Android, an installed app uses an accessibility service to read visible elements and perform gestures. You enable that service yourself in the phone's settings. Screenshots require a separate screen-sharing approval on the device. USB setup uses Android's debugging tools to connect the computer to the phone.

On iPhone, a computer runs the connection through WebDriverAgent and go-ios over USB. The iPhone needs Developer Mode and a signed WebDriverAgent installation. The iPhone guide explains signing, installation and the host process that must remain running. This preparation is a material part of setup, especially if you have not previously installed a device test runner.

Both platforms expose the same tool names, but individual behavior differs. For example, the iPhone implementation uses an edge swipe for Back, refuses the notifications action, and uses iOS points for coordinates. Check the platform guide before assuming an Android interaction transfers directly to iPhone.

Setup and a first task

The documented prerequisites include Node.js on the computer and a USB data cable. Android also needs platform-tools and an eligible device; iPhone needs the additional developer tooling above. Follow the current release and setup instructions in the README, then register the MCP server with your agent client. The CLI also provides diagnostic and screen-reading commands for checking the connection before adding an agent.

A useful first task is to read Settings and report a visible device detail. It exercises screen access and navigation while giving you an obvious way to check the answer. Use one controlling agent per phone: element numbers refer to the latest screen read, so another controller can invalidate the context.

Consent, stopping and network access

Android displays a Stop control while the agent reads or acts. Pausing blocks screen reads and actions, ends screen sharing and survives app restarts. Resuming requires someone at the phone. On iPhone, locking the device pauses access; unlocking it locally resumes it. Phone-Use provides no remote resume on either platform.

USB is the initial connection route, with optional network access through Tailscale or another encrypted tunnel. The project's HTTP API has token authentication but no TLS, and its documentation says to keep its ports off the public internet. Screenshots are returned on request without being stored on the phone, though the connected agent may retain them in conversation history.

Compatibility limits and Franklin's take

The testing notes describe specific Android emulator and iPhone checks. They also identify gaps, including physical-phone Tailscale testing and iPhone unplug/replug behavior. Treat that evidence as a starting point for checking your own device, rather than assuming universal app and hardware compatibility.

Phone-Use is a practical project to evaluate when an agent needs to work through a mobile interface. Start with a spare device and a short, observable task. Confirm screen reading, one action and the on-device Stop behavior before attempting a longer workflow. After an action times out, inspect the screen before retrying because the action may already have occurred.