Scrawl: hand a coding agent annotated screen evidence with matching narration

Scrawl helps someone show a coding agent an interface problem by talking, clicking and drawing over the screen. The output combines captured frames with narration, giving the agent a more concrete handoff than a written description of where a button or visual defect appears.

Capture the moments that carry meaning

The Scrawl website says it takes a frame when the user draws, clicks, says “look” or “now,” or when the screen changes. It pairs sentences with the frame they describe and crops frames to the window involved in the interaction.

That event-based approach focuses the handoff on moments the person highlighted. It is worth checking whether the resulting sequence preserves enough context to explain the problem, particularly when an issue depends on a transition between states rather than one static frame.

Keep narration connected to the evidence

Scrawl describes speech transcription on the user’s computer. Its agent workflow includes reading a contact sheet and transcript, opening relevant keyframes and requesting a denser grid around a moment identified by time or a spoken phrase.

The closer-look mechanism is useful when the first capture leaves an ambiguous detail. A good handoff should let the agent inspect the evidence supporting the user’s explanation, rather than require it to guess what happened between two images. This guide describes the published workflow without claiming an independent capture test.

Check the platform and capture route

The site lists a Mac release for Apple silicon on macOS 15 or later, a Windows x64 unsigned preview and Linux as forthcoming. It also describes mirrored iPhone capture and a ScrawlKit route for recording inside an iOS app.

For a first evaluation, record one small, reproducible interface issue and inspect the paste before handing it to an agent. Confirm that the relevant window, narration and key moment are present, and remove unrelated information from the capture. The tool’s value is the clarity of that evidence handoff.