The developer of Understudy stated that the tool was built to handle practical tasks spanning multiple environments, including native apps, browsers, terminals, and chat tools. It currently only supports macOS [Hacker News].
Understudy Released: A Desktop Agent That Learns Tasks from a Single Operation Demonstration
This article is a translation. Read the Japanese original
A distinctive feature is its ability to learn via demonstration. When a user performs a task once, the agent records the screen video and semantic events. It extracts intent rather than simple coordinates, converting the action into a reusable skill [Hacker News].
In the released demo, the agent is taught a sequence of actions: downloading a photo from Google Image Search, removing the background in Pixelmator Pro, and sending it via Telegram. Subsequently, the agent executes the same process for a different target [Hacker News].
This mechanism is not a simple macro. Skills save intent steps and routing options. The developer stated that when available, the agent can prioritize faster routes instead of repeating every GUI step [Hacker News].
The source code is available on GitHub and can be installed via npm [Hacker News].
Source: Show HN: Understudy – Teach a desktop agent by demonstrating a task once（HN 120pt・41コメント） (HN Search (backfill), 2026-03-13)