Computer Use

Computer Use enables programmatic control of desktop environments within sandboxes. It provides mouse, keyboard, screenshot, screen recording, accessibility, and display operations for automating GUI interactions and testing desktop applications.

Computer Use and VNC work together to enable both manual and automated desktop interactions. VNC provides the visual interface for users to manually interact with the desktop, while Computer Use provides the programmatic API for AI agents to automate operations.

The desktop stack (D-Bus, Xvfb, the window manager, and the x11vnc server) ships with the sandbox runtime as a self-contained add-on, so it works on any Linux sandbox image, including images with no desktop packages. The virtual display runs at 1280x720. All Computer Use calls go through the toolbox API and require an API key with the sandbox write permission or a sandbox access token with the desktop or toolbox operation.

Use cases

Computer Use is used when a task needs a graphical desktop instead of an API or a shell.

• Computer-use agents. An agent reads screenshots and the accessibility tree, then moves the mouse and types to operate desktop applications. The desktop stack ships with the sandbox runtime, so it works on any Linux sandbox image. All Computer Use calls go through the toolbox API and require an API key with the sandbox write permission or a sandbox access token with the desktop or toolbox operation.

• Desktop application tests. GUI tests click, type, and take screenshots to check the state of an application. Full-screen and region screenshots return a PNG image as a base64 string. The virtual display has a fixed resolution of 1280x720, with no setting per sandbox.

• Browser automation. A browser on the sandbox desktop is driven through mouse and keyboard input, for sites that have no API. Typing sends any text, including symbols and non-ASCII characters. A delay between characters slows typing for applications that drop fast input.

• Agents on Windows and macOS. Windows and macOS sandboxes provide screenshots, mouse and keyboard input, the accessibility tree, and screen recordings through the same API. On Windows, a desktop session must be open. When that session ends or another session replaces it, the desktop fails with a cause that names the reason.

• Human in the loop. A person watches or takes over the same desktop through VNC while an agent works. Starting Computer Use also starts the x11vnc server. The VNC server runs in shared mode, so several viewers attach to the same desktop at once.

Start Computer Use

Start all computer use processes (D-Bus, Xvfb, the window manager, and the x11vnc server) in the sandbox. The call blocks until the X server and the VNC server accept connections and returns the status of each process. If either does not come up within 5 s, the desktop is marked failed and the call returns an error. Calling start on a desktop that is already ready returns the current status. A start made while another start or a restart is in progress waits for it to finish, then returns the resulting status, or an error if the desktop failed. Calling start on a failed desktop returns an error; call restart instead.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()

result = sandbox.computer_use.start()
print("Computer use processes started:", result.message)
for name, process in result.status.items():
    print(f"{name}: running={process.running} pid={process.pid}")

Stop Computer Use

Stop all computer use processes in the sandbox. Stopping a ready desktop returns it to not started. Stopping a failed desktop kills any remaining processes but keeps the failed state and its cause; only restart leaves the failed state.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

result = sandbox.computer_use.stop()
print("Computer use processes stopped:", result.message)

Restart Computer Use

Tear the desktop down and build a new one. Restart is valid from every state and is the only way out of the failed state. Concurrent restart calls join the same in-flight restart and receive the same result. Application windows do not survive a restart.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

result = sandbox.computer_use.restart()
print(result.message, result.desktop)  # computer use restarted ready

Get status

Get the desktop status. The status field is active only while the desktop field is ready.

The cause object is present only when the desktop is failed. Possible cause codes:

Failure evidence is boot-scoped: a sandbox reboot resets the desktop to not started. The SDKs report a state or cause code they do not recognize as unknown and keep the API value in raw_status, raw_desktop, and cause.raw_code (rawStatus, rawDesktop, and cause.rawCode in TypeScript).

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()

status = sandbox.computer_use.status()
print("Computer use status:", status.status, status.desktop)
if status.desktop == "failed":
    print(f"Failure: {status.cause.code}: {status.cause.message}")

Get process status

Get the status of one desktop process: D-Bus, Xvfb, the window manager, or the x11vnc server. Without a name, it returns the status of every process.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

xvfb_status = sandbox.computer_use.service.status("xvfb")
print(xvfb_status.process_name, xvfb_status.running)

Get all process statuses

Get the status of every desktop process. The response is a map keyed by process name, with one entry each for D-Bus, Xvfb, the window manager, and the x11vnc server while a desktop exists. The map is empty before the first start, after stop, after a failed start, and after the add-on restarts. Each entry reports whether the process is alive, its supervisor restart state, and its startup priority.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

for name, process in sandbox.computer_use.service.status().items():
    print(f"{name}: {process.status} pid={process.pid} restarts={process.restart_attempts}")

Restart process

Restart a specific desktop process. Per-process restart works only while the desktop is ready. The Xvfb process cannot be restarted on its own because application windows live inside it; use restart instead.

The D-Bus, window manager, and x11vnc processes are restarted automatically when they exit. A process that exits within 30 s of starting is restarted at most 5 times in a row, after waits of 1, 2, 4, 8, and 16 s. The sixth such exit marks it failed in the status map, while the desktop stays ready. A run that lasts 30 s or longer resets the count. Restarting a failed process resets its count and starts it again.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

result = sandbox.computer_use.service.restart("wm")
print(result.message, result.process_name)

Get process logs

Get logs for a specific desktop process. Each log is checked every 5 s and whenever the process starts; once it exceeds 256 KiB, it is cut to its most recent 128 KiB. Logs accumulate across process and desktop restarts. Logs and errors are readable only while a desktop exists, including after Xvfb exits.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

logs = sandbox.computer_use.service.logs("x11vnc")
print("x11vnc logs:", logs.logs)

Get process errors

Get error logs for a specific desktop process.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

errors = sandbox.computer_use.service.errors("x11vnc")
print("x11vnc errors:", errors.errors)

Mouse operations

Every mouse, keyboard, screenshot, display, accessibility, and recording-start call requires the desktop to be ready; otherwise it returns 409 with a desktop not ready error. Coordinates are screen pixels in the range -32768 to 32767; values outside it return 400.

Click

Click the mouse at the specified coordinates with the left, right, or middle button, as a single or a double click.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

sandbox.computer_use.mouse.click(x=100, y=200)

Move

Move the mouse cursor to the given coordinates. The response holds the final pointer position.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

result = sandbox.computer_use.mouse.move(x=100, y=200)
print(f"Mouse moved to: {result.x}, {result.y}")

Drag

Drag the mouse from start to end coordinates while holding the selected button. The pointer moves in 10 interpolated steps so applications that ignore a teleporting drag still register it.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

result = sandbox.computer_use.mouse.drag(
    start_x=50, start_y=50, end_x=150, end_y=150, button="left"
)
print(f"Drag ended at {result.x}, {result.y}")

Scroll

Scroll the mouse wheel up or down at the specified coordinates by a number of wheel ticks. One tick is one notch of a physical mouse wheel.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

result = sandbox.computer_use.mouse.scroll(x=100, y=200, direction="down")
print(result.success)

Get position

Get the current mouse cursor position.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

position = sandbox.computer_use.mouse.position()
print(f"Mouse is at: {position.x}, {position.y}")

Keyboard operations

Send keyboard input to the desktop. Type a text string, press one key with optional modifiers, or press a hotkey combination. Key names resolve against the keyboard map of the virtual display, and, like mouse calls, keyboard calls require the desktop to be ready.

Type

Type arbitrary text, including uppercase letters, symbols, and non-ASCII characters. Newlines are sent as Enter and tabs as Tab. Characters with no key in the X11 keymap are typed through a temporary key mapping.

Set a delay between characters to slow typing down for applications that drop fast input.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

sandbox.computer_use.keyboard.type(text="Hello, World!")

Press

Press a key with optional modifiers. Modifiers are held in order, the key is pressed and released, and the modifiers are released in reverse, even if the press fails. See supported keys.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

sandbox.computer_use.keyboard.key(key="enter")

Hotkey

Press a hotkey combination. The combination is a plus-separated chord string: the last entry is the key, the others are modifiers.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

sandbox.computer_use.keyboard.hotkey(keys="ctrl+c")

Supported keys

Named keys and aliases are case-insensitive. A single character keeps its case: A presses Shift and a. The following are supported for the key and modifiers parameters and for the parts of the keys parameter:

The SDK modifiers type lists ctrl, alt, shift, and cmd; the API accepts those, their aliases, and any other supported key name.

Accessibility operations

Use accessibility operations to inspect the AT-SPI tree and interact with UI elements by node ID. Start Computer Use before calling accessibility methods.

Each node carries an ID, role, name, bounds (x, y, width, height in screen pixels), and states. The description, actions, and children appear only when not empty, and find matches never include children. A node without the AT-SPI component interface reports zero bounds. The role is the AT-SPI role name, passed through unchanged. Node states outside the list in find nodes appear as state:<n>, where <n> is the AT-SPI state number. On Linux, the value field is never set.

A node ID joins the bus name and the object path of the element with a colon, and is valid only while the owning application keeps the object alive. A malformed ID returns 400. An unknown ID returns 404 from focus and invoke, and 400 from set value.

The scope parameter selects the root of the search:

Traversal is bounded: 2000 nodes and 2 s for the tree, 5000 nodes and 2 s for find. When a bound is reached, the response is marked as truncated. Cutting the tree at the maximum depth does not mark it as truncated.

Get tree

Read an accessibility tree for the focused app, a specific process, or all apps. The maximum depth parameter limits the depth below the root, and 0 returns only the root. The SDKs return at most 256 levels. The API returns the whole subtree within the traversal budget when the parameter is omitted or -1.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

tree = sandbox.computer_use.a11y.tree(scope="focused")
print(tree.root.role, tree.root.name, tree.truncated)

Find nodes

Search the accessibility tree by role, accessible name, state, and scope. All filters must match. Name matching is case-sensitive. The regex mode uses the syntax of the Rust regex-lite crate and matches anywhere in the name unless the pattern is anchored with ^ and $; an invalid pattern returns 400. Find visits nodes depth first, last child first, so the first match is not necessarily the first element on screen.

Node states: invalid, active, armed, busy, checked, collapsed, defunct, editable, enabled, expandable, expanded, focusable, focused, has-tooltip, horizontal, iconified, modal, multi-line, multiselectable, opaque, pressed, resizable, selectable, selected, sensitive, showing, single-line, stale, transient, vertical, visible, manages-descendants, indeterminate, required, truncated, and animated.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

buttons = sandbox.computer_use.a11y.find(scope="focused", role="button", name="Submit")
for node in buttons.matches:
    print(node.id, node.name, node.bounds.x, node.bounds.y)

Focus node

Focus a node from get tree or find nodes. The call asks the application to give the node focus; if the application declines, the call returns 400.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

node = sandbox.computer_use.a11y.find(scope="focused", role="button", name="Submit").matches[0]
sandbox.computer_use.a11y.focus(id=node.id)

Invoke node

Run a node action, such as pressing a button. Without an action name, the node's click action is invoked, falling back to its first action. An action name must appear in the node's actions list and is case-sensitive. A node with an empty actions list, or an action the application rejects, returns 400. A node without the AT-SPI action interface returns 404.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

node = sandbox.computer_use.a11y.find(scope="focused", role="button", name="Submit").matches[0]
sandbox.computer_use.a11y.invoke(id=node.id)

Set node value

Write text content to nodes that implement editable text, such as entries and text areas. The value replaces the current content.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

node = sandbox.computer_use.a11y.find(scope="focused", role="entry").matches[0]
sandbox.computer_use.a11y.set_value(id=node.id, value="hello")

Screenshot operations

Screenshots return the image as a base64 string, its encoded size in bytes, and, when the cursor is requested, the pointer coordinates. The cursor is never drawn. Full-screen and region screenshots are PNG; compressed variants accept format, quality, and scale parameters. The SDKs take every variant with one screenshot method: a region selects the region variant, and setting the format, quality, or scale selects the compressed variant. The show cursor parameter works with every variant and does not select compression.

Take full screen

Take a screenshot of the entire screen.

import base64

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

screenshot = sandbox.computer_use.screen.screenshot()
print(f"Screenshot size: {screenshot.size_bytes} bytes")
with open("screen.png", "wb") as f:
    f.write(base64.b64decode(screenshot.screenshot))

Take region

Take a screenshot of a specific region. The width and height must be positive, and the region must lie within the screen.

from daytona import Daytona
from daytona.v2 import ComputerUseRegion

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

screenshot = sandbox.computer_use.screen.screenshot(
    region=ComputerUseRegion(x=100, y=100, width=300, height=200)
)
print(f"Captured region: {screenshot.size_bytes} bytes")

Take compressed

Take a compressed screenshot of the entire screen.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

screenshot = sandbox.computer_use.screen.screenshot(format="jpeg", quality=80)
print(f"Compressed size: {screenshot.size_bytes} bytes")

Take compressed region

Take a compressed screenshot of a specific region. Accepts the region parameters of take region and the compression parameters of take compressed.

from daytona import Daytona
from daytona.v2 import ComputerUseRegion

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

screenshot = sandbox.computer_use.screen.screenshot(
    region=ComputerUseRegion(x=0, y=0, width=800, height=600), format="jpeg", quality=80
)
print(f"Compressed size: {screenshot.size_bytes} bytes")

Screen Recording

Computer Use records desktop sessions for debugging, documentation, or automation workflows. Recordings are MP4 files captured at 10 frames per second and stored inside the sandbox; the file path field in every recording response reports the location. Video is encoded with MPEG-4 Part 2, not H.264, so some browsers cannot play it inline.

Each recording has the following fields. TypeScript and the API use the camelCase names; Python uses the snake_case names.

Limits:

• At most 2 recordings can be in progress at once, counting recordings that are recording, stopping, or finishing after the 15-minute or 512 MiB cap; a third start returns 400.
• A recording stops on its own after 15 minutes or 512 MiB, whichever comes first.
• Up to 20 finished recordings are kept, including failed and interrupted ones; when a stop or list call finds more, the oldest are deleted with their files.
• Labels longer than 120 characters are truncated to 120.

Start Recording

Start a screen recording with an optional label. The desktop must be ready.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

recording = sandbox.computer_use.recording.start()
print(f"Recording started: {recording.id}")

Stop Recording

Stop an active recording session by providing the recording ID. The call returns once the file is finalized, with the terminal recording.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
recording = sandbox.computer_use.recording.start()

stopped_recording = sandbox.computer_use.recording.stop(id=recording.id)
print(f"Recording stopped: {stopped_recording.duration_seconds} seconds")
print(f"Saved to: {stopped_recording.file_path}")

List Recordings

Get a list of all recordings in the sandbox, active and finished:

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

recordings_list = sandbox.computer_use.recording.list()
print(f"Total recordings: {len(recordings_list.recordings)}")
for rec in recordings_list.recordings:
    print(f"- {rec.file_name}: {rec.status} {rec.duration_seconds}s ({rec.size_bytes} bytes)")

Get Recording

Get details about a specific recording.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
recording = sandbox.computer_use.recording.start()

recording_detail = sandbox.computer_use.recording.get(recording.id)
print(f"Recording: {recording_detail.file_name}")
print(f"Status: {recording_detail.status}")
print(f"Duration: {recording_detail.duration_seconds}s")

Delete Recording

Delete a recording by ID. Deleting an active recording stops it first and then removes the file.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
recording = sandbox.computer_use.recording.start()

sandbox.computer_use.recording.delete(recording.id)
print("Recording deleted successfully")

Download Recording

Download a recording file from the sandbox. The daemon streams the file. The SDKs hold the whole file in memory and return it as bytes; write it to disk yourself. An active recording can be downloaded, and the response holds the file written so far.

from pathlib import Path

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
recording = sandbox.computer_use.recording.start()
sandbox.computer_use.recording.stop(id=recording.id)

data = sandbox.computer_use.recording.download(recording.id)
Path("local_recording.mp4").write_bytes(data)
print(f"Downloaded {len(data)} bytes")

Display operations

The virtual display is created when Computer Use starts and runs at 1280x720 with 24-bit color. The resolution is fixed, with no per-sandbox setting. Verify the live geometry with get info.

Get info

Get information about the displays. The response lists every X screen with its ID, position, size, and active flag. The virtual desktop has one screen.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

info = sandbox.computer_use.display.info()
print(f"Total displays: {len(info.displays)}")
for display in info.displays:
    print(f"Display {display.id}: {display.width}x{display.height} at {display.x},{display.y}")

Get windows

Get the list of open windows. Each window carries its ID, title, position, size, and active flag. The list is read from the client list property that the window manager maintains, which is why start refuses to run without a window manager.

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

windows = sandbox.computer_use.display.windows()
print(f"Found {len(windows.windows)} open windows:")
for window in windows.windows:
    print(f"- {window.title} (ID: {window.id}, active: {window.is_active})")

Errors

The SDKs raise these errors with the codes in the following table.

Permissions

Computer Use runs through the toolbox of a sandbox. Access is governed by one project permission, which you can grant to an organization role or an API key.

A sandbox access token with the desktop or toolbox operation grants the same access to a single sandbox, without an API key. The sandbox read permission alone does not grant toolbox access.