Computer Use
Programmatic control of desktop environments within Daytona sandboxes.
Computer Use enables programmatic control of desktop environments within sandboxes. It provides mouse, keyboard, screenshot, screen recording, accessibility, and display operations for automating GUI interactions and testing desktop applications.
Computer Use and VNC work together to enable both manual and automated desktop interactions. VNC provides the visual interface for users to manually interact with the desktop, while Computer Use provides the programmatic API for AI agents to automate operations.
The desktop stack (D-Bus, Xvfb, the window manager, and the x11vnc server) ships with the sandbox runtime as a self-contained add-on, so it works on any Linux sandbox image, including images with no desktop packages. The virtual display runs at 1280x720. All Computer Use calls go through the toolbox API and require an API key with the sandbox write permission.
Use cases
Computer Use is used when a task needs a graphical desktop instead of an API or a shell.
- Computer-use agents: an agent reads screenshots and the accessibility tree, then moves the mouse and types to operate desktop applications.
- Desktop application tests: GUI tests click, type, and take screenshots to check the state of an application.
- Browser automation: a browser on the sandbox desktop is driven through mouse and keyboard input, for sites that have no API.
- Element-level automation: the accessibility tree finds a button or a text field by role, name, and state. An agent invokes the element or sets its text without screen coordinates.
- Screenshots for models: region and compressed screenshots capture part of the screen or a smaller image. An agent sends less image data to a model on each step.
- Window-aware automation: the window list reports the title, position, size, and active flag of each window. An agent checks which application is in front before it acts.
- Agents on Windows and macOS: Windows and macOS sandboxes provide screenshots, mouse and keyboard input, the accessibility tree, and screen recordings through the same API. On Windows, a desktop session must be open.
- Recorded sessions: screen recordings capture what an agent or a test did, for review and debugging. Recordings are MP4 files stored in the sandbox and downloaded through the API.
- Human in the loop: a person watches or takes over the same desktop through VNC while an agent works.
Start Computer Use
Start all computer use processes (D-Bus, Xvfb, the window manager, and the x11vnc server) in the sandbox. The call blocks until the X server and the VNC server accept connections and returns the status of each process. If either does not come up within 5 s, the desktop is marked failed and the call returns an error. Calling start on a desktop that is already ready returns the current status; calling it while the desktop is starting or failed returns an error.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
result = sandbox.computer_use.start()
print("Computer use processes started:", result.message)
for name, process in result.status.items():
print(f"{name}: running={process.running} pid={process.pid}")Stop Computer Use
Stop all computer use processes in the sandbox. Stopping a ready desktop returns it to not started. Stopping a failed desktop kills any remaining processes but keeps the failed state and its cause; only restart leaves the failed state.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
result = sandbox.computer_use.stop()
print("Computer use processes stopped:", result.message)Restart Computer Use
Tear the desktop down and build a new one. Restart is valid from every state and is the only way out of the failed state. Concurrent restart calls join the same in-flight restart and receive the same result. Application windows do not survive a restart.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
result = sandbox.computer_use.restart()
print(result.message, result.desktop) # computer use restarted readyGet status
Get the desktop status. The status field is active only while the desktop field is ready.
desktop | status | Meaning |
|---|---|---|
not_started | inactive | No desktop exists. Call start. |
starting | inactive | Start is in progress. Poll status. |
ready | active | Actions are accepted. |
failed | inactive | The desktop is gone. cause.code and cause.message explain why. Call restart. |
The cause object is present only when the desktop is failed. Possible cause codes:
| Code | Meaning |
|---|---|
desktop_exited | The X server (xvfb) exited. Windows in it are lost. |
addon_crashed | The add-on that supervises the desktop died while a desktop existed. |
start_failed | Creating the desktop or probing its readiness failed, for example a missing binary. |
start_interrupted | The add-on died while the desktop was being created. |
session_ended | Windows only. The interactive session the desktop was bound to ended. |
session_replaced | Windows only. A different interactive session became active. |
state_corrupt | Retained desktop state could not be read or validated. |
state_persistence_failed | A state transition could not be persisted. |
Failure evidence is boot-scoped: a sandbox reboot resets the desktop to not started.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
status = sandbox.computer_use.status()
print("Computer use status:", status.status, status.desktop)
if status.desktop == "failed":
print(f"Failure: {status.cause.code}: {status.cause.message}")Get process status
Get the status of one desktop process: D-Bus, Xvfb, the window manager, or the x11vnc server. Without a name, it returns the status of every process.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
xvfb_status = sandbox.computer_use.service.status("xvfb")
print(xvfb_status.process_name, xvfb_status.running)Get all process statuses
Get the status of every desktop process. The response is a map keyed by process name, with one entry each for D-Bus, Xvfb, the window manager, and the x11vnc server. Each entry reports whether the process is alive, its supervisor restart state, and its startup priority.
| API field | Description |
|---|---|
Running | Whether the process is alive. |
Pid | Process ID while the process is running. |
Status | running, stopped, or failed. |
Failed | Whether the process entered the terminal failed state. |
AutoRestart | Whether the supervisor restarts the process automatically. |
RestartAttempts | Restart attempts consumed by the current generation of the process. |
RestartBackoffMs | Current restart backoff in milliseconds. |
Priority | Relative startup priority within the desktop stack. |
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
for name, process in sandbox.computer_use.service.status().items():
print(f"{name}: {process.status} pid={process.pid} restarts={process.restart_attempts}")Restart process
Restart a specific desktop process. Per-process restart works only while the desktop is ready. The Xvfb process cannot be restarted on its own because application windows live inside it; use restart instead.
The D-Bus, window manager, and x11vnc processes are restarted automatically if they exit: up to 6 attempts with exponential backoff from 1 s to 30 s, reset after 30 s of uptime. When attempts run out, the process is marked failed in the status map while the desktop stays ready.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
result = sandbox.computer_use.service.restart("wm")
print(result.message, result.process_name)Get process logs
Get logs for a specific desktop process. Logs are retained up to 256 KiB per process and trimmed to the most recent 128 KiB while the process runs.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
logs = sandbox.computer_use.service.logs("x11vnc")
print("x11vnc logs:", logs.logs)The response has the following fields:
| Field | Description |
|---|---|
logs | Holds the retained standard output of the process. |
Get process errors
Get error logs for a specific desktop process.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
errors = sandbox.computer_use.service.errors("x11vnc")
print("x11vnc errors:", errors.errors)The response has the following fields:
| Field | Description |
|---|---|
errors | Holds the retained standard error output of the process. The output is capped at 256 KiB, like the logs. The value is an empty string when the process has written nothing to standard error. |
Mouse operations
Every mouse, keyboard, screenshot, display, accessibility, and recording-start call requires the desktop to be ready; otherwise it returns 409 with a desktop not ready error. Coordinates are screen pixels in the range -32768 to 32767; values outside it return 400.
Click
Click the mouse at the specified coordinates with the left, right, or middle button, as a single or a double click.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
sandbox.computer_use.mouse.click(x=100, y=200)| Parameter | Type | Description |
|---|---|---|
button | ComputerUseMouseButton | Sets the mouse button. The default is "left". |
double | bool | Makes the click a double click. The default is False. |
Move
Move the mouse cursor to the given coordinates. The response holds the final pointer position.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
result = sandbox.computer_use.mouse.move(x=100, y=200)
print(f"Mouse moved to: {result.x}, {result.y}")Drag
Drag the mouse from start to end coordinates while holding the selected button. The pointer moves in 10 interpolated steps so applications that ignore a teleporting drag still register it.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
result = sandbox.computer_use.mouse.drag(
start_x=50, start_y=50, end_x=150, end_y=150, button="left"
)
print(f"Drag ended at {result.x}, {result.y}")| Parameter | Type | Description |
|---|---|---|
button | ComputerUseMouseButton | Sets the mouse button to hold. The default is "left". |
Scroll
Scroll the mouse wheel up or down at the specified coordinates by a number of wheel ticks. One tick is one notch of a physical mouse wheel.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
result = sandbox.computer_use.mouse.scroll(x=100, y=200, direction="down")
print(result.success)Get position
Get the current mouse cursor position.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
position = sandbox.computer_use.mouse.position()
print(f"Mouse is at: {position.x}, {position.y}")Keyboard operations
Send keyboard input to the desktop. Type a text string, press one key with optional modifiers, or press a hotkey combination. Key names resolve against the keyboard map of the virtual display, and, like mouse calls, keyboard calls require the desktop to be ready.
Type
Type arbitrary text, including uppercase letters, symbols, and non-ASCII characters. Newlines are sent as Enter and tabs as Tab. Characters with no key in the X11 keymap are typed through a temporary key mapping.
Set a delay between characters to slow typing down for applications that drop fast input.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
sandbox.computer_use.keyboard.type(text="Hello, World!")Press
Press a key with optional modifiers. Modifiers are held in order, the key is pressed and released, and the modifiers are released in reverse, even if the press fails. See supported keys.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
sandbox.computer_use.keyboard.key(key="enter")Hotkey
Press a hotkey combination. The combination is a plus-separated chord string: the last entry is the key, the others are modifiers.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
sandbox.computer_use.keyboard.hotkey(keys="ctrl+c")Supported keys
Key names are case-insensitive. The following are supported for the key and modifiers parameters and for the parts of the keys parameter:
| Category | Keys |
|---|---|
| Modifiers | ctrl (alias control), alt, shift, cmd (aliases super, meta, win) |
| Editing | enter (alias return), tab, escape (alias esc), backspace, delete (alias del), space |
| Navigation | home, end, pageup (alias page_up), pagedown (alias page_down), insert, up, down, left, right |
| Function keys | f1 through f12 |
| Symbols | minus, plus, equal (alias equals) |
| Characters | Any single character, for example a, A, 5, /, - |
The SDK modifiers type accepts ctrl, alt, shift, and cmd; the API also accepts the aliases.
An unknown key or modifier returns 400.
Accessibility operations
Use accessibility operations to inspect the AT-SPI tree and interact with UI elements by node ID. Start Computer Use before calling accessibility methods.
Each node carries an ID, role, name, description, bounds (x, y, width, height), states, actions, and children. Node IDs have the form <bus-name>:<object-path> and are valid only while the owning application keeps the object alive; an unknown or malformed ID returns 404 or 400.
The scope parameter selects the root of the search:
| Scope | Root |
|---|---|
focused | The application whose state includes active, else the first registered application. Default. |
pid | The application owned by process pid. Returns 404 if no accessible application has that PID. |
all | The AT-SPI registry root, covering every application. |
Traversal is bounded: 2000 nodes and 2 s for the tree, 5000 nodes and 2 s for find. When a bound is reached, the response is marked as truncated.
Get tree
Read an accessibility tree for the focused app, a specific process, or all apps. The maximum depth parameter limits the depth below the root: 0 returns only the root, and omitting it or passing -1 returns the whole subtree within the traversal budget.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
tree = sandbox.computer_use.a11y.tree(scope="focused")
print(tree.root.role, tree.root.name, tree.truncated)| Parameter | Type | Description |
|---|---|---|
scope | ComputerUseA11yScope | Sets the root of the search: "focused", "pid", or "all". With scope="pid", also pass pid. This parameter is required. |
max_depth | int | Limits the depth below the root. Omit the parameter to get the whole subtree. |
Find nodes
Search the accessibility tree by role, accessible name, state, and scope. All filters must match.
Node states: active, armed, busy, checked, collapsed, defunct, editable, enabled, expandable, expanded, focusable, focused, has-tooltip, horizontal, iconified, modal, multi-line, multiselectable, opaque, pressed, resizable, selectable, selected, sensitive, showing, single-line, stale, transient, vertical, visible, manages-descendants, indeterminate, required, truncated, and animated.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
buttons = sandbox.computer_use.a11y.find(scope="focused", role="button", name="Submit")
for node in buttons.matches:
print(node.id, node.name, node.bounds.x, node.bounds.y)| Parameter | Type | Description |
|---|---|---|
scope | ComputerUseA11yScope | Sets the root of the search. This parameter is required. |
role | str | Matches the node role exactly, for example button or entry. |
name | str | Matches the accessible name according to name_match. |
name_match | ComputerUseA11yNameMatch | Sets how name is compared: exact, substring, or regex. The default is exact. |
states | Sequence[str] | Requires the node to have every listed state, for example ["enabled", "focusable"]. |
limit | int | Caps the number of matches. With more matches, truncated is True. The default is 50. |
Focus node
Focus a node from get tree or find nodes.
sandbox.computer_use.a11y.focus(id=node.id)Invoke node
Run a node action, such as pressing a button. Without an action name, the node's click action is invoked, falling back to its first action. An action name must appear in the node's actions list.
sandbox.computer_use.a11y.invoke(id=node.id)Set node value
Write text content to nodes that implement editable text, such as entries and text areas. The value replaces the current content.
sandbox.computer_use.a11y.set_value(id=node.id, value="hello")Screenshot operations
Screenshots return the image as a base64 string, its encoded size in bytes, and, when the cursor is requested, the pointer coordinates. The cursor is never drawn. Full-screen and region screenshots are PNG; compressed variants accept format, quality, and scale parameters.
Take full screen
Take a screenshot of the entire screen.
import base64
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
screenshot = sandbox.computer_use.screen.capture()
print(f"Screenshot size: {screenshot.size_bytes} bytes")
with open("screen.png", "wb") as f:
f.write(base64.b64decode(screenshot.screenshot))Take region
Take a screenshot of a specific region. The width and height must be positive, and the region must lie within the screen.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
screenshot = sandbox.computer_use.screen.region(x=100, y=100, width=300, height=200)
print(f"Captured region: {screenshot.size_bytes} bytes")Take compressed
Take a compressed screenshot of the entire screen.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
screenshot = sandbox.computer_use.screen.capture_compressed(format="jpeg", quality=80)
print(f"Compressed size: {screenshot.size_bytes} bytes")| Parameter | Type | Description |
|---|---|---|
format | str | Sets the image format, png or jpeg. The default is png. |
quality | int | Sets the image quality, from 1 to 100. The quality applies to jpeg. The default is 85. |
scale | float | Resizes the captured image by a factor from 0.1 to 1.0. The default is 1.0. |
show_cursor | bool | Returns the pointer coordinates with the screenshot. The cursor is never drawn. The default is False. |
Take compressed region
Take a compressed screenshot of a specific region. Accepts the region parameters of take region and the compression parameters of take compressed.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
screenshot = sandbox.computer_use.screen.region_compressed(
x=0, y=0, width=800, height=600, format="jpeg", quality=80
)
print(f"Compressed size: {screenshot.size_bytes} bytes")Screen Recording
Computer Use records desktop sessions for debugging, documentation, or automation workflows. Recordings are MP4 files captured at 10 frames per second and stored inside the sandbox; the file path field in every recording response reports the location.
Each recording has the following fields:
| Field | Description |
|---|---|
id | Recording ID, prefixed rec_. Used for stop, get, download, and delete. |
file_name | <label>-<id>.mp4. The label is reduced to letters, digits, and -, and defaults to recording. |
file_path | Path of the file inside the sandbox. |
start_time | RFC 3339 timestamp. |
end_time | RFC 3339 timestamp, set once the recording is finalized. |
status | See the status table below. |
duration_seconds | Elapsed seconds while active, or the final duration. |
size_bytes | Current file size while active, or the final size. |
| Status | Meaning |
|---|---|
recording | Capturing. |
stopping | Stop was requested and the encoder is finalizing the file. |
duration_limit_reached | The 15 minute cap was hit and the recording is being finalized. |
size_limit_reached | The 512 MiB cap was hit and the recording is being finalized. |
completed | Terminal. The file is ready to download. |
failed | Terminal. The encoder exited abnormally or produced no usable file. |
Limits:
- At most 2 recordings can be active at once; a third start returns
400. - A recording stops on its own after 15 minutes or 512 MiB, whichever comes first.
- The 20 most recent finished recordings are kept; older ones are deleted as new ones finish.
- Labels are limited to 120 characters.
Start Recording
Start a screen recording with an optional label. The desktop must be ready.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
recording = sandbox.computer_use.recording.start()
print(f"Recording started: {recording.id}")Stop Recording
Stop an active recording session by providing the recording ID. The call returns once the file is finalized, with the terminal recording.
# Stop the recording
stopped_recording = sandbox.computer_use.recording.stop(id=recording.id)
print(f"Recording stopped: {stopped_recording.duration_seconds} seconds")
print(f"Saved to: {stopped_recording.file_path}")List Recordings
Get a list of all recordings in the sandbox, active and finished:
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
recordings_list = sandbox.computer_use.recording.list()
print(f"Total recordings: {len(recordings_list.recordings)}")
for rec in recordings_list.recordings:
print(f"- {rec.file_name}: {rec.status} {rec.duration_seconds}s ({rec.size_bytes} bytes)")Get Recording
Get details about a specific recording.
recording_detail = sandbox.computer_use.recording.get("recording-id")
print(f"Recording: {recording_detail.file_name}")
print(f"Status: {recording_detail.status}")
print(f"Duration: {recording_detail.duration_seconds}s")Delete Recording
Delete a recording by ID. Deleting an active recording stops it first and then removes the file.
sandbox.computer_use.recording.delete("recording-id")
print("Recording deleted successfully")Download Recording
Download a recording file from the sandbox. The SDKs return the file content as bytes; write it to disk yourself. The daemon streams the file, so downloads are not limited by a buffer size.
from pathlib import Path
data = sandbox.computer_use.recording.download(recording.id)
Path("local_recording.mp4").write_bytes(data)
print(f"Downloaded {len(data)} bytes")Display operations
The virtual display is created when Computer Use starts and runs at 1280x720 with 24-bit color. The resolution is fixed, with no per-sandbox setting. Verify the live geometry with get info.
Get info
Get information about the displays. The response lists every X screen with its ID, position, size, and active flag. The virtual desktop has one screen.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
info = sandbox.computer_use.display.info()
print(f"Total displays: {len(info.displays)}")
for display in info.displays:
print(f"Display {display.id}: {display.width}x{display.height} at {display.x},{display.y}")Get windows
Get the list of open windows. Each window carries its ID, title, position, size, and active flag. The list is read from the client list property that the window manager maintains, which is why start refuses to run without a window manager.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()
windows = sandbox.computer_use.display.windows()
print(f"Found {len(windows.windows)} open windows:")
for window in windows.windows:
print(f"- {window.title} (ID: {window.id}, active: {window.is_active})")