Desktop, viewer and human takeover

Desktop operations use display pixels, not screenshot pixels. Read the current display and convert image coordinates with the returned CE-Scale-X and CE-Scale-Y values. With scale 0.5, each image pixel represents two display pixels; cropped coordinates also need the crop origin applied by the application.

Display and images

Set one of 1024×768, 1280×800, 1440×900 or 1920×1080. No default display size is specified here. DPI is read-only; attempting to set it is unprocessable. Every desktop route returns desktop_disabled on a headless computer. Screenshot PNG/JPEG/WebP responses are bytes; JSON base64 fields remain encoded strings.

Actions

Submit at most 20 actions in one call. click, move, drag, scroll, type, key, key_down, key_up, mouse_down, mouse_up, hold_key and wait have separate validated shapes. Select exactly one of app or URL when launching. Batches exceeding the 120-second scheduled-duration bound are rejected before an action. type delay_ms is at most 1,000 per character; hold_key and wait are each bounded at 10 seconds.

Scroll uses notches only at launch. Positive horizontal values move right and positive vertical values move down; the resulting pixels depend on the application. Pixel scrolling is rejected. Keys are canonical names or valid modifier chords; use type for arbitrary Unicode. A literal plus key is named plus. Invalid chords fail before the batch executes.

Held input and people

Temporary chord/button inputs are released in a finally path. Explicit downs have a ten-second safety lease; matching downs renew it. Shutdown, pause/stop and authority expiry/revocation release API-held inputs. RFB input has its own socket lifetime. A connected exclusive human holder causes human_in_control unless force was explicitly authorized; the help workflow must not use force.

Session creation returns credentials, listing returns metadata. Follow Human takeover and RFB streaming for connected-control readback. Recording and accessibility-tree routes are not available at launch and return not_found.

Exact input vocabulary and cleanup

• click {x, y, button: left|right|middle|back|forward, count: 1|2|3, modifiers?}
• move {x, y}; drag {path:[{x,y},…], button?, modifiers?} (two points = simple drag)
• mouse_down {button} / mouse_up {button}
• scroll {x, y, dx, dy, unit: notches (default; the only launch value), modifiers?}
• type {text, delay_ms?}; key {keys: "ctrl+l"}; key_down {key} / key_up {key}; hold_key {key, ms ≤ 10000}
• wait {ms ≤ 10000}
• modifiers: any of shift, ctrl, alt, super.

Scroll moves the pointer to (x,y), then emits horizontal notches before vertical notches: positive dx means right, negative dx left, positive dy down, negative dy up. One notch is one X11 wheel-button press/release: up=4, down=5, left=6, right=7; the resulting pixel distance is application-dependent. Zero means no notches on that axis. Pixel scrolling is not supported at launch and unit:pixels returns 422 unprocessable. Drag defaults to left button and no modifiers; move to the first point, press the button, visit every subsequent point in order, then release.

Canonical key names are lowercase a-z, 0-9, f1-f12, enter, tab, escape, backspace, delete, insert, home, end, pageup, pagedown, left, right, up, down, space, plus, minus, equal, comma, period, slash, backslash, semicolon, apostrophe, grave, bracketleft, bracketright, shift, ctrl, alt and super. Names are ASCII case-insensitive; aliases return→enter, esc→escape, control→ctrl and meta→super are accepted; no other aliases are accepted. key.keys is a single canonical key or zero or more distinct modifiers (shift/ctrl/alt/super) separated by '+' followed by one non-modifier key, e.g. ctrl+l or ctrl+plus. No whitespace or escape syntax is accepted; use plus for the '+' key and type for arbitrary Unicode text. key_down/key_up/hold_key accept a single key, not a chord. Invalid keys/chords return 422 unprocessable before any action in the batch executes. Key presses the chord and releases only inputs it acquired, in reverse order; click/scroll/drag/hold_key similarly release their temporary inputs in a finally block. Explicit key_down/mouse_down state persists until matching up, but has a 10-second safety lease renewed only by another down for that same input; expiry releases it. Release API-held inputs on agent/desktop shutdown, pause/stop and token revocation/expiry; a short HTTP request closing is not an input-session close. RFB-held inputs are tracked separately and released when their control socket closes. Quickstart ctrl+l holds Control while pressing lowercase l; Return and Enter both normalize to enter. These spellings are identical in raw REST, SDKs, CLI action files and adapters; no client silently rewrites an unsupported key to another key.

Human takeover: while a control session with exclusive:true is connected, /desktop/actions returns 409 human_in_control unless force:true. Non-exclusive control sessions share input.

Check actions and PNG bytes

Set CE_COMPUTER_ID to a running desktop computer reserved for this example. The program moves the pointer and presses Escape, checks the actual executed count, reads a PNG and verifies its bytes through the file API. It removes only the unique guest file it created.

import os
import uuid
from computeengines import Client

with Client() as client:
    computer = client.computers.get(os.environ["CE_COMPUTER_ID"])
    screen = computer.desktop.display()
    result = computer.desktop.actions([
        {"type": "move", "x": screen.width // 2, "y": screen.height // 2},
        {"type": "key", "keys": "escape"},
    ])
    if result.executed != 2:
        raise RuntimeError("Desktop batch was incomplete")
    png = computer.desktop.screenshot(format="png")
    if not png.startswith(b"\x89PNG\r\n\x1a\n"):
        raise RuntimeError("Screenshot was not PNG")
    path = "/home/dev/docs-screen-" + uuid.uuid4().hex + ".png"
    try:
        computer.files.write(path, png)
        if computer.files.read(path) != png:
            raise RuntimeError("PNG file roundtrip changed bytes")
        print("Desktop actions and PNG byte roundtrip verified")
    finally:
        computer.files.remove(path)

Dory is in development; these pages describe the planned service and may change. Questions: support@computeengines.com.