Computer Use

tensorlake/ubuntu-vnc is a managed desktop image for browser automation and computer-use agents. It boots XFCE, TigerVNC, and Firefox for you, and the SDK connects through the authenticated sandbox proxy so you can drive the desktop without manually exposing port 5901. This guide builds on Sandboxes. If you already have a Tensorlake API key, you can create a desktop sandbox, capture screenshots, and send mouse and keyboard input in just a few lines.

Prerequisites

Python
JavaScript

pip install tensorlake
export TENSORLAKE_API_KEY=your-api-key

npm install tensorlake
export TENSORLAKE_API_KEY=your-api-key

The current managed tensorlake/ubuntu-vnc image uses tensorlake as its VNC password.

Launch a Desktop Sandbox

Use tensorlake/ubuntu-vnc when you want a full Linux desktop instead of a shell-only environment.

Python
JavaScript

from tensorlake.sandbox import Sandbox


sandbox = Sandbox.create(image="tensorlake/ubuntu-vnc")
    print(sandbox.sandbox_id)

import { Sandbox } from "tensorlake";


const sandbox = await Sandbox.create({
  image: "tensorlake/ubuntu-vnc",
});

try {
  console.log(sandbox.sandboxId);
} finally {
  await sandbox.terminate();
  client.close();
}

You still get a normal Sandbox object back, so computer use fits naturally alongside run(), file operations, PTY sessions, snapshots, and tunnels.

Capture Screenshots

Once the sandbox is running, attach to the desktop and save a PNG. This is the easiest way to inspect the layout and discover click coordinates before sending pointer events. Fresh desktop sandboxes can take a few seconds to finish starting XFCE and other desktop services. If your first screenshot is blank or input does not land where you expect, wait briefly after connecting and then retry.

Python
JavaScript

import time
from pathlib import Path
from tensorlake.sandbox import Sandbox


sandbox = Sandbox.create(image="tensorlake/ubuntu-vnc")
    with sandbox.connect_desktop(password="tensorlake") as desktop:
        time.sleep(4.0)
        screenshot = desktop.screenshot()
        Path("sandbox-desktop.png").write_bytes(screenshot)

        print(desktop.width, desktop.height)

import { writeFile } from "node:fs/promises";
import { Sandbox } from "tensorlake";


const sandbox = await Sandbox.create({
  image: "tensorlake/ubuntu-vnc",
});

try {
  const desktop = await sandbox.connectDesktop({
    password: "tensorlake",
  });

  try {
    await new Promise((resolve) => setTimeout(resolve, 4000));
    const screenshot = await desktop.screenshot();
    await writeFile("sandbox-desktop.png", screenshot);

    console.log(desktop.width, desktop.height);
  } finally {
    await desktop.close();
  }
} finally {
  await sandbox.terminate();
  client.close();
}

Send Keyboard and Mouse Input

The desktop client supports keyboard shortcuts, typed input, clicks, double-clicks, mouse movement, and scrolling. The example below uses a reliable keyboard-driven flow: open a terminal, type a command, and then verify the result from the sandbox shell.

Python
JavaScript

import time
from tensorlake.sandbox import Sandbox


sandbox = Sandbox.create(image="tensorlake/ubuntu-vnc")
    with sandbox.connect_desktop(password="tensorlake") as desktop:
        desktop.press(["ctrl", "alt", "t"])
        time.sleep(4.0)

        desktop.type_text("echo docs-test > /tmp/desktop-test.txt")
        desktop.press("enter")
        time.sleep(3.0)

        # Mouse helpers are also available when you know the coordinates.
        desktop.move_mouse(640, 400)
        desktop.scroll_down()

    result = sandbox.run("bash", ["-lc", "cat /tmp/desktop-test.txt"])
    print(result.stdout.strip())  # docs-test

import { Sandbox } from "tensorlake";


const sandbox = await Sandbox.create({
  image: "tensorlake/ubuntu-vnc",
});

try {
  const desktop = await sandbox.connectDesktop({
    password: "tensorlake",
  });

  try {
    await desktop.press(["ctrl", "alt", "t"]);
    await new Promise((resolve) => setTimeout(resolve, 4000));

    await desktop.typeText("echo docs-test > /tmp/desktop-test.txt");
    await desktop.press("enter");
    await new Promise((resolve) => setTimeout(resolve, 3000));

    // Mouse helpers are also available when you know the coordinates.
    await desktop.moveMouse(640, 400);
    await desktop.scrollDown();
  } finally {
    await desktop.close();
  }

  const result = await sandbox.run("bash", {
    args: ["-lc", "cat /tmp/desktop-test.txt"],
  });
  console.log(result.stdout.trim()); // docs-test
} finally {
  await sandbox.terminate();
  client.close();
}

Coordinate-based actions are screen-relative. A common workflow is:

Take a screenshot.
Inspect the desktop layout and note the coordinates you care about.
Use move_mouse() / moveMouse(), click(), double_click() / doubleClick(), and scroll() with those coordinates.

Reconnect to an Existing Sandbox

If a sandbox is already running, connect by sandbox ID and attach to the desktop without creating a new VM.

Python
JavaScript

from pathlib import Path
from tensorlake.sandbox import Sandbox

sandbox_id = "your-running-sandbox-id"


with Sandbox.connect(sandbox_id) as sandbox:
    with sandbox.connect_desktop(password="tensorlake") as desktop:
        Path("existing-sandbox.png").write_bytes(desktop.screenshot())

import { writeFile } from "node:fs/promises";
import { Sandbox } from "tensorlake";


const sandbox = Sandbox.connect("your-running-sandbox-id");

try {
  const desktop = await sandbox.connectDesktop({
    password: "tensorlake",
  });

  try {
    const screenshot = await desktop.screenshot();
    await writeFile("existing-sandbox.png", screenshot);
  } finally {
    await desktop.close();
  }
} finally {
  sandbox.close();
  client.close();
}

Connecting to an existing sandbox only closes the client connection when you are done. It does not terminate the running VM.

Use noVNC in the Browser

If you want a human to interact with the sandbox desktop in real time, use a real VNC client in the browser instead of polling screenshots. noVNC is a good fit here. The recommended architecture is:

Keep the Tensorlake API key on your backend.
Use the backend to open a TCP tunnel to the sandbox’s VNC port 5901.
Bridge that local tunnel to a browser WebSocket endpoint such as /vnc/<session-id>.
Point noVNC at your backend WebSocket and authenticate with the desktop password tensorlake.

This keeps sandbox proxy authentication server-side and gives the browser a low-latency live desktop stream. You do not need to expose port 5901 publicly yourself. If you are also running an agent loop, a good pattern is to use:

noVNC for the live human-facing desktop stream
sandbox.connectDesktop() for screenshots and high-level computer-use actions on the backend

That separation avoids turning the browser view into a screenshot polling loop.

Browser Client with noVNC

Install noVNC in your frontend:

npm install @novnc/novnc

Then connect the browser to your own WebSocket bridge:

import RFB from "@novnc/novnc/lib/rfb";

const host = document.getElementById("desktop");
if (!(host instanceof HTMLDivElement)) {
  throw new Error("Missing #desktop container");
}

const protocol = window.location.protocol === "https:" ? "wss:" : "ws:";
const url = `${protocol}//${window.location.host}/vnc`;

const rfb = new RFB(host, url, {
  credentials: { password: "tensorlake" },
  shared: true,
});

rfb.scaleViewport = true;
rfb.clipViewport = false;
rfb.showDotCursor = true;

Use a fixed-size container for the desktop surface:

<div id="desktop" style="width: 1200px; height: 800px; background: black;"></div>

Desktop API Surface

Python uses snake_case, while JavaScript uses camelCase, but both SDKs expose the same core capabilities:

Screenshots: screenshot()
Mouse input: move_mouse() / moveMouse(), mouse_press() / mousePress(), mouse_release() / mouseRelease(), click(), double_click() / doubleClick(), scroll(), scroll_up() / scrollUp(), and scroll_down() / scrollDown()
Keyboard input: key_down() / keyDown(), key_up() / keyUp(), press(), and type_text() / typeText()
Desktop size: width and height

connect_desktop() and connectDesktop() go through the authenticated sandbox proxy, so you do not need to bind or expose the VNC port yourself. If you need manual access for debugging, you can still create a TCP tunnel to port 5901.

tl sbx tunnel <sandbox_id> 5901 --listen-port 15901

After the tunnel is up, you can connect with any VNC client using localhost:15901.

Overview

Guides

Use Cases

Prerequisites

Launch a Desktop Sandbox

Capture Screenshots

Send Keyboard and Mouse Input

Reconnect to an Existing Sandbox

Use noVNC in the Browser

Browser Client with noVNC

Desktop API Surface

Overview

Guides

Use Cases

​Prerequisites

​Launch a Desktop Sandbox

​Capture Screenshots

​Send Keyboard and Mouse Input

​Reconnect to an Existing Sandbox

​Use noVNC in the Browser

​Browser Client with noVNC

​Desktop API Surface

Prerequisites

Launch a Desktop Sandbox

Capture Screenshots

Send Keyboard and Mouse Input

Reconnect to an Existing Sandbox

Use noVNC in the Browser

Browser Client with noVNC

Desktop API Surface