# Tensorlake skills
Source: https://docs.tensorlake.ai/agent-skills
Tensorlake skills teach your coding agents to build production workflows with TensorLake's Sandbox and Orchestration SDKs.
Instead of treating TensorLake as just another API, **Tensorlake Skills** teach agents how to use TensorLake as infrastructure: coordinate workflows with the Orchestration SDK, run tasks in isolated environments with the Sandbox SDK, and compose reliable agent systems for production use.
## What You Can Build
Use it when you want your coding agent to build:
* Multi-agent applications with an orchestrator and specialist sub-agents
* Sandboxed coding or execution workflows
* Agent teams with separate workspaces
* Long-running or stateful agent systems
* Production-ready orchestration patterns
## What the Skill Does
It guides agents to:
* Use the **Orchestration SDK** for workflow logic and multi-agent coordination
* Use the **Sandbox SDK** for isolated code execution and real agent workspaces
* Combine both SDKs to build production-style agent systems
* Choose TensorLake patterns that are better than a single-agent or stateless approach
Works with any LLM provider (OpenAI, Anthropic) and any agent framework (LangChain, CrewAI, LlamaIndex). TensorLake is the infrastructure layer. Bring your own models and frameworks.
## Supported Agents
| Agent | File | How to Install |
| ------------------------------------------------------------- | ----------- | ---------------------------------------- |
| [Claude Code](https://docs.anthropic.com/en/docs/claude-code) | `SKILL.md` | [Claude Code installation](#claude-code) |
| [Google ADK](https://google.github.io/adk-docs/skills/) | `SKILL.md` | [Google ADK installation](#google-adk) |
| [OpenAI Codex](https://openai.com/index/codex/) | `AGENTS.md` | [Codex installation](#openai-codex) |
## Installation
### Any Agent
```bash theme={null}
npx skills add tensorlakeai/tensorlake-skills
```
### Claude Code
Clone the repo and copy the skill into your project's `.claude/skills/` directory:
```bash theme={null}
git clone https://github.com/tensorlakeai/tensorlake-skills /tmp/tensorlake-skills
mkdir -p .claude/skills/tensorlake
cp -r /tmp/tensorlake-skills/SKILL.md /tmp/tensorlake-skills/references .claude/skills/tensorlake/
rm -rf /tmp/tensorlake-skills
```
Or for global access across all projects:
```bash theme={null}
git clone https://github.com/tensorlakeai/tensorlake-skills /tmp/tensorlake-skills
mkdir -p ~/.claude/skills/tensorlake
cp -r /tmp/tensorlake-skills/SKILL.md /tmp/tensorlake-skills/references ~/.claude/skills/tensorlake/
rm -rf /tmp/tensorlake-skills
```
### Google ADK
Install the skill by adding the `SKILL.md` file to your ADK agent's skill directory. See the [Google ADK skills documentation](https://google.github.io/adk-docs/skills/) for details.
### OpenAI Codex
Install the skill by adding the `AGENTS.md` file to your Codex agent configuration. See the [OpenAI Codex documentation](https://openai.com/index/codex/) for details.
Works with Claude Code, Cursor, Cline, GitHub Copilot, Windsurf, and more via [skills.sh](https://skills.sh).
## Setup
TensorLake requires a `TENSORLAKE_API_KEY` configured in the local environment.
1. Get an API key at [cloud.tensorlake.ai](https://cloud.tensorlake.ai)
2. Run `tl login` or configure the variable through your shell profile, `.env` file, or secret manager
Do not paste API keys into chat, commit them to source control, or print them in terminal output.
## The Skill Triggers Automatically
The skill activates when you ask the agent to:
* Build agentic workflows or multi-agent pipelines
* Run LLM-generated code in a secure sandbox
* Orchestrate complex multi-step AI applications
* Integrate TensorLake with any LLM, framework, database, or API
* Ask questions about TensorLake APIs or documentation
## Source
The skill is open source and available on GitHub: [tensorlakeai/tensorlake-skills](https://github.com/tensorlakeai/tensorlake-skills)
# Root
Source: https://docs.tensorlake.ai/api-reference/root
get /
# Introduction
Source: https://docs.tensorlake.ai/api-reference/v2/introduction
Tensorlake API Reference
## Sandbox APIs
The Tensorlake Sandbox API lets you create isolated runtimes, inspect their state, update public ingress settings, snapshot running sandboxes, restore new sandboxes from snapshots, and suspend or resume them.
Launch a sandbox for the current project.
Capture the current filesystem and memory state of a running sandbox.
Start a new sandbox from a previously created snapshot.
## Sandbox Runtime APIs
The sandbox proxy also exposes runtime endpoints for each running sandbox. These requests are routed through the sandbox proxy to the daemon inside the sandbox.
Start processes, inspect status, send signals, write stdin, and stream output from a sandbox.
Check sandbox daemon health and inspect runtime process counts.
Create interactive terminal sessions and attach over WebSocket.
Read, write, delete, and list files through the sandbox proxy.
Relay raw TCP bytes to a sandbox-local port over WebSocket.
Inspect or manage the in-sandbox SSH daemon used by Tensorlake's SSH proxy.
# Close Process Stdin
Source: https://docs.tensorlake.ai/api-reference/v2/processes/close-stdin
post /api/v1/processes/{pid}/stdin/close
Close a process stdin pipe and deliver EOF to the process.
Close a process's stdin pipe and deliver EOF.
The process must be started with `stdin_mode: "pipe"` for this endpoint to work.
## Endpoint
```http theme={null}
POST /api/v1/processes/{pid}/stdin/close
```
## Example Request
```bash theme={null}
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes/42/stdin/close \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `204 No Content` when stdin is closed.
If stdin was not opened in `pipe` mode, Tensorlake returns `400 Bad Request`. Missing processes return `404 Not Found`.
# Follow Process Output
Source: https://docs.tensorlake.ai/api-reference/v2/processes/follow-output
get /api/v1/processes/{pid}/output/follow
Replay captured output and follow live combined output over Server-Sent Events.
Replay captured output and follow new output over Server-Sent Events.
## Endpoint
```http theme={null}
GET /api/v1/processes/{pid}/output/follow
```
Use `curl -N` or an SSE client so the connection stays open.
## Example Request
```bash theme={null}
curl -N https://.sandbox.tensorlake.ai/api/v1/processes/42/output/follow \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## SSE Events
Tensorlake first replays any captured output, then streams live events:
```text theme={null}
event: output
data: {"line":"Processing item 1/10","timestamp":1710000000000,"stream":"stdout"}
event: output
data: {"line":"Processing item 2/10","timestamp":1710000001000,"stream":"stdout"}
event: eof
data: {}
```
`/output/follow` follows combined output and includes `stream` set to `stdout` or `stderr`.
If you need a single stream, use [Follow Process Stdout](/api-reference/v2/processes/follow-stdout) or [Follow Process Stderr](/api-reference/v2/processes/follow-stderr). When the process exits and the output stream closes, Tensorlake sends `event: eof`.
# Follow Process Stderr
Source: https://docs.tensorlake.ai/api-reference/v2/processes/follow-stderr
get /api/v1/processes/{pid}/stderr/follow
Replay captured stderr and follow live stderr over Server-Sent Events.
Replay captured stderr and follow new stderr over Server-Sent Events.
## Endpoint
```http theme={null}
GET /api/v1/processes/{pid}/stderr/follow
```
Use `curl -N` or an SSE client so the connection stays open.
## Example Request
```bash theme={null}
curl -N https://.sandbox.tensorlake.ai/api/v1/processes/42/stderr/follow \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## SSE Events
Tensorlake first replays any captured stderr, then streams live events:
```text theme={null}
event: output
data: {"line":"Traceback (most recent call last):","timestamp":0}
event: output
data: {"line":"ValueError: invalid input","timestamp":1710000001000}
event: eof
data: {}
```
Stderr-only events do not include a `stream` field. If you need merged stdout and stderr with stream tags, use [Follow Process Output](/api-reference/v2/processes/follow-output).
# Follow Process Stdout
Source: https://docs.tensorlake.ai/api-reference/v2/processes/follow-stdout
get /api/v1/processes/{pid}/stdout/follow
Replay captured stdout and follow live stdout over Server-Sent Events.
Replay captured stdout and follow new stdout over Server-Sent Events.
## Endpoint
```http theme={null}
GET /api/v1/processes/{pid}/stdout/follow
```
Use `curl -N` or an SSE client so the connection stays open.
## Example Request
```bash theme={null}
curl -N https://.sandbox.tensorlake.ai/api/v1/processes/42/stdout/follow \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## SSE Events
Tensorlake first replays any captured stdout, then streams live events:
```text theme={null}
event: output
data: {"line":"Processing item 1/10","timestamp":0}
event: output
data: {"line":"Processing item 2/10","timestamp":1710000001000}
event: eof
data: {}
```
Stdout-only events do not include a `stream` field. If you need merged stdout and stderr with stream tags, use [Follow Process Output](/api-reference/v2/processes/follow-output).
# Get Process
Source: https://docs.tensorlake.ai/api-reference/v2/processes/get
get /api/v1/processes/{pid}
Retrieve process metadata and current status for a sandbox process.
Get the current status and metadata for one process.
## Endpoint
```http theme={null}
GET /api/v1/processes/{pid}
```
## Example Request
```bash theme={null}
curl https://.sandbox.tensorlake.ai/api/v1/processes/42 \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK` with the process metadata:
```json theme={null}
{
"handle": 1,
"pid": 42,
"status": "running",
"exit_code": null,
"signal": null,
"stdin_writable": false,
"command": "python",
"args": ["-m", "http.server", "8080"],
"started_at": 1710000000000,
"ended_at": null
}
```
`handle` is a daemon-local stable identifier. Route paths use the operating-system `pid`.
If the process does not exist, Tensorlake returns `404 Not Found`.
# Sandbox Processes API Overview
Source: https://docs.tensorlake.ai/api-reference/v2/processes/introduction
Manage sandbox processes through the sandbox proxy.
The sandbox process API is exposed through each sandbox's management hostname, derived from the sandbox's `ingress_endpoint`, not `https://api.tensorlake.ai`.
```text theme={null}
https://.sandbox.tensorlake.ai
```
These endpoints are proxied through the sandbox proxy to the daemon running inside the sandbox. Use them to start background processes, inspect status, send signals, write stdin, and retrieve or follow captured output.
Include `Authorization: Bearer $TENSORLAKE_API_KEY` on requests to the sandbox proxy.
Launch a new process inside a sandbox.
Start a non-interactive process and stream output until exit.
Enumerate the processes tracked by the sandbox daemon.
Inspect the current status and metadata for one process.
Deliver a POSIX signal such as `SIGTERM` or `SIGKILL`.
Write to stdin or close stdin for a running process.
Read captured stdout, stderr, or combined output.
Replay existing output and follow new output over SSE.
Force-terminate a running process with `SIGKILL`.
# Kill Process
Source: https://docs.tensorlake.ai/api-reference/v2/processes/kill
delete /api/v1/processes/{pid}
Force-terminate a process with `SIGKILL`.
Kill a running process with `SIGKILL`.
## Endpoint
```http theme={null}
DELETE /api/v1/processes/{pid}
```
## Example Request
```bash theme={null}
curl -X DELETE https://.sandbox.tensorlake.ai/api/v1/processes/42 \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `204 No Content` when the process is terminated.
If you want a graceful shutdown first, use [Send Signal](/api-reference/v2/processes/signal) with `15` before falling back to `SIGKILL`.
# List Processes
Source: https://docs.tensorlake.ai/api-reference/v2/processes/list
get /api/v1/processes
List the processes tracked inside a sandbox through the sandbox proxy.
List the processes tracked inside a sandbox.
## Endpoint
```http theme={null}
GET /api/v1/processes
```
## Example Request
```bash theme={null}
curl https://.sandbox.tensorlake.ai/api/v1/processes \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK`:
```json theme={null}
{
"processes": [
{
"handle": 1,
"pid": 42,
"status": "running",
"exit_code": null,
"signal": null,
"stdin_writable": false,
"command": "python",
"args": ["-m", "http.server", "8080"],
"started_at": 1710000000000,
"ended_at": null
}
]
}
```
`status` is one of `running`, `exited`, `signaled`, or `oom_killed`. `handle` is a daemon-local stable identifier; route paths use the operating-system `pid`.
# Get Process Output
Source: https://docs.tensorlake.ai/api-reference/v2/processes/output
get /api/v1/processes/{pid}/output
Read the captured combined output for a process.
Read output that has already been captured for a process.
## Endpoint
```http theme={null}
GET /api/v1/processes/{pid}/output
```
## Example Request
```bash theme={null}
curl https://.sandbox.tensorlake.ai/api/v1/processes/42/output \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK`:
```json theme={null}
{
"pid": 42,
"lines": [
"Serving HTTP on 0.0.0.0 port 8080",
"127.0.0.1 - - [06/Apr/2026 22:20:00] \"GET / HTTP/1.1\" 200 -"
],
"line_count": 2
}
```
`/output` returns the combined captured lines from stdout and stderr and does not include per-line stream tags.
If you need a single stream, use [Get Process Stdout](/api-reference/v2/processes/stdout) or [Get Process Stderr](/api-reference/v2/processes/stderr). If you need stream-tagged live events, use [Follow Process Output](/api-reference/v2/processes/follow-output).
# Run Process
Source: https://docs.tensorlake.ai/api-reference/v2/processes/run
post /api/v1/processes/run
Start a non-interactive process, stream captured output over Server-Sent Events, and emit a final exit event. Stdin is closed for this endpoint.
Start a process, stream its captured output over Server-Sent Events, and receive a final exit event.
Use this endpoint when you want a single request for non-interactive command execution. If the process needs stdin, use [Start Process](/api-reference/v2/processes/start), [Process Stdin](/api-reference/v2/processes/stdin), and [Close Process Stdin](/api-reference/v2/processes/close-stdin) instead.
## Endpoint
```http theme={null}
POST /api/v1/processes/run
```
Use this endpoint on the sandbox proxy host:
```text theme={null}
https://.sandbox.tensorlake.ai/api/v1/processes/run
```
## Example Request
```bash theme={null}
curl -N -X POST https://.sandbox.tensorlake.ai/api/v1/processes/run \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"command": "python",
"args": ["-c", "import os; print(os.environ[\"MODE\"])"],
"env": {"MODE": "batch"},
"working_dir": "/workspace",
"timeout": 30
}'
```
## Request Body
```json theme={null}
{
"command": "python",
"args": ["-c", "print('hello')"],
"env": {"MODE": "batch"},
"working_dir": "/workspace",
"user": "tl-user",
"timeout": 30
}
```
* `command` is required.
* `args` defaults to `[]`.
* `env` defaults to `{}` and is applied to the process environment.
* `working_dir` is optional.
* `user` is optional and accepts a username, UID string, `uid:gid` string, or an object such as `{"uid": 1000, "gid": 1000}`.
* `timeout` is optional. When set, Tensorlake kills the process if it is still running after that many seconds.
`/processes/run` always starts the process with stdin closed and stdout/stderr captured, even if `stdin_mode`, `stdout_mode`, or `stderr_mode` are present in the request body.
## SSE Events
Tensorlake streams JSON payloads in SSE `data:` frames:
```text theme={null}
data: {"handle":1,"pid":42,"started_at":1710000000000}
data: {"line":"hello","timestamp":1710000000010,"stream":"stdout"}
data: {"exit_code":0}
```
The first event contains `handle`, `pid`, and `started_at`. Output events contain `line`, `timestamp`, and `stream`. The final event contains `exit_code` or `signal`; if the kernel OOM killer ended the process, it includes `signal: 9` and `oom_killed: true`.
# Send Signal
Source: https://docs.tensorlake.ai/api-reference/v2/processes/signal
post /api/v1/processes/{pid}/signal
Send a POSIX signal such as `SIGTERM` or `SIGKILL` to a running process.
Send a POSIX signal to a running process.
## Endpoint
```http theme={null}
POST /api/v1/processes/{pid}/signal
```
## Example Request
```bash theme={null}
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes/42/signal \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"signal": 15}'
```
## Request Body
```json theme={null}
{
"signal": 15
}
```
Common values include `15` for `SIGTERM` and `9` for `SIGKILL`.
## Response
Tensorlake returns `200 OK`:
```json theme={null}
{
"success": true
}
```
If the process does not exist, Tensorlake returns `404 Not Found`. Invalid signals or signals sent to non-running processes return `400 Bad Request`.
# Start Process
Source: https://docs.tensorlake.ai/api-reference/v2/processes/start
post /api/v1/processes
Start a new process through the sandbox proxy, for example on `https://.sandbox.tensorlake.ai`.
Start a new process inside a sandbox.
## Endpoint
```http theme={null}
POST /api/v1/processes
```
Use this endpoint on the sandbox proxy host:
```text theme={null}
https://.sandbox.tensorlake.ai/api/v1/processes
```
## Example Request
```bash theme={null}
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"command": "python",
"args": ["-m", "http.server", "8080"],
"env": {"PORT": "8080"},
"working_dir": "/workspace",
"user": "tl-user",
"stdin_mode": "pipe",
"stdout_mode": "capture",
"stderr_mode": "capture"
}'
```
## Request Body
```json theme={null}
{
"command": "python",
"args": ["-m", "http.server", "8080"],
"env": {"PORT": "8080"},
"working_dir": "/workspace",
"user": "tl-user",
"stdin_mode": "pipe",
"stdout_mode": "capture",
"stderr_mode": "capture"
}
```
* `command` is required.
* `args` defaults to `[]`.
* `env` defaults to `{}` and is applied to the process environment.
* `working_dir` is optional.
* `user` is optional and accepts a username, UID string, `uid:gid` string, or an object such as `{"uid": 1000, "gid": 1000}`. The default is `tl-user`.
* `stdin_mode` accepts `closed` or `pipe`. The default is `closed`.
* `stdout_mode` and `stderr_mode` accept `capture` or `discard`. The default is `capture`.
## Response
Tensorlake returns `201 Created` with the started process metadata:
```json theme={null}
{
"handle": 1,
"pid": 42,
"status": "running",
"exit_code": null,
"signal": null,
"stdin_writable": true,
"command": "python",
"args": ["-m", "http.server", "8080"],
"started_at": 1710000000000,
"ended_at": null
}
```
`handle` is a daemon-local stable identifier. Route paths use the operating-system `pid`.
# Get Process Stderr
Source: https://docs.tensorlake.ai/api-reference/v2/processes/stderr
get /api/v1/processes/{pid}/stderr
Read the captured stderr lines for a process.
Read stderr that has already been captured for a process.
## Endpoint
```http theme={null}
GET /api/v1/processes/{pid}/stderr
```
## Example Request
```bash theme={null}
curl https://.sandbox.tensorlake.ai/api/v1/processes/42/stderr \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK`:
```json theme={null}
{
"pid": 42,
"lines": [
"Traceback (most recent call last):",
"ValueError: invalid input"
],
"line_count": 2
}
```
If you need stdout only, use [Get Process Stdout](/api-reference/v2/processes/stdout). If you need both streams merged, use [Get Process Output](/api-reference/v2/processes/output).
# Process Stdin
Source: https://docs.tensorlake.ai/api-reference/v2/processes/stdin
post /api/v1/processes/{pid}/stdin
Write raw bytes to a process whose stdin was opened in `pipe` mode.
Write raw bytes to a process's stdin.
The process must be started with `stdin_mode: "pipe"` for this endpoint to work.
## Write to Stdin
```http theme={null}
POST /api/v1/processes/{pid}/stdin
```
```bash theme={null}
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes/42/stdin \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/octet-stream" \
--data-binary "print('hello')\n"
```
Tensorlake returns `204 No Content` when the bytes are accepted.
If stdin was not opened in `pipe` mode, Tensorlake returns `400 Bad Request`. Missing processes return `404 Not Found`.
To deliver EOF without killing the process, use [Close Process Stdin](/api-reference/v2/processes/close-stdin).
# Get Process Stdout
Source: https://docs.tensorlake.ai/api-reference/v2/processes/stdout
get /api/v1/processes/{pid}/stdout
Read the captured stdout lines for a process.
Read stdout that has already been captured for a process.
## Endpoint
```http theme={null}
GET /api/v1/processes/{pid}/stdout
```
## Example Request
```bash theme={null}
curl https://.sandbox.tensorlake.ai/api/v1/processes/42/stdout \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK`:
```json theme={null}
{
"pid": 42,
"lines": [
"Serving HTTP on 0.0.0.0 port 8080",
"127.0.0.1 - - [06/Apr/2026 22:20:00] \"GET / HTTP/1.1\" 200 -"
],
"line_count": 2
}
```
If you need stderr only, use [Get Process Stderr](/api-reference/v2/processes/stderr). If you need both streams merged, use [Get Process Output](/api-reference/v2/processes/output).
# Create PTY Session
Source: https://docs.tensorlake.ai/api-reference/v2/pty/create
post /api/v1/pty
Create a PTY-backed interactive terminal session through the sandbox proxy.
Create a PTY session for interactive terminal access.
## Endpoint
```http theme={null}
POST /api/v1/pty
```
Use this endpoint on the sandbox proxy host:
```text theme={null}
https://.sandbox.tensorlake.ai/api/v1/pty
```
## Example Request
```bash theme={null}
curl -X POST https://.sandbox.tensorlake.ai/api/v1/pty \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"command": "/bin/bash",
"args": ["-l"],
"env": {"TERM": "xterm-256color"},
"working_dir": "/workspace",
"rows": 24,
"cols": 80
}'
```
## Request Body
```json theme={null}
{
"command": "/bin/bash",
"args": ["-l"],
"env": {"TERM": "xterm-256color"},
"working_dir": "/workspace",
"rows": 24,
"cols": 80
}
```
* `command` is required.
* `args` is optional.
* `env` is optional.
* `working_dir` is optional.
* `rows` and `cols` are optional and default to `24` and `80`.
* Tensorlake clamps `rows` to `1..500` and `cols` to `1..1000`.
## Response
Tensorlake returns `201 Created`:
```json theme={null}
{
"session_id": "LYtJOrxE9Kz3bphPUDzuX",
"token": ""
}
```
Use `session_id` with the other PTY endpoints. Use `token` when connecting to the PTY WebSocket.
If the sandbox already has 64 PTY sessions, Tensorlake returns `429 Too Many Requests` with code `TOO_MANY_SESSIONS`.
# Get PTY Session
Source: https://docs.tensorlake.ai/api-reference/v2/pty/get
get /api/v1/pty/{session_id}
Retrieve metadata for a single PTY session.
Get metadata for one PTY session.
## Endpoint
```http theme={null}
GET /api/v1/pty/{session_id}
```
## Example Request
```bash theme={null}
curl https://.sandbox.tensorlake.ai/api/v1/pty/ \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK`:
```json theme={null}
{
"session_id": "LYtJOrxE9Kz3bphPUDzuX",
"pid": 42,
"command": "/bin/bash",
"args": ["-l"],
"rows": 24,
"cols": 80,
"created_at": 1710000000000,
"ended_at": null,
"exit_code": null,
"is_alive": true
}
```
The PTY token is not returned from this endpoint. If the kernel OOM killer terminated the PTY process, the response includes `oom_killed: true`. If the session does not exist, Tensorlake returns `404 Not Found`.
# Sandbox PTY API Overview
Source: https://docs.tensorlake.ai/api-reference/v2/pty/introduction
Create and manage interactive PTY sessions through the sandbox proxy.
The sandbox PTY API is exposed through each sandbox's management hostname, derived from the sandbox's `ingress_endpoint`, not `https://api.tensorlake.ai`.
```text theme={null}
https://.sandbox.tensorlake.ai
```
Use PTY sessions when you need an interactive terminal, shell, or full-screen TUI inside a sandbox.
Include `Authorization: Bearer $TENSORLAKE_API_KEY` on requests to the sandbox proxy. The WebSocket attach endpoint also requires the per-session PTY token returned from session creation.
Start a new PTY-backed interactive session.
Enumerate the PTY sessions tracked by the sandbox daemon.
Inspect session metadata, terminal size, and liveness.
Change the terminal rows and columns for an active session.
Connect to a session over WebSocket and exchange terminal bytes.
Terminate a PTY session.
# Kill PTY Session
Source: https://docs.tensorlake.ai/api-reference/v2/pty/kill
delete /api/v1/pty/{session_id}
Terminate a PTY session.
Terminate a PTY session.
## Endpoint
```http theme={null}
DELETE /api/v1/pty/{session_id}
```
## Example Request
```bash theme={null}
curl -X DELETE https://.sandbox.tensorlake.ai/api/v1/pty/ \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `204 No Content`.
The daemon first sends `SIGHUP` to the PTY session and, if it is still alive after a short grace period, follows up with `SIGKILL`.
If the session does not exist, Tensorlake returns `404 Not Found`.
# List PTY Sessions
Source: https://docs.tensorlake.ai/api-reference/v2/pty/list
get /api/v1/pty
List the PTY sessions tracked inside a sandbox.
List the PTY sessions tracked inside a sandbox.
## Endpoint
```http theme={null}
GET /api/v1/pty
```
## Example Request
```bash theme={null}
curl https://.sandbox.tensorlake.ai/api/v1/pty \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK`:
```json theme={null}
{
"sessions": [
{
"session_id": "LYtJOrxE9Kz3bphPUDzuX",
"pid": 42,
"command": "/bin/bash",
"args": ["-l"],
"rows": 24,
"cols": 80,
"created_at": 1710000000000,
"ended_at": null,
"exit_code": null,
"is_alive": true
}
]
}
```
The PTY token is not included in list responses. If the kernel OOM killer terminated a PTY process, that session includes `oom_killed: true`.
# Resize PTY Session
Source: https://docs.tensorlake.ai/api-reference/v2/pty/resize
post /api/v1/pty/{session_id}/resize
Resize the terminal dimensions for a PTY session.
Resize a PTY session.
## Endpoint
```http theme={null}
POST /api/v1/pty/{session_id}/resize
```
## Example Request
```bash theme={null}
curl -X POST https://.sandbox.tensorlake.ai/api/v1/pty//resize \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"rows": 40, "cols": 120}'
```
## Request Body
```json theme={null}
{
"rows": 40,
"cols": 120
}
```
Tensorlake clamps `rows` to `1..500` and `cols` to `1..1000`.
## Response
Tensorlake returns `204 No Content` when the resize is applied.
If the session does not exist, Tensorlake returns `404 Not Found`.
# Attach PTY WebSocket
Source: https://docs.tensorlake.ai/api-reference/v2/pty/websocket
get /api/v1/pty/{session_id}/ws
Upgrade to a WebSocket connection for an interactive PTY session.
Attach to a PTY session over WebSocket.
## Endpoint
```http theme={null}
GET /api/v1/pty/{session_id}/ws
```
Use the WebSocket endpoint on the sandbox proxy host:
```text theme={null}
wss://.sandbox.tensorlake.ai/api/v1/pty//ws
```
## Authentication
You must provide the PTY token returned from [Create PTY Session](/api-reference/v2/pty/create).
Tensorlake accepts the token in either place:
* Preferred: `X-PTY-Token: `
* Backward-compatible fallback: `?token=`
The header form is preferred because query parameters are more likely to appear in access logs.
## WebSocket Protocol
After connecting, send a binary `READY` frame immediately so Tensorlake can flush any buffered output.
For an end-to-end example that creates the session, sends `READY`, runs a command, reads output, and closes cleanly, see [PTY Sessions](/sandboxes/pty-sessions).
### Client-to-server opcodes
| Opcode | Meaning | Payload |
| ------ | ------- | ----------------------------------------------------------- |
| `0x00` | Data | Raw terminal input bytes |
| `0x01` | Resize | `cols` as big-endian `u16`, then `rows` as big-endian `u16` |
| `0x02` | Ready | No payload |
### Server-to-client opcodes
| Opcode | Meaning | Payload |
| ------ | ------- | ----------------------------- |
| `0x00` | Data | Raw terminal output bytes |
| `0x03` | Exit | Exit code as big-endian `i32` |
## Example Connection
Header-based token:
```bash theme={null}
wscat -H "X-PTY-Token: " -c "wss://.sandbox.tensorlake.ai/api/v1/pty//ws"
```
Query-string token:
```bash theme={null}
wscat -c "wss://.sandbox.tensorlake.ai/api/v1/pty//ws?token="
```
## Connection Semantics
* If the token is invalid, Tensorlake returns `403 Forbidden` with code `INVALID_TOKEN`.
* If the session does not exist, Tensorlake returns `404 Not Found` with code `SESSION_NOT_FOUND`.
* When the process exits, Tensorlake sends the `0x03` exit frame and then closes the WebSocket with reason `exit:`.
* If the PTY session is terminated while the socket is open, Tensorlake closes the WebSocket with code `1001` and reason `session terminated`.
* If you do not send `READY`, Tensorlake buffers output up to 1 MB before disconnecting the client.
# Runtime Health
Source: https://docs.tensorlake.ai/api-reference/v2/runtime/health
get /api/v1/health
Check whether the sandbox daemon is healthy.
Check whether the sandbox daemon is healthy.
## Endpoint
```http theme={null}
GET /api/v1/health
```
Use this endpoint on the sandbox proxy host:
```text theme={null}
https://.sandbox.tensorlake.ai/api/v1/health
```
## Example Request
```bash theme={null}
curl https://.sandbox.tensorlake.ai/api/v1/health \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK`:
```json theme={null}
{
"healthy": true
}
```
# Runtime Info
Source: https://docs.tensorlake.ai/api-reference/v2/runtime/info
get /api/v1/info
Retrieve sandbox daemon version, uptime, and process counts.
Get sandbox daemon metadata and process counts.
## Endpoint
```http theme={null}
GET /api/v1/info
```
Use this endpoint on the sandbox proxy host:
```text theme={null}
https://.sandbox.tensorlake.ai/api/v1/info
```
## Example Request
```bash theme={null}
curl https://.sandbox.tensorlake.ai/api/v1/info \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK`:
```json theme={null}
{
"version": "0.1.0",
"uptime_secs": 3600,
"running_processes": 2,
"total_processes": 5
}
```
# Delete File
Source: https://docs.tensorlake.ai/api-reference/v2/sandbox-files/delete
delete /api/v1/files
Delete a file through the sandbox proxy.
Delete a file from a sandbox path.
## Endpoint
```http theme={null}
DELETE /api/v1/files?path=
```
## Example Request
```bash theme={null}
curl -X DELETE "https://.sandbox.tensorlake.ai/api/v1/files?path=/workspace/temp.txt" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `204 No Content` when the file is deleted.
If the file does not exist, Tensorlake returns `404 Not Found`. Paths containing `..` are rejected with `403 Forbidden`.
# Sandbox Files API Overview
Source: https://docs.tensorlake.ai/api-reference/v2/sandbox-files/introduction
Read, write, delete, and list files through the sandbox proxy.
The sandbox file API is exposed through each sandbox's management hostname, derived from the sandbox's `ingress_endpoint`, not `https://api.tensorlake.ai`.
```text theme={null}
https://.sandbox.tensorlake.ai
```
Use these endpoints to read files, upload content, delete files, and list directory contents inside a sandbox.
Include `Authorization: Bearer $TENSORLAKE_API_KEY` on requests to the sandbox proxy.
Download file contents from a sandbox path.
Upload bytes to a sandbox path.
Remove a file from a sandbox.
List files and directories at a sandbox path.
# List Directory
Source: https://docs.tensorlake.ai/api-reference/v2/sandbox-files/list
get /api/v1/files/list
List directory contents through the sandbox proxy.
List the contents of a sandbox directory.
## Endpoint
```http theme={null}
GET /api/v1/files/list?path=
```
## Example Request
```bash theme={null}
curl "https://.sandbox.tensorlake.ai/api/v1/files/list?path=/workspace" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK`:
```json theme={null}
{
"path": "/workspace",
"entries": [
{
"name": "src",
"is_dir": true,
"size": null,
"modified_at": 1710000000000
},
{
"name": "data.csv",
"is_dir": false,
"size": 24,
"modified_at": 1710000001000
}
]
}
```
Entries are sorted with directories first and then alphabetically. If the path is not a directory, Tensorlake returns `400 Bad Request`. Missing paths return `404 Not Found`.
# Read File
Source: https://docs.tensorlake.ai/api-reference/v2/sandbox-files/read
get /api/v1/files
Read a file through the sandbox proxy.
Read a file from a sandbox path.
## Endpoint
```http theme={null}
GET /api/v1/files?path=
```
## Example Request
```bash theme={null}
curl "https://.sandbox.tensorlake.ai/api/v1/files?path=/workspace/data.csv" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK` with `Content-Type: application/octet-stream` and the raw file bytes.
If the path points to a directory, Tensorlake returns `400 Bad Request`. If the file does not exist, Tensorlake returns `404 Not Found`. Paths containing `..` are rejected with `403 Forbidden`.
# Write File
Source: https://docs.tensorlake.ai/api-reference/v2/sandbox-files/write
put /api/v1/files
Write raw bytes to a sandbox file path through the sandbox proxy.
Write bytes to a sandbox path.
## Endpoint
```http theme={null}
PUT /api/v1/files?path=
```
## Example Request
```bash theme={null}
curl -X PUT "https://.sandbox.tensorlake.ai/api/v1/files?path=/workspace/config.json" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/octet-stream" \
--data-binary '{"debug": true, "port": 8080}'
```
## Response
Tensorlake returns `204 No Content` when the write succeeds.
Parent directories are created automatically if needed. Paths containing `..` are rejected with `403 Forbidden`.
# Copy Sandbox
Source: https://docs.tensorlake.ai/api-reference/v2/sandboxes/copy
post /sandboxes/{sandbox_id}/copy
Boot one or more new sandboxes from a running or suspended source, restoring filesystem, memory, and running processes so each copy warm-starts. A running source is copied from the executor hosting it; a suspended source is copied from the snapshot its suspend produced. Copies inherit the source's image, resources, entrypoint, network policy, and exposed ports. Takes no request body.
Boot one or more new sandboxes from a running or suspended source, restoring filesystem, memory, and running processes so each copy warm-starts.
* This path accepts either the sandbox ID or the sandbox name.
* The source must be running or suspended. Any other state returns `400 Bad Request`.
* A running source is copied from the executor hosting it. A suspended source is copied from the snapshot its suspend already produced, so a copy does not leave a new checkpoint behind for you to clean up.
* Copies inherit the source's image, resources, entrypoint, network policy, and exposed ports.
* The request takes no body. `times` and `name` are query parameters.
## Endpoint
```http theme={null}
POST /sandboxes/{sandbox_id}/copy
```
## Example Request
```bash theme={null}
curl -X POST "https://api.tensorlake.ai/sandboxes//copy?times=1" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Example Response
```json theme={null}
{
"source_sandbox_id": "sbx_01HK9ZA4MT",
"sandboxes": [
{
"sandbox_id": "sbx_01HKA1B7QP",
"status": "running",
"ingress_endpoint": "https://sbx_01HKA1B7QP.sandbox.tensorlake.ai"
}
]
}
```
## Fanning Out
Pass `times` to create several copies from the same source in one call. Each copy is an independent sandbox with its own ID.
```bash theme={null}
curl -X POST "https://api.tensorlake.ai/sandboxes//copy?times=4" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Naming Copies
Sandbox names are unique per namespace, so copies cannot reuse the source's name. Names are derived instead:
| `name` | `times` | Source | Copy names |
| -------- | ------- | ----------------- | -------------------------------------- |
| `worker` | 1 | any | `worker` |
| `worker` | 3 | any | `worker-1`, `worker-2`, `worker-3` |
| omitted | 1 | named `build-env` | `build-env-copy` |
| omitted | 2 | named `build-env` | `build-env-copy-1`, `build-env-copy-2` |
| omitted | any | unnamed | unnamed |
```bash theme={null}
curl -X POST "https://api.tensorlake.ai/sandboxes//copy?times=3&name=worker" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
Naming matters for lifecycle: only named sandboxes can be [suspended and resumed](/sandboxes/lifecycle). An unnamed copy terminates at its idle timeout instead. A named source therefore keeps its copies named by default, so a copy behaves like the sandbox it came from.
Pass `name` explicitly to give an unnamed source's copies a name.
Derived names are validated before any copy is created:
* `400` if a derived name is malformed, or exceeds the 63-character DNS label limit once suffixed. A base that fits on its own can still overflow with `-N` appended.
* `409` if a derived name is already claimed by a live sandbox in the namespace. This is checked up front, so a rejected request creates nothing. Re-running the same named fan-out is safe and will not leave partial copies behind.
## Partial Failures
A `200` means every requested copy is running. Two other statuses return the same body shape, so inspect each entry's `status` to see which copies succeeded:
* `422`: one or more copies failed before becoming ready.
* `504`: one or more copies did not become ready within the timeout.
## Related
Capture a reusable artifact and restore it into new sandboxes later.
Create, suspend, resume, and terminate sandboxes.
# Create Sandbox
Source: https://docs.tensorlake.ai/api-reference/v2/sandboxes/create
post /sandboxes
Create an ephemeral or named sandbox.
Launch an ephemeral or named sandbox.
The response includes the sandbox's `ingress_endpoint`. Prefer it over constructing `*.sandbox.tensorlake.ai` hostnames yourself when building sandbox-specific URLs in client code.
* Omit `name` to create an ephemeral sandbox.
* Set `name` to create a named sandbox that supports suspend and resume.
* Set `snapshot_id` to restore from a snapshot, or `image` to boot from a registered Sandbox Image.
* For fresh creates, if `resources.disk_mb` is omitted, the sandbox uses the default 10 GB root disk (`10240` MiB).
* With `image`, `resources.disk_mb` can be used to grow the root disk at create time (growth-only).
* With `snapshot_id` from a filesystem snapshot, `resources.disk_mb` can be used to grow the root disk at create time (growth-only).
# Delete Sandbox
Source: https://docs.tensorlake.ai/api-reference/v2/sandboxes/delete
delete /sandboxes/{sandbox_id}
Terminate a sandbox. This operation is idempotent and returns success if the sandbox was already terminated.
Terminate a sandbox.
This call is idempotent. If the sandbox is already terminated, Tensorlake still returns success.
# Get Sandbox
Source: https://docs.tensorlake.ai/api-reference/v2/sandboxes/get
get /sandboxes/{sandbox_id}
Retrieve metadata for a sandbox in the current project, including its `ingress_endpoint`, the base ingress origin for the sandbox's current placement, and its `sandbox_url`, the sandbox-specific management URL derived from it.
Retrieve metadata for a single sandbox.
The response includes `ingress_endpoint`, the base ingress origin for this sandbox's current placement, and `sandbox_url`, the sandbox-specific management URL derived from it.
# List Sandboxes
Source: https://docs.tensorlake.ai/api-reference/v2/sandboxes/list
get /sandboxes
List sandboxes.
List sandboxes for the current project.
Use `status=running` to query only sandboxes that are currently live in memory. Omitting the filter returns full sandbox history, including terminated sandboxes.
# Restart Sandbox
Source: https://docs.tensorlake.ai/api-reference/v2/sandboxes/restart
post /sandboxes/{sandbox_id}/restart
Restart a terminated sandbox under its original ID and name.
Restart a terminated sandbox. The sandbox keeps its original ID and name.
* This path accepts either the sandbox ID or the sandbox name.
* Only terminated sandboxes can be restarted. Anything else returns `400 Bad Request`; to wake a suspended sandbox, use [Resume](/api-reference/v2/sandboxes/resume) instead.
* Terminated sandboxes stay restartable for 48 hours after termination.
* When the sandbox has a usable snapshot, the restart restores from the most recent one. When no snapshot exists, the sandbox cold boots from its image with a fresh filesystem.
* Tensorlake returns `202 Accepted` when restart starts; the sandbox re-enters `Pending` and boots like a newly created sandbox.
* Tensorlake returns `409 Conflict` when a new sandbox has claimed the name since termination, or when the snapshot the restart would restore from is being deleted.
# Restore Sandbox
Source: https://docs.tensorlake.ai/api-reference/v2/sandboxes/restore
post /sandboxes
Restore a sandbox from a previously created snapshot by calling the create endpoint with a `snapshot_id`.
Restore a new sandbox from a previously created snapshot.
## Endpoint
```http theme={null}
POST /sandboxes
```
To restore from a snapshot, call the standard create sandbox endpoint and include `snapshot_id` in the request body.
* If the snapshot type is filesystem (default), the new sandbox restores the captured filesystem. You can override launch settings (including resources).
* If the snapshot type is memory, the new sandbox restores filesystem, memory, and running processes exactly as they were. Image, resources (CPUs, memory), entrypoint, and secrets come from the snapshot and cannot be changed at restore time.
## Example Request
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"snapshot_id": ""
}'
```
For filesystem snapshots, `resources.disk_mb` can be used at restore time to grow the root disk (growth-only).
For the full request and response schema of `POST /sandboxes`, see [Create Sandbox](/api-reference/v2/sandboxes/create). For the end-to-end snapshot workflow, see [Snapshots](/sandboxes/snapshots).
# Resume Sandbox
Source: https://docs.tensorlake.ai/api-reference/v2/sandboxes/resume
post /sandboxes/{sandbox_id}/resume
Resume a suspended named sandbox from its suspend snapshot. Returns `202 Accepted` when resume begins and `200 OK` when the sandbox is already running.
Resume a suspended named sandbox from its suspend snapshot.
* This path accepts either the sandbox ID or the sandbox name.
* Tensorlake returns `202 Accepted` when resume starts.
* Tensorlake returns `200 OK` when the sandbox is already running.
* If the sandbox is not suspended, or its suspend snapshot is missing or not ready, Tensorlake returns `400 Bad Request`.
Most sandbox-proxy requests to a suspended named sandbox also resume it automatically, so this endpoint is mainly useful when you want to wake the sandbox proactively before sending traffic.
# Snapshot Sandbox
Source: https://docs.tensorlake.ai/api-reference/v2/sandboxes/snapshot
post /sandboxes/{sandbox_id}/snapshot
Create a snapshot of a running sandbox so you can restore the same filesystem and memory state later.
Create a snapshot of a running sandbox so you can restore the same filesystem and memory state later.
You can optionally pass `snapshot_type` in the request body:
* `filesystem` (default): captures filesystem state only and restores with a cold boot.
* `memory`: captures filesystem, memory, and running process state and restores with a warm start.
## Endpoint
```http theme={null}
POST /sandboxes/{sandbox_id}/snapshot
```
## Example Request
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandboxes//snapshot \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"snapshot_type":"memory"}'
```
Use the created snapshot with [Restore Sandbox](/api-reference/v2/sandboxes/restore) when you want to boot a new sandbox from that saved state.
For the broader snapshot lifecycle, including listing and deleting snapshots, see [Snapshots](/sandboxes/snapshots).
# Suspend Sandbox
Source: https://docs.tensorlake.ai/api-reference/v2/sandboxes/suspend
post /sandboxes/{sandbox_id}/suspend
Suspend a named running sandbox by snapshotting it and terminating the live container. Returns `202 Accepted` when suspension begins or is already in progress, and `200 OK` when the sandbox is already suspended.
Suspend a named running sandbox by snapshotting it and terminating the live container.
* This path accepts either the sandbox ID or the sandbox name.
* Only named sandboxes can be suspended. Ephemeral sandboxes return `400 Bad Request`.
* Tensorlake returns `202 Accepted` when suspension starts or is already in progress.
* Tensorlake returns `200 OK` when the sandbox is already suspended.
# Update Sandbox
Source: https://docs.tensorlake.ai/api-reference/v2/sandboxes/update
patch /sandboxes/{sandbox_id}
Update proxy-visible sandbox settings such as public exposed ports and whether ingress can skip authentication checks.
Update settings of a running sandbox.
This endpoint controls the sandbox proxy allowlist (`exposed_ports` and `allow_unauthenticated_access`), the sandbox `name`, and the egress `network` policy. The `network` field is tri-state: omit it to leave the policy unchanged, send an object to replace it, or send `null` to clear it. See [Networking](/sandboxes/networking#update-the-policy-on-a-running-sandbox) for details.
# Disable SSH
Source: https://docs.tensorlake.ai/api-reference/v2/ssh/disable
post /api/v1/ssh/disable
Stop the sandbox's internal SSH daemon.
Stop the sandbox's internal SSH server.
This endpoint is normally managed by Tensorlake's SSH proxy. Most users should disconnect their SSH client or use the CLI instead of calling it directly.
The public sandbox proxy authenticates your API key and injects the internal forwarded-auth headers required by the daemon.
## Endpoint
```http theme={null}
POST /api/v1/ssh/disable
```
## Example Request
```bash theme={null}
curl -X POST https://.sandbox.tensorlake.ai/api/v1/ssh/disable \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK` with the updated SSH daemon status:
```json theme={null}
{
"enabled": false,
"pid": null,
"host_key_fingerprint": "SHA256:..."
}
```
# Enable SSH
Source: https://docs.tensorlake.ai/api-reference/v2/ssh/enable
post /api/v1/ssh/enable
Enable the sandbox's internal SSH daemon for sandbox-proxy backend connections.
Enable the sandbox's internal SSH server for authenticated sandbox-proxy connections.
This endpoint is normally called by Tensorlake's SSH proxy. Most users should use [SSH and PTY Sessions](/sandboxes/pty-sessions) or `tl sbx ssh` instead of calling this endpoint directly.
The public sandbox proxy authenticates your API key and injects the internal forwarded-auth headers required by the daemon.
## Endpoint
```http theme={null}
POST /api/v1/ssh/enable
```
Use this endpoint on the sandbox proxy host:
```text theme={null}
https://.sandbox.tensorlake.ai/api/v1/ssh/enable
```
## Example Request
```bash theme={null}
curl -X POST https://.sandbox.tensorlake.ai/api/v1/ssh/enable \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"proxy_pubkey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...",
"backend_user": "tl-user"
}'
```
## Request Body
```json theme={null}
{
"proxy_pubkey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...",
"backend_user": "tl-user"
}
```
* `proxy_pubkey` is required. It is the sandbox proxy's backend public key.
* `backend_user` is optional and defaults to `tl-user`.
## Response
Tensorlake returns `200 OK` with SSH daemon status:
```json theme={null}
{
"enabled": true,
"pid": 123,
"host_key_fingerprint": "SHA256:..."
}
```
# SSH Status
Source: https://docs.tensorlake.ai/api-reference/v2/ssh/status
get /api/v1/ssh/status
Retrieve the sandbox's internal SSH daemon status.
Get the sandbox's internal SSH server status.
This endpoint reports the in-sandbox `sshd` lifecycle used by Tensorlake's SSH proxy. To connect with SSH as an end user, see [SSH and PTY Sessions](/sandboxes/pty-sessions).
The public sandbox proxy authenticates your API key and injects the internal forwarded-auth headers required by the daemon.
## Endpoint
```http theme={null}
GET /api/v1/ssh/status
```
## Example Request
```bash theme={null}
curl https://.sandbox.tensorlake.ai/api/v1/ssh/status \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## Response
Tensorlake returns `200 OK`:
```json theme={null}
{
"enabled": true,
"pid": 123,
"host_key_fingerprint": "SHA256:..."
}
```
# TCP Tunnel WebSocket
Source: https://docs.tensorlake.ai/api-reference/v2/tunnels/tcp
get /api/v1/tunnels/tcp
Upgrade to a WebSocket that relays binary frames to and from a sandbox-local TCP port.
Open an authenticated WebSocket tunnel to a TCP port listening inside the sandbox.
Use this endpoint when you need raw TCP byte forwarding, such as VNC, Postgres, Redis, or Chrome DevTools over a private local tunnel. For most users, the `tl sbx tunnel` CLI or SDK helper manages this protocol for you.
## Endpoint
```http theme={null}
GET /api/v1/tunnels/tcp?port=
```
Use the WebSocket endpoint on the sandbox proxy host:
```text theme={null}
wss://.sandbox.tensorlake.ai/api/v1/tunnels/tcp?port=9222
```
## Example Connection
Authenticate to the sandbox proxy with your API key. The proxy injects the internal forwarded-auth headers that the daemon requires.
```bash theme={null}
wscat -c "wss://.sandbox.tensorlake.ai/api/v1/tunnels/tcp?port=9222" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
## WebSocket Protocol
After the WebSocket upgrade succeeds, binary frames are relayed verbatim:
* Client binary frames are written to `127.0.0.1:` inside the sandbox.
* Bytes read from that TCP connection are sent back as WebSocket binary frames.
* Text frames are rejected.
## Response Semantics
* `101 Switching Protocols` means the tunnel is open.
* `400 Bad Request` means `port` is missing, invalid, or `0`.
* `401 Unauthorized` means authenticated sandbox-proxy forwarding was not present.
* `502 Bad Gateway` means nothing accepted the TCP connection on `127.0.0.1:` inside the sandbox.
For local-port forwarding examples, see [Local Tunnels](/sandboxes/tunnels).
# Architecture
Source: https://docs.tensorlake.ai/applications/architecture
How Tensorlake's Application Runtime runs your code under the hood
This page describes the architecture of Tensorlake's Application Runtime. Tensorlake is a complex system with many moving parts. To help users build a mental model of how it works, this page documents the system architecture.
**Advanced topic.** You do not need to understand these details to effectively use Tensorlake. The details are documented here for those who wish to learn about them without having to go spelunking through the source code.
## High-Level Overview
When a request hits your application, the runtime creates a new sandbox in milliseconds and your agent starts in an isolated environment with its own filesystem. Every function decorated with `@function()` can run in its own remote sandbox with dedicated resources. From your code it looks like a normal function call, but under the hood the runtime is scheduling containers, managing state, and handling failures.
At a high level, the system looks like this:
```mermaid theme={null}
graph TD
Client["Client (SDK / HTTP)"] --> Server
subgraph Server["Server (Control Plane)"]
direction LR
API["HTTP API"]
AppSched["Application
Scheduler"]
ContSched["Container
Scheduler"]
StateDB["State Store"]
end
Server -- "gRPC stream" --> DPA
Server -- "gRPC stream" --> DPB
subgraph DPA["Dataplane A"]
LR1["Language Runtime"]
LR2["Language Runtime"]
end
subgraph DPB["Dataplane B"]
LR3["Language Runtime"]
LR4["Language Runtime"]
end
```
The **server** is the control plane. It receives requests from clients, persists all state, and runs two schedulers. The **application scheduler** manages the lifecycle of function calls: it builds the execution graph for each request, creates allocations, checkpoints outputs, and handles replay on failure. The **container scheduler** manages the infrastructure layer: it tracks resources across all dataplanes, places containers on worker nodes, manages warm pools, and scales containers up and down based on demand.
A **dataplane** manages containers on a pool of worker nodes. You can think of it as a regional cluster of compute capacity. Multiple dataplanes can run in parallel, and the server distributes work across them. Each dataplane maintains a persistent bidirectional gRPC stream with the server. It reports its current state (running containers, resource usage, allocation results) and receives new work assignments in return.
A **language runtime** is the sandbox that runs your code. Every `@function()` call runs in its own isolated container with its own filesystem, dependencies, and resource limits. When a function calls another function, the child runs in a separate sandbox. A lightweight orchestrator can dispatch work to GPU-equipped containers without needing GPU resources itself. From your code, this is invisible.
## Why a Custom Scheduler
A common question is why Tensorlake built its own container scheduler instead of using Kubernetes. The short answer is that Kubernetes was designed for long-running services, not for workloads that create a new container for every request and need it running in milliseconds.
Tensorlake's execution model is fundamentally different from what Kubernetes expects. When a request arrives, the runtime creates a fresh sandbox (an isolated container with its own filesystem) in single-digit milliseconds. At peak load, the scheduler creates hundreds of these per second. In Kubernetes, creating a pod involves writing to etcd, passing through admission controllers, waiting for the kubelet to sync, and pulling images. This takes seconds at best, often longer. Creating a pod per request at this rate would overwhelm the Kubernetes control plane.
Beyond raw speed, the container scheduler is tightly integrated with the application scheduler in ways that a general-purpose orchestrator can't be. It understands function-level container pools (warm pools, minimum counts, and buffer sizes per function) and uses this to make smarter placement decisions. Its eviction algorithm knows which containers have active allocations and never evicts them, prioritizing containers above pool buffers first. It tracks container affinity per function so it can route work to dataplanes that already have warm containers, avoiding cold starts entirely. None of these concepts exist in Kubernetes scheduling.
The desired state model is also purpose-built for this workload. The server pushes desired state to dataplanes over a persistent gRPC stream, and dataplanes reconcile in real-time. Kubernetes uses a watch/list model over etcd that works well for long-running services but adds latency when you need the scheduling loop to react in milliseconds, for example when a function completes and the next step in a workflow needs to start immediately.
Finally, there is an operational argument. Deploying agents on Kubernetes means writing YAMLs, configuring Horizontal Pod Autoscalers, managing image pull policies, setting up KEDA or Knative for scale-to-zero, and running a separate durable execution server for crash recovery. Tensorlake collapses all of that into a single runtime. You deploy your Python code, and the scheduler handles the rest.
## The Server
The server is the single control plane for the entire system. It exposes the HTTP API that receives requests from clients and the SDK, persists all state to a durable store, and runs the two schedulers that coordinate all work. Every request, every function call, every allocation, and every container decision flows through the server.
When a request arrives, the server creates a **request context**: a record that tracks the full state of the request, including the function call graph, all function runs, and the final outcome. The request context is persisted immediately. From this point, the application scheduler and container scheduler work together to execute the request.
### Container Scheduler
The container scheduler is responsible for the infrastructure layer: deciding which containers run on which machines, and managing their lifecycle.
```mermaid theme={null}
graph LR
subgraph CS["Container Scheduler"]
direction TB
RT["Resource Tracker
CPU, memory, GPU per executor"]
CP["Container Pools
min, max, buffer per function"]
PL["Placement Engine
constraints, affinity, eviction"]
end
CS -- "create / terminate" --> DPA["Dataplane A"]
CS -- "create / terminate" --> DPB["Dataplane B"]
```
The container scheduler maintains a real-time view of every executor (worker node) in the system: its total and free resources (CPU, memory, GPU), and every container running on it. It also tracks **container pools**, which group containers by function. Each pool has configurable minimums, maximums, and buffer sizes that control scaling behavior.
When the application scheduler needs a container for a function, the container scheduler first checks whether a **warm container** already exists in the function's pool, a pre-initialized container with no active work. If one is available, it claims it immediately, avoiding cold-start latency entirely.
If no warm container is available, the scheduler runs its **placement engine**. It finds candidate executors that satisfy the function's resource requirements and constraints, then selects one. If no executor has enough free resources, the scheduler runs a **vacuum pass**: it looks for lower-priority containers that can be evicted to free up space. Eviction follows a priority order: containers above the pool's buffer count are evicted first, then those above the minimum, and only as a last resort those at or below the minimum. Containers with active allocations are never evicted.
The container scheduler communicates with dataplanes through a **desired state model**. Rather than issuing imperative commands, it declares the desired state of each container (running or terminated) and the dataplane reconciles its actual state to match. This makes the system resilient to transient failures: if a message is lost, the next reconciliation cycle corrects the drift.
Scaling is driven by demand. When requests arrive, new containers are created. When traffic drops, idle containers are terminated. Functions with no traffic have no running containers and incur no cost. You can configure **warm pools** to keep a buffer of pre-initialized containers ready for latency-sensitive functions, or set **concurrency caps** to limit the total number of concurrent instances.
### Application Scheduler
The application scheduler manages the execution of your code: the function call graph, allocations, checkpointing, and replay.
```mermaid theme={null}
graph TD
Req["Incoming Request"] --> RC["Request Context
function call graph, runs, outcome"]
RC --> FC["Function Calls
nodes in the execution DAG"]
FC --> FR["Function Runs
execution instances with checkpointed outputs"]
FR --> AL["Allocations
unit of work assigned to a container"]
AL --> CS["Container Scheduler
finds or creates a container"]
```
When a request arrives, the application scheduler creates an initial **function call**, a node in the execution graph that represents a function to invoke with specific inputs. For each function call, it creates a **function run**, an execution instance that tracks status (pending, running, completed) and stores the checkpointed output when the function finishes.
To actually execute a function run, the application scheduler creates an **allocation**, a unit of work that binds a function run to a specific container. It first checks whether an existing container has capacity (based on the function's `max_concurrency` setting). If not, it asks the container scheduler to create a new one. The allocation is persisted and pushed to the dataplane through the desired state stream.
When a function calls another function, the language runtime reports the child function call back to the server. The application scheduler adds a new node to the execution graph, creates a function run for it, and the cycle repeats. This is how Tensorlake builds the full DAG of function calls for each request: the graph grows dynamically as your code executes.
When a function run completes, the application scheduler checkpoints its output. The output data is stored in object storage (not in the database), so your agents can pass large files between functions without workarounds. The scheduler then propagates the output to any downstream function calls that depend on it, creating new function runs as inputs become available.
**Replay** is how the system recovers from failures. When a request is replayed, the application scheduler walks the function call graph from the beginning. Function runs that already have checkpointed outputs return their results instantly without re-executing. The replay fast-forwards through completed work until it reaches the function that failed, then starts running it again from scratch. From the application's perspective, it picks up right where it left off.
**Retries** handle individual function failures. When a function run fails (whether from an exception, a container crash, or a timeout), the application scheduler checks the function's retry policy. If retries are available, it creates a new allocation and runs the function again with the same inputs.
## Dataplanes
A dataplane manages the containers on a pool of worker nodes. It is the bridge between the server's scheduling decisions and actual code execution.
```mermaid theme={null}
graph LR
subgraph DP["Dataplane"]
direction TB
SR["State Reconciler"]
HR["Heartbeat Reporter"]
FEC1["Language Runtime Controller"]
FEC2["Language Runtime Controller"]
end
Server -- "desired state
(gRPC stream)" --> DP
DP -- "heartbeats
(every 5s)" --> Server
```
Each dataplane maintains two communication channels with the server. A **bidirectional gRPC stream** carries the desired state: the server pushes container specifications and allocations to the dataplane, and the dataplane acknowledges receipt. A **heartbeat** fires every five seconds, reporting the dataplane's current state: which containers are running, their resource usage, and the results of completed allocations.
When the dataplane receives a new desired state, its **state reconciler** compares it against the actual state of the local system. If a container should exist but doesn't, it creates one. If a container should be terminated, it shuts it down. If an allocation needs to be executed, it routes it to the appropriate language runtime controller.
Allocation results flow back to the server through the heartbeat channel. The dataplane buffers results and fragments large payloads across multiple heartbeats (with a 10MB limit per message) to avoid overwhelming the connection. Results are only removed from the buffer after the server acknowledges receipt, ensuring nothing is lost in transit.
If the gRPC stream disconnects, the dataplane reconnects automatically. If heartbeats fail, it uses exponential backoff. The desired state model means that temporary disconnections don't cause inconsistency: the next successful sync brings everything back in line.
## Language Runtimes
A language runtime is the sandbox that runs your function code. It is a container with its own filesystem, dependencies, and resource limits, managed by a controller on the dataplane.
```mermaid theme={null}
graph LR
AL["Allocation"] --> P["Preparing
download inputs,
presign URLs"]
P --> R["Running
execute function,
stream state"]
R --> F["Finalizing
upload outputs,
clean up"]
```
When a language runtime receives an allocation, it processes it through a three-phase pipeline. In the **preparing** phase, the runtime downloads input data and presigns blob URLs for outputs. This phase does not occupy a concurrency slot, so the container can prepare multiple allocations in parallel while running others.
In the **running** phase, the function code executes. The language runtime streams state updates back to the dataplane controller in real-time: progress updates, output blob requests, child function calls, and the final result. If the function calls another `@function()`-decorated function, the language runtime reports the child call to the server, which creates a new allocation for it. For **blocking calls**, the language runtime registers a watcher and pauses until the child function's result arrives from the server.
In the **finalizing** phase, the runtime completes any multipart uploads, cleans up blob handles, and releases the concurrency slot.
Each language runtime has a configurable `max_concurrency` that limits how many allocations it can execute simultaneously in the running phase. The application scheduler respects this limit when placing allocations: if all slots are full, it either queues the work or asks the container scheduler for a new container.
## Getting in Depth
This has been a high-level overview of the Application Runtime architecture. The [durable execution model](/applications/durability), [crash recovery behavior](/applications/crash-recovery), [scaling configuration](/applications/scaling-agents), and [queuing behavior](/applications/scale-out-queuing) are all documented in more detail.
How checkpointing and replay work to make your functions resilient to failures.
How the server detects failures and re-schedules only the work that needs to re-run.
Configure scaling behavior, warm pools, and concurrency limits for your functions.
Functions, applications, decorators, and resource configuration.
# Async Functions
Source: https://docs.tensorlake.ai/applications/async-functions
Use Python async/await with Tensorlake async functions. Run them concurrently to optimize resource usage and reduce latency.
An `async` Tensorlake function behaves like a regular Python `async` function. Calling it returns a coroutine
that doesn't run until it's awaited or started with `asyncio.create_task()` or other `asyncio` module functions.
```python theme={null}
from tensorlake.applications import application, function
@function()
async def capitalize(text: str) -> str:
return text.upper()
@application()
@function()
async def greet(name: str) -> str:
# Calling an async Tensorlake function `capitalize` returns a coroutine.
# `await` is available inside async Tensorlake functions `greet`.
# `await` starts the `capitalize` coroutine and waits for it to complete, returning the result.
capitalized: str = await capitalize(name)
return f"Hello, {capitalized}!"
```
coroutines returned by async Tensorlake functions behave almost the same way as [Futures](/applications/futures)
used with sync Tensorlake functions.
### asyncio.create\_task
Use `asyncio.create_task()` to run a coroutine in the background without blocking on it. This returns an `asyncio.Task`
that can be awaited later to get the result.
```python theme={null}
import asyncio
from tensorlake.applications import application, function
@function()
async def double(x: int) -> int:
return x * 2
@application()
@function()
async def my_app(x: int) -> int:
coroutine = double(x)
# Starts the coroutine in the background and returns an asyncio.Task.
task: asyncio.Task = asyncio.create_task(coroutine)
# Do something else and then await the task to get the result.
return await task
```
### Running coroutines in parallel with asyncio.gather
Use `asyncio.gather()` to run multiple coroutines in parallel and collect their results. This is the
standard Python way to run async functions concurrently.
```python theme={null}
import asyncio
from tensorlake.applications import application, function
@function()
async def capitalize(text: str) -> str:
return text.upper()
@function()
async def make_joke(name: str) -> str:
return f"Why did {name} cross the road? To get to the other side!"
@application()
@function()
async def greet(name: str) -> str:
# Start both function calls in parallel.
capitalized, joke = await asyncio.gather(
capitalize(name),
make_joke(name),
)
return f"Hello, {capitalized}! {joke}"
```
### Non-blocking map and reduce operations
Calling `function.map(...)` or `function.reduce(...)` on an async function returns a coroutine.
```python theme={null}
from tensorlake.applications import application, function
@function()
async def double(x: int) -> int:
return x * 2
@function()
async def add(a: int, b: int) -> int:
return a + b
@application()
@function()
async def process_numbers(numbers: list[int]) -> int:
# Calling .map() on an async function returns a coroutine.
# `await` runs the map operation and blocks until all items are processed.
doubled: list[int] = await double.map(numbers)
# Calling .reduce() on an async function also returns a coroutine.
total: int = await add.reduce(doubled)
return total
```
The coroutines returned by `function.map()` or `function.reduce()` behave exactly the same as coroutines returned
by async `function(...)` calls.
### Passing coroutines and asyncio.Tasks as inputs
Coroutines returned from async Tensorlake functions and `asyncio.Task` objects created with `asyncio.create_task()` from
such coroutines can be passed as arguments to other function calls.
Tensorlake automatically runs the coroutines or `asyncio.Task` objects, waits for them to complete, and uses their results
as the argument values. This works exactly like [passing Futures as inputs](/applications/futures#passing-futures-as-inputs).
```python theme={null}
from tensorlake.applications import application, function
@function()
async def double(x: int) -> int:
return x * 2
@function()
async def add(a: int, b: int) -> int:
return a + b
@application()
@function()
async def my_app(x: int) -> int:
a = double(x)
b = double(x + 1)
# Pass coroutines as function call arguments. Tensorlake runs both in parallel,
# waits for them to complete, and uses their results as the arguments for `add`.
return await add(a, b)
```
All input coroutines that don't depend on each other run in parallel, allowing Tensorlake to optimize resource usage and
reduce overall application latency. A function call or a map-reduce operation are only blocked while their input coroutines
are running. Once all input coroutines complete, Tensorlake automatically runs the function call or the map-reduce operation.
### Wrapping coroutines and asyncio.Tasks into Python objects is not allowed
When passing Tensorlake coroutines or `asyncio.Task` objects create from them as arguments to function calls,
or returning them as tail calls, they cannot be wrapped into other Python objects. For example, returning a list with a
coroutine inside is not allowed. Tensorlake will not recognize the coroutine wrapped into the list.
This is the same restriction as with [Futures](/applications/futures#wrapping-futures-into-python-objects-is-not-allowed).
Map and reduce operations accept a Future/coroutine/`asyncio.Task` or a list as input items.
If a list is passed then the Futures/coroutines/asyncio tasks in the list are recognized by
Tensorlake and run automatically.
### Tail calls
Returning a Tensorlake function coroutine or its `asyncio.Task` makes a [tail call](/applications/futures#tail-calls).
The returning function completes immediately and its function container is freed to process the next request.
Tensorlake runs the returned coroutine or task and uses its result as the function's return value.
This works exactly like returning a Future as a tail call.
```python theme={null}
from tensorlake.applications import application, function
@function()
async def double(x: int) -> int:
return x * 2
@application()
@function()
async def my_app(x: int) -> int:
# Returns a coroutine as a tail call. The function completes immediately
# and Tensorlake runs the coroutine in the background.
return double(x)
```
Futures can also be returned as tail calls from async functions.
```python theme={null}
from tensorlake.applications import application, function
@function()
def double(x: int) -> int:
return x * 2
@application()
@function()
async def my_app(x: int) -> int:
return double.future(x)
```
### Calling sync functions from async functions
Sync Tensorlake functions can be called directly from async functions. The call blocks the asyncio event loop
until the sync function completes. No other asyncio tasks can run while the asyncio event loop is blocked.
Because of this, calling sync Tensorlake functions directly is an anti-pattern and should be avoided.
Use `function.future()` to call sync functions without blocking the event loop. Call `future.run()` to start the Future
in the background. Use `await future` to wait for the Future to complete and get its result. If this doesn't fit the use case,
use `future.coroutine()` to convert the Future into a coroutine that can be used the same way as any coroutine returned by
an async Tensorlake function.
```python theme={null}
from tensorlake.applications import application, function, Future
@function()
def sync_double(x: int) -> int:
return x * 2
@application()
@function()
async def my_app(x: int) -> int:
# Simple await of the sync function call.
return await sync_double.future(x)
@application()
@function()
async def my_app_tail_call(x: int) -> int:
# Tail call.
return sync_double.future(x)
@application()
@function()
async def my_app_background_task(x: int) -> int:
double_task: asyncio.Task = asyncio.create_task(sync_double.future(x).coroutine())
return await double_task
```
### Calling async functions from sync functions
Sync functions cannot `await` coroutines. To call an async Tensorlake function from a sync function,
use `function.future()` to create a Future and call `.result()` to block until it completes.
```python theme={null}
from tensorlake.applications import application, function
@function()
async def async_double(x: int) -> int:
return x * 2
@function()
async def async_add(a: int, b: int) -> int:
return a + b
@application()
@function()
def my_app(x: int) -> int:
doubled: int = async_double.future(x).result()
return async_add.future(x, doubled).result()
```
## See Also
Use Futures for parallel execution and tail calls.
Parallel processing over lists of data.
# Building Workflows
Source: https://docs.tensorlake.ai/applications/building-workflows
Build multi-step data workflows with parallel execution and optimized resource usage
Data workflows involve multiple steps: fetching, transforming, validating, enriching, and loading data. Tensorlake lets you define these pipelines as composed functions that automatically run in parallel where possible, with built-in durability and resource optimization.
Your workflows are exposed as HTTP endpoints, that can be called on-demand. They scale up when they are called, and scale down when they are idle.
## Your First Workflow
Workflows in Tensorlake use **futures** to define function calls without executing them immediately. This allows Tensorlake to optimize execution by running independent steps in parallel. When you return a future from a function (called a **tail call**), the function completes immediately without blocking, and Tensorlake orchestrates the remaining work.
Here's a simple workflow that processes and formats data from multiple sources:
```python theme={null}
from tensorlake.applications import application, function
@application()
@function()
def enrich_record(record_id: str) -> dict:
# Create futures - these don't run yet, just define the function calls
profile = fetch_profile.future(record_id)
history = fetch_history.future(record_id)
# Return a tail call - enrich_record() completes immediately without blocking
# Tensorlake then automatically:
# 1. Runs fetch_profile() and fetch_history() in parallel (no dependencies between them)
# 2. Once both complete, runs merge_data() with their results
# 3. Uses merge_data()'s return value as enrich_record()'s final result
return merge_data.future(profile, history)
@function()
def fetch_profile(record_id: str) -> dict:
# Fetch from profile service
return {"id": record_id, "name": "Example Corp", "tier": "enterprise"}
@function()
def fetch_history(record_id: str) -> list:
# Fetch transaction history
return [{"date": "2024-01-15", "amount": 5000}]
@function()
def merge_data(profile: dict, history: list) -> dict:
return {"profile": profile, "transactions": history}
```
**What happens when you call this workflow:**
```bash theme={null}
curl https://api.tensorlake.ai/applications/enrich_record \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
--json '"rec_123"'
```
1. `enrich_record` starts and immediately returns (doesn't block)
2. `fetch_profile("rec_123")` and `fetch_history("rec_123")` run **in parallel**
3. When both complete, `merge_data` runs with both results
4. Final response contains the merged data
**Key benefits:**
* **Parallel execution** where possible (lower latency)
* **No blocking**: the orchestrator container is freed immediately
* **Automatic dependency tracking**: no manual coordination needed
* **Built-in durability**: failures resume from checkpoints
For a deep dive on futures and tail calls, see [Futures](/applications/futures#tail-calls).
See [async functions](/applications/async-functions) on how to build non-blocking workflows using Python async/await.
Each function in your workflow can be configured with retry policies. If a step fails, Tensorlake automatically retries it based on your [retry configuration](/applications/retries).
## Best Practices
### Design for Parallelism
Identify steps that can run independently:
```python theme={null}
# Sequential: slow
@function()
def slow_pipeline(data: str) -> str:
result1 = step1(data)
result2 = step2(data) # Could have run in parallel
return combine(result1, result2)
# Parallel: fast
@function()
def fast_pipeline(data: str) -> str:
result1 = step1.future(data)
result2 = step2.future(data) # Runs in parallel with step1
return combine.future(result1, result2)
```
### Use Tail Calls for Efficiency
Return futures instead of blocking. When you return a future as a tail call, the current function's container is freed immediately, so you're not paying for idle containers waiting for downstream results.
```python theme={null}
# Blocks container unnecessarily
@function()
def inefficient(data: str) -> str:
result = expensive_operation(data) # Container blocked here
return result
# Frees container immediately
@function()
def efficient(data: str) -> str:
return expensive_operation.future(data) # Container freed right away
```
### Process Lists with Map-Reduce
For workflows that process collections of items, use map-reduce operations to parallelize the work:
```python theme={null}
from pydantic import BaseModel
class ProcessingResult(BaseModel):
total_processed: int = 0
total_value: float = 0.0
@application()
@function()
def process_batch(record_ids: list[str]) -> ProcessingResult:
# Map: process each record in parallel
results = process_record.future.map(record_ids)
# Reduce: aggregate results as they complete
return aggregate_results.future.reduce(results, ProcessingResult())
@function()
def process_record(record_id: str) -> dict:
# Each record processed in its own container
return {"id": record_id, "value": 100.0}
@function()
def aggregate_results(summary: ProcessingResult, record: dict) -> ProcessingResult:
summary.total_processed += 1
summary.total_value += record["value"]
return summary
```
Map-reduce operations automatically run in parallel and scale to handle large datasets efficiently. See [Map-Reduce](/applications/map-reduce) for more details.
## Learn More
Deep dive on futures, tail calls, and parallel execution.
Async functions are another way to define workflows with parallel execution.
How workflows recover from failures.
# SDK Reference
Source: https://docs.tensorlake.ai/applications/concepts
Functions, applications, decorators, request context, and lifecycle reference
## Applications
Applications are the top-level decorators that define entry points for your applications. You can define as many applications as you want in
your project. Each one of them will be assigned a unique HTTP entry point based on the name of the Python function.
```python theme={null}
from tensorlake.applications import application, function
# This application's name will be `hello_world`.
@application()
@function()
def hello_world():
print("Hello, world!")
# This application's name will be `hola_mundo`.
@application()
@function()
def hola_mundo():
print("Hola, mundo!")
```
### Configuring Applications
The `@application` decorator allows you to specify the following attributes:
1. `tags` - dict of tags to categorize the application.
2. `retries` - Retry policy for every function in the application unless a function specifies its own retry policy. No retries by default if function failed. See [Retries](/applications/concepts#retries).
3. `region` - The region where every function in the application will be deployed unless a function specifies its own region. Either `us-east-1` or `eu-west-1`. The default is any of the regions.
4. `allow` - List of application capabilities to enable. The supported capability is `unauthenticated_requests`, which creates a public endpoint that you can invoke without a Tensorlake API key. See [Public endpoints](/applications/public-endpoints).
The following code snippet shows an example with custom application metadata, retries, and region:
```python theme={null}
from tensorlake.applications import application, function, Retries
@application(
tags={"language": "python"},
retries=Retries(max_retries=3),
region="us-east-1",
)
@function()
def hello_world():
print("Hello, world!")
```
### Application inputs and output
Application functions take zero or more arguments which are the current request inputs.
The inputs get deserialized from their JSON representation into Python objects specified
in the arguments' type hints. A reverse process happens for the request output.
The object returned from the application function gets JSON serialized according to the return
type hint of the function. The resulting JSON is returned as the HTTP response body of the application request.
For example if your application function takes a single `str` argument and returns a `str`, then the
request input and output should be JSON strings:
```python theme={null}
from tensorlake.applications import function, application
@application()
@function()
def greet(data: str) -> str:
return data + " from greet!"
```
```json request input theme={null}
"Hello, world!"
```
```json request output theme={null}
"Hello, world! from greet!"
```
If you want to use multiple application request inputs with complex data structures,
you can add more arguments and use type hints with your Pydantic model classes.
Each type hint needs to be [supported in Pydantic JSON mode](https://docs.pydantic.dev/latest/concepts/serialization/#json-mode).
All basic type hints like `str`, `int`, `float`, `bool`, `list`, `dict`, `set`, `tuple`, `None`, `Any`, `|`,
Pydantic model classes and more are supported. If a type hint is a union of multiple Python types,
like `str | int`, then the request JSON input can match any of the types in the union.
```python theme={null}
from pydantic import BaseModel
from tensorlake.applications import application, function
class PersonSearchQuery(BaseModel):
name: str
age: int
class PersonSearchResult(BaseModel):
matches: list[dict]
@application()
@function()
def process_data(query: PersonSearchQuery | list[PersonSearchQuery], limit: int | None = None) -> PersonSearchResult:
if isinstance(query, list):
return PersonSearchResult(
matches=[
{"name": q.name, "age": q.age, "id": i+1} for i, q in enumerate(query) if limit is None or i < limit
]
)
else:
return PersonSearchResult(matches=[{"name": query.name, "age": query.age, "id": 1}])
```
```json request input theme={null}
{"name": "John", "age": 30}
```
```json response output theme={null}
{"matches":[{"name":"John","age":30,"id":1}]}
```
The limit argument is optional so we can omit it from the request input.
If an argument type hint is `Any`, then the corresponding request input can be any valid JSON value
(string, number, object, array, boolean, null). The JSON value gets deserialized into the corresponding
Python object (`str`, `int`/`float`, `dict`, `list`, `bool`, `None`). The same applies to `Any` return type hint
(i.e. a Python dict gets serialized as JSON object, a Python list as JSON array, etc.).
If type hints are not provided then they are treated as `Any`.
### Calling Applications
You can call applications remotely using any HTTP client or Tensorlake Python SDK.
Use empty POST request body if the application takes no arguments.
A JSON serialized request body is passed if the application takes one argument.
Use multipart/form-data request body if the application takes multiple arguments
or one or more files (see [uploading and downloading files](/applications/concepts#uploading-and-downloading-files)).
i.e. to call the `hello_world` application defined above (takes no arguments):
```bash bash theme={null}
curl \
https://api.tensorlake.ai/applications/hello_world \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
--json ''
```
```python python theme={null}
from tensorlake.applications import run_remote_application
run_remote_application("hello_world")
```
i.e. to call the `greet` application defined above (takes a single string argument):
```bash bash theme={null}
curl \
https://api.tensorlake.ai/applications/greet \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
--json '"Hello, John"'
```
```python python theme={null}
from tensorlake.applications import run_remote_application
run_remote_application("greet", "Hello, John")
# Or:
# run_remote_application("greet", data="Hello, John")
```
i.e. to call the `process_data` application defined above (with multiple arguments):
```bash bash theme={null}
query_value='[{"name": "Alice", "age": 25}, {"name": "Bob", "age": 24}]'
limit_value='10'
curl \
https://api.tensorlake.ai/applications/process_data \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Accept: application/json" \
-F "query=$query_value;type=application/json" \
-F "limit=$limit_value;type=application/json"
```
```python python theme={null}
from tensorlake.applications import run_remote_application
run_remote_application(
"process_data",
query=[
PersonSearchQuery(name="Alice", age=25),
PersonSearchQuery(name="Bob", age=24)
],
limit=10
)
```
If you pass an argument and application function doesn't have it then it's simply ignored.
If an argument has a default value then you can omit it from the request input.
Both of these features make it easy to update application code without breaking existing clients.
### Uploading and downloading files
Application functions can receive files as current request inputs and return a file as a current request output.
This makes it easy to build applications which process input files or produce an output file. File sizes of up
to 5 TB are supported. The file input type is represented by the `File` class in Tensorlake SDK with the following
interface:
```python theme={null}
class File:
content: bytes # Raw bytes of the file
content_type: str # MIME content type of the file
```
When an argument has a `File` type hint, Tensorlake SDK doesn't attempt to deserialize the input from JSON and
instead passes a `File` object with original request input binary content and content type. When the return type
hint is `File`, Tensorlake SDK doesn't JSON serialize the returned `File` object. It instead sets the request
output content type to the `File.content_type` and the HTTP response body to the raw bytes in `File.content`.
```python theme={null}
from tensorlake.applications import function, application, File
@application()
@function()
def process_file(input: File) -> File:
print(
"Got file content type:",
input.content_type,
"size:",
len(input.content),
"bytes"
)
return File(
# HTTP response body is the raw bytes of the input file.
content=input.content,
# HTTP response content type is the same as input file content type.
content_type=input.content_type
)
```
`File.content` field holds the raw bytes of the file uploaded to the application endpoint.
At the moment, the SDK doesn't support lazy loading large files, so the entire file is loaded into memory
when the function is called.
To pass a local file at `/foo/bar/file_name.txt` path to an application, you need to use a multipart/form-data
HTTP request or just use Python SDK.
```bash bash theme={null}
input_value='@/foo/bar/file_name.txt'
curl \
https://api.tensorlake.ai/applications/process_file \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Accept: application/json" \
-F "input=$input_value"
```
```python python theme={null}
from tensorlake.applications import run_remote_application, File
# Note: File object from Tensorlake SDK is not the same as File object from Python standard library.
with open("/foo/bar/file_name.txt", "rb") as local_file:
local_file_content: bytes = local_file.read()
run_remote_application(
"process_file",
File(
content=local_file_content,
content_type="text/plain"
)
)
```
## Functions
Functions are the building blocks of applications. They are Python functions decorated with the `@function` decorator.
Tensorlake functions can call other Tensorlake functions. The function call blocks until the called function returns
its output to the calling function.
For example, a simple application function which calls another function to process its input:
```python theme={null}
from tensorlake.applications import application, function
# Define an application function which is an HTTP entry point for the application.
@application()
@function()
def greet(name: str) -> str:
if name.startswith("A"):
return "Hello, A-name!"
else:
# Call another function to perform a specific processing of non-A names.
return process_non_a_name(name) + " from greet!"
@function()
def process_non_a_name(name: str) -> str:
return "A" + name[1:]
```
Every Tensorlake function call:
* is executed in its own function container,
* supports durable execution,
* can run in parallel with other function calls,
* can be retried independently if it fails,
* has its own resource limits (CPU, memory, disk, GPU, timeout),
* has its logs available in Tensorlake logging tools,
* has its execution timeline available in Tensorlake tracing tools,
* can report progress updates to extend its timeout,
* can share state with other function calls of the same application request.
Every Python function decorated with `@function` becomes a Tensorlake function and thus gets
all these capabilities automatically.
### Configuring Tensorlake functions
The `@function` decorator allows you to set the following attributes:
1. `description` - A description of the function's purpose and behavior.
2. `cpu` - The number of CPUs available to the function. The default is `1.0` CPU. See [CPU](/applications/concepts#cpu).
3. `memory` - The memory GB available to the function. The default is `1.0` GB. See [Memory](/applications/concepts#memory).
4. `ephemeral_disk` - The ephemeral `/tmp` disk space available to the function in GB. The default is `2.0` GB. See [Ephemeral Disk](/applications/concepts#ephemeral-disk).
5. `gpu` - The GPU model available to the function. The default is `None` (no GPU). Please contact `support@tensorlake.ai` to enable GPU support.
6. `timeout` - The timeout for the function in seconds. The default is 5 minutes. See [Timeouts](/applications/concepts#timeouts).
7. `image` - The image to use for the function container. A basic Debian based image by default. See [Images](/applications/images).
8. `secrets` - The secrets available to the function in its environment variables. No secrets by default. See [Secrets](/applications/secrets).
9. `retries` - Retry policy for the function. No retries by default if function failed. See [Retries](/applications/concepts#retries).
10. `region` - The region where the function will be deployed. Either `us-east-1` or `eu-west-1`. The default is any of the regions.
The following code snippet shows an example of all the function attributes set to custom values.
```python theme={null}
from tensorlake.applications import function, Image, Retries
@function(
# Use Ubuntu as a base image instead of the default Debian
image=Image(base_image="ubuntu:latest"),
# Make my_secret available to the function as an environment variable
secrets=["my_secret"],
# Description of the function in the workflow
description="Measures the string using its length",
# Retry the function twice if it fails
retries=Retries(max_retries=2),
# Function fails if it was running for more than 30 seconds and didn't report any progress
timeout=30,
# 2 CPUs are available to the function
cpu=2,
# 4 GB of memory is available to the function
memory=4,
# 2 GB of ephemeral /tmp disk space is available to the function
ephemeral_disk=2,
# Run the function in a container with GPU support
gpu="H100",
# Run the function in us-east-1 region only
region="us-east-1",
)
def string_length(s: str) -> int:
return len(s)
```
### Function inputs and output
Tensorlake functions are not exposed as HTTP entry points unlike application functions.
Because of this Tensorlake functions have minimal limitations on their signatures.
Arguments and return value don't require any type hints but have to be picklable.
Most Python objects are picklable, except special cases like Processes, Threads,
database connections, etc.
If a Tensorlake function argument or return value is a Tensorlake SDK `File` object, then it
bypasses pickling and is passed as-is to and from Tensorlake functions.
### Application functions and Tensorlake Functions
Every application function decorated with `@application()` decorator is also a Tensorlake function.
This is why every application function needs to be decorated with `@function()` decorator in addition
to `@application()`.
As application functions are HTTP entry points into Tensorlake applications, they have some differences
compared to regular Tensorlake functions. Application functions:
* Require JSON serializable type hints for all arguments and the return value.
* Don't support `/` and `*` in function arguments.
* Don't support `*args` and `**kwargs`.
* Ignore function call arguments that are not defined in the function signature. This simplifies
code migrations, i.e. if HTTP client sends extra arguments that the application function doesn't
take anymore after its code update.
Application functions can be called from regular Tensorlake functions. The call is executed in the current
application request without creating a new one. So it behaves like a regular Tensorlake function call
inside an application.
### Classes
Sometimes a function needs one-time initialization, like loading a large model into memory.
This is achieved by defining a class using the `@cls` decorator. Classes use their `__init__(self)` constructor to run any initialization code once on function container startup.
The constructor can not have any arguments other than `self`. Any number of class methods can be decorated with `@function`.
```python theme={null}
from large_model import load_large_model
from tensorlake.applications import application, cls, function, run_remote_application
@cls()
class MyCompute:
def __init__(self):
# Run initialization code once on function container startup
self.model = load_large_model()
@application()
@function(cpu=4, memory=16)
def run(self, data: str) -> int:
return self.model.run(data)
if __name__ == "__main__":
run_remote_application("MyCompute.run", data="some input data")
```
### Timeouts
When a function runs longer than its timeout, it is terminated and marked as failed. The timeout in seconds is set using the `timeout` attribute.
The default timeout is `300` (5 minutes). Minimum is `1`, maximum is `172800` (48 hours). Progress updates can be sent by the function to extend the
timeout. See [Request Context](/applications/concepts#request-context).
```python theme={null}
from tensorlake.applications import function
# Set a 30 minute timeout for long-running agent tasks
@function(timeout=1800)
def deep_research(prompt: str) -> str:
...
```
### Retries
When a function fails by raising an exception or timing out, it gets retried according to its retry policy.
The default retry policy is to not retry the function call. You can specify a custom retry policy using the `retries` attribute.
If you allow retries, it's typically a best practice to ensure that the function is idempotent unless this is not required for your use case.
```python theme={null}
from tensorlake.applications import function, Retries
# Retry the function once if it failed
@function(retries=Retries(max_retries=1))
def my_function() -> int:
raise Exception("Something went wrong")
```
You can set default retry policy for all the functions in the application decorator. See the [Configuring Applications](/applications/concepts#configuring-applications) guide.
### Request Context
Functions can use a request context to share state between function calls of the same request. The context has information about the current request and provides access
to APIs for the current request. You can access the request context directly from the `RequestContext` class.
```python theme={null}
from tensorlake.applications import RequestContext, function
@function()
def my_function(data: str) -> int:
ctx: RequestContext = RequestContext.get()
print(f"Request ID: {ctx.request_id}")
...
```
#### Request ID
Each request has a unique identifier accessible via `ctx.request_id`. This is useful for logging and debugging.
```python theme={null}
from tensorlake.applications import RequestContext, function
@function()
def my_function(data: str) -> str:
ctx = RequestContext.get()
print(f"Processing request: {ctx.request_id}")
return ctx.request_id
```
#### Request Headers
A sanitized list of HTTP headers from the application invocation are available through `ctx.headers`. The collection is immutable and case-insensitive.
```python theme={null}
from tensorlake.applications import RequestContext, function
@function()
def inspect_request() -> dict:
headers = RequestContext.get().headers
return {
"content_type": headers.get("Content-Type"),
"event_type": headers.get("X-GitHub-Event"),
"signatures": headers.getlist("X-Hub-Signature-256"),
}
```
Use `headers[name]` when a header is required, `headers.get(name)` when it is optional, and `headers.getlist(name)` to retrieve every value of a repeated header.
Tensorlake removes sensitive and connection-specific headers before creating the request context. Removed headers include `Authorization`, `Authentication`, `Cookie`, `Host`, proxy credentials and hop-by-hop headers.
See [Public endpoints](/applications/public-endpoints) for an end-to-end signature verification example.
#### Request State
The state API allows you to set and get key-value pairs scoped per request. Each new request starts with an empty state. Values can be any picklable object.
| Method | Description |
| ----------------------------------------------------------------- | ------------------------------------------------ |
| `state.set(key: str, value: Any) -> None` | Set a key-value pair |
| `state.get(key: str, default: Any \| None = None) -> Any \| None` | Get a value by key, returns default if not found |
```python theme={null}
from tensorlake.applications import RequestContext, function
@function()
def first_function(data: str) -> int:
ctx = RequestContext.get()
ctx.state.set("user_data", data)
ctx.state.set("processed", True)
return second_function()
@function()
def second_function() -> int:
ctx = RequestContext.get()
data = ctx.state.get("user_data")
return len(data)
```
#### Streaming Progress Updates
The progress API allows you to stream execution progress from your functions. This is useful for monitoring long-running tasks and providing real-time feedback to users.
```python theme={null}
from tensorlake.applications import RequestContext, function
@function()
def process_items(items: list) -> dict:
ctx = RequestContext.get()
results = []
for i, item in enumerate(items):
ctx.progress.update(i + 1, len(items), f"Processing item {i + 1}")
results.append(process(item))
return {"results": results}
```
**Key features:**
* **Automatic timeout reset** - Each progress update resets the function timeout, allowing long-running agents to run indefinitely
* **Real-time streaming** - Stream updates to frontends via Server-Sent Events (SSE)
* **HTTP API access** - Query progress updates programmatically for custom dashboards and monitoring
See the [Streaming Progress guide](/applications/guides/streaming-progress) for detailed API reference, frontend integration examples, and best practices.
#### Request Metrics
The metrics API allows you to record custom metrics for monitoring and debugging.
| Method | Description |
| ------------------------------------------------------- | ------------------------------------------------------------------ |
| `metrics.timer(name: str, value: int \| float) -> None` | Record a duration metric (in seconds) |
| `metrics.counter(name: str, value: int = 1) -> None` | Increment a counter by the given value. Every counter starts at 0. |
```python theme={null}
import time
from tensorlake.applications import RequestContext, function
@function()
def my_function(data: str) -> int:
ctx = RequestContext.get()
start_time = time.monotonic()
result = len(data)
# Record metrics
ctx.metrics.timer("processing_time", time.monotonic() - start_time)
ctx.metrics.counter("items_processed", result)
return result
```
### CPU
The number of CPUs available to the function is set using the `cpu` attribute. Minimum is `1.0`, maximum is `8.0`.
The default is `1.0`. This is usually sufficient for functions that only call external APIs and do simple data processing.
Adding more CPUs is recommended for functions that do complex data processing or work with large datasets.
If functions use large multi-gigabyte inputs or produce large multi-gigabyte outputs, then at least 3 CPUs are recommended.
This results in the fastest download and upload speeds for the data.
```python theme={null}
from tensorlake.applications import function
# Allocate 4 CPUs for data processing
@function(cpu=4)
def process_data(data: list) -> dict:
...
```
### Memory
GB memory available to the function is set using the `memory` attribute. Minimum is `1.0`, maximum is `32.0`.
The default is `1.0`. This is usually sufficient for functions that only call external APIs and do simple data processing.
Adding more memory is recommended for functions that do complex data processing or work with large datasets.
It's recommended to set `memory` to at least 2x the size of the largest inputs and outputs of the function.
This is because when the inputs/outputs are deserialized/serialized both serialized and deserialized representations are
kept in memory.
```python theme={null}
from tensorlake.applications import function
# Allocate 8 GB memory for loading large models
@function(memory=8)
def run_model(data: str) -> str:
...
```
### Ephemeral disk
Ephemeral disk space is a temporary storage space available to functions at `/tmp` path. It gets erased when its
function container gets terminated. It's optimal for storing temporary files that are not needed after the function
execution is completed. Ephemeral disks are backed by fast SSD drives. Using other filesystem paths like `/home/ubuntu`
for storing temporary files will result in slower performance. Temporary files created using Python modules like `tempfile`
are stored in ephemeral disk space inside `/tmp`.
GB of ephemeral disk space available to the function is set using `ephemeral_disk` attribute. Minimum is `2.0`, maximum is `50.0`.
The default is `2.0` GB. This is usually sufficient for functions that only call external APIs and do simple data processing.
If the function needs to temporarily store large files or datasets on disk, then the `ephemeral_disk` attribute should be increased
accordingly.
```python theme={null}
from tensorlake.applications import function
# Allocate 20 GB disk for downloading and processing large files
@function(ephemeral_disk=20)
def process_video(url: str) -> str:
# Download video to /tmp
# Process and return results
...
```
# Crash Recovery
Source: https://docs.tensorlake.ai/applications/crash-recovery
How agents survive failures and resume without losing work
**Durable Execution is in Technical Preview**
This feature is currently in technical preview and under active development. Please contact us on Slack if you'd like to ask a question or try it out.
Agents call LLMs, scrape websites, query databases, and invoke external APIs. Any of these can fail: rate limits, timeouts, transient network errors, OOM kills. Without durability, a failure means restarting the entire agent from scratch, repeating every LLM call and API request.
Tensorlake checkpoints every `@function()` call automatically. When a request fails, you [replay](/applications/durability#request-replay-api) it and only the failed step re-executes. Everything before it is served from the checkpoint.
This page covers the recovery patterns. For automatic retries on transient failures (rate limits, validation errors), see [Retries & Rate Limits](/applications/retries). For long-running functions that need to extend their deadline as they make progress, see [Timeouts](/applications/timeouts). For try/except patterns and graceful degradation, see [Error Handling](/applications/error-handling).
## Why LLM Calls Must Be Durable
LLM calls are unlike normal API calls. They are **non-deterministic**: the same prompt can produce a different response on every invocation. This makes re-execution dangerous, not just wasteful.
Consider a travel agent that plans a trip. On the first run, the LLM decides on flights to Whistler. The agent books the flights, then crashes while searching for hotels. Without durable execution, the agent restarts from scratch. This time the LLM decides on Japan instead. Now the user has unwanted Whistler flights and a completely different trip plan.
Making LLM calls durable solves three problems at once:
* **Consistency**: Prior LLM decisions are preserved on replay. The agent resumes searching for Whistler hotels, not re-planning the entire trip.
* **Cost**: LLM inference is expensive. Re-executing 14 successful tool-calling iterations because the 15th failed wastes tokens and money.
* **Rate limits**: Agentic applications multiply downstream calls by an order of magnitude. Re-executing all of them increases the chance of hitting rate limits again.
On Tensorlake, every `@function()` call is automatically checkpointed. When a request is replayed, previously successful LLM calls return their recorded outputs. The model is not called again.
## Durable Tool Calls
The most common agent pattern is a loop: the LLM decides which tool to call, the tool runs, the result feeds back into the LLM. Each iteration is an expensive operation: an LLM inference plus a tool execution.
Wrap each tool in its own `@function()` to make every tool call a checkpoint:
```python theme={null}
from tensorlake.applications import application, function, Image
llm_image = Image().run("pip install openai")
@function()
def search_web(query: str) -> list[dict]:
import requests
response = requests.get("https://api.search.com/v1/search", params={"q": query})
return response.json()["results"]
@function()
def read_document(url: str) -> str:
import requests
return requests.get(url).text
@function(image=llm_image)
def call_llm(messages: list[dict]) -> dict:
from openai import OpenAI
response = OpenAI().chat.completions.create(
model="gpt-4o",
messages=messages,
tools=[
{"type": "function", "function": {"name": "search_web", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}},
{"type": "function", "function": {"name": "read_document", "parameters": {"type": "object", "properties": {"url": {"type": "string"}}}}},
]
)
return response.choices[0].message
@application()
@function(image=llm_image, timeout=1800)
def research_agent(topic: str) -> str:
tools = {"search_web": search_web, "read_document": read_document}
messages = [{"role": "user", "content": f"Research this topic: {topic}"}]
for _ in range(20): # max iterations
response = call_llm(messages) # checkpointed
messages.append(response)
if not response.get("tool_calls"):
return response["content"]
for tool_call in response["tool_calls"]:
fn = tools[tool_call["function"]["name"]]
result = fn(**tool_call["function"]["arguments"]) # checkpointed
messages.append({"role": "tool", "content": str(result), "tool_call_id": tool_call["id"]})
return messages[-1].get("content", "Max iterations reached")
```
If the agent crashes on iteration 15, a replay skips the first 14 iterations entirely. The LLM calls, web searches, and document reads from those iterations are all served from checkpoints. The agent resumes from iteration 15 with the full conversation history intact.
## Surviving Partial Failures in Fan-Out
When you process a batch of items in parallel using [map](/applications/map-reduce), each item is an independent function call with its own checkpoint. If 3 out of 1,000 items fail, replay only re-processes those 3.
```python theme={null}
from tensorlake.applications import application, function
@function(timeout=120)
def process_document(doc_url: str) -> dict:
"""Parse a single document. Each call is independently checkpointed."""
content = fetch_and_parse(doc_url)
extracted = extract_fields(content)
return extracted
@function()
def aggregate_results(results: list[dict], acc: dict) -> dict:
"""Combine results as they arrive."""
acc["documents"].append(results)
return acc
@application()
@function()
def batch_processor(doc_urls: list[str]) -> dict:
results = process_document.map(doc_urls)
summary = results.reduce(aggregate_results, {"documents": []})
return summary
```
This is the pattern behind durable data ingestion pipelines. Whether you're processing SEC filings, insurance forms, or research papers, partial failures don't lose the work already completed.
## Idempotent Side Effects
When a function sends an email, charges a credit card, or writes to an external database, you don't want that action repeated on replay. Wrap the side effect in its own `@function()`. Since the function's output is checkpointed, replay skips it entirely.
```python theme={null}
@function()
def send_notification(user_id: str, message: str) -> str:
"""Send once, skip on replay."""
response = email_api.send(user_id, message)
return response.message_id
@application()
@function()
def onboarding_agent(user_id: str) -> str:
profile = build_profile(user_id)
send_notification(user_id, f"Welcome, {profile['name']}!") # sent once
return setup_account(profile)
```
## Functions That Must Always Run Fresh
Some function calls should never be replayed from a checkpoint because they need live data every time. Mark them with `durable=False`:
```python theme={null}
@function(durable=False)
def get_current_price(ticker: str) -> float:
"""Always fetches the latest price, even on replay."""
return stock_api.get_price(ticker)
@function()
def get_historical_data(ticker: str) -> list[dict]:
"""Historical data doesn't change, so it is safe to checkpoint."""
return stock_api.get_history(ticker, days=30)
```
See [Disabling durable execution](/applications/durability#disabling-durable-execution) for the full implications.
## When to Use Durable Execution
| Scenario | Benefit |
| ----------------------------------------------- | --------------------------------------------- |
| Agent loops with 10+ tool calls | Crash on call #N resumes from #N, not #1 |
| Batch processing 100s-1000s of documents | Partial failures only re-process failed items |
| Pipelines with expensive LLM calls | No repeated inference costs on retry |
| Multi-step workflows with external side effects | Emails, payments, API calls aren't duplicated |
For the technical details of how checkpointing, fingerprinting, and replay modes work, see [Durable Execution](/applications/durability).
## Related Guides
Replay API, adaptive vs. strict modes, and fingerprinting internals.
Auto-retry on rate limits, validation errors, and transient failures.
Heartbeat-based timeouts so long agent loops don't fail prematurely.
Try/except, futures, and degrading gracefully when a step fails.
# Cron Scheduler
Source: https://docs.tensorlake.ai/applications/cron-scheduler
Schedule recurring invocations of your Orchestration endpoints.
Cron schedules trigger your deployed orchestration endpoints on a recurring basis. You can manage schedules programmatically via the API or through the Applications UI.
## Creating a Schedule
Send a `POST` request to create a cron schedule for a deployed application. The schedule starts immediately after creation.
```
POST /applications/{application}/cron-schedules
```
```python Python theme={null}
import requests, base64, json
application = "my-app"
payload = {"cron_expression": "0 * * * *"} # Every hour
# Optional: pass input data to each invocation
input_data = json.dumps({"report_type": "daily"}).encode()
payload["input_base64"] = base64.b64encode(input_data).decode()
response = requests.post(
f"https://api.tensorlake.ai/applications/{application}/cron-schedules",
json=payload,
headers={"Authorization": "Bearer TENSORLAKE_API_KEY"},
)
response.raise_for_status()
schedule_id = response.json()["schedule_id"]
print(f"Created schedule: {schedule_id}")
```
```typescript TypeScript theme={null}
async function createCronSchedule(
application: string,
cronExpression: string,
inputBytes?: Uint8Array,
) {
const body: Record = { cron_expression: cronExpression };
if (inputBytes) {
body.input_base64 = btoa(String.fromCharCode(...inputBytes));
}
const res = await fetch(
`/applications/${application}/cron-schedules`,
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
},
);
if (!res.ok) {
const err = await res.json();
throw new Error(err.error ?? `HTTP ${res.status}`);
}
const { schedule_id } = await res.json();
return schedule_id as string;
}
```
### Request fields
| Field | Type | Required | Description |
| ----------------- | ------ | -------- | -------------------------------------------------------------------------------- |
| `cron_expression` | string | Yes | A valid 5-field cron expression |
| `input_base64` | string | No | Base64-encoded bytes passed as input on every invocation. Maximum 1 MiB decoded. |
The response returns a `schedule_id`. Save this. It is required to delete the schedule later.
The minimum allowed interval is 60 seconds. `* * * * *` (every minute) is the fastest supported expression. Sub-minute expressions are rejected with a `400` error.
## Listing Schedules
Retrieve all cron schedules for an application:
```
GET /v1/namespaces/{namespace}/applications/{application}/cron-schedules
```
```python Python theme={null}
response = requests.get(
f"https://api.tensorlake.ai/applications/{application}/cron-schedules",
headers={"Authorization": "Bearer TENSORLAKE_API_KEY"},
)
response.raise_for_status()
for schedule in response.json()["schedules"]:
print(schedule["id"], schedule["cron_expression"], schedule["next_fire_time_ms"])
```
```typescript TypeScript theme={null}
interface CronSchedule {
id: string;
application_name: string;
cron_expression: string;
next_fire_time_ms: number;
last_fired_at_ms: number | null;
created_at: number;
enabled: boolean;
}
async function listCronSchedules(application: string) {
const res = await fetch(
`/applications/${application}/cron-schedules`,
);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const { schedules } = await res.json();
return schedules as CronSchedule[];
}
```
### Response fields
| Field | Type | Description |
| ------------------- | -------------- | ------------------------------------------------------------------------------------ |
| `id` | string | Unique ID for this schedule |
| `application_name` | string | The application this schedule belongs to |
| `cron_expression` | string | The schedule expression as stored |
| `next_fire_time_ms` | number | Unix timestamp (ms) of the next scheduled invocation |
| `last_fired_at_ms` | number \| null | Unix timestamp (ms) of the last invocation. `null` if the schedule has never fired. |
| `created_at` | number | Monotonic counter for ordering, not a wall-clock timestamp. Do not display as a date |
| `enabled` | boolean | Always `true` (reserved for future use) |
`next_fire_time_ms` and `last_fired_at_ms` are standard Unix millisecond timestamps. In JavaScript: `new Date(next_fire_time_ms)`.
## Deleting a Schedule
```
DELETE /applications/{application}/cron-schedules/{schedule_id}
```
```python Python theme={null}
response = requests.delete(
f"https://api.tensorlake.ai/applications/{application}/cron-schedules/{schedule_id}",
headers={"Authorization": "Bearer TENSORLAKE_API_KEY"},
)
response.raise_for_status()
```
```typescript TypeScript theme={null}
async function deleteCronSchedule(
application: string,
scheduleId: string,
) {
const res = await fetch(
`/applications/${application}/cron-schedules/${scheduleId}`,
{ method: "DELETE" },
);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
}
```
Deletion is permanent. To modify a schedule, delete it and recreate it. You can reuse the `cron_expression` from the list response to pre-populate the new request.
## Limits
| Limit | Value |
| --------------------------------- | --------------- |
| Minimum interval | 60 seconds |
| Maximum schedules per application | 100 |
| Maximum input payload | 1 MiB (decoded) |
## Related
Monitor scheduled invocations alongside the rest of your application activity.
Configure automatic retries for functions triggered by the scheduler.
Pass secrets securely to functions that run on a schedule.
# Durable Execution
Source: https://docs.tensorlake.ai/applications/durability
Tensorlake automatically persists function outputs so retries and replays skip already-succeeded work, avoiding costly restarts of long-running agent workflows.
Agentic applications and AI workflows are often **long-running** (seconds to hours) and interact with **unreliable dependencies**
(LLMs, external APIs, tools). A failure in a dependency call requires implementing retry logic and restarting the agent or workflow
from scratch when out of retries. This can be costly and adds significant complexity and latency.
When running on Tensorlake, your application automatically saves outputs of every Tensorlake function call in the current application request.
This means that outputs of successful Tensorlake function calls will be available without re-execution when you [replay](#request-replay-api)
an application request after it failed. The same applies to [automatic retries](/applications/retries). When a Tensorlake function gets retried
and runs the same previously succeeded Tensorlake function calls again, it will use their saved outputs instead of re-executing them.
For a worked example of how durable execution lets agents survive crashes mid-loop, see [Crash Recovery](/applications/crash-recovery). For tuning retry counts, rate limits, and validation-driven retries on a single function, see [Retries & Rate Limits](/applications/retries). For functions that should keep running across long agent loops without tripping the timeout, see [Timeouts](/applications/timeouts).
Storing outputs of successful function calls in an application request and re-using the outputs in the same request without re-executing
the same function calls again is called **durable execution**. It works out-of-the-box for all Tensorlake applications.
**Durable execution** is in technical preview mode. Please [contact us on Slack](https://join.slack.com/t/tensorlakecloud/shared_invite/zt-32fq4nmib-gO0OM5RIar3zLOBm~ZGqKg)
if you'd like to ask a question or try it out.
## Request Replay API
You can use Request Replay API to restart a failed Tensorlake application request from where it failed without re-executing the previously successful Tensorlake function calls in it.
```bash bash theme={null}
curl \
"https://api.tensorlake.ai/applications/$APPLICATION_NAME/requests/$REQUEST_ID/replay" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
--json '{}'
```
```python python theme={null}
from tensorlake.applications import Request, get_remote_request
application_name: str = "my_durable_application"
request_id: str = "abc123def456ghi789"
request: Request = get_remote_request(application_name, request_id)
request.replay()
# Blocks until the request replay completes and prints request output.
print(request.output())
```
When you replay a request, Tensorlake doesn't create a new request. Instead, it re-runs the same request with the same request ID. The request runs again and the request
output is updated when the replay completes.
### Application code upgrade
When request gets replayed it runs the same application code version as in the previous run. You can upgrade it to the latest application code version by passing
`--json '{ "upgrade_to_latest_version": true }'` in HTTP replay API call or passing `request.replay(upgrade_to_latest_version=True)` in Python. This is handy if you fixed
a bug in your application code and want to re-run the request with the fix applied. If you replay with a code upgrade, please ensure that the latest application code
can handle the original request inputs. This typically requires backward compatibility implemented at your application function parameters level.
### Replay modes
Tensorlake detects when a replayed request follows a different execution path comparing to the original request run or any its past replays.
For example, a replayed request may execute a new function call if it uses a random number generator to do it:
```python theme={null}
import random
from tensorlake.applications import application, function
@function()
def foo():
print("foo")
@function()
def bar():
print("bar")
@application()
@function()
def my_workflow_app():
# succeeded in the original request run,
# skipped in the replayed run
foo()
if random.random() < 0.5:
# never called in the original request run,
# called in the replayed run, Tensorlake detects this
bar()
# ... more Tensorlake function calls
```
Other common causes of a replayed requests following a different execution path:
* Conditional execution of code depending on current time, database state, values returned by external APIs, etc.
* Changing order of Tensorlake function calls depending on duration of external API calls, LLM calls, etc (aka race conditions).
* Change of Tensorlake function calls in [upgraded application code](#application-code-upgrade).
For some applications, a replayed request following a different execution path is expected and acceptable and for others it is not. Tensorlake provides two
different replay modes to suite the needs of both types of applications. [Adaptive replay](#adaptive-replay) allows this scenario and [Strict replay](#strict-replay)
doesn't allow it and fails the replay if it happens.
#### Adaptive replay
By default, Tensorlake uses **adaptive replay**. In this mode, all new Tensorlake function calls are allowed to execute, even if the replayed request doesn't run
some function calls that were executed in the original request run or in previous replayed runs. This mode is useful when the user just wants to re-run the request
from where it failed without being concerned about potential behavioral changes or non-determinism in their application code.
To explicitly enable adaptive replay, pass `--json '{ "mode": "adaptive" }'` in HTTP replay API call or pass `request.replay(mode=ReplayMode.ADAPTIVE)` in Python.
This is not necessary since adaptive replay is the default mode.
#### Strict replay
In this mode, if a new Tensorlake function call is detected during the request replay and one or more Tensorlake function call from the original request run
or from previous replayed runs are not executed in the current replayed run, then the request replay fails with a `ReplayError`. This mode is useful when the
user wants to ensure that the request behavior remains the same during replays. i.e. that all the resources claimed during the original request run are reused
during the replayed run without claiming more resources again (i.e. to not redo cross-service transactions).
To enable strict replay, pass `--json '{ "mode": "strict" }'` in HTTP replay API call or pass `request.replay(mode=ReplayMode.STRICT)` in Python.
### How function calls are matched
Tensorlake makes a fingerprint of every Tensorlake function call made in an application request. It then compares fingerprints of new function calls made during
a request replay with fingerprints of previously executed function calls in the same request to determine whether the function call has been made previously.
A function call fingerprint includes:
* Function call type (i.e. "function\_call", "map", "reduce").
* Function name.
* Parent function call fingerprint.
* Function call sequence number in the parent function call.
* Other information to ensure that changes in function call tree structures are detected.
Function parameters are not included in the function call fingerprint.
Takeaways from this:
* Changing function parameters in application code doesn't affect replay behavior. A new function call with different
parameters still matches the previous function call. This enables seemless application code upgrades without affecting
replays.
* Passing different values (e.g., random numbers, current time) as function parameters doesn't affect replay behavior.
A function call with a different random number passed into it still matches its previous function call where the random
number was different.
* If sequence of function calls changed in the latest application code then the replayed function calls will not match
the previous function calls. In this case the replay behavior depends on the selected replay mode (adaptive or strict).
* If function calls are started in an arbitrary order (i.e. with a random delay) then the order of function calls would
differ between the original request run and the replayed run even without application code changes. In this case the
replay behavior depends on the selected replay mode (adaptive or strict). Application code should avoid arbitrary function
call ordering to ensure consistent behavior during request replays and reuse of previously completed work.
## Automatic retries
When a Tensorlake function call gets [retried automatically](/applications/retries), it uses the same durable execution mechanism
to re-use outputs of previously successful Tensorlake function calls from the same request. In this case, adaptive replay mode is always used.
See [Retries & Rate Limits](/applications/retries) for the `retries=` parameter, validation-driven retries, and `max_containers` × `concurrency` for capping concurrent calls. For broader exception-handling patterns (try/except, futures, graceful degradation), see [Error Handling](/applications/error-handling).
## Disabling durable execution
Durable execution is enabled by default for all Tensorlake functions. You can disable it for a function by setting the `durable` attribute to `False` in the `@function` decorator.
```python theme={null}
from tensorlake.applications import application, function
from magic_llm import ask_llm
@function(durable=False)
def get_current_weather() -> str:
return ask_llm("What's the weather now in San Francisco?")
```
Disabling durable execution for a function means that when its parent function calls are re-executed during request replays or automatic retries,
then the non-durable function calls will always be re-executed and their outputs will never be reused from previous executions. Disabling durability
is useful for functions that must always run fresh (e.g., functions that return current time or current weather or stock price). It's not recommended
to call other Tensorlake functions from a non-durable function because all such function calls will also be non-durable and will always be re-executed.
If the same functions are called from durable functions then their outputs will still be saved and reused as normal if the called functions are durable.
With strict replay mode, no validation is done on non-durable function calls. A non-durable function call and all its child Tensorlake function calls just get
re-executed on each replay.
## Best practices for durable Tensorlake applications
* Wrap every external call (LLM, API, database, etc.) in a Tensorlake function to make these
calls durable and avoid repeating work. If a framework is doing these calls then use framework customization
points (e.g., callbacks, hooks, decorators, etc.) to wrap the calls in Tensorlake functions.
* Design your application code to be deterministic to ensure that replays follow the same execution
path and thus reuse previously finished work.
* If your Tensorlake functions have external side effects (e.g., sending emails, modifying databases),
ensure that these side effects are idempotent or can be safely retried without causing issues.
* [Disable durability](#disabling-durable-execution) for functions that must always run on request replay or retry.
* If strict mode and code upgrade to latest are used in a replay then the latest application code needs to be fully
backward compatible with the original request code to avoid failing the replay.
## Human in the loop and external events
The replay API can be used for resuming requests that timed out while waiting for external inputs (e.g., human review, external event).
```python theme={null}
from tensorlake.applications import application, function, RequestError
from approval_system import wait_for_approval_request, get_approval_message, create_approval_request, ApprovalDeniedError
@function()
def create_approval(user_id: str, action: str) -> str:
"""Wraps create_approval_request call to make it durable."""
return create_approval_request(user_id, action)
@function(timeout=300) # 5 minute timeout
def wait_approval(approval_id: str) -> str:
"""Waits for an approval request to be completed and returns its approval message.
Raises RequestError if the approval was denied. This fails the current request immediately.
"""
try:
wait_for_approval_request(approval_id)
except ApprovalDeniedError as e:
raise RequestError(f"Approval '{approval_id}' was denied: {e}") from e
return get_approval_message(approval_id)
@application()
@function()
def user_authorization_workflow(user_id: str, action: str) -> str:
approval_id: str = create_approval(user_id, action)
approval_message: str = wait_approval(approval_id)
return f"Approval received: {approval_message}"
```
`wait_approval` function times out after waiting for 5 minutes. The wait can be resumed once the approval is granted by replaying the request
using the Request Replay API. The replayed request will skip the already completed `create_approval` function call and re-execute the
`wait_approval` function call which will now be able to complete successfully.
## Comparison with Temporal
Both Tensorlake and Temporal provide durable execution, they achieve it through different architectures. Temporal relies on event history replay,
whereas Tensorlake saves and retrieves function outputs and matches function calls using their fingerprints. This removes many constraints that
Temporal imposes on application code.
| Feature | Tensorlake Applications | Temporal |
| :------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **User Code Constraints** | **Adaptive.** By default, a replayed request can change its function calls. | **Strict Determinism.** Workflow logic must be perfectly deterministic or replay crashes. |
| **Handling Code Updates** | **Adaptive.** By default, Tensorlake adapts to new code. New function calls execute normally, and removed function calls are ignored. No special versioning logic is required. | **Complex.** Requires explicit "Versioning" logic (`workflow.patched()`) or creating new task queues to prevent "Non-Determinism Errors" when replays encounter new code. |
| **History Limits** | **Unlimited.** There are no event history size limits. You can have infinite loops or long-running applications without resetting execution state. | **Limited.** Event history has hard size limits (typically 50K events). Large loops or long-running workflows must use "Continue-As-New" to truncate history. |
| **Replay Behavior** | **Adaptive.** By default, if the code execution path deviates, Tensorlake simply executes the new path while reusing cached outputs where possible. | **Strict.** If the code execution path deviates from the saved history, the workflow fails (Block/Retry loop). |
| **Code Failures** | **Fails Fast.** If a function fails and runs out of retries, the request fails immediately, allowing you to debug and [Replay](#request-replay-api) it later when fixed. | **Blocks and Retries.** If a workflow task fails (e.g., a bug in logic), it blocks and retries indefinitely until fixed. |
| **Code Structure** | **Flexible.** You can structure your application code freely, using any programming constructs without worrying about replay constraints. | **Constrained.** You must split the code into workflows and activities and use them carefully to avoid non-determinism and ensure replayability. |
| **Strict Replay Mode** | **Available.** You can enable strict replay mode to enforce exact function calls matching during replays to avoid non-determinism. | **Available.** Temporal always enforces strict determinism in workflow code |
| **Non-durable Functions** | **Supported.** You can disable durability for specific functions that must always run fresh on replays or retries. | **Not Supported.** All external data must be recorded in history. Retrieving fresh data during replay is generally forbidden to prevent non-determinism. |
## Next
Walkthrough: how durable execution survives mid-agent crashes and partial fan-out failures.
Configure retry counts, validation-driven retries, and concurrency caps.
Bounded function timeouts with progress-update heartbeats.
Try/except patterns, future failures, and graceful degradation.
# Error handling
Source: https://docs.tensorlake.ai/applications/error-handling
How errors propagate in Tensorlake Applications, covering function exceptions, timeouts, retries, and patterns for building resilient agentic workflows.
Agentic applications interact with unreliable dependencies (LLMs, tools, external APIs). This guide explains how errors propagate in Tensorlake Applications and common patterns for building resilient workflows on the Agentic Runtime.
## How failures propagate
* **A function can fail** by raising an exception or timing out (see [Timeouts](/applications/timeouts)).
* **If an exception is not handled**, it bubbles up to the caller and can fail the overall request. Failed requests can be re-run with the [Replay API](/applications/durability#request-replay-api), and previously successful nested calls are served from [checkpoints](/applications/durability) instead of re-executing.
* **Retries** can be configured per-function or at the application level. See [Retries & Rate Limits](/applications/retries).
* **Mid-loop crashes** in long-running agents are covered in [Crash Recovery](/applications/crash-recovery).
## Pattern: catch errors and continue
Use `try/except` inside your application to decide whether to fail the request or degrade gracefully.
```python theme={null}
from tensorlake.applications import application, function
@function()
def call_tool(x: str) -> str:
# e.g., LLM/tool/API call that can fail
raise RuntimeError("tool failed")
@application()
@function()
def workflow(user_input: str) -> dict:
try:
tool_output = call_tool(user_input)
return {"status": "ok", "tool_output": tool_output}
except Exception as e:
# Decide how your agent/workflow should behave on failure
return {"status": "degraded", "error": str(e)}
```
## Pattern: retries for flaky dependencies
Retries are a good fit for transient failures (timeouts, 429s, temporary upstream errors). Configure them on the function (or set defaults on the application).
```python theme={null}
from tensorlake.applications import function, Retries
@function(retries=Retries(max_retries=3))
def flaky_step() -> str:
...
```
## Futures: handling parallel failures
When using [Futures](/applications/futures), errors are raised when you call `.result()`:
```python theme={null}
from tensorlake.applications import application, function, Future
@function()
def maybe_fails() -> str:
raise RuntimeError("boom")
@application()
@function()
def parallel_work() -> str:
fut: Future = maybe_fails.future().run()
try:
return fut.result()
except Exception as e:
return f"handled: {e}"
```
## Debugging tips
* **Start by reproducing locally**: Tensorlake Applications run as normal Python functions locally. See [Testing locally](/applications/quickstart#testing-locally).
* **Add structured logs**: log inputs/outputs (excluding secrets) so you can diagnose failures.
* **Make side effects idempotent**: if a function can retry, avoid double-charging or double-writing.
## Related Guides
Configure auto-retries for transient failures and structured-output validation.
Resume long agent loops from the failed step instead of restarting.
Replay API, adaptive vs. strict modes, and how checkpoints survive failures.
Per-function deadlines and progress-update heartbeats.
# Futures
Source: https://docs.tensorlake.ai/applications/futures
Use Futures to run multiple function calls in parallel to optimize resource usage and reduce latency.
A Future object defines, runs and tracks execution of a function call or another operation like map or reduce
It is created using the `function.future` factory. i.e. calling `my_function.future(1, 2, 3)` returns a Future object
for the `my_function(1, 2, 3)` function call. The Future doesn't start running until it's started with its `.run()` or
`.result()` methods, used as a function call argument, or returned from a function. `.result()` method blocks until the
Future completes and returns the value returned by the function call or raises an exception on failure.
```python theme={null}
from tensorlake.applications import application, function, Future
@application()
@function()
def my_application(name: str) -> str:
# Creates a Future object for the `capitalize(name)` function call and runs it immediately.
# `Future.run()` blocks the calling function to start the function call, not to finish it.
capitalized_name_future: Future = capitalize.future(name).run()
# `Future.result()` blocks until the `capitalize` function call completes.
# It returns the value returned by the function call or raises an exception on failure.
capitalized_name: str = capitalized_name_future.result()
return f"Hello, {capitalized_name}!"
@function()
def capitalize(text: str) -> str:
return text.upper()
```
The main purpose of Futures is to allow running multiple function calls in parallel and getting their results later.
This allows building applications that can process multiple independent tasks concurrently, reducing overall latency.
Class method `Future.wait(futures: Iterable[Future])` can be used to wait for multiple Futures to complete.
See more details at [waiting for multiple Futures to complete](#waiting-for-multiple-futures-to-complete).
### Example: Running multiple function calls in parallel
```python theme={null}
from tensorlake.applications import application, function, Future, RETURN_WHEN
@function()
def capitalize(text: str) -> str:
return text.upper()
@application()
@function()
def greet(name: str) -> str:
# Start two function calls in parallel.
capitalized_name: Future = capitalize.future(name).run()
joke: Future = make_joke.future(name).run()
# Wait for both function calls to complete.
Future.wait([capitalized_name, joke], return_when=RETURN_WHEN.ALL_COMPLETED)
# Call `say_hello_and_say_joke` with the values returned by both function calls.
# Block until `say_hello_and_say_joke` completes and return its return value.
return say_hello_and_say_joke(capitalized_name.result(), joke=joke.result())
@function()
def say_hello_and_say_joke(name: str, joke: str) -> str:
return f"Hello, {name}! Here's a joke for you: {joke}"
@function()
def make_joke(name: str) -> str:
return f"Why did {name} cross the road? To get to the other side!"
```
### Example: Non-blocking map and reduce operations
Use `function.future.map(...)` and `function.future.reduce(...)` to create Futures for map and reduce operations.
The arguments of these methods are the same for `function.map(...)` and `function.reduce(...)` described at
[Map-Reduce](/applications/map-reduce) page.
```python theme={null}
from tensorlake.applications import application, function, Future
@application()
@function()
def process_numbers(numbers: list[int]) -> int:
# Start a map operation to double the numbers in parallel with another function call.
doubled_numbers: Future = double_number.future.map(numbers).run()
# Start another function call in parallel.
log_processing.future(len(numbers)).run()
# Wait for the map operation to complete and get the doubled numbers.
doubled_numbers_result: list[int] = doubled_numbers.result()
# Make sure that log_processing call is completed.
log_processing.result()
# Start a reduce operation to sum the doubled numbers and return its result.
return sum.future.reduce(doubled_numbers_result).result()
@function()
def double_number(number: int) -> int:
return number * 2
@function()
def sum(a: int, b: int) -> int:
return a + b
@function()
def log_processing(count: int) -> None:
print(f"Processing {count} numbers")
```
### Waiting for multiple Futures to complete
`Future.wait` class method can be used to wait for multiple Futures to complete. This class method is inspired by the standard `concurrent.futures.wait` in Python.
It's full signature is:
```python theme={null}
from tensorlake.applications import Future, RETURN_WHEN
Future.wait(
futures: Iterable[Future],
timeout: float|None = None,
return_when=RETURN_WHEN.ALL_COMPLETED
) -> tuple[list[Future], list[Future]]
```
* `futures`: An iterable of Future objects to wait for.
* `timeout`: An optional timeout in seconds. If specified, the method will return after the timeout even if not all Futures have completed.
* `return_when`: A flag indicating when to return. It can be one of the following values from the `RETURN_WHEN` enum:
* `RETURN_WHEN.ALL_COMPLETED`: Wait until all Futures have completed.
* `RETURN_WHEN.FIRST_COMPLETED`: Wait until at least one Future has completed.
* `RETURN_WHEN.FIRST_EXCEPTION`: Wait until at least one Future has raised an exception or all have completed.
The method returns a tuple of two lists: `(done, not_done)`, where `done` is a list of Futures that have completed, and `not_done` is a list of Futures that have not completed yet.
If a future is not running yet, it's started automatically when passed to `Future.wait`.
### Future object
Future object has the following methods and properties:
* `exception -> TensorlakeError|None`: If the function call or another operation associated with this Future failed then this property will return the exception associated with the failure.
Otherwise, it will return `None`. If the operation is not yet complete, this property will also return `None`.
* `result(timeout: float|None = None) -> Any`: Blocks until the operation completes and returns the result of the operation (i.e. value returned by function call).
If the operation fails, the `FunctionError` will be raised. See more about [error handling](/applications/error-handling).
An optional timeout in seconds can be specified. If timeout is reached before the Future completes, a `TimeoutError` will be raised.
* `done() -> bool`: Returns `True` if the operation has completed (either successfully or with an exception), otherwise returns `False`.
* `run() -> Future`: Starts the Future's operation. Returns the same Future object for chaining. A Future that hasn't been started
with `.run()` will be started automatically when passed as another operation input or returned as a [tail call](#tail-calls).
* `__await__() -> Generator[Any]`: Allows awaiting the Future in async functions. This is equivalent to calling `.result()`, but the call will not block the
async event loop.
* `coroutine() -> Coroutine`: Converts the Future into a coroutine that can be used the same way as any coroutine returned by an async Tensorlake function.
Returns the same coroutine object if called multiple times on the same Future. Can only be called before a Future is started with `.run()`.
### Passing Futures as inputs
Futures can be passed as arguments to function calls. When a Future gets passed this way, Tensorlake automatically runs it if not running,
waits for the Future to complete and uses its result as the function call argument value. This allows building applications that can run
multiple function calls in parallel without blocking on their results until it's necessary.
```python theme={null}
from tensorlake.applications import application, function, Future
@function()
def double(x: int) -> int:
return x * 2
@function()
def add(a: int, b: int) -> int:
return a + b
@application()
@function()
async def my_app(x: int) -> int:
a: Future = double.future(x)
b: Future = double.future(x + 1)
# Pass Futures as function call arguments. Tensorlake runs both Futures in parallel,
# waits for them to complete, and uses their results as the arguments for `add`.
return add(a, b)
```
All input futures that don't depend on each other run in parallel, allowing Tensorlake to optimize resource usage and
reduce overall application latency. A function call or a map-reduce operation are only blocked while their input Futures
are running. Once all input Futures complete, Tensorlake automatically runs the function call or the map-reduce operation.
#### Wrapping Futures into Python objects is not allowed
When passing Futures as arguments to function calls, or returning them as tail calls,
the Futures cannot be wrapped into other Python objects. For example:
```python theme={null}
from tensorlake.applications import application, function, Future
@function()
def capitalize(text: str) -> str:
return text.upper()
@application()
@function()
def my_application(name: str) -> list[str]:
capitalized_name: Future = capitalize.future(name)
names: list[str | Future] = [capitalized_name, name]
# Passing Python list with a Future as an argument here is not allowed.
# Tensorlake will not recognize the Future wrapped into the list
# and will not run it or wait for it to complete.
return concat(names)
@function()
def concat(strings: list[str]) -> str:
return "".join(strings)
```
Map and reduce operations accept a Future or a list as input items.
If a list is passed then the Futures in the list are recognized by Tensorlake and run automatically.
### Tail calls
When a Tensorlake function calls another Tensorlake function or calls `future.result()`, the calling function blocks until the
function call or the future completes and returns its result.
Applications that make many of such calls can face multiple challenges:
1. **Wasted Resources**: While waiting for the result, the calling function container cannot perform other tasks while still consuming its compute resources.
2. **Higher Resource Usage**: More function containers are required to handle the same number of concurrent application requests if each request blocks multiple function containers.
3. **Higher Latency**: Sequential blocking function calls or `future.result()` calls can lead to increased overall latency, especially when multiple function calls are involved.
To address these challenges, Tensorlake introduced **Tail Calls**. A function makes a tail call when it returns a Future.
The result of the future, when available, becomes the return value of the function. Once the Future is returned, it immediately
starts running and frees the calling function container to process next tasks. This allows building applications that can run multiple
function calls in parallel without blocking on their results until it's necessary, significantly reducing overall latency and resource usage.
With tail calls the example `greet(...)` application doesn't have to wait for completion of any of its function calls.
`greet(...)` just returns almost immediately after telling Tensorlake what it needs to do for the request.
`greet(...)` then frees its container to process another request while Tensorlake is orchestrating the execution the most efficient
way possible. Once all function calls complete, Tensorlake will return the final result to the user.
```python theme={null}
from tensorlake.applications import application, function, Future
@function()
def capitalize(text: str) -> str:
return text.upper()
@function()
def make_joke(name: str) -> str:
return f"Why did {name} cross the road? To get to the other side!"
@function()
def say_hello_and_say_joke(name: str, joke: str) -> str:
return f"Hello, {name}! Here's a joke for you: {joke}"
@application()
@function()
def greet(name: str) -> str:
# Returns a future for `say_hello_and_say_joke(capitalize(name), make_joke(name))` function call.
# This is a tail call. `greet` doesn't block waiting for any of the function calls to complete.
# Once returned Tensorlake will run the Future and use `say_hello_and_say_joke` return value as the
# return value of `greet`. The `say_hello_and_say_joke` function call will run as soon as both its
# arguments are available. Both arguments are computed in parallel because they don't depend on each other.
capitalized_name: Future = capitalize.future(name)
joke: Future = make_joke.future(name)
return say_hello_and_say_joke.future(capitalized_name, joke=joke)
```
Same as with input futures, wrapping a Future returned from a function into another Python object is not allowed.
For example, returning a list with a Future inside is not allowed. Tensorlake will not recognize the Future wrapped into the list.
## See Also
Learn how to use async functions in Tensorlake applications.
Learn how to use map-reduce operations to run function calls in parallel and aggregate results.
# Logging
Source: https://docs.tensorlake.ai/applications/guides/logging
Emit logs from Tensorlake applications with `print`, the built-in application logger, or structlog for structured JSON logs that are easier to analyze and visualize.
You can print logs in your Tensorlake application to help you debug and monitor your application's behavior. Logs can be printed using the `print` function or by using a logging library such as `logging` or `structlog`.
We recommend you using structured logs for better analysis and visualization. They're usually JSON that contain key-value pairs, making them easier to parse. The following guide will help you configure `structlog` to take full advantage of structured logs in Tensorlake.
## Adding Structured Logs to Your Application
### Using Tensorlake's built-in application logger
The Tensorlake SDK provides a built-in application logger that outputs messages in a predefined JSON format. This logger is designed to be easy to use and provides a simple way to log messages with structured data.
To initialize it, you need to import the `Logger` class from the `tensorlake.applications` module and use the `get_logger` method:
```python theme={null}
from tensorlake.applications import Logger
logger = Logger.get_logger(module="my_app")
```
Then you can use it to log messages with structured data:
```python theme={null}
@application()
@function(description="An example of logging in Tensorlake")
def logging_example(name: str) -> str:
logger.info("User logged in", user_id=123)
return f"Hello, {name}. This is a logging example!"
```
You can also log exceptions as structured data by using the `exc_info=True` parameter:
```python theme={null}
@application()
@function(description="An example of logging in Tensorlake")
def logging_example(name: str) -> str:
try:
# some code that may raise an exception
except Exception:
logger.error("An error occurred", exc_info=True)
return f"Hello, {name}. This is a logging example!"
```
Finally, if you need to bind additional context to your logs, you can use the `bind()` method:
```python theme={null}
@application()
@function(description="An example of logging in Tensorlake")
def logging_example(name: str) -> str:
logger = logger.bind(user_id=123)
logger.info("User logged in")
logger.debug("Debug message")
return f"Hello, {name}. This is a logging example!"
```
### Using a custom StructLog configuration
If you don't want to use the Tensorlake's built-in application logger, you can use [structlog](https://www.structlog.org/en/stable/) to add structured logs to your application. Structlog is a Python library that provides a simple and flexible way to create structured logs.
To configure structlog to print JSON logs, including stack traces, we recommend using the following code:
```python theme={null}
import structlog
structlog.configure(
processors=[
structlog.stdlib.add_log_level, # Add log level
structlog.processors.TimeStamper(fmt="iso", key="timestamp", utc=True), # Add timestamp in RFC3339 format
structlog.processors.StackInfoRenderer(), # Add stack info for exceptions
structlog.processors.dict_tracebacks, # Formats exception info
structlog.processors.JSONRenderer(), # Render the log entry as JSON
],
cache_logger_on_first_use=True,
)
```
Before you start printing any logs in your library, you need to initialize the logger with the previous configuration. You can do this by calling the `structlog.get_logger()` function:
```python theme={null}
logger = structlog.get_logger("my-tensorlake-application")
```
After initializing the logger, you can start printing logs using the `logger` object inside your application. Look at this next example putting all the code together:
```python theme={null}
import structlog
from tensorlake.applications import (
application,
function,
)
# Configure structlog to output in JSON format
structlog.configure(
processors=[
structlog.stdlib.add_log_level, # Add log level
structlog.processors.TimeStamper(fmt="iso", key="timestamp", utc=True), # Add timestamp in RFC3339 format
structlog.processors.StackInfoRenderer(), # Add stack info for exceptions
structlog.processors.dict_tracebacks, # Formats exception info
structlog.processors.JSONRenderer(), # Render the log entry as JSON
],
cache_logger_on_first_use=True,
)
# Create a logger instance
logger = structlog.get_logger("logging_example")
@application()
@function(description="An example of logging in Tensorlake")
def logging_example(name: str) -> str:
logger.info("Logging example started", status="started")
logger.debug("Debugging the payload", name=name)
logger.warning("The application is about to crash")
try:
1 / 0
except ZeroDivisionError:
logger.error("Division by zero error", exc_info=True)
return f"Hello, {name}. This is a logging example!"
```
### Setting levels for your logs
By default, when you print any information with `print` in your application, we assign the level `INFO` to those logs.
Tensorlake supports the 5 standard levels of logging, `TRACE`, `DEBUG`, `INFO`, `WARNING`, and `ERROR`. These levels
are represented with numbers from Trace(1) to Error(5).
Our built-in application logger, as well as Structlog, provides helpers that will set the log level for you directly, like `logger.debug` and `logger.warning`.
To set the logging level manually, you have to print JSON objects that include a `level` attribute. We take the string representation of these
levels from the JSON objects and transform them into our internal representation:
```python theme={null}
@application()
@function(description="An example of logging in Tensorlake")
def manual_log_level_example(_name: str):
print('{"level": "DEBUG", "message": "Debugging the payload"}')
```
## Log retention
By default, all application logs are retained for 7 days. This retention period can be increased to 30 days or 1 year maximum.
If you want to increase the retention period contact Tensorlake support at `support@tensorlake.ai`.
## Visualizing the logs in Tensorlake's Dashboard
The logs that you print in your applications can be visualized in each application page of the [Tensorlake's Dashboard](https://cloud.tensorlake.ai).
That page allows you to filter logs by different parameters, like request IDs, function names, and logging levels.
## Get Application logs via API
Application logs are also accessible via the Tensorlake API. You can use `curl` or any other HTTP client to retrieve logs for your application. The following section explains how to do that:
```bash theme={null}
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/logs" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
**Response:**
```json theme={null}
{
"logs": [
{
"timestamp": 1717171717171717171,
"uuid": "550e8400-e29b-41d4-a716-446655440000",
"namespace": "my-namespace",
"application": "my-application",
"body": "Processing started for item 1",
"level": 3,
"logAttributes": "{\"level\": \"info\"}"
}
],
"nextToken": "1717171717171717172.550e8400-e29b-41d4-a716-446655440000"
}
```
### Filtering Logs
You can filter logs using query parameters to narrow down results:
**Filter by Request ID**
```bash theme={null}
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/logs?requestId={request_id}" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
**Filter by Function Name**
```bash theme={null}
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/logs?function={function_name}" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
**Filter system events out**
By default, we add system and application events to the logs, so you can keep track of the lifecycle of your requests.
Use `events` if you want to filter out system events:
```bash theme={null}
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/logs?events=3" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
**Filter by log levels**
Log levels are identified by numbers from Trace(1) to Error(5). By default, we show logs for all levels.
These are all the possible values for the different levels:
1. Trace
2. Debug
3. Info
4. Warning
5. Error
If you want to learn how to set these log levels, check out our [Logging reference](/applications/guides/logging).
```bash theme={null}
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/logs?level={level}" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
**Combine Filters**
Use the `gate` parameter to combine multiple filters with AND (default) or OR logic:
```bash theme={null}
# Get logs matching BOTH request ID AND function name
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/logs?requestId={request_id}&function={function_name}&gate=and" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
# Get logs matching EITHER request ID OR function name
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/logs?requestId={request_id}&function={function_name}&gate=or" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
### Pagination and Ordering
**Get Most Recent Logs (Default)**
By default, logs are returned in descending order (newest first). Use `tail` to specify the number of logs:
```bash theme={null}
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/logs?tail=50" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
**Get Oldest Logs First**
Use `head` to get logs in ascending order (oldest first):
```bash theme={null}
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/logs?head=50" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
**Paginate Through Logs**
Use the `nextToken` from the response to fetch the next page:
```bash theme={null}
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/logs?nextToken={next_token}" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
### Query Parameters Reference
| Parameter | Type | Description |
| ------------------ | ------------- | ----------------------------------------------------------- |
| `requestId` | String | Filter logs for specific request IDs |
| `function` | String | Filter logs for specific function names |
| `functionExecutor` | String | Filter logs for specific function executor containers |
| `functionRunId` | String | Filter logs for specific function runs |
| `allocationId` | String | Filter logs for specific allocations |
| `level` | Integer | Filter logs for specific log levels |
| `events` | Integer | Filter system and application events |
| `gate` | `and` \| `or` | Logic for combining multiple filters (default: `and`) |
| `head` | Integer | Number of logs to return in ascending order (default: 100) |
| `tail` | Integer | Number of logs to return in descending order (default: 100) |
| `nextToken` | String | Pagination token from previous response |
The parameter that filter logs (requestId, function, functionExecutor, functionRunId, allocationId, and level) can be repeated one or multiple times. If you add more than one parameter with the same name, Tensorlake will search for both parameters using the `gate` parameter as connector.
For example, filtering DEBUG and INFO logs:
```bash theme={null}
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/logs?level=2&level=3" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
# Progress Updates
Source: https://docs.tensorlake.ai/applications/guides/streaming-progress
Stream real-time progress updates from functions
## Progress API
### Getting the Request Context
First, get access to the request context in your function:
```python theme={null}
from tensorlake.applications import RequestContext, function
@function()
def my_function(data: str) -> str:
# Get the current request context
ctx = RequestContext.get()
# Now you can use ctx.progress.update()
ctx.progress.update(1, 10, "Starting processing...")
return "done"
```
### Method: `progress.update()`
Stream progress updates to monitoring systems and frontends.
```python theme={null}
ctx.progress.update(
current: int | float,
total: int | float,
message: str | None = None,
attributes: dict[str, str] | None = None
)
```
**Parameters:**
| Parameter | Type | Required | Description |
| ------------ | ------------------------ | -------- | ------------------------------------- |
| `current` | `int \| float` | ✅ Yes | Current step or percentage complete |
| `total` | `int \| float` | ✅ Yes | Total steps or 100 for percentage |
| `message` | `str \| None` | ❌ No | Human-readable progress message |
| `attributes` | `dict[str, str] \| None` | ❌ No | Additional metadata (key-value pairs) |
### Basic Usage
**Simple progress tracking:**
```python theme={null}
@function()
def process_items(items: list) -> dict:
ctx = RequestContext.get()
for i, item in enumerate(items):
# Update progress: current step, total steps
ctx.progress.update(i + 1, len(items))
process(item)
return {"processed": len(items)}
```
**With a message:**
```python theme={null}
@function()
def multi_step_workflow() -> str:
ctx = RequestContext.get()
ctx.progress.update(1, 3, "Fetching data from API...")
data = fetch_data()
ctx.progress.update(2, 3, "Processing data...")
processed = process_data(data)
ctx.progress.update(3, 3, "Storing results...")
store_results(processed)
return "complete"
```
**With additional metadata:**
```python theme={null}
@function()
def batch_processor(items: list) -> dict:
ctx = RequestContext.get()
errors = 0
for i, item in enumerate(items):
try:
process(item)
except Exception:
errors += 1
# Include metadata about the processing
ctx.progress.update(
current=i + 1,
total=len(items),
message=f"Processing item {i + 1}",
attributes={
"error_count": str(errors),
"success_rate": f"{((i + 1 - errors) / (i + 1) * 100):.1f}%"
}
)
return {"total": len(items), "errors": errors}
```
### Using Percentages
You can use percentages instead of step counts:
```python theme={null}
@function()
def long_operation() -> str:
ctx = RequestContext.get()
# 0-100 scale
ctx.progress.update(0, 100, "Starting...")
# 25% complete
ctx.progress.update(25, 100, "Quarter way through...")
# 50% complete
ctx.progress.update(50, 100, "Halfway done...")
# 100% complete
ctx.progress.update(100, 100, "Finished!")
return "done"
```
## Consuming Progress Streams
Progress updates are available through the Tensorlake API in real-time.
### Polling for Progress Updates
```bash theme={null}
# Get progress updates for a specific request
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/requests/{request_id}/progress" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
**Response:**
```json theme={null}
{
"current": 45,
"total": 100,
"message": "Processing batch 3 of 10",
"attributes": {
"batch_id": "batch_003",
"records_processed": "4500"
},
"timestamp": 1704067200000
}
```
## Learn More
Full context API reference.
# Container Images
Source: https://docs.tensorlake.ai/applications/images
Define per-function container images declaratively with Tensorlake's `Image` API. Set the base image, install Python and system packages, and customize per-deploy.
Tensorlake functions run in function containers. To install dependencies in the containers, we use container images that are built when you deploy an application.
Functions can use any Python or system packages installed into their container images. Tensorlake provides a declarative API
to define function container images with their dependencies.
## Defining Images
An image is defined using an `Image` object. You can modify the base image, run commands to install dependencies at build time, and modify other image attributes, like its name.
```python theme={null}
from tensorlake.applications import Image
image = (
Image(
name="my-pdf-parser-image",
base_image="ubuntu:24.04",
)
.run("apt update")
.run("pip install langchain")
)
```
```python theme={null}
from tensorlake.applications import function
@function(image=image)
def parse_pdf(pdf_path: str) -> str:
import langchain
# All the packages installed in the image are available inside the function.
# They need to be imported here because they might not be available
# in the Python environment used to deploy the application.
...
```
#### Default Base Image
We use a Debian based image `python:{LOCAL_PYTHON_VERSION}-slim-bookworm` as the default.
`LOCAL_PYTHON_VERSION` represents the Python version in your current Python environment.
#### Private Base Images
If your `base_image` lives in a **private** container registry, Tensorlake authenticates the pull using your local Docker config
(`$DOCKER_CONFIG/config.json`, default `~/.docker/config.json`), for example after `docker login`. Function images build through
the same path as sandbox images, so the setup is identical. See [Private Registries](/sandboxes/images#private-registries) for
supported auth types, CI setup, and the `DOCKER_CONFIG` scoping pattern.
# Introduction
Source: https://docs.tensorlake.ai/applications/introduction
Add serverless orchestration to any agent
Orchestrate is a serverless runtime for adding data orchestration capabilities to Agents. You can build orchestration APIs without deploying containers, workers or queues. Functions starts running when they are called and scale down to zero after finishing work.
Some use cases are -
1. Creating multi-stage tools that needs to be retried until they complete.
2. Data ingestion worklfow APIs.
3. Scale out processing using distributed map and reduce.
```python theme={null}
from tensorlake.applications import application, function
@function()
def summarize(doc: str) -> int:
summary = call_llm(doc)
return summary
@function()
def summarize_files(docs: List[str]) -> List[str]:
summaries = docs.map(summarize)
return summaries
```
1. Tensorlake functions are a unit of compute which is executed in a sandbox and retried based on a user provided retry policy.
2. Functions decorated with `@applications` becomes callable from external systems and exposed as HTTP APIs.
3. Function calls are automatically queued durably when they are called when there is not enough compute to handle the requests.
4. Each function’s inputs and outputs are check-pointed durably so they can be retried.
5. Every function can have different resource asks, making it possible to allocate more resources to functions which are more compute or memory intensive.
### Quickstart
Let's build a simple application that greets a user by name.
The `tl` CLI scaffolds and deploys applications. Install it with the install script:
```bash theme={null}
curl -fsSL https://tensorlake.ai/install | sh
```
Then install the Python SDK, which provides the `tensorlake` package your application code imports:
```bash theme={null}
pip install tensorlake
```
You can get an [API key](/platform/authentication#api-keys) from the Tensorlake Dashboard.
```bash theme={null}
export TENSORLAKE_API_KEY=
```
Applications are defined by Python functions. Let's start with a template, that greets a user by name.
```bash theme={null}
tl app new hello_world
```
This creates a file named `hello_world/hello_world.py` with the following content:
```python hello_world.py theme={null}
from tensorlake.applications import application, function
@application()
@function()
def greet(name: str) -> str:
return f"Hello, {name}!"
```
Deploy your application referencing your application's source file.
```bash theme={null}
tl app deploy hello_world/hello_world.py
```
## Invoke Orchestrate Functions
Orchestrate endpoints can be invoked using HTTP requests or the Python SDK.
### HTTP Endpoint
```
https://api.tensorlake.ai/applications/
```
```bash bash theme={null}
curl https://api.tensorlake.ai/applications/hello_world \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
--json '"John"'
# {"request_id":"beae8736ece31ef9"}
```
```python python theme={null}
from tensorlake.applications import run_remote_application, Request
request: Request = run_remote_application(greet, 'John')
print(request.id)
# "beae8736ece31ef9"
```
This will return a request ID that you can use to track the progress of your request.
Requests may run seconds to hours depending on your workload.
```bash bash theme={null}
curl -X GET https://api.tensorlake.ai/applications/hello_world/requests/{request_id} \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
# {
# "id":"B0IwzHibTTfn5mCXHPGsu",
# "outcome":"success",
# "failure_reason":null,
# "request_error":null,
# .... other fields ...
#}
```
```python python theme={null}
# You don't need to poll for request completion. Retrieving the output will wait for the request to complete.
```
The `outcome` field will be `success` or `failure` depending on whether the request completed successfully. It will be null if the request is still in progress.
```bash bash theme={null}
curl -X GET https://api.tensorlake.ai/applications/hello_world/requests/{request_id}/output \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
# "Hello, John!"
```
```python python theme={null}
from tensorlake.applications import run_remote_application, Request
request: Request = run_remote_application(greet, 'John')
output: str = request.output()
print(output)
# "Hello, John!"
```
## Testing Locally
Tensorlake Applications can run locally on your laptop. You can run them like regular python scripts.
```python hello_world.py theme={null}
# At the end of the file
from tensorlake.applications import run_local_application, Request
if __name__ == "__main__":
request: Request = run_local_application(greet, 'John')
output: str = request.output()
print(output)
# "Hello, John!"
```
Deploying agents which starts complex workflows, or multi-stage tool calls requires building complex distributed systems with queues, workers or orchestration
engines. It takes away time from focusing and building the agentic logic. Orchestrate helps to solve this problem by letting you write orchestration
endpoints and solves the coordination of functions, retries and autoscaling.
## Examples
Multi-agent research pipeline with parallel web search and report synthesis using OpenAI Agents SDK.
Execute LLM-generated code safely in isolated containers with data science libraries.
Claude agentic loop that chains tool calls, each running in its own isolated container.
Parse bank statements, categorize transactions, and answer spending questions with Claude.
Serverless web crawler that scrapes websites N levels deep using headless Chrome.
Conversational weather agent powered by Claude, deployed as an HTTP API.
## Next Steps
Follow our quick start guide to build and deploy a serverless agentic code interpreter in under 5 minutes.
# Map-Reduce
Source: https://docs.tensorlake.ai/applications/map-reduce
Use map-reduce patterns in Tensorlake Applications to parallelize large-scale ETL. Apply a function across items and aggregate the results.
*Map-Reduce* is supported by Tensorlake Applications to support large scale ETL of data.
**Map** is the process of applying a function to each item of a list in parallel.
**Reduce** is the process of aggregating the results of the map phase.
The example below visualizes mapping of a list of numbers to their squares and reducing the results by summing the squares:
```mermaid theme={null}
flowchart TD
inputs(1, 2, 3, 4, 5)
map(square.map)
inputs --> map
square1(1)
map --> square1
square2(4)
map --> square2
square3(9)
map --> square3
square4(16)
map --> square4
square5(25)
map --> square5
reduce1("sum(1, 4)")
square1 --> reduce1
square2 --> reduce1
reduce2("sum(5, 9)")
reduce1 --> reduce2
square3 --> reduce2
reduce3("sum(14, 16)")
reduce2 --> reduce3
square4 --> reduce3
reduce4("sum(30, 25)")
reduce3 --> reduce4
square5 --> reduce4
final_result(55)
reduce4 --> final_result
```
Tensorlake automatically parallelizes function calls across multiple function containers when you map a function to a list.
The reducer function is applied to each pair of mapped values sequentially in their original order in the list.
Tensorlake runs each reduce function call as soon as its input values are available.
## Blocking Map-Reduce
In the following code example, we calculate the square of each number and once we have all the squares, we sum them.
```python theme={null}
from pydantic import BaseModel
from tensorlake.applications import application, function
class TotalSum(BaseModel):
value: int = 0
@application()
@function()
def sum_squares(total_numbers: int) -> TotalSum:
# Blocks until all map calls complete.
# The behavior and signature of function.map is very similar to Python's built-in map except it's distributed and parallel.
squares: List[int] = square.map([i for i in range(total_numbers)])
# Blocks until all reduce calls complete.
# The behavior and signature of function.reduce is very similar to Python's functools.reduce except it's distributed.
total: TotalSum = sum_total.reduce(squares, TotalSum(value=0))
return total
@function()
def square(number: int) -> int:
return number ** 2
@function()
def sum_total(total: TotalSum, number: int) -> TotalSum:
total.value += number
# This value will be passed to the next sum_total call as the first argument.
# Unless this is the last call, in which case it will be returned as the final
# result of the reduce operation.
return total
```
## Non-blocking Map-Reduce
In the following code example, we calculate the square of each number and as soon as each square is available, we sum them.
This is achieved using [futures and tail calls](/applications/futures#tail-calls).
This reduces the overall duration of the Map-Reduce operation. The reduce function is still called sequentially in the original
order of the list.
```python theme={null}
from pydantic import BaseModel
from tensorlake.applications import application, function, Future
class TotalSum(BaseModel):
value: int = 0
@application()
@function()
def sum_squares(total_numbers: int) -> TotalSum:
# Defines map function calls but doesn't run them.
squares: Future = square.future.map([i for i in range(total_numbers)])
# Defines reduce function calls that will run as soon as each mapped value is available.
# Returns the reduce operation definition as a tail call. Tensorlake will take care of running it.
# The final value of the reduce operation will be assigned as the request output like if this function
# returns it here.
return sum_total.future.reduce(squares, TotalSum(value=0))
@function()
def square(number: int) -> int:
return number ** 2
@function()
def sum_total(total: TotalSum, number: int) -> TotalSum:
total.value += number
# This value will be passed to the next sum_total call as the first argument.
# Unless this is the last call, in which case it will be returned as the final
# result of the reduce operation.
return total
```
## Inputs
### List
Both map and reduce operations accept a list as operation inputs.
Each item in the list can be a value, a Future, a Tensorlake coroutine, or an `asyncio.Task` object.
Tensorlake recognizes these Futures/coroutines/`asyncio.Task` objects, runs them automatically, and uses their
results as the input values for the operation.
```python theme={null}
from tensorlake.applications import application, function, Future
@function()
def double(number: int) -> int:
return number * 2
@function()
def sum(a: int, b: int) -> int:
return a + b
@application()
@function()
def sum_doubled(numbers: list[int]) -> int:
# Reduce operation input is a list of Futures.
doubled: list[Future] = [double.future(number) for number in numbers]
# sum is called on each pair of doubled values as soon as they are available.
return sum.reduce(doubled, 0)
```
### Future / Coroutine / Task
Map and reduce operations accept a single [Future](/applications/futures)/[coroutine/`asyncio.Task`](/applications/async-functions.mdx)
object as their input. The Future/coroutine/`asyncio.Task` object has to resolve to a list of items.
Tensorlake automatically waits for it to complete and uses the returned list as the operation input.
This is useful when the input list is produced by another Tensorlake function.
```python theme={null}
from tensorlake.applications import application, function, Future
@function()
def generate_numbers(count: int) -> list[int]:
return list(range(count))
@function()
def square(number: int) -> int:
return number ** 2
@function()
def sum(a: int, b: int) -> int:
return a + b
@application()
@function()
def sum_of_squares(count: int) -> int:
# generate_numbers returns a list[int] when it completes.
numbers_future: Future = generate_numbers.future(count)
# Pass the Future as input to map. Tensorlake waits for generate_numbers
# to complete and maps square over the returned list.
squares: Future = square.future.map(numbers_future)
# Pass the map Future as input to reduce operation.
return sum.reduce(squares)
```
## Tail calls
Map and reduce operation Futures can be returned from functions as [tail calls](/applications/futures#tail-calls).
The returning function completes immediately and frees its container while Tensorlake orchestrates the map-reduce operation.
This allows Tensorlake to optimize resource usage and reduce overall application latency.
Learn how to use async functions in Tensorlake applications.
Use Futures for parallel execution and tail calls.
# Observability
Source: https://docs.tensorlake.ai/applications/observability
Built-in tracing, execution timelines and monitoring
Every Tensorlake function call is automatically traced. You get execution timelines, logs, metrics, and error details without configuring any observability infrastructure.
## Execution Timelines
When a request flows through your application, Tensorlake records every function call in an execution timeline. You can see:
* **Function call sequence**: which functions ran and in what order
* **Timing**: how long each function took, including cold start time
* **Dependencies**: which function calls ran in parallel vs. sequentially
* **Status**: success, failure, or retry for each function call
This is available in the [Tensorlake Dashboard](https://cloud.tensorlake.ai/applications) for every application request.
## Structured Logging
Use Python's standard `print()` or `logging` module inside your functions. Logs are captured automatically and associated with the specific function call and request.
```python theme={null}
from tensorlake.applications import function
import logging
logger = logging.getLogger(__name__)
@function()
def process_data(data: str) -> str:
logger.info(f"Processing {len(data)} characters")
result = transform(data)
logger.info(f"Transformation complete, output size: {len(result)}")
return result
```
Logs are available in the dashboard and through the [Logging guide](/applications/guides/logging) for configuration details.
## Learn More
Structured logging configuration.
# Programming Agents
Source: https://docs.tensorlake.ai/applications/overview
Core concepts and common patterns for running agents on Tensorlake
Tensorlake is a **compute platform for agents**: it runs your agents, it doesn't replace your agent framework. You bring the agent logic (OpenAI Agents SDK, LangGraph, Claude SDK, or plain Python), and Tensorlake provides the infrastructure: serverless containers, durable execution, sandboxes, and observability.
## Patterns
## Agent Loop in a Single Function
The simplest pattern: your entire agent loop runs inside one `@function()`. Tensorlake handles deployment, scaling, and durability.
```python theme={null}
from tensorlake.applications import application, function
@application()
@function(timeout=3600)
def research_agent(topic: str) -> str:
from agents import Agent, Runner, WebSearchTool
agent = Agent(
name="ResearchAgent",
instructions="Thoroughly research the given topic using web search.",
tools=[WebSearchTool()]
)
result = Runner.run_sync(agent, topic)
return result.final_output
```
This works well for agents that:
* Run a single loop with tool calls
* Don't need to fan out work to other agents
* Have predictable resource requirements
## Sandboxing Functions
When your agent calls tools with different resource needs (CPU, memory, GPU, dependencies), wrap each tool in its own `@function()`. Each function runs in its own container with its own resource limits and dependencies.
```python theme={null}
from tensorlake.applications import application, function, Image
heavy_image = Image().run("pip install torch transformers")
@function(image=heavy_image, memory=8, gpu="T4")
def classify_image(image_url: str) -> str:
"""Runs in a GPU container with 8GB memory."""
from transformers import pipeline
classifier = pipeline("image-classification")
return classifier(image_url)[0]["label"]
@function()
def search_web(query: str) -> list[str]:
"""Runs in a lightweight container."""
import requests
# Call search API
return ["result1", "result2"]
@application()
@function(timeout=1800)
def research_agent(topic: str) -> dict:
# Agent loop calls tools that run in separate containers
image_label = classify_image("https://example.com/photo.jpg")
web_results = search_web(topic)
return {"image": image_label, "web": web_results}
```
Each `@function()`:
* Runs in its own isolated container
* Has its own dependencies, CPU, memory, and GPU allocation
* Is independently retryable and durable
* Scales independently based on demand
## Harness Pattern: Agent as Orchestrator
For complex agents, separate the **harness** (orchestration logic) from the **work** (tool execution). The harness is a lightweight function that coordinates heavier worker functions.
```python theme={null}
from tensorlake.applications import application, function, Image
worker_image = Image().run("pip install openai langchain")
@application()
@function(timeout=3600)
def analyst_agent(query: str) -> dict:
"""Lightweight harness that orchestrates worker functions."""
from openai import OpenAI
client = OpenAI()
# Agent decides what to do
plan = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": f"Plan research for: {query}"}]
).choices[0].message.content
# Dispatch to worker functions
data = fetch_data(query)
analysis = analyze_data(data)
return {"plan": plan, "analysis": analysis}
@function(image=worker_image, cpu=4, memory=8)
def fetch_data(query: str) -> dict:
"""Heavy data fetching in a dedicated container."""
...
@function(image=worker_image, cpu=2, memory=4)
def analyze_data(data: dict) -> str:
"""Analysis with different resource needs."""
...
```
## Running Agent Frameworks on Tensorlake
### OpenAI Agents SDK
```python theme={null}
from tensorlake.applications import application, function
@application()
@function(timeout=1800)
def openai_agent(prompt: str) -> str:
from agents import Agent, Runner, WebSearchTool
agent = Agent(
name="Assistant",
instructions="You are a helpful assistant.",
tools=[WebSearchTool()]
)
result = Runner.run_sync(agent, prompt)
return result.final_output
```
### LangGraph
```python theme={null}
from tensorlake.applications import application, function, Image
image = Image().run("pip install langgraph langchain-openai")
@application()
@function(image=image, timeout=1800)
def langgraph_agent(query: str) -> str:
from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI
model = ChatOpenAI(model="gpt-4")
agent = create_react_agent(model, tools=[])
result = agent.invoke({"messages": [("human", query)]})
return result["messages"][-1].content
```
### Claude SDK
```python theme={null}
from tensorlake.applications import application, function
@application()
@function(timeout=3600, ephemeral_disk=4)
def claude_agent(prompt: str) -> str:
import asyncio
from claude_agent_sdk import query, ClaudeAgentOptions
async def run():
options = ClaudeAgentOptions(
system_prompt="You are an expert developer.",
permission_mode="acceptEdits",
cwd="/tmp/workspace"
)
result = ""
async for message in query(prompt=prompt, options=options):
result = str(message)
return result
return asyncio.run(run())
```
## Parallel Sub-Agents
When your workflow involves multiple specialist agents, fan them out using [futures](/applications/futures) or [async functions](/applications/async-functions) so they run in parallel:
```python theme={null}
@application()
@function()
def analyze_proposal(text: str) -> dict:
financial = financial_agent.future(text)
legal = legal_agent.future(text)
technical = technical_agent.future(text)
return synthesize.future(financial, legal, technical)
```
See [Parallel Sub-Agents](/applications/parallel-sub-agents) for detailed patterns.
## Core Concepts
**Building blocks of applications.** Functions are Python functions that run in isolated containers with their own dependencies, compute, and storage.
**HTTP-triggered entry points.** Applications are functions exposed as HTTP endpoints that receive requests and orchestrate work across multiple functions.
**Resume from failures, not restart.** Checkpoints are automatically created so retries continue from the last successful step instead of starting over.
**Run untrusted code safely.** Every function runs in an isolated sandbox with configurable resource limits and network restrictions.
**Parallel data processing.** Fan out work across a list in parallel, then aggregate results, with no queue setup required.
**Built-in tracing and logging.** Every function call is automatically traced with timing, logs, and execution timelines.
# Parallel Sub-Agents
Source: https://docs.tensorlake.ai/applications/parallel-sub-agents
Fan out work to specialist agents that run in parallel
Every agent framework has converged on the same pattern: break a complex task into independent subtasks, run specialist agents on each subtask in parallel, and synthesize the results. LangGraph does this with `Send` and `@task` futures. OpenAI Agents SDK uses `asyncio.gather` and `agent.as_tool()`. Claude Agent SDK spawns subagents via the `Task` tool. Deep Agents dispatches parallel `task` tool calls.
On Tensorlake, you get the same fan-out/fan-in pattern, but each sub-agent runs in its own container with dedicated resources, independent retries, and durable checkpointing. No `asyncio` plumbing, no graph DSL, no shared memory coordination.
## Basic Pattern: Fan-Out and Combine
Define each sub-agent as a `@function()`, create futures for each, and pass them to a combiner function as a tail call:
```python theme={null}
from tensorlake.applications import application, function, Image
research_image = Image().run("pip install openai requests")
@application()
@function()
def analyze_company(company_name: str) -> dict:
# Fan out to specialist agents: all run in parallel
financials = financial_agent.future(company_name)
market = market_agent.future(company_name)
sentiment = sentiment_agent.future(company_name)
# Combine results: runs after all agents complete
return compile_report.future(financials, market, sentiment, company_name)
@function(image=research_image, timeout=600)
def financial_agent(company: str) -> dict:
"""Analyze financial data for a company."""
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": f"Analyze financials for {company}"}]
)
return {"analysis": response.choices[0].message.content}
@function(image=research_image, timeout=600)
def market_agent(company: str) -> dict:
"""Analyze market position and competitors."""
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": f"Analyze market position for {company}"}]
)
return {"analysis": response.choices[0].message.content}
@function(image=research_image, timeout=600)
def sentiment_agent(company: str) -> dict:
"""Analyze public sentiment."""
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": f"Analyze sentiment for {company}"}]
)
return {"analysis": response.choices[0].message.content}
@function()
def compile_report(financials: dict, market: dict, sentiment: dict, company: str) -> dict:
return {
"company": company,
"financials": financials,
"market": market,
"sentiment": sentiment
}
```
**Execution flow:**
```mermaid theme={null}
graph LR
A["analyze_company()"] --> B["financial_agent()"]
A --> C["market_agent()"]
A --> D["sentiment_agent()"]
B --> E["compile_report()"]
C --> E
D --> E
E --> F["result"]
```
## How It Works
1. The orchestrator function creates futures for each sub-agent: this defines the calls without running them
2. Futures are passed as arguments to the combiner function, which is returned as a **tail call**
3. Tensorlake detects that the future arguments have no dependencies on each other and runs all sub-agents **in parallel**
4. When all sub-agents complete, the combiner runs with their results
5. The orchestrator's container is freed immediately after returning the tail call
The orchestrator's container is freed immediately after returning the tail call. You're not paying for an idle container while sub-agents work.
## Real-World Patterns
These patterns are inspired by what teams are building in production with LangGraph, OpenAI Agents SDK, Claude Agent SDK, and Deep Agents, reimplemented on Tensorlake with container isolation, independent scaling, and durable execution.
### Parallel Research with Synthesis
The most common multi-agent pattern across every framework: decompose a research question into subtopics, investigate each in parallel, and synthesize the findings. This is the pattern behind GPT Researcher, Exa's web research system, and Anthropic's multi-agent research system.
```python theme={null}
from tensorlake.applications import application, function, Image
research_image = Image().run("pip install openai requests beautifulsoup4")
@function(image=research_image, timeout=900, retries=2)
def research_subtopic(topic: str, subtopic: str) -> dict:
"""Each researcher runs in its own container, searches the web,
reads sources, and produces a structured summary."""
from openai import OpenAI
client = OpenAI(max_retries=0)
# Step 1: Generate search queries for this subtopic
queries = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": f"Generate 3 search queries to research '{subtopic}' in the context of '{topic}'."}],
).choices[0].message.content
# Step 2: Search and gather sources
sources = search_and_read(queries)
# Step 3: Analyze and summarize
analysis = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "Summarize research findings with citations."},
{"role": "user", "content": f"Topic: {subtopic}\n\nSources:\n{sources}"},
],
).choices[0].message.content
return {"subtopic": subtopic, "analysis": analysis, "source_count": len(sources)}
@function(image=research_image, timeout=300)
def synthesize_research(results: list[dict], topic: str) -> dict:
"""Combine all parallel research into a cohesive report."""
from openai import OpenAI
combined = "\n\n---\n\n".join(
f"## {r['subtopic']}\n{r['analysis']}" for r in results
)
report = OpenAI(max_retries=0).chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "Synthesize research findings into a cohesive report. Resolve contradictions and highlight consensus."},
{"role": "user", "content": f"Topic: {topic}\n\nFindings:\n{combined}"},
],
).choices[0].message.content
return {"topic": topic, "report": report, "sections": len(results)}
@application()
@function(image=research_image, timeout=120)
def deep_research(topic: str) -> dict:
"""Orchestrator: decompose, fan out, synthesize."""
from openai import OpenAI
import json
# Plan the research
plan = OpenAI(max_retries=0).chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": f"Break this topic into 3-5 independent research subtopics: {topic}"}],
response_format={"type": "json_object"},
).choices[0].message.content
subtopics = json.loads(plan)["subtopics"]
# Fan out: each subtopic researched in parallel
findings = [research_subtopic.future(topic, sub) for sub in subtopics]
# Synthesize: runs after all research completes
return synthesize_research.future(findings, topic)
```
Each researcher runs in its own container with its own 15-minute timeout and 2 retries. If one subtopic's research fails (rate limit, network error), only that subtopic is retried. The other researchers' work is preserved.
### Multi-Perspective Analysis
Multiple specialist agents examine the same input from different analytical perspectives, a pattern used in production for investment analysis, proposal review, and compliance checks.
```python theme={null}
from pydantic import BaseModel
from tensorlake.applications import application, function, Image
analyst_image = Image().run("pip install anthropic")
class AnalystReport(BaseModel):
perspective: str
assessment: str
risk_score: float
key_findings: list[str]
@function(image=analyst_image, timeout=600, retries=2)
def growth_analyst(company_data: dict) -> AnalystReport:
"""Evaluate revenue growth, market expansion, and competitive moats."""
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-4-5-20250929",
max_tokens=2000,
messages=[{"role": "user", "content": f"As a growth analyst, evaluate:\n{company_data}"}],
)
return parse_report("growth", response.content[0].text)
@function(image=analyst_image, timeout=600, retries=2)
def value_analyst(company_data: dict) -> AnalystReport:
"""Evaluate cash flow, margins, and intrinsic value."""
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-4-5-20250929",
max_tokens=2000,
messages=[{"role": "user", "content": f"As a value analyst, evaluate:\n{company_data}"}],
)
return parse_report("value", response.content[0].text)
@function(image=analyst_image, timeout=600, retries=2)
def risk_analyst(company_data: dict) -> AnalystReport:
"""Evaluate regulatory risk, market volatility, and operational risk."""
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-4-5-20250929",
max_tokens=2000,
messages=[{"role": "user", "content": f"As a risk analyst, evaluate:\n{company_data}"}],
)
return parse_report("risk", response.content[0].text)
@function(image=analyst_image, timeout=300)
def investment_committee(growth: AnalystReport, value: AnalystReport, risk: AnalystReport) -> dict:
"""Weigh all perspectives and produce a final recommendation."""
import anthropic
client = anthropic.Anthropic()
combined = f"Growth: {growth.model_dump()}\nValue: {value.model_dump()}\nRisk: {risk.model_dump()}"
response = client.messages.create(
model="claude-sonnet-4-5-20250929",
max_tokens=2000,
messages=[{"role": "user", "content": f"As an investment committee, synthesize these analyst reports into a buy/hold/sell recommendation:\n{combined}"}],
)
return {"recommendation": response.content[0].text, "analyst_reports": [growth.model_dump(), value.model_dump(), risk.model_dump()]}
@application()
@function()
def analyze_investment(company_data: dict) -> dict:
growth = growth_analyst.future(company_data)
value = value_analyst.future(company_data)
risk = risk_analyst.future(company_data)
return investment_committee.future(growth, value, risk)
```
This mirrors the multi-agent portfolio collaboration pattern from the OpenAI Agents SDK cookbook, but each analyst runs in an isolated container with its own timeout and retry policy.
### Document Processing Pipeline
Process a batch of documents through parallel specialist agents, a common pattern for intake automation in insurance, legal, and financial services.
```python theme={null}
from tensorlake.applications import application, function, Image
ocr_image = Image().run("pip install pytesseract pillow pdf2image")
llm_image = Image().run("pip install openai")
@function(image=ocr_image, cpu=2, memory=4, timeout=120)
def extract_text(doc_url: str) -> dict:
"""OCR and text extraction: needs CPU for image processing."""
content = download_and_ocr(doc_url)
return {"url": doc_url, "text": content}
@function(image=llm_image, timeout=300, retries=2)
def classify_document(doc: dict) -> dict:
"""Determine document type and extract key fields."""
from openai import OpenAI
response = OpenAI(max_retries=0).chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": f"Classify this document and extract key fields:\n{doc['text'][:4000]}"}],
response_format={"type": "json_object"},
)
return {**doc, "classification": response.choices[0].message.content}
@function(image=llm_image, timeout=300, retries=2)
def check_compliance(doc: dict) -> dict:
"""Check for missing signatures, dates, required fields."""
from openai import OpenAI
response = OpenAI(max_retries=0).chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": f"Check this document for compliance issues:\n{doc['text'][:4000]}"}],
response_format={"type": "json_object"},
)
return {**doc, "compliance": response.choices[0].message.content}
@function(timeout=60)
def merge_results(classified: dict, compliance: dict) -> dict:
return {
"url": classified["url"],
"classification": classified["classification"],
"compliance": compliance["compliance"],
}
@application()
@function()
def process_document(doc_url: str) -> dict:
extracted = extract_text.future(doc_url)
classified = classify_document.future(extracted)
compliance = check_compliance.future(extracted)
return merge_results.future(classified, compliance)
```
```mermaid theme={null}
graph LR
A["process_document()"] --> B["extract_text()"]
B --> C["classify_document()"]
B --> D["check_compliance()"]
C --> E["merge_results()"]
D --> E
E --> F["result"]
```
After extraction, classification and compliance checking run in parallel. They both depend on the extracted text but not on each other. If the compliance check hits a rate limit, it retries independently without re-running OCR or classification.
## Use Any Agent Framework
Each sub-agent can use whatever framework you want internally. The `@function()` boundary is a container boundary: what runs inside is up to you. Define each specialist as a focused function using its framework, then fan them out with `.future()`.
```python theme={null}
from tensorlake.applications import application, function, Image
# Each framework gets its own container image with its own dependencies
langgraph_image = Image().run("pip install langgraph langchain-openai tavily-python")
openai_image = Image().run("pip install openai-agents")
claude_image = Image().run("pip install claude-agent-sdk")
deep_image = Image().run("pip install deepagents langchain-openai")
@function(image=langgraph_image, timeout=600)
def market_researcher(company: str) -> str:
"""Market research using a LangGraph ReAct agent with web search."""
from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI
from langchain_community.tools import TavilySearchResults
agent = create_react_agent(
ChatOpenAI(model="gpt-4o"),
tools=[TavilySearchResults(max_results=5)],
)
result = agent.invoke({"messages": [
("human", f"Research the market position, competitors, and recent news for {company}.")
]})
return result["messages"][-1].content
@function(image=openai_image, timeout=600)
def financial_analyst(company: str) -> str:
"""Financial analysis using an OpenAI Agents SDK agent with tool use."""
from agents import Agent, Runner, WebSearchTool
agent = Agent(
name="FinancialAnalyst",
instructions=(
"You are a financial analyst. Analyze revenue, margins, cash flow, "
"and valuation metrics. Use web search to find the latest filings."
),
tools=[WebSearchTool()],
)
result = Runner.run_sync(agent, f"Analyze the financials for {company}")
return result.final_output
@function(image=claude_image, timeout=900, ephemeral_disk=4)
def risk_assessor(company: str) -> str:
"""Risk assessment using a Claude agent with deep reasoning."""
import asyncio
from claude_agent_sdk import query, ClaudeAgentOptions
async def run():
result = ""
async for message in query(
prompt=f"Assess regulatory, operational, and market risks for {company}.",
options=ClaudeAgentOptions(
system_prompt="You are a risk analyst. Identify and score key risks.",
permission_mode="acceptEdits",
cwd="/tmp/workspace",
),
):
result = str(message)
return result
return asyncio.run(run())
@function(image=deep_image, timeout=900)
def technical_reviewer(company: str) -> str:
"""Technical deep-dive using a Deep Agent with planning and web search."""
from deepagents import create_deep_agent
agent = create_deep_agent(
model="openai:gpt-4o",
system_prompt="Evaluate the company's technology stack, patents, and engineering culture.",
)
result = agent.invoke({
"messages": [{"role": "user", "content": f"Technical review of {company}"}]
})
return result["messages"][-1].content
@function(timeout=300)
def compile_analysis(market: str, financials: str, risks: str, technical: str, company: str) -> dict:
"""Combine all analyst reports into a final recommendation."""
return {
"company": company,
"market_research": market,
"financial_analysis": financials,
"risk_assessment": risks,
"technical_review": technical,
}
@application()
@function()
def analyze_company(company: str) -> dict:
# Four frameworks, four containers, all running in parallel
market = market_researcher.future(company)
financials = financial_analyst.future(company)
risks = risk_assessor.future(company)
technical = technical_reviewer.future(company)
return compile_analysis.future(market, financials, risks, technical, company)
```
```mermaid theme={null}
graph LR
A["analyze_company()"] --> B["market_researcher()\nLangGraph"]
A --> C["financial_analyst()\nOpenAI Agents SDK"]
A --> D["risk_assessor()\nClaude Agent SDK"]
A --> E["technical_reviewer()\nDeep Agents"]
B --> F["compile_analysis()"]
C --> F
D --> F
E --> F
F --> G["result"]
```
Each agent runs in its own container with its own dependencies: no version conflicts, no shared memory, no `asyncio` event loop contention. If the risk assessment takes longer than the others, the completed agents' results are checkpointed and preserved.
## Different Resources Per Agent
Each sub-agent can have its own container configuration:
```python theme={null}
gpu_image = Image().run("pip install torch transformers")
@function(cpu=1, memory=2, timeout=300)
def text_agent(prompt: str) -> str:
"""Lightweight text analysis."""
...
@function(image=gpu_image, cpu=4, memory=16, gpu="T4", timeout=600)
def vision_agent(image_url: str) -> dict:
"""GPU-heavy image analysis."""
...
@function(cpu=2, memory=4, timeout=900)
def data_agent(query: str) -> list:
"""Medium resources for data fetching."""
...
@application()
@function()
def multimodal_analysis(prompt: str, image_url: str) -> dict:
text_result = text_agent.future(prompt)
vision_result = vision_agent.future(image_url)
data_result = data_agent.future(prompt)
return combine_results.future(text_result, vision_result, data_result)
```
## Chaining Parallel Stages
You can chain stages where each stage fans out in parallel:
```python theme={null}
@application()
@function()
def pipeline(query: str) -> dict:
# Stage 1: Gather data in parallel
web = search_web.future(query)
papers = search_papers.future(query)
news = search_news.future(query)
# Stage 2: Analyze each source (runs after stage 1)
analysis = analyze_sources.future(web, papers, news)
# Stage 3: Generate final output
return generate_report.future(analysis, query)
```
Each stage waits for its dependencies automatically. Stages without dependencies run in parallel.
## Using Futures for More Control
When you need to do work in the orchestrator while sub-agents run, use [Futures](/applications/futures) instead of tail calls:
```python theme={null}
from tensorlake.applications import application, function, Future, RETURN_WHEN
@application()
@function(timeout=1800)
def interactive_analysis(query: str) -> dict:
# Start sub-agents
agent_a: Future = agent_a_work.future(query).run()
agent_b: Future = agent_b_work.future(query).run()
# Do local work while agents run
local_context = prepare_context(query)
# Wait for both agents
Future.wait([agent_a, agent_b], return_when=RETURN_WHEN.ALL_COMPLETED)
return {
"context": local_context,
"agent_a": agent_a.result(),
"agent_b": agent_b.result()
}
```
## Learn More
Deep dive on futures, tail calls, and parallel execution.
Use Python async/await for parallel workflows.
Parallel processing over lists of data.
# Troubleshooting
Source: https://docs.tensorlake.ai/applications/production/troubleshooting
Common issues building Tensorlake applications and how to debug them
## Common Issues
### Function Timeout
If your function is timing out, consider:
1. **Increase the timeout** - Set a higher `timeout` value in your `@function` decorator
2. **Report progress** - Use `ctx.progress.update()` to reset the timeout. See [Streaming Progress Updates](/applications/concepts#streaming-progress-updates)
3. **Check the logs** - Use the Logs API above to see what your function was doing before it timed out
### Request Failed
To investigate a failed request:
1. **Check request state** - Get the full request state including failure reason:
```bash theme={null}
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/requests/{request_id}" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
2. **Review logs** - Filter logs by the request ID to see what happened:
```bash theme={null}
curl -X GET \
"https://api.tensorlake.ai/applications/{application}/logs?requestId={request_id}" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
### Out of Memory
If your function is running out of memory:
1. **Check current allocation** - Review the `memory` setting in your `@function` decorator
2. **Increase memory** - Set `memory` to a higher value (up to 32 GB). See [Memory](/applications/concepts#memory)
3. **Process in batches** - Break large datasets into smaller chunks
### Debugging Tips
* Add `print()` statements in your code to log intermediate values
* Use `ctx.request_id` to correlate logs across function calls. See [Request ID](/applications/concepts#request-id)
* Check that your function has sufficient CPU, memory, and disk resources
* Review retry settings if functions are failing intermittently. See [Retries](/applications/concepts#retries)
# Public Endpoints
Source: https://docs.tensorlake.ai/applications/public-endpoints
Expose an application endpoint that callers can invoke without a Tensorlake API key.
By default, applications require an [API Key](/platform/access-control#api-keys) for authorization. Public endpoints let you call applications without Tensorlake credentials.
## Enable a public endpoint
Add the `unauthenticated_requests` capability to the `@application` decorator:
```python public_api.py theme={null}
from tensorlake.applications import application, function
@application(allow=["unauthenticated_requests"])
@function()
def public_api(payload: dict) -> dict:
return {"status": "accepted", "payload": payload}
```
The `allow` attribute accepts a list of application capabilities. At the moment, `unauthenticated_requests` is the only supported capability.
## Deploy the application
Deploy the application normally:
```bash theme={null}
tl app deploy public_api.py
```
On the first deployment, Tensorlake assigns the application a stable, opaque `public_endpoint_id`. Redeploying the application preserves this ID. The public URL has this form:
```text theme={null}
https://api.tensorlake.ai/applications/public/
```
After the application is deployed, you can call the public endpoint without an `Authorization` header:
```bash theme={null}
curl https://api.tensorlake.ai/applications/public/ \
--json '{"event": "created"}'
```
## Read request headers
A sanitized list of HTTP request headers are available inside your application's request context when you use public endpoints. You can access them through `RequestContext.get().headers`. The `Headers` collection is immutable and supports case-insensitive lookups:
```python theme={null}
from tensorlake.applications import RequestContext
headers = RequestContext.get().headers
request_type = headers["X-Request-Type"] # Required header
signature = headers.get("x-request-signature") # Optional header
all_values = headers.getlist("X-Provider-Tag") # Repeated header
```
When a header has multiple values, `get()` return the last value. `getlist()` returns all values in their received order.
Tensorlake prevents the following headers from being forwarded into application invocations:
* Credentials and browser state: `Authorization`, `Authentication`, `Cookie`, and proxy authentication headers
* Routing and transport details: `Host`, standard hop-by-hop headers, and headers named by `Connection`
## Receive a raw request body
Use the type `HttpBody` when you need to parse the request body sent into a public endpoint.
```python theme={null}
from tensorlake.applications import HttpBody, application, function
@application(allow=["unauthenticated_requests"])
@function()
def receive_raw_body(body: HttpBody) -> dict:
return {
"content_type": body.content_type,
"size": len(body.content),
"payload": body.json(),
}
```
`HttpBody` exposes the raw bytes through `body.content`, along with `body.content_type`, `body.text()`, and `body.json()`.
## Usage example: Receiving webhooks
Webhook providers are a common use for public endpoints. The following handler receives GitHub `workflow_job` events and validates each request using GitHub's `X-Hub-Signature-256` header and the exact request body:
```python github_webhook.py theme={null}
import hmac
import os
from hashlib import sha256
from tensorlake.applications import (
HttpBody,
RequestContext,
application,
function,
)
def verify_signature(
raw_body: bytes,
signature: str | None,
secret: str,
) -> bool:
if not signature or not signature.startswith("sha256="):
return False
expected = "sha256=" + hmac.new(
secret.encode("utf-8"),
raw_body,
sha256,
).hexdigest()
return hmac.compare_digest(expected, signature)
@application(allow=["unauthenticated_requests"])
@function(secrets=["GITHUB_WEBHOOK_SECRET"])
def github_webhook(body: HttpBody) -> dict[str, str]:
headers = RequestContext.get().headers
if headers.get("X-GitHub-Event") != "workflow_job":
return {"status": "ignored"}
if not verify_signature(
body.content,
headers.get("X-Hub-Signature-256"),
os.environ["GITHUB_WEBHOOK_SECRET"],
):
return {"status": "rejected"}
event = body.json()
# Process the verified event.
return {"status": "accepted", "action": event.get("action", "")}
```
Store the same secret that you configure with GitHub, then deploy the application:
```bash theme={null}
tl secrets set GITHUB_WEBHOOK_SECRET=
tl app deploy github_webhook.py
```
## Disable public access
If you want to disable your application's public endpoint, remove `unauthenticated_requests` from the `allow` list and redeploy:
```python theme={null}
@application()
@function()
def github_webhook(body: HttpBody) -> dict[str, str]:
...
```
# Applications Quickstart
Source: https://docs.tensorlake.ai/applications/quickstart
Write, deploy, and call your first Tensorlake Application, a serverless agentic web-scraper with the Claude Agent SDK in under five minutes.
This guide will walk you through the process of writing, deploying, and calling Tensorlake Applications. You will learn how to build a serverless
agentic web-scrapper with Anthropic's Claude Agent SDK under 5 minutes.
Let's start with a simple "Hello, World!" application, to make sure your environment is set up correctly.
The `tl` CLI scaffolds and deploys applications. Install it with the install script:
```bash theme={null}
curl -fsSL https://tensorlake.ai/install | sh
```
Then install the Python SDK, which provides the `tensorlake` package your application code imports:
```bash theme={null}
pip install tensorlake
```
You can get an [API key](/platform/authentication#api-keys) from the Tensorlake Dashboard.
```bash theme={null}
export TENSORLAKE_API_KEY=
```
Applications are defined by Python functions. Let's start with a template, that greets a user by name.
```bash theme={null}
tl app new hello_world
```
This creates a file named `hello_world/hello_world.py` with the following content:
```python hello_world.py theme={null}
from tensorlake.applications import application, function
@application()
@function()
def greet(name: str) -> str:
return f"Hello, {name}!"
```
Deploy your application referencing your application's source file.
```bash theme={null}
tl app deploy hello_world/hello_world.py
```
That's it. You now have a distributed app running in the cloud.
## Call Applications
Tensorlake gives you an HTTP endpoint, for calling your application remotely.
```
https://api.tensorlake.ai/applications/
```
Fetch a key from the [Tensorlake Dashboard](/platform/authentication#api-keys) and export it as an environment variable:
```bash theme={null}
export TENSORLAKE_API_KEY=
```
```bash bash theme={null}
curl https://api.tensorlake.ai/applications/hello_world \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
--json '"John"'
# {"request_id":"beae8736ece31ef9"}
```
```python python theme={null}
from tensorlake.applications import run_remote_application, Request
request: Request = run_remote_application(greet, 'John')
print(request.id)
# "beae8736ece31ef9"
```
This will return a request ID that you can use to track the progress of your request.
Requests may run seconds to hours depending on your workload.
```bash bash theme={null}
curl -X GET https://api.tensorlake.ai/applications/hello_world/requests/{request_id} \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
# {
# "id":"B0IwzHibTTfn5mCXHPGsu",
# "outcome":"success",
# "failure_reason":null,
# "request_error":null,
# .... other fields ...
#}
```
```python python theme={null}
# You don't need to poll for request completion. Retrieving the output will wait for the request to complete.
```
The `outcome` field will be `success` or `failure` depending on whether the request completed successfully. It will be null if the request is still in progress.
```bash bash theme={null}
curl -X GET https://api.tensorlake.ai/applications/hello_world/requests/{request_id}/output \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
# "Hello, John!"
```
```python python theme={null}
from tensorlake.applications import run_remote_application, Request
request: Request = run_remote_application(greet, 'John')
output: str = request.output()
print(output)
# "Hello, John!"
```
## Testing Locally
Tensorlake Applications can run locally on your laptop. You can run them like regular python scripts.
```python hello_world.py theme={null}
# At the end of the file
from tensorlake.applications import run_local_application, Request
if __name__ == "__main__":
request: Request = run_local_application(greet, 'John')
output: str = request.output()
print(output)
# "Hello, John!"
```
## Building an Agentic Code Interpreter
Now let's build a real agentic application. We will build a code interpreter agent with OpenAI Agent SDK.
The tensorlake application function will be the main agentic loop, and we will use a Tensorlake function to execute code,
and pass it as a tool to the agent. Whenever the agent needs to execute code, it will call the Tensorlake function and pass the code as a tool call.
The Tensorlake function will execute the code in an isolated container and return the output to the agent.
The agent needs access to the OpenAI API. Add your API key as a secret using the Tensorlake CLI:
```bash theme={null}
tl secrets set OPENAI_API_KEY=
```
This securely stores your API key so it can be injected into your application at runtime. The secret is referenced in the function decorator which uses the OpenAI Agent SDK and
will be available as an environment variable.
```python code_interpreter.py theme={null}
import sys
from io import StringIO
from tensorlake.applications import application, function, Image
# Image for the code execution container - has data science libraries
code_exec_image = (
Image(name="python:3.11-slim")
.run("pip install numpy pandas matplotlib")
)
# Image for the agent container - has the OpenAI Agent SDK
agent_image = (
Image(name="python:3.11-slim")
.run("pip install openai-agents")
)
@function(image=code_exec_image, cpu=2, memory=4)
def execute_code(code: str) -> str:
"""Execute Python code in a secure sandbox and return the output."""
stdout_capture = StringIO()
old_stdout = sys.stdout
try:
sys.stdout = stdout_capture
exec_globals = {"__builtins__": __builtins__}
exec(code, exec_globals)
sys.stdout = old_stdout
return stdout_capture.getvalue()
except Exception as e:
sys.stdout = old_stdout
return f"Error: {e}\nOutput: {stdout_capture.getvalue()}"
@application()
@function(image=agent_image, secrets=["OPENAI_API_KEY"])
def code_interpreter_agent(user_request: str) -> str:
"""Run the agentic loop and return the final answer."""
from agents import Agent, Runner, function_tool
@function_tool
def execute_python(code: str) -> str:
"""Execute Python code in a secure sandbox. Use this for calculations or data analysis."""
return execute_code(code)
agent = Agent(
name="Code interpreter",
model="gpt-4o",
instructions="You are a helpful assistant that can execute Python code to solve problems.",
tools=[execute_python],
)
result = Runner.run_sync(agent, user_request)
return result.final_output
```
Deploy your application and call it:
```bash theme={null}
tl app deploy code_interpreter.py
```
```bash theme={null}
curl https://api.tensorlake.ai/applications/code_interpreter_agent \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
--json '"What is the square root of 273 * 312821 plus 1782?"'
```
On Lambda or Vercel, running arbitrary code execution would require complex sandboxing, security policies, and resource management, all in the same container as your main application.
With Tensorlake, the `execute_code` function runs in a completely isolated container with its own CPU, memory, and dependencies. If code execution needs heavy compute or specialized libraries, it scales independently from your agent logic.
You get secure, isolated code execution without managing infrastructure.
Tensorlake handles the infrastructure complexity so you can focus on building powerful AI tools.
## Next Steps
Here are some of the next things to learn about:
Learn key concepts and APIs to program applications.
Learn how to add dependencies for your applications.
Learn how to manage secrets that your applications access.
Learn how to use map-reduce to process large datasets.
Learn how to build multi-step workflows with parallel execution and optimized resource usage.
Learn how to run multiple function calls in parallel using Futures.
Learn how to use Python async/await with Tensorlake functions.
# Retries & Rate Limits
Source: https://docs.tensorlake.ai/applications/retries
Handle LLM rate limits, transient failures, and structured output validation with durable retries
LLM providers return rate-limit errors, APIs time out, and web scrapes hit transient failures. Tensorlake handles retries at the platform level. Each retry is durable, meaning any nested function calls that already succeeded are served from checkpoints instead of re-executing. See [Durable Execution](/applications/durability) for how the checkpoint mechanism works and [Crash Recovery](/applications/crash-recovery) for the agent-loop walkthrough.
## Configuring Retries
Set the `retries` parameter on any `@function()` to automatically retry on failure. This is especially useful for LLM calls that return structured output. If the LLM returns malformed data, Pydantic validation fails and Tensorlake retries the entire call:
```python theme={null}
from pydantic import BaseModel
from tensorlake.applications import function
class ResearchFindings(BaseModel):
summary: str
sources: list[str]
confidence: float
@function(retries=3)
def extract_findings(text: str) -> ResearchFindings:
from openai import OpenAI
# Disable client-level retries to avoid unpredictable behavior
response = OpenAI(max_retries=0).chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "Extract research findings as JSON."},
{"role": "user", "content": text},
],
response_format={"type": "json_object"},
)
# If validation fails, Tensorlake retries the entire function
return ResearchFindings.model_validate_json(response.choices[0].message.content)
```
**How retries work:**
* Rate limit errors, timeouts, or exceptions trigger automatic retries
* Validation failures (e.g., Pydantic `ValidationError`) also trigger retries
* Tensorlake retries up to 3 times with exponential backoff
* Any nested function calls that already succeeded are served from checkpoints, not re-executed (see [Durable Execution](/applications/durability))
If retries are exhausted, the request fails and can be re-run later via the [Replay API](/applications/durability#request-replay-api). For broader exception-handling patterns (try/except, futures, fallbacks), see [Error Handling](/applications/error-handling). For controlling per-function deadlines, see [Timeouts](/applications/timeouts).
Disable client-level retries (e.g., OpenAI's `max_retries=0`) when using Tensorlake retries. Layering both creates unpredictable behavior and inflated retry counts.
## Rate Limiting External APIs
When calling external APIs with rate limits, you can control the total number of concurrent calls using the formula:
**Total concurrent calls = `max_containers` × `concurrency`**
This allows you to respect API rate limits by capping the maximum number of parallel requests your function can make:
```python theme={null}
from tensorlake.applications import function
@function(
retries=3,
max_containers=5, # Maximum 5 containers
concurrency=2 # Each container handles 2 concurrent requests
)
def call_rate_limited_api(query: str) -> dict:
# Total concurrent calls: 5 × 2 = 10 requests max
import requests
response = requests.get(f"https://api.example.com/search?q={query}")
return response.json()
```
**Use cases:**
* **Respect API quotas**: If an API allows 100 requests/second, set `max_containers=50` and `concurrency=2`
* **Control costs**: Limit concurrent LLM calls to manage token spend
* **Prevent overload**: Cap requests to internal services that can't handle high concurrency
See [Scale-Out & Queuing](/applications/scale-out-queuing) for more on `max_containers` and request queuing.
## Related Guides
How checkpointing makes every retry cheap.
Surviving mid-loop crashes when retries run out.
Per-function deadlines and progress-update heartbeats.
Try/except, futures, and graceful degradation patterns.
# Sandboxes
Source: https://docs.tensorlake.ai/applications/sandboxes
Two patterns for running agents with isolated code execution
Agents that generate and execute code need a workspace: a computer where they can run code, install packages, and access files. That workspace needs to be isolated so the agent can't access your credentials, files, or network. Sandboxes provide this isolation.
The question isn't whether to use sandboxes. It's how to integrate them with your agent. There are two architectural patterns, based on where the agent runs: inside the sandbox or outside of it.
## Pattern 1: Agent in Sandbox
The agent runs inside an isolated container. Your application communicates with it over the network.
```mermaid theme={null}
graph LR
A["Your Application"] -- "HTTP" --> B["Sandbox Container"]
subgraph B["Sandbox Container"]
C["Agent Code"]
D["Filesystem"]
E["Packages"]
end
```
This is what Tensorlake's `@function()` does. When you deploy a function, your agent code runs inside an isolated container with its own filesystem, dependencies, and resource limits. The agent has direct access to its environment: it can read and write files, install packages, and execute code, all within the container boundary.
```python theme={null}
from tensorlake.applications import application, function, Image
agent_image = Image().run("pip install openai")
@application()
@function(image=agent_image, timeout=1800, memory=4, ephemeral_disk=10)
def coding_agent(task: str) -> str:
"""Agent runs inside the container with full filesystem access."""
from openai import OpenAI
import subprocess
client = OpenAI()
messages = [{"role": "user", "content": task}]
for _ in range(20):
response = client.chat.completions.create(model="gpt-4o", messages=messages)
reply = response.choices[0].message
if not reply.tool_calls:
return reply.content
for tool_call in reply.tool_calls:
if tool_call.function.name == "run_code":
# Code executes directly: agent is already in the sandbox
result = subprocess.run(
["python", "-c", tool_call.function.arguments],
capture_output=True, text=True, timeout=30
)
messages.append({"role": "tool", "content": result.stdout or result.stderr})
```
**When to use this pattern:**
* The agent and execution environment are tightly coupled
* The agent needs persistent filesystem access across tool calls
* You want production to mirror local development: same code, same environment
**Trade-offs:**
* API keys must live inside the container for the agent to make inference calls
* Updating agent logic requires redeploying the function
With Tensorlake, every `@function()` is already a sandbox. You get process isolation, resource limits, timeout enforcement, and dependency isolation without any extra setup.
## Pattern 2: Sandbox as Tool
The agent runs in a Tensorlake function and gets sandboxes as tools it can use for code execution. When the agent needs to run untrusted or LLM-generated code, it creates a sandbox on demand, executes code there, and reads the results back.
```mermaid theme={null}
graph LR
A["Agent Function"] -- "Create" --> B["Sandbox 1"]
A -- "Create" --> C["Sandbox 2"]
A -- "Create" --> D["Sandbox 3"]
```
Tensorlake's [Sandbox API](/sandboxes/introduction) provides this pattern. Your agent logic runs in a `@function()`, and when it needs to execute code, it creates a sandbox with the `Sandbox` SDK and uses it as a tool.
```python theme={null}
from tensorlake.applications import application, function, Image
agent_image = Image().run("pip install openai tensorlake")
@application()
@function(image=agent_image, timeout=1800)
def coding_agent(task: str) -> str:
"""Agent uses a sandbox as a tool for code execution."""
from openai import OpenAI
from tensorlake.sandbox import Sandbox
client = OpenAI()
# Create an on-demand sandbox for code execution
sandbox = Sandbox.create(
image="tensorlake/ubuntu-minimal",
cpus=1.0,
memory_mb=1024,
timeout_secs=60,
)
try:
messages = [{"role": "user", "content": task}]
for _ in range(20):
response = client.chat.completions.create(model="gpt-4o", messages=messages)
reply = response.choices[0].message
if not reply.tool_calls:
return reply.content
for tool_call in reply.tool_calls:
if tool_call.function.name == "run_code":
# Code executes in the remote sandbox, not here
result = sandbox.execute(tool_call.function.arguments)
messages.append({"role": "tool", "content": result.output})
finally:
sandbox_client.delete(sandbox.sandbox_id)
```
**When to use this pattern:**
* You need to execute untrusted or LLM-generated code
* API keys should stay outside the code execution environment
* You want to spin up multiple sandboxes in parallel for concurrent code execution
* The agent needs to create, inspect, and tear down environments dynamically
**Trade-offs:**
* Network latency on each execution call
* Two layers of containers (agent function + sandbox)
## Learn More
Install Tensorlake and create your first sandbox.
Sandbox states, resources, timeouts, and lifecycle operations.
Control internet access and blocked destinations.
# Scale-Out & Queuing
Source: https://docs.tensorlake.ai/applications/scale-out-queuing
Workflows scale automatically as endpoints are called, with configurable scaling per function
Workflows scale out automatically as their endpoints are called. When you invoke a workflow, Tensorlake spins up containers for each function as needed, processes the request, and scales back down when idle.
Each function in your workflow can have its own scaling configuration. You control scaling behavior with two parameters: `warm_containers` and `max_containers`.
**Example workflow with scaling:**
```python theme={null}
from tensorlake.applications import application, function
@function(warm_containers=2, max_containers=10)
def enrich_data(record_id: str) -> dict:
# 2 containers always warm, scales up to 10
...
@function()
def transform(data: dict) -> dict:
# Transform the enriched data
...
@application()
@function()
def process_workflow(record_id: str) -> dict:
enriched = enrich_data.future(record_id)
return transform.future(enriched)
```
When you call `POST /applications/process_workflow`, the workflow endpoint scales automatically, and each function scales based on its configuration.
## Scaling Parameters
Configure scaling in the `@function()` decorator:
```python theme={null}
from tensorlake.applications import function
@function(
warm_containers=2,
max_containers=10
)
def process_data(data: str) -> str:
...
```
### `warm_containers`
Number of pre-warmed containers to keep ready. Warm containers have your code and dependencies loaded, eliminating cold start latency for incoming requests.
```python theme={null}
@function(warm_containers=3)
def classify_document(content: str) -> str:
"""3 containers are always warm and ready to handle requests."""
# Critical first step in workflow - needs low latency
...
```
Use warm containers when:
* You need low-latency responses
* Cold starts are unacceptable for your use case
* You have predictable baseline traffic
### `max_containers`
Maximum number of containers. Once this limit is reached, additional requests are automatically queued and processed in FIFO order as containers become available.
```python theme={null}
@function(max_containers=5)
def bounded_processing(data: str) -> str:
"""No more than 5 containers will run simultaneously."""
...
```
## Automatic Queuing
When all containers for a function are busy and `max_containers` has been reached, Tensorlake automatically queues incoming requests. No configuration is needed. Queuing is built into the platform.
* Requests are processed in **FIFO order**
* Queued requests begin processing as soon as a container becomes available
* No separate queue infrastructure (Redis, SQS, RabbitMQ) is required
```python theme={null}
@function(max_containers=3)
def process_with_llm(data: str) -> str:
"""At most 3 concurrent LLM calls. Additional requests are queued."""
# Expensive workflow step - limit concurrency to control costs
...
```
## Combined Behaviors
Combine parameters for fine-grained control:
### Low-latency with bounded scale
```python theme={null}
@function(
warm_containers=2, # 2 containers ready for instant response
max_containers=10 # Scale up to 10, then queue
)
def extract_entities(document: str) -> dict:
# First step in document workflow - needs low latency
...
```
### High-throughput with bounded cost
```python theme={null}
@function(
warm_containers=4, # 4 containers pre-warmed
max_containers=50 # Scale up to 50, then queue
)
def enrich_from_api(record_id: str) -> dict:
# High-volume workflow step with bounded scale
...
```
## Scaling in Workflows
Each function in your workflow scales independently. This allows different workflow steps to have different scaling profiles based on their resource requirements and latency needs:
```python theme={null}
@function(warm_containers=5, max_containers=50)
def fetch_data(record_id: str) -> dict:
"""High-throughput data fetching with low latency."""
...
@function(max_containers=3)
def analyze_with_llm(data: dict) -> dict:
"""Expensive LLM analysis, bounded concurrency to control costs."""
...
@application()
@function()
def process_record(record_id: str) -> dict:
# fetch_data can handle 50 concurrent requests
data = fetch_data.future(record_id)
# analyze_with_llm is limited to 3 concurrent executions
return analyze_with_llm.future(data)
```
In this workflow, `fetch_data` can scale to 50 containers for high throughput, while `analyze_with_llm` is capped at 3 to control costs. When you call the `process_record` endpoint, both functions scale independently based on their configuration.
## Default Behavior
Without any scaling parameters, workflow functions scale dynamically:
* Containers scale from zero based on demand when the workflow endpoint is called
* There is no upper bound on container count
* Cold starts occur for the first request after an idle period
* No automatic queuing (unlimited scaling)
## Learn More
Full @function() decorator reference.
Structuring agents for scale.
Multi-step data workflows.
# Autoscaling
Source: https://docs.tensorlake.ai/applications/scaling-agents
Autoscaling guide for Orchestration endpoints
Tensorlake scales your `@function()` sandboxes automatically.
In most cases, you do not need to configure anything. Start with defaults, then tune only if you have a specific latency or cost goal.
## Default Behavior
With just `@function()`, Tensorlake does this automatically:
* Creates containers when requests arrive
* Scales to zero when idle
* Adds more containers as traffic grows
```python theme={null}
from tensorlake.applications import function
@function()
def agent(prompt: str) -> str:
...
```
This is the simplest and most cost-efficient setup for many async and internal workloads.
## Scaling Settings
Use these only when default on-demand scaling is not enough:
| Setting | What it controls | What happens |
| ----------------- | --------------------- | --------------------------------------------------------------- |
| `warm_containers` | Ready-to-serve buffer | Keeps extra pre-started containers ready so bursts start faster |
| `max_containers` | Capacity ceiling | Caps total containers so scale and cost stay bounded |
How they work together:
* `warm_containers` adds ready capacity above current demand.
* `max_containers` limits the final upper bound.
* If demand exceeds `max_containers`, requests wait in queue.
## Practical Examples
### 1) Reduce cold starts
If this is a user-facing endpoint and startup delay is noticeable:
```python theme={null}
@function(warm_containers=2)
def agent(prompt: str) -> str:
...
```
### 2) Cap spend or protect downstream APIs
If you need to bound scale:
```python theme={null}
@function(max_containers=10)
def agent(prompt: str) -> str:
...
```
When all 10 are busy, new requests wait in queue.
### 3) Balance low latency with bounded scale
If you want faster startup plus bounded scaling:
```python theme={null}
@function(warm_containers=2, max_containers=20)
def agent(prompt: str) -> str:
...
```
Result:
* 2 warm containers are ready for faster responses
* Scale is still capped at 20 containers
### 4) High-throughput with a safety ceiling
```python theme={null}
@function(
warm_containers=4,
max_containers=50,
)
def agent(prompt: str) -> str:
...
```
## How to Choose Values
Start with `@function()` and add knobs only for a specific goal:
* Lower first-request latency: set `warm_containers=1`, then increase gradually.
* Budget or downstream protection: set `max_containers` to a safe upper limit.
* Stable setup: add a small `warm_containers` buffer, then cap with `max_containers`.
* Keep changes incremental: update one knob, test, then adjust.
## Learn More
How queueing works when demand exceeds available capacity
Pattern for handling transient API failures safely
# Secrets
Source: https://docs.tensorlake.ai/applications/secrets
Providing secrets to Tensorlake functions
Secrets allow providing sensitive values to your Tensorlake functions in a secure manner without having to put them into your code.
## Storing secrets
You can store secrets on Tensorlake Cloud using the CLI:
```bash theme={null}
tl secrets set AWS_ACCESS_KEY=MY_AWS_ACCESS_KEY
tl secrets set OPENAI_API_KEY=MY_OPENAI_API_KEY
```
## Using secrets
Stored secrets are available as environment variables within your Tensorlake functions:
```
@application()
@function(secrets=["AWS_ACCESS_KEY", "OPENAI_API_KEY"])
def my_function() -> str:
aws_access_key = os.environ["AWS_ACCESS_KEY"]
openai_api_key = os.environ["OPENAI_API_KEY"]
...
```
### Secrets and application deployment
When you add or update a secret used by an already deployed application, it needs to get redeployed for the new secret values to take effect.
## CLI Commands
### List Secrets
List secrets that have been previously set. Values are not shown for security reasons.
```bash theme={null}
$ tl secrets list
| Name | Created At |
| ----------- | ---------- |
| SECRET_NAME | Date |
```
### Set a Secret
Set a secret will create or update a secret.
```bash theme={null}
$ tl secrets set = [=]
```
### Unset a Secret
```bash theme={null}
tl secrets unset []
```
## Security
Secrets use envelope encryption with AES-256-GCM, providing strong confidentiality and integrity.
Each project has a dedicated Data Encryption Key (DEK) wrapped by a root Key Encryption Key (KEK) managed by AWS KMS, creating strict
isolation boundaries.
Secrets remain encrypted at rest and are only decrypted in-memory on dataplane machines running workflows
that requires those secrets, with all communication secured through mutual TLS (mTLS).
# Timeouts
Source: https://docs.tensorlake.ai/applications/timeouts
How function timeouts work and how progress updates reset them
Agents can take an unpredictable amount of time to do their work: an LLM tool-calling loop might finish in seconds or run for hours depending on the task. Tensorlake handles this by letting functions run indefinitely as long as they keep sending heartbeats in the form of progress updates. A timeout only kicks in if a function stops making progress: it doesn't finish within the allotted time *and* it doesn't send any progress updates to reset the clock.
## Setting Timeouts
Set the `timeout` attribute on the `@function()` decorator to control how long a function can run before it is terminated and marked as failed.
```python theme={null}
from tensorlake.applications import function
@function(timeout=1800) # 30 minutes
def deep_research(prompt: str) -> str:
...
```
| | Value |
| ----------- | ------------------- |
| **Default** | `300` (5 minutes) |
| **Minimum** | `1` second |
| **Maximum** | `172800` (48 hours) |
When a function times out, it is terminated and marked as failed. If the function has a [retry policy](/applications/retries), it will be retried according to that policy. Already-completed nested calls are served from checkpoints on retry (see [Durable Execution](/applications/durability)).
## Automatic Timeout Reset
When a function reports progress via `ctx.progress.update()`, its timeout automatically resets. This allows functions to run indefinitely as long as they continue making progress.
```python theme={null}
from tensorlake.applications import function, RequestContext
@function(timeout=300) # 5 minute timeout
def long_running_task(items: list) -> dict:
ctx = RequestContext.get()
# After 3 minutes of processing...
ctx.progress.update(50, 100, "Halfway done")
# Timeout just reset to 5 minutes from NOW
# Function can run another 5 minutes before next update
# Continue processing...
```
**Timeline:**
1. Function starts with 5 minute timeout
2. At 3 minutes: `progress.update()` called
3. Timeout resets to 5 minutes from this point
4. Function can now run until minute 8 (3 + 5)
5. Next `progress.update()` resets timeout again
Set a short timeout (e.g., 5 minutes) and rely on progress updates to extend it. This way, if a function gets stuck and stops reporting progress, it fails fast instead of running silently for hours.
## Examples
### Agent Loops
An agent that runs hundreds of iterations can use a short timeout per iteration. Each progress update resets the clock:
```python theme={null}
@function(timeout=300) # 5 minute timeout per iteration
def persistent_agent(task: str) -> str:
ctx = RequestContext.get()
for iteration in range(1000):
ctx.progress.update(iteration, 1000) # resets timeout
result = agent_iteration(task)
if is_complete(result):
return result
```
### Batch Processing
Process an unbounded stream of items. The function runs as long as items keep arriving:
```python theme={null}
@function(timeout=600) # 10 minute timeout
def process_stream(stream_url: str) -> dict:
ctx = RequestContext.get()
count = 0
for item in stream_items(stream_url):
ctx.progress.update(count, count + 1, f"Processed {count} items")
process(item)
count += 1
return {"total": count}
```
### Video/Audio Processing
Report progress every N frames to keep the timeout from firing during long media processing:
```python theme={null}
@function(timeout=300)
def process_video(video_url: str) -> str:
ctx = RequestContext.get()
frames = extract_frames(video_url)
total_frames = len(frames)
for i, frame in enumerate(frames):
if i % 100 == 0:
ctx.progress.update(i, total_frames, f"Processing frame {i}/{total_frames}")
process_frame(frame)
return "complete"
```
## Learn More
The full progress API, frontend integration, and SSE streaming.
What happens after a timeout fires: auto-retry with checkpoint reuse.
Resume a request from where it timed out instead of restarting.
How nested completed calls are reused across timeout-triggered retries.
Catch timeouts in caller code and degrade gracefully.
Functions, retries, resource limits, and request context.
# Agent with Tool Calling
Source: https://docs.tensorlake.ai/examples/agentic-applications/agent-with-tools
Build a Claude agent that orchestrates complex workflows using tool calls.
Check out the full source code for this example on GitHub.
This tutorial demonstrates how to build an **Agent with Tool Calling** using Tensorlake and the Anthropic API. This agent orchestrates a multi-step workflow where Claude decides which tools to call and in what order to answer user queries effectively.
## Overview
The Agent with Tool Calling follows this pattern:
1. **User Query**: The user asks a question that requires external data or actions (e.g., "What's the weather like at my current location?").
2. **Tool Selection**: Claude analyzes the query and selects the appropriate tool(s) to call (e.g., `get_ip_address`, `get_location_info`).
3. **Tool Execution**: The selected tool functions are executed within Tensorlake's isolated environment.
4. **Information Synthesis**: The tool outputs are fed back to Claude, which then synthesizes a final answer or decides to call more tools.
## Prerequisites
* **Python 3.11+**
* **Tensorlake Account** and CLI installed.
* **Anthropic API Key**
## Implementation (`app.py`)
Here is the complete implementation for the Agent with Tool Calling.
```python theme={null}
import os
from typing import Dict, Any
from anthropic import Anthropic
from pydantic import BaseModel, Field
from tensorlake import application, function, Image
# Define the runtime environment
image = Image(name="agent-with-tools").run("pip install anthropic requests tensorlake pydantic")
class ToolInput(BaseModel):
name: str
arguments: Dict[str, Any]
@application()
@function(image=image, secrets=["ANTHROPIC_API_KEY"])
def agent_loop(query: str) -> str:
"""
Main entry point for the Agent with Tool Calling.
Orchestrates the conversation loop between Claude and the tools.
"""
client = Anthropic()
messages = [{"role": "user", "content": query}]
# Define available tools for Claude
tools = [
{
"name": "get_ip_address",
"description": "Get the public IP address of the current execution environment.",
"input_schema": {"type": "object", "properties": {}}
},
{
"name": "get_location_info",
"description": "Get location information based on an IP address.",
"input_schema": {
"type": "object",
"properties": {
"ip_address": {"type": "string", "description": "The IP address to lookup."}
},
"required": ["ip_address"]
}
},
{
"name": "get_weather_alerts",
"description": "Get current weather alerts for a location.",
"input_schema": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "City or location name."}
},
"required": ["location"]
}
}
]
while True:
# Ask Claude for the next step
response = client.messages.create(
model="claude-3-opus-20240229",
max_tokens=1024,
tools=tools,
messages=messages
)
# If Claude decides to stop and give an answer
if response.stop_reason == "end_turn":
return response.content[0].text
# If Claude wants to use a tool
if response.stop_reason == "tool_use":
tool_use = next(block for block in response.content if block.type == "tool_use")
tool_name = tool_use.name
tool_input = tool_use.input
tool_use_id = tool_use.id
print(f"Tool Call: {tool_name} with input: {tool_input}")
# Execute the requested tool
if tool_name == "get_ip_address":
tool_result = get_ip_address()
elif tool_name == "get_location_info":
tool_result = get_location_info(tool_input["ip_address"])
elif tool_name == "get_weather_alerts":
tool_result = get_weather_alerts(tool_input["location"])
else:
tool_result = f"Error: Tool {tool_name} not found."
# Add tool result to conversation history
messages.append({"role": "assistant", "content": response.content})
messages.append({
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": tool_use_id,
"content": str(tool_result)
}
]
})
@function(image=image)
def get_ip_address() -> str:
"""Simulates getting the public IP address."""
# In a real scenario, you might use `requests.get('https://api.ipify.org').text`
return "203.0.113.1"
@function(image=image)
def get_location_info(ip_address: str) -> str:
"""Simulates getting location info for an IP."""
return f"Location for {ip_address}: San Francisco, CA"
@function(image=image)
def get_weather_alerts(location: str) -> str:
"""Simulates getting weather alerts."""
return f"No active weather alerts for {location}."
```
## Running Locally
To test the agent locally, add this code block to the end of `app.py`:
```python theme={null}
if __name__ == "__main__":
from tensorlake.applications import run_local_application
# Run the application locally
result = run_local_application(agent_loop, "What are the current weather alerts for my location?")
print(f"Agent Output: {result}")
```
Then run the script:
```bash theme={null}
export ANTHROPIC_API_KEY=your_key_here
python app.py
```
## Deploying to Tensorlake
Deploy your agent to the cloud for production use:
```bash theme={null}
tl secrets set ANTHROPIC_API_KEY=your_key_here
tl app deploy app.py
```
Your agent is now live! It can autonomously chain tool calls to solve complex user requests, all running within secure, scalable Tensorlake functions.
# Code Interpreter Agent
Source: https://docs.tensorlake.ai/examples/agentic-applications/code-interpreter
Build a secure code execution environment using Tensorlake and OpenAI.
Check out the full source code for this example on GitHub.
This tutorial demonstrates how to build a **Code Interpreter Agent** that can safely execute Python code generated by an LLM. By leveraging Tensorlake's isolated sandboxing, you can run arbitrary code without compromising your local environment or production servers.
## Overview
The Code Interpreter Agent follows this workflow:
1. **User Request**: The user asks a question that requires code execution (e.g., "Calculate the Fibonacci sequence up to 100").
2. **Code Generation**: An OpenAI agent interprets the request and generates the necessary Python code.
3. **Secure Execution**: The code is sent to a Tensorlake function running in a secure, isolated container.
4. **Result Retrieval**: The execution output (stdout, stderr) is captured and returned to the agent.
5. **Final Answer**: The agent formulates a final response based on the code output.
## Prerequisites
* **Python 3.11+**
* **Tensorlake Account** and CLI installed.
* **OpenAI API Key**
## Implementation (`app.py`)
Here is the complete implementation for the Code Interpreter Agent.
```python theme={null}
import sys
import io
import contextlib
from typing import Optional
from openai import OpenAI
from pydantic import BaseModel, Field
from tensorlake import application, function, Image
# Define the execution environment
# We install pandas and numpy to support data analysis tasks
image = Image(name="code-interpreter").run("pip install openai pandas numpy")
class ExecutionResult(BaseModel):
stdout: str
stderr: str
result: Optional[str] = None
@application()
@function(image=image, secrets=["OPENAI_API_KEY"])
def code_interpreter(prompt: str) -> str:
"""
Main entry point for the Code Interpreter Agent.
"""
client = OpenAI()
# Step 1: Generate code based on user prompt
completion = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a Python code generator. Output only valid Python code to solve the user's problem. Do not include markdown blocks."},
{"role": "user", "content": prompt}
]
)
code = completion.choices[0].message.content
# Step 2: Execute the generated code securely
print(f"Executing generated code:
{code}")
execution_result = execute_python_code(code)
# Step 3: formulate final answer
final_response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "Answer the user's question based on the code execution result."},
{"role": "user", "content": f"Question: {prompt}
Code Output:
{execution_result.stdout}
Errors:
{execution_result.stderr}"}
]
)
return final_response.choices[0].message.content
@function(image=image)
def execute_python_code(code: str) -> ExecutionResult:
"""
Executes Python code in a secure sandbox and captures output.
"""
stdout = io.StringIO()
stderr = io.StringIO()
try:
# Redirect stdout and stderr to capture print statements
with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr):
exec(code, {"__name__": "__main__"})
except Exception as e:
print(f"Execution error: {e}", file=stderr)
return ExecutionResult(
stdout=stdout.getvalue(),
stderr=stderr.getvalue()
)
```
## Running Locally
To test the interpreter locally, add this code block to the end of `app.py`:
```python theme={null}
if __name__ == "__main__":
from tensorlake.applications import run_local_application
# Run the application locally
result = run_local_application(code_interpreter, "Calculate the sum of the first 50 prime numbers.")
print(f"Code Interpreter Output:\n{result}")
```
Then run the script:
```bash theme={null}
export OPENAI_API_KEY=your_key_here
python app.py
```
## Deploying to Tensorlake
Deploy your secure code interpreter to the cloud with a single command:
```bash theme={null}
tl secrets set OPENAI_API_KEY=your_key_here
tl app deploy app.py
```
This deployment creates a dedicated, isolated environment for every execution request, ensuring complete safety and scalability for your code interpretation tasks.
# Deep Research Agent
Source: https://docs.tensorlake.ai/examples/agentic-applications/deep-research
Build a multi-agent deep research pipeline with Tensorlake and OpenAI.
Check out the full source code for this example on GitHub.
This tutorial demonstrates how to build a **Deep Research Agent** using Tensorlake and the OpenAI Agents SDK. This application orchestrates multiple agents to plan, search, and write comprehensive research reports on any given topic.
## Overview
The Deep Research Agent consists of three specialized agents that work together in a pipeline:
1. **Planner Agent**: Breaks down the user's research topic into specific search queries and steps.
2. **Search Agent**: Executes the planned search queries in parallel, retrieving and summarizing relevant information from the web.
3. **Writer Agent**: Synthesizes the gathered information into a structured, comprehensive markdown report.
Each agent runs as an isolated, serverless function on Tensorlake, ensuring scalability and fault tolerance.
## Prerequisites
* **Python 3.11+**
* **Tensorlake Account** and CLI installed.
* **OpenAI API Key**
## Project Structure
Your project should look like this:
```text theme={null}
deep-research/
├── app.py # Main application logic and Tensorlake functions
├── models.py # Pydantic data models for structured inputs/outputs
├── prompts.py # System prompts for the agents
└── requirements.txt
```
## Implementation
### 1. Define Data Models (`models.py`)
First, we define the data structures that our agents will use to communicate. This ensures type safety and clear interfaces between the agents.
```python theme={null}
from pydantic import BaseModel, Field
from typing import List
class SearchQuery(BaseModel):
query: str = Field(..., description="A specific search query to execute.")
rationale: str = Field(..., description="Why this query is important.")
class ResearchPlan(BaseModel):
topic: str
search_queries: List[SearchQuery]
class SearchResult(BaseModel):
url: str
title: str
content: str
summary: str
class ResearchReport(BaseModel):
topic: str
markdown_content: str
references: List[str]
```
### 2. Create the Agents (`app.py`)
In `app.py`, we define our Tensorlake functions. Each function represents a stage in the pipeline and utilizes an OpenAI agent.
```python theme={null}
import os
from typing import List
from openai import OpenAI
from tensorlake import application, function, Image
from pydantic import BaseModel
from models import ResearchPlan, SearchResult, ResearchReport
# Import your prompts here
# from prompts import PLANNER_PROMPT, SEARCH_PROMPT, WRITER_PROMPT
# Define the runtime image
image = Image(name="deep-research-agent").run("pip install openai tensorlake pydantic")
@application()
@function(image=image, secrets=["OPENAI_API_KEY"])
def deep_research_pipeline(topic: str) -> ResearchReport:
"""
Orchestrates the deep research pipeline.
"""
print(f"Starting deep research on: {topic}")
# Phase 1: Planning
plan = create_research_plan(topic)
print(f"Plan created with {len(plan.search_queries)} queries.")
# Phase 2: Searching (Parallel Execution)
# We map the search function over the queries to run them in parallel
search_results = execute_search.map(plan.search_queries)
print(f"Completed {len(search_results)} searches.")
# Phase 3: Writing
report = write_report(topic, search_results)
print("Report generation complete.")
return report
@function(image=image, secrets=["OPENAI_API_KEY"])
def create_research_plan(topic: str) -> ResearchPlan:
client = OpenAI()
completion = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are an expert research planner."},
{"role": "user", "content": f"Create a research plan for: {topic}"}
],
response_format=ResearchPlan
)
return completion.choices[0].message.parsed
@function(image=image, secrets=["OPENAI_API_KEY"])
def execute_search(query_obj) -> SearchResult:
# In a real implementation, you would use a search tool or API here.
# For this example, we'll simulate a search result.
# You could use tools like Tavily, Serper, or a custom scraper.
client = OpenAI()
# Simulate processing the query
summary = f"Simulated search results for: {query_obj.query}"
return SearchResult(
url="https://example.com",
title=f"Results for {query_obj.query}",
content="Full content would go here...",
summary=summary
)
@function(image=image, secrets=["OPENAI_API_KEY"])
def write_report(topic: str, results: List[SearchResult]) -> ResearchReport:
client = OpenAI()
# Compile context from search results
context = "
".join([f"Source: {r.url}
Summary: {r.summary}" for r in results])
completion = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are an expert research writer. Write a comprehensive report based on the provided context."},
{"role": "user", "content": f"Topic: {topic}
Context:
{context}"}
],
response_format=ResearchReport
)
return completion.choices[0].message.parsed
```
## Running Locally
To test your pipeline locally, add this code block to the end of `app.py` and run it with python.
```python theme={null}
if __name__ == "__main__":
from tensorlake.applications import run_local_application
# Run the application locally
run_local_application(deep_research_pipeline, "The Future of Quantum Computing")
```
Then run the script:
```bash theme={null}
export OPENAI_API_KEY=your_key_here
python app.py
```
## Deploying to Tensorlake
When you're ready to deploy, use the `tl app deploy` command.
```bash theme={null}
tl secrets set OPENAI_API_KEY=your_key_here
tl app deploy app.py
```
Your deep research agent is now live and scalable! You can invoke it via the provided HTTP endpoint or the Tensorlake SDK.
# Personal Finance Manager
Source: https://docs.tensorlake.ai/examples/agentic-applications/personal-finance-manager
Build an AI agent on Tensorlake that analyzes bank statements and answers spending questions.
Check out the full source code for this example on GitHub.
This tutorial demonstrates how to build a **Personal Finance Manager** using Tensorlake and Claude. This application can parse PDF bank statements, categorize transactions using LLMs, store them in a PostgreSQL database, and answer natural language questions about your spending.
## Overview
The Personal Finance Manager consists of two main agents:
1. **Finance Analyzer Agent**: Parses PDF statements, extracts transactions, categorizes them using Claude, and stores them in a database.
2. **Finance Query Agent**: Translates natural language questions (e.g., "How much did I spend on groceries?") into SQL queries, executes them, and visualizes the results.
## Prerequisites
* **Python 3.11+**
* **Tensorlake Account** and CLI installed.
* **Anthropic API Key**
* **PostgreSQL Database** (e.g., Neon, Supabase, or local)
## Project Structure
```text theme={null}
personal-finance/
├── app.py # Main application logic and agents
├── models.py # Pydantic models for transactions and queries
├── config.py # Configuration and prompts
└── requirements.txt
```
## Implementation (`app.py`)
Here is a simplified view of the core logic for the Finance Analyzer Agent.
```python theme={null}
import os
from typing import List
from anthropic import Anthropic
from tensorlake import Agent, Assistant, File, User, configure_logging, get_logger, Image, application, function
# import your models and config here
# Define the runtime environment with necessary dependencies
image = Image(name="finance-manager").run("pip install anthropic pandas psycopg2-binary tensorlake pydantic")
class FinanceAnalyzerAgent(Agent):
"""
Parses PDF statements and stores categorized transactions.
"""
def __init__(self, name: str, system_prompt: str, tools: List):
super().__init__(name, system_prompt, tools)
# Initialize Anthropic client and DB connection
self.anthropic_client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
def call(self, user_message: User, files: List[File]) -> Assistant:
if not files:
return Assistant(content="Please upload a PDF statement.")
pdf_file = files[0]
# 1. Extract text from PDF (using a helper or library)
pdf_text = self._extract_text(pdf_file)
# 2. Extract transactions using Claude
transactions = self._extract_transactions_from_text(pdf_text)
# 3. Categorize transactions using Claude
categorized = self._categorize_transactions(transactions)
# 4. Store in Database
self._insert_into_db(categorized)
return Assistant(content=f"Processed statement. Added {len(categorized)} transactions.")
# Helper methods implementation...
```
And the Finance Query Agent:
```python theme={null}
class FinanceQueryAgent(Agent):
"""
Answers questions about financial data using SQL.
"""
def call(self, user_message: User, files: List) -> Assistant:
# 1. Get Database Schema
schema = self._get_db_schema()
# 2. Generate SQL query using Claude based on user question and schema
sql_query = self._generate_sql(user_message.content, schema)
# 3. Execute SQL
results = self._execute_sql(sql_query)
# 4. Formulate answer
answer = self._generate_answer(user_message.content, results)
return Assistant(content=answer)
```
## Running Locally
1. Set up your environment variables:
```bash theme={null}
export ANTHROPIC_API_KEY=your_key
export DATABASE_URL=postgresql://user:password@host:port/dbname
```
2. Run the application:
```bash theme={null}
python app.py path/to/statement.pdf
```
## Deploying to Tensorlake
Deploy your finance manager to the cloud securely.
```bash theme={null}
tl secrets set ANTHROPIC_API_KEY=your_key
tl secrets set DATABASE_URL=your_db_url
tl app deploy app.py
```
Your personal finance assistant is now ready to help you track your spending!
# Weather Agent
Source: https://docs.tensorlake.ai/examples/agentic-applications/weather-agent
Build a conversational weather agent using Tensorlake and OpenWeatherMap.
Check out the full source code for this example on GitHub.
This tutorial demonstrates how to build a **Weather Agent** using Tensorlake and the OpenWeatherMap API. This agent can fetch real-time weather data for any location and answer natural language questions about it.
## Overview
The Weather Agent consists of:
1. **Weather Tool**: A Python function that calls the OpenWeatherMap API.
2. **Weather Agent**: An LLM-powered agent that understands user queries (e.g., "Will I need an umbrella in London today?") and decides when to call the weather tool.
## Prerequisites
* **Python 3.11+**
* **Tensorlake Account** and CLI installed.
* **OpenWeatherMap API Key**
* **Anthropic API Key**
## Project Structure
```text theme={null}
weather-app/
├── agent.py # Weather Agent logic and API calls
├── tensorlake_app.py # Main application entry point
├── config.py # Configuration and prompts
└── requirements.txt
```
## Implementation (`agent.py`)
Here is the core logic for the Weather Agent.
```python theme={null}
import os
import requests
from typing import List
from anthropic import Anthropic
from pydantic import BaseModel, Field
from tensorlake import Agent, Assistant, Function, FunctionTool, User, configure_logging, get_logger, Image, function
# Define the runtime environment
image = Image(name="weather-agent").run("pip install anthropic requests tensorlake pydantic")
class GetCurrentWeatherInput(BaseModel):
location: str = Field(..., description="The city and country, e.g., 'London, UK' or 'New York, USA'.")
@function(image=image, secrets=["OPENWEATHER_API_KEY"])
def get_current_weather(location: str) -> str:
"""
Fetches current weather data for a specified location using the OpenWeatherMap API.
"""
api_key = os.getenv("OPENWEATHER_API_KEY")
base_url = "http://api.openweathermap.org/data/2.5/weather"
params = {
"q": location,
"appid": api_key,
"units": "metric",
}
response = requests.get(base_url, params=params)
if response.status_code == 200:
data = response.json()
desc = data['weather'][0]['description']
temp = data['main']['temp']
return f"Weather in {location}: {desc}, {temp}°C"
else:
return f"Could not fetch weather for {location}."
class WeatherAgent(Agent):
def __init__(self, name: str, system_prompt: str, tools: List[FunctionTool]):
super().__init__(name, system_prompt, tools)
self.anthropic_client = Anthropic()
def call(self, user_message: User, files: List) -> Assistant:
messages = [{"role": "user", "content": user_message.content}]
response = self.anthropic_client.messages.create(
model="claude-3-opus-20240229",
max_tokens=1024,
system=self.system_prompt,
messages=messages,
tools=self.tools,
)
if response.stop_reason == "tool_use":
tool_use = next(block for block in response.content if block.type == "tool_use")
if tool_use.name == "get_current_weather":
location = tool_use.input["location"]
weather_info = get_current_weather(location)
messages.append({"role": "assistant", "content": response.content})
messages.append({
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": tool_use.id,
"content": weather_info
}
]
})
final_response = self.anthropic_client.messages.create(
model="claude-3-opus-20240229",
max_tokens=1024,
system=self.system_prompt,
messages=messages,
tools=self.tools,
)
return Assistant(content=final_response.content)
return Assistant(content=response.content)
# Initialize the agent
weather_agent = WeatherAgent(
name="weather_agent",
system_prompt="You are a helpful weather assistant.",
tools=[
FunctionTool(
name="get_current_weather",
description="Get current weather for a location.",
function=Function(
name="get_current_weather",
description="Get current weather for a location.",
input_schema=GetCurrentWeatherInput.model_json_schema(),
),
)
]
)
```
## Running Locally
1. Set up environment variables:
```bash theme={null}
export OPENWEATHER_API_KEY=your_openweather_key
export ANTHROPIC_API_KEY=your_anthropic_key
```
2. Run the agent:
```bash theme={null}
python agent.py
```
## Deploying to Tensorlake
Deploy your weather agent to the cloud.
```bash theme={null}
tl secrets set OPENWEATHER_API_KEY=your_openweather_key
tl secrets set ANTHROPIC_API_KEY=your_anthropic_key
tl app deploy agent.py
```
Your weather agent is now live and ready to answer queries!
# Web Scraper to MongoDB Atlas
Source: https://docs.tensorlake.ai/examples/agentic-applications/web-scraper
Build a scalable web scraper that stores vector embeddings in MongoDB Atlas.
Check out the full source code for this example on GitHub.
This tutorial demonstrates how to build a production-grade **Web Scraper** that crawls websites, processes content into clean Markdown, generates embeddings using Voyage AI, and stores them in MongoDB Atlas Vector Search.
## Overview
This application showcases the power of Tensorlake's parallel processing capabilities:
1. **Parallel Crawling**: Uses Breadth-First Search (BFS) with Tensorlake's `.map()` to fetch multiple pages concurrently at each depth level.
2. **Headless Browsing**: Utilizes **PyDoll** (based on Chromium) to render JavaScript-heavy websites.
3. **Content Cleaning**: Converts HTML and PDFs to clean Markdown, automatically removing boilerplate like headers, footers, and ads.
4. **Vector Embeddings**: Generates high-quality embeddings for document chunks using **Voyage AI**.
5. **Vector Search**: Stores the processed chunks and embeddings directly into **MongoDB Atlas** for RAG applications.
## Prerequisites
* **Python 3.11+**
* **Tensorlake Account** and CLI installed.
* **MongoDB Atlas** cluster URI.
* **Voyage AI** API Key.
## Implementation
The application is defined in a single file, `scraper_to_atlas.py`. It defines two custom runtime images: one for scraping (with Chromium) and one for embedding (lightweight).
### 1. Define Dependencies and Images
```python theme={null}
from tensorlake.applications import Image
# Image with Chromium, pydoll, and dependencies for web scraping
scraper_image = (
Image(name="scraper-to-atlas-image", base_image="python:3.11.0")
.env("DEBIAN_FRONTEND", "noninteractive")
.run("apt-get update && apt-get install -y chromium ...") # System deps
.run("pip install pydoll-python tensorlake beautifulsoup4 markdownify pymupdf4llm")
)
# Image for embedding and MongoDB operations
embedding_image = (
Image(name="embedding-image", base_image="python:3.11.0")
.run("pip install tensorlake voyageai pymongo")
)
```
### 2. Main Scraping Logic (`scraper_to_atlas.py`)
The `@application` entry point orchestrates the crawling process. It manages the BFS queue and dispatches parallel tasks using `fetch_and_convert.map()`.
```python theme={null}
@application()
@function(secrets=["VOYAGE_API_KEY", "MONGO_URI"])
def scrape_and_embed(input: ScrapeAndEmbedInput) -> dict:
# ... setup BFS ...
# Phase 1: Parallel BFS
for depth in range(max_depth + 1):
# ... deduce URLs to fetch ...
# Parallel fetch all URLs at this depth level using map()
results = fetch_and_convert.map(urls_to_fetch)
# ... process results and collect new links ...
# Phase 2: Process PDFs in parallel
if pdf_urls:
pdf_results = fetch_and_convert_pdf.map(list(pdf_urls))
# Phase 3: Generate embeddings and store
embed_and_store(all_documents, ...)
```
### 3. Page Fetching and Conversion
The `fetch_and_convert` function runs in the `scraper_image` and uses PyDoll to render pages.
```python theme={null}
@function(image=scraper_image, timeout=120, memory=4)
def fetch_and_convert(url: str) -> dict:
return asyncio.run(_fetch_and_convert_async(url))
async def _fetch_and_convert_async(url: str) -> dict:
async with Chrome() as browser:
page = await browser.new_page()
await page.goto(url)
html = await page.content()
# ... extract title and links ...
# Convert to clean markdown
markdown = _html_to_markdown(html)
chunks = _chunk_text(markdown)
return {"url": url, "chunks": chunks, ...}
```
### 4. Embedding and Storage
The `embed_and_store` function runs in the `embedding_image` and handles interaction with Voyage AI and MongoDB.
```python theme={null}
@function(image=embedding_image, secrets=["VOYAGE_API_KEY", "MONGO_URI"])
def embed_and_store(documents, mongo_uri, voyage_api_key, ...):
# Initialize Voyage AI
vo = voyageai.Client(api_key=voyage_api_key)
# Generate embeddings
embeddings = vo.embed(texts=[d["text"] for d in documents], model="voyage-4-large")
# Store in MongoDB
client = pymongo.MongoClient(mongo_uri)
collection = client[db_name][col_name]
collection.insert_many([{...} for ...])
```
## Running Locally
1. Set your environment variables:
```bash theme={null}
export MONGO_URI="mongodb+srv://..."
export VOYAGE_API_KEY="voyage-..."
```
2. Run the application:
```bash theme={null}
python scraper_to_atlas.py
```
## Deploying to Tensorlake
Deploy your scalable scraper to the cloud.
```bash theme={null}
tl secrets set MONGO_URI="mongodb+srv://..."
tl secrets set VOYAGE_API_KEY="voyage-..."
tl app deploy scraper_to_atlas.py
```
Your scraper will now run in the cloud, automatically scaling to handle hundreds of pages in parallel!
# How much does Tensorlake cost and what are the limits?
Source: https://docs.tensorlake.ai/faqs/pricing-limits-faq
Tensorlake pricing, covering the free tier, On-Demand, Pro, and Enterprise plans, per-second sandbox billing, CPU and RAM rates, plan limits, and resource defaults.
Common questions about Tensorlake's pricing model, plan tiers, and the resource limits that apply to sandboxes and image builds. For the live pricing page, see [tensorlake.ai/pricing](https://www.tensorlake.ai/pricing).
## How is sandbox-as-a-service pricing typically structured?
Sandbox-as-a-service products are usually billed on usage rather than a flat seat fee. The common dimensions are:
* **Compute time**: CPU-seconds or vCPU-hours while the sandbox is running.
* **Memory-time**: GB-seconds or GB-hours of allocated memory.
* **Storage**: disk for snapshots, suspended-state preservation, and persisted volumes.
* **Network egress**: outbound bandwidth.
* **Build minutes**: time spent building custom images, sometimes metered separately.
Tensorlake Cloud uses **per-second sandbox billing** with separate CPU and RAM rates, plus per-GB-hour storage for suspended snapshots.
## Does Tensorlake have a free tier?
Yes. The **Free tier is \$0 forever with no credit card required.** It includes:
* 2 concurrent sandboxes
* 1 core / 1 GB RAM / 10 GB disk per sandbox
* Unmetered sessions
* Self-serve docs and community Slack support
* SOC 2 Type 2 compliance
The Free tier's 1 core / 1 GB RAM / 10 GB disk allowance matches the [SDK defaults](#what-are-the-default-resources-for-a-tensorlake-sandbox), so most starter scripts will run unchanged.
## How much does a Tensorlake Sandbox cost?
| | On-Demand | Pro |
| -------------------- | -------------------- | -------------------------------- |
| Base fee | \$0 | \$250 / month |
| CPU | \$0.05 / hr per core | \$0.03 / hr per core *(40% off)* |
| RAM | \$0.015 / hr per GB | 40% off metered usage |
| Concurrent sandboxes | up to 100 | 1,000 |
Sandbox runtime is **billed by the second**. One credit is worth \$0.01. Snapshots are billed **per GB-hour while suspended**.
## What are the Tensorlake plan tiers?
| Plan | Price | Concurrent sandboxes | Highlights |
| -------------- | --------------------- | -------------------- | ----------------------------------------------------------------------------------- |
| **Free** | \$0 forever (no card) | 2 | 1 core / 1 GB RAM / 10 GB disk per sandbox; unmetered sessions; SOC 2 Type 2 |
| **On-Demand** | \$0 base + usage | up to 100 | $0.05/hr per core, $0.015/hr per GB RAM; best-effort support |
| **Pro** | \$250 / month + usage | 1,000 | \$0.03/hr per core (40% off); snapshot + resume; 24×7 Slack/email; 24h P1 SLA |
| **Enterprise** | Custom quote | Unlimited | HIPAA + SOC 2 Type 2; SSO/SAML; RBAC; DPA; in-VPC / on-prem; 1h P1 SLA; resident SA |
See [tensorlake.ai/pricing](https://www.tensorlake.ai/pricing) for current rates.
## Is sandbox billing per-second or per-hour in Tensorlake?
**Per-second.** Rates are quoted per hour for clarity ($0.05/hr per core on On-Demand, $0.03/hr per core on Pro), but you only pay for the seconds the sandbox is running. This is why suspending a named sandbox between agent turns is meaningfully cheaper than leaving it running idle. See [suspend/resume](/sandboxes/lifecycle).
## Do I pay for a suspended Tensorlake Sandbox?
A suspended sandbox **consumes no compute**, so CPU and RAM time are not billed while suspended. Snapshots are billed **per GB-hour** while suspended.
## What are the default resources for a Tensorlake Sandbox?
By default, `Sandbox.create()` allocates:
| Resource | Default | How to override |
| --------- | -------------------- | ---------------------------------- |
| CPUs | `1.0` | `cpus=` (SDK) or `--cpus` (CLI) |
| Memory | `1024 MB` | `memory_mb=` / `--memory` |
| Root disk | `10240 MiB` (10 GiB) | `disk_mb=` / `--disk_mb` |
| Timeout | none | `timeout_secs=` / `--timeout-secs` |
## What are the memory and disk limits for a Tensorlake Sandbox?
* **Memory:** between `1024` and `8192` MB per CPU core.
* **Disk:** between `10240` and `102400` MiB (10–100 GiB).
Disk size is **growth-only**: when creating a sandbox from an `image`, `disk_mb` can grow the root disk at create time. When restoring from a filesystem snapshot, `disk_mb` can grow the root disk at restore time. You cannot shrink the disk below the source.
See the [SDK reference](/sandboxes/sdk-reference) and [Lifecycle: resources](/sandboxes/lifecycle) for the full parameter table.
## How many concurrent sandboxes can I run on each Tensorlake plan?
| Plan | Concurrent sandboxes |
| ---------- | -------------------- |
| Free | 2 |
| On-Demand | up to 100 |
| Pro | 1,000 |
| Enterprise | Unlimited |
## What is the default sandbox timeout in Tensorlake?
The default `timeout_secs` is `600` (10 minutes). The maximum allowed value depends on your plan:
| Plan | Max `timeout_secs` |
| ------------------------- | -------------------------------------------------------------- |
| Free (unverified) | 3600 (1 hour) |
| Free (verified) | 7200 (2 hours) |
| On-Demand (pay-as-you-go) | 86400 (24 hours) |
| Pro / Enterprise | See [tensorlake.ai/pricing](https://www.tensorlake.ai/pricing) |
Verify your free account by adding a credit card. It's used for identity verification only.
Setting `timeout_secs=0` requests the plan maximum.
When `timeout_secs` elapses:
* **Named sandboxes** auto-**suspend**: state is preserved and you can `resume`.
* **Ephemeral sandboxes** auto-**terminate**: the state is gone.
```python theme={null}
# Auto-suspend a named sandbox after 30 minutes of inactivity
sandbox = Sandbox.create(name="my-env", timeout_secs=1800)
```
## What are the build-time defaults for a Tensorlake sandbox image?
Build-time defaults are `cpus=2.0`, `memory=4096 MB`, and `disk=10 GiB`. Override them with `--cpus`, `--memory`, and `--disk_mb` in the CLI, or `cpus`, `memoryMb`, and `diskMb` in SDK options.
See [Build and Import Images](/sandboxes/images) for full image-build parameters.
# How do sandbox images work in Tensorlake?
Source: https://docs.tensorlake.ai/faqs/sandbox-images-faq
Frequently asked questions about Tensorlake sandbox images, covering base images, custom images, and building from Python, TypeScript, or a Dockerfile.
Sandbox images let you prebuild dependencies, files, and environment setup once, then launch fresh sandboxes from that prepared state. Below are the most common questions about images.
## What is a sandbox image and how is it different from a Docker image?
A sandbox image is a prebuilt, named environment used to launch isolated VMs or sandboxes. It packages dependencies, files, and environment setup so each new sandbox starts from the same prepared state.
The shape is similar to a Docker image (base layer, build operations like `run`, `copy`, `env`, `workdir`), but:
* The runtime target is a **MicroVM**, not a container.
* The build artifact is a **sandbox snapshot**, not an OCI image.
* The image is **scoped to a project**, not a registry.
Tensorlake supports defining sandbox images in Python, TypeScript, or a raw Dockerfile. See [Build and Import Images](/sandboxes/images).
## What sandbox images does Tensorlake provide?
Tensorlake ships several base images:
* **`ubuntu-minimal`** *(default)*: minimal Ubuntu without systemd. Boots in a few hundred milliseconds. Best for fast cold boot.
* **`ubuntu-systemd`**: Ubuntu with a full systemd init system. Use when you need to install packages like Docker or Kubernetes inside the sandbox.
* **`debian-minimal`**: minimal Debian 13.
* **`ubuntu-vnc`**: desktop-enabled Ubuntu (based on `ubuntu-systemd`) with XFCE, TigerVNC, and Firefox preinstalled. Use for browser automation and [computer-use](/sandboxes/computer-use) workloads.
## Which sandbox image should I use?
| Workload | Recommended image |
| ------------------------------------------- | ----------------- |
| Fastest cold boot | `ubuntu-minimal` |
| Need systemd, Docker, or Kubernetes | `ubuntu-systemd` |
| Debian-based environment | `debian-minimal` |
| Browser automation / desktop / computer use | `ubuntu-vnc` |
## How do I build a custom sandbox image?
You can build a custom image on top of a base image from Python, TypeScript, or a raw Dockerfile. When you build an image, Tensorlake:
1. Parses the image definition DSL and local source context.
2. Starts a temporary sandbox from the selected base image.
3. Translates supported build operations such as `run`, `copy`, `add`, `env`, and `workdir` into sandbox build steps and executes them there.
4. Creates a snapshot of the prepared sandbox.
5. Registers the image name for that snapshot in your current project.
Then create new sandboxes with:
```bash theme={null}
tl sbx new --image
```
See [Build and Import Images](/sandboxes/images) for full examples.
## Are Tensorlake sandbox images project-scoped?
Yes. Custom images are scoped to the project selected in the CLI. Before registering one:
```bash theme={null}
# CLI: for `tl login` and image commands
curl -fsSL https://tensorlake.ai/install | sh
tl login
# Python SDK: for programmatic image registration
pip install tensorlake
```
```bash theme={null}
# CLI: for `tl login` and image commands
curl -fsSL https://tensorlake.ai/install | sh
tl login
# TypeScript SDK: for programmatic image registration
npm install tensorlake
```
Programmatic image registration from the TypeScript SDK additionally needs `TENSORLAKE_API_KEY`, `TENSORLAKE_ORGANIZATION_ID`, and `TENSORLAKE_PROJECT_ID`.
## Why does `pip install` require `--break-system-packages` in a Tensorlake sandbox?
The Tensorlake Ubuntu and Debian images ship a PEP 668–managed system Python, so `pip install` requires `--break-system-packages` (or an explicit virtualenv). Without the flag, `pip` exits with `error: externally-managed-environment`. The flag is a requirement, not a stylistic choice.
## Can I use my existing Dockerfile with Tensorlake?
Yes. Tensorlake can build a sandbox image from a raw Dockerfile alongside the Python and TypeScript image DSLs. Tensorlake parses the Dockerfile, starts a temporary sandbox from the selected base image, runs supported build operations (`run`, `copy`, `add`, `env`, `workdir`), captures a snapshot, and registers the resulting image name in your project. After that, launch sandboxes with `tl sbx new --image `.
# How does the sandbox lifecycle work?
Source: https://docs.tensorlake.ai/faqs/sandbox-lifecycle-faq
Frequently asked questions about the Tensorlake Sandbox lifecycle, including ephemeral vs named sandboxes, suspend, resume, terminate, and timeouts.
## How do you keep an AI agent's environment alive between turns or sessions?
There are two general approaches:
* **Pause-in-place**: suspend the live VM so its filesystem, memory, and running processes are preserved, then resume under the same identifier. Useful for keeping an agent's working memory and open processes alive between turns.
* **Snapshot-and-restore**: capture a reusable artifact you can boot into a fresh VM later. Useful for retrying from a checkpoint or branching experiments.
In Tensorlake, these map to [suspend/resume](#how-do-i-suspend-a-tensorlake-sandbox) on named sandboxes and [snapshot/restore](/sandboxes/snapshots) via filesystem or memory snapshots.
## What's the difference between ephemeral and named sandboxes?
| | Ephemeral | Named |
| -------------------- | ------------------------------------ | ---------------------------------------- |
| **Created with** | `tl sbx create` | `tl sbx create ` |
| **Suspend / Resume** | Not supported | Supported |
| **Reference by** | ID only | ID **or** name |
| **Use when** | Short-lived tasks, one-off execution | Multi-step work, persistent environments |
Ephemeral sandboxes have no name and run until you terminate them or they time out. Named sandboxes support suspend/resume so you can pause between tasks and pick up where you left off.
## What states does a Tensorlake Sandbox move through?
Every sandbox moves through these states. Create starts the sandbox in `Pending`; from `Running`, you can suspend (named only), snapshot, or terminate. Ephemeral sandboxes skip `Suspending`/`Suspended`.
| State | What it means |
| ---------------- | ------------------------------------------------------------------------------- |
| **Pending** | Sandbox is being scheduled and booted. Transitions to `Running` automatically. |
| **Running** | Sandbox is live and accepting commands, file operations, and process execution. |
| **Snapshotting** | A reusable snapshot artifact is being captured. Returns to `Running` when done. |
| **Suspending** | Named sandbox is being paused, manually or via `timeout_secs`. |
| **Suspended** | Named sandbox is paused. Consumes no compute; state preserved. |
| **Terminated** | Sandbox has stopped. Final state; cannot be reversed. |
For the full state diagram, see [Lifecycle](/sandboxes/lifecycle).
## How do I suspend a Tensorlake Sandbox?
Call `suspend` on a named sandbox. The sandbox transitions to `Suspended`, consumes no compute, and preserves its state. Call `resume` on the same sandbox ID to bring it back to `Running`. Suspend is not supported for ephemeral sandboxes.
## When should I use suspend vs. snapshot?
Both preserve sandbox state, but they serve different purposes:
* **Suspend** pauses *this* sandbox so you can resume it later under the same ID.
* **Snapshot** captures a reusable artifact you can restore into a *new* sandbox.
| Scenario | Use Suspend | Use Snapshot |
| ------------------------------- | ----------- | ------------ |
| Pause and resume later | ✅ | ❌ |
| Save cost when idle | ✅ | ❌ |
| Keep agent memory alive | ✅ | ❌ |
| Retry from a checkpoint | ❌ | ✅ |
| Run experiments from same state | ❌ | ✅ |
| Clone environment | ❌ | ✅ |
See [Snapshots](/sandboxes/snapshots) for save-and-restore.
## What happens when a sandbox times out?
Ephemeral sandboxes are **terminated** when their timeout elapses. Named sandboxes transition to **Suspending** then **Suspended** when `timeout_secs` elapses, preserving state for later resume rather than terminating.
## Can I reverse a terminated sandbox?
No. `Terminated` is the final state. To restore prior state into a new sandbox, capture a snapshot before termination and create a new sandbox from that snapshot.
**Coming soon:** a `restart` operation that re-boots a terminated sandbox from
its last snapshot.
## How is suspend/resume different from stopping and restarting a Docker container?
Stopping a Docker container ends its processes and discards memory state; restarting goes through a cold boot and re-initializes the application. Tensorlake `suspend` pauses a named sandbox in place: filesystem, memory, and running processes are preserved. `resume` brings it back to `Running` under the same sandbox ID with the same in-memory state, so an agent harness can pick up exactly where it left off.
# What are Tensorlake Sandboxes?
Source: https://docs.tensorlake.ai/faqs/sandboxes-faq
Frequently asked questions about Tensorlake Sandboxes, isolated MicroVMs for AI agents, tool calls, builds, and IDEs.
## What is a MicroVM sandbox?
A MicroVM sandbox is a lightweight virtual machine (typically backed by [Firecracker](https://firecracker-microvm.github.io/) or CloudHypervisor) designed to start in milliseconds and run a single workload in hardware-isolated form. Unlike containers, each MicroVM has its own kernel, which makes them safer for running untrusted or AI-generated code. They're commonly used for AI agents, code execution, serverless functions, and CI/build workloads.
Tensorlake Sandboxes are MicroVMs built on Firecracker and CloudHypervisor.
## What are Tensorlake Sandboxes?
Tensorlake Sandboxes are isolated MicroVMs that boot in hundreds of milliseconds, with memory and filesystem preserved across suspend and resume. You can use them to run agent harnesses, execute tool calls, or as VMs for coding agents, builds, and IDEs.
## How are Tensorlake Sandboxes isolated?
Each sandbox is a MicroVM backed by [Firecracker](https://firecracker-microvm.github.io/) and CloudHypervisor. Sandboxes provide hardware-level isolation rather than container-level isolation, so untrusted or AI-generated code can run safely without sharing a kernel with other workloads.
## How fast does a Tensorlake Sandbox start?
Tensorlake creates a fresh sandbox in **single-digit milliseconds**; OS boot then completes in a few hundred milliseconds for the default `tensorlake/ubuntu-minimal` image. `tensorlake/ubuntu-systemd`, which includes a full init system and additional tooling (like Docker and Kubernetes support), takes around one second to boot.
At peak load, the scheduler creates hundreds of sandboxes per second. See [Architecture](/applications/architecture) for how this differs from Kubernetes pod creation.
## How do I create a Tensorlake Sandbox?
Create one on demand from the CLI or the SDK. Pass `image`, `cpus`, and memory to control the runtime.
```bash cli theme={null}
tl sbx create
```
```python sandbox.py theme={null}
from tensorlake.sandbox import Sandbox
resp = Sandbox.create(
image="tensorlake/ubuntu-minimal",
cpus=4,
memory_mb=8192,
)
```
```typescript sandbox.ts theme={null}
import { Sandbox } from "tensorlake";
const resp = await Sandbox.create({
image: "tensorlake/ubuntu-minimal",
cpus: 4,
memoryMb: 8192,
});
```
See the [Quickstart](/sandboxes/quickstart) for a full walkthrough.
## What can I run inside a Tensorlake Sandbox?
Anything the OS supports. Common workloads include:
* Agent harnesses and [tool calls](/sandboxes/tool-calls)
* LLM-generated or untrusted code
* Browser automation and [computer use](/sandboxes/computer-use)
* Builds, tests, and CI workloads
* Long-running processes and [PTY sessions](/sandboxes/pty-sessions)
* [Networking](/sandboxes/networking) and [tunnels](/sandboxes/tunnels)
## Is Tensorlake compliant with HIPAA and SOC 2?
Yes. Tensorlake is HIPAA and SOC 2 Type II compliant, supports EU data residency, and offers zero data retention.
## How are Tensorlake Sandboxes different from Docker containers?
Tensorlake Sandboxes are MicroVMs backed by Firecracker and CloudHypervisor, which means each sandbox has its own kernel and hardware-level isolation. Docker containers share the host kernel: faster to start, but weaker isolation for running untrusted or AI-generated code. Tensorlake also provides built-in [suspend/resume](/sandboxes/lifecycle) and [snapshots](/sandboxes/snapshots), which aren't part of the standard Docker runtime.
If you have an existing Dockerfile, Tensorlake can build a sandbox image from it. See [Build and Import Images](/sandboxes/images).
## Why would I use a sandbox if agents can already run on my laptop?
A laptop works for developing agents and running one or two interactively. It becomes the bottleneck when agents need to run unattended, in parallel, or against untrusted code:
* **Isolation**: an agent that installs packages, modifies files, or runs LLM-generated code does it inside its own MicroVM, not on your machine.
* **Persistence**: sandbox state (filesystem and memory) survives [suspend and resume](/sandboxes/lifecycle), so long-running work doesn't depend on your laptop staying open.
* **Reproducibility**: each sandbox starts from a defined [image](/sandboxes/images), not from whatever happens to be installed locally.
* **Snapshot and fork**: checkpoint an agent's environment and [fork parallel workers](/sandboxes/snapshots) from it, which a single machine can't do.
* **Scale**: going from one agent to dozens means creating more sandboxes, not buying more hardware.
## How is Tensorlake different from running Claude Code with git worktrees?
Git worktrees let you check out several branches of a repository side by side, so a coding agent like Claude Code can work on multiple tasks at once on your machine. The isolation is at the source-code level: every worktree shares your laptop's OS, dependencies, processes, and network.
Tensorlake Sandboxes isolate the entire execution environment. Each sandbox is a MicroVM with its own filesystem, processes, and dependencies, and can be [suspended, resumed](/sandboxes/lifecycle), [snapshotted, and forked](/sandboxes/snapshots) independently, and it keeps running when your laptop doesn't.
In short:
* Worktrees isolate source code.
* Sandboxes isolate entire execution environments.
If your goal is to have Claude work on several branches locally, worktrees may be all you need. If your agents need their own runtime, persistent state, browser sessions, or custom dependencies, or need to run independently of a developer laptop, sandboxes provide that infrastructure. The two also compose: you can run Claude Code inside a sandbox and still use worktrees there.
# What are Tensorlake Workflows?
Source: https://docs.tensorlake.ai/faqs/workflows-faq
Tensorlake Workflows automate and orchestrate complex tasks by composing functions into a Graph of parallel or sequential steps.
Tensorlake Workflows let you compose functions into a Graph and execute them in parallel or serially. Below are the most common questions about how Workflows work.
## What is durable execution?
Durable execution is a runtime model where the outputs of each step in a long-running program are checkpointed, so a crash, timeout, or retry can resume from the last completed step instead of restarting from scratch. It's the foundation behind systems like Temporal, Inngest, and Restate, and is commonly used for AI agents, long-running data pipelines, multi-step orchestration, and workflows that span minutes, hours, or days.
Tensorlake Workflows implement durable execution natively in Python: function outputs are checkpointed to object storage, and on failure the scheduler [replays](/applications/architecture#replay) the call graph and skips already-completed steps. See [Durable Execution](/applications/durability) for the full model.
## What are Tensorlake Workflows?
Tensorlake Workflows are a way to automate and orchestrate complex tasks. You define a series of functions that execute in parallel or sequentially, and Tensorlake handles distribution, persistence, and recovery.
## What is a Graph in a Tensorlake Workflow?
A Graph connects multiple functions together into a workflow. It contains:
* **Node**: a function that operates on data.
* **Start Node**: the first function executed when the graph is invoked.
* **Edges**: represent data flow between functions.
* **Conditional Edge**: evaluates input data from the previous function and decides which edges to take. Like an if-else statement.
Graphs are workflows whose functions can be executed in parallel, while
Pipelines are linear workflows that execute functions serially.
## How do I define a function in a Tensorlake Workflow?
Functions are regular Python functions decorated with `@tensorlake_function()`.
A function executes in a distributed manner and its output is stored, so if downstream functions fail they can resume from that output. The decorator accepts parameters to configure retry behavior, placement constraints, and more.
## How do I run a sequential pipeline in Tensorlake?
Chain nodes with `add_edge` so each function transforms the output of the previous one until reaching the end node.
```mermaid theme={null}
flowchart TD
node1 --> node2
node2 --> node3
```
```python theme={null}
@tensorlake_function()
def node1(input: int) -> int:
return input + 1
@tensorlake_function()
def node2(input2: int) -> int:
return input2 + 2
@tensorlake_function()
def node3(input3: int) -> int:
return input3 + 3
graph = Graph(name="pipeline", start_node=node1)
graph.add_edge(node1, node2)
graph.add_edge(node2, node3)
```
***Use case:*** Transforming a video into text by first extracting the audio, and then doing Automatic Speech Recognition (ASR) on the extracted audio.
## How do I run workflow steps in parallel in Tensorlake?
Add multiple edges from one start node to different downstream functions. Each branch produces an output for the same input in parallel.
```mermaid theme={null}
flowchart TD
start_node --> add_two
start_node --> is_odd
```
```python theme={null}
@tensorlake_function()
def start_node(input: int) -> int:
return input + 1
@tensorlake_function()
def add_two(input: int) -> int:
return input + 2
@tensorlake_function()
def is_even(input: int) -> int:
return input % 2 == 0
graph = Graph(name="pipeline", start_node=start_node)
graph.add_edge(start_node, add_two)
graph.add_edge(start_node, is_even)
```
***Use case:*** Extracting embeddings and structured data from the same unstructured data.
## How do I parallelize a function across many items (map) in Tensorlake?
When an upstream function returns a sequence and the downstream function accepts a single element of that sequence, Tensorlake automatically parallelizes the downstream function (one invocation per element) across machines and worker processes.
```mermaid theme={null}
flowchart TD
map(map)
node1(node)
node2(node)
node3(node)
map --> node1
map --> node2
map --> node3
```
```python theme={null}
@tensorlake_function()
def fetch_urls() -> list[str]:
return [
'https://example.com/page1',
'https://example.com/page2',
'https://example.com/page3',
]
# scrape_page is called in parallel for every element of fetch_url across
# many machines in a cluster or across many worker processes in a machine
@tensorlake_function()
def scrape_page(url: str) -> str:
content = requests.get(url).text
return content
```
***Use case:*** Generating an embedding for every chunk of a document.
## How do I aggregate results across many items (reduce) in Tensorlake?
Reduce functions aggregate outputs from one or more functions that return sequences. They have two key properties:
* **Lazy evaluation**: reduce functions are invoked incrementally as elements become available, so they stream over large datasets efficiently.
* **Stateful aggregation**: the aggregated value persists between invocations. Each call receives the current accumulated state along with the new element to process.
```mermaid theme={null}
flowchart TD
map(map)
reducer1(reducer)
reducer2(reducer)
reducer3(reducer
output=accumulator)
map --> reducer1
reducer1 --> reducer2
map --> reducer2
reducer2 --> reducer3
map --> reducer3
```
```python theme={null}
@tensorlake_function()
def fetch_numbers() -> list[int]:
return [1, 2, 3, 4, 5]
class Total(BaseModel):
value: int = 0
@tensorlake_function(accumulate=Total)
def accumulate_total(total: Total, number: int) -> Total:
total.value += number
return total
```
***Use case:*** Aggregating a summary from hundreds of web pages.
## How do I conditionally route data between functions in Tensorlake?
Use `@tensorlake_router` on a function that returns the list of downstream functions to invoke based on custom logic. The router decides at runtime which branch(es) to take.
```mermaid theme={null}
flowchart TD
router{router}
router -->|if <condition>| node1
router -->|else| node2
```
```python theme={null}
@tensorlake_function()
def handle_error(text: str):
# Logic to handle error messages
pass
@tensorlake_function()
def handle_normal(text: str):
# Logic to process normal text
pass
# The function routes data into the handle_error and handle_normal based on the
# logic of the function.
@tensorlake_router()
def analyze_text(text: str) -> List[Union[handle_error, handle_normal]]:
if 'error' in text.lower():
return [handle_error]
else:
return [handle_normal]
```
***Use case:*** Processing outputs differently based on classification results.
## How do Tensorlake Workflows compare to durable-execution systems like Temporal or Inngest?
Tensorlake Workflows are a durable-execution runtime in the same category as Temporal, Inngest, and Restate: function outputs are checkpointed, and on failure the scheduler replays the call graph from the last completed checkpoint instead of re-running everything from scratch.
The differences are surface and integration:
* **Authored as plain Python.** Functions are decorated with `@tensorlake_function`, with no separate worker SDK or activity/workflow split.
* **One runtime for code and isolation.** Workflows run on the same platform as [Tensorlake Sandboxes](/sandboxes/introduction), so the durable functions and the isolated environments they call into are managed by one scheduler.
* **Output storage built in.** Function outputs are persisted to object storage by default, so you can pass large files between steps without external workarounds.
See [Architecture](/applications/architecture) and [Durable Execution](/applications/durability) for how checkpointing and replay work.
# Architecture
Source: https://docs.tensorlake.ai/filesystems/architecture
How versioned file systems and repositories store metadata, deliver file content, merge changes, and expose operational state.
This page is a deep dive. You do not need it for day-to-day use. Start with [Versioned File Systems](/filesystems/introduction) or [Git Repositories](/git/introduction) first. File systems and repositories share the mount client and product concepts but store history in [two different engines](#two-storage-engines-one-mount-client).
## Overview
Tensorlake separates metadata from file content.
* The **control plane** stores history metadata (for repositories that's Git commits, refs, branches, and private workspace WAL checkpoints; for file systems it's native content-addressed snapshots and per-session checkpoint journals), plus sessions, workspaces, and operation history.
* The **data plane** stores file content in blob storage (Git objects for repositories, content-addressed blobs for file systems).
* Mounts resolve metadata through the control plane and fetch file content lazily from the data plane.
This split keeps session creation, snapshot listing, and history fast even when a file system or repository is large.
```mermaid theme={null}
graph TD
Client["CLI / API"] --> Meta
subgraph CP["Control Plane"]
direction LR
Meta["Metadata Store
native snapshots · journals · Git refs · sessions"]
Reconcile["Reconcile / Merge
change-set rebase (fs) · 3-way merge (git)"]
Bg["Background Maintenance
optimization, dedup, retention, GC"]
end
subgraph DP["Data Plane"]
Blob["Blob Store
file content"]
end
Meta --> Reconcile
Bg -.-> Blob
Meta -- "resolves ref to tree" --> Mount["File System Mount"]
Blob -- "lazy, parallel fetch" --> Mount
Mount --> S1["Sandbox"]
Mount --> S2["Sandbox"]
Mount --> S3["Sandbox"]
```
## Two storage engines, one mount client
File systems and repositories share the mount/overlay client, crash-safe local journal, and lazy content delivery, but they store and publish history through **different engines**:
* **Repositories** are Git. History is Git commits, refs, and branches; content is packed into Git objects; they speak Git smart-HTTP so `git clone`/`push` work.
* **File systems** run the native snapshot engine. History is immutable content-addressed checkpoints and snapshots (not Git commits); file content is deduplicated and packed into content-addressed blobs; and a per-session **checkpoint journal** is the crash-safe replication path into the shared timeline. File systems have no Git access: no commits, refs, branches, packs, or Git wire protocol.
A session or workspace is metadata, not a full copy. It records what it belongs to, the point it started from, and its durable WAL or snapshot chain. A Git read-only mount creates no workspace; a writable Git workspace is created lazily when its first remote WAL checkpoint arrives. Saving uploads only changed content. Optimization, deduplication, retention, and garbage collection run in the background so agents never wait on storage maintenance.
## One local journal, two product surfaces
Writable file-system and repository mounts share one crash-safe client pipeline:
```mermaid theme={null}
graph LR
W["File writes"] --> J["Local journal
crash recovery"]
J --> U["Upload changed content"]
U --> C["Server WAL checkpoint
durable replication boundary"]
C --> F{"Surface"}
F -->|"tl fs autosave"| H["Shared file-system timeline"]
F -->|"tl git snapshot"| G["Workspace Git commit"]
G -->|"tl git promote"| B["Branch"]
C --> P["Prune confirmed local generations"]
```
The difference is publication policy. On `tl fs`, the server verifies, orders, and applies each autosave checkpoint to the shared drive before acknowledging it; `tl fs snapshot` adds a permanent retention point. On `tl git`, autosave checkpoints remain private workspace WAL: they create neither a Git commit nor a branch update. `tl git snapshot` materializes the current WAL as a workspace commit, and `tl git promote` deliberately lands it on a branch.
## Content Delivery
Mounting a repository or file system does not copy it into the sandbox. The mount resolves the tree, then fetches file content lazily as processes read paths.
That means:
* Large repositories can mount quickly.
* A sandbox only downloads paths it actually reads.
* Many sandboxes can mount the same repository without a single shared serving path becoming the bottleneck.
Following read-only mounts add branch tracking. When the followed branch moves, Tensorlake compares the old and new commits and invalidates only changed paths. Unchanged files keep their warm page cache.
## Reconciling concurrent writes
The two engines reconcile concurrent writes differently, because they model history differently.
**File systems have one linear timeline.** When checkpoints race, the server orders them and rebases each update's changed paths onto the current head before acknowledging it. Cost is proportional to changed paths, not file-system size. Disjoint paths merge automatically; two writers on the same path are last-writer-wins: the later ordered update's version of that file sticks, silently. There is no three-way merge and no conflict marker on a file system, because a single timeline has nothing to diverge from. See [Concurrent Writes](/filesystems/concurrent-writes).
**Repositories can have divergent branches**, so they use a server-side three-way merge engine, verified against Git's merge behavior, for promote, `git merge`, and rebase. It compares the common ancestor, the target branch, and the workspace, resolving non-overlapping edits automatically and surfacing genuine conflicts. `tl git sync` has a narrower role: it refreshes or switches a read-only or snapshot-free view, carrying its WAL tail forward, and refuses to rewrite an established workspace snapshot chain. Two conflict behaviors apply on the repository surface: **Fail** (the default for promotion: nothing lands, a structured conflict report is returned) and **Materialize** (writes Git conflict markers plus a queryable conflict record). Both preserve the losing content in Git history.
## Promotion
Promotion is the repository surface's path through the merge engine. It first materializes any dirty workspace WAL, then lands the workspace as one squashed commit. See [Repository Mounts](/git/workspace-mounts) for the workflow. File systems have no promote step: replicated autosaves advance the shared timeline automatically, while snapshots make selected points permanent.
## Observability
Durable file-system and repository state is queryable through the API and available to the control plane:
* Which workspaces are live.
* Which are detached and resumable.
* Which durable WAL checkpoints and snapshots exist.
* Which principal created each snapshot.
* Which paths changed.
For file systems, the durable record is the single timeline's checkpoint and snapshot history. For repositories, branch activity is also recorded: pushes, promotions, rebases, merges, and materialized conflict records. Mount heartbeats expose liveness, but the unsealed tail of edits remains in the sandbox's local journal until autosave. The control plane does not observe every in-progress file edit.
Detached Git workspaces are collected by lifecycle tier. An actively mounted workspace is retained; a WAL-only detached workspace defaults to 48 hours, and a detached workspace with snapshots defaults to 14 days. The deployment configuration controls these periods.
File systems, mounts, local journals, autosave checkpoints, snapshots, and publishing.
Mount, autosave, create permanent snapshots, inspect status, restore, and clean up.
Clone, branch, commit, merge, push, and fetch with plain Git.
Run agents in isolated sandboxes and mount file systems into them.
# Concurrent Writes
Source: https://docs.tensorlake.ai/filesystems/concurrent-writes
How several mounts writing to one file system reconcile. Disjoint paths merge automatically, same-path writes are last-writer-wins.
Several mounts can write to the same file system at once. Autosave sends each mount's settled work through its session WAL. Before acknowledging a checkpoint, the server orders that update and applies its changed paths to the file system's single shared timeline. A file system is a shared drive, not a branching repository, so reconciliation is simple and automatic, with one rule for the rare overlap.
## Disjoint changes merge automatically
Replicated updates that touch different paths, or different files in the same directory, reconcile cleanly and never notice each other. The server rebases each update's changed paths onto the current state of the timeline, so two agents writing to `sessions/a/` and `sessions/b/` both land, in full, with no coordination and no conflict.
This is the common case, and it's the case worth designing for: **partition writers by directory or file** and concurrent access is effortless. It's exactly how one file system per user, mounted across many sandboxes, is meant to work: each session owns its own subtree.
## Same-path writes are last-writer-wins
If two mounts change the *same* file concurrently, the later server-ordered checkpoint wins. The file on the timeline becomes the later writer's version; the earlier writer's change to that file is overwritten. There are no conflict markers, no merge to resolve, and no manual step. A file system behaves like a shared disk, where the last published update to a path is the one that sticks.
This is deliberate. A file system is a single linear timeline, not a set of branches that diverge and merge, so there is nothing to three-way merge, and binary files can't carry text conflict markers anyway. If you need private work with an explicit publication step (review before landing, or long-lived divergence with conflict resolution), use a [Git repository workspace](/git/workspace-mounts) instead, where branches genuinely diverge and merges surface conflicts.
## Making concurrency safe
Because same-path overwrites are silent, the way to keep concurrent agents from stepping on each other is to keep their writes disjoint:
* **Give each agent or task its own subtree** (`sessions//`, `users//`). This is the natural shape for agent workloads and eliminates same-path contention entirely.
* **Treat shared files as append-only or single-owner.** If one file must be written by many agents, funnel writes through one of them, or have each agent write its own file and merge at read time.
* **Wait for settled-write replication, not an explicit snapshot.** A small quiet edit normally reaches the shared head in about a second plus upload time; continuous writers flush at least every 5 seconds. Following mounts refresh after that checkpoint commits. They do not see bytes that are still in another mount's open local batch.
## Next Steps
Status, history, resume, and restore.
How checkpoints are ordered and reconciled internally.
# Core Concepts
Source: https://docs.tensorlake.ai/filesystems/core-concepts
Short definitions for the versioned file system model: mounts, local journals, autosave checkpoints, permanent snapshots, retention, and publishing.
These are the terms used across the versioned file system docs.
## File System
A **file system** is the durable, versioned store: one shared timeline. Autosave checkpoints are kept as a recent recovery window; permanent snapshots are kept until you delete them (see [Retention](#retention)).
```bash theme={null}
tl fs create agent-scratch
```
```python theme={null}
from tensorlake.filesystem import FilesystemClient
client = FilesystemClient()
fs = client.create("agent-scratch")
```
```typescript theme={null}
import { FilesystemClient } from "tensorlake";
const client = new FilesystemClient();
const fs = await client.create("agent-scratch");
```
A new file system starts empty and is ready to mount immediately.
## Mount
A **mount** gives a sandbox a directory backed by a file system.
```bash theme={null}
tl fs mount agent-scratch /work
```
```python theme={null}
mount = client.mount("agent-scratch", "/work")
# or, from a filesystem object: mount = fs.mount("/work")
```
```typescript theme={null}
const mount = await client.mount("agent-scratch", "/work");
// or, from a filesystem object: const mount = await fs.mount("/work");
```
The mount path is ephemeral. Processes read and write `/work` like any other directory while the sandbox is running. Reads stream in lazily, so mounting is fast regardless of how much the file system holds.
## Session
A **session** is the state behind one writable mount. It records which checkpoint the mount started from and owns a crash-safe local journal for changes that have not reached the server yet.
Sessions survive unmounts and sandbox crashes. Remounting a file system this machine has a detached session for resumes that session:
```bash theme={null}
tl fs mount agent-scratch /work2
```
```python theme={null}
mount = client.mount("agent-scratch", "/work2")
```
```typescript theme={null}
const mount = await client.mount("agent-scratch", "/work2");
```
`tl fs ls agent-scratch` lists a file system's sessions.
## Autosave Checkpoint
An **autosave checkpoint** is a durable, shared recovery point for the mounted file state. Writable mounts replicate settled changes through a per-session server WAL automatically, including at a bounded interval when an agent writes continuously.
The server verifies and applies each acknowledged checkpoint to the shared file-system timeline before returning success. Following mounts can observe it on their next head poll, typically within seconds of the original writes settling. Automatic points appear under `Recent autosave WAL` in `tl fs history`. They are ephemeral: Tensorlake retains the newest 256 and all points from at least the most recent 24 hours, then truncates older entries automatically.
## Permanent Snapshot
A **snapshot** keeps the invocation's exact state as a permanent, billed retention point:
```bash theme={null}
tl fs snapshot /work -m "baseline benchmarks"
```
```python theme={null}
mount.snapshot("baseline benchmarks")
```
```typescript theme={null}
await mount.snapshot("baseline benchmarks");
```
If autosave already published that state, the command promotes the current automatic save to
permanent retention in place. It uploads no file bytes and creates no duplicate content version.
If the current save is already permanent, the command is a quiet no-op. Permanent snapshots appear
separately in `tl fs history`, can be restored or mounted read-only, and remain until you run
`tl fs delete-snapshot`.
## Retention
Autosave checkpoints are retained as a bounded recent window so long-running mounts do not accumulate history without limit. Permanent snapshots are billed storage, structurally exempt from automatic expiry, and remain until you delete them.
## Publishing
Writable mounts **publish**: every acknowledged autosave checkpoint advances the file system's shared timeline, and other mounts converge to it automatically. The server WAL is the crash-safe commit path, not a private 15-minute staging window.
`tl fs status` shows the mode:
```bash theme={null}
mode: writable — shared current state
```
There is no separate promote step on the file system surface. If you want private work that publishes only when you say so, use a [Git repository workspace](/git/workspace-mounts) instead.
## Concurrent Writes
Several mounts can write to one file system at once. Crystallized updates that touch different paths merge automatically. Two mounts publishing the *same* file in overlapping windows are last-writer-wins: the later ordered update's version sticks, silently. A file system is a shared drive, not a set of branches, so there are no conflict markers.
See [Concurrent Writes](/filesystems/concurrent-writes).
## Retained Files
After a checkpoint publishes, the mount keeps local copies of its files as a byte cache: reads stay local and future autosaves stay incremental. `tl fs status` counts them under `retained:`. They are already durable; the cache only costs local disk.
## Credentials
`tl login` stores a Tensorlake CLI credential. `tl fs` commands mint and refresh the short-lived credentials they need automatically, including inside sandboxes, where `tl fs token ` mints the scoped credential a guest needs to attach one file system.
You only handle credentials yourself on the Git surface. See [Authentication](/git/authentication).
# Distribute Files to Agents
Source: https://docs.tensorlake.ai/filesystems/distribute-files
Distribute versioned manuals, skills, configs, and binary tools to agents with read-only mounts.
Use a versioned file system when many agents need the same files at a stable path.
Put operating manuals, skills, configs, test fixtures, or binary tools in a file system. Update it from a laptop, CI job, or backend service. Agents mount it read-only. When an autosave checkpoint or permanent snapshot commits, following mounts refresh changed paths automatically.
## Pattern
1. Store shared files in a versioned file system.
2. Publish updates from outside the sandbox with `tl fs push`.
3. Mount the file system into agents as a read-only directory.
4. Follow the file system for automatic distribution, or pin a permanent snapshot for fixed releases.
## Create an Asset File System
```bash theme={null}
$ tl fs create agent-assets
Created filesystem agent-assets (empty).
```
```python theme={null}
from tensorlake.filesystem import FilesystemClient
client = FilesystemClient()
fs = client.create("agent-assets")
```
```typescript theme={null}
import { FilesystemClient } from "tensorlake";
const client = new FilesystemClient();
const fs = await client.create("agent-assets");
```
Keep the layout simple and stable:
```text theme={null}
agent-assets/
manuals/
skills/
bin/
configs/
```
Agents can refer to paths like `/opt/agent-assets/manuals/operator.md`, `/opt/agent-assets/skills/research/SKILL.md`, or `/opt/agent-assets/bin/validator`.
## Publish From Outside a Sandbox
The producer does not need a sandbox, and it does not need a mount. Push a directory and create a permanent snapshot for the release:
```bash theme={null}
$ tl fs push ./agent-assets agent-assets -m "publish agent assets"
Pushed ./agent-assets to agent-assets (48 file(s)).
```
Pushing the same directory again uploads only what changed, the right shape for a CI job or release service that republishes on every change. Pass `-m` on the changed push that should create a permanent snapshot; without it, the push creates an ephemeral autosave checkpoint. A push with no changes is a quiet no-op.
Pushes honor `.gitignore`, preserve symlinks, and preserve executable bits on regular files. A file system has no special `.git` handling: only `.gitignore` governs what is excluded.
## Mount Into Agents
Mount the file system read-only at a predictable path:
```bash theme={null}
$ tl fs mount agent-assets /opt/agent-assets --ro
Mounted agent-assets at /opt/agent-assets (read-only, follows the filesystem)
```
```python theme={null}
mount = client.mount("agent-assets", "/opt/agent-assets", readonly=True)
```
```typescript theme={null}
const mount = await client.mount("agent-assets", "/opt/agent-assets", true);
```
New sandboxes read the current state. Running following mounts refresh as the shared timeline advances, so updated manuals, skills, configs, and tools appear without rebuilding sandbox images.
Agents should write outputs to a separate writable mount. Keep shared assets read-only so every agent sees the same source files.
## Version Releases
Use a pinned permanent snapshot when a run must be reproducible:
```bash theme={null}
tl fs mount agent-assets:9f2a1c8e4d6b1a0f3c7e9d2b8a4f6c1e0d3b7a99fedcba987654321001234567 /opt/agent-assets --ro
```
Pinned mounts stay fixed for the lifetime of the sandbox. Find permanent snapshot IDs under `Snapshots` in `tl fs history agent-assets`. Recent autosave IDs can expire and should not be used as long-lived release anchors.
If you need named release channels (`stable`, `canary`) that you advance deliberately, back the assets with a [Git repository](/git/introduction) and use branches as channels. That's the surface built for explicit publication.
## Binary Tools
Put tools under a stable directory such as `bin/` with the executable bit set, and push:
```bash theme={null}
chmod +x agent-assets/bin/validator
tl fs push ./agent-assets agent-assets -m "add validator tool"
```
Agents can call the tool directly:
```bash theme={null}
/opt/agent-assets/bin/validator --input /work/result.json
```
## Choose a Mount
| Need | Use |
| ------------------------------------------------------------ | ---------------------------------------------------------------- |
| Roll out the latest manuals, skills, or tools to many agents | Following read-only mount |
| Keep an eval, benchmark, or release run fixed | Pinned read-only mount |
| Publish assets from CI or an external service | `tl fs push` |
| Let an agent create or modify files | [Writable mount](/filesystems/filesystem-mounts#writable-mounts) |
## Next Steps
Choose between following the file system and pinning a permanent snapshot.
Give agents a separate place to write outputs.
# File System Mounts
Source: https://docs.tensorlake.ai/filesystems/filesystem-mounts
Choose the right versioned file system mount for agent work, fixed inputs, and shared assets.
A file system mount exposes a versioned file system as an ordinary directory in a sandbox. The mount path is ephemeral. For writable mounts, Tensorlake replicates settled changes through a durable server WAL into the shared file system.
Start with a writable mount unless the sandbox should not write.
Install the file-system extension once on macOS. Linux needs no setup.
```bash theme={null}
tl fs setup --check
tl fs setup
```
## Choose a Mode
Every mode uses the same command shape:
```bash theme={null}
tl fs mount [:]
```
| Mode | Command | Use it for |
| -------------------- | ------------------------------------------------------- | -------------------------------------------------------------------------- |
| Writable mount | `tl fs mount agent-scratch /work` | Agents creating or modifying files; autosaves become durable automatically |
| Read-only, following | `tl fs mount agent-scratch /skills --ro` | Shared skills, prompts, docs, configs, and assets |
| Read-only, pinned | `tl fs mount agent-scratch: /release --ro` | Reproducible builds, evals, and fixed releases |
## Writable Mounts
Writable mounts are for agent sessions that create or modify files. The agent writes files in the mount; writes enter a crash-safe local journal and settled changes replicate as durable server WAL checkpoints. The server applies each acknowledged checkpoint to the shared timeline.
Mount the filesystem:
```bash theme={null}
$ tl fs mount agent-scratch /work
Mounted filesystem agent-scratch at /work (session 54398548341c, saves publish automatically)
At save e3f421a78c8cbba09c79294131835fe0da8b4433a1b2c3d4e5f60718293a4b5c. Changes save automatically; tl fs snapshot /work makes a permanent snapshot.
Autosave: settled changes replicate in about 1s (5s max while continuously writing).
```
```python theme={null}
from tensorlake.filesystem import FilesystemClient
client = FilesystemClient()
mount = client.mount("agent-scratch", "/work")
```
```typescript theme={null}
import { FilesystemClient } from "tensorlake";
const client = new FilesystemClient();
const mount = await client.mount("agent-scratch", "/work");
```
Autosave is always on for writable mounts. Changes normally reach the shared timeline about 750ms after they settle plus upload/server time; a continuously-writing agent checkpoints at least every 5 seconds. The server does not acknowledge the checkpoint until verification and shared-head publication complete. `tl fs snapshot /work -m "..."` keeps the command's exact state permanently and returns only after the durable snapshot receipt exists. If autosave already published that state, the server promotes the existing automatic save in place—no file bytes are uploaded and no duplicate content version is created. You can safely unmount as soon as the command succeeds. It is a quiet no-op only when the current save is already permanent.
Remounting a file system this machine has a detached session for resumes that session, unsaved local changes included. A session already mounted elsewhere mounts read-only.
## Read-only Mounts
Read-only mounts are for inputs and shared assets. Follow the file system's current state to roll out updates to many sandboxes, or pin a permanent snapshot for reproducibility.
```bash theme={null}
tl fs mount agent-scratch /skills --ro
```
```python theme={null}
mount = client.mount("agent-scratch", "/skills", readonly=True)
```
```typescript theme={null}
const mount = await client.mount("agent-scratch", "/skills", true);
```
See [Read-only Mounts](/filesystems/read-only-mounts).
## Session Operations
Use session operations to inspect local changes, resume after a sandbox restart, browse history, create or restore permanent snapshots, repair a session, and clean up.
```bash theme={null}
tl fs status /work # unsaved changes, retained + ignored counts
tl fs snapshot /work -m "milestone" # permanent snapshot (--clear also trims its retained byte cache)
tl fs history agent-scratch # browse the timeline
tl fs delete-snapshot agent-scratch # remove a permanent snapshot
tl fs restore /work # restore a snapshot or retained autosave
tl fs doctor /work # inspect / repair local session state
```
See [Manage Sessions](/filesystems/manage-sessions).
## Mounting in Sandboxes
A sandbox guest needs one scoped credential to attach a file system:
```bash theme={null}
$ tl fs token agent-scratch
```
The command prints the credential and the environment recipe for the guest. Inside the sandbox, the same `tl fs mount` commands work unchanged.
Use pinned and following mounts for fixed inputs and shared assets.
Roll out manuals, skills, configs, and tools to agent fleets.
Inspect status, resume, restore, and clean up.
File systems, mounts, sessions, autosave checkpoints, snapshots, and publishing.
# Cloud Volumes
Source: https://docs.tensorlake.ai/filesystems/introduction
Cloud Volumes are directories that can be mounted on one or more sandboxes. Volumes can be versioned and can be time-travelled.
Agents produce state and artifacts worth keeping: code, documents, artifacts, working notes.
Volumes can be mounted as a directory on the file system into a sandbox, all writes are automatically saved in durable storage asynchronously. They offer
SSD grade write performance, and reads are cached and can be fetched either lazily or pre-fetched from remote storage.
These directories can be snapshotted to create point-in-time checkpoints and can be time travelled to restore state at a given time.
Volumes can be used on any sandbox provider or AWS/GCP/Azure or even Kubernetes containers. They are portable directories that can be mounted
on any Linux or OSX machines.
These are some use cases for versioned file systems:
* Version an agent's working state without teaching it version control
* Persist a long-running session so a crashed sandbox loses nothing
* Share one file system across several coding agents at once. Disjoint work merges automatically
* Distribute documents, skills, and tools to fleets of agents with read-only mounts
## Quickstart
Install the Tensorlake CLI and sign in:
```bash theme={null}
curl -fsSL https://tensorlake.ai/install | sh
tl login
```
Then create a file system, mount it, and watch changes save themselves.
```bash theme={null}
$ tl fs create agent-scratch
Created filesystem agent-scratch (empty).
mount it: tl fs mount agent-scratch
or push a folder: tl fs push agent-scratch
```
```python theme={null}
from tensorlake.filesystem import FilesystemClient
client = FilesystemClient()
fs = client.create("agent-scratch")
```
```typescript theme={null}
import { FilesystemClient } from "tensorlake";
const client = new FilesystemClient();
const fs = await client.create("agent-scratch");
```
A new empty file system is created on our servers. It's ready to be mounted immediately on a sandbox or machine.
```bash theme={null}
$ tl fs mount agent-scratch /work
Mounted filesystem agent-scratch at /work (session 54398548341c, saves publish automatically)
At save e3f421a78c8cbba09c79294131835fe0da8b4433a1b2c3d4e5f60718293a4b5c. Changes save automatically; tl fs snapshot /work makes a permanent snapshot.
Autosave: settled changes replicate in about 1s (5s max while continuously writing).
```
```python theme={null}
from pathlib import Path
# Tilde paths are not expanded by the SDK; pass a resolved path.
mount = fs.mount(str(Path.home() / "work"))
```
```typescript theme={null}
import { homedir } from "node:os";
import { join } from "node:path";
// Tilde paths are not expanded by the SDK; pass a resolved path.
const mount = await fs.mount(join(homedir(), "work"));
```
The file system is mounted at /work on the local machine. It's a normal POSIX compliant directory.
You can write to the file system through the Python or TypeScript SDKs without mounting them. The SDKs use the
HTTP API of the remote file system directly.
Write into the mounted directory with POSIX read/write APIs. if you use the Python/TypeScript SDKs, writes go straight to the file system without a mount.
```bash theme={null}
$ echo "hypothesis: the parser is quadratic" > /work/notes.md
$ mkdir /work/results && cp bench.json /work/results/
```
```python theme={null}
fs.write_file("notes.md", "hypothesis: the parser is quadratic\n")
```
```typescript theme={null}
import { readFile } from "node:fs/promises";
await fs.writeFile("notes.md", "hypothesis: the parser is quadratic\n");
await fs.writeFile("results/bench.json", await readFile("bench.json"));
```
The writes are asynchronously replicated to Tensorlake's remote file system. The server advances the file-system timeline.
The file systems can be mounted on multiple machines at once. The other mounts will see the changes within seconds.
You can create a permanent snapshot to create a point-in-time checkpoint of the file system. You can time travel across the snapshots, restore a specific
checkpoint or fork the file system from that point.
```bash theme={null}
$ tl fs snapshot /work -m "baseline benchmarks"
```
```python theme={null}
mount.snapshot("baseline benchmarks")
```
```typescript theme={null}
await mount.snapshot("baseline benchmarks");
```
If autosave already published those changes, snapshotting a clean mount promotes that current
automatic save to permanent retention in place. It uploads no file bytes and creates no second
content version. If the current save is already permanent, the command is a quiet no-op.
```bash theme={null}
$ tl fs status /work
filesystem: agent-scratch
session: 54398548341c (created 5m ago)
mode: writable — every save becomes the filesystem's current state
autosave: settled changes replicate in about 1s (5s max while continuously writing)
last autosave: 2m ago
permanent snapshots: 1
daemon: serving save 8b21f6a9
local: clean
```
```python theme={null}
status = mount.status()
print(status.filesystem, status.mounted)
```
```typescript theme={null}
const status = await mount.status();
console.log(status.filesystem, status.mounted);
```
History is browsed with the CLI:
```bash theme={null}
$ tl fs history agent-scratch
Autosave WAL (fixed native_fs_v1): each checkpoint is synchronously replicated to the shared drive; keep the newest 256 generations and all generations from the last 24h. Snapshots are permanent until you delete them.
Snapshots (permanent — kept until deleted)
4d9a2f7e baseline benchmarks 4m ago
Recent autosave WAL (ephemeral — truncated automatically)
8b21f6a9 2m ago
e3f421a7 5m ago
```
`local: clean` means the local journal has no changes waiting for autosave. History separates permanent snapshots from recent autosave recovery points.
The mount path is disposable; the session behind it is durable. If the sandbox crashes or you unmount and walk away, remount the file
system and pick up from its last durable state:
```bash theme={null}
$ tl fs mount agent-scratch /work2
Resumed session 54398548341c
```
```python theme={null}
mount = client.mount("agent-scratch", "/work2")
```
```typescript theme={null}
const mount = await client.mount("agent-scratch", "/work2");
```
From **another** machine you recover everything through the last autosave checkpoint, and because autosave is frequent, that is
typically seconds of work at most; anything written since the last checkpoint stays on the machine that wrote it.
Remounting on the **same** machine can additionally recover a detached session's unsaved local changes from its overlay.
## Push a Folder Without Mounting
When you just want a directory's contents in a file system, you can push the directory into the file system:
```bash theme={null}
$ tl fs push ./results agent-scratch -m "run 42 results"
Pushed ./results to agent-scratch (14 file(s)).
```
Pushing the same directory again uploads only what changed. Passing `-m` creates a permanent snapshot.
## Mental Model
1. A **file system** is the durable, versioned store.
2. A **mount** gives a sandbox an ordinary directory backed by it.
3. A **session** is the state behind one mount, resumable after crashes and unmounts.
4. An **autosave checkpoint** replicates settled work through a session WAL and advances the shared timeline before it is acknowledged.
5. A **snapshot** keeps the invocation's exact state as a permanent, billed retention point until you delete it. If autosave already published that state, the existing save is promoted in place without uploading bytes or creating duplicate content.
6. Other mounts converge as acknowledged autosaves or permanent snapshots advance the shared timeline.
See [Core Concepts](/filesystems/core-concepts) for short definitions of each term.
## Summary of Features
* **Autosave**: settled changes replicate through durable server WAL and into the shared timeline automatically; a continuously-writing agent still checkpoints at a bounded interval.
* **Permanent snapshots and history**: browse both kinds of history with `tl fs history`; create a permanent, billed point with `tl fs snapshot` and remove it with `tl fs delete-snapshot`.
* **Time travel**: restore the mount to any retained autosave or permanent snapshot with `tl fs restore`.
* **Durable sessions**: crash a sandbox and remount elsewhere to recover through the last server autosave. On the same machine, the local journal can also recover changes that had not reached the server yet.
* **Shared file systems**: many writers on one file system; server-ordered disjoint changes merge automatically, same-path writes are last-writer-wins.
* **Read-only mounts**: serve a fixed snapshot or follow the file system's current state across many running sandboxes.
## Use Cases
Roll out manuals, skills, configs, and tools to many agents with versioned read-only mounts.
Prefer a Git repository when generated projects need branches and explicit publication.
## Where To Go Next
Learn the vocabulary: file systems, mounts, sessions, autosave checkpoints, snapshots, and publishing.
Choose between writable and read-only mounts.
Inspect status, resume, restore an earlier checkpoint or snapshot, and clean up.
How several mounts writing at once reconcile: disjoint paths merge, same-path is last-writer-wins.
# Manage Sessions
Source: https://docs.tensorlake.ai/filesystems/manage-sessions
Inspect, resume, restore, and clean up versioned file system sessions.
A mount path is ephemeral. Server autosaves and permanent snapshots are the durable state behind it. A writable session also owns a local crash-safe journal, so the same machine can recover changes that had not reached the server when the mount stopped.
Use these commands, or their SDK equivalents, to inspect a session, resume it, browse its history, restore an earlier checkpoint or snapshot, or delete a file system.
## Check Status
```bash theme={null}
$ tl fs status /work
filesystem: agent-scratch
session: 54398548341c (created 12m ago)
mode: writable — every save becomes the filesystem's current state
autosave: settled changes replicate in about 1s (5s max while continuously writing)
last autosave: 2m ago
permanent snapshots: 1
daemon: serving save 8b21f6a9
log: ~/.local/share/tensorlake/mounts/54398548341c.../daemon.log
local: clean
```
```python theme={null}
from tensorlake.filesystem import FilesystemClient
client = FilesystemClient()
status = client.mount_status("/work")
print(status.filesystem, status.mounted)
```
```typescript theme={null}
import { FilesystemClient } from "tensorlake";
const client = new FilesystemClient();
const status = await client.mountStatus("/work");
console.log(status.filesystem, status.mounted);
```
With unsaved changes, status lists dirty paths:
```bash theme={null}
local: 2 change(s):
M src/parser.py
D src/old_parser.py
```
Two more lines appear as a session ages:
* `retained:` counts files already published and kept locally as the byte cache. They are durable; the local copies only make reads and future autosaves fast. `tl fs snapshot --clear` trims cache entries covered by that snapshot while preserving later writes and ignored files.
* `ignored:` counts local-only files that never enter an autosave or snapshot (build output and the like, per the file system's ignore rules).
Add `--json` for machine-readable output; the SDK's `MountStatus.raw` carries the same structured payload.
## Browse History
```bash theme={null}
$ tl fs history agent-scratch
Autosave WAL (fixed native_fs_v1): each checkpoint is synchronously replicated to the shared drive; keep the newest 256 generations and all generations from the last 24h. Snapshots are permanent until you delete them.
Snapshots (permanent — kept until deleted)
4d9a2f7e baseline benchmarks 1h ago
Recent autosave WAL (ephemeral — truncated automatically)
8b21f6a9 2m ago
e3f421a7 3h ago
```
Autosaves provide a recent recovery window. Create a permanent, billed snapshot as part of the changed generation you need to keep:
```bash theme={null}
$ tl fs snapshot /work -m "baseline benchmarks"
Snapshot 4d9a2f7e "baseline benchmarks" — kept until deleted.
```
```python theme={null}
mount.snapshot("baseline benchmarks")
```
```typescript theme={null}
await mount.snapshot("baseline benchmarks");
```
Permanent snapshots are exempt from automatic retention. Snapshotting a clean mount promotes its
current automatic save to permanent retention in place, without uploading file bytes or creating a
duplicate content version. It is a quiet no-op only when the current save is already permanent.
Drop a permanent snapshot you no longer need with `tl fs delete-snapshot agent-scratch `.
## List Sessions
```bash theme={null}
$ tl fs ls agent-scratch
Session Filesystem Base Saves Mode Mounted Age
54398548341c agent-scratch e3f421a7 yes publishing /work 12m
```
`tl fs ls` with no argument lists your file systems.
## Resume a Session
Unmounting keeps the session by default:
```bash theme={null}
$ tl fs unmount /work
Unmounted /work. Session 54398548341c kept — `tl fs mount agent-scratch ` resumes it.
```
```python theme={null}
mount.unmount()
# or, without a Mount object: client.unmount("/work")
```
```typescript theme={null}
await mount.unmount();
// or, without a mount object: await client.unmount("/work");
```
Remounting the file system on a machine that has a detached session resumes that session, unsaved local changes included. The new mount path can be different:
```bash theme={null}
$ tl fs mount agent-scratch /work2
Resumed session 54398548341c at its last durable checkpoint.
```
```python theme={null}
mount = client.mount("agent-scratch", "/work2")
```
```typescript theme={null}
const mount = await client.mount("agent-scratch", "/work2");
```
A session already mounted elsewhere mounts read-only if you mount its file system again. Unmount it there first to take writes.
## Discard Local Changes
Throw away unsaved changes (and ignored files under the mount) with the mount:
```bash theme={null}
$ tl fs unmount /work --discard
Unmounted /work (unsaved local changes discarded).
```
```python theme={null}
mount.unmount(discard=True)
```
```typescript theme={null}
await mount.unmount(true);
```
Everything already published is untouched: autosave checkpoints and snapshots are immutable.
## Restore
Restore the mount contents to an earlier checkpoint or snapshot:
```bash theme={null}
$ tl fs restore /work 4d9a2f7e5b3d8c6a4f2e0d9b7c5a3f1e8d6b4c2a1029384756abcdef01234567
Restored /work to 4d9a2f7e as save 7c31e8a2 (no file bytes transferred).
```
You can restore to any autosave still inside the retention window or to any permanent snapshot.
Restore does not move the timeline backward or copy file bytes. It publishes a new automatic save
whose content points at the selected historical root, then waits for the mounted directory and its
kernel caches to converge before returning. Other writable and following mounts observe that new
shared state normally. Restoring over local changes requires `--discard`; create a permanent
snapshot first if they should survive.
## Repair a Session
If a session's local state is ever inconsistent (a hard sandbox kill mid-write, an interrupted resume), `tl fs doctor` inspects it and can repair the local journal. **Unmount first:** doctor operates on a detached session and never contacts the server, so it cannot run under a live mount.
```bash theme={null}
$ tl fs doctor /work --json
```
Pass `--repair-journal` to rebuild a damaged local journal. To re-point the session's base as part of that repair, add `--base ` (or `--base empty` to reset to an empty base); `--base` requires `--repair-journal`. Doctor only touches local session state: durable history is never modified, because it never talks to the server.
## Delete a File System
```bash theme={null}
$ tl fs rm agent-scratch
Delete filesystem agent-scratch and all of its history? This cannot be undone. [y/N]
```
```python theme={null}
client.delete("agent-scratch")
```
```typescript theme={null}
await client.delete("agent-scratch");
```
Pass `-f` to skip the confirmation; the SDK clients delete without prompting. Deletion removes the file system, its history, and its sessions.
# Read-only Mounts
Source: https://docs.tensorlake.ai/filesystems/read-only-mounts
Mount permanent snapshots or follow a file system's current state for reproducible inputs and shared assets.
Use read-only mounts when a sandbox needs files but should not write to them.
There are two read-only shapes:
* **Following**: tracks the file system's current state and refreshes as replicated autosaves or permanent snapshots land.
* **Pinned**: resolves a specific snapshot or retained autosave once and never changes.
## Following Read-only Mount
A following mount is best for shared skills, prompts, docs, configs, and dependencies.
```bash theme={null}
$ tl fs mount agent-assets /skills --ro
Mounted agent-assets at /skills (read-only, follows the filesystem)
```
```python theme={null}
from tensorlake.filesystem import FilesystemClient
client = FilesystemClient()
mount = client.mount("agent-assets", "/skills", readonly=True)
```
```typescript theme={null}
import { FilesystemClient } from "tensorlake";
const client = new FilesystemClient();
const mount = await client.mount("agent-assets", "/skills", true);
```
When the shared timeline advances, Tensorlake refreshes only the paths that changed. Unchanged files keep their warm cache.
## Pinned Read-only Mount
A pinned mount is best for reproducible builds, evals, benchmarks, and released assets.
```bash theme={null}
$ tl fs mount agent-assets:9f2a1c8e4d6b1a0f3c7e9d2b8a4f6c1e0d3b7a99fedcba987654321001234567 /release --ro
```
Use a permanent snapshot from `tl fs history` when an input must remain available indefinitely. A recent autosave can also be mounted, but it is an ephemeral recovery point and may age out of retention.
For a complete asset distribution workflow, see [Distribute Files to Agents](/filesystems/distribute-files).
## Choosing Between Them
| Need | Use |
| ------------------------------------------------------------ | ---------------------------------------------------------------- |
| Every run must see the same files | Pinned read-only mount |
| Many sandboxes should receive updates without image rebuilds | Following read-only mount |
| An agent needs to write files | [Writable mount](/filesystems/filesystem-mounts#writable-mounts) |
# Authentication
Source: https://docs.tensorlake.ai/git/authentication
Authenticate the Tensorlake CLI, plain Git clients, and repository mounts.
Most users only need one command:
```bash theme={null}
tl login
```
`tl login` opens a browser, authenticates your account, and stores a local CLI token. After that, `tl git` commands mint the short-lived credentials they need automatically.
You only handle a Git credential yourself when you use plain `git`, CI, or another HTTP client.
## How Authentication Fits Together
There are two layers:
| Layer | What it is | Used by |
| ----------------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| Tensorlake CLI/API auth | Your account, API key, or personal access token. This authorizes calls to Tensorlake. | `tl login`, `tl git`, API clients |
| Git credential | A short-lived credential minted for repository access. This authorizes repository operations. | `git clone`, `git push`, SDK calls, `tl git` mounts |
Your CLI/API credential is not sent to Git. Tensorlake uses it to mint a Git credential, then Git clients and mounts use that Git credential against the repository service.
## How Git Credentials Work
When a command needs repository access:
1. The CLI authenticates to Tensorlake with your local CLI token, API key, or PAT.
2. Tensorlake checks the current project and authorized principal.
3. Tensorlake mints a short-lived Git credential for that principal.
4. The Git service verifies the credential's signature, project, repo pattern, expiration, revocation status, and scopes on each request.
The credential contains:
| Field | Meaning |
| ------------- | --------------------------------------------------------------- |
| `gitUsername` | The Git username. It is always `t`. |
| `token` | The password Git sends with HTTP Basic auth. |
| `expiresAt` | When the credential stops working. |
| `repoPattern` | The repository or repository pattern the credential can access. |
| `scopes` | The operations the credential can perform. |
Most CLI paths mint a repo-scoped credential. That keeps clone, push, snapshot, and promotion operations narrow to the repository they are working on.
## Mint a Git Credential
```bash theme={null}
$ tl git token agent-outputs
project: project_9f3c2a1b
repo: agent-outputs
username: t
password: eyJhbGciOiJFUzI1NiJ9.eyJpc3MiOiJwcm9qZWN0XzlmM2MyYTFiIiwicmVwbyI6ImFnZW50LW91dHB1dHMifQ.Kx9f... (truncated)
expires: 2026-07-07T13:00:00Z
scopes: git:read, git:write
Remote URL
https://git.tensorlake.ai/project_9f3c2a1b/agent-outputs
Use this credential with Git or SDK clients
username: t
password: the token above
```
The username is always `t`. The password is the token.
## Use It With Git
Cache the credential with Git's credential helper:
```bash theme={null}
git config --global credential.helper store
git clone https://git.tensorlake.ai/project_9f3c2a1b/agent-outputs
Username for 'https://git.tensorlake.ai': t
Password for 'https://t@git.tensorlake.ai':
```
Git remembers the credential for later commands against the same remote.
Put the token directly in the remote URL for scripts and CI:
```bash theme={null}
TOKEN=$(tl git token agent-outputs --json | jq -r .token)
git clone https://t:$TOKEN@git.tensorlake.ai/project_9f3c2a1b/agent-outputs
```
Or update an existing remote:
```bash theme={null}
git remote set-url origin https://t:$TOKEN@git.tensorlake.ai/project_9f3c2a1b/agent-outputs
git push origin main
```
Treat a token in a URL or credential store like any other secret. It expires automatically, but it should not be committed, logged, or shared.
## Scopes
Every credential carries one or more scopes:
| Scope | Grants |
| --------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `git:read` | Clone, fetch, and read-only mount |
| `git:write` | Push, snapshot, and promote. Implies `git:read`. |
| `repo:write` | Create, fork, delete, archive, and restore repositories |
| `project:read` | List and read every repository in the project. Implies `git:read`. |
| `project:admin` | Project administration, workspace fleet management, quotas, and operation history. Implies `project:read` and `git:read`. |
`tl git token ` mints `git:read` and `git:write` for that repository only. It is enough to clone, push, snapshot, and promote. It cannot create, delete, or list other repositories.
Repo-scoped credentials cannot create repositories, delete repositories, manage keys, or revoke tokens.
## Token Lifetime
Git credentials are short-lived by design. The default lifetime is one hour.
When a token expires, mint a new one:
```bash theme={null}
tl git token
```
`tl git` mounts handle this automatically. The CLI caches fresh Git credentials for later commands and a running mount daemon rotates its credential before expiry.
For the full plain Git workflow, see [Use with Git](/git/git-repositories).
Mint a credential, clone, branch, commit, merge, and push.
API keys, personal access tokens, SSO, and broader Tensorlake API authentication.
# Git Repositories
Source: https://docs.tensorlake.ai/git/git-repositories
Scalable Tensorlake-hosted git repositories for agents
Tensorlake repositories are ordinary Git repositories. They speak the standard Git smart HTTP protocol, so you can clone, branch, commit, merge, push, and fetch with ordinary `git`.
This page covers the plain Git interface: everything here happens in a clone, with ordinary `git`. Use [Repository Mounts](/git/workspace-mounts) when an agent should work inside a mounted directory, persist snapshots, and promote when ready; [Git or the tl CLI?](/git/introduction#git-or-the-tl-cli) is the one-rule guide for choosing between them.
Use [Repository SDKs](/git/repository-sdks) when you want to create repositories, inspect refs, push worktrees, or merge branches from Python or TypeScript.
## Create a Repository
```bash theme={null}
$ tl git create agent-outputs --default-branch main
created https://git.tensorlake.ai/project_9f3c2a1b/agent-outputs
```
`project_9f3c2a1b` is the project your `tl` login is scoped to. You do not pass it to every command.
You can also create a repository lazily by pushing to a repository path that does not exist yet. Lazy creation only applies to `git push`; `git clone` and `git fetch` require the repository to already exist.
## Get a Credential
Tensorlake authenticates Git operations with HTTP Basic Auth, not SSH keys.
First make sure you are logged in:
```bash theme={null}
tl login
```
Then mint a short-lived repository credential:
```bash theme={null}
$ tl git token agent-outputs
project: project_9f3c2a1b
repo: agent-outputs
username: t
password: eyJhbGciOiJFUzI1NiJ9.eyJpc3MiOiJwcm9qZWN0XzlmM2MyYTFiIiwicmVwbyI6ImFnZW50LW91dHB1dHMifQ.Kx9f... (truncated)
expires: 2026-07-07T13:00:00Z
scopes: git:read, git:write
```
The username is always `t`. The password is the token.
## Clone
Use Git's credential store:
```bash theme={null}
git config --global credential.helper store
git clone https://git.tensorlake.ai/project_9f3c2a1b/agent-outputs
Username for 'https://git.tensorlake.ai': t
Password for 'https://t@git.tensorlake.ai':
```
Or embed the token in the URL for scripts and CI:
```bash theme={null}
TOKEN=$(tl git token agent-outputs --json | jq -r .token)
git clone https://t:$TOKEN@git.tensorlake.ai/project_9f3c2a1b/agent-outputs
```
If the token is in a URL, treat the URL as a secret.
## Branch, Commit, and Push
From here, it is ordinary Git:
```bash theme={null}
git checkout -b feature
# edit files
git add .
git commit -m "implement parser"
git push origin feature
```
## Merge Changes
Tensorlake does not add a pull-request layer on top of Git repositories. Merge locally, then push the result:
```bash theme={null}
git checkout main
git pull origin main
git merge feature
git push origin main
```
A non-fast-forward push is rejected. Pull, resolve conflicts, and push again.
Mount repositories as directories for agent sessions.
Land workspaces on moved branches and merge branches directly.
Create repositories, push worktrees, and merge branches from Python or TypeScript.
`tl login`, Git credentials, scopes, and token lifetime.
How storage, lazy content delivery, merging, and observability work.
# Git Repositories
Source: https://docs.tensorlake.ai/git/introduction
Managed Git repositories built for agents that scales to tens of millions of repositories and hundreds of commits per second per repository.
We built a disaggregated Git infrastructure to scale to tens of millions of repositories and absorb hundreds of thousands of pushes per second,
by re-engineering how Git's metadata, ingestion, and storage work. Merges run server-side and cost is proportional to
changed paths, not repository size.
These are some use cases for Tensorlake repositories:
* Store code and assets produced by coding agents, one repository per generated project
* Give agents a durable Git history with branches, merges, and activity attribution
* Create repositories at product scale from your control plane with the SDKs
* Mount a repository as a live directory in a sandbox, without cloning it first
* Track durable workspace checkpoints, snapshots, branch activity, and mount liveness from the control plane
![Git Repository Dashboard]()
## Quickstart
Install the Tensorlake CLI and sign in:
```bash theme={null}
curl -fsSL https://tensorlake.ai/install | sh
tl login
```
Then create a repository, commit files to `main`, and verify the history with Git.
```bash theme={null}
$ tl git create agent-outputs --default-branch main
created https://git.tensorlake.ai/project_9f3c2a1b/agent-outputs
```
```bash theme={null}
$ pip install tensorlake
$ export TENSORLAKE_API_KEY=tlk_...
```
```python theme={null}
from tensorlake import RepositoryClient
with RepositoryClient.from_env() as repos:
repo = repos.create("agent-outputs", default_branch="main")
print(repo.url)
```
```bash theme={null}
$ npm install tensorlake
$ export TENSORLAKE_API_KEY=tlk_...
```
```typescript theme={null}
import { RepositoryClient } from "tensorlake";
const repos = await RepositoryClient.fromEnv();
const repo = await repos.create("agent-outputs", {
defaultBranch: "main",
});
console.log(repo.url);
```
`agent-outputs` is an ordinary Git repository. Its default branch is `main`.
Use Git directly, or push a local worktree from your application.
Clone with a short-lived credential, then use Git as usual:
```bash theme={null}
$ TOKEN=$(tl git token agent-outputs --json | jq -r .token)
$ git clone https://t:$TOKEN@git.tensorlake.ai/project_9f3c2a1b/agent-outputs
$ cd agent-outputs
```
First commit on a fresh machine? Git needs an identity: `git config user.name "Agent User"` and `git config user.email "agent@example.com"`.
```bash theme={null}
$ echo "# Agent Outputs" > README.md
$ git add README.md
$ git commit -m "initial app"
$ git push origin main
$ git log --oneline -1
4f8c2a1 initial app
```
```bash theme={null}
$ mkdir -p initial-app
$ echo "# Agent Outputs" > initial-app/README.md
```
```python theme={null}
from tensorlake import RepositoryClient
with RepositoryClient.from_env() as repos:
report = repos.push_worktree(
"agent-outputs",
root="./initial-app",
branch="main",
message="initial app",
)
print(report.commit)
```
```bash theme={null}
$ mkdir -p initial-app
$ echo "# Agent Outputs" > initial-app/README.md
```
```typescript theme={null}
import { RepositoryClient } from "tensorlake";
const repos = await RepositoryClient.fromEnv();
const report = await repos.pushWorktree("agent-outputs", {
path: "./initial-app",
branch: "main",
message: "initial app",
});
console.log(report.commit);
```
The repository now has a normal Git commit on `main`.
```bash theme={null}
$ tl git mount agent-outputs /work
Mounted repository agent-outputs at /work (workspace 3f9a2b7e1c4d).
$ ls /work
README.md
```
So far, `agent-outputs` has behaved like a normal Git repository, and `git clone` would work here too. Mounting skips the clone: content streams in as it's read. A writable mount is the default; it creates its private server workspace lazily when the first autosave WAL checkpoint arrives. Add `--ro` for a stateless read-only view that follows the branch or pins a commit.
```bash theme={null}
$ printf "\n## Parser Notes\n" >> /work/README.md
$ tl git snapshot /work -m "add notes"
Snapshot 8b21f6a9c3d5 retained permanently.
$ tl git promote /work main
Snapshot 8b21f6a9c3d5 retained permanently.
Promoted workspace 3f9a2b7e1c4d -> refs/heads/main at 1c4d9a2f7e5b.
$ git -C agent-outputs fetch origin main
$ git -C agent-outputs log --oneline -2 origin/main
1c4d9a2 add notes
4f8c2a1 initial app
```
Autosave already protects the mounted file state in the workspace's durable private WAL. Snapshotting materializes that state as a Git commit; the branch is unchanged until promotion. After promotion, `main` contains the change. New mounts, `git clone`, and `git fetch` all see the same files.
## Mental Model
The workflow is:
1. A **repository** stores the durable Git history.
2. A **mount** gives a sandbox an ephemeral directory backed by the repository, no clone required. A mount is writable by default; add `--ro` for a stateless read-only view of a branch, a pinned commit, or a subtree of a monorepo.
3. The shared **local journal** records writes crash-safely. Autosave publishes those writes to a durable private **workspace WAL** without creating a commit or moving a branch.
4. A **workspace** is created lazily on the first remote WAL checkpoint and holds an agent's isolated state.
5. A **snapshot** materializes the current WAL as a Git commit on the workspace.
6. **Promotion** publishes a workspace snapshot to a branch so future mounts and Git users can use that version. **Rebase** replays a workspace onto a moved branch, server-side.
7. The control plane can observe mount liveness and durable workspace operations. The unsealed local edit tail stays in the sandbox until autosave.
```mermaid theme={null}
graph LR
subgraph SB["Sandbox"]
M["Mount at /work
lazy view of the workspace
+ local edits"]
end
subgraph R["Repository (server)"]
direction TB
B["Branch main
published history"]
W["Workspace
private WAL · snapshots: s1 → s2 → s3"]
end
B -- "first WAL checkpoint" --> W
W -- "mounted into" --> M
W -- "promote" --> B
```
Durable published history lives in the repository branch. Durable in-progress work lives in each workspace's WAL and snapshot chain. The sandbox holds the lazy mount plus the newest unsealed local edit tail. Work flows in a circle: autosave privately, snapshot deliberate commits, then promote the result back to a branch.
## Git or the tl CLI?
Both work against the same repository. Which one you use depends on what is on disk in front of you:
* **You cloned** (`git clone`) → use ordinary Git. A clone is a real checkout with a `.git` directory; Tensorlake is a normal remote. Commit, rebase, merge, and push exactly as you would anywhere.
* **You mounted** (`tl git mount`) → use the `tl git` verbs. A mount has no `.git` and Git commands do not run inside it; the verbs are the mount's interface, and each maps onto a Git habit:
| Git habit on a clone | Equivalent on a mount |
| ------------------------------------ | ------------------------------ |
| `git commit` | `tl git snapshot` |
| `git fetch` / switch a pristine view | `tl git sync` |
| `git rebase origin/main` | `tl git rebase` |
| `git push` | `tl git promote` |
| `git status` / `git log` | `tl git status` / `tl git log` |
* **You have no files at all** (a control plane, CI, an SDK caller) → use `tl git merge` and the SDKs. Merges, preflights, and conflict queries run server-side without any checkout.
The two sides always converge through the repository: a promoted workspace is a normal commit that `git fetch` sees, and a `git push` shows up in mounts on their next refresh or sync.
## Summary of Features
* **Plain Git repositories**: clone, branch, commit, merge, push, and fetch over Git smart HTTP.
* **Server-side merges and rebases**: merge branches, preflight conflicts, rebase workspaces, and query structured conflict records without a checkout.
* **Repository mounts**: instant read-only views of any branch, commit, or subtree, and isolated workspaces with private autosave WAL plus explicit snapshot, promote, and rebase when agents need to write.
* **Fleet observability**: durable workspace checkpoints, snapshots, operations, and mount liveness are available to the control plane, with actor attribution in activity history.
* **SDKs**: create repositories, push worktrees, and merge branches from Python or TypeScript.
## Where To Go Next
Mint a credential, clone, branch, commit, merge, and push with ordinary Git.
Mount any branch, commit, or subtree into a sandbox; snapshot progress and promote finished work.
Create repositories, push worktrees, and merge branches from Python or TypeScript.
Land workspaces on moved branches, resolve conflicts, and merge branches directly.
How CLI credentials and short-lived Git credentials fit together.
Store generated apps, docs, and assets with snapshots, promotion, and activity history.
# Merging Changes
Source: https://docs.tensorlake.ai/git/merging
Land workspace changes on branches that moved, rebase a workspace, resolve conflicts, and merge branches directly.
Two situations bring you to this page:
* **A workspace and its branch both changed.** This needs a [mounted workspace](/git/workspace-mounts): the commands take a mount path like `/work`, and the mounted directory is where conflicts get resolved. The normal loop is: rebase onto the branch, fix any conflicts, snapshot the resolution, then promote. The daemon seals pending local WAL before the rebase starts.
* **Two branches diverged.** [`tl git merge`](#merge-branches-directly) merges them directly on the server: no mount, no clone, no working copy. It runs from any machine with the CLI, or from your control plane with the SDKs.
Tensorlake never force-overwrites a branch. If a merge cannot land cleanly, nothing is published unless you explicitly choose a mode that materializes conflict markers.
If you are working in a **clone**, none of this applies. Use ordinary `git merge`, `git rebase`, and `git push`; Tensorlake is a normal remote. See [Git or the tl CLI?](/git/introduction#git-or-the-tl-cli) for the one-rule guide.
## Promote a Workspace
If the target branch has not moved since the workspace was created, promotion lands the workspace on that branch:
```bash theme={null}
$ tl git promote /work main
Promoted workspace 3f9a2b7e1c4d -> refs/heads/main at 1c4d9a2f7e5b.
```
If the branch moved, a plain promotion fails instead of overwriting someone else's work. Add `--merge` when Tensorlake should merge the workspace with the latest branch tip:
```bash theme={null}
$ tl git promote /work main --merge
Promoted workspace 3f9a2b7e1c4d -> refs/heads/main at 7a1c3e5b9d2f.
```
When the changes do not overlap, this lands one merge commit combining both histories. If the branch did not move, it fast-forwards like a plain promotion.
## When Promotion Conflicts
If the workspace and branch changed the same content, promotion publishes nothing. The target branch and workspace stay unchanged, and the command reports each conflicted path:
```bash theme={null}
$ tl git promote /work main --merge
error: promote to main conflicts on 2 path(s); nothing was published:
content src/parser.py
delete_modify notes/plan.md
```
Resolve the conflict in the workspace:
1. Run `tl git rebase /work main`.
2. Edit the conflicted files.
3. Snapshot the resolved files.
4. Promote again.
## Rebase the Branch into a Workspace
`tl git rebase` replays the workspace onto the target branch. The mounted directory updates in place:
```bash theme={null}
$ tl git rebase /work main
Rebased workspace onto main (3 path(s) changed).
```
If the rebase conflicts, Tensorlake writes standard Git diff3 markers into the workspace files:
```bash theme={null}
$ tl git rebase /work main
note: 1 conflict(s) materialized as diff3 markers:
content src/parser.py
Resolve the markers, then `tl git snapshot /work`.
```
The file carries the branch version, the merge base, and the workspace version:
```text theme={null}
<<<<<<< main
def parse(text: str) -> Ast:
||||||| base
def parse(text):
=======
def parse(text, *, strict=False):
>>>>>>> workspace 3f9a2b7e1c4d
```
Edit the file to resolve the conflict, then snapshot and promote:
```bash theme={null}
$ tl git snapshot /work -m "resolve parser conflict"
$ tl git promote /work main --merge
Promoted workspace 3f9a2b7e1c4d -> refs/heads/main at 4b6d8e0a2c4f.
```
The daemon serializes the transition with autosave and seals pending local WAL before rebasing, so edits do not race the base change.
## Sync a View
`tl git sync /work []` has a narrower role than rebase. It refreshes the current source or switches a read-only or snapshot-free writable view, carrying any unsnapshotted WAL tail onto the new base. It refuses a switch that would rewrite an established workspace snapshot chain and directs you to `tl git rebase` instead.
Use sync for checkout-like source changes before a workspace has snapshot history. Use rebase once workspace commits must be replayed onto another base.
## Structured Conflict Records
A merge that materializes conflicts (a `--materialize` direct merge or a conflicted rebase) records more than markers. Each conflicted commit carries a structured record naming every path and the three-way terms it was merged from, available to the control plane and queryable without parsing file contents:
```bash theme={null}
$ tl git conflicts agent-outputs 7a1c3e5b9d2f4b6d8e0a2c4f6e8b0d3a5c7e9f1b
ours: 4b6d8e0a2c4f
theirs: 9c4e2a7b1f3d
base: 5f2c8e1a9b3d
conflicts: 1 path(s):
content src/parser.py
```
Nothing is silently overwritten: both sides of every conflict remain reachable from the record.
## Merge Branches Directly
Use `tl git merge` when you want to merge one branch into another without creating a workspace.
Preflight first to see what would happen. It never writes:
```bash theme={null}
$ tl git merge agent-outputs main dev --preflight
merge base: 5f2c8e1a9b3d
changed paths: 12
Clean merge.
```
A shallow preflight reports same-file collisions as *potential* conflicts:
```bash theme={null}
$ tl git merge agent-outputs main dev --preflight
merge base: 5f2c8e1a9b3d
changed paths: 12
conflicts: 1 conflict(s):
content src/parser.py (potential)
(run with --deep for exact content-merge answers)
```
Add `--deep` when you need exact text-merge results. It runs the text merges and drops the
`(potential)` qualifier:
```bash theme={null}
$ tl git merge agent-outputs main dev --preflight --deep
merge base: 5f2c8e1a9b3d
changed paths: 12
conflicts: 1 conflict(s):
content src/parser.py
```
Drop `--preflight` to land the merge:
```bash theme={null}
$ tl git merge agent-outputs main dev -m "land dev"
merge base: 5f2c8e1a9b3d
changed paths: 12
Merged dev into main at 7a1c3e5b9d2f
```
A conflicted direct merge publishes nothing and exits non-zero. Add `--materialize` to land it anyway with markers and a structured conflict record. Add `--json` for machine-readable reports.
## How Merging Runs
Merges run server-side with a three-way merge engine. Tensorlake does not clone the repository, check out a working tree, or scan every file.
Merge cost scales with the number of changed paths, not total repository size.
## Conflict Kinds
Conflict reports classify each path:
| Kind | Meaning |
| --------------- | ---------------------------------------------------------------------------- |
| `content` | Both sides edited the same region of a text file. |
| `delete_modify` | One side deleted the path; the other modified it. The modified side is kept. |
| `add_add` | Both sides added the path with different content. |
| `kind_mismatch` | The path is a file on one side and a directory or symlink on the other. |
| `mode` | Both sides changed the file mode to different values. |
| `too_large` | A side is binary or over the text-merge size limit; never text-merged. |
## Next Steps
Mount, snapshot, and promote: the workflow merging builds on.
How the merge engine and conflict policies work internally.
# Repository SDKs
Source: https://docs.tensorlake.ai/git/repository-sdks
Create repositories, inspect refs, push worktrees, and merge branches from Python and TypeScript.
Use the repository SDKs to manage Tensorlake Git repositories from application code: create repositories, inspect branches and refs, mint Git credentials, push a local worktree, and run server-side merges.
Use the `tl git` mount family for mounted workflows: see [Repository Mounts](/git/workspace-mounts).
## Install and Configure
The SDKs use a Tensorlake API key from the environment.
```bash theme={null}
pip install tensorlake
export TENSORLAKE_API_KEY=tlk_...
```
```bash theme={null}
npm install tensorlake
export TENSORLAKE_API_KEY=tlk_...
```
PATs are CLI-only. Use an API key with the SDKs.
## Create and List Repositories
`RepositoryClient` is the entry point for repository operations.
```python theme={null}
from tensorlake import RepositoryClient
with RepositoryClient.from_env() as repos:
repo = repos.create("agent-outputs", default_branch="main")
print(repo.url)
for item in repos.list():
print(item.name, item.default_branch, item.status)
```
```typescript theme={null}
import { RepositoryClient } from "tensorlake";
const repos = await RepositoryClient.fromEnv();
const repo = await repos.create("agent-outputs", {
defaultBranch: "main",
});
console.log(repo.url);
for (const item of await repos.list()) {
console.log(item.name, item.defaultBranch, item.status);
}
```
Later examples assume `repos` is a `RepositoryClient` created with `from_env()` or `await fromEnv()`.
## Inspect Branches and Refs
Use `info` when you want the repository URL, branches, and refs in one call.
```python theme={null}
from tensorlake import RepositoryClient
repos = RepositoryClient.from_env()
info = repos.info("agent-outputs")
print(info.url)
for branch in info.branches:
print(branch.name, branch.oid)
```
```typescript theme={null}
import { RepositoryClient } from "tensorlake";
const repos = await RepositoryClient.fromEnv();
const info = await repos.info("agent-outputs");
console.log(info.url);
for (const branch of info.branches) {
console.log(branch.name, branch.oid);
}
```
## Get a Git Credential
Most SDK methods mint short-lived Git credentials automatically. Call `credential` only when you need to hand a token to `git`, CI, or another HTTP client.
```python theme={null}
credential = repos.credential("agent-outputs")
print(credential.git_username) # always "t"
print(credential.token) # use as the Git HTTP password
```
```typescript theme={null}
const credential = await repos.credential("agent-outputs");
console.log(credential.gitUsername); // always "t"
console.log(credential.token); // use as the Git HTTP password
```
## Push a Local Worktree
`push_worktree` creates one commit from a local directory and updates a branch. It skips `.git` and honors `.gitignore`.
```python theme={null}
report = repos.push_worktree(
"agent-outputs",
root=".",
branch="main",
message="sync generated output",
)
print(report.commit)
print(report.ref_name)
```
```typescript theme={null}
const report = await repos.pushWorktree("agent-outputs", {
path: process.cwd(),
branch: "main",
message: "sync generated output",
});
console.log(report.commit);
console.log(report.refName);
```
Pass `expect_oid` in Python or `expectOid` in TypeScript when you want the push to fail if the branch moved.
## Merge Branches
Use `merge` to merge one branch or commit into another without cloning the repository.
Preflight first when you want to see whether a merge is clean:
```python theme={null}
report = repos.merge(
"agent-outputs",
"main",
"feature",
preflight=True,
deep=True,
)
if report.clean:
landed = repos.merge(
"agent-outputs",
"main",
"feature",
message="merge feature into main",
)
print(landed.commit)
else:
for conflict in report.conflicts:
print(conflict.kind, conflict.path)
```
```typescript theme={null}
const report = await repos.merge("agent-outputs", "main", "feature", {
preflight: true,
deep: true,
});
if (report.clean) {
const landed = await repos.merge("agent-outputs", "main", "feature", {
message: "merge feature into main",
});
console.log(landed.commit);
} else {
for (const conflict of report.conflicts) {
console.log(conflict.kind, conflict.path);
}
}
```
By default, a conflicted commit-mode merge publishes nothing and returns a report with `clean: false` and no `commit`.
Use `materialize` when you want conflicted files to land with standard Git diff3 markers:
```python theme={null}
report = repos.merge(
"agent-outputs",
"main",
"feature",
materialize=True,
message="merge feature into main",
)
if report.commit and not report.clean:
record = repos.commit_conflicts("agent-outputs", report.commit)
for path in (record.paths if record else []):
print(path.kind, path.path)
```
```typescript theme={null}
const report = await repos.merge("agent-outputs", "main", "feature", {
materialize: true,
message: "merge feature into main",
});
if (report.commit && !report.clean) {
const record = await repos.commitConflicts("agent-outputs", report.commit);
for (const path of record?.paths ?? []) {
console.log(path.kind, path.path);
}
}
```
## API Surface
| Task | Python | TypeScript |
| ------------------- | ------------------------------------------------------------- | ---------------------------------------------------------- |
| Create a repository | `create(repo, default_branch=None)` | `create(repo, { defaultBranch })` |
| List repositories | `list()` | `list()` |
| Delete a repository | `delete(repo)` | `delete(repo)` |
| Fork a repository | `fork(repo, base_repo)` | `fork(repo, baseRepo)` |
| Archive or restore | `archive(repo)`, `restore(repo)` | `archive(repo)`, `restore(repo)` |
| Repository URL | `url(repo)` | `url(repo)` |
| Branches and refs | `info(repo)`, `branches(repo)`, `refs(repo)` | `info(repo)`, `branches(repo)`, `refs(repo)` |
| Delete a branch | `delete_branch(repo, branch)` | `deleteBranch(repo, branch)` |
| Operation history | `operations(repo)` | `operations(repo)` |
| Git credential | `credential(repo=None)` | `credential(repo)` |
| Push local files | `push_worktree(repo, root, branch, message, expect_oid=None)` | `pushWorktree(repo, { path, branch, message, expectOid })` |
| Push job status | `commit_status(repo, job_id)` | `commitStatus(repo, jobId)` |
| Merge branches | `merge(repo, ours, theirs, ...)` | `merge(repo, ours, theirs, options)` |
| Conflict records | `commit_conflicts(repo, commit)` | `commitConflicts(repo, commit)` |
## Next Steps
Use the same repositories with ordinary `git clone`, `git push`, and `git fetch`.
Store code, docs, and assets produced by coding agents at repository scale.
Learn the merge behavior behind SDK and CLI merge operations.
# Store Agent-Generated Code
Source: https://docs.tensorlake.ai/git/store-generated-code
Store code, docs, and assets produced by coding agents in versioned repositories.
Use Tensorlake repositories when your product creates or updates many user projects with coding agents.
A coding agent platform may create thousands of repositories each day and keep updating existing applications as users ask for changes. Tensorlake gives every generated project a durable Git source of truth, private autosave WAL for active agents, snapshots for deliberate commits, and activity history for visibility.
## Pattern
1. Create one repository per generated app, site, package, or user project.
2. Mount the repository when an agent needs to change it.
3. Let autosave protect work during the run; snapshot deliberate milestones as Git commits.
4. Promote accepted changes back to the project branch.
5. Use repository activity to see what agents created, updated, merged, or published.
## Create Repositories at Product Scale
Create repositories from your control plane when users start new projects:
```bash theme={null}
export TENSORLAKE_API_KEY=tlk_...
```
```python theme={null}
from tensorlake import RepositoryClient
with RepositoryClient.from_env() as repos:
repo = repos.create("app-7f3c2a1b", default_branch="main")
print(repo.url)
```
```bash theme={null}
export TENSORLAKE_API_KEY=tlk_...
```
```typescript theme={null}
import { RepositoryClient } from "tensorlake";
const repos = await RepositoryClient.fromEnv();
const repo = await repos.create("app-7f3c2a1b", {
defaultBranch: "main",
});
console.log(repo.url);
```
Use stable repository names, such as an internal app id. Human-facing app names can change without changing the storage identity.
## Why This Scales
A repository is the durable unit for a user project. A mount is the ephemeral file-system path for one sandbox. A workspace is the isolated snapshot history for one agent run.
That separation keeps high-volume platforms simple: your control plane can create, list, fork, archive, and update repositories through the SDKs, while agents work through writable mounts. Mounting does not copy the whole repository into the sandbox. File content is fetched as processes read it, autosave persists changed content into private workspace WAL, and snapshots materialize deliberate Git commits without forcing every run to become one large final commit.
Use branches for product states such as `main`, `preview`, or `release`. Use workspaces for in-progress agent attempts.
## Store Generated Files
When a backend process already has a generated worktree, push it directly:
```python theme={null}
report = repos.push_worktree(
"app-7f3c2a1b",
root="./generated-app",
branch="main",
message="create initial app",
)
print(report.commit)
```
```typescript theme={null}
const report = await repos.pushWorktree("app-7f3c2a1b", {
path: "./generated-app",
branch: "main",
message: "create initial app",
});
console.log(report.commit);
```
`push_worktree` and `pushWorktree` create one commit from a local directory. They skip `.git`, honor `.gitignore`, preserve symlinks, and preserve executable bits on regular files.
Use this for generated code, app assets, docs, migrations, configuration, and deployment metadata:
```text theme={null}
generated-app/
src/
public/
docs/
migrations/
package.json
```
## Let Agents Work in a Mounted Workspace
When an agent is editing an existing project, mount the repository into the sandbox:
```bash theme={null}
$ tl git mount app-7f3c2a1b /work
Mounted repository app-7f3c2a1b at /work (workspace 3f9a2b7e1c4d).
```
The agent edits `/work` like a normal directory. Autosave protects the changing state in private workspace WAL. Create snapshots when the run reaches meaningful Git history boundaries:
```bash theme={null}
$ tl git snapshot /work -m "scaffold billing page"
Snapshot 8b21f6a9c3d5 retained permanently.
$ tl git snapshot /work -m "wire billing API"
Snapshot 4d9a2f7e5b3d retained permanently.
```
Autosave makes long-running agent work durable without producing a stream of commits. If a sandbox stops, the workspace can be reattached through its latest server WAL checkpoint. Snapshots provide deliberate points you can inspect, rebase, and promote.
When the user accepts the result, publish it. Promotion first seals and materializes any dirty WAL, so the latest edits are included even if the agent did not run one final snapshot command:
```bash theme={null}
$ tl git promote /work main
Promoted workspace 3f9a2b7e1c4d -> refs/heads/main at 1c4d9a2f7e5b.
```
After promotion, future agents that mount `app-7f3c2a1b`, build systems that clone it, and developers who fetch it get the published version.
## Update Existing Repositories Safely
Generated applications keep changing after the first version. Users ask agents to add pages, fix bugs, change copy, update dependencies, or regenerate docs.
For backend pushes, pass `expect_oid` in Python or `expectOid` in TypeScript when the update should fail if the branch moved:
```python theme={null}
info = repos.info("app-7f3c2a1b")
current = next(b.oid for b in info.branches if b.name == "main")
report = repos.push_worktree(
"app-7f3c2a1b",
root="./generated-app",
branch="main",
message="update homepage",
expect_oid=current,
)
```
```typescript theme={null}
const info = await repos.info("app-7f3c2a1b");
const current = info.branches.find((branch) => branch.name === "main")?.oid;
const report = await repos.pushWorktree("app-7f3c2a1b", {
path: "./generated-app",
branch: "main",
message: "update homepage",
expectOid: current,
});
```
Use server-side merge APIs when two agents or a user and an agent changed the same project branch. Preflight first when you want to know whether a merge is clean before publishing.
## Visibility for Agent Fleets
Every repository and workspace has activity you can inspect through the API and dashboard:
* Which repositories exist for user projects.
* Which branches changed.
* Which workspaces are live, detached, or resumable.
* Which durable WAL checkpoints and snapshots each agent created.
* Which paths changed in a snapshot.
* Which principal pushed, promoted, merged, or reconciled changes.
This gives the product control plane enough state to answer operational questions: which users have active generations, which runs published to production branches, and where a failed run can resume. Fine-grained edits remain local until autosave; the control plane observes durable checkpoints rather than every keystroke.
## Scale Model
| Need | Use |
| ------------------------------------- | -------------------------------------------------------------------- |
| New user app or project | Create a repository |
| Agent edits an existing app | Mount `main` (writable by default) |
| Long-running generation | Rely on autosave for recovery; snapshot meaningful commit boundaries |
| User accepts a result | Promote the workspace to the project branch |
| Backend writes generated code or docs | Repository SDK `push_worktree` |
| Prevent overwriting newer work | `expect_oid` / `expectOid` |
| Inspect what agents did | Workspace status, snapshots, operations, and branch activity |
| Build or deploy generated code | Ordinary `git clone`, `git fetch`, or a read-only mount |
## Next Steps
Create repositories, push worktrees, merge branches, and inspect refs from application code.
Mount, snapshot, and promote code generated by agents.
Handle branches that moved while an agent was working.
Understand snapshots, lazy content delivery, merge behavior, and observability.
# Repository Mounts
Source: https://docs.tensorlake.ai/git/workspace-mounts
Mount a repository lazily, autosave into private workspace WAL, materialize deliberate commits, and land changes with server-side promote and rebase.
A mount gives a sandbox a live directory backed by a repository, without cloning it. Attach takes about a second regardless of repository size; file content streams in as processes read it, and unopened content never transfers.
By default a mount is **writable**. The agent edits `/work` like any directory, and changes autosave to a durable private workspace WAL. The server workspace is created lazily on the first WAL checkpoint. Autosave never creates a Git commit or updates a branch; snapshots and promotion remain deliberate. Add `--ro` for a read-only view that creates no workspace and needs no cleanup.
Because durable WAL checkpoints, snapshots, operations, and mount heartbeats live on the server, your control plane can track workspace state across a fleet of sandboxes. See [Observe Workspaces](#observe-workspaces).
Directory mounts also automatically save your work and if your sandbox crashes, you can resume from the last auto-saved checkpoint.
Install the file-system extension once on macOS. Linux needs no setup.
```bash theme={null}
tl fs setup --check
tl fs setup
```
## Choose a Mount
The command shape:
```bash theme={null}
tl git mount [:][//] [--ro] [--workspace ] [--publish]
```
| Mount | Command | Use it for |
| ------------------------ | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------- |
| Writable mount (default) | `tl git mount agent-outputs /work` | An agent editing privately before promotion; workspace activates on first autosave |
| Read-only branch view | `tl git mount agent-outputs /code --ro` | Read a repository at the moving branch tip |
| Read-only pinned commit | `tl git mount agent-outputs: /release --ro` | Fixed inputs, reproducible builds, agent context |
| Read-only subtree | `tl git mount agent-outputs:main//services/auth /code --ro` | One directory of a monorepo |
| Resume a workspace | `tl git mount agent-outputs /work --workspace ` | Reattach an existing workspace on another machine |
| Publish on snapshot | `tl git mount agent-outputs:main /work --publish` | Every explicit snapshot also promotes to the branch; autosaves stay private |
## Writable Mounts (the Default)
```bash theme={null}
$ tl git mount agent-outputs /work
Mounted repository agent-outputs at /work (workspace 3f9a2b7e1c4d).
```
`/work` is an ordinary writable directory based on the branch tip; name a different base with `agent-outputs:`. Writes enter a crash-safe local journal and autosave to private server WAL; the branch does not change while the agent works. Workspace mounts can take a subtree too; snapshots record paths relative to the repository root.
## Autosave WAL
Writable mounts autosave every 30 seconds. Each autosave uploads changed content and advances the workspace's durable WAL checkpoint, but creates no Git commit and moves no branch. This is the recovery layer for an agent run: another machine can resume through the latest server checkpoint, while the same machine can additionally recover a newer unsealed tail from its local journal.
Autosave is intentionally separate from Git history. Use `tl git snapshot` when the current state should become an attributable workspace commit.
## Read-Only Mounts
Add `--ro` for a view that creates no workspace and needs no cleanup:
```bash theme={null}
$ tl git mount agent-outputs /code --ro
Mounted repository agent-outputs read-only at /code (refs/heads/main at 9f2a1c8e4d6b; no workspace).
```
A read-only branch mount follows the branch: when it moves, Tensorlake refreshes only the paths that changed. Reads are always consistent: a file open when the branch moves keeps serving the version it was opened against.
Pin a commit when the input must never change (pinning takes a **full** commit id, not an abbreviation):
```bash theme={null}
$ tl git mount agent-outputs:9f2a1c8e4d6b1a0f3c7e9d2b8a4f6c1e0d3b7a99 /release --ro
Mounted repository agent-outputs read-only at /release (9f2a1c8e4d6b1a0f3c7e9d2b8a4f6c1e0d3b7a99 at 9f2a1c8e4d6b; no workspace).
```
Mount a subtree when the task only needs part of a large repository. The mount root becomes that directory, and nothing outside it is fetched:
```bash theme={null}
$ tl git mount agent-outputs:main//services/auth /code --ro
Mounted repository agent-outputs read-only at /code (refs/heads/main at 9f2a1c8e4d6b; no workspace).
```
Subtree and commit forms combine: `agent-outputs:9f2a1c8e4d6b1a0f3c7e9d2b8a4f6c1e0d3b7a99//services/auth`.
## Snapshot Changes
```bash theme={null}
$ tl git snapshot /work -m "implemented parser and tests"
Snapshot 8b21f6a9c3d5 retained permanently.
```
Snapshotting first seals any pending local changes, then materializes the resulting WAL state as one commit on the workspace. It is a deliberate history boundary, not the first point of durability. Snapshotting with no changes is a no-op:
```bash theme={null}
$ tl git snapshot /work
Snapshot 8b21f6a9c3d5 retained permanently.
```
A clean snapshot mints no new commit; it reports the snapshot already at the workspace tip.
## Promote to a Branch
```bash theme={null}
$ tl git promote /work main
Snapshot 8b21f6a9c3d5 retained permanently.
Promoted workspace 3f9a2b7e1c4d -> refs/heads/main at 1c4d9a2f7e5b.
```
Promotion is the deliberate path to a branch. Before landing, it autosaves and materializes any dirty WAL, then lands the workspace as a squashed commit. Compare-and-swap safety prevents a plain promotion from overwriting a branch that moved. Add `--merge` to land a two-parent merge instead. See [Merging Changes](/git/merging) for conflict reports and the resolve loop.
Activity history shows who published the promotion and which workspace it came from.
## Rebase a Workspace
When the branch has moved and you want the workspace replayed on top of it (a linear history instead of a merge), rebase it onto a target ref or commit:
```bash theme={null}
$ tl git rebase /work main
Rebased workspace onto main (3 path(s) changed).
```
The target is required; rebase onto any branch, tag, or full commit. The daemon first seals pending local WAL, then the replay runs server-side and the mounted directory updates in place. If both sides changed the same content, the conflicting files carry standard diff3 markers: resolve, snapshot, and promote. After a clean rebase, promotion fast-forwards. Add `--fail-on-conflict` to report conflicts without materializing markers into the workspace.
## Check Status
```bash theme={null}
$ tl git status /work
repository: agent-outputs
reference: refs/heads/main
state: workspace_snapshotted_unpromoted
workspace: 3f9a2b7e1c4d
base: 9c4e2a7b1f3d
snapshot: 8b21f6a9c3d5
target: 9c4e2a7b1f3d
relationship: ahead
local changes: 0 path(s)
next: tl git promote
next: tl git rebase
```
Daemon and credential diagnostics follow those lines; they are for support, not for the workflow.
With unsealed changes, status reports how many paths are dirty and makes snapshotting the next valid transition:
```bash theme={null}
state: workspace_locally_dirty
local changes: 2 path(s)
next: tl git snapshot
```
Status names the state a workspace is in together with the commands that leave it, and never recommends a transition the state does not permit — a dirty overlay is told to snapshot, because rebase and sync reject one. Besides the two above, the states are `workspace_clean`, `workspace_target_advanced` (the branch moved past your base), `read_only_following`, `read_only_pinned`, `source_ref_deleted`, and `server_unreachable_stale_view` for a mount that cannot currently reach the server. Add `--json` for machine-readable output.
## Observe Workspaces
Durable workspace state is server-side, so you can inspect a fleet without touching its sandboxes. List workspace WAL, snapshot, and attachment state first:
```bash theme={null}
tl git workspaces agent-outputs
```
`tl git smartlog` shows every branch, tag, workspace, and snapshot chain and how they sit relative to each other:
```bash theme={null}
$ tl git smartlog agent-outputs
repository project_9f3c2a1b/agent-outputs
main 1c4d9a2f7e5b
├─ 3f9a2b7e1c4d 8b21f6a9c3d5 implemented parser and tests ahead 1
├─ 7d4c1e9b2a6f 5e3a7c1d9b2f refactor config loader ahead 2
└─ b2e8f4a6c0d1 (no snapshots) at base
```
`tl git log` shows one mount's own workspace snapshot chain; `tl git smartlog --project` widens the view to the whole project. The same graph is in the dashboard's repository page, with each workspace's snapshot chain and actor attribution. Every snapshot, promote, rebase, and merge is also a durable entry in the repository's activity history: the audit trail of what your agents did, queryable long after the sandboxes are gone.
Live mounts also report a **liveness heartbeat** (that a mount exists, where it's mounted, and that it's still alive) so the control plane can tell which sessions are active. Fine-grained edits remain local until autosave; durable WAL checkpoints report progress without exposing every in-progress edit.
## Reattach a Workspace
The mount path is disposable. The workspace is the resumable state behind it. Unmounting keeps the workspace by default:
```bash theme={null}
$ tl git unmount /work
Unmounted /work.
```
Reattach it later from the same sandbox, another sandbox, or another machine. The new mount path can be different. `--workspace` takes the **full** workspace id (`tl git workspaces ` lists them; mount and status print an abbreviation):
```bash theme={null}
$ tl git mount agent-outputs /work2 --workspace 3f9a2b7e1c4d5e7f1a2b4c6d8e0f3a5b7c9d1e3f
Mounted repository agent-outputs at /work2 (workspace 3f9a2b7e1c4d).
```
A workspace that already has a live writable mount refuses a second mount. Unmount it there first to take writes.
## Delete a Workspace
Delete the workspace while unmounting when its history is no longer needed:
```bash theme={null}
$ tl git unmount /work --delete
Unmounted /work.
```
The workspace WAL and unpublished snapshots become unreachable. Promote first if the work should survive as branch history.
Detached workspaces are also collected automatically. By default, a WAL-only workspace is retained for 48 hours and a workspace with snapshots for 14 days. An actively mounted workspace is retained. These periods are deployment configuration, so use explicit deletion when your application requires deterministic cleanup.
## What a Mount Is Not
* **Not a Git checkout.** There is no `.git` directory inside a mount, and `git` commands do not run there. The verbs on this page are the interface; see [Git or the tl CLI?](/git/introduction#git-or-the-tl-cli) for how they map onto Git habits. `tl git clone` exists when you need a real clone.
* **Not auto-publishing.** A repository mount autosaves to private workspace WAL, but autosave never creates a commit or changes a branch. For a shared directory whose autosave windows periodically advance the common state, use a [file system](/filesystems/introduction).
* **Not fully offline.** Cached content and the crash-safe local journal can remain usable during a disconnection. Fetching uncached content and making work remotely durable require reconnection.
## Next Steps
Land workspaces on moved branches, resolve conflicts, and merge branches directly.
Learn how promotion and merge conflict handling work internally.
# Access Control
Source: https://docs.tensorlake.ai/platform/access-control
Organization and project hierarchy, role-based permissions, and user management.
This guide covers Tensorlake's access control system, including user roles and permissions for both dashboard users and programmatic API access. The system manages access through a hierarchical structure of organizations and projects, with role-based permissions that apply to both human users and API keys.
Dashboard users interact with organizations and projects through [Tensorlake Cloud dashboard](https://cloud.tensorlake.ai), while developers can also use API keys for programmatic access. API keys operate at the project level with project-member permissions, making them ideal for integrating Tensorlake into applications and automated workflows.
[//]: # "TODO: create a diagram to explain the access control"
## Entities and Relationships
### Organizations
Organizations are the top-level entity in our system. Each organization can contain multiple projects and has its own set of members. Organizations implement a role-based access control system with two distinct roles: admin and member. These roles determine what actions users can perform within the organization and its projects.
### Projects
Projects exist within organizations and serve as containers for related resources that require similar access control. Unlike team-based structures, projects are designed to group resources that should be protected and accessed in a consistent manner. This resource-centric approach allows for fine-grained access control based on the nature of the resources rather than organizational hierarchy.
### API Keys
API Keys function exclusively at the project level and have the same permissions as project members. They can:
* Access project resources and data
* Make API calls within the project scope
* Cannot perform any administrative actions
API Keys are ideal for service accounts, automated processes, and integrations that need programmatic access to project resources.
## Membership Rules
Project membership is tied to organization membership. A user must first be a member of an organization before they can be added to any projects within that organization. This hierarchical structure ensures proper access control across your resources.
API keys have the same permissions as project members. This means they can access project resources but cannot perform administrative actions that are reserved for project admins.
## Roles and Permissions
The following table categorizes permissions by functional area to clearly show what each role can do:
### Organization Management Permissions
| Permission | Org Admin | Org Member | Project Admin | Project Member | API Key |
| -------------------------------- | :-------: | :--------: | :-----------: | :------------: | :-----: |
| Create new projects | ✅ | ❌ | ❌ | ❌ | ❌ |
| Invite users to organization | ✅ | ❌ | ❌ | ❌ | ❌ |
| View organization members | ✅ | ✅ | ❌ | ❌ | ❌ |
| Manage organization member roles | ✅ | ❌ | ❌ | ❌ | ❌ |
| Remove members from organization | ✅ | ❌ | ❌ | ❌ | ❌ |
### Project Access Control Permissions
| Permission | Org Admin | Org Member | Project Admin | Project Member | API Key |
| ------------------------------------- | :-------: | :--------: | :-----------: | :------------: | :-----: |
| Access all projects automatically | ✅ | ❌ | ❌ | ❌ | ❌ |
| Add organization members to a project | ✅ | ❌ | ✅ | ❌ | ❌ |
| Remove members from a project | ✅ | ❌ | ✅ | ❌ | ❌ |
| Change project member roles | ✅ | ❌ | ✅ | ❌ | ❌ |
| View projects they are members of | ✅ | ✅ | ✅ | ✅ | N/A |
### Resource Access Permissions
| Permission | Org Admin | Org Member | Project Admin | Project Member | API Key |
| ----------------------------- | :-------: | :--------: | :-----------: | :------------: | :-----: |
| View project resources\* | ✅ | ❌ | ✅ | ✅ | ✅ |
| Manage project resources\* | ✅ | ❌ | ✅ | ❌ | ❌ |
| Create API keys for a project | ✅ | ❌ | ✅ | ❌ | ❌ |
| Create Webhooks for a project | ✅ | ❌ | ✅ | ❌ | ❌ |
\*Project resources include Files, Datasets, and Webhooks. API Keys are specific to a project *and* user.
### Organization Roles in Detail
#### Organization Admin
Organization admins have complete control over the organization. They have full access to all projects within the organization, regardless of whether they are explicitly added as project members. Organization admins are the only users who can create new projects, invite users to join the organization, manage the roles of organization members, and remove members from the organization. Org admins can also [configure and enforce SSO](/platform/sso) for the organization.
#### Organization Member
Organization members have limited access within the organization. They can view the member list of the organization but can only access projects to which they have been explicitly added. Their permissions within accessible projects are determined by their project role.
### Project Roles in Detail
#### Project Admin
Project admins have management capabilities within their specific project. They can add existing organization members to their project, remove members from the project, and change the roles of project members. However, project admins cannot invite new users to the organization. This capability is reserved for organization admins.
#### Project Member
Project members have basic access to the project resources according to the system's permission model. They can view and interact with the project but cannot modify membership or roles.
## Invitation Process
User invitations can only be created by organization admins. When creating an invitation, the admin specifies the invitee's email address, organization role, and a default project and project role.
Upon invitation creation, an email is sent to the invitee with a unique link. After the invitee authenticates and accepts the invitation, the system verifies that the account email matches the invitation email. Once verified, the user is added to the organization with the specified role and to the default project contained in the invitation.
Invitations expire 7 days after creation. The invitation can only be accepted if the account accepting it has the same email as the invitation.
## Usage Guidelines
Projects should be used strategically to group resources that require similar access control patterns. Rather than organizing by teams or departments, consider organizing projects based on resource types, security requirements, or functional boundaries.
Consider the following best practices:
* Create projects based on resource sensitivity and access requirements
* Group resources that are commonly accessed together in the same project
* Use projects to implement the principle of least privilege by limiting access to only necessary resources
* Regularly audit project membership and permissions
* Rotate API keys periodically for enhanced security
## Frequently Asked Questions
Organization roles (admin/member) control access to organization-wide functions like creating projects and inviting users. Project roles (admin/member) control access to specific project resources and project-level management.
Yes, your organization role and project roles are independent. An organization member can be a project admin for specific projects they're added to.
Only organization admins can invite new users. Go to your organization settings and create an invitation with the user's email, organization role, and default project assignment.
No, API keys have the same permissions as project members. They can access project resources but cannot manage users, create projects, or perform administrative functions.
Invitations expire after 7 days. If expired, an organization admin will need to create a new invitation for the user.
Yes, as long as there is one Organization admin, other admins can be removed or changed to be a member, regardless of if they made the Organization.
Organization admins automatically have access to all projects. Organization members can only see and access projects they've been explicitly added to. To find out which projects
you have acces to, go to the organization and click on the dropdown menu to select a project. You can also get a full list by going to
`https://cloud.tensorlake.ai/organizations/[YOUR_ORG_ID]]/projects`.
Yes, both project admins and project members can create API keys for their projects. Only organization admins and project admins can manage other aspects of projects.
Organize projects based on resource sensitivity and access requirements rather than team structure. Group resources that need similar access controls and are commonly used together.
Datasets, files, API keys, and Webhooks are organized by project.
Regularly review project membership, especially when team members change roles or leave. Also rotate API keys periodically for enhanced security.
# Authentication
Source: https://docs.tensorlake.ai/platform/authentication
Learn how to make API requests to the Tensorlake APIs
## Tensorlake Account
You need to have a Tensorlake Cloud account to make API requests if you're using the Python SDK, the `tensorlake` npm package, a generated TypeScript client, or directly
calling the REST API. You can create an account on [cloud.tensorlake.ai](https://cloud.tensorlake.ai).
## API keys
API keys are project-specific credentials that allow programmatic access to resources within a project. Each API key exists
solely within the context of its project and has the [same permissions as a project member](/platform/access-control#project-access-control-permissions).
API keys cannot have organization-level permissions.
#### Creating API keys
1. Go to the [Tensorlake Dashboard](https://cloud.tensorlake.ai)
2. Select the project to make API calls against.
3. Create an API key.
![Create API key]()
Every tensorlake API key starts with `tl_apiKey_*`.
## Tensorlake Python SDK
The [Tensorlake SDK](https://github.com/tensorlakeai/tensorlake) leverages API keys for authentication.
For example:
```python your_app.py theme={null}
from tensorlake import Sandbox
API_KEY = "tl_apiKey_xxxx"
sandbox = Sandbox.create(api_key=API_KEY)
```
## TypeScript
For sandbox and cloud/application APIs, use the official `tensorlake` npm package.
```bash theme={null}
npm install tensorlake
export TENSORLAKE_API_KEY=your-api-key-here
```
```ts theme={null}
import { Sandbox } from "tensorlake";
const sandbox = Sandbox.create({
apiKey: process.env.TENSORLAKE_API_KEY,
});
```
These examples assume Node.js 18+, Bun, or Deno so `fetch` is available globally.
The npm package covers sandboxes plus cloud/application APIs. See [Sandboxes](/sandboxes/introduction) to get started.
## REST API
REST API requests needs to include the API key in the header as a Bearer Token.
For example, to make a request to the Sandboxes API, you would use the following curl command:
```bash theme={null}
curl --request POST \
--url https://api.tensorlake.ai/sandboxes \
--header 'Authorization: Bearer ' \
--header 'Content-Type: application/json'
```
For enterprise SSO, see [Single Sign-On (SSO)](/platform/sso).
## Frequently Asked Questions
You can regenerate your API key from the Tensorlake Dashboard. Go to your project settings, find the API keys section, and create a new key. Remember to update your applications with the new key.
No, API keys are project-specific. Each API key only works within the context
of the project where it was created. You'll need separate API keys for each
project.
API keys have the same permissions as a project member. They cannot have
organization-level permissions and are limited to project-specific operations.
Create a new API key first, update your applications to use the new key, then
delete the old key from the dashboard. This ensures no downtime during
rotation.
Immediately delete the compromised API key from the Tensorlake Dashboard and
generate a new one. Update all applications using the old key as soon as
possible.
API keys do not have explicit expiration dates. Each API key will remain
active until it is deleted.
This usually means your API key is invalid, was deleted, or is not properly
included in the Authorization header as a Bearer token. Verify your key exists
in the project you expect on the Tensorlake Dashboard, then verify format and
header structure.
Use environment variables or secure credential management systems. Never
hardcode API keys in your source code or commit them to version control.
API keys are for programmatic access and machine-to-machine communication,
while user authentication is for interactive dashboard access. API keys don't
expire with user sessions.
API keys inherit project member permissions.
# Billing
Source: https://docs.tensorlake.ai/platform/billing
Tensorlake Cloud uses usage-based billing. See the Billing page in your dashboard for current usage and invoices.
Billing in Tensorlake Cloud is usage-based. For detailed information, please refer to the Billing page.
![Billing page]()
# EU Endpoints
Source: https://docs.tensorlake.ai/platform/eu-data-residency
Use Tensorlake's EU endpoints for data residency and compute in Europe.
Tensorlake APIs are available in the EU region to provide data residency and compute in Europe.
## EU HTTP Endpoint
EU HTTP Endpoint is `https://api.eu.tensorlake.ai/`
## Workflows API
```python theme={null}
from tensorlake.functions_sdk import Graph
graph = Graph(name="my_graph", region=Region.EU)
```
## API Keys and Webhooks
The same API key can be used for both US and EU regions.
Webhooks are supported in both regions.
# Security Policies
Source: https://docs.tensorlake.ai/platform/security
Tensorlake's data storage, encryption, and compliance practices for enterprise customers in healthcare, financial services, legal, and government.
At Tensorlake, we take data security and privacy extremely seriously. We serve customers across healthcare, financial services,
legal, and government sectors who entrust us with high-stakes personally identifiable information (PII) and mission-critical data.
We understand that protecting this sensitive information isn't just important. It's essential to our customers' operations and
regulatory compliance. We have implemented robust, enterprise-grade security measures to ensure the highest level of protection.
This report outlines our data storage practices, encryption protocols, and compliance adherence.
## Data Storage
The data that may be stored on Tensorlake includes the files and state inside your sandboxes, snapshots, and the inputs and outputs of your workflows. Below are the default policies around data storage.
There are options for Hybrid and Fully-Disconnected On Prem usage of Tensorlake. Contact us at [support@tensorlake.ai](mailto:support@tensorlake.ai)
if you have to ensure your data never leaves your servers.
1. **Sandbox data**: Files, filesystem changes, and snapshots created inside your sandboxes are stored in accordance with our data storage policy below.
2. **Workflow data**: The inputs and outputs of your workflow runs are stored in accordance with our data storage policy below.
### Storage Policies
1. **Storage Location**: We utilize Amazon Web Services (AWS) S3 for storing data. Data is encrypted at rest and in transit.
2. **Access Permissions**: Access to AWS S3 storage is strictly limited to the internal services that operate on your behalf. This ensures that only authorized and authenticated processes can interact with the stored data, minimizing the risk of unauthorized access.
3. **Data Retention**: For all users, you can delete your data from our servers at any time using our APIs.
4. **Data Usage**: For all users, we never use any of your data for training purposes. We respect the privacy of our customers and ensure only they have access to the data from their requests.
## Deleting Your Data
While your data is stored securely in accordance with our storage policies outlined above, we understand that you may want to remove specific data at any time. If you need to request complete data deletion and/or access audit logs from Tensorlake, please contact us at [support@tensorlake.ai](mailto:support@tensorlake.ai) .
1. **Sandbox Deletion**: When you delete a sandbox, its files and state are permanently removed from our storage and cannot be recovered. Snapshots you created are retained until you delete them.
2. **Workflow Data Deletion**: You can delete the inputs and outputs associated with your workflow runs. Once deleted, the data is permanently removed and cannot be recovered.
3. **Immediate Deletion**: When you request deletion, the data is immediately removed from our active systems. This ensures that you maintain full control over your data lifecycle.
4. **API Access**: Data deletion can be performed through our API endpoints, allowing you to integrate data management into your workflows and compliance processes.
Whether you need to comply with data retention policies, respond to data subject requests, or simply manage your storage usage, you have the tools to delete your data whenever needed.
## Encryption
1. **Encryption at Rest**: All data stored in AWS S3 is encrypted at rest using industry-standard encryption algorithms. This means that even if unauthorized individuals were to gain access to the stored data, they would not be able to decipher it without the proper encryption keys.
2. **Encryption in Transit**: We employ encryption protocols to protect data in transit. All communication between our systems and the data storage is conducted over secure channels using encryption mechanisms such as SSL/TLS. This ensures that data remains confidential and tamper-proof during transmission.
If you have any further questions or require additional information regarding our security practices, please don't hesitate to reach out to [support@tensorlake.ai](mailto:support@tensorlake.ai).
### List of Authorized Subprocessors
| Company | Description | Country (where subprocessing takes place) |
| ------------------------------- | ----------------------- | ----------------------------------------- |
| Amazon Web Services, Inc. (AWS) | Cloud Infrastructure | United States, EU |
| OpenAI, LLC | Artificial Intelligence | United States, EU |
| Anthropic PBC | Artificial Intelligence | United States, EU |
| Datadog | Error Monitoring | United States |
| PostHog, Inc. | Product Analytics | United States |
| Google Cloud | Cloud Infrastructure | United States, EU |
| Microsoft Azure | Cloud Infrastructure | United States, EU |
| Lambda Labs. | Cloud Infrastructure | United States, EU |
# Single Sign-On (SSO)
Source: https://docs.tensorlake.ai/platform/sso
Configure and enforce SSO for your organization using OIDC or SAML 2.0 identity providers.
Single Sign-On (SSO) lets your team sign in to Tensorlake through your company's identity provider (IdP). Once configured, members authenticate with your IdP instead of managing separate Tensorlake credentials.
## Prerequisites
* You must be an [organization admin](/platform/access-control#organization-admin).
* SSO access must be enabled for your organization. If you don't have access, request it from the SSO settings page in the dashboard.
## Setup
Navigate to **Organization Settings > SSO** and click **Configure SSO Connection**.
Tensorlake supports two protocols:
* **OIDC (OpenID Connect)**: recommended for providers like Google Workspace, Okta, and Auth0.
* **SAML 2.0**: supported for providers like Azure AD (Entra ID), OneLogin, and other SAML-compatible IdPs.
Provide the following details:
| Field | Description |
| -------------------------- | --------------------------------------------------------------------------------------------------------------- |
| **Domain** | Your organization's email domain (e.g. `yourcompany.com`). Users with this domain will be directed to your IdP. |
| **Issuer URL** | The issuer or entity ID from your IdP. |
| **Client ID** | The application/client ID assigned by your IdP. |
| **Client Secret** | The client secret from your IdP (OIDC only). |
| **Authorization Endpoint** | The URL where users are sent to authenticate (OIDC). |
| **Token Endpoint** | The URL used to exchange authorization codes for tokens (OIDC). |
| **ACS URL / SSO URL** | The Assertion Consumer Service URL (SAML). Provided by Tensorlake. |
| **Certificate** | The X.509 signing certificate from your IdP (SAML). |
**Attribute mapping:** Ensure your IdP sends at minimum the user's email address. Name attributes (first name, last name) are recommended for a complete profile.
After saving your configuration, test the connection by performing a test login.
1. Click **Test Connection** in the SSO settings.
2. You will be redirected to your IdP to authenticate.
3. After successful authentication, you are redirected back to Tensorlake and the provider is marked as **Verified**.
SSO enforcement cannot be enabled until you have completed a successful test login. Enforcing an untested configuration could lock users out of your organization.
SSO enforcement requires all organization members to sign in through your IdP. When enabled, password-based login is disabled for all members. The only way to sign in is through the IdP.
To enable enforcement:
1. Designate at least one organization admin as a **bypass user**. This is required before enforcement can be enabled.
2. Toggle **Enforce SSO** in the SSO settings.
**Bypass users** are organization admins who retain password-based login for emergency recovery (for example, if your IdP goes down and you need to access the dashboard to disable enforcement). Only organization admins can be designated as bypass users.
Enabling SSO enforcement invalidates all existing sessions for the organization. All members (except bypass users) will be signed out and must re-authenticate through the IdP.
Only organization admins can enable or disable SSO enforcement and manage bypass users.
## How SSO login works
When SSO is configured for your organization, the login flow works as follows:
1. A user enters their email on the Tensorlake login page.
2. Tensorlake checks whether the email domain has an SSO provider configured.
3. If SSO is configured and enforced, the user is redirected to the IdP with an `SSO_REQUIRED` response. Password-based login is not available.
4. If SSO is configured but not enforced, the user can choose to sign in with SSO or with their Tensorlake password.
5. After authenticating with the IdP, the user is redirected back to Tensorlake and signed in.
New users who sign in via SSO on their first login are automatically provisioned with a Tensorlake account. Users who previously signed in with email OTP may need to be [invited to the organization](/platform/access-control#invitation-process) before SSO login will work for them.
## Frequently Asked Questions
Tensorlake supports any IdP that implements OIDC or SAML 2.0. Common providers include Okta, Azure AD (Entra ID), Google Workspace, OneLogin, and Auth0.
Each organization supports one SSO provider at a time. If you need to switch providers, update the existing SSO configuration with the new provider's details.
All members must authenticate through your IdP. Password-based login is disabled for the entire organization, except for designated bypass users.
Only organization admins can be designated as bypass users. At least one bypass user is required before SSO enforcement can be enabled. Bypass users retain password-based login solely for emergency recovery, such as disabling enforcement if your IdP becomes unavailable.
API keys are not affected by SSO enforcement. Existing API keys continue to work regardless of SSO settings. API keys authenticate directly with the Tensorlake API and do not go through the IdP login flow.
If your IdP is unavailable, members will not be able to sign in. A bypass user (org admin with password-based login retained) can sign in and disable enforcement until the IdP is restored.
Yes. SSO is not enforced until you explicitly enable enforcement. During setup and testing, all users can continue to sign in with their existing credentials.
No. Existing organization members continue to have access. They will simply be redirected to the IdP on their next login if enforcement is enabled.
# Configure Webhooks
Source: https://docs.tensorlake.ai/platform/webhooks/configuration
Create and manage a project webhook destination, HTTPS endpoint, event subscriptions, and delivery status.
## Prerequisites
Before creating a webhook, you need:
* a Tensorlake Cloud organization and project;
* organization administrator or project administrator access; and
* a public HTTPS endpoint that accepts `POST` requests.
## Create a destination
In [Tensorlake Cloud](https://cloud.tensorlake.ai), select the project that
produces the events, then select **Webhooks**.
Select **Create webhook** and provide:
* **Webhook name**: a name that identifies the receiver or environment.
* **Endpoint URL**: an absolute HTTPS URL without embedded credentials.
Select one or more sandbox, application, or application-request lifecycle
events. The destination receives only the selected event types from this
project.
Select **Create webhook**, then open the new destination to copy its signing
secret and send a test event.
Treat the signing secret like a password. Store it in your secret manager and
never commit it to source control or log it.
## Edit or pause delivery
Open a destination and select **Edit webhook** to change its name, endpoint URL,
subscribed events, or delivery status.
Turn off **Delivery enabled** to pause attempts to that destination without
deleting its configuration. Re-enable delivery when the receiver is ready.
## Delete a destination
Delete a destination when it should no longer receive events. Deleting a
webhook removes that endpoint configuration.
# Webhook Event Reference
Source: https://docs.tensorlake.ai/platform/webhooks/events
Event names, delivery envelope, payload fields, lifecycle semantics, and examples for Tensorlake webhooks.
Every Tensorlake lifecycle webhook is a versioned JSON object. The common
envelope identifies the event, project, producer ordering domain, and
event-specific data.
## Common envelope
| Field | Type | Description |
| ----------------- | ------- | -------------------------------------------------------------------------------------- |
| `event_id` | string | Stable event identifier beginning with `evt_`. Use it as the deduplication key. |
| `event_type` | string | One supported lifecycle event name. |
| `event_version` | integer | Payload schema version. The current version is `1`. |
| `occurred_at` | string | RFC 3339 UTC time when the lifecycle transition was recorded. |
| `project_id` | string | Public ID of the project that produced the event. |
| `source_id` | string | Opaque identifier for the producer ordering domain. |
| `source_revision` | string | Unsigned decimal source revision. Keep it as a string to avoid integer precision loss. |
| `source_ordinal` | integer | Zero-based order when one source revision produces multiple events. |
| `data` | object | Event-specific sandbox, application, or request data. |
`event_id` remains the same across retries. Delivery order is not guaranteed.
For related events from the same `source_id`, compare `source_revision`
numerically and then `source_ordinal` when you need source order.
## Event catalog
### Sandbox events
| Event type | Emitted when |
| ------------------------------- | -------------------------------------------------------------------------------------- |
| `tensorlake.sandbox.created` | A sandbox is persisted for the first time. |
| `tensorlake.sandbox.running` | A sandbox enters the running state. |
| `tensorlake.sandbox.suspended` | A sandbox enters the suspended state. |
| `tensorlake.sandbox.resumed` | A suspended sandbox begins a new generation. Its public status is initially `pending`. |
| `tensorlake.sandbox.terminated` | A sandbox terminates without a failure outcome. |
| `tensorlake.sandbox.failed` | A sandbox terminates with a failure outcome. |
Sandbox event `data` contains `previous_status` and a `sandbox` object.
| Sandbox field | Type | Description |
| ---------------- | -------------- | ----------------------------------------------------------------- |
| `id` | string | Sandbox ID. |
| `name` | string or null | Name of a named sandbox, when present. |
| `type` | string | `named` or `ephemeral`. |
| `image` | string or null | Sandbox image, when present. |
| `generation_id` | integer | Sandbox generation, starting at `1` and increasing on resume. |
| `status` | string | `pending`, `running`, `suspended`, or `terminated`. |
| `outcome` | string or null | `success`, `failure`, or `null` while no terminal outcome exists. |
| `outcome_reason` | string or null | Allowlisted terminal reason, when available. |
| `snapshot_id` | string or null | Snapshot associated with the sandbox, when present. |
| `created_at` | string | RFC 3339 sandbox creation time. |
| `resources` | object | Requested CPU, memory, disk, and GPU resources. |
```json Sandbox failed event theme={null}
{
"event_id": "evt_019f95fb-7f33-7000-8000-000000000006",
"event_type": "tensorlake.sandbox.failed",
"event_version": 1,
"occurred_at": "2026-07-24T21:15:15.123Z",
"project_id": "project_123",
"source_id": "src_01k0example",
"source_revision": "81247",
"source_ordinal": 0,
"data": {
"previous_status": "running",
"sandbox": {
"id": "9q2j1f4n6y8x0r3k5m7pa",
"name": "build-runner",
"type": "named",
"image": "tensorlake/default",
"generation_id": 2,
"status": "terminated",
"outcome": "failure",
"outcome_reason": "out_of_memory",
"snapshot_id": null,
"created_at": "2026-07-24T20:10:00Z",
"resources": {
"cpus": 2,
"memory_mb": 4096,
"disk_mb": 20480,
"gpu_count": 0,
"gpu_model": null
}
}
}
}
```
Possible sandbox failure reasons are `unknown`, `user_terminated`, `timeout`,
`internal_error`, `constraint_unsatisfiable`, `executor_removed`,
`out_of_memory`, `container_startup_failed`, `pool_deleted`,
`container_terminated`, `bad_image`, `image_not_found`,
`container_unhealthy`, `function_error`, `function_timeout`,
`function_cancelled`, `desired_state_removed`, `process_crash`, and
`executor_drained`. A successful termination can report `unknown`,
`user_terminated`, or `timeout`. `outcome_reason` can be `null` when no safe
reason is available.
### Application events
| Event type | Emitted when |
| -------------------------------- | ----------------------------------------------------------------------------- |
| `tensorlake.application.created` | An application is persisted for the first time. |
| `tensorlake.application.updated` | A deployment, metadata, capability, or application state change is persisted. |
| `tensorlake.application.deleted` | An existing application is deleted. |
Application event `data` contains `previous_version`, `previous_state`, and an
`application` object. The previous fields are `null` for a creation event.
| Application field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------- |
| `name` | string | Application name. |
| `version` | string | Application version. |
| `state` | string | `active` or `disabled`. |
| `disabled_reason` | string or null | Reason the application is disabled, when present. |
| `created_at` | string | RFC 3339 application creation time. |
```json Application updated event theme={null}
{
"event_id": "evt_019f95fb-8703-7000-8000-000000000008",
"event_type": "tensorlake.application.updated",
"event_version": 1,
"occurred_at": "2026-07-24T21:15:17.123Z",
"project_id": "project_123",
"source_id": "src_01k0example",
"source_revision": "81249",
"source_ordinal": 0,
"data": {
"previous_version": "2026-07-23",
"previous_state": "active",
"application": {
"name": "invoice-parser",
"version": "2026-07-24",
"state": "active",
"disabled_reason": null,
"created_at": "2026-04-11T18:12:00Z"
}
}
}
```
### Application-request events
| Event type | Emitted when |
| ------------------------------------------ | ------------------------------------------------------- |
| `tensorlake.application.request.created` | An application request is created. |
| `tensorlake.application.request.completed` | An application request completes successfully. |
| `tensorlake.application.request.failed` | An application request reaches a failed terminal state. |
Application-request event `data` contains an `application` identity and a
`request` object.
| Request field | Type | Description |
| ---------------- | -------------- | ---------------------------------------------------------- |
| `id` | string | Application request ID. |
| `status` | string | `created`, `completed`, or `failed`. |
| `failure_reason` | string or null | Allowlisted reason for a failed request. |
| `created_at` | string | RFC 3339 request creation time. |
| `finished_at` | string or null | RFC 3339 terminal time; `null` until the request finishes. |
```json Application request completed event theme={null}
{
"event_id": "evt_019f9604-13d0-7000-8000-00000000000b",
"event_type": "tensorlake.application.request.completed",
"event_version": 1,
"occurred_at": "2026-07-24T21:24:37.456Z",
"project_id": "project_123",
"source_id": "src_01k0example",
"source_revision": "81252",
"source_ordinal": 0,
"data": {
"application": {
"name": "invoice-parser",
"version": "2026-07-24"
},
"request": {
"id": "req_01k0example",
"status": "completed",
"failure_reason": null,
"created_at": "2026-07-24T21:24:35.101Z",
"finished_at": "2026-07-24T21:24:37.456Z"
}
}
}
```
Possible application-request failure reasons are `unknown`, `internal_error`,
`function_error`, `function_timeout`, `request_error`,
`constraint_unsatisfiable`, `cancelled`, and `out_of_memory`.
# Webhooks
Source: https://docs.tensorlake.ai/platform/webhooks/overview
Receive signed, project-scoped notifications for sandbox, application, and application-request lifecycle events.
Tensorlake webhooks send signed HTTPS notifications when sandboxes,
applications, and application requests change state. Use them to react to
lifecycle changes without polling an API.
Each webhook belongs to one project. It receives only events produced by that
project and only the event types selected for that destination.
## Supported events
| Resource | Events |
| ------------------- | ------------------------------------------------------------ |
| Sandbox | Created, running, suspended, resumed, terminated, and failed |
| Application | Created, updated, and deleted |
| Application request | Created, completed, and failed |
See the [event reference](./events) for event names, payload
fields, and examples.
## Create a webhook
Your endpoint must accept `POST` requests over HTTPS. Before processing an
event, verify its Svix signature using the destination's signing secret.
In [Tensorlake Cloud](https://cloud.tensorlake.ai), select an organization
and project, then select **Webhooks** in the project navigation.
Select **Create webhook**, enter a name and absolute HTTPS endpoint URL,
then select at least one lifecycle event.
Open the webhook details page, copy the signing secret, and send a
synthetic event with **Test webhook**.
Only organization administrators and project administrators can create or
edit webhook destinations. Project members can view destinations for projects
they can access.
## Delivery contract
Webhook delivery is **at least once** and is not ordered. The same event can be
delivered more than once, and a later lifecycle event can arrive before an
earlier one.
Design your receiver to:
* verify the signature before parsing or processing the payload;
* deduplicate events by the stable `event_id`;
* use `source_revision` and `source_ordinal` when ordering events from the same
producer is necessary;
* durably record accepted work before returning a `2xx` response; and
* process expensive or failure-prone work asynchronously.
Tensorlake uses Svix to sign messages and deliver them to your endpoint. Svix
retries failed endpoint attempts. A successful response from your endpoint
acknowledges that delivery attempt; it does not change the underlying
Tensorlake resource.
## Next steps
* [Configure and manage a destination](./configuration)
* [Understand event payloads](./events)
* [Verify webhook signatures](./signature-verification)
* [Send a test event](./testing)
# Verify Webhook Signatures
Source: https://docs.tensorlake.ai/platform/webhooks/signature-verification
Verify the Svix signature on every Tensorlake webhook before accepting or processing its payload.
Every Tensorlake webhook delivery is signed by Svix with the destination's
signing secret. Make sure you verify the signature before parsing or processing the event.
## Get the signing secret
1. Open your project in [Tensorlake Cloud](https://cloud.tensorlake.ai).
2. Select **Webhooks** and open the destination.
3. Under **Signing secret**, copy the value beginning with `whsec_`.
4. Store the secret in your application's secret manager.
## Verify a request
Svix includes these headers with each delivery:
* `svix-id`
* `svix-timestamp`
* `svix-signature`
You can use an [official Svix library](https://docs.svix.com/receiving/verifying-payloads/how)
to verify the **raw request body** and the request headers with your signing
secret.
Reject requests with missing or invalid signature headers. After successful
verification, parse the returned payload as the appropriate
[lifecycle event](./events).
For implementations that cannot use an official library, follow Svix's
[manual verification procedure](https://docs.svix.com/receiving/verifying-payloads/how-manual).
# Test a Webhook
Source: https://docs.tensorlake.ai/platform/webhooks/testing
Send a signed synthetic lifecycle event from Tensorlake Cloud to validate a webhook destination.
Test a destination before relying on it for production lifecycle events.
## Send a test event
In [Tensorlake Cloud](https://cloud.tensorlake.ai), select your project,
select **Webhooks**, then open the destination you want to test.
Under **Test delivery**, select one of the event types subscribed by this
destination.
Select **Test webhook**. Tensorlake sends a signed synthetic payload to the
configured endpoint.
Verify that your endpoint validates the signature, records the `event_id`,
parses the expected [event shape](./events), and returns a
`2xx` response.
A test delivery uses a valid example payload and the destination's normal Svix
signing and endpoint-delivery path. It does not create, update, suspend,
terminate, or delete a real Tensorlake resource.
## Troubleshooting
| Symptom | Check |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Signature verification fails | Verify with the raw request body, all three Svix headers, and this destination's signing secret. |
| Endpoint returns `404` | Confirm the complete URL path. Paths are case-sensitive. |
| Endpoint returns `403` | Confirm the route accepts server-to-server `POST` requests and does not require browser session or CSRF credentials. |
| Event is delivered repeatedly | Return `2xx` after durable acceptance and deduplicate by `event_id`. |
| An expected event never arrives | Confirm delivery is enabled and the event type is selected for this project-scoped destination. |
# Agentic Autoresearch Loop
Source: https://docs.tensorlake.ai/sandboxes/agentic-autoresearch
Autonomously improve an ML training script overnight using an LLM agent that proposes code modifications, races them in parallel sandboxes, and hill-climbs toward lower validation loss.
Automate ML research iteration with an LLM agent that reads your training script, proposes targeted code changes, and validates each change by running it in an isolated sandbox. Inspired by [Karpathy's autoresearch](https://github.com/karpathy/autoresearch) (March 2026), the loop runs unattended. Each accepted modification becomes the new baseline, and the agent builds on what it has already learned.
## How it works
1. **Calibrate**: Run the baseline training script in a sandbox to establish a starting validation loss.
2. **Propose**: The agent reads the current best script and experiment history, then proposes *N* candidate modifications (with increasing temperature for diversity).
3. **Race**: All *N* candidates run in parallel TensorLake sandboxes for a fixed step budget.
4. **Evaluate**: Parse `val_loss` from each sandbox's stdout. The candidate with the lowest loss wins the round.
5. **Hill-climb**: Accept the winner only if it beats the current best. Update the baseline script and history.
6. **Repeat**: Loop until the iteration budget is exhausted.
### Why sandboxes are required
The agent emits complete, self-contained Python scripts. Running untrusted LLM-generated training code in your host process would be unsafe. The model could write arbitrary filesystem operations or import unexpected packages. Each candidate runs in an isolated sandbox with a fixed memory ceiling and is killed automatically when the step budget completes.
***
## Prerequisites
```bash theme={null}
pip install tensorlake openai rich python-dotenv
```
Create a `.env` file in your project root:
```
TENSORLAKE_API_KEY="your-api-key-here"
OPENAI_API_KEY="your-openai-key-here"
```
***
## TypeScript SDK starter
The Node.js version follows the same core loop: propose candidates, race them in parallel sandboxes, parse `val_loss`, and keep the winner.
```typescript theme={null}
import { Sandbox } from "tensorlake";
async function evaluateCandidate(script: string) {
const sandbox = await Sandbox.create({
cpus: 2.0,
memoryMb: 4096,
timeoutSecs: 900,
allowInternetAccess: false,
});
try {
await sandbox.writeFile(
"/workspace/train.py",
new TextEncoder().encode(script),
);
const result = await sandbox.run("python", {
args: ["/workspace/train.py"],
workingDir: "/workspace",
timeout: 900,
});
const match = result.stdout.match(/val_loss:\s*([0-9.]+)/);
return {
valLoss: match ? Number(match[1]) : Number.POSITIVE_INFINITY,
stdout: result.stdout,
stderr: result.stderr,
};
} finally {
await sandbox.terminate();
}
}
const candidates = [
"print('val_loss: 1.2345')",
"print('val_loss: 1.1021')",
];
const results = await Promise.all(candidates.map(evaluateCandidate));
const winner = results.reduce((best, cur) =>
cur.valLoss < best.valLoss ? cur : best,
);
console.log(winner);
client.close();
```
Use the OpenAI response to generate each candidate script, then keep appending accepted experiments to your history exactly like the Python version below.
***
## Full example
Pass `--smoke` for a fast proof-of-concept run (3 iterations, 2 candidates, 150 training steps, \~5 minutes). The full run uses 8 iterations, 3 candidates, and 300 steps (\~20 minutes).
````python theme={null}
"""
Karpathy Autoresearch Loop + TensorLake Sandboxes
==================================================
Inspired by github.com/karpathy/autoresearch (April 2026, 64k⭐).
The "Karpathy Loop":
1. Give an AI agent a training script and a plain-English program.md
2. Agent proposes one targeted code modification per iteration
3. Run the modified script in isolation for a fixed step budget
4. If val_loss improves → accept, update the baseline
5. Repeat overnight → hundreds of validated improvements
TensorLake sandboxes are the right tool here:
• Modified training code is untrusted (the agent could emit anything)
• Multiple candidate modifications can race in parallel sandboxes
• Each sandbox is killed after the time budget: no runaway experiments
• The host process never imports/executes model weights or agent code
This example:
State : current best train.py + full experiment history
Action : LLM proposes one self-contained code modification
Sandbox : TensorLake runs the modified script for STEPS training steps
Reward : Δval_loss = best_val_loss − new_val_loss (positive = improvement)
Update : Greedy hill-climbing (accept if reward > 0)
Parallelism: CANDIDATES sandboxes race each iteration (ThreadPoolExecutor)
Smoke : --smoke → 3 iters, 2 candidates, 150 steps/run (~5 min)
Full : 8 iters, 3 candidates, 300 steps/run (~20 min)
"""
from dotenv import load_dotenv
load_dotenv()
import re
import sys
import time
from dataclasses import dataclass, field
from typing import List, Optional
from openai import OpenAI
from tensorlake.sandbox import Sandbox
from tensorlake.applications import application, function
from rich.console import Console
from rich.panel import Panel
from rich.table import Table
from rich.rule import Rule
from rich import box
console = Console()
SMOKE = "--smoke" in sys.argv
# ─── program.md: plain-English guidance for the agent ───────────────────────
# This is what Karpathy calls "the human's job": describe the search space.
PROGRAM_GUIDANCE = """\
You are an ML research agent optimising a character-level MLP language model
trained on a small text corpus using numpy only (no torch/tensorflow).
The training script defines these tunable constants near the top:
CTX (context window / n-gram size, int)
HIDDEN (hidden layer size, int)
LR (learning rate, float)
BATCH (batch size, int)
WDECAY (L2 weight decay, float)
STEPS (DO NOT CHANGE - fixed budget)
Good things to try:
• Learning rate: sweep 1e-4 → 0.1
• Learning rate decay: multiply LR by 0.999 each step (add near opt.step)
• Hidden size: 32 / 64 / 128 / 256
• Context window CTX: 2 / 4 / 8
• Weight decay WDECAY: 0 → 1e-4
• Activation: replace np.tanh with np.maximum(0, x) (ReLU) or np.clip(x,0,None)
• Initialization scale: change 0.01 to 0.1 or use He/Xavier init
• Add a second hidden layer (W3, b3) with size HIDDEN//2
• Momentum: track velocity vectors, apply SGD+momentum
• Batch size: 16 / 32 / 64
Constraints:
• numpy only: do not import torch, tensorflow, sklearn
• STEPS must stay unchanged
• Output format: last printed line must be val_loss: X.XXXX
"""
# ─── Baseline training script ─────────────────────────────────────────────────
# ~130 K-param nano-GPT on a small public-domain text.
# Runs in ~15 s on CPU (150 steps) / ~30 s (300 steps).
BASELINE_SCRIPT = '''\
import subprocess, sys
subprocess.run(["python3","-m","pip","install","numpy","-q","--target","/tmp/pkgs"],
capture_output=True, check=False)
sys.path.insert(0, "/tmp/pkgs")
import numpy as np
np.random.seed(42)
# ── Corpus (opening of Alice in Wonderland, public domain) ──────────────────
TEXT = (
"Alice was beginning to get very tired of sitting by her sister on the bank,"
" and of having nothing to do: once or twice she had peeped into the book her"
" sister was reading, but it had no pictures or conversations in it, and what"
" is the use of a book thought Alice without pictures or conversations so she"
" was considering in her own mind as well as she could for the hot day made"
" her feel very sleepy and stupid whether the pleasure of making a daisy-chain"
" would be worth the trouble of getting up and picking the daisies when"
" suddenly a White Rabbit with pink eyes ran close by her there was nothing so"
" very remarkable in that nor did Alice think it so very much out of the way"
" to hear the Rabbit say to itself oh dear oh dear I shall be late when she"
" thought it over afterwards it occurred to her that she ought to have wondered"
" at this but at the time it all seemed quite natural but when the Rabbit"
" actually took a watch out of its waistcoat-pocket and looked at it and then"
" hurried on Alice started to her feet for it flashed across her mind that she"
" had never before seen a rabbit with either a waistcoat-pocket or a watch to"
" take out of it and burning with curiosity she ran across the field after it"
) * 4 # ~4 800 chars
# ── Tokeniser ────────────────────────────────────────────────────────────────
chars = sorted(set(TEXT))
vocab = len(chars)
stoi = {c: i for i, c in enumerate(chars)}
data = [stoi[c] for c in TEXT]
split = int(0.9 * len(data))
train_d, val_d = data[:split], data[split:]
# ── Hyperparameters (agent modifies these) ───────────────────────────────────
CTX = 4 # context window (n-gram)
HIDDEN = 64 # hidden layer size
LR = 0.05 # learning rate
BATCH = 32 # mini-batch size
WDECAY = 0.0 # L2 weight decay
STEPS = STEPS_PLACEHOLDER # fixed budget - do not change
# ── Parameters ───────────────────────────────────────────────────────────────
W1 = np.random.randn(vocab * CTX, HIDDEN) * 0.01
b1 = np.zeros(HIDDEN)
W2 = np.random.randn(HIDDEN, vocab) * 0.01
b2 = np.zeros(vocab)
def get_batch(d):
idx = np.random.randint(0, len(d) - CTX, BATCH)
X = np.zeros((BATCH, vocab * CTX))
for i, start in enumerate(idx):
for j in range(CTX):
X[i, j * vocab + d[start + j]] = 1.0
Y = np.array([d[i + CTX] for i in idx])
return X, Y
def forward(X):
H = np.tanh(X @ W1 + b1)
logits = H @ W2 + b2
logits -= logits.max(1, keepdims=True)
probs = np.exp(logits)
probs /= probs.sum(1, keepdims=True)
return H, probs
def ce_loss(probs, Y):
return -np.log(probs[np.arange(len(Y)), Y] + 1e-8).mean()
# ── Training loop ─────────────────────────────────────────────────────────────
for step in range(STEPS):
X, Y = get_batch(train_d)
H, probs = forward(X)
dl = probs.copy(); dl[np.arange(BATCH), Y] -= 1; dl /= BATCH
dW2 = H.T @ dl; db2 = dl.sum(0)
dH = dl @ W2.T * (1 - H**2)
dW1 = X.T @ dH; db1 = dH.sum(0)
W1 -= LR * (dW1 + WDECAY * W1)
b1 -= LR * db1
W2 -= LR * (dW2 + WDECAY * W2)
b2 -= LR * db2
# ── Evaluate ──────────────────────────────────────────────────────────────────
losses = [ce_loss(forward(get_batch(val_d)[0])[1], get_batch(val_d)[1]) for _ in range(30)]
print(f"val_loss: {np.mean(losses):.4f}")
'''
# ─── Data models ──────────────────────────────────────────────────────────────
@dataclass
class Experiment:
iteration: int
candidate: int
description: str
script: str
val_loss: Optional[float] = None
delta: Optional[float] = None # positive = improvement
accepted: bool = False
error: Optional[str] = None
@dataclass
class ResearchState:
best_script: str
best_val_loss: float = 999.0
history: List[Experiment] = field(default_factory=list)
def history_summary(self) -> str:
if not self.history:
return "No experiments yet."
lines = []
for e in self.history[-8:]: # last 8
status = "✓ ACCEPTED" if e.accepted else ("✗ error" if e.error else "✗ rejected")
vl = f"{e.val_loss:.4f}" if e.val_loss else "—"
d = f"Δ{e.delta:+.4f}" if e.delta is not None else ""
lines.append(f" [{status}] iter={e.iteration} val={vl} {d} - {e.description}")
return "\n".join(lines)
# ─── Agent: propose one code modification ────────────────────────────────────
def propose_modification(state: ResearchState, candidate_idx: int) -> tuple[str, str]:
"""Returns (description, modified_script)."""
client = OpenAI()
prompt = f"""{PROGRAM_GUIDANCE}
Current best val_loss: {state.best_val_loss:.4f}
Experiment history:
{state.history_summary()}
Current best script:
```python
{state.best_script}
```
Propose modification #{candidate_idx + 1} (make it different from recent attempts).
Return ONLY a JSON object with two keys:
"description": one sentence describing the change
"script": the complete modified Python script
No markdown fences around the JSON. Just the raw JSON object."""
resp = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
temperature=0.9 + candidate_idx * 0.1, # more exploration for later candidates
response_format={"type": "json_object"},
)
import json
data = json.loads(resp.choices[0].message.content)
return data["description"], data["script"]
# ─── TensorLake @function for map-reduce ──────────────────────────────────────────
@function()
def run_experiment_in_sandbox(exp_data: dict) -> dict:
"""Run exp.script in a TensorLake sandbox via map-reduce."""
import re
iteration = exp_data["iteration"]
candidate = exp_data["candidate"]
description = exp_data["description"]
script = exp_data["script"]
max_retries = 5
last_error = None
for attempt in range(max_retries):
try:
box = Sandbox.create(memory_mb=4096, timeout_secs=900)
ex = box.run("python3", ["-c", script], timeout=300)
stdout = (ex.stdout or "").strip()
stderr = (ex.stderr or "").strip()
m = re.search(r"val_loss:\s*([0-9.]+)", stdout)
if not m:
return {
"iteration": iteration,
"candidate": candidate,
"description": description,
"val_loss": None,
"error": (stderr or stdout)[:120] or "no val_loss in output"
}
return {
"iteration": iteration,
"candidate": candidate,
"description": description,
"val_loss": float(m.group(1)),
"error": None
}
except Exception as exc:
last_error = str(exc)[:150]
if attempt < max_retries - 1:
wait_time = 2 ** attempt
time.sleep(wait_time)
continue
return {
"iteration": iteration,
"candidate": candidate,
"description": description,
"val_loss": None,
"error": last_error
}
return {
"iteration": iteration,
"candidate": candidate,
"description": description,
"val_loss": None,
"error": last_error
}
# ─── Main autoresearch loop (TensorLake @application) ──────────────────────────────
@application()
@function()
def autoresearch(iterations: int = 8, candidates: int = 3):
steps = 150 if SMOKE else 300
console.print(Panel(
"[bold cyan]Karpathy Autoresearch Loop + TensorLake Map-Reduce[/bold cyan]\n\n"
"[dim]Inspired by github.com/karpathy/autoresearch (April 2026)\n\n"
"Loop:\n"
" 1. Agent reads current best script + experiment history\n"
" 2. Proposes CANDIDATES modifications (different temperatures)\n"
" 3. All CANDIDATES run in parallel via Tensorlake map-reduce\n"
" 4. Best val_loss wins; accepted if it beats the current best\n"
" 5. Accepted script becomes the new baseline\n\n"
"Reward = Δval_loss (positive = improvement)\n"
"Policy = GPT-4o prompted with program.md + experiment history\n"
"Update = greedy hill-climbing (accept if reward > 0)\n"
f"Mode = {'SMOKE (3 iters, 2 candidates, 150 steps)' if SMOKE else f'Full ({iterations} iters, {candidates} candidates, {steps} steps)'}[/dim]",
border_style="cyan",
))
# ── Calibrate baseline ───────────────────────────────────────────────────
baseline = BASELINE_SCRIPT.replace("STEPS_PLACEHOLDER", str(steps))
console.print(Rule("[yellow]Calibrating baseline[/yellow]", style="yellow"))
console.print("[dim]Running baseline script in sandbox to establish starting val_loss...[/dim]")
calib_result = run_experiment_in_sandbox({
"iteration": 0,
"candidate": 0,
"description": "baseline",
"script": baseline
})
if calib_result["error"] or calib_result["val_loss"] is None:
console.print(f"[red]Baseline failed: {calib_result['error']}[/red]")
return
calib_val_loss = calib_result["val_loss"]
state = ResearchState(best_script=baseline, best_val_loss=calib_val_loss)
console.print(f" Baseline val_loss: [bold yellow]{state.best_val_loss:.4f}[/bold yellow]\n")
try:
# ── Research iterations ──────────────────────────────────────────────────
for it in range(1, iterations + 1):
console.print(Rule(f"[cyan]Iteration {it}/{iterations}[/cyan]", style="cyan"))
console.print(f" [dim]Current best: {state.best_val_loss:.4f} "
f"Proposing {candidates} candidates (map-reduce)...[/dim]")
# Propose candidates (sequentially, they call the LLM)
proposals_data = []
for c in range(candidates):
desc, script = propose_modification(state, c)
proposals_data.append({
"iteration": it,
"candidate": c,
"description": desc,
"script": script
})
# Run all candidates in parallel using Tensorlake map-reduce
console.print(f" [dim]Executing {candidates} candidates in parallel (Tensorlake map-reduce)...[/dim]")
result_dicts = run_experiment_in_sandbox.map(proposals_data)
results = []
for res in result_dicts:
exp = Experiment(
iteration=res["iteration"],
candidate=res["candidate"],
description=res["description"],
script=proposals_data[res["candidate"]]["script"],
val_loss=res["val_loss"],
error=res["error"]
)
results.append(exp)
# Score & rank
valid = [r for r in results if r.val_loss is not None]
for r in valid:
r.delta = state.best_val_loss - r.val_loss # positive = improvement
valid.sort(key=lambda r: r.val_loss)
# Print iteration table
t = Table(box=box.SIMPLE, show_header=True, header_style="bold white")
t.add_column("C", width=3)
t.add_column("Modification", width=52)
t.add_column("val_loss", width=9, justify="right")
t.add_column("Δ", width=9, justify="right")
t.add_column("", width=3)
for r in sorted(results, key=lambda r: (r.val_loss or 999)):
if r.error:
t.add_row(str(r.candidate), r.description[:50], "—", "—",
f"[red]✗[/red]")
continue
delta_str = (f"[green]{r.delta:+.4f}[/green]" if r.delta and r.delta > 0
else f"[red]{r.delta:+.4f}[/red]" if r.delta
else "—")
t.add_row(str(r.candidate), r.description[:50],
f"{r.val_loss:.4f}", delta_str, "")
console.print(t)
# Accept best if improved
if valid and valid[0].delta is not None and valid[0].delta > 0:
winner = valid[0]
winner.accepted = True
state.best_val_loss = winner.val_loss
state.best_script = winner.script
console.print(
f" [bold green]✓ Accepted: {winner.description}\n"
f" val_loss {calib_val_loss:.4f} → {state.best_val_loss:.4f} "
f"(Δ{winner.delta:+.4f})[/bold green]"
)
else:
console.print(" [dim]No improvement this iteration; baseline unchanged.[/dim]")
state.history.extend(results)
# ── Final summary ────────────────────────────────────────────────────────
finally:
accepted = [e for e in state.history if e.accepted]
total_improvement = calib_val_loss - state.best_val_loss
pct = total_improvement / calib_val_loss * 100
color = "bold green" if pct > 5 else "yellow" if pct > 0 else "red"
console.print(Panel(
f"[bold green]Autoresearch complete[/bold green]\n\n"
f"Baseline val_loss : [yellow]{calib_val_loss:.4f}[/yellow]\n"
f"Final val_loss : [bold green]{state.best_val_loss:.4f}[/bold green]\n"
f"Total improvement : [{color}]{total_improvement:+.4f} ({pct:+.1f}%)[/{color}]\n"
f"Accepted changes : {len(accepted)} / {len(state.history)}\n\n"
+ ("\n".join(f" ✓ iter {e.iteration}: {e.description}" for e in accepted)
if accepted else " (none)"),
title="[bold]Research Summary[/bold]",
border_style="green",
))
# ── Result interpretation ─────────────────────────────────────────────────
accept_rate = len(accepted) / len(state.history) * 100 if state.history else 0
console.print(Panel(
f"[bold]What these numbers mean[/bold]\n\n"
f"val_loss is cross-entropy on held-out characters (nats).\n"
f"Lower = the model assigns higher probability to the correct next character.\n\n"
f" Baseline {calib_val_loss:.4f} → Final {state.best_val_loss:.4f} "
f"([{color}]{pct:+.1f}%[/{color}])\n\n"
f"Context:\n"
f" • A random character predictor on this ~50-char vocabulary scores ln(50) ≈ 3.91\n"
f" • The baseline MLP ({calib_val_loss:.2f}) already beats random: it learned\n"
f" that 'e', space, and 't' are far more likely than 'Z'\n"
f" • Each accepted change is a genuine algorithmic improvement:\n"
f" the agent modified real training code and the sandbox verified it\n"
f" on held-out data, not on the training set\n\n"
f"Acceptance rate: {len(accepted)}/{len(state.history)} ({accept_rate:.0f}%)\n"
f" • Typical for greedy hill-climbing on a small model: 25–40% is normal\n"
f" • Rejected experiments are still informative: they update the agent's\n"
f" memory so it avoids the same dead ends next iteration\n\n"
f"Smoke vs full run:\n"
f" • Smoke (3 iters, 2 candidates, 150 steps) is a proof-of-concept\n"
f" • Full run (8 iters, 3 candidates, 300 steps) gives the agent\n"
f" enough budget to explore LR schedules, second layers, momentum,\n"
f" and architecture changes; improvements compound across iterations\n"
f" • Karpathy's original loop ran ~700 experiments overnight and found\n"
f" 11% speed improvement; the same pattern scales here",
title="[bold cyan]Score interpretation[/bold cyan]",
border_style="cyan",
))
if accepted:
console.print(Rule("[green]Final best script[/green]", style="green"))
console.print(Panel(state.best_script[:1200] + ("..." if len(state.best_script) > 1200 else ""),
border_style="green"))
if __name__ == "__main__":
from tensorlake.applications import run_local_application, Request
request: Request = run_local_application(
autoresearch,
iterations=3 if SMOKE else 8,
candidates=2 if SMOKE else 3,
)
````
***
## What happens step-by-step
| Step | Component | Action |
| :---- | :----------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **1** | **Calibration** | Baseline script runs in a sandbox. The resulting `val_loss` becomes the threshold every candidate must beat. |
| **2** | **Proposal** | GPT-4o receives the current best script, `program.md` guidance, and the last 8 experiments. It returns *N* JSON objects, each with a `description` and a complete modified `script`. |
| **3** | **Parallel race** | All *N* candidates are submitted to a `ThreadPoolExecutor`. Each thread creates a TensorLake sandbox, runs the script for the fixed step budget, and parses `val_loss` from stdout. |
| **4** | **Selection** | Candidates are ranked by `val_loss`. The winner is accepted only if its loss is strictly lower than the current best (greedy hill-climbing). |
| **5** | **State update** | The accepted script replaces the baseline. All results (accepted and rejected) are appended to the history so the agent avoids revisiting dead ends. |
| **6** | **Next iteration** | The agent is prompted again with the updated script and history. Improvements compound across iterations. |
***
## Key design decisions
### Increasing temperature across candidates
Each candidate is proposed with a slightly higher temperature (`0.9 + candidate_idx * 0.1`). The first candidate is a focused, conservative change. Later candidates are more exploratory. This covers both the safe and speculative ends of the search space in a single iteration.
### Experiment history as agent memory
The agent receives a rolling window of the last 8 experiments (accepted and rejected), each annotated with `val_loss` and `Δ`. This prevents the agent from re-proposing changes that already failed and nudges it toward unexplored directions without any external memory store.
### Fixed `STEPS` budget enforced in `program.md`
The `STEPS` constant is explicitly marked as off-limits in the guidance. Without this constraint, the agent could trivially reduce `val_loss` by running more training steps, a form of reward hacking that would make comparisons between candidates meaningless.
### Greedy hill-climbing over rollout
The loop uses simple greedy acceptance (accept if `Δval_loss > 0`) rather than beam search or rollout. For an overnight research loop where each experiment costs real CPU time, greedy hill-climbing maximises the number of validated improvements within the time budget.
***
This example uses `python-dotenv` to load your API keys. Create a `.env` file in your project root:
```
TENSORLAKE_API_KEY="your-api-key-here"
OPENAI_API_KEY="your-openai-key-here"
```
Both clients pick them up automatically.
## What to build next
Train a model directly with RL using sandboxes as the reward oracle, a complementary approach to autoresearch.
Dispatch parallel sandboxes across a swarm of specialized worker agents.
# Agentic Dungeons & Dragons
Source: https://docs.tensorlake.ai/sandboxes/agentic-d&g
Build a dynamic D&D-style game where parallel AI agents act as scene writers and a Dungeon Master agent orchestrates the story.
Create a dynamic, unpredictable storytelling game using a swarm of AI agents. This guide demonstrates how to build a Dungeons & Dragons-style RPG where multiple "Scene Agents" draft possible outcomes in parallel, and a "Dungeon Master" agent weaves them into a coherent narrative based on player choice.
This pattern uses a "Map-Reduce" model: parallel workers generate possibilities (map), and a lead agent synthesizes them (reduce).
## How it works
1. **Branching Possibilities**: For a given player choice, the application imagines several potential actions (e.g., "Fight," "Flee," "Negotiate").
2. **Map (Parallel Scene Writers)**: A `scene_agent` is spawned for each potential action. These run in parallel, each in its own sandbox.
3. **Sandbox Execution**: Each `scene_agent` uses an LLM to generate a Python script that simulates a dice roll and determines the outcome of its assigned action. The script runs securely in the sandbox and outputs a JSON with narrative text, consequences, and an ASCII art illustration.
4. **Reduce (Dungeon Master)**: A `dungeon_master` agent receives the drafted scenes from all parallel workers.
5. **Narrate & Update State**: The DM selects the draft corresponding to the player's *actual* choice, applies the consequences (e.g., HP loss, new item), and uses an LLM to write the next part of the story, complete with new choices for the player.
***
## Prerequisites
You'll need the Tensorlake SDK, an OpenAI client, and the `rich` library for the terminal UI.
```bash theme={null}
pip install tensorlake openai pydantic rich python-dotenv
```
This example uses the `python-dotenv` library to load your API keys from a `.env` file. Create a file named `.env` in your project root and add your keys:
```
TENSORLAKE_API_KEY="your-api-key-here"
OPENAI_API_KEY="your-openai-key-here"
```
The clients will automatically use these keys.
***
## TypeScript SDK starter
In Node.js, model each branch as `LLM -> sandbox -> JSON scene draft`, then reduce the drafts with your Dungeon Master step:
````typescript theme={null}
import OpenAI from "openai";
import { Sandbox } from "tensorlake";
type SceneDraft = {
branch_id: number;
branch_label: string;
narrative: string;
consequences: string;
image_prompt: string;
ascii_art: string;
};
const openai = new OpenAI();
async function sceneAgent(branchId: number, branchLabel: string) {
const prompt = `Write Python that simulates the "${branchLabel}" branch and prints one JSON object.`;
const response = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: prompt }],
});
const generatedCode =
response.choices[0].message.content
?.replace("```python", "")
.replace("```", "")
.trim() ?? "";
const sandbox = await Sandbox.create({
allowInternetAccess: false,
timeoutSecs: 600,
});
try {
const execution = await sandbox.run("python3", {
args: ["-c", generatedCode],
});
return JSON.parse(execution.stdout) as SceneDraft;
} finally {
await sandbox.terminate();
}
}
const drafts = await Promise.all([
sceneAgent(0, "Fight"),
sceneAgent(1, "Flee"),
sceneAgent(2, "Negotiate"),
]);
console.log(drafts);
sandboxes.close();
````
This gives you the same parallel-map stage as the Python example. Your Dungeon Master step can stay in Node.js and operate on the returned JSON drafts.
***
## Full Example
The complete script below orchestrates the entire game loop. You can run it directly to play in your terminal.
````python theme={null}
from dotenv import load_dotenv
load_dotenv()
from tensorlake.sandbox import Sandbox
from pydantic import BaseModel
from typing import List, Optional
from openai import OpenAI
from rich.console import Console
from rich.panel import Panel
from rich.text import Text
from rich.rule import Rule
from rich import box
import json
from concurrent.futures import ThreadPoolExecutor
import time
console = Console()
# ─── Data Models ─────────────────────────────────────────────────────────────
class PlayerState(BaseModel):
player_name: str
hp: int = 20
max_hp: int = 20
inventory: List[str] = ["torch", "dagger"]
story_history: List[str] = []
current_choice: Optional[str] = None
turn: int = 0
class SceneDraft(BaseModel):
branch_id: int
branch_label: str
narrative: str
consequences: str
image_prompt: str
ascii_art: str
class StoryBeat(BaseModel):
scene_narrative: str
choices: List[str]
image_prompt: str
ascii_art: str
updated_state: PlayerState
# ─── Agent 1: Scene Writer (runs in parallel per branch) ─────────────────────
def scene_agent(args: dict) -> SceneDraft:
"""
Each scene_agent drafts ONE possible branch outcome in an isolated sandbox.
Runs in parallel, one sandbox per branch.
"""
branch_id = args["branch_id"]
branch_label = args["branch_label"]
player_state = PlayerState(**args["player_state"])
setting = args["setting"]
print(f"⚔️ Scene Agent [{branch_label}]: Drafting branch in sandbox...")
client = OpenAI()
prompt = f"""
You are a D&D scene writer. The player chose: "{branch_label}".
Setting: {setting}
Player: {player_state.player_name}, HP: {player_state.hp}/{player_state.max_hp}
Inventory: {player_state.inventory}
Story so far: {' | '.join(player_state.story_history[-3:]) or 'Adventure begins.'}
Write a Python script that:
1. Uses 'random' to simulate a D20 dice roll
2. Determines success/failure of the action "{branch_label}" based on the roll (>=10 is success)
3. Prints a single valid JSON (no markdown, no extra text) with these exact keys:
- branch_id: {branch_id}
- branch_label: "{branch_label}"
- narrative: vivid 3-sentence scene description of the outcome
- consequences: one of "-N HP", "+item_name", "no change", or "unlocked secret"
- image_prompt: a DALL-E prompt for this scene in dark fantasy style
- ascii_art: a 10-15 line ASCII art illustration using / \\ | _ . * # @ ~ ^
that depicts the scene visually. Must be a single string with \\n for newlines.
Make it evocative of the environment: dungeon, dragon, forest, castle, monster, etc.
IMPORTANT: The script must print ONLY valid JSON to stdout. No markdown, no code fences.
"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}]
)
generated_code = (
response.choices[0].message.content
.replace("```python", "")
.replace("```", "")
.strip()
)
print(f"⚔️ Scene Agent [{branch_label}]: Executing dice logic in Sandbox...")
sandbox = Sandbox.create()
execution = sandbox.run("python3", ["-c", generated_code])
output = execution.stdout.strip()
print(f"⚔️ Scene Agent [{branch_label}]: Result -> {output[:80]}...")
data = json.loads(output)
return SceneDraft(**data)
# ─── Agent 2: Dungeon Master (aggregator + narrator) ─────────────────────────
def dungeon_master(args: dict) -> StoryBeat:
"""
The DM receives all parallel branch drafts, picks the player's chosen one,
narrates the next scene with 3 new choices, and updates state.
"""
drafts = [SceneDraft(**d) for d in args["drafts"]]
player_state = PlayerState(**args["player_state"])
chosen_label = player_state.current_choice
print(f"🎲 Dungeon Master: Received {len(drafts)} branch drafts. Chosen: '{chosen_label}'")
chosen = next((d for d in drafts if d.branch_label == chosen_label), drafts[0])
client = OpenAI()
prompt = f"""
You are an epic Dungeon Master continuing a D&D adventure.
The player chose: "{chosen.branch_label}"
What happened: {chosen.narrative}
Consequences: {chosen.consequences}
Player state: HP={player_state.hp}/{player_state.max_hp}, Inventory={player_state.inventory}
Turn number: {player_state.turn}
Now write the next story beat. Respond ONLY as raw JSON (no markdown, no code fences) with:
- scene_narrative: 2 vivid paragraphs in second person ("You...") describing what unfolds
- choices: list of exactly 3 short action choices for the player (action verbs, max 5 words each)
- image_prompt: a DALL-E prompt for the scene illustration in dark fantasy oil painting style
"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}]
)
raw = (
response.choices[0].message.content
.replace("```json", "")
.replace("```", "")
.strip()
)
data = json.loads(raw)
# Apply consequences to player state
updated = player_state.model_copy(deep=True)
cons = chosen.consequences.lower()
if "hp" in cons and "-" in cons:
try:
dmg = int(''.join(filter(str.isdigit, cons.split("hp")[0])))
updated.hp = max(0, updated.hp - dmg)
except ValueError:
pass
elif "hp" in cons and "+" in cons:
try:
heal = int(''.join(filter(str.isdigit, cons.split("hp")[0])))
updated.hp = min(updated.max_hp, updated.hp + heal)
except ValueError:
pass
if "+" in cons and "hp" not in cons:
item = chosen.consequences.replace("+", "").strip()
if item and item not in updated.inventory:
updated.inventory.append(item)
updated.story_history.append(chosen.narrative[:100])
updated.turn += 1
return StoryBeat(
scene_narrative=data["scene_narrative"],
choices=data["choices"],
image_prompt=data["image_prompt"],
ascii_art=chosen.ascii_art,
updated_state=updated,
)
# ─── Application: One Full RPG Turn ──────────────────────────────────────────
def rpg_adventure(player_name: str, choice: str, state_json: str, setting: str) -> str:
"""
One full turn:
1. Fan out 3 parallel scene agents (one per branch)
2. DM aggregates and narrates the chosen branch
3. Returns next StoryBeat as JSON
"""
print(f"\n🧙 RPG Turn: Player='{player_name}', Choice='{choice}'")
state = json.loads(state_json)
state["current_choice"] = choice
branches = [
{"branch_id": 0, "branch_label": "Fight"},
{"branch_id": 1, "branch_label": "Flee"},
{"branch_id": 2, "branch_label": "Negotiate"},
]
scene_args = [
{**b, "player_state": state, "setting": setting}
for b in branches
]
# Threads are needed to run multiple sandboxes concurrently.
# The sandboxes THEMSELVES run in parallel on the server, but threads allow
# our script to wait for multiple results simultaneously.
with ThreadPoolExecutor(max_workers=len(branches)) as executor:
drafts = list(executor.map(scene_agent, scene_args))
beat = dungeon_master({
"drafts": [d.model_dump() for d in drafts],
"player_state": state,
})
return beat.model_dump_json(indent=2)
# ─── UI Helpers ──────────────────────────────────────────────────────────────
TITLE_SCREEN = r"""
____ ____ _ __ ___ __ ______________ _________
/ __ \/ __ \/ | / / / _ | / / / / __/ ___/ _ \/ ___/ __/
/ /_/ / / / / |/ / / __ |/ /_/ / _// (_ / // / /__/ _/
\____/_/ /_/_/|_/ /_/ |_|\____/___/\___/____/\___/___/
🐉 A N A I - P O W E R E D A D V E N T U R E 🐉
"""
def print_title():
console.print()
console.print(Text(TITLE_SCREEN, style="bold red"))
console.print(Rule(style="red"))
console.print()
def print_scene(beat: StoryBeat):
console.print()
console.print(Rule("⚔ NEW SCENE", style="yellow"))
# ASCII art panel
console.print(
Panel(
Text(beat.ascii_art, style="bold green", justify="center"),
border_style="dim green",
padding=(1, 4),
)
)
# Narrative panel
console.print(
Panel(
beat.scene_narrative,
title="[bold cyan]📖 What Unfolds[/bold cyan]",
border_style="cyan",
padding=(1, 2),
)
)
def print_stats(state: PlayerState):
hp_color = "bold green" if state.hp > 10 else "bold yellow" if state.hp > 5 else "bold red"
hp_bar = "█" * state.hp + "░" * (state.max_hp - state.hp)
inv_str = ", ".join(state.inventory) if state.inventory else "nothing"
console.print(
Panel(
f"[{hp_color}]❤ HP: {state.hp}/{state.max_hp} [{hp_bar}][/{hp_color}]\n"
f"[bold white]🎒 Inventory:[/bold white] [dim]{inv_str}[/dim]\n"
f"[bold white]📜 Turn:[/bold white] [dim]{state.turn}[/dim]",
title=f"[bold magenta]🧙 {state.player_name}[/bold magenta]",
border_style="magenta",
box=box.SIMPLE,
)
)
def print_choices(choices: List[str]):
console.print()
console.print(Rule("🎮 YOUR MOVE", style="bold yellow"))
for i, c in enumerate(choices, 1):
console.print(f" [bold yellow]{i}.[/bold yellow] [white]{c}[/white]")
console.print(f" [dim]q. Quit adventure[/dim]")
console.print()
def get_player_choice(choices: List[str]) -> Optional[str]:
while True:
console.print("[bold]Enter 1, 2, 3 or q:[/bold] ", end="")
raw = input().strip().lower()
if raw == "q":
return None
if raw in ("1", "2", "3"):
idx = int(raw) - 1
if idx < len(choices):
return choices[idx]
console.print(" [bold red]⚠ Invalid input. Try 1, 2, 3 or q.[/bold red]")
# ─── Entry Point ─────────────────────────────────────────────────────────────
if __name__ == "__main__":
print_title()
# Hero name
console.print("[bold]Enter your hero's name[/bold] (or press Enter for 'Aldric the Bold'): ", end="")
player_name = input().strip() or "Aldric the Bold"
console.print(f"\n[bold green]Welcome, {player_name}! Your legend begins...[/bold green]\n")
time.sleep(1)
# Initial state & setting
state = PlayerState(player_name=player_name)
setting = (
"A crumbling dungeon entrance lit by sickly green torchlight. "
"Ancient runes glow on the walls. Something massive growls in the darkness ahead. "
"The air smells of sulfur and old bones."
)
current_choice = "Explore"
# ── Game Loop ──
while True:
console.print(f"\n[dim]⏳ Generating scene for: '[italic]{current_choice}[/italic]'...[/dim]")
try:
# Directly call the function instead of using run_local_application
result_json = rpg_adventure(
player_name=player_name,
choice=current_choice,
state_json=state.model_dump_json(),
setting=setting,
)
beat = StoryBeat.model_validate_json(result_json)
except Exception as e:
console.print(f"\n[bold red]❌ Error generating scene: {e}[/bold red]")
console.print("[dim]Retrying...[/dim]")
continue
# Update state
state = beat.updated_state
# Render scene
print_scene(beat)
print_stats(state)
# Death check
if state.hp <= 0:
console.print(
Panel(
"[bold red]💀 You have fallen in battle.\n\nYour legend ends here... for now.[/bold red]",
border_style="red",
padding=(1, 2),
)
)
break
# Choices
print_choices(beat.choices)
chosen = get_player_choice(beat.choices)
if chosen is None:
console.print(
"\n[bold yellow]🏰 You sheathe your sword and walk away into the mist.\n"
"Farewell, adventurer. Your story is unfinished.[/bold yellow]\n"
)
break
# Roll forward
current_choice = chosen
setting = beat.scene_narrative[-300:] # tail of scene becomes new setting context
````
***
### What Happens Step-by-Step
| Step | Component | Action |
| :---- | :----------------- | :-------------------------------------------------------------------------------------------------------------------------- |
| **1** | **Orchestrator** | Triggers 3 parallel `scene_agent` tasks using `.map()`, one for each potential action ("Fight", "Flee", "Negotiate"). |
| **2** | **Scene Agent** | Uses GPT-4o to generate a Python script that simulates a dice roll and determines the outcome for its assigned branch. |
| **3** | **Sandbox** | Securely executes the generated script, capturing the JSON output containing the narrative, consequences, and ASCII art. |
| **4** | **Dungeon Master** | Receives all drafted scenes and selects the one matching the player's actual choice. |
| **5** | **Dungeon Master** | Applies consequences (e.g., HP loss, new item) to the player's state based on the sandbox output. |
| **6** | **Dungeon Master** | Uses GPT-4o to narrate the next story beat and generate three new, context-aware choices for the player. |
| **7** | **UI Loop** | The main game loop receives the final `StoryBeat`, renders the scene and stats, and prompts the player for their next move. |
***
## How to Extend This Example
### Generate Images
The agents already create image prompts for DALL-E. You could extend the `dungeon_master` to call an image generation API and display the resulting image, creating a true multimedia experience.
### Add More Complex Logic
The sandbox is perfect for running more complex game mechanics. You could:
* Implement a full combat system with multiple enemy types.
* Create skill checks that depend on the player's inventory or stats.
* Generate dynamic loot tables or environmental puzzles.
### Use Snapshots for Faster Turns
If your `scene_agent` sandboxes needed to install libraries like `numpy` for more complex simulations, the `pip install` on every turn would add latency. You can pre-install dependencies into a base sandbox and create a **Snapshot**. Future turns can then launch from that snapshot instantly.
```python theme={null}
# In your scene_agent:
sandbox = Sandbox.create(snapshot_id="your-snapshot-id")
# Dependencies are already installed!
execution = sandbox.run("python3", ["-c", generated_code])
```
***
## What to build next
See another example of the Map-Reduce pattern with parallel agents.
Optimize your game's turn speed by pre-baking dependencies.
# Reproducible Environments for RL Rollouts
Source: https://docs.tensorlake.ai/sandboxes/agentic-rl-reproducible-env
Use Tensorlake sandboxes to guarantee isolated, deterministic rollouts for reinforcement learning training.
A rollout is a complete episode of agent-environment interaction: an agent takes actions, the environment transitions, and rewards accumulate until the episode ends. Reproducibility means that given the same random seed and the same action sequence, every rollout produces exactly the same observations, transitions, and rewards. This property is foundational for RL engineering: without it, you cannot reliably compare two policy versions, reproduce a bug seen during training, or verify that a reward spike was real and not noise.
The hard part is that real training runs hundreds or thousands of rollouts in parallel. Each worker must be completely isolated from the others: no shared filesystem, no shared process state, no network side-effects leaking across episodes. If any state bleeds between workers, your "reproducible" seed no longer controls the outcome and you lose the guarantee. Tensorlake sandboxes enforce this isolation at the infrastructure level: every rollout gets its own fresh environment, and the seed is the only variable in play.
***
## Core concepts
**Isolation** means each rollout runs in its own compute environment with no shared resources. Two workers seeded with different values must not be able to influence each other's trajectories through any shared channel: not a shared pip cache, not a shared `/tmp`, not a shared network state. In production, this matters most when you are running hundreds of rollouts per training step: any shared state becomes a source of variance that your reward signal cannot explain.
**Stateful resets** mean the environment always starts from a known, controlled baseline when a new rollout begins. A reset that partially inherits state from a previous episode is one of the most common and hardest-to-debug sources of non-reproducibility. Because each sandbox is created fresh per rollout, the reset is total: there is no prior episode state to inherit.
**Determinism** means the environment's random number generator is seeded before any interaction begins, and the seed is the sole source of randomness for the entire episode. Given the same seed, the same initial observation, and the same action sequence, the trajectory must be identical byte-for-byte. This lets you replay any episode from training history, compare policy versions on equal footing, and write regression tests against specific trajectories.
***
## How Tensorlake sandboxes provide this
`Sandbox.create()` starts a fresh, isolated compute environment and returns a `box` handle. Every sandbox is a separate process tree with its own filesystem and memory. There is no shared state between two sandboxes created from the same client.
The seed is passed into the environment harness as a string literal embedded in the Python script that runs inside the sandbox, not set on the host process. This keeps the host's random state completely separate from the environment's, which is important when you dispatch many rollouts from a single host thread pool.
Parallel rollouts map cleanly onto `ThreadPoolExecutor`: each thread creates its own sandbox, runs its episode, collects its trajectory, and the sandbox is destroyed when the context manager exits. The executor manages concurrency; the sandboxes manage isolation.
***
## Prerequisites
```bash theme={null}
pip install tensorlake gymnasium python-dotenv
```
Create a `.env` file in your project root:
```
TENSORLAKE_API_KEY="your-api-key-here"
```
***
## TypeScript SDK starter
The same reproducibility pattern works from Node.js: embed the seed in the harness, run one rollout per sandbox, and compare trajectories across identical seeds.
```typescript theme={null}
import { Sandbox } from "tensorlake";
function gymHarness(seed: number) {
return `
import gymnasium as gym, json
seed = ${seed}
env = gym.make("CartPole-v1")
obs, _ = env.reset(seed=seed)
env.action_space.seed(seed)
trajectory = []
total_reward = 0.0
for _ in range(200):
action = env.action_space.sample()
next_obs, reward, terminated, truncated, _ = env.step(action)
trajectory.append((obs.tolist(), int(action), float(reward), bool(terminated)))
total_reward += reward
obs = next_obs
if terminated or truncated:
break
print(json.dumps({"seed": seed, "total_reward": total_reward, "trajectory": trajectory}))
`;
}
async function runSingleRollout(seed: number) {
const sandbox = await Sandbox.create({
memoryMb: 2048,
allowInternetAccess: false,
});
try {
await sandbox.run("python3", {
args: ["-m", "pip", "install", "gymnasium", "--break-system-packages", "-q"],
});
const result = await sandbox.run("python3", {
args: ["-c", gymHarness(seed)],
});
return JSON.parse(result.stdout);
} finally {
await sandbox.terminate();
}
}
const [rolloutA, rolloutB] = await Promise.all([
runSingleRollout(42),
runSingleRollout(42),
]);
console.assert(
JSON.stringify(rolloutA.trajectory) === JSON.stringify(rolloutB.trajectory),
"same seed should produce the same trajectory",
);
client.close();
```
For batch collection, replace the final `Promise.all()` with one rollout per seed and aggregate the returned JSON results by seed.
***
## Full example
```python theme={null}
"""
Reproducible RL Rollouts with Tensorlake Sandboxes
===================================================
Demonstrates three properties:
1. Isolation - each rollout runs in its own sandbox
2. Determinism - same seed → same trajectory, verified by assertion
3. Parallelism - multiple seeds dispatched concurrently via ThreadPoolExecutor
"""
from dotenv import load_dotenv
load_dotenv()
import json
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, field
from typing import List, Tuple
from tensorlake.sandbox import Sandbox
# ─── Data models ──────────────────────────────────────────────────────────────
@dataclass
class RolloutConfig:
seed: int
env_name: str = "CartPole-v1"
max_steps: int = 200
@dataclass
class RolloutResult:
seed: int
total_reward: float
steps: int
# Each element is (observation, action, reward, terminated)
trajectory: List[Tuple] = field(default_factory=list)
# ─── Gymnasium harness ────────────────────────────────────────────────────────
# This script runs inside the sandbox. It is a self-contained string so that
# the host's Python environment has no influence on the episode's random state.
_GYM_HARNESS = """
import gymnasium as gym
import json
import sys
seed = {seed}
env_name = {env_name!r}
max_steps = {max_steps}
env = gym.make(env_name)
obs, _ = env.reset(seed=seed)
# env.reset(seed=) only seeds the observation/transition RNG.
# The action space has its own RNG that must be seeded separately.
env.action_space.seed(seed)
trajectory = []
total_reward = 0.0
steps = 0
for _ in range(max_steps):
action = env.action_space.sample()
next_obs, reward, terminated, truncated, _ = env.step(action)
trajectory.append((obs.tolist(), int(action), float(reward), bool(terminated)))
total_reward += reward
steps += 1
obs = next_obs
if terminated or truncated:
break
env.close()
result = {{
"seed": seed,
"total_reward": total_reward,
"steps": steps,
"trajectory": trajectory,
}}
# The only output is the JSON result: the caller reads stdout
print(json.dumps(result))
"""
# ─── Single rollout ───────────────────────────────────────────────────────────
def run_single_rollout(config: RolloutConfig) -> RolloutResult:
"""
Run one complete RL episode in a fresh, isolated sandbox.
A new sandbox is created for every call so there is no shared filesystem
or process state between concurrent rollouts. The seed is embedded in the
harness string rather than set on the host, which keeps the host's random
state fully separate from the environment's.
"""
harness = _GYM_HARNESS.format(
seed=config.seed,
env_name=config.env_name,
max_steps=config.max_steps,
)
box = Sandbox.create(memory_mb=2048)
# Use python3 -m pip to install into the sandbox's managed environment
box.run("python3", ["-m", "pip", "install", "gymnasium",
"--break-system-packages", "-q"])
execution = box.run("python3", ["-c", harness])
raw = (execution.stdout or "").strip()
data = json.loads(raw)
return RolloutResult(
seed=data["seed"],
total_reward=data["total_reward"],
steps=data["steps"],
trajectory=data["trajectory"],
)
# ─── Parallel rollout collection ──────────────────────────────────────────────
def collect_parallel_rollouts(
seeds: List[int],
env_name: str = "CartPole-v1",
max_steps: int = 200,
) -> List[RolloutResult]:
"""
Dispatch one sandbox per seed, all running concurrently.
ThreadPoolExecutor manages the concurrency; the sandboxes manage isolation.
Results are returned in seed order regardless of completion order.
"""
configs = [RolloutConfig(seed=s, env_name=env_name, max_steps=max_steps) for s in seeds]
results_by_seed = {}
with ThreadPoolExecutor(max_workers=len(configs)) as pool:
future_to_seed = {pool.submit(run_single_rollout, cfg): cfg.seed for cfg in configs}
for future in as_completed(future_to_seed):
seed = future_to_seed[future]
results_by_seed[seed] = future.result()
return [results_by_seed[s] for s in seeds]
# ─── Reproducibility check ────────────────────────────────────────────────────
def verify_reproducibility(
seed: int = 42,
env_name: str = "CartPole-v1",
max_steps: int = 200,
) -> None:
"""
Run the same seed twice in independent sandboxes and assert the trajectories
are identical. This is the core guarantee: isolation + determinism means the
seed fully determines the episode.
"""
print(f"Verifying reproducibility for seed={seed}...")
config = RolloutConfig(seed=seed, env_name=env_name, max_steps=max_steps)
result_a = run_single_rollout(config)
result_b = run_single_rollout(config)
assert result_a.steps == result_b.steps, (
f"Step count mismatch: {result_a.steps} vs {result_b.steps}"
)
assert result_a.total_reward == result_b.total_reward, (
f"Reward mismatch: {result_a.total_reward} vs {result_b.total_reward}"
)
assert result_a.trajectory == result_b.trajectory, (
"Trajectory mismatch: observations or actions differed between runs"
)
print(
f" Passed. seed={seed} → {result_a.steps} steps, "
f"reward={result_a.total_reward:.1f} (identical across both runs)"
)
# ─── Main ─────────────────────────────────────────────────────────────────────
if __name__ == "__main__":
# Step 1: Verify that the same seed always produces the same trajectory
verify_reproducibility(seed=42)
print(
" → Same seed, two independent sandboxes, identical trajectory.\n"
" The seed is the only source of variation: no shared state, no host RNG leakage."
)
# Step 2: Collect 4 rollouts in parallel, one sandbox per seed
seeds = [0, 1, 2, 3]
print(f"\nCollecting {len(seeds)} parallel rollouts...")
results = collect_parallel_rollouts(seeds)
# Step 3: Print a summary table
print(f"\n{'Seed':>6} {'Steps':>6} {'Total Reward':>14}")
print("-" * 32)
for r in results:
print(f"{r.seed:>6} {r.steps:>6} {r.total_reward:>14.1f}")
best = max(results, key=lambda r: r.total_reward)
worst = min(results, key=lambda r: r.total_reward)
print(
f"\n → CartPole rewards 1.0 per step, so total reward equals episode length.\n"
f" Seed {best.seed} balanced the longest ({int(best.total_reward)} steps); "
f"seed {worst.seed} fell first ({int(worst.total_reward)} steps).\n"
f" Different seeds produce different episodes because the initial pole\n"
f" angle varies. Run again with the same seeds and you get identical numbers."
)
```
**Expected output:**
```
Verifying reproducibility for seed=42...
Passed. seed=42 → 30 steps, reward=30.0 (identical across both runs)
Collecting 4 parallel rollouts...
Seed Steps Total Reward
--------------------------------
0 18 18.0
1 29 29.0
2 14 14.0
3 15 15.0
```
In CartPole, the reward is 1.0 per step regardless of action, so total reward equals step count. The episode ends when the pole tips past 12 degrees or the cart leaves the track. Different seeds produce different episode lengths because the initial pole angle varies. The reproducibility assertion confirms that seed=42 always produces the exact same 30-step trajectory in two independent sandboxes.
***
## Tic-tac-toe: policy evaluation
This example extends the CartPole infrastructure to a custom two-player environment and shows where sandboxes are more directly necessary. Policies are defined as code strings, the same pattern used in [RL Training with GSPO](/sandboxes/gspo-agentic-rl) for LLM-generated completions. A policy that crashes, loops, or behaves unexpectedly only kills its own sandbox; the rest of the evaluation runs unaffected.
The data model is the same as CartPole: `TttConfig` extends `RolloutConfig` by replacing `env_name` with `policy_x` and `policy_o`; `run_ttt_batch` returns the same `RolloutResult`. The `total_reward` field becomes the mean return per game from X's perspective (+1 win, −1 loss, 0 draw). `evaluate_matchup` follows the same parallel dispatch pattern as `collect_parallel_rollouts`, running one sandbox per seed to get a reliable return estimate. This is the **policy evaluation** step in policy iteration. You would call it after each policy update to measure how much the return improved.
```python theme={null}
from dotenv import load_dotenv
load_dotenv()
import json
import statistics
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, field
from typing import Dict, List, Tuple
from tensorlake.sandbox import Sandbox
# ─── Reuse RolloutResult from the CartPole section ────────────────────────────
# total_reward = mean reward per game, X's perspective (+1 win, -1 loss, 0 draw)
# steps = total moves across all games in the batch
# trajectory = list of per-game outcomes
@dataclass
class RolloutResult:
seed: int
total_reward: float
steps: int
trajectory: List[dict] = field(default_factory=list)
# ─── Tic-tac-toe config ───────────────────────────────────────────────────────
# Extends the RolloutConfig pattern: swap env_name for policy_x / policy_o,
# add n_games (games per sandbox call = one rollout batch).
@dataclass
class TttConfig:
seed: int
policy_x: str # key into POLICIES
policy_o: str
n_games: int = 50
# ─── Policies as code strings ─────────────────────────────────────────────────
# Treat these like LLM-generated completions: they run inside the sandbox,
# never in the host process. A buggy policy crashes its sandbox, not the loop.
POLICIES: Dict[str, str] = {
"random": """
def choose_action(board, player, rng):
moves = [i for i, v in enumerate(board) if v is None]
return rng.choice(moves)
""",
"greedy": """
def choose_action(board, player, rng):
WINS = [(0,1,2),(3,4,5),(6,7,8),(0,3,6),(1,4,7),(2,5,8),(0,4,8),(2,4,6)]
opponent = "O" if player == "X" else "X"
moves = [i for i, v in enumerate(board) if v is None]
# Take the win if available
for move in moves:
b = board[:]; b[move] = player
for a, c, d in WINS:
if b[a] and b[a] == b[c] == b[d]: return move
# Block the opponent's win
for move in moves:
b = board[:]; b[move] = opponent
for a, c, d in WINS:
if b[a] and b[a] == b[c] == b[d]: return move
return rng.choice(moves)
""",
}
# ─── Harness ──────────────────────────────────────────────────────────────────
# Runs n_games games inside a single sandbox and returns the batch return.
# Both policies execute in separate namespaces so they can't overwrite each
# other's globals. This matters when policies come from different sources.
_TTT_HARNESS = """
import json, random
WINS = [(0,1,2),(3,4,5),(6,7,8),(0,3,6),(1,4,7),(2,5,8),(0,4,8),(2,4,6)]
ns_x, ns_o = {{}}, {{}}
exec({policy_x!r}, ns_x); exec({policy_o!r}, ns_o)
choose_x = ns_x["choose_action"]; choose_o = ns_o["choose_action"]
def winner(b):
for a, c, d in WINS:
if b[a] and b[a] == b[c] == b[d]: return b[a]
return None
rng = random.Random({seed})
games = []
for _ in range({n_games}):
board, moves_played = [None] * 9, 0
for turn in range(9):
player = "X" if turn % 2 == 0 else "O"
action = (choose_x if player == "X" else choose_o)(board[:], player, rng)
board[action] = player; moves_played += 1
w = winner(board)
if w:
games.append({{"outcome": w + " wins", "reward": 1 if w == "X" else -1, "moves": moves_played}})
break
else:
games.append({{"outcome": "draw", "reward": 0, "moves": moves_played}})
print(json.dumps({{
"total_reward": sum(g["reward"] for g in games) / len(games),
"steps": sum(g["moves"] for g in games),
"trajectory": games,
}}))
"""
# ─── Interactive move oracle ──────────────────────────────────────────────────
# For interactive play the sandbox stays open for the whole game session.
# Each opponent turn sends the current board and gets one action back.
# timeout_secs gives the human up to 5 minutes of total think time.
_MOVE_HARNESS = """
import random
ns = {{}}
exec({policy!r}, ns)
action = ns["choose_action"]({board!r}, {player!r}, random.Random({seed}))
print(action)
"""
WINS = [(0,1,2),(3,4,5),(6,7,8),(0,3,6),(1,4,7),(2,5,8),(0,4,8),(2,4,6)]
def _winner(board: list):
for a, c, d in WINS:
if board[a] and board[a] == board[c] == board[d]:
return board[a]
return None
def _display(board: list) -> None:
row = lambda i: " | ".join(
str(i * 3 + j) if board[i * 3 + j] is None else board[i * 3 + j]
for j in range(3)
)
print(f" {row(0)}\n---+---+---\n {row(1)}\n---+---+---\n {row(2)}\n")
def play_against(human_side: str = "X", opponent_policy: str = "greedy") -> None:
"""
Play a game of tic-tac-toe against a policy running in a sandbox.
The sandbox opens once at the start of the game and stays live until the
game ends. Each opponent turn is a single box.run() call: the policy code
never executes in the host process.
human_side: "X" (you move first) or "O" (opponent moves first)
opponent_policy: any key in POLICIES
"""
assert human_side in ("X", "O"), "human_side must be 'X' or 'O'"
opponent_side = "O" if human_side == "X" else "X"
board = [None] * 9
print(f"\nYou are {human_side}. Opponent: {opponent_policy}.")
print("Empty squares show their position number (0–8).\n")
_display(board)
# Keep one sandbox alive for the whole game: no re-creation per move
box = Sandbox.create(memory_mb=1024, timeout_secs=300)
for turn in range(9):
player = "X" if turn % 2 == 0 else "O"
available = [i for i, v in enumerate(board) if v is None]
if player == human_side:
while True:
try:
move = int(input(f"Your move ({human_side}), choose from {available}: "))
if move in available:
break
print(f" Square {move} is taken. Choose from {available}.")
except ValueError:
print(f" Enter a number from {available}.")
else:
# The seed is the turn number: deterministic but varies per turn
harness = _MOVE_HARNESS.format(
policy=POLICIES[opponent_policy],
board=board,
player=player,
seed=turn,
)
ex = box.run("python3", ["-c", harness])
move = int((ex.stdout or "").strip())
print(f" {opponent_side} ({opponent_policy}) plays {move}")
board[move] = player
_display(board)
w = _winner(board)
if w:
print("You win!" if w == human_side else f"{opponent_policy} wins!")
return
print("Draw!")
# ─── Single batch rollout ─────────────────────────────────────────────────────
def run_ttt_batch(config: TttConfig) -> RolloutResult:
"""
Run one batch of n_games in a fresh sandbox and return a RolloutResult.
Follows the same signature as run_single_rollout from the CartPole section:
one config in, one RolloutResult out, one sandbox per call.
"""
harness = _TTT_HARNESS.format(
policy_x=POLICIES[config.policy_x],
policy_o=POLICIES[config.policy_o],
seed=config.seed,
n_games=config.n_games,
)
box = Sandbox.create(memory_mb=1024)
ex = box.run("python3", ["-c", harness])
data = json.loads((ex.stdout or "").strip())
return RolloutResult(
seed=config.seed,
total_reward=data["total_reward"],
steps=data["steps"],
trajectory=data["trajectory"],
)
# ─── Policy evaluation ────────────────────────────────────────────────────────
def evaluate_matchup(
policy_x: str,
policy_o: str,
seeds: List[int],
n_games: int = 50,
) -> Tuple[float, float]:
"""
Run one batch per seed in parallel; return (mean_return, std_return).
Follows the same parallel dispatch pattern as collect_parallel_rollouts:
one sandbox per seed, all running concurrently. More seeds = tighter
estimate of the true policy return.
"""
configs = [
TttConfig(seed=s, policy_x=policy_x, policy_o=policy_o, n_games=n_games)
for s in seeds
]
returns: List[float] = [0.0] * len(configs)
with ThreadPoolExecutor(max_workers=len(configs)) as pool:
futures = {pool.submit(run_ttt_batch, cfg): i for i, cfg in enumerate(configs)}
for future in as_completed(futures):
returns[futures[future]] = future.result().total_reward
return statistics.mean(returns), statistics.stdev(returns)
# ─── Q-learning ───────────────────────────────────────────────────────────────
# Uses str(s)+","+str(a) as Q-key to avoid f-string braces conflicting
# with .format() when the harness template is rendered on the host.
_QLEARN_HARNESS = """
import json, random
WINS = [(0,1,2),(3,4,5),(6,7,8),(0,3,6),(1,4,7),(2,5,8),(0,4,8),(2,4,6)]
def greedy_move(board, rng):
moves = [i for i, v in enumerate(board) if v is None]
for move in moves:
b = board[:]; b[move] = "O"
for a, c, d in WINS:
if b[a] and b[a] == b[c] == b[d]: return move
for move in moves:
b = board[:]; b[move] = "X"
for a, c, d in WINS:
if b[a] and b[a] == b[c] == b[d]: return move
return rng.choice(moves)
def winner(b):
for a, c, d in WINS:
if b[a] and b[a] == b[c] == b[d]: return b[a]
return None
def skey(b): return tuple(0 if v is None else 1 if v == "X" else 2 for v in b)
def qkey(s, a): return str(s) + "," + str(a)
def qv(q, s, a): return q.get(qkey(s, a), 0.0)
q = json.loads({q_json!r})
rng = random.Random({seed})
alpha, gamma, epsilon = {alpha}, {gamma}, {epsilon}
ep_rewards = []
for _ in range({n_episodes}):
board = [None] * 9
ep_r = 0.0
while True:
moves = [i for i, v in enumerate(board) if v is None]
if not moves: ep_rewards.append(ep_r); break
s = skey(board)
a = rng.choice(moves) if rng.random() < epsilon else max(moves, key=lambda x: qv(q, s, x))
board[a] = "X"
w = winner(board)
if w or not any(v is None for v in board):
r = 1.0 if w == "X" else -1.0 if w == "O" else 0.0
q[qkey(s, a)] = qv(q, s, a) + alpha * (r - qv(q, s, a))
ep_r += r; ep_rewards.append(ep_r); break
board[greedy_move(board[:], rng)] = "O"
w = winner(board)
r = 1.0 if w == "X" else -1.0 if w == "O" else 0.0
s2 = skey(board)
moves2 = [i for i, v in enumerate(board) if v is None]
nq = max((qv(q, s2, x) for x in moves2), default=0.0) if moves2 else 0.0
q[qkey(s, a)] = qv(q, s, a) + alpha * (r + gamma * nq - qv(q, s, a))
ep_r += r
if w or not moves2: ep_rewards.append(ep_r); break
print(json.dumps({{"q_table": q, "mean_reward": sum(ep_rewards)/len(ep_rewards), "n_states": len(q)}}))
"""
@dataclass
class QConfig:
seed: int
q_table: dict = field(default_factory=dict)
epsilon: float = 0.3 # exploration rate: high early, can decay over iterations
alpha: float = 0.5 # learning rate
gamma: float = 0.9 # discount factor
n_episodes: int = 300
def run_qlearning_iter(config: QConfig) -> dict:
"""Run one training iteration in a sandbox; return updated Q-table + stats."""
harness = _QLEARN_HARNESS.format(
q_json=json.dumps(config.q_table),
seed=config.seed,
alpha=config.alpha,
gamma=config.gamma,
epsilon=config.epsilon,
n_episodes=config.n_episodes,
)
box = Sandbox.create(memory_mb=1024)
ex = box.run("python3", ["-c", harness])
return json.loads((ex.stdout or "").strip())
def train_q(n_iter: int = 8, episodes_per_iter: int = 300) -> dict:
"""
Train a Q-table over n_iter sequential sandbox calls.
Each call receives the Q-table from the previous iteration and returns
an updated one. Mean reward moving from negative to positive confirms
the policy is improving against the greedy opponent.
"""
q_table: dict = {}
print(f"{'Iter':>5} {'Mean reward':>13} {'Q-states':>10}")
print("-" * 34)
for i in range(n_iter):
result = run_qlearning_iter(QConfig(seed=i, q_table=q_table, n_episodes=episodes_per_iter))
q_table = result["q_table"]
print(f"{i+1:>5} {result['mean_reward']:>+13.3f} {result['n_states']:>10}")
return q_table
def q_policy_code(q_table: dict) -> str:
"""
Serialize the Q-table into a choose_action string compatible with POLICIES.
This lets the learned policy plug directly into evaluate_matchup and
play_against without any changes to those functions.
"""
q_json = json.dumps(q_table)
return (
"import json as _j\n"
"_Q = _j.loads(" + repr(q_json) + ")\n"
"def choose_action(board, player, rng):\n"
" def skey(b): return tuple(0 if v is None else 1 if v == 'X' else 2 for v in b)\n"
" def qkey(s, a): return str(s) + ',' + str(a)\n"
" moves = [i for i, v in enumerate(board) if v is None]\n"
" return max(moves, key=lambda a: _Q.get(qkey(skey(board), a), 0.0))\n"
)
# ─── Main ─────────────────────────────────────────────────────────────────────
if __name__ == "__main__":
matchups = [
("random", "random"),
("greedy", "random"),
("random", "greedy"),
("greedy", "greedy"),
]
seeds = [0, 1, 2, 3] # one sandbox per seed per matchup = 16 sandboxes total
print("Evaluating all matchups (4 seeds × 50 games each, one sandbox per seed)...")
print(f"\n{'X policy':>10} {'O policy':>10} {'mean return':>13} {'std':>6}")
print("-" * 48)
eval_results = {}
for x, o in matchups:
mean, std = evaluate_matchup(x, o, seeds=seeds)
eval_results[(x, o)] = (mean, std)
print(f"{x:>10} {o:>10} {mean:>+13.3f} {std:>6.3f}")
print(
f"\n → Mean return is the expected reward per game from X's perspective\n"
f" (+1 win, −1 loss, 0 draw), averaged over {seeds} seeds × 50 games.\n"
f" greedy-vs-random ({eval_results[('greedy','random')][0]:+.3f}) shows how\n"
f" strongly a win/block heuristic dominates pure chance.\n"
f" greedy-vs-greedy ({eval_results[('greedy','greedy')][0]:+.3f} ≠ 0) reveals a\n"
f" fork vulnerability: X can reach positions that greedy-O cannot\n"
f" simultaneously block, which a stronger policy would eliminate.\n"
f" Low std (0.04–0.10) confirms 4 seeds × 50 games is enough to\n"
f" rank policies reliably. Scale up seeds for tighter confidence intervals."
)
# ── Train and add the learned policy ─────────────────────────────────────
print("\nTraining Q-learner vs greedy opponent (8 iterations × 300 episodes)...")
q_table = train_q(n_iter=8, episodes_per_iter=300)
# Serialize the Q-table into a choose_action string, same interface as
# random and greedy, so evaluate_matchup works without any changes.
POLICIES["q_learned"] = q_policy_code(q_table)
print("\nEvaluating learned policy against baselines:")
print(f"\n{'Matchup':>28} {'mean return':>13} {'std':>6}")
print("-" * 54)
for x, o in [("q_learned", "greedy"), ("greedy", "q_learned"), ("q_learned", "random")]:
mean, std = evaluate_matchup(x, o, seeds=seeds)
print(f"{x+' vs '+o:>28} {mean:>+13.3f} {std:>6.3f}")
print(
"\n → q_learned was trained as X against greedy O.\n"
" It does not know how to play as O. greedy vs q_learned\n"
" exposes this: the policy is role-specialized, not general."
)
# ── Play a game ───────────────────────────────────────────────────────────
side = input("\nPlay a game? Choose your side [X/O] (or press Enter to skip): ").strip().upper()
if side in ("X", "O"):
available_policies = list(POLICIES.keys())
opp = input(f"Opponent policy {available_policies} (default: greedy): ").strip().lower()
if opp not in POLICIES:
opp = "greedy"
play_against(human_side=side, opponent_policy=opp)
```
**Expected output (evaluation):**
```
Evaluating all matchups (4 seeds × 50 games each, one sandbox per seed)...
X policy O policy mean return std
------------------------------------------------
random random +0.255 0.100
greedy random +0.900 0.043
random greedy -0.640 0.069
greedy greedy +0.180 0.059
→ Mean return is the expected reward per game from X's perspective
(+1 win, −1 loss, 0 draw), averaged over [0, 1, 2, 3] seeds × 50 games.
greedy-vs-random (+0.900) shows how strongly a win/block heuristic dominates pure chance.
greedy-vs-greedy (+0.180 ≠ 0) reveals a fork vulnerability: X can reach positions
that greedy-O cannot simultaneously block, which a stronger policy would eliminate.
Low std (0.04–0.10) confirms 4 seeds × 50 games is enough to rank policies reliably.
```
**Expected output (Q-learning training):**
```
Training Q-learner vs greedy opponent (8 iterations × 300 episodes)...
Iter Mean reward Q-states
----------------------------------
1 -0.470 393
2 -0.113 592
3 -0.177 823
4 -0.080 963
5 +0.117 1051
6 +0.087 1159
7 +0.073 1226
8 +0.053 1297
Evaluating learned policy against baselines:
Matchup mean return std
------------------------------------------------------
q_learned vs greedy +0.927 0.034
greedy vs q_learned +0.990 0.008
q_learned vs random +0.785 0.051
→ q_learned was trained as X against greedy O.
It does not know how to play as O. greedy vs q_learned
exposes this: the policy is role-specialized, not general.
```
The training loop passes the Q-table from each iteration into the next via JSON. Mean reward moving from −0.47 to +0.05 over 8 iterations shows the policy improving against a greedy opponent. Each iteration is a separate sandbox call: the host owns the Q-table and the loop control; the sandbox owns the episode dynamics.
The jump in Q-states from 393 to 1297 reflects the agent exploring new board positions as its policy improves. Early iterations barely escape losing positions; later ones have enough coverage to exploit the greedy opponent's fork blindspot.
After training, `q_policy_code()` serializes the Q-table into a `choose_action` string with the same interface as `random` and `greedy`. This lets the learned policy drop into `evaluate_matchup` and `play_against` with zero changes to those functions.
After the evaluation the script prompts you to play. Choose `X` to move first or `O` to let the opponent open.
**Expected output (interactive game as O against greedy):**
```
Play a game? Choose your side [X/O] (or press Enter to skip): O
Opponent policy ['random', 'greedy', 'q_learned'] (default: greedy):
You are O. Opponent: greedy.
Empty squares show their position number (0–8).
0 | 1 | 2
---+---+---
3 | 4 | 5
---+---+---
6 | 7 | 8
X (greedy) plays 4
0 | 1 | 2
---+---+---
3 | X | 5
---+---+---
6 | 7 | 8
Your move (O), choose from [0, 1, 2, 3, 5, 6, 7, 8]: 0
O | 1 | 2
---+---+---
3 | X | 5
---+---+---
6 | 7 | 8
X (greedy) plays 8
...
```
The opponent's policy code runs inside the sandbox on every turn. The `choose_action` function never executes in your host process. The sandbox stays open for the whole game session (`timeout_secs=300`); only the move oracle harness re-runs on each turn.
***
## Key design callouts
### Why the seed is embedded in the harness string
The seed is formatted directly into the Python script that runs inside the sandbox, not set via an environment variable or a host-side call. This means the host process's random state has no path into the episode. If you set the seed on the host and then passed the environment object into the sandbox, any host-side RNG calls between setup and rollout would shift the environment's random state relative to what you expected. Embedding it in the harness makes the episode fully self-contained.
In gymnasium specifically, `env.reset(seed=seed)` only seeds the observation and transition RNG. The action space has a separate RNG that must be seeded independently with `env.action_space.seed(seed)`. Forgetting the second call produces non-deterministic trajectories even when everything else is correct.
### Why each rollout gets its own sandbox
Sharing a sandbox across rollouts would mean sharing filesystem state, installed package versions, and any residual process state from prior episodes. Even if you call `env.reset()` correctly, state outside the environment object (temporary files, cached computations, mutated globals) can persist and affect the next episode. Creating a fresh sandbox per rollout makes the isolation structural rather than depending on careful cleanup.
### How this relates to the GSPO pattern
In [RL Training with GSPO](/sandboxes/gspo-agentic-rl), the sandbox is a reward oracle: each model completion is sent to a sandbox that runs a hidden test suite and returns a score. The reproducibility concern is different there: you need each completion to be evaluated fairly, not that the environment is deterministic. But the underlying mechanism is the same: one sandbox per evaluation, no shared state. The reproducibility pattern here is what you would use when the environment itself (not just the evaluator) needs to be deterministic across training runs.
***
This example uses `python-dotenv` to load your Tensorlake API key. Create a `.env` file in your project root:
```
TENSORLAKE_API_KEY="your-api-key-here"
```
The SDK will pick it up automatically.
***
## What to build next
Use sandboxes as a reward oracle to fine-tune a language model on code generation tasks.
Dispatch parallel sandboxes across a swarm of worker agents for large-scale rollout collection.
Freeze environment state mid-rollout to create branching experiments without re-running from scratch.
# Agentic Swarm Intelligence
Source: https://docs.tensorlake.ai/sandboxes/agentic-swarm-intelligence
Orchestrate a swarm of LLM agents running specialized tasks in parallel sandboxes.
Combine the power of LLM orchestration with secure, isolated execution environments. This guide shows how to build a "swarm" of agents, where multiple worker agents generate and execute code in parallel sandboxes to analyze a problem from different perspectives, and a lead agent synthesizes their findings.
## How it works
1. **Define Worker Agents**: Create a function that uses an LLM to generate code for a specific perspective (e.g., Scientific, Economic).
2. **Execute in Sandboxes**: Each worker spins up a secure Tensorlake Sandbox to run the generated code and capture the output.
3. **Map (Parallelize)**: Launch multiple instances of the worker agent in parallel.
4. **Reduce (Aggregate)**: A lead agent receives all the reports and synthesizes a final insight.
***
## Prerequisites
```bash theme={null}
pip install tensorlake openai pydantic python-dotenv
```
## TypeScript SDK starter
If your orchestrator already runs in Node.js, use the same pattern: LLM generates code, one sandbox executes it, and `Promise.all()` fans the scouts out in parallel.
````typescript theme={null}
import OpenAI from "openai";
import { Sandbox } from "tensorlake";
type ScoutReport = {
perspective: string;
score: number;
insight: string;
};
const openai = new OpenAI();
async function scoutAgent(perspective: string): Promise {
const prompt = `You are a ${perspective} analyst for a Mars mission.
Write Python that prints one JSON object with perspective, score, and insight.`;
const response = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: prompt }],
});
const generatedCode =
response.choices[0].message.content
?.replace("```python", "")
.replace("```", "")
.trim() ?? "";
const sandbox = await Sandbox.create({
allowInternetAccess: false,
timeoutSecs: 600,
});
try {
await sandbox.run("pip", {
args: ["install", "numpy", "--user", "--break-system-packages"],
});
const execution = await sandbox.run("python3", {
args: ["-c", generatedCode],
});
return JSON.parse(execution.stdout) as ScoutReport;
} finally {
await sandbox.terminate();
}
}
const reports = await Promise.all(
["Scientific", "Economic", "Ethical"].map(scoutAgent),
);
console.log(reports);
sandboxes.close();
````
## Full example
This example simulates a Mars mission planning scenario where "scout" agents analyze different risks (Scientific, Economic, Ethical, etc.) by writing and running simulations in isolated sandboxes.
````python theme={null}
from dotenv import load_dotenv
load_dotenv() # Load environment variables from .env file
from tensorlake.sandbox import Sandbox
from pydantic import BaseModel
from typing import List
from openai import OpenAI
from concurrent.futures import ThreadPoolExecutor
class ScoutReport(BaseModel):
agent_id: int
raw_data: str
class FinalInsight(BaseModel):
summary: str
# 1. Worker Agent: LLM + Sandbox Execution
def scout_agent(task_id: int) -> ScoutReport:
"""Each scout analyzes a specific aspect of the mission."""
perspectives = ["Scientific", "Economic", "Ethical", "Logistical", "Psychological"]
perspective = perspectives[task_id % len(perspectives)]
print(f"🕵️ Scout {task_id}: Analyzing {perspective} perspective...")
client = OpenAI()
# Step A: LLM decides what to do
prompt = f"""
You are a {perspective} analyst for a Mars mission.
Write a Python script to perform a simple simulation using the 'numpy' library.
The simulation should model a key factor from your perspective (e.g., scientific sensor data, economic cost projection, logistical supply levels).
The script MUST print a single valid JSON string to standard output. This JSON should contain:
'perspective': '{perspective}',
'score': an integer from 0-100 derived from your simulation (higher is better),
'insight': a brief, unique risk or opportunity revealed by the simulation.
Do NOT use markdown blocks."""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}]
)
# Clean up markdown formatting (remove ```python ... ```)
generated_code = response.choices[0].message.content.replace("```python", "").replace("```", "").strip()
print(f"🕵️ Scout {task_id}: Generated code -> {generated_code}")
# Step B: Secure execution in a Sandbox
sandbox = Sandbox.create()
print(f"🕵️ Scout {task_id}: Installing dependencies in Sandbox...")
sandbox.run("pip", ["install", "numpy", "--user", "--break-system-packages"])
print(f"🕵️ Scout {task_id}: Running simulation in Sandbox...")
execution = sandbox.run("python3", ["-c", generated_code])
output = execution.stdout.strip()
print(f"🕵️ Scout {task_id}: Execution complete. Output: {output}")
return ScoutReport(agent_id=task_id, raw_data=output)
# 2. Lead Agent: LLM Aggregator (The "Reducer")
def lead_aggregator(reports: List[ScoutReport]) -> FinalInsight:
"""The Lead LLM reviews all sandbox outputs to find patterns."""
print(f"👑 Lead Agent: Received {len(reports)} scout reports. Aggregating...")
client = OpenAI()
combined_reports = "\n".join([f"Report {r.agent_id}: {r.raw_data}" for r in reports])
# The 'Intelligence' step: synthesizing multiple sources
prompt = (
f"You are the Mission Commander for Mars Colonization. Review these viability reports:\n{combined_reports}\n\n"
"1. Calculate the average viability score.\n"
"2. Synthesize a strategic Go/No-Go recommendation.\n"
"3. Summarize key risks."
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}]
)
return FinalInsight(summary=response.choices[0].message.content)
# 3. The Swarm Application
def intelligence_swarm(count: int) -> str:
print(f"🚀 Launching a swarm of {count} scouts...")
# Parallel Map: Launch multiple sandboxed scouts
with ThreadPoolExecutor() as executor:
reports = list(executor.map(scout_agent, range(count)))
# Reduce: Use the Lead Agent to combine results
final_insight = lead_aggregator(reports)
return final_insight.summary
if __name__ == "__main__":
# This runs 3 parallel LLMs, 3 parallel Sandboxes, and 1 Aggregator LLM
result = intelligence_swarm(count=5)
print(f"\n--- SWARM INTELLIGENCE REPORT ---\n{result}")
````
### Workflow: Step-by-Step Execution
| Step | Component | Action |
| ----- | ---------------- | -------------------------------------------------------------------------------------- |
| **1** | **Orchestrator** | Triggers 5 parallel scout tasks using the `scout_agent.map()` function. |
| **2** | **Scout Agent** | Leverages GPT-4o to draft a custom simulation script based on a specific perspective. |
| **3** | **Sandbox** | Securely installs `numpy`, handles dependencies, and executes the script in isolation. |
| **4** | **Scout Agent** | Compiles simulation data into a structured `ScoutReport` for return. |
| **5** | **Lead Agent** | Aggregates all reports and prompts GPT-4o for a final **Go/No-Go** decision. |
***
This example uses the `python-dotenv` library to load your Tensorlake API key from a `.env` file. Create a file named `.env` in your project root and add your key:
```
TENSORLAKE_API_KEY="your-api-key-here"
```
The SDK will automatically use this key.
## Production Tips
### Reduce Latency with Snapshots
The example above runs `pip install numpy` inside every scout's sandbox. In a real swarm with dozens of agents, this adds unnecessary latency and bandwidth usage.
For production, create a "base" sandbox, install your common dependencies, and create a **Snapshot**. Then, have your agents initialize from that snapshot instantly.
```python theme={null}
# 1. Create a snapshot ID (do this once)
# snapshot = sandbox.checkpoint()
# 2. Use it in your agent
sandbox = Sandbox.create(snapshot_id="snps_abc123")
# Numpy is already installed!
sandbox.run("python", ["-c", generated_code])
```
See the Snapshots guide for details.
### Security: Lock down the network
Since the scouts run code generated by an LLM, it is safer to disable internet access to prevent data exfiltration or malicious downloads.
```python theme={null}
sandbox = Sandbox.create(allow_internet_access=False)
# ...
```
## What to build next
Learn how to build a stateful code interpreter for a single agent.
Optimize your swarm's startup time by pre-baking dependencies.
# Async SDK (Python)
Source: https://docs.tensorlake.ai/sandboxes/async
Use the AsyncSandbox class to drive sandboxes from asyncio code.
The Python SDK ships an async-native variant of the sandbox API on top of asyncio. Every method on the sync [`Sandbox`](/sandboxes/sdk-reference#sandbox-handle) handle has a one-to-one async counterpart on `AsyncSandbox`: same names, same parameters, just `async def` and awaited.
## When to use it
Reach for the async API when:
* You're driving multiple sandboxes concurrently (e.g. fanning out work with `asyncio.gather`).
* Your application is already asyncio-based (FastAPI, aiohttp, an LLM agent loop, etc.) and you don't want to mix in blocking calls.
* You're streaming output from many processes at once.
If you only ever use one sandbox at a time and your code is otherwise synchronous, the sync `Sandbox` API is simpler and equivalent.
## The shape of the API
```python theme={null}
from tensorlake.sandbox import AsyncSandbox
```
`AsyncSandbox` is the runtime handle for a single sandbox. Use `await AsyncSandbox.create(...)` to provision and connect, or `await AsyncSandbox.connect(sandbox_id)` to attach to an existing one. Every instance method is awaited:
```python theme={null}
sandbox = await AsyncSandbox.create()
result = await sandbox.run("python", ["-c", "print('hello')"])
await sandbox.write_file("/workspace/data.csv", b"name,score\nAlice,95\n")
content = await sandbox.read_file("/workspace/data.csv")
```
Refer to the [SDK Reference](/sandboxes/sdk-reference) for the full method list; the names, parameters, and return types are identical to the sync API. The pages below walk through the same workflow with async syntax.
## Create and run
```python theme={null}
import asyncio
from tensorlake.sandbox import AsyncSandbox
async def main():
sandbox = await AsyncSandbox.create(cpus=2.0, memory_mb=2048)
try:
result = await sandbox.run("python", ["-c", "print('hello')"])
print(result.stdout)
finally:
await sandbox.terminate()
asyncio.run(main())
```
`AsyncSandbox` is also an async context manager. Use `async with` to terminate the sandbox automatically when the block exits:
```python theme={null}
async with await AsyncSandbox.create(cpus=2.0, memory_mb=2048) as sandbox:
result = await sandbox.run("python", ["-c", "print('hello')"])
print(result.stdout)
# sandbox is terminated here
```
## Run many sandboxes in parallel
The async API is designed for fan-out. Use `asyncio.gather` to start and run sandboxes concurrently:
```python theme={null}
import asyncio
from tensorlake.sandbox import AsyncSandbox
async def evaluate(prompt: str) -> str:
async with await AsyncSandbox.create(cpus=1.0, memory_mb=1024) as sandbox:
result = await sandbox.run("python", ["-c", prompt])
return result.stdout
async def main():
prompts = [
"print(2 + 2)",
"print(sum(range(100)))",
"import math; print(math.pi)",
]
outputs = await asyncio.gather(*(evaluate(p) for p in prompts))
for out in outputs:
print(out.strip())
asyncio.run(main())
```
Each `evaluate` call creates, executes against, and terminates its own sandbox in parallel with the others.
## Connect to an existing sandbox
Reattach to a named sandbox after `resume`, or operate on a sandbox another process created:
```python theme={null}
sandbox = await AsyncSandbox.connect("my-env")
info = await sandbox.info()
print(info.sandbox_id) # sandbox.sandbox_id is now populated too
```
Unlike the sync `Sandbox.sandbox_id` property, which transparently fetches
sandbox info on first access, the async `AsyncSandbox.sandbox_id` cannot
block on a network call. Call `await sandbox.info()` (or any other awaited
method that resolves the sandbox, like `status()`) once before reading
`sandbox.sandbox_id` on a freshly connected handle.
## Background processes and streaming output
Start a process, keep the handle, and collect its output once it finishes:
```python theme={null}
proc = await sandbox.start_process("python", ["-c", """
import time
for i in range(5):
print(f'tick {i}')
time.sleep(1)
"""])
print(proc.pid)
# follow_output blocks until the process exits, then returns a TracedIterator
# of the captured events you can iterate normally.
events = await sandbox.follow_output(proc.pid)
for event in events:
print(event.line, end="")
```
For long-running processes you want to stop yourself, send a signal directly. Don't `follow_output` first, since it would block waiting for the process to exit:
```python theme={null}
import signal
proc = await sandbox.start_process("python", ["-m", "http.server", "8080"])
# ... do work that talks to the server ...
await sandbox.send_signal(proc.pid, signal.SIGTERM)
```
## File operations
```python theme={null}
await sandbox.write_file("/workspace/data.csv", b"name,score\nAlice,95\n")
content = await sandbox.read_file("/workspace/data.csv")
print(content.value.decode("utf-8"))
listing = await sandbox.list_directory("/workspace")
for entry in listing.value.entries:
print(entry.name, entry.is_dir, entry.size)
```
## Suspend, resume, and snapshot
Suspend and resume require a named sandbox, so pass `name=` at creation time. `checkpoint` works on any sandbox, including ephemeral ones.
```python theme={null}
sandbox = await AsyncSandbox.create(name="my-env", cpus=1.0)
await sandbox.suspend()
await sandbox.resume()
snapshot = await sandbox.checkpoint()
restored = await AsyncSandbox.create(snapshot_id=snapshot.snapshot_id)
```
## Learn more
Full method list that applies to both sync and async APIs.
State machine, suspend/resume, timeouts.
Run commands, capture output, and manage background processes.
Capture and restore full VM state.
# Drive Chrome over CDP
Source: https://docs.tensorlake.ai/sandboxes/chrome-cdp
Run Google Chrome inside an ubuntu-vnc sandbox and drive it locally through the Chrome DevTools Protocol over a tunnel.
The `tensorlake/ubuntu-vnc` image ships with Google Chrome pre-installed. Combined with [Local Tunnels](/sandboxes/tunnels), this gives you a real, sandboxed Chrome that any DevTools-Protocol client (Playwright, Puppeteer, `chrome-remote-interface`, plain WebSocket) can drive from your laptop as if it were running locally: no headless container, no screenshot polling, no public port.
If you want to drive the whole XFCE desktop (mouse, keyboard, screenshots) instead of just Chrome, use the higher-level [Computer Use](/sandboxes/computer-use) API, which talks to the same `tensorlake/ubuntu-vnc` image through `connect_desktop()` / `connectDesktop()`. The two workflows compose: keep the agent loop on CDP and attach a human reviewer over VNC.
This guide walks through:
1. Launching `tensorlake/ubuntu-vnc`.
2. Starting Chrome with CDP enabled on the desktop session.
3. Tunneling the CDP port to `127.0.0.1`.
4. Driving the browser from Python or Playwright.
## Prerequisites
```bash theme={null}
curl -fsSL https://tensorlake.ai/install | sh
export TENSORLAKE_API_KEY=your-api-key
```
You can also use `tl login` to obtain a Personal Access Token interactively. The desktop password for the managed `tensorlake/ubuntu-vnc` image is `tensorlake`.
## 1. Launch the Sandbox
```bash theme={null}
tl sbx create -i tensorlake/ubuntu-vnc -c 4 -m 4096 chrome-cdp
```
`chrome-cdp` is a name (optional, but it lets you suspend and resume later). The CLI prints the new sandbox id; reuse it as `` below. Four CPUs and 4 GiB of RAM is a comfortable default for a single Chrome session.
## 2. Start Chrome with CDP Enabled
Start Chrome on the existing VNC display (`:1`) as the desktop user (`tl-user`). Two flags matter:
* `--remote-debugging-port=9222` opens the DevTools Protocol endpoint on `127.0.0.1:9222` inside the sandbox.
* `--remote-allow-origins=*` is required by current Chrome versions before they will accept a WebSocket whose `Origin` is anything other than the request host. Without it the HTTP `/json/version` endpoint works but `ws://127.0.0.1:9222/devtools/...` returns `403 Forbidden`.
```bash theme={null}
tl sbx exec -- bash -lc '
sudo -u tl-user bash -c "
nohup env DISPLAY=:1 XAUTHORITY=/home/tl-user/.Xauthority \
google-chrome \
--no-first-run \
--no-default-browser-check \
--remote-debugging-port=9222 \
--remote-allow-origins=* \
--user-data-dir=/tmp/chrome-cdp \
> /tmp/chrome-cdp.log 2>&1 &
disown
"
'
```
```python theme={null}
from tensorlake.sandbox import Sandbox
with Sandbox.connect("") as sandbox:
sandbox.start_process(
"sudo",
args=[
"-u", "tl-user",
"env",
"DISPLAY=:1",
"XAUTHORITY=/home/tl-user/.Xauthority",
"google-chrome",
"--no-first-run",
"--no-default-browser-check",
"--remote-debugging-port=9222",
"--remote-allow-origins=*",
"--user-data-dir=/tmp/chrome-cdp",
],
)
```
`start_process` returns immediately and the sandbox daemon keeps Chrome alive: no `nohup`, no shell, no log redirection. Stdout and stderr are captured by the daemon and reachable via `sandbox.get_stdout(pid)` / `sandbox.get_stderr(pid)` if you want to inspect them.
```javascript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.connect({ sandboxId: "" });
await sandbox.startProcess("sudo", {
args: [
"-u", "tl-user",
"env",
"DISPLAY=:1",
"XAUTHORITY=/home/tl-user/.Xauthority",
"google-chrome",
"--no-first-run",
"--no-default-browser-check",
"--remote-debugging-port=9222",
"--remote-allow-origins=*",
"--user-data-dir=/tmp/chrome-cdp",
],
});
```
`startProcess` returns once the daemon has spawned the child; Chrome keeps running in the background. Read its output later with `sandbox.getStdout(pid)` / `sandbox.getStderr(pid)`.
Confirm CDP is up:
```bash theme={null}
tl sbx exec -- bash -lc 'curl -s http://127.0.0.1:9222/json/version'
```
You should see a JSON response with `Browser`, `Protocol-Version`, and `webSocketDebuggerUrl`.
Because Chrome is running on the VNC display `:1`, you can also attach a VNC viewer through the [Local Tunnels](/sandboxes/tunnels) workflow and watch it operate in real time. CDP control and human observation can run side by side.
## 3. Open a Tunnel
Forward `127.0.0.1:9222` on your laptop to `127.0.0.1:9222` inside the sandbox:
```bash theme={null}
tl sbx tunnel 9222
```
Leave the command running. Open a second terminal for the rest of this guide.
```javascript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.connect({ sandboxId: "" });
const tunnel = await sandbox.createTunnel(9222, { localPort: 9222 });
console.log(`CDP at http://127.0.0.1:${tunnel.address().port}`);
// ... drive the browser ...
await tunnel.close();
```
Verify locally:
```bash theme={null}
curl http://127.0.0.1:9222/json/version
```
Same JSON, but reached from your laptop. Every byte transits an authenticated WebSocket, so port `9222` never has to be in `exposed_ports`.
## 4. Drive the Browser
### Open a Tab
CDP exposes an HTTP control surface on the same port. Open a fresh tab with a `PUT`:
```bash theme={null}
curl -X PUT "http://127.0.0.1:9222/json/new?https://news.ycombinator.com"
```
The response includes a `webSocketDebuggerUrl` for the new tab. List all tabs with `curl http://127.0.0.1:9222/json/list` and close one with `curl http://127.0.0.1:9222/json/close/`.
### Playwright
```python theme={null}
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.connect_over_cdp("http://127.0.0.1:9222")
context = browser.contexts[0]
page = context.new_page()
page.goto("https://news.ycombinator.com")
titles = page.locator(".titleline > a").all_text_contents()
print(titles[:5])
```
```javascript theme={null}
import { chromium } from "playwright";
const browser = await chromium.connectOverCDP("http://127.0.0.1:9222");
const [context] = browser.contexts();
const page = await context.newPage();
await page.goto("https://news.ycombinator.com");
const titles = await page.locator(".titleline > a").allTextContents();
console.log(titles.slice(0, 5));
```
### Raw CDP via WebSocket
When you want to issue protocol calls directly (`Runtime.evaluate`, `Page.navigate`, `DOM.getDocument`), connect to the per-tab WebSocket and exchange JSON messages:
```python theme={null}
import json
import urllib.request
import websocket # pip install websocket-client
targets = json.loads(urllib.request.urlopen("http://127.0.0.1:9222/json/list").read())
page = next(t for t in targets if t["type"] == "page")
ws = websocket.create_connection(page["webSocketDebuggerUrl"])
ws.send(json.dumps({
"id": 1,
"method": "Runtime.evaluate",
"params": {
"expression": "document.title",
"returnByValue": True,
},
}))
print(json.loads(ws.recv())["result"]["result"]["value"])
ws.close()
```
This is also the path you take when wiring CDP into an LLM agent: expose `open_url`, `evaluate`, and `list_targets` as tools that wrap these calls.
### Coding Agents (`chrome-devtools` MCP)
Claude Code and OpenAI Codex can both drive the same sandboxed Chrome through the official [`chrome-devtools-mcp`](https://github.com/ChromeDevTools/chrome-devtools-mcp) server. The MCP attaches to an existing Chrome via `--browser-url`; match that URL to the tunnel's local port and no other configuration is needed. Using Chrome's canonical `9222` on both sides keeps everything default-on-default.
Register the MCP once for your user:
```bash theme={null}
claude mcp add chrome-devtools -- npx chrome-devtools-mcp@latest \
--browser-url http://127.0.0.1:9222
```
Stored at user scope by default. Pass `--scope project` to write it to the current project's `.mcp.json` instead.
```bash theme={null}
codex mcp add chrome-devtools -- npx chrome-devtools-mcp@latest \
--browser-url http://127.0.0.1:9222
```
Writes to `~/.codex/config.toml` (or `$CODEX_HOME/config.toml`). Codex has no project-vs-user scope. The file is always user-global. The equivalent block, if you prefer to edit the file by hand:
```toml theme={null}
[mcp_servers.chrome-devtools]
command = "npx"
args = ["chrome-devtools-mcp@latest", "--browser-url", "http://127.0.0.1:9222"]
```
The `--browser-url` flag is what tells the MCP to attach to an existing Chrome instead of launching its own.
With Chrome already running inside the sandbox (step 2) and a tunnel open at the default local port:
```bash theme={null}
tl sbx tunnel 9222
```
restart the agent so it picks up the new MCP (Claude Code re-reads on launch; Codex reads `config.toml` at startup and does not hot-reload), then ask it to do something in the browser:
```
> open https://news.ycombinator.com and read the first headline
```
The agent routes that through `chrome-devtools` → `127.0.0.1:9222` → tunnel → sandbox Chrome on display `:1`.
If port `9222` is already taken on your laptop (a local Chrome with debugging on, another tunnel, etc.), pick any free port for both sides and keep them aligned:
```bash theme={null}
# tunnel the sandbox's 9222 to local 12222
tl sbx tunnel 9222 --listen-port 12222
# point the MCP at the same local port
claude mcp add chrome-devtools -- npx chrome-devtools-mcp@latest \
--browser-url http://127.0.0.1:12222
```
```bash theme={null}
# tunnel the sandbox's 9222 to local 12222
tl sbx tunnel 9222 --listen-port 12222
# point the MCP at the same local port
codex mcp add chrome-devtools -- npx chrome-devtools-mcp@latest \
--browser-url http://127.0.0.1:12222
```
Verify the path before you point an agent at it: `curl http://127.0.0.1:9222/json/version` should return Chrome's JSON. The tunnel CLI keeps the local port bound even when the sandbox upstream goes away (terminated, suspended without auto-resume), so a hung `curl` usually means the sandbox is gone, not that the MCP is misconfigured.
## 5. Tear Down
Stop the tunnel with `Ctrl+C`. Stop Chrome inside the sandbox when you no longer need it:
```bash theme={null}
tl sbx exec -- bash -lc 'sudo -u tl-user pkill -f google-chrome || true'
```
Suspend the sandbox to keep the user-data-dir warm for next time, or terminate it to release resources:
```bash theme={null}
tl sbx suspend # named sandboxes only
tl sbx terminate
```
## Notes and Pitfalls
* **`--remote-allow-origins=*` is required** for Chrome ≥ 111. Without it, the HTTP CDP endpoints work but every WebSocket handshake fails with `403`. Restart Chrome with the flag if you forget.
* **Bind address.** `--remote-debugging-port` only listens on `127.0.0.1` by default, which is exactly what you want: the tunnel forwards to `127.0.0.1` inside the sandbox, so DevTools stays unreachable from anywhere else.
* **`--user-data-dir` is required for CDP.** Chrome ≥ 136 refuses to enable `--remote-debugging-port` against the default profile and prints `DevTools remote debugging requires a non-default data directory. Specify this using --user-data-dir.` to its log. Always pass `--user-data-dir=/tmp/` (or any path other than `~/.config/google-chrome`).
* **Headless mode.** If you do not need the VNC view, you can launch with `--headless=new` instead of attaching to display `:1`. The tunneling and CDP usage remain identical.
* **Sandboxing inside containers.** Chrome's setuid sandbox sometimes fails inside container/VM combinations. If you see `Failed to move to new namespace` errors, add `--no-sandbox` to the launch flags.
* **Multiple agents.** Each tab has its own `webSocketDebuggerUrl`. Two clients can drive different tabs of the same Chrome at the same time, which is useful when an agent loop and a human reviewer both want a window.
## Related Guides
* [Computer Use](/sandboxes/computer-use): drive the full XFCE desktop (mouse, keyboard, screenshots) on the same `tensorlake/ubuntu-vnc` image.
* [Local Tunnels](/sandboxes/tunnels): the tunneling primitive that carries CDP traffic from your laptop into the sandbox.
* [Snapshots](/sandboxes/snapshots): fork a warmed-up Chrome profile so parallel agents start with cookies, history, and extensions already in place.
# CICD & Build Systems
Source: https://docs.tensorlake.ai/sandboxes/cicd-build
Execute build steps and run tests in isolated, reproducible environments.
Build systems and CI/CD pipelines often require clean, isolated environments to ensure reproducibility and prevent dependency conflicts. Tensorlake Sandboxes allow you to spin up ephemeral containers on demand, upload source code, run tests, and retrieve artifacts.
This example demonstrates a complete mini-CI pipeline that creates a dummy project, runs tests, and builds a distribution package inside a sandbox.
## TypeScript SDK starter
If your build runner is already in Node.js, the workflow is the same: stream project files into a sandbox, run each CI step, and pull artifacts back out if needed.
```typescript theme={null}
import { readFile, readdir } from "node:fs/promises";
import path from "node:path";
import { Sandbox } from "tensorlake";
async function copyTree(sandbox: Sandbox, localDir: string, remoteDir: string) {
await sandbox.run("mkdir", { args: ["-p", remoteDir] });
for (const entry of await readdir(localDir, { withFileTypes: true })) {
const localPath = path.join(localDir, entry.name);
const remotePath = path.posix.join(remoteDir, entry.name);
if (entry.isDirectory()) {
await copyTree(sandbox, localPath, remotePath);
} else {
await sandbox.writeFile(remotePath, await readFile(localPath));
}
}
}
async function runStep(
sandbox: Sandbox,
name: string,
command: string,
args: string[],
) {
const result = await sandbox.run(command, {
args,
workingDir: "/workspace/project",
});
if (result.exitCode !== 0) {
throw new Error(`${name} failed\n${result.stderr}`);
}
}
const sandbox = await Sandbox.create({ timeoutSecs: 900 });
try {
await copyTree(sandbox, "./my_cool_project", "/workspace/project");
await runStep(sandbox, "Install Dependencies", "pip", [
"install",
"-r",
"requirements.txt",
"--user",
"--break-system-packages",
]);
await runStep(sandbox, "Run Tests", "python", [
"-m",
"pytest",
"/workspace/project",
]);
await runStep(sandbox, "Build Package", "python", [
"setup.py",
"sdist",
"bdist_wheel",
]);
} finally {
await sandbox.terminate();
client.close();
}
```
## Example: CI/CD Pipeline
The following script simulates a CI pipeline. It generates a simple Python project, uploads it to a sandbox, installs dependencies, runs `pytest`, and builds a wheel file.
```python theme={null}
import os
import tempfile
import shutil
from dotenv import load_dotenv
from setuptools import setup, find_packages
load_dotenv()
from tensorlake.sandbox import Sandbox
def create_dummy_project(base_dir):
"""
Helper to create a simple Python project layout locally
so we have something to build/test.
"""
project_root = os.path.join(base_dir, "my_cool_project")
src_dir = os.path.join(project_root, "src", "my_cool_project")
tests_dir = os.path.join(project_root, "tests")
os.makedirs(src_dir, exist_ok=True)
os.makedirs(tests_dir, exist_ok=True)
# Create setup.py
with open(os.path.join(project_root, "setup.py"), "w") as f:
f.write("from setuptools import setup, find_packages\n"
"setup(name='my_cool_project', version='0.1.0', "
"package_dir={'': 'src'}, packages=find_packages(where='src'))")
# Create source code
with open(os.path.join(src_dir, "__init__.py"), "w") as f:
f.write("def add(a, b): return a + b")
# Create test
with open(os.path.join(tests_dir, "test_logic.py"), "w") as f:
f.write("from my_cool_project import add\ndef test_add(): assert add(2, 3) == 5")
# Create requirements.txt
with open(os.path.join(project_root, "requirements.txt"), "w") as f:
f.write("pytest\nsetuptools\nwheel\n")
return project_root
def run_ci_step(sandbox, name, command, working_dir="/workspace/project", env=None):
"""Runs a command in the sandbox, prints the output, and checks for errors."""
print(f"--- Running Step: {name} ---")
result = sandbox.run(command[0], command[1:], env=env, working_dir=working_dir)
print(f"STDOUT:\n{result.stdout}")
if result.stderr:
print(f"STDERR:\n{result.stderr}")
if result.exit_code != 0:
print(f"❌ Step '{name}' FAILED with exit code {result.exit_code}")
raise RuntimeError(f"CI step '{name}' failed.")
else:
print(f"✅ Step '{name}' PASSED")
print("-" * (len(name) + 20))
def copy_to_sandbox(sandbox, local_path, remote_path):
"""Recursively copies a local directory to the sandbox."""
print(f"Copying {local_path} -> {remote_path} ...")
sandbox.run("mkdir", ["-p", remote_path])
for root, dirs, files in os.walk(local_path):
rel_root = os.path.relpath(root, local_path)
remote_root = remote_path if rel_root == "." else os.path.join(remote_path, rel_root)
for d in dirs:
sandbox.run("mkdir", ["-p", os.path.join(remote_root, d)])
for file in files:
local_file = os.path.join(root, file)
remote_file = os.path.join(remote_root, file)
with open(local_file, "rb") as f:
sandbox.write_file(remote_file, f.read())
async def main():
# 1. Setup local dummy project
temp_dir = tempfile.mkdtemp()
project_path = create_dummy_project(temp_dir)
print(f"Dummy project created at: {project_path}")
try:
# 2. Create Sandbox
sandbox = Sandbox.create()
print("🚀 Sandbox created for CI/CD pipeline.")
# 3. Upload Code
copy_to_sandbox(sandbox, project_path, "/workspace/project")
# 4. Install Dependencies
run_ci_step(
sandbox,
"Install Dependencies",
["pip", "install", "-r", "requirements.txt", "--user", "--break-system-packages"],
working_dir="/workspace/project"
)
# 5. Run Tests
run_ci_step(
sandbox,
"Run Tests",
["python", "-m", "pytest", "/workspace/project"],
env={"PYTHONPATH": "/workspace/project/src"},
)
# 6. Build Artifacts
run_ci_step(sandbox, "Build Package", ["python", "setup.py", "sdist", "bdist_wheel"])
print("\n🎉 CI/CD Pipeline finished successfully! 🎉")
except Exception as e:
print(f"\n🔥 CI/CD Pipeline FAILED: {e}")
finally:
shutil.rmtree(temp_dir)
if __name__ == "__main__":
import asyncio
asyncio.run(main())
```
## How It Works
1. **Environment Creation**: The script instantiates a fresh sandbox. This ensures no leftover files or environment variables from previous builds affect the current run.
2. **File Injection**: The custom `copy_to_sandbox` function walks the local directory tree and streams files into the sandbox using `sandbox.write_file()`. This simulates the "checkout" phase of a CI pipeline.
3. **Step Execution**: The `run_ci_step` helper function executes shell commands (like `pip` and `pytest`) inside the sandbox using `sandbox.run()`. It captures `stdout`, `stderr`, and exit codes to determine success or failure.
4. **Artifact Generation**: The build step generates `.whl` and `.tar.gz` files inside the sandbox. In a real-world scenario, you would use `sandbox.read_file()` to download these artifacts back to your storage.
## Learn More
Learn how to efficiently move large files and directories in and out of sandboxes.
Run commands, manage long-running processes, and handle exit codes.
# Run Claude Managed Agents on Tensorlake Sandboxes
Source: https://docs.tensorlake.ai/sandboxes/claude-managed-agents
Use Tensorlake sandboxes as the self-hosted execution environment for Claude Managed Agents: Anthropic runs the agent loop, every tool call runs in a sandbox you control.
Run Anthropic's agent loop on Anthropic's infrastructure, and run every tool call inside a Tensorlake sandbox you control.
## The model: brain vs. hands
Claude Managed Agents splits an agent into two halves. **Claude is the brain**: the LLM, the agent loop, session state, and the work queue all live on Anthropic's infrastructure. It decides *which* tool to call but never executes one. **The sandbox is the hands**: every `bash`, `read`, `write`, `edit`, `glob`, `grep` call actually runs inside an execution environment you own.
A Claude *Environment* with hosting type **Self-hosted** doesn't run tools itself. Instead it places a work item on a queue. Your **orchestrator** consumes that queue and turns each session into a Tensorlake sandbox. The sandbox runs a thin worker that attaches back to Anthropic and executes tool calls for the life of the session.
```mermaid theme={null}
flowchart LR
app["Your app (Anthropic SDK)"] <-->|events| anthropic["Anthropic
agent loop · session state · work queue"]
anthropic -->|"work item per session run"| orch["Orchestrator
drain queue → sandbox per session"]
orch -->|"Sandbox.create + start_process"| sbx["Tensorlake sandbox
per session"]
sbx -->|"worker attaches back · executes tool calls"| anthropic
```
## Why Tensorlake for this role
* **Fast boot / sub-second wake.** An agent loop is a tight decide→execute→decide cycle: many short tool calls (`bash`, `read`, `grep`) separated by model think-time. A suspended sandbox resumes from its memory snapshot in **\~0.6s** (a restore, not a cold app boot), so the hands are ready the instant the brain calls a tool. Low per-tool-call latency without keeping a sandbox warm between turns, which is what makes Tensorlake a good home for executing the agent loop's tool calls.
* **Snapshots & fork-from-snapshot.** `sandbox.checkpoint()` captures filesystem (or full memory) state; `Sandbox.create(snapshot_id=...)` boots a fresh sandbox from it. Run it N times → N children exploring in parallel from one known-good state. Basis for best-of-N tool execution, parallel sub-agents, and retry-with-divergence.
* **Suspend / resume.** Named sandboxes suspend when idle and resume with state intact: no cold-start on every turn, and the basis for the wake-on-request trick below.
* **Public port exposure.** `expose_ports(...)` serves a process inside the sandbox at `https://{port}-{id}.sandbox.tensorlake.ai` with TLS terminated by Tensorlake's proxy, so you don't need a reverse proxy of your own.
## Three orchestrator modes
The orchestrator's job is identical in all three (`orchestrator_lib.py`: get-or-create a sandbox per session, drain the queue). Only *where it runs* differs:
| Mode | Where it runs | Spawn latency | Needs |
| ------------------------------------ | -------------------------------------------------- | ------------------------- | ----------------------------------------------- |
| **Webhook-in-sandbox** (recommended) | Inside a Tensorlake sandbox, port exposed publicly | Sub-second, scale-to-zero | Nothing running on your side (wakes on request) |
| **Polling** | Your machine / server | Seconds | A long-running host process |
| **Webhook** | Your machine / server | \~Instant | A public HTTPS endpoint + TLS |
Run exactly one orchestrator per `ANTHROPIC_ENVIRONMENT_ID`.
## Webhook-in-sandbox: scale-to-zero push
In the recommended mode the FastAPI receiver runs *inside* a Tensorlake sandbox, with port 5051 exposed at a public HTTPS URL. Anthropic pushes webhooks directly to Tensorlake: no host process, no TLS of yours. The sandbox is created with a short idle timeout, so when no inbound traffic arrives it suspends (memory and the running uvicorn process preserved). The next inbound webhook resumes it automatically.
Measured: a confirmed-`suspended` sandbox served `GET /healthz` in \~0.6s and flipped to `running` (a memory-snapshot restore, not a cold app boot). Outbound polling from inside the sandbox does *not* keep it awake, so it still suspends on schedule. The result is push latency with nothing running and nothing billed while idle, which suits bursty workloads better than a long-running host process.
## Long-running sessions: resume instead of recreate
The same suspend/resume primitive applies one layer down, to the **per-session** sandbox. A long-running agent is rarely busy throughout: it's bursts of work separated by idle gaps (waiting on a human approval, a slow CI job, or think-time between turns). At `SANDBOX_TIMEOUT_SECONDS` an idle per-session sandbox auto-suspends: filesystem and processes frozen to a snapshot, nothing billed.
The question on the next burst is whether you **rebuild or restore**:
| | Recreate from base (default) | Resume the suspended sandbox |
| --------------------- | ------------------------------------------------ | ----------------------------------------------- |
| Idle cost | Zero (suspended) | Zero (suspended), identical |
| Resume latency | Full create + image pull + setup | Memory-snapshot restore, sub-second |
| Session working state | Lost: re-clone repo, re-install deps, redo setup | Intact: `/workspace`, deps, warm caches as left |
| Best for | Independent, cheap-to-setup bursts | A session whose accumulated state *is* the work |
In the reference code this is one decision in `orchestrator_lib.py`: `_find_live_sandbox` treats a suspended sandbox as not-live and recreates it. Set `RESUME_SUSPENDED_SESSIONS=true` and it instead connects to the suspended sandbox; the runner relaunch is the inbound op that resumes it (the relaunch path already exists for the max-idle case). The clean-slate default stays available because a fresh sandbox can't inherit a half-broken state from a previous burst: resume is opt-in for when continuity matters more than a guaranteed clean slate.
## Quickstart
The full, working integration (with the exact credentials, the Console-only steps, mode-by-mode setup, verification, and troubleshooting) lives in the [reference repo's `examples/managed-agent` README](https://github.com/tensorlakeai/claude-managed-agents-tensorlake-sandbox/blob/main/examples/managed-agent/README.md). That README is the source of truth; this section is just the shape of it so you know what you're signing up for.
The setup has four stages:
1. **Configure**: `cd examples/managed-agent`, copy `.env.example`→`.env` and `.env.local.example`→`.env.local`, then `uv sync`.
2. **Tensorlake**: set `TENSORLAKE_API_KEY` in `.env`, `uv run tl login`, then `make build` to build the per-session sandbox image. (Keep the SDK key and the `tl login` session pointed at the *same* project. See the gotchas below.)
3. **Claude Platform**: in a non-default workspace: create an Agent (`make agent`), create a **Self-hosted** Environment, and generate its environment key (Console-only). This populates `ANTHROPIC_API_KEY` + `ANTHROPIC_AGENT_ID` in `.env.local` and `ANTHROPIC_ENVIRONMENT_ID` + `ANTHROPIC_ENVIRONMENT_KEY` in `.env`.
4. **Orchestrator**: pick one mode. For webhook-in-sandbox: `make build-webhook`, register the printed URL as a Webhook (`Session lifecycle → Run started`) with its signing secret in `ANTHROPIC_WEBHOOK_SIGNING_KEY`, *then* `make webhook-sandbox` (the secret is baked in at launch, so it must be set first).
Then drive a session from anywhere:
```bash theme={null}
make session PROMPT="create hello.txt with 'hi' then read it back"
```
Success looks like a stream of `running` / `thinking` / tool calls (`→ write`, `→ read`) ending in `· done`. If a session never streams, the README's `work.stats` check and troubleshooting section diagnose the common causes (import-order 401, mismatched Tensorlake projects, `workers_polling: 0` in webhook modes).
## Injecting credentials and naming sandboxes
Two Tensorlake SDK facts shape how the orchestrator passes secrets and names sandboxes:
* \*\*Inject env vars per command, not on create. Pass every credential and per-session var (`ANTHROPIC_ENVIRONMENT_KEY`, and the session, work, and environment IDs) via `start_process(env={...})`, which merges on top of the sandbox base environment.
* **Sandbox names must be slugs**: lowercase letters, digits, and hyphens only. Derive a sandbox name from a session id by slugifying it (e.g. `agent-`).
Setup footguns specific to the reference code (`.env` precedence, the order in which you import the SDK relative to loading credentials, and keeping the Python SDK key and the `tl` CLI session on the same project) are covered in the [repo README's troubleshooting section](https://github.com/tensorlakeai/claude-managed-agents-tensorlake-sandbox/blob/main/examples/managed-agent/README.md#troubleshooting).
## When you want parallelism
The [parallel sub-agents example](/applications/parallel-sub-agents) forks N sandboxes from one snapshot so a single agent session can explore N candidate solutions at once: `parent.checkpoint()` then `Sandbox.create(snapshot_id=...)` × N. Expose it to the agent as a bundled CLI helper (callable via `bash`), an MCP tool, or an orchestrator-side action gated by session metadata.
## Next steps
The full integration: image build, in-sandbox worker, and one orchestrator in three runnable modes.
Anthropic's docs for the agent harness, sessions, and self-hosted environments.
The suspend/resume and snapshot model that powers wake-on-request.
Fork N sandboxes from one snapshot for best-of-N tool execution.
# Commands & Processes
Source: https://docs.tensorlake.ai/sandboxes/commands
Run commands, capture and stream output, and start, manage, and signal background processes
Run shell commands inside sandboxes with full stdout/stderr capture, real-time streaming, and configurable timeouts. Then keep long-running services alive with background processes, signals, stdin, and supervised restarts.
Commands and processes share one API. Use `run` / `exec` when you want to **run something and wait for the result**; use `start_process` when you want to **start something that keeps running** and manage its lifecycle.
Use [Sandbox Process Logs](/sandboxes/process-logs) when you want to browse, search, and filter retained stdout/stderr across processes in the Console.
Sandbox-specific operations use sandbox-specific ingress, commonly seen as `https://.sandbox.tensorlake.ai`, derived from the sandbox's `ingress_endpoint`.
For named sandboxes, you can use the sandbox **name** in place of the ID, both in the proxy hostname and in CLI/API commands. For example, `https://my-env.sandbox.tensorlake.ai/api/v1/processes` and `tl sbx exec my-env python main.py` work the same as their ID-based equivalents. The proxy resolves the name to the underlying sandbox automatically.
The command and process APIs documented here run on the management URL on port `9501`, which always requires authentication. Unauthenticated proxy access applies only to exposed user ports.
## Basic Execution
```bash theme={null}
# Run in an existing sandbox: use the sandbox ID or name
tl sbx exec my-env python -c 'print("Hello from sandbox!")'
# Or create, run, and tear down in one step
tl sbx run python -c 'print("Hello from sandbox!")'
```
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create()
result = sandbox.run("python", ["-c", "print('Hello from sandbox!')"])
print(result.stdout) # Hello from sandbox!
print(result.exit_code) # 0
```
```typescript theme={null}
const result = await sandbox.run("python", {
args: ["-c", "print('Hello from sandbox!')"],
});
console.log(result.stdout); // Hello from sandbox!
console.log(result.exitCode); // 0
```
```bash theme={null}
# Start a Python process inside the sandbox
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"command": "python",
"args": ["-c", "print(\"Hello from sandbox!\")"]
}'
```
**Response:**
```json theme={null}
{
"pid": 294,
"status": "running",
"exit_code": null,
"signal": null,
"stdin_writable": false,
"command": "python",
"args": ["-c", "print(\"Hello from sandbox!\")"],
"started_at": 1773950042728,
"ended_at": null
}
```
## CLI Options
```bash theme={null}
# Timeout is in seconds
tl sbx exec --timeout 10 python -c 'print("hi")'
# Run from a specific working directory
tl sbx exec --workdir /workspace python main.py
# Inject environment variables into a single command
tl sbx exec --env MODE=prod --env DEBUG=0 /bin/sh -lc 'printf "%s %s\n" "$MODE" "$DEBUG"'
# Keep the sandbox after a one-shot run so you can inspect it afterwards
tl sbx run --keep /bin/sh -lc 'echo KEEP_TEST && sleep 1'
```
A verified `--env` run printed `prod 0`. A verified `--keep` run ended with `Sandbox kept alive.`, and `tl sbx ls --all` then showed that sandbox as `running`.
```python theme={null}
# Run with a timeout, working directory, and per-command environment
result = sandbox.run(
"python",
["main.py"],
env={"MODE": "prod", "DEBUG": "0"},
working_dir="/workspace",
timeout=10,
)
```
```typescript theme={null}
const result = await sandbox.run("python", {
args: ["main.py"],
env: { MODE: "prod", DEBUG: "0" },
workingDir: "/workspace",
timeout: 10,
});
console.log(result.exitCode);
```
```bash theme={null}
# Start a process with custom environment variables and working directory
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"command": "python",
"args": ["main.py"],
"env": {"MODE": "prod", "DEBUG": "0"},
"working_dir": "/workspace"
}'
```
## Shell Commands
```bash theme={null}
# Count Python files in /workspace with a pipe
tl sbx exec bash -c "ls -la /workspace | grep '.py' | wc -l"
# Redirect stdout and stderr to files
tl sbx exec bash -c "python script.py > output.txt 2> errors.txt"
# Chain setup and execution in one shell command
tl sbx exec bash -c "cd /workspace && pip install -r requirements.txt && python main.py"
```
```python theme={null}
# Pipes
result = sandbox.run("bash", ["-c", "ls -la /workspace | grep '.py' | wc -l"])
print(result.stdout)
# Redirects
sandbox.run("bash", ["-c", "python script.py > output.txt 2> errors.txt"])
# Command chaining
sandbox.run("bash", ["-c", "cd /workspace && pip install -r requirements.txt && python main.py"])
```
```typescript theme={null}
const count = await sandbox.run("bash", {
args: ["-lc", "ls -la /workspace | grep '.py' | wc -l"],
});
console.log(count.stdout);
await sandbox.run("bash", {
args: ["-lc", "python script.py > output.txt 2> errors.txt"],
});
await sandbox.run("bash", {
args: [
"-lc",
"cd /workspace && pip install -r requirements.txt && python main.py",
],
});
```
```bash theme={null}
# Use bash when you need pipes, redirects, or command chaining
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"command": "bash",
"args": ["-c", "ls -la /workspace | grep .py | wc -l"]
}'
```
## Get Process Output
Fetch the buffered stdout, stderr, or combined output of a running or finished process.
```python theme={null}
sandbox = Sandbox.create()
result = sandbox.run("python", ["-c", "print('hello')"])
print(result.stdout)
print(result.stderr)
```
```typescript theme={null}
import { ProcessStatus } from "tensorlake";
const proc = await sandbox.startProcess("python", {
args: [
"-c",
"import sys; print('hello'); print('oops', file=sys.stderr)",
],
});
let info = await sandbox.getProcess(proc.pid);
while (info.status === ProcessStatus.RUNNING) {
await new Promise((resolve) => setTimeout(resolve, 100));
info = await sandbox.getProcess(proc.pid);
}
console.log((await sandbox.getStdout(proc.pid)).lines);
console.log((await sandbox.getStderr(proc.pid)).lines);
console.log((await sandbox.getOutput(proc.pid)).lines);
```
```bash theme={null}
# Get stdout
curl https://.sandbox.tensorlake.ai/api/v1/processes//stdout \
-H "Authorization: Bearer $TL_API_KEY"
# Get stderr
curl https://.sandbox.tensorlake.ai/api/v1/processes//stderr \
-H "Authorization: Bearer $TL_API_KEY"
# Get combined output
curl https://.sandbox.tensorlake.ai/api/v1/processes//output \
-H "Authorization: Bearer $TL_API_KEY"
```
**Combined output response:**
```json theme={null}
{
"pid": 297,
"lines": ["hello", "oops"],
"line_count": 2
}
```
Not supported in the CLI.
## Error Handling
```bash theme={null}
# The CLI prints stderr and returns a non-zero exit code on failure
tl sbx exec python -c "import nonexistent_module"
```
```python theme={null}
sandbox = Sandbox.create()
result = sandbox.run("python", ["-c", "import nonexistent_module"])
if result.exit_code != 0:
print(f"Command failed with exit code {result.exit_code}")
print(f"stderr: {result.stderr}")
```
```typescript theme={null}
const result = await sandbox.run("python", {
args: ["-c", "import nonexistent_module"],
});
if (result.exitCode !== 0) {
console.log(`Command failed with exit code ${result.exitCode}`);
console.log(`stderr: ${result.stderr}`);
}
```
```bash theme={null}
# Check the exited process status
curl https://.sandbox.tensorlake.ai/api/v1/processes/ \
-H "Authorization: Bearer $TL_API_KEY"
```
**Process status response:**
```json theme={null}
{
"pid": 305,
"status": "exited",
"exit_code": 1,
"signal": null,
"stdin_writable": false,
"command": "python",
"args": ["-c", "import nonexistent_module"],
"started_at": 1773950228855,
"ended_at": 1773950228866
}
```
**stderr response:**
```json theme={null}
{
"pid": 305,
"lines": [
"Traceback (most recent call last):",
" File \"\", line 1, in ",
"ModuleNotFoundError: No module named 'nonexistent_module'"
],
"line_count": 3
}
```
## Start a Background Process
Use `start_process` for work that should keep running after the call returns: servers, watchers, and other long-lived jobs.
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create()
# Start a background process
proc = sandbox.start_process("python", ["-m", "http.server", "8080"])
print(f"PID: {proc.pid}")
```
```typescript theme={null}
const proc = await sandbox.startProcess("python", {
args: ["-m", "http.server", "8080"],
});
console.log(`PID: ${proc.pid}`);
```
```bash theme={null}
# Run a command in the background using shell syntax
tl sbx exec bash -c "python -m http.server 8080 &"
```
```bash theme={null}
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"command": "python",
"args": ["-m", "http.server", "8080"]
}'
```
**Response** (`201 Created`):
```json theme={null}
{
"pid": 42,
"status": "running",
"command": "python",
"args": ["-m", "http.server", "8080"],
"stdin_writable": false,
"started_at": 1710000000000
}
```
**Request options:**
```json theme={null}
{
"command": "python",
"args": ["-m", "http.server", "8080"],
"env": {"PORT": "8080"},
"working_dir": "/workspace",
"stdin_mode": "pipe",
"stdout_mode": "capture",
"stderr_mode": "capture"
}
```
## List Processes
```python theme={null}
procs = sandbox.list_processes()
for p in procs:
print(f"PID {p.pid}: {p.status}")
```
```typescript theme={null}
const processes = await sandbox.listProcesses();
for (const proc of processes) {
console.log(`PID ${proc.pid}: ${proc.status}`);
}
```
```bash theme={null}
curl https://.sandbox.tensorlake.ai/api/v1/processes \
-H "Authorization: Bearer $TL_API_KEY"
```
**Response:**
```json theme={null}
{
"processes": [
{
"pid": 42,
"status": "running",
"command": "python",
"args": ["-m", "http.server", "8080"],
"stdin_writable": false,
"started_at": 1710000000000
}
]
}
```
## Stream Output
Stream stdout/stderr in real time for long-running commands using Server-Sent Events:
`tl sbx exec` streams combined output to your terminal while the process runs.
```python theme={null}
# Start a long-running process
sandbox = Sandbox.create()
proc = sandbox.start_process("python", ["-c", """
import time
for i in range(5):
print(f"Step {i+1}/5")
time.sleep(1)
"""])
# Stream output as it arrives
for event in sandbox.follow_output(proc.pid):
print(event.line, end="")
```
```typescript theme={null}
const proc = await sandbox.startProcess("python", {
args: [
"-c",
"import time\nfor i in range(5):\n print(f'Step {i+1}/5')\n time.sleep(1)",
],
});
for await (const event of sandbox.followOutput(proc.pid)) {
process.stdout.write(event.line);
}
```
```bash theme={null}
# Follow stdout via SSE
curl -N https://.sandbox.tensorlake.ai/api/v1/processes//stdout/follow \
-H "Authorization: Bearer $TL_API_KEY"
# Follow combined output via SSE
curl -N https://.sandbox.tensorlake.ai/api/v1/processes//output/follow \
-H "Authorization: Bearer $TL_API_KEY"
```
**SSE stream:**
```
event: output
data: {"line":"Step 1/2","timestamp":1773950220162,"stream":"stdout"}
event: output
data: {"line":"Step 2/2","timestamp":1773950220162,"stream":"stdout"}
event: eof
data: {}
```
## Send Signals
Send POSIX signals to running processes:
```python theme={null}
import signal
proc = sandbox.start_process("python", ["-m", "http.server", "8080"])
# Gracefully stop the process
sandbox.send_signal(proc.pid, signal.SIGTERM)
```
```typescript theme={null}
await sandbox.sendSignal(proc.pid, 15);
```
```bash theme={null}
# Send SIGTERM (15)
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes//signal \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"signal": 15}'
# Send SIGKILL (9)
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes//signal \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"signal": 9}'
```
## Kill a Process
```python theme={null}
import signal
sandbox.send_signal(proc.pid, signal.SIGKILL)
```
```typescript theme={null}
await sandbox.killProcess(proc.pid);
```
```bash theme={null}
curl -X DELETE https://.sandbox.tensorlake.ai/api/v1/processes/ \
-H "Authorization: Bearer $TL_API_KEY"
```
## Write to Stdin
Send input to a running process started with stdin in pipe mode:
```typescript theme={null}
import { StdinMode } from "tensorlake";
const proc = await sandbox.startProcess("python", {
args: ["-i"],
stdinMode: StdinMode.PIPE,
});
await sandbox.writeStdin(
proc.pid,
new TextEncoder().encode("print('hello')\n"),
);
await sandbox.closeStdin(proc.pid);
```
```bash theme={null}
# Start a process with stdin pipe
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"command": "python", "args": ["-i"], "stdin_mode": "pipe"}'
# Write to stdin
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes//stdin \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/octet-stream" \
--data-binary "print('hello')\n"
# Close stdin
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes//stdin/close \
-H "Authorization: Bearer $TL_API_KEY"
```
## Managed Processes
Use a managed process for long-running services that should restart after a crash or failed health check. Managed processes use the same process API as normal background commands, but opt into supervision when you provide a `name`, a restart policy, or a health check.
Managed process flags start a background process. In the CLI, these flags require `--detach`. For blocking one-shot commands that stream output and return an exit code, use plain `tl sbx exec` or `sandbox.run(...)`.
```bash theme={null}
# Start and supervise a web server. The CLI prints the PID.
tl sbx exec --detach \
--name dev-server \
--restart always \
--health-http 8080 \
python -m http.server 8080
# Inspect the managed process
tl sbx ps --json
# Manually restart it through the supervisor
tl sbx restart
# Stop the process and remove it from supervision
tl sbx kill
```
```python theme={null}
from tensorlake.sandbox import (
ProcessHealthCheck,
ProcessHealthCheckType,
RestartPolicy,
RestartPolicyConfig,
Sandbox,
)
sandbox = Sandbox.create()
proc = sandbox.start_process(
"python",
args=["-m", "http.server", "8080"],
user="root",
name="dev-server",
restart=RestartPolicyConfig(
policy=RestartPolicy.ALWAYS,
max_restarts=10,
initial_backoff_ms=500,
max_backoff_ms=30_000,
),
health_check=ProcessHealthCheck(
type=ProcessHealthCheckType.HTTP,
port=8080,
path="/",
interval_ms=1_000,
failure_threshold=3,
),
)
print(proc.pid)
print(proc.managed.status)
print(proc.managed.health_status)
current = sandbox.get_process(proc.pid)
restarted = sandbox.restart_process(proc.pid)
```
```typescript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.create();
const proc = await sandbox.startProcess("python", {
args: ["-m", "http.server", "8080"],
user: "root",
name: "dev-server",
restart: {
policy: "always",
maxRestarts: 10,
initialBackoffMs: 500,
maxBackoffMs: 30_000,
},
healthCheck: {
type: "http",
port: 8080,
path: "/",
intervalMs: 1_000,
failureThreshold: 3,
},
});
console.log(proc.pid);
console.log(proc.managed?.status);
console.log(proc.managed?.healthStatus);
const current = await sandbox.getProcess(proc.pid);
const restarted = await sandbox.restartProcess(proc.pid);
```
```bash theme={null}
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"command": "python",
"args": ["-m", "http.server", "8080"],
"user": "root",
"name": "dev-server",
"restart": {
"policy": "always",
"max_restarts": 10,
"initial_backoff_ms": 500,
"max_backoff_ms": 30000
},
"health_check": {
"type": "http",
"port": 8080,
"path": "/",
"interval_ms": 1000,
"failure_threshold": 3
}
}'
```
**Managed response fields:**
```json theme={null}
{
"pid": 42,
"status": "running",
"command": "python",
"args": ["-m", "http.server", "8080"],
"managed": {
"id": "dev-server",
"name": "dev-server",
"status": "running",
"restart_count": 0,
"restart": {"policy": "always"},
"health_status": "starting",
"consecutive_health_failures": 0
}
}
```
```bash theme={null}
# Manual managed restart
curl -X POST https://.sandbox.tensorlake.ai/api/v1/processes//restart \
-H "Authorization: Bearer $TL_API_KEY"
```
Restart policies are `never`, `on_failure`, and `always`. Health checks can be HTTP checks against a local sandbox port and optional path, or TCP checks against a local port. Process user selection accepts a username such as `root`, a UID string, a `uid:gid` string, or an object such as `{"uid": 1000, "gid": 1000}`. The default user is [`tl-user`](/sandboxes/tensorlake-images#default-user-and-working-directory).
## Interactive Shell
```bash theme={null}
# Open an interactive shell in the sandbox
tl sbx ssh
# Use a custom shell
tl sbx ssh --shell /bin/sh
```
```python theme={null}
pty = sandbox.create_pty(
command="/bin/bash",
rows=24,
cols=80,
)
pty.send_input("pwd\nexit\n")
print(pty.wait())
```
```typescript theme={null}
const pty = await sandbox.createPty({
command: "/bin/bash",
rows: 24,
cols: 80,
});
await pty.sendInput("pwd\nexit\n");
console.log(await pty.wait());
```
```bash theme={null}
# 1. Create a PTY session
curl -X POST https://.sandbox.tensorlake.ai/api/v1/pty \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"command": "/bin/bash", "rows": 24, "cols": 80}'
```
**Response:**
```json theme={null}
{
"session_id": "LYtJOrxE9Kz3bphPUDzuX",
"token": ""
}
```
```bash theme={null}
# 2. Connect via WebSocket
wscat -c "wss://.sandbox.tensorlake.ai/api/v1/pty//ws?token="
```
`tl sbx ssh` requires an interactive terminal and automatically resumes a suspended sandbox before opening the PTY session.
For the full programmatic PTY flow, including the `READY` handshake, binary WebSocket opcodes, and clean shutdown, see [PTY Sessions](/sandboxes/pty-sessions).
## Learn More
Interactive shells over WebSocket.
Read, write, and copy files.
Sandbox states, resources, and timeouts.
Control internet access and outbound destinations.
# Computer Use
Source: https://docs.tensorlake.ai/sandboxes/computer-use
Launch ubuntu-vnc sandboxes and drive their desktop from Python or JavaScript.
`tensorlake/ubuntu-vnc` is a managed desktop image for browser automation and computer-use agents. It boots XFCE, TigerVNC, and Firefox for you, and the SDK connects through the authenticated sandbox proxy so you can drive the desktop without manually exposing port `5901`.
This guide builds on [Sandboxes](/sandboxes/introduction). If you already have a Tensorlake API key, you can create a desktop sandbox, capture screenshots, and send mouse and keyboard input in just a few lines.
If you specifically want to automate **Chrome** rather than the desktop, see [Drive Chrome over CDP](/sandboxes/chrome-cdp). It pairs `tensorlake/ubuntu-vnc` with a [tunnel](/sandboxes/tunnels) so Playwright, Puppeteer, or `chrome-devtools-mcp` can drive the in-sandbox browser as if it were running locally.
## Prerequisites
```bash theme={null}
pip install tensorlake
export TENSORLAKE_API_KEY=your-api-key
```
```bash theme={null}
npm install tensorlake
export TENSORLAKE_API_KEY=your-api-key
```
```bash theme={null}
curl -fsSL https://tensorlake.ai/install | sh
export TENSORLAKE_API_KEY=your-api-key
```
Prefer an interactive login? Run `tl login` instead of setting
`TENSORLAKE_API_KEY`; it stores a Personal Access Token in
`~/.config/tensorlake/credentials.toml`.
The current managed `tensorlake/ubuntu-vnc` image uses `tensorlake` as its VNC password.
## Launch a Desktop Sandbox
Use `tensorlake/ubuntu-vnc` when you want a full Linux desktop instead of a shell-only environment.
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create(image="tensorlake/ubuntu-vnc")
print(sandbox.sandbox_id)
```
```javascript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.create({
image: "tensorlake/ubuntu-vnc",
});
try {
console.log(sandbox.sandboxId);
} finally {
await sandbox.terminate();
}
```
```bash theme={null}
tl sbx create -i tensorlake/ubuntu-vnc
```
`tl sbx create` prints the sandbox id on stdout. Reuse it with
`tl sbx tunnel`, `tl sbx ssh`, `tl sbx exec`, and the rest of the
`tl sbx ...` subcommands. List running sandboxes with `tl sbx ls` and
terminate one with `tl sbx terminate `.
You still get a normal `Sandbox` object back, so computer use fits naturally alongside `run()`, file operations, PTY sessions, snapshots, and tunnels.
## Capture Screenshots
Once the sandbox is running, attach to the desktop and save a PNG. This is the easiest way to inspect the layout and discover click coordinates before sending pointer events.
Fresh desktop sandboxes can take a few seconds to finish starting XFCE and other desktop services. If your first screenshot is blank or input does not land where you expect, wait briefly after connecting and then retry.
```python theme={null}
import time
from pathlib import Path
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create(image="tensorlake/ubuntu-vnc")
with sandbox.connect_desktop(password="tensorlake") as desktop:
time.sleep(4.0)
screenshot = desktop.screenshot()
Path("sandbox-desktop.png").write_bytes(screenshot)
print(desktop.width, desktop.height)
```
```javascript theme={null}
import { writeFile } from "node:fs/promises";
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.create({
image: "tensorlake/ubuntu-vnc",
});
try {
const desktop = await sandbox.connectDesktop({
password: "tensorlake",
});
try {
await new Promise((resolve) => setTimeout(resolve, 4000));
const screenshot = await desktop.screenshot();
await writeFile("sandbox-desktop.png", screenshot);
console.log(desktop.width, desktop.height);
} finally {
await desktop.close();
}
} finally {
await sandbox.terminate();
}
```
## Send Keyboard and Mouse Input
The desktop client supports keyboard shortcuts, typed input, clicks, double-clicks, mouse movement, and scrolling. The example below uses a reliable keyboard-driven flow: open a terminal, type a command, and then verify the result from the sandbox shell.
```python theme={null}
import time
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create(image="tensorlake/ubuntu-vnc")
with sandbox.connect_desktop(password="tensorlake") as desktop:
# Give XFCE a moment to finish initializing the keybind daemon and
# window manager. On a freshly-restored snapshot the in-VM `vncserver`
# is up before XFCE has finished settling, so the very first
# `Ctrl+Alt+T` can be lost if it lands before the keybind handler
# registers.
time.sleep(5.0)
desktop.press(["ctrl", "alt", "t"])
time.sleep(4.0)
desktop.type_text("echo docs-test > /tmp/desktop-test.txt")
desktop.press("enter")
time.sleep(3.0)
# Mouse helpers are also available when you know the coordinates.
desktop.move_mouse(640, 400)
desktop.scroll_down()
result = sandbox.run("bash", ["-lc", "cat /tmp/desktop-test.txt"])
print(result.stdout.strip()) # docs-test
```
```javascript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.create({
image: "tensorlake/ubuntu-vnc",
});
try {
const desktop = await sandbox.connectDesktop({
password: "tensorlake",
});
try {
// Give XFCE a moment to finish initializing the keybind daemon and
// window manager. On a freshly-restored snapshot the in-VM `vncserver`
// is up before XFCE has finished settling, so the very first
// `Ctrl+Alt+T` can be lost if it lands before the keybind handler
// registers.
await new Promise((resolve) => setTimeout(resolve, 5000));
await desktop.press(["ctrl", "alt", "t"]);
await new Promise((resolve) => setTimeout(resolve, 4000));
await desktop.typeText("echo docs-test > /tmp/desktop-test.txt");
await desktop.press("enter");
await new Promise((resolve) => setTimeout(resolve, 3000));
// Mouse helpers are also available when you know the coordinates.
await desktop.moveMouse(640, 400);
await desktop.scrollDown();
} finally {
await desktop.close();
}
const result = await sandbox.run("bash", {
args: ["-lc", "cat /tmp/desktop-test.txt"],
});
console.log(result.stdout.trim()); // docs-test
} finally {
await sandbox.terminate();
}
```
Coordinate-based actions are screen-relative. A common workflow is:
1. Take a screenshot.
2. Inspect the desktop layout and note the coordinates you care about.
3. Use `move_mouse()` / `moveMouse()`, `click()`, `double_click()` / `doubleClick()`, and `scroll()` with those coordinates.
## Reconnect to an Existing Sandbox
If a sandbox is already running, connect by sandbox ID and attach to the desktop without creating a new VM.
```python theme={null}
from pathlib import Path
from tensorlake.sandbox import Sandbox
sandbox_id = "your-running-sandbox-id"
with Sandbox.connect(sandbox_id) as sandbox:
with sandbox.connect_desktop(password="tensorlake") as desktop:
Path("existing-sandbox.png").write_bytes(desktop.screenshot())
```
```javascript theme={null}
import { writeFile } from "node:fs/promises";
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.connect({
sandboxId: "your-running-sandbox-id",
});
try {
const desktop = await sandbox.connectDesktop({
password: "tensorlake",
});
try {
const screenshot = await desktop.screenshot();
await writeFile("existing-sandbox.png", screenshot);
} finally {
await desktop.close();
}
} finally {
sandbox.close();
}
```
Connecting to an existing sandbox only closes the client connection when you are done. It does not terminate the running VM.
## Connect with a VNC Client
If you want to drive the desktop from a real VNC viewer (Screen Sharing on macOS, TigerVNC, RealVNC, Remmina, etc.) rather than the SDK, open a TCP tunnel to the sandbox's VNC port and point your client at the local end. The tunnel keeps sandbox-proxy authentication local, so you do **not** need to expose `5901` publicly.
Open the tunnel with `tl sbx tunnel`. Replace `` with the id printed by `tl sbx create` (or `tl sbx ls`):
```bash theme={null}
tl sbx tunnel 5901 --listen-port 15901
```
Leave that command running. It forwards `127.0.0.1:15901` on your machine to port `5901` inside the sandbox over an authenticated WebSocket. Then connect any VNC client to `localhost:15901` using the desktop password `tensorlake`:
Use the built-in Screen Sharing client:
```bash theme={null}
open vnc://localhost:15901
```
Enter `tensorlake` when macOS prompts for the password.
```bash theme={null}
vncviewer localhost:15901
```
Most distributions ship `vncviewer` in the `tigervnc-viewer` package
(`apt install tigervnc-viewer` on Debian/Ubuntu,
`dnf install tigervnc` on Fedora).
Any RFB-compatible viewer works: RealVNC Viewer, TightVNC, Remmina,
KRDC, etc. Point it at `localhost:15901` and use `tensorlake` as the
password.
Stop the tunnel with `Ctrl+C` when you are done. Closing the tunnel does not terminate the sandbox; reopen it any time with the same command.
## Use noVNC in the Browser
If you want a human to interact with the sandbox desktop in real time, use a real VNC client in the browser instead of polling screenshots. [`noVNC`](https://novnc.com/info.html) is a good fit here.
The recommended architecture is:
1. Keep the Tensorlake API key on your backend.
2. Use the backend to open a TCP tunnel to the sandbox's VNC port `5901`.
3. Bridge that local tunnel to a browser WebSocket endpoint such as `/vnc/`.
4. Point `noVNC` at your backend WebSocket and authenticate with the desktop password `tensorlake`.
This keeps sandbox proxy authentication server-side and gives the browser a low-latency live desktop stream. You do **not** need to expose port `5901` publicly yourself.
If you are also running an agent loop, a good pattern is to use:
* `noVNC` for the live human-facing desktop stream
* `sandbox.connectDesktop()` for screenshots and high-level computer-use actions on the backend
That separation avoids turning the browser view into a screenshot polling loop.
### Browser Client with noVNC
Install `noVNC` in your frontend:
```bash theme={null}
npm install @novnc/novnc
```
Then connect the browser to your own WebSocket bridge:
```ts theme={null}
import RFB from "@novnc/novnc/lib/rfb";
const host = document.getElementById("desktop");
if (!(host instanceof HTMLDivElement)) {
throw new Error("Missing #desktop container");
}
const protocol = window.location.protocol === "https:" ? "wss:" : "ws:";
const url = `${protocol}//${window.location.host}/vnc`;
const rfb = new RFB(host, url, {
credentials: { password: "tensorlake" },
shared: true,
});
rfb.scaleViewport = true;
rfb.clipViewport = false;
rfb.showDotCursor = true;
```
Use a fixed-size container for the desktop surface:
```html theme={null}
```
## Desktop API Surface
Python uses `snake_case`, while JavaScript uses `camelCase`, but both SDKs expose the same core capabilities:
* Screenshots: `screenshot()`
* Mouse input: `move_mouse()` / `moveMouse()`, `mouse_press()` / `mousePress()`, `mouse_release()` / `mouseRelease()`, `click()`, `double_click()` / `doubleClick()`, `scroll()`, `scroll_up()` / `scrollUp()`, and `scroll_down()` / `scrollDown()`
* Keyboard input: `key_down()` / `keyDown()`, `key_up()` / `keyUp()`, `press()`, and `type_text()` / `typeText()`
* Desktop size: `width` and `height`
`connect_desktop()` and `connectDesktop()` go through the authenticated sandbox proxy, so you do not need to bind or expose the VNC port yourself. For interactive debugging through a real VNC viewer, see [Connect with a VNC Client](#connect-with-a-vnc-client) above.
## Related Guides
* [Drive Chrome over CDP](/sandboxes/chrome-cdp): point Playwright, Puppeteer, or `chrome-devtools-mcp` at the Chrome that ships in `tensorlake/ubuntu-vnc`.
* [Local Tunnels](/sandboxes/tunnels): the tunneling primitive used by both the VNC viewer and Chrome CDP workflows.
* [Snapshots](/sandboxes/snapshots): fork warm desktops to parallelize agent runs without re-launching XFCE.
# Run Your Test Suite with Crabbox
Source: https://docs.tensorlake.ai/sandboxes/crabbox
Crabbox's Tensorlake provider drops any command into a Firecracker microVM: warm a sandbox, sync your working tree, and run your tests with one command.
[Crabbox](https://crabbox.sh) is an open-source CLI from OpenClaw whose whole loop is *warm a box, sync the diff, run the suite*. Its [Tensorlake provider](https://crabbox.sh/providers/tensorlake.html) delegates the sandbox to the `tensorlake` CLI, so one command puts your test run in an isolated Firecracker microVM:
```sh theme={null}
crabbox run --provider tensorlake --tensorlake-image tl-crabbox -- pnpm test
```
`tl-crabbox` is a public image we publish for Crabbox: the standard Ubuntu base plus a writable `/workspace` (Crabbox's default workdir) and pnpm preinstalled.
Crabbox owns the local workflow: config, repo claims, sync manifests, and guardrails. Tensorlake owns the microVM and command transport: under the hood, Crabbox shells out to `tensorlake sbx create`, `cp`, `exec`, and `terminate`. See [Sandbox lifecycle](/sandboxes/lifecycle) for what happens on the Tensorlake side.
## Prerequisites
* A Tensorlake account and API key. Sign up at [cloud.tensorlake.ai](https://cloud.tensorlake.ai).
* Run Crabbox from inside a git repository. It builds its sync file list from `git ls-files`, so a plain directory fails with `build sync file list: exit status 128`.
## Setup
```sh theme={null}
brew install openclaw/tap/crabbox
curl -fsSL https://tensorlake.ai/install | sh
```
No Homebrew? Grab a release archive from [Crabbox's GitHub releases](https://github.com/openclaw/crabbox/releases). The `tensorlake` CLI must be on your `PATH`, or point Crabbox at it with `--tensorlake-cli`.
```sh theme={null}
export TENSORLAKE_API_KEY=tl_apiKey_...
```
Crabbox passes the key to the CLI through the environment. It never appears on the command line. If your account spans multiple organizations or projects, also set `TENSORLAKE_ORGANIZATION_ID` and `TENSORLAKE_PROJECT_ID`.
Add a `.crabbox.yaml` at your repo root:
```yaml theme={null}
provider: tensorlake
tensorlake:
image: tl-crabbox
```
Pinning `tl-crabbox` here means every run and warmup picks it up, with no `--tensorlake-image` flag to retype.
The `image` line matters. Crabbox's default workdir is `/workspace/crabbox`, and in Tensorlake's standard images commands run as [`tl-user`](/sandboxes/tensorlake-images#default-user-and-working-directory), which cannot create `/workspace`. Without the pin, every run fails with `tensorlake exec "mkdir -p '/workspace/crabbox'" exited 1`. Prefer a standard image anyway? Set `tensorlake.workdir: /home/tl-user/crabbox` instead.
```sh theme={null}
crabbox warmup --provider tensorlake --tensorlake-cpus 2 --tensorlake-memory-mb 2048
```
Crabbox creates a named sandbox and prints a friendly slug (like `harbor-barnacle`) you can reuse across runs with `--id `. Every `tensorlake.*` config field has a matching `--tensorlake-*` flag and `CRABBOX_TENSORLAKE_*` environment override.
```sh theme={null}
crabbox run --provider tensorlake -- pnpm test
```
Crabbox syncs your **git-tracked files** into the sandbox and streams output back as the command runs. For shell pipelines, use `--shell`:
```sh theme={null}
crabbox run --provider tensorlake --shell 'pnpm install && pnpm test'
```
`tl-crabbox` ships with `node`, `npm`, `pnpm`, `corepack`, `python3`, and `git`. Need more toolchain? Register your own image with [`tensorlake sbx image create`](/sandboxes/images) and pin it via `tensorlake.image` instead. To forward secrets from your shell, allowlist them with `--allow-env API_TOKEN`. Values are injected for the command and removed after.
One-off runs lease a sandbox and terminate it automatically. Warmed sandboxes stick around until you release them:
```sh theme={null}
crabbox stop --provider tensorlake harbor-barnacle
```
Add `--keep-on-failure` to a run to keep the sandbox alive after a failing command.
## Troubleshooting
| Error | Cause | Fix |
| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `build sync file list: exit status 128` | You're not inside a git repository, and Crabbox builds its sync list from `git ls-files`. | Run from a repo root (`git init` if needed). |
| `tensorlake exec "mkdir -p '/workspace/crabbox'" exited 1` | The default workdir isn't writable by `tl-user` in Tensorlake's standard images. | Set `tensorlake.image: tl-crabbox` in `.crabbox.yaml` (or override `tensorlake.workdir` to `/home/tl-user/crabbox`). |
| `Failed to spawn process in : No such file or directory` | The command's executable (e.g. `pnpm`) isn't in the sandbox image. The message blames the directory, but it's the missing binary. | Pin `tl-crabbox` (ships pnpm), use a tool the image has (`npm`, `corepack`), or register an image with your toolchain. |
For every flag, gotcha, and lifecycle detail, see [Crabbox's Tensorlake provider reference](https://crabbox.sh/providers/tensorlake.html).
# Data Analysis
Source: https://docs.tensorlake.ai/sandboxes/data-analysis
Perform parallel data analysis and model benchmarking in isolated sandboxes.
Run parallel data analysis, model training, and benchmarking tasks in secure, isolated sandbox environments. Each sandbox can have its own dependencies and resource limits, allowing you to compare different models or process large datasets concurrently.
This example demonstrates how to benchmark several `scikit-learn` classification models in parallel by running each in its own sandbox.
## TypeScript SDK starter
The same benchmarking pattern works in Node.js: one model per sandbox, `Promise.all()` for fan-out, and JSON on stdout for aggregation.
```typescript theme={null}
import { Sandbox } from "tensorlake";
async function runModelBenchmark(modelName: string, sklearnPath: string) {
const splitAt = sklearnPath.lastIndexOf(".");
const modulePath = sklearnPath.slice(0, splitAt);
const className = sklearnPath.slice(splitAt + 1);
const code = `
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from ${modulePath} import ${className}
import json
data = load_iris()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.3)
model = ${className}()
model.fit(X_train, y_train)
print(json.dumps({"model": "${modelName}", "accuracy": model.score(X_test, y_test)}))
`;
const sandbox = await Sandbox.create({
timeoutSecs: 900,
allowInternetAccess: false,
});
try {
await sandbox.run("pip", {
args: [
"install",
"numpy",
"scikit-learn",
"--user",
"--break-system-packages",
],
});
const result = await sandbox.run("python", {
args: ["-c", code],
});
return JSON.parse(result.stdout);
} finally {
await sandbox.terminate();
}
}
const modelsToTest = {
"Random Forest": "sklearn.ensemble.RandomForestClassifier",
SVM: "sklearn.svm.SVC",
"Logistic Regression": "sklearn.linear_model.LogisticRegression",
};
const results = await Promise.all(
Object.entries(modelsToTest).map(([name, path]) =>
runModelBenchmark(name, path),
),
);
console.table(results);
client.close();
```
## Example: Parallel Model Benchmarking
The following script benchmarks five different `scikit-learn` models on the Iris dataset. Each model is trained and evaluated in a separate, concurrent sandbox.
```python theme={null}
import asyncio
import json
from dotenv import load_dotenv
load_dotenv()
from tensorlake.sandbox import Sandbox
async def run_model_benchmark(model_name, sklearn_path):
"""
Runs a model benchmark inside an isolated sandbox.
Returns a dict with model name and accuracy.
"""
module_path, class_name = sklearn_path.rsplit('.', 1)
code = f"""
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from {module_path} import {class_name}
import json
data = load_iris()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.3)
model = {class_name}()
model.fit(X_train, y_train)
score = model.score(X_test, y_test)
print(json.dumps({{"model": "{model_name}", "accuracy": score}}))
"""
def _sync_benchmark():
sandbox = Sandbox.create()
print(f"🚀 Sandbox started for {model_name}...")
# install scikit-learn and its dependencies in the sandbox
sandbox.run("pip", ["install", "--user", "--break-system-packages", "numpy", "scikit-learn"])
# run the code in the sandbox
result = sandbox.run("python", ["-c", code])
output_data = json.loads(result.stdout.strip())
return output_data
return await asyncio.to_thread(_sync_benchmark)
async def main():
models_to_test: dict[str, str] = {
"Random Forest": "sklearn.ensemble.RandomForestClassifier",
"SVM": "sklearn.svm.SVC",
"Logistic Regression": "sklearn.linear_model.LogisticRegression",
"Decision Tree": "sklearn.tree.DecisionTreeClassifier",
"KNN": "sklearn.neighbors.KNeighborsClassifier",
}
tasks = [run_model_benchmark(name, path) for name, path in models_to_test.items()]
print("Gathering results from all sandboxes...\n")
results = await asyncio.gather(*tasks)
print("--- Benchmark Results ---")
for r in results:
print(f"{r['model']:<20}: {r['accuracy']:.4f}")
if __name__ == "__main__":
asyncio.run(main())
```
## How It Works
The script orchestrates the parallel execution of model benchmarks using Python's `asyncio` library.
**1. Parallel Execution:** The `main` function defines a dictionary of models to test and creates a list of asynchronous tasks using a list comprehension. `asyncio.gather` runs all these tasks concurrently.
**2. Sandbox Task:** The `run_model_benchmark` function is responsible for a single benchmark. For each model, it:
* Creates a new, isolated sandbox.
* Installs the necessary Python libraries (`numpy` and `scikit-learn`) inside the sandbox using `sandbox.run()`. The `--break-system-packages` flag is used to comply with PEP 668 in newer Python environments.
* Executes a Python script that trains the model on the Iris dataset and calculates its accuracy.
* Prints the results as a JSON string to standard output.
* Captures the `stdout`, parses the JSON, and returns the result.
**3. Aggregate Results:** Once all sandboxes have completed their tasks, `asyncio.gather` returns a list of all the results, which are then printed to the console.
This example uses the `python-dotenv` library to load your Tensorlake API key from a `.env` file. Create a file named `.env` in your project root and add your key:
```
TENSORLAKE_API_KEY="your-api-key-here"
```
The SDK will automatically use this key.
## Pro Tips
### Faster Execution with Snapshots
The example installs dependencies every time a sandbox is created. This is simple but inefficient for repeated runs. To significantly speed up your workflow, you can use **Snapshots**.
1. Create a "base" sandbox and install all your dependencies.
2. Create a snapshot of that sandbox.
3. Start new sandboxes from the snapshot ID. The new sandboxes will have all the dependencies pre-installed, saving you valuable setup time.
Learn more in the [Snapshots guide](/sandboxes/snapshots).
## Learn More
Install Tensorlake and create your first sandbox.
Learn how to upload custom datasets and other files to your sandboxes.
# Run Devin Outposts on Tensorlake Sandboxes
Source: https://docs.tensorlake.ai/sandboxes/devin-outposts
Serve Devin Outposts sessions on Tensorlake sandboxes. Devin runs the agent loop; every command, file edit, and repo checkout runs in a Firecracker microVM you control.
[Devin Outposts](https://docs.devin.ai/cloud/outposts) lets you run Devin sessions inside infrastructure you control. The agent loop stays in Cognition's cloud. The session machine, where commands run, files change, and repos get checked out, moves into a Tensorlake sandbox you own.
Use an outpost when the agent needs a custom environment: private CA certificates, preinstalled toolchains, pre-cloned repositories, or services only reachable from your network, all defined in the Tensorlake image and sandbox configuration. And because Tensorlake can run on self-hosted compute, the sandboxes can sit inside your network, where Devin can read your source code, query your databases, and test against the systems it is writing code for.
An **orchestrator** watches Devin's session queue, claims each session, and runs it in a Tensorlake sandbox. The orchestrator itself runs inside a long-lived Tensorlake sandbox, so no long-running process stays on your laptop. When a session goes idle the sandbox suspends; when it wakes the same sandbox resumes; when it ends the sandbox is terminated. Each Devin session maps to one sandbox.
```mermaid theme={null}
graph LR
D["Devin cloud
agent loop + session queue"]
O["Orchestrator
one per outpost, runs in a Tensorlake sandbox"]
S["Tensorlake sandbox
one per session, runs devin-remote
(Devin's session agent binary)"]
O <-->|"watch · claim · release"| D
O -->|"create · suspend · resume · launch remote"| S
S -->|"outbound connection"| D
```
## Prerequisites
* A Devin account with Outposts enabled, and an org admin who can connect the outpost.
* A Tensorlake account and API key from [cloud.tensorlake.ai](https://cloud.tensorlake.ai).
* Python 3.10+ on the machine that runs the setup and launcher commands. The orchestrator itself runs inside a Tensorlake sandbox.
Tensorlake sandboxes are Linux microVMs, so the outpost is Linux only. Run one orchestrator per outpost.
## Setup
The reference implementation is [tensorlakeai/devin-outposts-tensorlake](https://github.com/tensorlakeai/devin-outposts-tensorlake), a Python package that is mainly the orchestrator. Installing it also provides the CLI commands used in the steps below: `outposts-connect`, the two image builders, for orchestrator and session sandboxes respectively, and the sandbox launcher.
```bash theme={null}
git clone https://github.com/tensorlakeai/devin-outposts-tensorlake.git
cd devin-outposts-tensorlake
python3 -m venv .venv && . .venv/bin/activate
pip install -e .
cp .env.example .env
```
Add your `TENSORLAKE_API_KEY` to `.env`.
Run this on the same machine as your browser:
```bash theme={null}
outposts-connect --platform linux
```
`outposts-connect` is the package's local implementation of [Devin's partner connection flow](https://docs.devin.ai/cloud/outposts/partners). It opens Devin's connection page, where a Devin org admin confirms the outpost and clicks **Connect**. The browser returns a one-time code to a temporary listener on `localhost`, which is why the command and browser must share a machine. The command exchanges the code for a machine-serving token and writes `DEVIN_OUTPOSTS_TOKEN`, `DEVIN_API_URL`, and `OUTPOST_ID` to `.env`. The token never passes through the browser.
This one-time authorization creates the outpost and a service user whose token the orchestrator uses for every queue call. The outpost then appears in Devin's environment picker whenever someone in your org creates a session.
For headless hosts or manual outpost creation, see [manual setup in the repo README](https://github.com/tensorlakeai/devin-outposts-tensorlake#manual-outpost-setup).
```bash theme={null}
set -a && . ./.env && set +a
build-devin-outposts-image
```
This builds an image with the remote's required system packages: `git`, `curl`, and CA certificates. It also tries to install the GitHub CLI and Chromium, but those steps are best-effort; if your sessions depend on either tool, verify the built image. Devin also lists `ffmpeg` as an optional dependency for screen recording; this image skips it, so add it to the build recipe if you want recordings. Copy the printed name into `IMAGE_NAME` in `.env`.
```bash theme={null}
set -a && . ./.env && set +a
build-devin-outposts-dispatcher-image
```
This packages the orchestrator itself into its own image, distinct from the session image that serving sandboxes boot from. Build it once; rebuild only when you update the package. (The repo calls this the dispatcher image.)
```bash theme={null}
set -a && . ./.env && set +a
devin-outposts-orchestrator-sandbox
```
This starts the orchestrator inside a long-lived Tensorlake sandbox. The command is idempotent: run it again and it resumes the sandbox and re-ensures the orchestrator process. The orchestration work now happens in the sandbox, not on your local; the only local piece left is a small keep-alive cron, covered in [Operate the orchestrator](#operate-the-orchestrator).
The launcher reads your local `.env` and injects the credentials the orchestrator needs (`TENSORLAKE_API_KEY`, the Devin machine token, and `GIT_TOKEN` if set) into the orchestrator process's environment at start. They are never baked into the orchestrator image. Treat this sandbox as a trusted control-plane host; session sandboxes never receive the machine token, each remote gets only its own session's connect token. To rotate a credential, update `.env`, then `--terminate` and relaunch.
Confirm the orchestrator is watching, then create a session in the Devin UI or Slack and select your outpost. The orchestrator claims it and runs it in a sandbox.
```bash theme={null}
devin-outposts-orchestrator-sandbox --status
```
Each claim pins `devin-remote`, Devin's session binary, by SHA; the orchestrator downloads it into the sandbox and verifies the checksum before executing it. The binary dials out to Devin's gateway over HTTPS, so the sandbox needs no inbound ports.
Session credentials follow the same injected-at-start pattern as the orchestrator's. The claim response carries that session's connect token and gateway URL, which the orchestrator sets as environment variables when it launches `devin-remote` in the sandbox. If `REPOS` is set, the orchestrator pre-clones those repositories into the sandbox first, passing `GIT_USERNAME` and `GIT_TOKEN` to the clone command only. Nothing is baked into the session image, and the session sandbox never receives your Tensorlake API key or the Devin machine token.
## Operate the orchestrator
Observe and manage the orchestrator sandbox from your machine:
```bash theme={null}
devin-outposts-orchestrator-sandbox --status
devin-outposts-orchestrator-sandbox --logs
devin-outposts-orchestrator-sandbox --terminate
```
Outposts is watch-based, so the orchestrator runs continuously (not scale-to-zero). Tensorlake suspends a named sandbox after your plan's maximum idle window, and a suspended orchestrator cannot claim sessions. The launcher is idempotent (it resumes the sandbox and re-ensures the orchestrator process), so schedule it on a cron to keep it watching across that window:
```bash theme={null}
# crontab -e: resume the orchestrator sandbox if it suspended
*/15 * * * * cd /path/to/devin-outposts-tensorlake && set -a && . ./.env && set +a && .venv/bin/devin-outposts-orchestrator-sandbox
```
The cron host needs the repo, the venv, and `.env`, and it needs to be awake when the tick fires, so an always-on machine (a small server or any host that does not sleep) is the better home for it. A laptop works too, with one caveat: while the laptop sleeps the orchestrator can stay suspended, and queued sessions wait until the next tick after it wakes. Keep the interval shorter than your plan's idle window.
## Watch session logs
The orchestrator log (`--logs`) shows lifecycle events only (claim, sandbox created, serving, released). The session's processing output, every tool call `devin-remote` executes, is written inside the serving sandbox to `/tmp/devin-outposts/.log`. The reference implementation includes `session_logs.py`, a helper script we wrote to monitor session logs from your machine:
```bash theme={null}
python session_logs.py # list this outpost's serving sandboxes
python session_logs.py # dump the last 500 lines
python session_logs.py -f # stream live (ctrl-C to stop)
```
The sandbox name is the `dvo-...` string the orchestrator logs when it claims a session (visible in `--logs`). Live mode runs `tail -F` in the sandbox over a PTY websocket (the SDK's streaming channel) and follows the session lifecycle: it reconnects if the connection idles out, waits while the sandbox is suspended (Devin put the session to sleep) and re-attaches on resume, and exits when the sandbox is terminated because the session ended.
## Configuration
Set these in `.env`. `outposts-connect` writes the Devin values, you add `TENSORLAKE_API_KEY` during install, and the image build prints `IMAGE_NAME`; the rest have defaults.
| Variable | Required | Purpose |
| -------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DEVIN_OUTPOSTS_TOKEN` | Yes | Machine-serving token for queue watch/claim/release. |
| `DEVIN_API_URL` | No | Devin API base without `/opbeta`. Defaults to `https://api.devin.ai`; `outposts-connect` writes it. |
| `TENSORLAKE_API_KEY` | Yes | Authenticates the Tensorlake SDK for images and sandbox lifecycle. |
| `OUTPOST_ID` | Yes | Your outpost. |
| `IMAGE_NAME` | Yes | Image the orchestrator boots sandboxes from. |
| `MAX_CONCURRENT_SESSIONS` | No | Concurrent sessions (default `5`). |
| `SANDBOX_CPUS` / `SANDBOX_MEMORY_MB` / `SANDBOX_DISK_MB` | No | Per-session sizing (2 vCPU, 8 GiB, 10 GiB). |
| `SANDBOX_TIMEOUT_SECS` | No | Idle seconds before auto-suspend (default `1800`). |
| `REPOS` / `GIT_USERNAME` / `GIT_TOKEN` | No | Comma-separated clone URLs to pre-clone into each sandbox, and credentials for private ones. The credentials are passed to the clone command only, never stored in the image. |
## Next steps
The full orchestrator and the CLI commands that drive it.
The suspend/resume model that keeps idle sessions cheap.
Build a custom session image with your toolchain.
The same orchestrator pattern for Anthropic's managed agents.
# Run Docker
Source: https://docs.tensorlake.ai/sandboxes/docker
Run Docker containers inside Tensorlake sandboxes using the tensorlake/ubuntu-systemd base image, with full systemd support for compose, networking, and daemons.
### Create the sandbox
```bash theme={null}
tl sbx create my-docker-sandbox --image tensorlake/ubuntu-systemd --cpus 2.0 --memory 2048
```
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create(
name="my-docker-sandbox",
image="tensorlake/ubuntu-systemd",
cpus=2.0,
memory_mb=2048,
)
```
```typescript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.create({
name: "my-docker-sandbox",
image: "tensorlake/ubuntu-systemd",
cpus: 2.0,
memoryMb: 2048,
});
```
### Install Docker
Install Docker from the [official Ubuntu repository](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository):
```bash theme={null}
tl sbx exec my-docker-sandbox bash -c '
set -e
apt-get update
apt-get install -y ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
. /etc/os-release && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu ${UBUNTU_CODENAME:-$VERSION_CODENAME} stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
'
```
```python theme={null}
script = """
set -e
apt-get update
apt-get install -y ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
. /etc/os-release && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu ${UBUNTU_CODENAME:-$VERSION_CODENAME} stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
"""
result = sandbox.run("bash", ["-c", script])
if result.exit_code != 0:
raise RuntimeError(result.stderr)
```
```typescript theme={null}
const script = [
"set -e",
"apt-get update",
"apt-get install -y ca-certificates curl",
"install -m 0755 -d /etc/apt/keyrings",
"curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc",
"chmod a+r /etc/apt/keyrings/docker.asc",
'. /etc/os-release && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu ${UBUNTU_CODENAME:-$VERSION_CODENAME} stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null',
"apt-get update",
"apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y",
].join("\n");
const result = await sandbox.run("bash", { args: ["-c", script] });
if (result.exitCode !== 0) throw new Error(result.stderr);
```
### Verify
```bash theme={null}
tl sbx exec my-docker-sandbox docker run hello-world
```
### SSH into the sandbox to run Docker commands interactively:
```bash theme={null}
tl sbx ssh my-docker-sandbox
sudo docker run hello-world
```
# Environment Variables
Source: https://docs.tensorlake.ai/sandboxes/environment-variables
Set per-command and per-PTY environment variables in the CLI, Python, and TypeScript.
Use this page as a quick reference for environment variable scopes in Sandboxes.
For endpoint-level and workflow details, also see [Execute Commands](./commands) and [PTY Sessions](./pty-sessions).
| Scope | Use this when | API surface |
| -------------- | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Command-scoped | Variables should apply to one command only | `tl sbx exec --env KEY=VALUE ...`, `sandbox.run(..., env=...)` |
| PTY-scoped | Variables should apply to one interactive PTY session only | `tl sbx ssh --env KEY=VALUE ...`, `create_pty(..., env=...)` / `createPty({ env: ... })` |
## Prerequisites
* You have the `tl` CLI installed and authenticated.
* You have a running sandbox connection in the CLI, Python, or TypeScript.
* You already set `TENSORLAKE_API_KEY` in your local environment.
## 1. Env vars for a single command in a sandbox
Use command-scoped env when values should not persist across all sandbox processes.
```bash theme={null}
# Set command-scoped env vars with repeated --env flags
tl sbx exec \
--env MODE=prod \
--env DEBUG=0 \
bash -lc 'echo MODE=$MODE DEBUG=$DEBUG'
```
```bash theme={null}
# Same pattern when creating a one-shot sandbox with tl sbx run
tl sbx run \
--env MODE=prod \
--env DEBUG=0 \
bash -lc 'echo MODE=$MODE DEBUG=$DEBUG'
```
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create()
result = sandbox.run(
"bash",
["-lc", "echo MODE=$MODE DEBUG=$DEBUG"],
env={"MODE": "prod", "DEBUG": "0"},
)
print(result.stdout)
```
```python theme={null}
# Advanced: set additional command-only variables.
result = sandbox.run(
"bash",
["-lc", "echo APP_ENV=$APP_ENV TRACE_ID=$TRACE_ID"],
env={"APP_ENV": "staging", "TRACE_ID": "req-123"},
)
```
```typescript theme={null}
const result = await sandbox.run("bash", {
args: ["-lc", "echo MODE=$MODE DEBUG=$DEBUG"],
env: { MODE: "prod", DEBUG: "0" },
});
console.log(result.stdout);
```
```typescript theme={null}
// Advanced: set additional command-only variables.
const overrideResult = await sandbox.run("bash", {
args: ["-lc", "echo APP_ENV=$APP_ENV TRACE_ID=$TRACE_ID"],
env: { APP_ENV: "staging", TRACE_ID: "req-123" },
});
```
## 2. Env vars when creating a PTY session
Use PTY-scoped env when you need custom variables inside one interactive terminal session.
```bash theme={null}
# Open an interactive PTY session
tl sbx ssh
```
```bash theme={null}
# Set custom PTY env vars
tl sbx ssh \
--env APP_ENV=dev \
--env TERM=screen-256color
```
```bash theme={null}
# Optional: custom shell, shell args, and working directory
tl sbx ssh \
--shell /bin/zsh \
--shell-arg -l \
--workdir /workspace \
--env APP_ENV=dev
```
`tl sbx ssh` always creates a PTY session with defaults like `TERM` and
`COLORTERM=truecolor`; your `--env` values are merged in and can override
those defaults.
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create()
pty = sandbox.create_pty(
command="/bin/bash",
args=["-l"],
env={"TERM": "xterm-256color", "APP_ENV": "dev"},
working_dir="/workspace",
rows=24,
cols=80,
)
pty.send_input("echo APP_ENV=$APP_ENV\nexit\n")
pty.wait()
```
```typescript theme={null}
const pty = await sandbox.createPty({
command: "/bin/bash",
args: ["-l"],
env: { TERM: "xterm-256color", APP_ENV: "dev" },
workingDir: "/workspace",
rows: 24,
cols: 80,
});
await pty.sendInput("echo APP_ENV=$APP_ENV\nexit\n");
await pty.wait();
```
## Choosing the right scope
* Use `run(..., env=...)` for one-off command values.
* Use PTY `env` for interactive terminal sessions.
# File Operations
Source: https://docs.tensorlake.ai/sandboxes/file-operations
Copy, read, write, and manage files inside Tensorlake sandboxes. Transfer between your local machine and the sandbox filesystem over the proxy URL.
File operations use sandbox-specific ingress, commonly seen as `https://.sandbox.tensorlake.ai`, derived from the sandbox's `ingress_endpoint`.
Named sandboxes can use the sandbox name in place of the ID in the proxy hostname. The file APIs documented here run on the management URL on port `9501`, which always requires authentication. Unauthenticated proxy access applies only to exposed user ports.
## Copy Files
```bash theme={null}
# Copy a local file into the sandbox
tl sbx cp data.csv :/workspace/data.csv
# Copy a file from the sandbox to local
tl sbx cp :/workspace/data.csv ./data.csv
```
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create()
sandbox.write_file("/workspace/data.csv", b"name,score\nAlice,95\nBob,87")
content = sandbox.read_file("/workspace/data.csv")
print(bytes(content).decode("utf-8"))
```
```typescript theme={null}
await sandbox.writeFile(
"/workspace/data.csv",
new TextEncoder().encode("name,score\nAlice,95\nBob,87"),
);
const content = await sandbox.readFile("/workspace/data.csv");
console.log(new TextDecoder().decode(content));
```
```bash theme={null}
curl -X PUT "https://.sandbox.tensorlake.ai/api/v1/files?path=/workspace/data.csv" \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/octet-stream" \
--data-binary "name,score\nAlice,95\nBob,87"
curl "https://.sandbox.tensorlake.ai/api/v1/files?path=/workspace/data.csv" \
-H "Authorization: Bearer $TL_API_KEY"
```
`tl sbx cp` is file-only today. Directory copy workflows should use the Python SDK, TypeScript SDK, or the raw file API.
## Read Files
```bash theme={null}
tl sbx cp :/workspace/data.csv ./data.csv
tl sbx exec cat /workspace/data.csv
```
```python theme={null}
content = sandbox.read_file("/workspace/data.csv")
print(bytes(content).decode("utf-8"))
image_bytes = sandbox.read_file("/workspace/chart.png")
with open("chart.png", "wb") as f:
f.write(image_bytes)
```
```typescript theme={null}
const content = await sandbox.readFile("/workspace/data.csv");
console.log(new TextDecoder().decode(content));
const bytes = await sandbox.readFile("/workspace/chart.png");
console.log(`Read ${bytes.byteLength} bytes`);
```
```bash theme={null}
curl "https://.sandbox.tensorlake.ai/api/v1/files?path=/workspace/data.csv" \
-H "Authorization: Bearer $TL_API_KEY"
curl "https://.sandbox.tensorlake.ai/api/v1/files?path=/workspace/chart.png" \
-H "Authorization: Bearer $TL_API_KEY" \
-o chart.png
```
## Write Files
```bash theme={null}
tl sbx cp config.json :/workspace/config.json
```
```python theme={null}
sandbox.write_file("/workspace/config.json", b'{"debug": true, "port": 8080}')
with open("model.pkl", "rb") as f:
sandbox.write_file("/workspace/model.pkl", f.read())
```
```typescript theme={null}
import { readFile } from "node:fs/promises";
await sandbox.writeFile(
"/workspace/config.json",
new TextEncoder().encode('{"debug": true, "port": 8080}'),
);
const modelBytes = await readFile("model.pkl");
await sandbox.writeFile("/workspace/model.pkl", modelBytes);
```
```bash theme={null}
curl -X PUT "https://.sandbox.tensorlake.ai/api/v1/files?path=/workspace/config.json" \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/octet-stream" \
--data-binary '{"debug": true, "port": 8080}'
curl -X PUT "https://.sandbox.tensorlake.ai/api/v1/files?path=/workspace/model.pkl" \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/octet-stream" \
--data-binary @model.pkl
```
## List Directory Contents
```bash theme={null}
tl sbx exec ls -la /workspace
```
```python theme={null}
entries = sandbox.list_directory("/workspace")
for entry in entries.entries:
print(f"{entry.name} ({entry.size} bytes)")
```
```typescript theme={null}
const listing = await sandbox.listDirectory("/workspace");
for (const entry of listing.entries) {
console.log(`${entry.name} (${entry.size ?? 0} bytes)`);
}
```
```bash theme={null}
curl "https://.sandbox.tensorlake.ai/api/v1/files/list?path=/workspace" \
-H "Authorization: Bearer $TL_API_KEY"
```
## Delete Files
```bash theme={null}
tl sbx exec rm -rf /workspace/temp
```
```python theme={null}
sandbox.delete_file("/workspace/temp")
```
```typescript theme={null}
await sandbox.deleteFile("/workspace/temp");
```
```bash theme={null}
curl -X DELETE "https://.sandbox.tensorlake.ai/api/v1/files?path=/workspace/temp" \
-H "Authorization: Bearer $TL_API_KEY"
```
## Organize Files
```bash theme={null}
tl sbx exec mkdir -p /workspace/src/components
tl sbx exec mv /workspace/old.txt /workspace/new.txt
```
```python theme={null}
sandbox.run("mkdir", ["-p", "/workspace/src/components"])
sandbox.run("mv", ["/workspace/old.txt", "/workspace/new.txt"])
```
```typescript theme={null}
await sandbox.run("mkdir", {
args: ["-p", "/workspace/src/components"],
});
await sandbox.run("mv", {
args: ["/workspace/old.txt", "/workspace/new.txt"],
});
```
Not supported in the HTTP API.
## Best Practices
* Use `/workspace` as the default directory for application files.
* Use absolute paths to avoid ambiguity.
* Use `write_file` / `read_file` for programmatic access.
* Use `tl sbx cp` for single-file transfers.
* Use the Python SDK, TypeScript SDK, or raw file API for directory-oriented workflows.
## Learn More
Execute commands in sandboxes.
Save and restore sandbox filesystem, memory, and running processes.
Sandbox states, resources, and timeouts.
# RL Training with GSPO
Source: https://docs.tensorlake.ai/sandboxes/gspo-agentic-rl
Fine-tune a language model on code generation tasks using Group Sequence Policy Optimization, with TensorLake sandboxes as the reward oracle.
Train a language model to write correct Python functions using reinforcement learning, without ever running untrusted model-generated code in your training process. This guide walks through a two-phase setup: a supervised fine-tuning warmup followed by GSPO fine-tuning, where every completion is evaluated inside an isolated TensorLake sandbox running a hidden pytest suite.
## How it works
1. **SFT warmup (Phase 1)**: Supervised pass on correct solutions so the model starts generating valid Python. Without this, all completions score 0, reward variance is 0, and the RL trainer has no gradient signal.
2. **GSPO fine-tuning (Phase 2)**: The `GRPOTrainer` (with `importance_sampling_level="sequence"`) generates *G* completions per step and dispatches them to *G* parallel sandboxes.
3. **Sandbox reward**: Each sandbox runs a hidden pytest suite against the model's code and returns `tests_passed / total` as the reward signal (0.0–1.0).
4. **Why sandboxes are required**: Model-generated code is untrusted. Running it in-process during training would be unsafe. Each completion is fully isolated.
### GSPO vs GRPO
Both algorithms use clipped importance sampling, but at different granularities:
| Algorithm | IS clipping |
| :-------- | :---------------------------------------------- |
| **GRPO** | `clip(π_θ(t) / π_old(t))` per token |
| **GSPO** | `clip(∏_t π_θ(t) / π_old(t))` once per sequence |
For long function bodies, token-level clipping lets noisy individual tokens dominate the gradient. Sequence-level clipping treats the entire trajectory as one unit, which is a better fit for code generation tasks.
***
## Prerequisites
```bash theme={null}
pip install tensorlake transformers trl datasets torch rich python-dotenv
```
Create a `.env` file in your project root with your Tensorlake API key:
```
TENSORLAKE_API_KEY="your-api-key-here"
```
***
## TypeScript SDK starter
In Node.js, the critical part is still the reward oracle: each completion gets written into its own sandbox, the hidden pytest suite runs there, and the pass ratio becomes the reward.
```typescript theme={null}
import { Sandbox } from "tensorlake";
const encoder = new TextEncoder();
async function scoreCompletion(
solutionSource: string,
hiddenTests: string,
): Promise {
const sandbox = await Sandbox.create({
cpus: 1.0,
memoryMb: 1024,
timeoutSecs: 300,
allowInternetAccess: false,
});
try {
await sandbox.writeFile("/workspace/solution.py", encoder.encode(solutionSource));
await sandbox.writeFile("/workspace/test_hidden.py", encoder.encode(hiddenTests));
await sandbox.run("python", {
args: ["-m", "pip", "install", "pytest", "--user", "--break-system-packages"],
});
const result = await sandbox.run("python", {
args: ["-m", "pytest", "-q", "/workspace/test_hidden.py"],
workingDir: "/workspace",
timeout: 300,
});
const passed = Number(result.stdout.match(/(\d+) passed/)?.[1] ?? 0);
const failed = Number(result.stdout.match(/(\d+) failed/)?.[1] ?? 0);
return passed / Math.max(1, passed + failed);
} finally {
await sandbox.terminate();
}
}
const hiddenTests = `
from solution import sum_list
def test_basic():
assert sum_list([1, 2, 3]) == 6
`;
const completions = [
"def sum_list(nums):\n return sum(nums)",
"def sum_list(nums):\n return 0",
];
const rewards = await Promise.all(
completions.map((completion) => scoreCompletion(completion, hiddenTests)),
);
console.log(rewards);
client.close();
```
That reward function plugs into the same GSPO loop described below. The model/trainer side can stay in Python, but the sandbox evaluation path can be moved to TypeScript if your orchestration layer already lives there.
***
## Full example
The script below runs end-to-end: baseline evaluation → SFT warmup → GSPO fine-tuning → final evaluation. Pass `--smoke` for a fast 5-minute CPU run (3 tasks, 20 SFT steps, 1 GSPO epoch).
````python theme={null}
"""
RL GSPO Reasoner: Code Generation with Hidden Test Suites
===========================================================
Algorithm : GSPO, Group Sequence Policy Optimization (Zheng et al., 2507.18071)
GRPOConfig(importance_sampling_level="sequence")
Why sandboxes are non-negotiable here
--------------------------------------
The model generates arbitrary Python function bodies. Running untrusted
model-generated code in the training process directly would be unsafe.
Each completion is executed inside an isolated TensorLake sandbox.
The sandbox runs a hidden pytest suite and returns tests_passed/total as reward.
Training strategy
-----------------
Phase 1 - SFT warmup (N steps):
Supervised pass on correct solutions so the model outputs valid Python.
Without this, all G completions score 0 → reward_std=0 → no gradient.
Phase 2 - GSPO fine-tuning:
GRPOTrainer with sequence-level IS. The reward function dispatches G
parallel sandboxes per step and prints every completion that scores > 0.
Smoke : --smoke → 3 functions, 20 SFT steps, 1 GSPO epoch (~5 min CPU)
Full : 10 functions, 60 SFT steps, 3 GSPO epochs (~30 min CPU)
"""
from dotenv import load_dotenv
load_dotenv()
import re
import sys
import textwrap
import torch
from concurrent.futures import ThreadPoolExecutor, as_completed
from datasets import Dataset
from torch.optim import AdamW
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import GRPOTrainer, GRPOConfig
from tensorlake.sandbox import Sandbox
from rich.console import Console
from rich.panel import Panel
from rich.table import Table
from rich.rule import Rule
from rich import box
from typing import List
console = Console()
MODEL_NAME = "HuggingFaceTB/SmolLM2-135M-Instruct"
OUTPUT_DIR = "./gspo_coder"
SMOKE = "--smoke" in sys.argv
# ─── Dataset ──────────────────────────────────────────────────────────────────
TASKS = [
dict(
name="sum_list",
prompt=(
"Write a Python function:\n\n"
"def sum_list(nums: list) -> int:\n"
' """Return the sum of all integers in nums."""'
),
tests=textwrap.dedent("""\
from solution import sum_list
def test_empty(): assert sum_list([]) == 0
def test_single(): assert sum_list([5]) == 5
def test_mixed(): assert sum_list([1, 2, 3]) == 6
def test_neg(): assert sum_list([-1, -2, 3]) == 0
"""),
solution="def sum_list(nums: list) -> int:\n return sum(nums)",
),
dict(
name="is_palindrome",
prompt=(
"Write a Python function:\n\n"
"def is_palindrome(s: str) -> bool:\n"
' """Return True if s reads the same forwards and backwards."""'
),
tests=textwrap.dedent("""\
from solution import is_palindrome
def test_yes(): assert is_palindrome("racecar") is True
def test_no(): assert is_palindrome("hello") is False
def test_empty(): assert is_palindrome("") is True
def test_single(): assert is_palindrome("a") is True
"""),
solution="def is_palindrome(s: str) -> bool:\n return s == s[::-1]",
),
dict(
name="fizzbuzz",
prompt=(
"Write a Python function:\n\n"
"def fizzbuzz(n: int) -> list:\n"
' """Return a list 1..n: "Fizz" div by 3, "Buzz" div by 5,\n'
' "FizzBuzz" both, else the number as a string."""'
),
tests=textwrap.dedent("""\
from solution import fizzbuzz
def test_basic():
r = fizzbuzz(15)
assert r[2] == "Fizz"
assert r[4] == "Buzz"
assert r[14] == "FizzBuzz"
assert r[0] == "1"
def test_length(): assert len(fizzbuzz(5)) == 5
"""),
solution=(
'def fizzbuzz(n: int) -> list:\n'
' out = []\n'
' for i in range(1, n + 1):\n'
' if i % 15 == 0: out.append("FizzBuzz")\n'
' elif i % 3 == 0: out.append("Fizz")\n'
' elif i % 5 == 0: out.append("Buzz")\n'
' else: out.append(str(i))\n'
' return out'
),
),
dict(
name="count_vowels",
prompt=(
"Write a Python function:\n\n"
"def count_vowels(s: str) -> int:\n"
' """Return the number of vowels (a,e,i,o,u, case-insensitive) in s."""'
),
tests=textwrap.dedent("""\
from solution import count_vowels
def test_basic(): assert count_vowels("hello") == 2
def test_upper(): assert count_vowels("AEIOU") == 5
def test_none(): assert count_vowels("bcdf") == 0
def test_empty(): assert count_vowels("") == 0
"""),
solution=(
"def count_vowels(s: str) -> int:\n"
" return sum(1 for c in s.lower() if c in 'aeiou')"
),
),
dict(
name="flatten",
prompt=(
"Write a Python function:\n\n"
"def flatten(lst: list) -> list:\n"
' """Flatten one level of nesting: [[1,2],[3]] -> [1,2,3]."""'
),
tests=textwrap.dedent("""\
from solution import flatten
def test_basic(): assert flatten([[1,2],[3,4]]) == [1,2,3,4]
def test_empty(): assert flatten([]) == []
def test_single(): assert flatten([[1]]) == [1]
def test_mixed(): assert flatten([[1,2],[]]) == [1,2]
"""),
solution=(
"def flatten(lst: list) -> list:\n"
" return [x for sub in lst for x in sub]"
),
),
dict(
name="max_consecutive",
prompt=(
"Write a Python function:\n\n"
"def max_consecutive(nums: list) -> int:\n"
' """Return the length of the longest run of equal consecutive elements."""'
),
tests=textwrap.dedent("""\
from solution import max_consecutive
def test_basic(): assert max_consecutive([1,1,2,2,2,3]) == 3
def test_single(): assert max_consecutive([5]) == 1
def test_empty(): assert max_consecutive([]) == 0
def test_all(): assert max_consecutive([7,7,7]) == 3
"""),
solution=(
"def max_consecutive(nums: list) -> int:\n"
" if not nums: return 0\n"
" best = cur = 1\n"
" for a, b in zip(nums, nums[1:]):\n"
" cur = cur + 1 if a == b else 1\n"
" best = max(best, cur)\n"
" return best"
),
),
dict(
name="second_largest",
prompt=(
"Write a Python function:\n\n"
"def second_largest(nums: list) -> int | None:\n"
' """Return the second largest unique value, or None if fewer than 2 unique values."""'
),
tests=textwrap.dedent("""\
from solution import second_largest
def test_basic(): assert second_largest([3,1,4,1,5]) == 4
def test_two(): assert second_largest([2,1]) == 1
def test_dupes(): assert second_largest([1,1,1]) is None
def test_empty(): assert second_largest([]) is None
"""),
solution=(
"def second_largest(nums: list):\n"
" u = sorted(set(nums), reverse=True)\n"
" return u[1] if len(u) >= 2 else None"
),
),
dict(
name="run_length_encode",
prompt=(
"Write a Python function:\n\n"
"def run_length_encode(s: str) -> str:\n"
' """Run-length encode s: "aaabbc" -> "a3b2c1"."""'
),
tests=textwrap.dedent("""\
from solution import run_length_encode
def test_basic(): assert run_length_encode("aaabbc") == "a3b2c1"
def test_single(): assert run_length_encode("a") == "a1"
def test_empty(): assert run_length_encode("") == ""
def test_mixed(): assert run_length_encode("abcd") == "a1b1c1d1"
"""),
solution=(
"def run_length_encode(s: str) -> str:\n"
" if not s: return ''\n"
" out, cur, n = [], s[0], 1\n"
" for c in s[1:]:\n"
" if c == cur: n += 1\n"
" else: out.append(f'{cur}{n}'); cur, n = c, 1\n"
" out.append(f'{cur}{n}')\n"
" return ''.join(out)"
),
),
dict(
name="rotate_list",
prompt=(
"Write a Python function:\n\n"
"def rotate_list(lst: list, k: int) -> list:\n"
' """Return lst rotated right by k positions."""'
),
tests=textwrap.dedent("""\
from solution import rotate_list
def test_basic(): assert rotate_list([1,2,3,4,5], 2) == [4,5,1,2,3]
def test_zero(): assert rotate_list([1,2,3], 0) == [1,2,3]
def test_empty(): assert rotate_list([], 3) == []
def test_full(): assert rotate_list([1,2,3], 3) == [1,2,3]
"""),
solution=(
"def rotate_list(lst: list, k: int) -> list:\n"
" if not lst: return []\n"
" k = k % len(lst)\n"
" return lst[-k:] + lst[:-k] if k else lst[:]"
),
),
dict(
name="word_frequency",
prompt=(
"Write a Python function:\n\n"
"def word_frequency(text: str) -> dict:\n"
' """Return word -> count (case-insensitive, split on whitespace)."""'
),
tests=textwrap.dedent("""\
from solution import word_frequency
def test_basic(): assert word_frequency("the cat sat") == {"the":1,"cat":1,"sat":1}
def test_repeat(): assert word_frequency("a a b") == {"a":2,"b":1}
def test_case(): assert word_frequency("A a") == {"a":2}
def test_empty(): assert word_frequency("") == {}
"""),
solution=(
"def word_frequency(text: str) -> dict:\n"
" d = {}\n"
" for w in text.lower().split():\n"
" d[w] = d.get(w, 0) + 1\n"
" return d"
),
),
]
SYSTEM_PROMPT = (
"You are a Python coding assistant. "
"Write ONLY the function: no imports, no test code, no explanation. "
"Output raw Python starting with `def`."
)
# ─── Dataset helpers ──────────────────────────────────────────────────────────
def build_dataset(tasks: list) -> Dataset:
return Dataset.from_dict({
"prompt": [
[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": t["prompt"]},
]
for t in tasks
],
"tests": [t["tests"] for t in tasks],
})
def _extract_code(text) -> str:
if isinstance(text, list):
text = text[0]["content"] if text else ""
text = text or ""
m = re.search(r"```(?:python)?\s*(.*?)```", text, re.DOTALL)
return (m.group(1) if m else text).strip()
# ─── Sandbox reward ───────────────────────────────────────────────────────────
_HARNESS = """\
import sys, os, subprocess, re
sys.path.insert(0, "/tmp/pkgs")
if not os.path.isdir("/tmp/pkgs"):
subprocess.run(
["python3", "-m", "pip", "install", "pytest", "-q", "--target", "/tmp/pkgs"],
capture_output=True, check=False,
)
sys.path.insert(0, "/tmp/pkgs")
os.makedirs("/tmp/sol", exist_ok=True)
open("/tmp/sol/solution.py", "w").write({code!r})
open("/tmp/sol/test_sol.py", "w").write({tests!r})
r = subprocess.run(
["python3", "-m", "pytest", "/tmp/sol/test_sol.py", "--tb=no", "-q",
"--import-mode=importlib"],
capture_output=True, text=True,
env={{**os.environ, "PYTHONPATH": "/tmp/pkgs:/tmp/sol"}},
)
p = int((re.search(r"(\\d+) passed", r.stdout) or [0,0])[1])
f = int((re.search(r"(\\d+) failed", r.stdout) or [0,0])[1])
t = p + f
print(f"{{p}}/{{t}}")
"""
def _run_sandbox(code: str, tests: str) -> float:
harness = _HARNESS.format(code=code, tests=tests)
try:
box = Sandbox.create(memory_mb=2048)
ex = box.run("python3", ["-c", harness])
last = (ex.stdout or "").strip().splitlines()
last = last[-1] if last else "0/0"
p, t = (int(x) for x in last.split("/"))
return p / t if t > 0 else 0.0
except Exception:
return 0.0
# ─── Reward function: logs best completion of every batch ───────────────────
_reward_log: List[dict] = [] # accumulates {code, score, step} across training
_step = [0] # mutable counter (closure-friendly)
def reward_sandbox(completions, tests: List[str], **kwargs) -> List[float]:
"""
Reward = fraction of hidden pytest tests that pass (0.0–1.0).
G completions are dispatched to G parallel sandboxes.
Every batch whose best score > 0 is printed immediately.
"""
codes = [_extract_code(c) for c in completions]
_step[0] += 1
with ThreadPoolExecutor(max_workers=len(codes)) as pool:
futures = {pool.submit(_run_sandbox, code, test): i
for i, (code, test) in enumerate(zip(codes, tests))}
scores = [0.0] * len(codes)
for fut in as_completed(futures):
i = futures[fut]
scores[i] = fut.result()
_reward_log.append({"step": _step[0], "code": codes[i], "score": scores[i]})
best_i = max(range(len(scores)), key=lambda i: scores[i])
if scores[best_i] > 0:
console.print(
f"\n [bold green]↑ step {_step[0]} reward={scores[best_i]:.0%}"
f" ({int(scores[best_i]*4)}/4 tests)[/bold green]"
)
console.print(Panel(
codes[best_i],
title=f"[bold green]Best completion - step {_step[0]}[/bold green]",
border_style="green",
))
return scores
def print_top_completions(n: int = 3):
nonzero = [e for e in _reward_log if e["score"] > 0]
if not nonzero:
console.print("[yellow]No non-zero rewards recorded during training.[/yellow]")
return
top = sorted(nonzero, key=lambda e: e["score"], reverse=True)[:n]
console.print(Rule(f"[bold green]Top {len(top)} completions by reward[/bold green]", style="green"))
for rank, entry in enumerate(top, 1):
color = "green" if entry["score"] >= 0.75 else "yellow"
console.print(Panel(
entry["code"],
title=f"[bold]#{rank} reward={entry['score']:.0%} step={entry['step']}[/bold]",
border_style=color,
))
# ─── Phase 1: SFT warmup ──────────────────────────────────────────────────────
def sft_warmup(model, tokenizer, tasks: list, steps: int = 30):
"""
Brief supervised pass on correct solutions.
Teaches the model to emit valid Python before GSPO takes over.
Without this, reward_std=0 every step and GSPO has no gradient signal.
"""
console.print(Rule("[magenta]Phase 1: SFT warmup[/magenta]", style="magenta"))
console.print(
f"[dim]{steps} gradient steps on correct solutions "
f"({len(tasks)} tasks, cycling). Goal: non-zero reward_std in Phase 2.[/dim]\n"
)
optimizer = AdamW(model.parameters(), lr=2e-5)
model.train()
texts = []
for task in tasks:
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": task["prompt"]},
{"role": "assistant", "content": task["solution"]},
]
texts.append(tokenizer.apply_chat_template(messages, tokenize=False))
for step in range(1, steps + 1):
text = texts[(step - 1) % len(texts)]
enc = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
labels = enc["input_ids"].clone()
outputs = model(**enc, labels=labels)
outputs.loss.backward()
optimizer.step()
optimizer.zero_grad()
if step % max(1, steps // 5) == 0 or step == steps:
console.print(f" SFT step {step:3d}/{steps} loss={outputs.loss.item():.4f}")
del optimizer
console.print("[dim]SFT warmup done.\n[/dim]")
# ─── Evaluation ───────────────────────────────────────────────────────────────
def evaluate(model, tokenizer, tasks: list):
model.eval()
device = next(model.parameters()).device
t = Table(box=box.SIMPLE, show_header=True, header_style="bold white")
t.add_column("Function", width=20)
t.add_column("Tests", width=7, justify="right")
t.add_column("Generated code (first 55 chars)", width=57)
t.add_column("", width=5)
total = 0.0
for task in tasks:
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": task["prompt"]},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
enc = tokenizer(text, return_tensors="pt")
input_ids = enc["input_ids"].to(device)
attention_mask = enc["attention_mask"].to(device)
with torch.no_grad():
out = model.generate(
input_ids, attention_mask=attention_mask,
max_new_tokens=160, do_sample=False,
pad_token_id=tokenizer.eos_token_id,
)
response = tokenizer.decode(out[0][input_ids.shape[1]:], skip_special_tokens=True)
code = _extract_code(response)
score = _run_sandbox(code, task["tests"])
total += score
bar = "█" * int(score * 5) + "░" * (5 - int(score * 5))
color = "green" if score == 1.0 else "yellow" if score > 0 else "red"
t.add_row(
task["name"],
f"[{color}]{score:.0%}[/{color}]",
code.replace("\n", "↵ ")[:55],
f"[{color}]{bar}[/{color}]",
)
console.print(t)
avg = total / len(tasks)
console.print(f" Average test pass rate: [bold cyan]{avg:.1%}[/bold cyan]")
return avg
# ─── Main ─────────────────────────────────────────────────────────────────────
def train_gspo():
tasks = TASKS[:3] if SMOKE else TASKS
sft_steps = 20 if SMOKE else 60
gspo_epochs = 1 if SMOKE else 3
console.print(Panel(
"[bold green]RL GSPO: Code Generation with Hidden Test Suites[/bold green]\n\n"
"[dim]Algorithm : GSPO (sequence-level IS) - GRPOConfig(importance_sampling_level='sequence')\n"
"Model : SmolLM2-135M-Instruct (135 M params, CPU-friendly)\n"
"Task : Implement Python functions from docstrings\n"
"Reward : fraction of hidden pytest tests passing (sandbox oracle)\n"
"Sandboxes : G parallel TensorLake sandboxes per GSPO step\n"
"Phase 1 : SFT warmup - correct solutions so model starts generating valid Python\n"
"Phase 2 : GSPO - refines via reward signal from sandbox test results\n"
"GPU needed : No\n"
f"Mode : {'SMOKE (3 tasks, 20 SFT steps, 1 GSPO epoch)' if SMOKE else f'Full ({len(tasks)} tasks, {sft_steps} SFT steps, {gspo_epochs} GSPO epochs)'}[/dim]",
border_style="green",
))
console.print("\n[dim]Loading SmolLM2-135M-Instruct...[/dim]")
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, dtype=torch.float32)
split = max(1, int(0.75 * len(tasks)))
train_tasks = tasks[:split]
eval_tasks = tasks[split:]
console.print(f"[dim]{split} train tasks / {len(eval_tasks)} eval tasks[/dim]\n")
# ── Baseline ────────────────────────────────────────────────────────────
console.print(Rule("[cyan]Baseline: before any training[/cyan]", style="cyan"))
evaluate(model, tokenizer, eval_tasks)
# ── Phase 1: SFT warmup ─────────────────────────────────────────────────
sft_warmup(model, tokenizer, train_tasks, steps=sft_steps)
console.print(Rule("[cyan]After SFT warmup[/cyan]", style="cyan"))
evaluate(model, tokenizer, eval_tasks)
# ── Phase 2: GSPO ────────────────────────────────────────────────────────
console.print(Rule("[yellow]Phase 2: GSPO fine-tuning[/yellow]", style="yellow"))
console.print(
"[dim]Best completions printed live as reward > 0 is observed.\n"
"reward_std > 0 confirms the policy is exploring.[/dim]\n"
)
config = GRPOConfig(
output_dir=OUTPUT_DIR,
importance_sampling_level="sequence", # ← GSPO vs GRPO
num_generations=2 if SMOKE else 4,
max_completion_length=200,
temperature=1.4, # high temp forces diverse G completions → reward_std > 0
learning_rate=2e-6,
num_train_epochs=gspo_epochs,
per_device_train_batch_size=1,
gradient_accumulation_steps=2 if SMOKE else 4,
warmup_steps=5,
beta=0.001,
epsilon=0.2,
logging_steps=1,
save_steps=999,
seed=42,
report_to="none",
bf16=False,
fp16=False,
)
trainer = GRPOTrainer(
model=model,
args=config,
train_dataset=build_dataset(train_tasks),
reward_funcs=[reward_sandbox],
processing_class=tokenizer,
)
trainer.train()
# ── Results ──────────────────────────────────────────────────────────────
print_top_completions(n=3)
console.print(Rule("[cyan]After GSPO training[/cyan]", style="cyan"))
final_acc = evaluate(model, tokenizer, eval_tasks)
console.print(Panel(
f"[bold]Result: {final_acc:.0%} average test pass rate on held-out functions[/bold]\n\n"
"Context:\n"
" • Eval functions were [bold]never seen[/bold] during SFT or GSPO training\n"
" • Baseline before any training: [red]0%[/red]\n"
f" • After GSPO: [bold green]{final_acc:.0%}[/bold green]"
" ← model generalised from 7 training functions to unseen ones\n\n"
"Why 25 % is a reasonable outcome for this setup:\n"
" • 135 M params is the [italic]smallest[/italic] publicly available instruct model\n"
" • Only 60 SFT steps on 7 reference solutions (~5 min CPU)\n"
" • 25 % means 1 / 4 tests pass per function: the model correctly\n"
" handles the empty-input edge case on all three unseen functions,\n"
" showing the pattern [italic]transferred[/italic] across task types\n"
" • Typical zero-shot pass@1 for 135 M models on HumanEval is < 5 %\n\n"
"Cheap ways to push higher (no extra hardware):\n"
" 1. [cyan]temperature=1.4[/cyan] (already set) - forces reward_std > 0 so GSPO\n"
" has a gradient signal instead of collapsing to all-zero advantages\n"
" 2. More SFT examples (50+ functions, ~10 min) before GSPO\n"
" 3. Switch to [cyan]Qwen2.5-0.5B-Instruct[/cyan] (4× more params, same CPU time)",
title="[bold cyan]Score interpretation[/bold cyan]",
border_style="cyan",
))
console.print(Panel(
"[bold]Why GSPO + sandbox here?[/bold]\n\n"
"1. [cyan]Sandboxes required[/cyan]: model code is untrusted and cannot run in-process.\n\n"
"2. [cyan]Hidden test suites[/cyan]: the model never sees the tests.\n"
" Sandbox is the only oracle → no reward hacking.\n\n"
"3. [cyan]GSPO over GRPO[/cyan]: long function bodies mean many tokens.\n"
" Token-level IS clipping (GRPO) lets noisy tokens dominate the gradient.\n"
" Sequence-level clipping (GSPO) clips the whole trajectory once:\n\n"
" GRPO: clip( π_θ(t)/π_old(t) ) per token\n"
" GSPO: clip( Π_t π_θ(t)/π_old(t) ) once per sequence",
title="[bold cyan]Design rationale[/bold cyan]",
border_style="cyan",
))
if __name__ == "__main__":
train_gspo()
````
***
## What happens step-by-step
| Step | Phase | What happens |
| :---- | :------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **1** | Setup | Model and tokenizer loaded. Tasks split 75/25 into train and eval sets. |
| **2** | Baseline | Eval tasks run through the untrained model and scored via sandbox. Typically \~0%. |
| **3** | SFT warmup | N supervised gradient steps on correct reference solutions. Ensures the model produces parseable Python before RL begins. |
| **4** | After SFT | Eval re-run. Reward variance should now be non-zero, a prerequisite for GSPO to have a gradient signal. |
| **5** | GSPO loop | For each training step, *G* completions are generated and dispatched to *G* parallel sandboxes. Each sandbox runs the hidden pytest suite and returns a score. |
| **6** | Reward signal | `reward_sandbox` collects scores, logs the best completion, and returns the score list to `GRPOTrainer`. |
| **7** | Final eval | Held-out functions (never seen during training) are evaluated. A 25% pass rate on a 135M parameter model is the expected outcome. |
***
## Key design decisions
### Why `temperature=1.4`
GSPO requires diversity across the *G* completions in each group to produce a non-zero reward standard deviation. If all completions are identical (low temperature), `reward_std = 0` and the advantage normalization produces zero gradients, so training stalls. Setting temperature high forces the model to explore different implementations.
### Why SFT warmup is required
Without warmup, a randomly-initialized or instruction-tuned model produces malformed Python that scores 0 on every test case. All-zero rewards mean all-zero advantages after normalization, and GSPO has nothing to optimize. Even 20 supervised steps on correct solutions is enough to bootstrap non-zero reward variance.
### Why sandboxes prevent reward hacking
The model never has access to the test file. The only feedback is the pass rate returned by the sandbox. This makes it impossible for the model to overfit to specific assertion patterns. It must actually implement the correct logic.
***
This example uses `python-dotenv` to load your Tensorlake API key. Create a `.env` file in your project root:
```
TENSORLAKE_API_KEY="your-api-key-here"
```
The SDK will pick it up automatically.
## What to build next
Use a sandbox as a tool inside an agentic LLM loop.
Dispatch parallel sandboxes across a swarm of worker agents.
# Harbor
Source: https://docs.tensorlake.ai/sandboxes/harbor
Run Harbor evaluations and RL rollouts on Tensorlake Sandboxes: fresh isolation per trial, pre-warmed snapshots for expensive environments, and independent test verification.
[Harbor](https://github.com/harbor-framework/harbor) is a framework from the creators of [Terminal-Bench](https://www.tbench.ai/) for evaluating and optimizing agents and language models. With Harbor you can evaluate arbitrary agents (Claude Code, OpenHands, Codex CLI, and others) against curated datasets like Terminal-Bench, SWE-Bench, and Aider Polyglot, build and share your own benchmarks, run thousands of trials in parallel across cloud providers, and generate rollouts for RL optimization.
Harbor abstracts the execution backend behind an `--env` flag. Tensorlake plugs in as one of those providers (alongside other sandboxes and local Docker), so the same Harbor commands run on Tensorlake sandboxes without changing your tasks, agents, or evaluators.
This guide focuses on running CLI-agent evaluations against benchmarks like Terminal-Bench. Harbor also supports generating rollouts for RL optimization. We'll cover those workflows in follow-up guides.
New to Tensorlake? Sign up at the [dashboard](https://cloud.tensorlake.ai). New accounts include free credits, enough to run a full Terminal-Bench sweep before you pay for anything.
## Quick start
Grab one from the [Tensorlake Dashboard](https://cloud.tensorlake.ai). You'll also need an API key for whichever agent provider you want to evaluate (e.g., Anthropic).
The `harbor[tensorlake]` extra installs the `TensorLakeEnvironment` provider alongside Harbor.
```bash theme={null}
uv pip install "harbor[tensorlake]"
```
```bash theme={null}
pip install "harbor[tensorlake]"
```
```bash theme={null}
export TENSORLAKE_API_KEY="tl_..."
export ANTHROPIC_API_KEY="sk-ant-..." # or another agent provider
```
Run a single Terminal-Bench task on Tensorlake with Claude Code as the agent:
```bash theme={null}
harbor run --env tensorlake \
--include-task-name terminal-bench/pytorch-model-cli \
--dataset terminal-bench/terminal-bench-2-1 \
--agent claude-code \
--model anthropic/claude-sonnet-4-6 \
--ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY
```
Drop `--include-task-name` to run the full Terminal-Bench 2.1 suite. `--ae KEY=VALUE` forwards an environment variable from your shell into the sandbox where the agent runs. Add more `--ae` flags for any other secrets the agent needs.
## Why Tensorlake for Harbor
Harbor's value comes from running large fleets of environments in parallel and trusting the results. Tensorlake's runtime is designed for exactly that workload:
* **Per-trial sandboxes**: each task starts on a clean machine and is destroyed at the end. No shared kernel state between trials, which matters for both eval reproducibility and RL reward integrity.
* **Full task-environment support**: Tensorlake imports a task's real Docker image and converts it into a sandbox image that boots directly, so every trial runs the exact environment the benchmark defines rather than one approximated by replaying a Dockerfile. That closes the environment gap that otherwise quietly skews results.
* **Pre-warmed snapshots**: environments with heavy `apt`/`pip` installs (PyTorch, CUDA toolchains, full Linux desktops) can be built once, snapshotted, and restored under a second for every subsequent trial or rollout.
* **Independent verification**: Harbor's test script runs inside the sandbox and writes `1.0` or `0.0` to `reward.txt`. The agent never sees or touches the verifier, so "the agent said it worked" is never confused with "the tests pass."
* **Parallel scale**: Tensorlake schedules thousands of sandboxes concurrently, which is what RL rollout generation and full benchmark sweeps need.
## Anatomy of a Harbor task
Harbor expects each task to be laid out like this - take [gcode-to-text](https://github.com/harbor-framework/terminal-bench-2/tree/main/gcode-to-text) as an example:
```
gcode-to-text
├── environment
│ ├── Dockerfile
│ └── text.gcode.gz
├── instruction.md
├── solution
│ └── solve.sh
├── task.toml
└── tests
├── test_outputs.py
└── test.sh
```
* `environment/Dockerfile` defines the base image and any setup steps.
* `instruction.md` is the prompt the agent receives.
* `solution/` is an oracle reference used to validate the environment itself.
* `tests/test.sh` runs after the agent finishes and produces `reward.txt`.
## Tune sandbox resources
Each task's `task.toml` controls the sandbox Harbor provisions on Tensorlake. Set resources in the `[environment]` block:
```toml task.toml theme={null}
[environment]
cpus = 2
memory_mb = 4096
storage_mb = 20480
allow_internet = true
```
| Field | Default | Forwarded to Tensorlake |
| ---------------- | ------- | ----------------------- |
| `cpus` | `1` | `cpus` |
| `memory_mb` | `2048` | `memory_mb` |
| `storage_mb` | `10240` | `ephemeral_disk_mb` |
| `allow_internet` | `true` | `allow_internet_access` |
Tensorlake requires `memory_mb` to be between 1024 and 8192 MB per CPU core.
A few rules of thumb:
* **Large or heavy images**: if your `environment/Dockerfile` pulls in big toolchains (PyTorch, CUDA, full Linux desktops, large datasets), bump `cpus` and `memory_mb` so the build and runtime have headroom, and raise `storage_mb` past the image size plus working-set room. Underprovisioned sandboxes show up as build timeouts or OOMs mid-trial.
* **Lock down `allow_internet`**: set `allow_internet = false` to stop the agent from searching the web for answers. If the verifier needs network access, bake those dependencies into the Dockerfile. Per-host allowlists are coming soon, so you'll be able to block search engines while leaving package mirrors reachable.
## Image build & caching
Each trial boots from an image. Harbor uses a prebuilt image when the task declares one, and otherwise builds the task's Dockerfile:
| Source | How to set it | When to use |
| ------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------- |
| **Prebuilt image** | `docker_image` in `task.toml` | Fastest start: boot directly from an image with the environment already baked in. Terminal-Bench ships these. |
| **Dockerfile** | `environment/Dockerfile` | No prebuilt image declared. Built once, cached, and reused across trials. |
Either way the image is built or imported **once** and reused. You only pay the cost on the first trial, then every later trial boots directly from the cached image. If a task sets both, the prebuilt image wins over the Dockerfile if it's exist.
Reusing one heavy environment across many runs (RL rollouts or repeated eval of the same task) can restore in under a second from a pre-warmed snapshot instead of rebuilding. See [Snapshots](/sandboxes/snapshots).
### Prebuilt image
If a task declares a `docker_image` in `task.toml`, Harbor boots directly from that image and skips the Dockerfile entirely:
```toml task.toml theme={null}
[environment]
docker_image = "myorg/my-task-env:2025-06"
```
Harbor looks the image up in Tensorlake by name and boots from it; if it isn't registered yet, it imports the image once and reuses it on every later trial. The registered name is derived from the reference string you put here, so the **first** import of a given reference is what every later run boots.
**Always publish with an immutable tag, never `latest`.** Because the registered name comes from the reference string (not the image contents), a tag like `latest` is captured at its first import and then frozen: if you push new content to `latest`, Harbor keeps booting the old image and never re-pulls. Use an immutable tag or digest (e.g. `myorg/my-task-env:2025-06` or `...@sha256:...`) so a new build means a new reference, which is what triggers a fresh import. This is the convention the published Terminal-Bench images follow.
To refresh an image that's already registered, point `docker_image` at a new immutable tag/digest, or delete the registered Tensorlake image so the next run re-imports it. (`--force-build` does **not** re-import: it builds from the Dockerfile instead.)
**Terminal-Bench 2.1 images are already published.** We've registered every Terminal-Bench 2.1 task image publicly, so anyone with Tensorlake access boots straight from them: no build, no import. Just run the dataset as usual and each task picks up its published image.
**Set your org and project context.** Looking an image up by name requires organization and project context. Without it, Harbor can't find the published image and falls back to importing it fresh. You'll see a line like this in the logs:
```
importing docker_image alexgshaw/pypi-server:20251031 directly: Looking up a sandbox image by name requires organization and project context (TENSORLAKE_ORGANIZATION_ID and TENSORLAKE_PROJECT_ID).
```
Set both before running so the lookup resolves and the boot stays instant:
```bash theme={null}
export TENSORLAKE_ORGANIZATION_ID="..."
export TENSORLAKE_PROJECT_ID="..."
```
Harbor also reads these from `~/.tensorlake/config.toml` (`organization` and `project`) if it's present.
### Dockerfile
If a task has no `docker_image`, Harbor builds its `environment/Dockerfile` once via Tensorlake's image builder, caches it, and boots every later trial directly from the cached image: no per-trial `apt`/`pip` work. The cache is keyed on the Dockerfile **and every file in the build context**, so editing a `requirements.txt` pin or any `COPY`'d file automatically triggers a rebuild.
```bash theme={null}
harbor run --env tensorlake \
--dataset terminal-bench/terminal-bench-2-1 \
--agent claude-code \
--model anthropic/claude-sonnet-4-6
```
If a build ever fails, Harbor automatically falls back to replaying the Dockerfile's `RUN`/`COPY` steps on each trial, so a trial is never blocked. It just runs a little slower. The fallback is also available as an explicit escape hatch while you iterate on a Dockerfile:
```bash theme={null}
harbor run --env tensorlake ... --ek use_oci_image_build=false
```
**Dockerfile requirements**
The image builder is stricter than a local `docker build`, so a few Docker conventions need small adjustments:
* **`COPY` does not auto-create parent directories**: `COPY x /a/b/c` fails if `/a/b` doesn't exist yet. Add `RUN mkdir -p /a/b` before the `COPY`.
* **Don't pin exact apt versions** (`apt-get install curl=8.5.0-2ubuntu10.6`): drop the pin or pick a version that exists in the target distro.
* **Use a FROM image that ships the Python you need** (e.g. `python:3.10-bookworm`) rather than relying on a non-native version being fetched at build time.
To force a fresh rebuild even when a valid cached image exists, add `--force-build`. This applies to that run only and doesn't disturb the cache used by subsequent normal runs.
```bash theme={null}
harbor run --env tensorlake \
--force-build \
--dataset terminal-bench/terminal-bench-2-1 \
--agent claude-code \
--model anthropic/claude-sonnet-4-6
```
### Sharing images publicly
By default, images Harbor builds or imports are **private to your organization**: only your org can boot from them. Add `--ek is_public=true` to register a freshly built or imported image as public, so any organization with Tensorlake access can reuse it:
```bash theme={null}
harbor run --env tensorlake \
--ek is_public=true \
--dataset terminal-bench/terminal-bench-2-1 \
--agent claude-code \
--model anthropic/claude-sonnet-4-6
```
The flag applies to both Dockerfile-built images and prebuilt `docker_image` imports. Automatic boot-from-public by another organization is wired through the **prebuilt `docker_image`** path (that's how the published Terminal-Bench images are reused), so if your goal is to publish an environment others boot directly, prefer a `docker_image` reference.
Publishing public images is gated to an allow list. If your account isn't on it, the flag is ignored and the image stays private. Reach out to Tensorlake to be added.
`is_public` only takes effect when the image is **newly registered**. If an image with the same name already exists (a private copy from an earlier run, or an existing public one), Harbor boots it as-is and won't republish it. To turn an already-private image public, delete it first (or change the build context so it gets a new name), then rerun with `--ek is_public=true`.
### Ad-hoc native dependencies
If a task just needs a couple of extra apt packages and you don't want to edit the Dockerfile or maintain a snapshot, use `preinstall_packages`:
```bash theme={null}
harbor run --env tensorlake \
--ek 'preinstall_packages=["build-essential","rustc","cargo"]' \
--dataset terminal-bench/terminal-bench-2-1 \
--agent claude-code \
--model anthropic/claude-sonnet-4-6
```
The packages are installed at the start of each trial. Prefer snapshots when the package set is large or reused across many runs so you pay the install cost once.
## Interactive debugging
When a trial fails and you want to poke around the live environment, attach to the session:
```bash theme={null}
harbor env attach
```
Drop directly into the running sandbox to inspect state, rerun tests by hand, and confirm whether the failure was the agent or the environment.
## Structured logs
Each trial produces structured artifacts, e.g.:
```
gcode-to-text__UFALMLv
├── agent/
├── verifier/
├── result.json
└── trial.log
```
So you can trace:
* The agent's actions and outputs
* What the verifier checked
* Why the trial passed or failed
## What to build next
Build an environment once, snapshot it, and restore in seconds for every trial.
Use sandboxes as a deterministic reward oracle for RL training loops.
# Build and Import Images
Source: https://docs.tensorlake.ai/sandboxes/images
Import or build custom images and run them in sandboxes.
Sandbox images let you set up dependencies, files, and environment once, then launch fresh sandboxes from that prepared state.
Define an image with a Dockerfile, the Python SDK, or the TypeScript SDK (or import an existing registry image directly), then pass the registered name to `image=` when creating sandboxes.
The usual flow is:
1. Choose a base image.
2. Define the setup steps with a Dockerfile or `Image` object.
3. Build and register the image name in your project.
4. Create sandboxes from that registered name.
## Choose a Base Image
You can use any image the build can pull as your `FROM` base: a [Tensorlake image](/sandboxes/tensorlake-images), a public OCI reference, or a private registry image.
The [Tensorlake images](/sandboxes/tensorlake-images) (`tensorlake/ubuntu-minimal`, `tensorlake/ubuntu-systemd`, `tensorlake/debian-minimal`) boot quickly and are tuned for sandbox workloads. Building on one does not carry over its runtime behavior: the [`tl-user` default user and working directory](/sandboxes/tensorlake-images#default-user-and-working-directory) apply only when you run a Tensorlake image directly. Your built image runs with whatever `USER` and `WORKDIR` its Dockerfile sets, or Docker defaults otherwise. If your image needs systemd services such as Docker or Kubernetes, base it on `tensorlake/ubuntu-systemd`.
## Build and Register an Image
You can define the same image with a Dockerfile, Python, or TypeScript. The build runs the setup steps and registers the result under the image name in your project.
```dockerfile Dockerfile theme={null}
FROM tensorlake/ubuntu-systemd
RUN apt-get update && apt-get install -y python3 python3-pip
COPY requirements.txt /tmp/requirements.txt
RUN python3 -m pip install --break-system-packages -r /tmp/requirements.txt
RUN mkdir -p /workspace/cache
ENV APP_ENV=prod
WORKDIR /workspace
```
```bash theme={null}
tl sbx image create ./Dockerfile --registered-name data-tools-image
```
```python theme={null}
from tensorlake import Image
image = (
Image(name="data-tools-image", base_image="tensorlake/ubuntu-systemd")
.copy("requirements.txt", "/tmp/requirements.txt")
.run("apt-get update && apt-get install -y python3 python3-pip")
.run("python3 -m pip install --break-system-packages -r /tmp/requirements.txt")
.run("mkdir -p /workspace/cache")
.env("APP_ENV", "prod")
.workdir("/workspace")
)
image.build(registered_name="data-tools-image", context_dir=".")
```
```typescript theme={null}
import { Image } from "tensorlake";
const image = new Image({
name: "data-tools-image",
baseImage: "tensorlake/ubuntu-systemd",
})
.copy("requirements.txt", "/tmp/requirements.txt")
.run("apt-get update && apt-get install -y python3 python3-pip")
.run("python3 -m pip install --break-system-packages -r /tmp/requirements.txt")
.run("mkdir -p /workspace/cache")
.env("APP_ENV", "prod")
.workdir("/workspace");
await image.build({
registeredName: "data-tools-image",
contextDir: ".",
});
```
In the SDKs, `context_dir` (`contextDir` in TypeScript) is optional and works like the build context in `docker build `. Pass it when the `Image` reads host files (through `copy()`, `add()`, or a `RUN --mount=type=bind`) so those sources resolve relative to it. Omit it otherwise.
### Build from an OCI Base
The build base can be any standard OCI image reference, not just `tensorlake/*`, for example `python:3.12-slim`, `debian:bookworm-slim`, `node:22-alpine`, `ghcr.io/...`, or `public.ecr.aws/...`.
```dockerfile Dockerfile theme={null}
FROM python:3.12-slim
RUN apt-get update && apt-get install -y curl
RUN python3 -m pip install pandas pyarrow duckdb
WORKDIR /workspace
```
```bash theme={null}
tl sbx image create ./Dockerfile --registered-name py-data-tools
```
The first build from a new OCI base takes longer because the upstream image has to be fetched and prepared. Subsequent builds are faster.
### Private Registries
If you can `docker pull` an image from a private registry, you can use it as a base or dependency in your sandbox image's Dockerfile. Authenticate with `docker login`, then run the build:
```bash theme={null}
docker login ghcr.io
tl sbx image create ./Dockerfile --registered-name my-private-image
```
`docker login` works with all private registries, including Docker Hub, GHCR, ECR, GCR, Quay, and self-hosted. During the build, the Tensorlake CLI and SDKs read registry credentials from `~/.docker/config.json` (or `$DOCKER_CONFIG/config.json` if `DOCKER_CONFIG` is set) and use them to pull private base images and dependencies. If the credentials are missing or expired, the build fails when it tries to pull from the private registry.
This also works in CI. For example, if you authenticate to ECR with [amazon-ecr-login](https://github.com/aws-actions/amazon-ecr-login) in a GitHub Actions workflow, `tl sbx image create` and SDK calls in the same workflow pick up those credentials.
## Import an Image from a Registry
To use an existing registry image as a sandbox image without adding any build steps, import it directly. There is no Dockerfile and no build context, and the reference is always pulled fresh from the registry. Sandboxes run the imported image with whatever user, working directory, and environment it defines.
Use this when you want a published image (`ubuntu:24.04`, `pytorch/pytorch:2.4.1-cuda12.1-cudnn9-runtime`, `ghcr.io/org/app:v1`) as-is. If you need to layer extra packages, files, or environment on top, write a Dockerfile that uses it as a `FROM` base instead. See [Build from an OCI Base](#build-from-an-oci-base).
```bash theme={null}
tl sbx image import pytorch/pytorch:2.4.1-cuda12.1-cudnn9-runtime \
--registered-name pytorch-runtime
```
```python theme={null}
from tensorlake import import_sandbox_image
import_sandbox_image(
"pytorch/pytorch:2.4.1-cuda12.1-cudnn9-runtime",
registered_name="pytorch-runtime",
)
```
```typescript theme={null}
import { importSandboxImage } from "tensorlake";
await importSandboxImage(
"pytorch/pytorch:2.4.1-cuda12.1-cudnn9-runtime",
{ registeredName: "pytorch-runtime" },
);
```
If you omit the registered name, it defaults to the reference's last path segment with any tag or digest stripped (`pytorch/pytorch:2.4.1` → `pytorch`, `ghcr.io/org/app@sha256:...` → `app`).
Imports use the same `docker login` credentials as Dockerfile builds, so private references work the same way (see [Private Registries](#private-registries)). The same CPU, memory, disk, and visibility options apply as for builds (see [Build Resources](#build-resources) and [Public Images](#public-images)).
## Launch Sandboxes from an Image
Create a sandbox from the registered image name. You can still override CPU, memory, disk, timeout, and entrypoint when the sandbox starts.
```bash theme={null}
tl sbx create --image data-tools-image
```
```bash theme={null}
tl sbx create \
--image data-tools-image \
--cpus 4.0 \
--memory 4096 \
--disk_mb 51200 \
--timeout 1800
```
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create(
image="data-tools-image",
cpus=4.0,
memory_mb=4096,
disk_mb=51200,
timeout_secs=1800,
)
try:
result = sandbox.run(
"python3",
["-c", "import pandas, pyarrow; print('ready')"],
)
print(result.stdout)
finally:
sandbox.terminate()
```
```typescript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.create({
image: "data-tools-image",
cpus: 4.0,
memoryMb: 4096,
diskMb: 51200,
timeoutSecs: 1800,
});
try {
const result = await sandbox.run("python3", {
args: ["-c", "import pandas, pyarrow; print('ready')"],
});
console.log(result.stdout);
} finally {
await sandbox.terminate();
}
```
You can't launch a sandbox directly from a Docker/registry image reference. It has to be registered as a Tensorlake image first. The quickest way to do that for an unmodified image is [Import an Image from a Registry](#import-an-image-from-a-registry), which registers it in one step with no Dockerfile. We are working on launching public registry images directly without a separate registration step.
## Build Resources
Builds run in a temporary builder sandbox. You can allocate more CPU, memory, or disk for the builder, and separately set the root disk size of the resulting image.
```bash theme={null}
tl sbx image create ./Dockerfile \
--registered-name data-tools-image \
--cpus 4 \
--memory 4096 \
--disk_mb 25600 \
--builder_disk_mb 32768
```
```python theme={null}
image.build(
registered_name="data-tools-image",
cpus=4.0,
memory_mb=4096,
disk_mb=25600,
builder_disk_mb=32768,
)
```
```typescript theme={null}
await image.build({
registeredName: "data-tools-image",
cpus: 4.0,
memoryMb: 4096,
diskMb: 25600,
builderDiskMb: 32768,
});
```
`disk_mb` / `diskMb` sets the root disk size for sandboxes created from the registered image. `builder_disk_mb` / `builderDiskMb` only affects the temporary builder sandbox.
Build defaults are `cpus=2.0`, `memory=4096 MB`, and a generated root disk of `10240 MiB` (10 GiB).
### Docker Compatibility Mode
`--docker_compat` runs the build or import with standard Docker/BuildKit instead of Tensorlake's default builder. Turn it on if a build or import fails or produces an unexpected result under the default builder, it trades speed and disk for maximum compatibility. Budget at least 3× the builder disk and memory (via the resource flags above). The flag works on both builds and imports; leave it off unless you need it.
```bash theme={null}
tl sbx image create ./Dockerfile \
--registered-name data-tools-image \
--docker_compat
```
```python theme={null}
image.build(registered_name="data-tools-image", docker_compat=True)
```
```typescript theme={null}
await image.build({
registeredName: "data-tools-image",
dockerCompat: true,
});
```
## Register an Existing Snapshot as an Image
If you already have a completed filesystem snapshot, you can give it a reusable image name without rebuilding:
```bash theme={null}
tl sbx image register data-tools-image snap_01HX... \
--dockerfile ./Dockerfile
```
The first positional argument is the image name to register, the second is the completed snapshot ID, and `--dockerfile` is stored alongside the image so `tl sbx image describe` can show how it was built. Add `--public` to make the name resolvable from any namespace (see [Public Images](#public-images)).
The snapshot must be in `Completed` status with a durable `snapshot_uri`; `tl sbx image register` rejects snapshots that haven't finished uploading.
## Inspect and List Registered Images
List the images registered in your project, or look one up by name, from the CLI or the SDKs.
```bash theme={null}
tl sbx image ls # list every image registered in the current project
tl sbx image describe data-tools-image # show Dockerfile, snapshot ID, image size
```
`describe` accepts either the registered image name or the underlying sandbox-template ID.
```python theme={null}
from tensorlake import find_sandbox_image_by_name, list_sandbox_images
images = list_sandbox_images() # every image registered in the current project
image = find_sandbox_image_by_name("data-tools-image") # None if no such image exists
if image is not None:
print(image["id"], image["snapshot_id"])
```
```typescript theme={null}
import { findSandboxImageByName, listSandboxImages } from "tensorlake";
const images = await listSandboxImages(); // every image registered in the current project
const image = await findSandboxImageByName("data-tools-image"); // null if no such image exists
if (image) {
console.log(image.id, image.snapshotId);
}
```
The SDK list and lookup calls use the same environment-based Tensorlake auth as image builds, and require organization and project context (`TENSORLAKE_ORGANIZATION_ID` and `TENSORLAKE_PROJECT_ID`).
## Public Images
By default a registered image is namespace-scoped. Pass `--public`, `is_public=True`, or `isPublic: true` to make the image name resolvable from any namespace. This is how the `tensorlake/*` base images work.
```bash theme={null}
tl sbx image create ./Dockerfile --registered-name shared-base --public
```
```python theme={null}
image.build(registered_name="shared-base", is_public=True)
```
```typescript theme={null}
await image.build({
registeredName: "shared-base",
isPublic: true,
});
```
Public image names must be globally unique for the registry. Names that collide with an already-registered public image will be rejected at creation time.
## Examples
### Skills Image
This variant preloads the [Tensorlake skills repo](/agent-skills) so coding agents can auto-discover it at startup:
```dockerfile Dockerfile theme={null}
FROM tensorlake/ubuntu-systemd
RUN apt-get update && apt-get install -y git nodejs npm python3 python3-pip
RUN npm install -g skills
RUN skills add tensorlakeai/tensorlake-skills --all -y --copy
RUN python3 -m pip install --break-system-packages tensorlake
```
If the file is named `Dockerfile`, the registered name defaults to the parent directory name. Otherwise it defaults to the file stem. Registered image names must be unique within a project.
## Supported Build Operations and Limitations
Sandbox image builds support most of the standard Dockerfile commands and features, but with some limitations:
* Dockerfile `$VAR` and environment variable substitution is not working in `FROM` commands
* Dockerfile `ONBUILD` commands are ignored and do not run during child image builds
* The following Dockerfile commands work as expected during image builds but do not have any effect when running sandboxes from the images:
* `ONBUILD`
* `SHELL`
* `EXPOSE`
* `HEALTHCHECK`
* `LABEL`
* `STOPSIGNAL`
* `VOLUME`
## See Also
The managed `tensorlake/*` images: what ships in them and how they behave at runtime.
Understand the underlying snapshot primitive used to save and restore sandbox state.
Learn which sandbox settings you can still override when launching from an image.
Ship Tensorlake SDK docs inside sandbox images for agents and tools.
# Sandbox and Orchestration Infrastructure for Agents
Source: https://docs.tensorlake.ai/sandboxes/introduction
Tensorlake provides isolated MicroVM sandboxes that boot in hundreds of milliseconds, with memory and filesystem preserved across suspend/resume.
Get setup in a few minutes, and start a sandbox in a few seconds.
Sandboxes can be used to run agent harnesses, run tool calls or even as VMs for running coding agents, builds and IDEs.
## How it works
Sandboxes are created on-demand via API calls, and they are MicroVMs backed by Firecracker and CloudHypervisor. You can specify images and
resources when creating them. The default image, `tensorlake/ubuntu-minimal`, starts up in a few hundred milliseconds, while `tensorlake/ubuntu-systemd`
has a full init system and more tools and takes around 1 second to boot.
```bash cli theme={null}
tl sbx create
```
```python sandbox.py theme={null}
from tensorlake.sandbox import Sandbox
resp = Sandbox.create(
image="tensorlake/ubuntu-minimal",
cpus=4,
memory_mb=8192,
)
```
```typescript sandbox.ts theme={null}
import { Sandbox } from "tensorlake";
const resp = await Sandbox.create({
image: "tensorlake/ubuntu-minimal",
cpus: 4,
memoryMb: 8192,
});
```
#### Start Using Sandboxes
Install the SDK and run your first sandbox.
The mental model behind ephemeral, named, suspend, and snapshot.
Use and customize sandbox images for your use case.
How to persist sandbox state across runs with suspend and snapshots.
## Trust and support
Tensorlake is HIPAA and SOC 2 Type II compliant, supports EU data residency, and offers zero data retention.
Chat with our engineers.
[support@tensorlake.ai](mailto:support@tensorlake.ai)
Use cases and product updates.
# Lifecycle
Source: https://docs.tensorlake.ai/sandboxes/lifecycle
Sandbox states, creation, suspend/resume, and cleanup
## Overview
Sandboxes come in two flavors:
* **Ephemeral**: no name. Runs until you terminate it or it times out. Cannot be suspended.
* **Named**: a name given at creation (or assigned later). Supports suspend and resume, so you can pause between tasks and pick up exactly where you left off.
| | Ephemeral | Named |
| -------------------- | ------------------------------------ | ---------------------------------------- |
| **Created with** | `tl sbx create` | `tl sbx create ` |
| **Suspend / Resume** | Not supported | Supported |
| **Reference by** | ID only | ID **or** name |
| **Use when** | Short-lived tasks, one-off execution | Multi-step work, persistent environments |
## Lifecycle states
Every sandbox moves through the states below. Create starts the sandbox in `Pending`; from `Running`, you can suspend (named only), snapshot, or terminate. Ephemeral sandboxes follow the same flow but skip `Suspending`/`Suspended`.
```mermaid theme={null}
stateDiagram-v2
[*] --> Pending: • create
• restore from snapshot
Pending --> Running
Running --> Snapshotting: snapshot
Snapshotting --> Running: snapshot complete
Running --> Suspending: named sandbox
• suspend
• timeout
Suspending --> Suspended
Suspended --> Running: resume
Running --> Terminated: • terminate
• timeout (ephemeral)
Suspended --> Terminated: terminate
Terminated --> Pending: restart
(within 48 hours)
Terminated --> [*]
style Suspending fill:#E8F4FF,stroke:#1D70B8,color:#0B3C6F,stroke-width:2px
style Suspended fill:#E8F4FF,stroke:#1D70B8,color:#0B3C6F,stroke-width:2px
```
| State | What it means | How you exit it |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| **Pending** | Sandbox is being scheduled and booted. Not yet ready to accept commands. | Transitions to `Running` automatically once boot completes. |
| **Running** | Sandbox is live and accepting commands, file operations, and process execution. Snapshots can be taken from this state. | Call `suspend` (named only) or `terminate`. |
| **Snapshotting** | A reusable snapshot artifact is being captured from the sandbox's filesystem, memory, and running processes. | Returns to `Running` when capture completes. |
| **Suspending** | Named sandbox is being paused in place. Triggered by manual suspend or by `timeout_secs` elapsing. | Transitions to `Suspended` automatically. |
| **Suspended** | Named sandbox is paused. Consumes no compute; state is preserved for resume under the same sandbox ID. | Call `resume` to return to `Running`, or `terminate` to end it. |
| **Terminated** | Sandbox has stopped, either manually or via timeout for ephemeral sandboxes. | Call `restart` within 48 hours of termination to boot it again under the same ID and name; after that the sandbox is gone. |
## Suspend vs. snapshot
Suspend and snapshot both preserve sandbox state, but they serve different purposes:
* **Suspend** pauses *this* sandbox so you can resume it later under the same ID.
* **Snapshot** captures a reusable artifact you can restore into a *new* sandbox.
Suspend/resume is covered on this page; see [Snapshots](/sandboxes/snapshots) for save-and-restore.
### When should I use what?
| Scenario | Use Suspend | Use Snapshot |
| ------------------------------- | ----------- | ------------ |
| Pause and resume later | ✅ | ❌ |
| Save cost when idle | ✅ | ❌ |
| Keep agent memory alive | ✅ | ❌ |
| Retry from a checkpoint | ❌ | ✅ |
| Run experiments from same state | ❌ | ✅ |
| Clone environment | ❌ | ✅ |
## Create a sandbox
Create an ephemeral sandbox by calling create with no name. Add a name to make the sandbox persistent and eligible for suspend/resume.
You can also boot a sandbox from an existing snapshot to restore a previously captured filesystem, memory, and running processes. See [Restoring from a snapshot](/sandboxes/snapshots#restoring-from-a-snapshot) for details.
```bash theme={null}
# Ephemeral: runs until terminated or timed out
tl sbx create
# Named: can be suspended and resumed
tl sbx create my-env
```
```python theme={null}
from tensorlake.sandbox import Sandbox
# Ephemeral sandbox: no name, cannot be suspended
ephemeral = Sandbox.create()
# Named sandbox: can be suspended and resumed
named = Sandbox.create(name="my-env")
print(f"Sandbox ID: {named.sandbox_id}")
print(f"Status: {named.status}")
```
```typescript theme={null}
// Ephemeral sandbox: no name, cannot be suspended
const ephemeral = await Sandbox.create();
// Named sandbox: can be suspended and resumed
const named = await Sandbox.create({ name: "my-env" });
console.log(named.sandboxId, named.status);
```
```bash theme={null}
# Ephemeral
curl -X POST https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{}'
# Named
curl -X POST https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "my-env"}'
```
### Resources
Configure CPU, memory, and disk size per sandbox. These are fixed when the sandbox is created and cannot be changed afterwards. Create a new sandbox if you need different resources.
```bash theme={null}
tl sbx create --cpus 2.0 --memory 2048 --disk_mb 25600
```
```python theme={null}
sandbox = Sandbox.create(
cpus=2.0,
memory_mb=2048,
disk_mb=25600,
)
```
```typescript theme={null}
const sandbox = await Sandbox.create({
cpus: 2.0,
memoryMb: 2048,
diskMb: 25600,
});
```
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"resources": {
"cpus": 2.0,
"memory_mb": 2048,
"disk_mb": 25600
}
}'
```
| Parameter | Type | Default | Description |
| ----------- | ------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cpus` | `float` | `1.0` | Number of CPUs to allocate |
| `memory_mb` | `int` | `1024` | Memory in megabytes. Must be between 1024–8192 MB per CPU core. |
| `disk_mb` | `int` | `10240` | Root filesystem size in MiB. Defaults to 10240 (10 GiB) when omitted. Must be between 10240 and 102400 (10–100 GiB). The CLI accepts `--disk_mb` in MiB. Accepted for fresh creates. With `image`, `disk_mb` can be used to grow the root disk at create time (growth-only). With `snapshot_id` from a filesystem snapshot, `disk_mb` can also be used to grow the root disk at restore time (growth-only). |
### Timeout
`timeout_secs` is an **idle threshold**, not a wall-clock lifetime. A sandbox stays running as long as it is handling traffic through the sandbox proxy: an open SSH session, a connected WebSocket PTY, a request to an exposed user port, or any SDK/CLI call. Once no proxied traffic has been in flight for `timeout_secs`, the sandbox times out. What happens next depends on the sandbox type:
* **Named sandboxes**: suspend on timeout. Filesystem, memory, and running processes are preserved so you can resume later under the same name.
* **Ephemeral sandboxes**: terminate on timeout (final state). The sandbox cannot be resumed.
If `timeout_secs` is not set, it goes with the default value which is `600` sec (10 minutes).
The maximum allowed `timeout_secs` depends on your plan: **1 hour** on Free (unverified), **2 hours** on Free (verified), and **24 hours** on On-Demand (pay-as-you-go). Setting `timeout_secs=0` requests the plan maximum. See [tensorlake.ai/pricing](https://www.tensorlake.ai/pricing) for higher limits on committed plans.
```bash theme={null}
tl sbx create --timeout 300
```
```python theme={null}
sandbox = Sandbox.create(timeout_secs=300)
```
```typescript theme={null}
const sandbox = await Sandbox.create({ timeoutSecs: 300 });
```
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"timeout_secs": 300}'
```
### Runtime environment
Sandboxes run on Tensorlake's managed Ubuntu 24.04 environment by default. If you need reusable setup or preinstalled dependencies, create a [sandbox image](/sandboxes/images) and launch sandboxes with `--image`. For one-off startup setup, create the sandbox and then use [command execution](/sandboxes/commands) to run those steps explicitly.
## Name and reference a sandbox
You can assign or update a sandbox's name after it is created. This is how you convert an ephemeral sandbox into a named one so it becomes eligible for suspend and resume.
```bash theme={null}
# Assign a name to a running sandbox
tl sbx name my-env
# Change an existing name
tl sbx name my-env new-name
```
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create()
named_sbx = sandbox.update(name="my-env")
print(named_sbx.name)
```
```typescript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.create();
const named_sbx = await sandbox.update({ name: "my-env" });
console.log(named_sbx.name);
```
```bash theme={null}
curl -X PATCH https://api.tensorlake.ai/sandboxes/ \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "my-env"}'
```
Once a sandbox has a name, you can use either the name or the UUID anywhere a sandbox identifier is accepted. Use `connect` to get an operable handle from either identifier.
```bash theme={null}
# All of these work with either the ID or the name
tl sbx exec my-env python main.py
tl sbx ssh my-env
tl sbx cp ./file.py my-env:/workspace/file.py
tl sbx suspend my-env
tl sbx resume my-env
tl sbx terminate my-env
tl sbx checkpoint my-env
tl sbx name my-env new-name
```
```python theme={null}
info = Sandbox.connect("my-env")
print(info.status)
sandbox = Sandbox.connect(identifier="my-env")
print(sandbox.sandbox_id) # server UUID, e.g. "s7jus08qec4axzgbpq76h"
print(sandbox.name) # "my-env"
result = sandbox.run("python", ["main.py"])
print(result.stdout)
renamed = sandbox.update(name= "new-name")
print(renamed.name)
sandbox.terminate()
```
```typescript theme={null}
const info = await Sandbox.connect("my-env");
console.log(info.status);
const sandbox = Sandbox.connect("my-env");
console.log(sandbox.sandboxId); // server UUID
console.log(sandbox.name); // "my-env"
const result = await sandbox.run("python", { args: ["main.py"] });
console.log(result.stdout);
await sandbox.update({ name: "new-name" });
await sandbox.terminate();
```
```bash theme={null}
curl https://api.tensorlake.ai/sandboxes/my-env \
-H "Authorization: Bearer $TL_API_KEY"
```
Authenticated requests can use either the sandbox ID or sandbox name. Unauthenticated proxy requests can also use sandbox names for exposed user ports when `allow_unauthenticated_access` is enabled. The management URL on port `9501` still requires authentication.
## Inspect and list
Use `get` to check a single sandbox's status and configuration, or `list` to see all sandboxes in your namespace.
```bash theme={null}
# List active sandboxes
tl sbx ls
# Running sandboxes only
tl sbx ls --running
# Include all sandboxes regardless of state
tl sbx ls --all
```
```python theme={null}
from tensorlake.sandbox import Sandbox
# Connect returns a Sandbox handle (not SandboxInfo)
sandbox = Sandbox.connect("my-env")
print(sandbox.status) # property: fetches fresh from server
print(sandbox.name)
print(sandbox.sandbox_id)
# Get the full metadata (image, resources, timeouts, etc.)
info = sandbox.info()
print(info.image)
print(f"{info.resources.cpus} CPUs, {info.resources.memory_mb} MB RAM")
# List all sandboxes in the namespace
for sb in Sandbox.list():
print(f"{sb.sandbox_id}: {sb.status}")
```
```typescript theme={null}
import { Sandbox } from "tensorlake";
// Connect returns a Sandbox handle (not SandboxInfo)
const sandbox = await Sandbox.connect("my-env");
console.log(await sandbox.status()); // status is an async method in TS
console.log(sandbox.name); // name is a getter
console.log(sandbox.sandboxId);
// Get the full metadata (image, resources, timeouts, etc.)
const info = await sandbox.info();
console.log(info.image, info.resources.cpus, info.resources.memoryMb);
// List all sandboxes in the namespace
const sandboxes = await Sandbox.list();
for (const sb of sandboxes) {
console.log(`${sb.sandboxId}: ${sb.status}`);
}
```
```bash theme={null}
# Get one sandbox (by name or ID)
curl https://api.tensorlake.ai/sandboxes/my-env \
-H "Authorization: Bearer $TL_API_KEY"
# List all sandboxes
curl https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TL_API_KEY"
```
## Suspend and resume
Suspend a running named sandbox to pause it in place, then resume the same sandbox later exactly where it left off. Suspend and resume do not create a reusable artifact. For that, use [Snapshots](/sandboxes/snapshots). Ephemeral sandboxes cannot be suspended, and suspend calls on them return an error.
```bash theme={null}
# Suspend a named sandbox (by name or ID)
tl sbx suspend my-env
# Resume it later
tl sbx resume my-env
```
```python theme={null}
sandbox.suspend()
sandbox.resume()
```
```typescript theme={null}
await sandbox.suspend();
await sandbox.resume();
```
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandboxes/my-env/suspend \
-H "Authorization: Bearer $TL_API_KEY"
curl -X POST https://api.tensorlake.ai/sandboxes/my-env/resume \
-H "Authorization: Bearer $TL_API_KEY"
```
## Terminate
Terminate a sandbox when the work is done. Sandboxes with `timeout_secs` set also terminate automatically once the timeout elapses. A terminated sandbox can be brought back with [restart](#restart-a-terminated-sandbox) for up to 48 hours; after that it is gone.
```bash theme={null}
tl sbx terminate my-env
```
```python theme={null}
sandbox.terminate()
```
```typescript theme={null}
await sandbox.terminate();
```
```bash theme={null}
curl -X DELETE https://api.tensorlake.ai/sandboxes/my-env \
-H "Authorization: Bearer $TL_API_KEY"
```
## Restart a terminated sandbox
Restart boots a terminated sandbox again under its original ID and name. Use it to recover a sandbox that was terminated by mistake, or one that ended up terminated because it could not be resumed. Terminated sandboxes stay restartable for 48 hours after termination.
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandboxes/my-env/restart \
-H "Authorization: Bearer $TL_API_KEY"
```
* Only terminated sandboxes can be restarted. To wake a suspended sandbox, use [resume](#suspend-and-resume) instead.
* When the sandbox has a usable snapshot, the restart restores from the most recent one, so filesystem and memory state come back. When no snapshot exists, the sandbox cold boots from its image with a fresh filesystem.
* The restarted sandbox re-enters `Pending` and boots like a newly created sandbox.
* If another sandbox has claimed the same name since termination, the restart fails with `409 Conflict`.
See the [Restart Sandbox API reference](/api-reference/v2/sandboxes/restart) for full details.
## End-to-end example
If you want a single example that creates a sandbox, inspects it, lists sandboxes, and cleans up when finished, use one of the sessions below.
```bash theme={null}
# Create an ephemeral sandbox (no name, cannot be suspended)
tl sbx create --cpus 1.0 --memory 1024 --timeout 300
# Create a named sandbox (can be suspended and resumed)
tl sbx create my-env --cpus 1.0 --memory 1024 --timeout 300
# Check status or list sandboxes
tl sbx ls
tl sbx ls --all
# Terminate the sandbox when you are done (by name or ID)
tl sbx terminate my-env
```
```python theme={null}
from tensorlake.sandbox import Sandbox
# Ephemeral sandbox: no name, cannot be suspended
ephemeral = Sandbox.create(
cpus=1.0,
memory_mb=1024,
disk_mb=10240,
timeout_secs=300,
)
# Named sandbox: can be suspended and resumed
named = Sandbox.create(
name="my-env",
cpus=1.0,
memory_mb=1024,
disk_mb=10240,
timeout_secs=300,
)
print(f"Sandbox ID: {named.sandbox_id}")
print(f"Status: {named.status}")
# Get the full metadata (image, resources, timeouts, etc.)
info = sandbox.info()
print(f"Image: {info.image}")
print(f"Resources: {info.resources.cpus} CPUs, {info.resources.memory_mb} MB RAM")
sandboxes = Sandbox.list()
for sb in sandboxes:
print(f"{sb.sandbox_id}: {sb.status}")
sandbox.terminate()
print("Sandboxes terminated")
```
```typescript theme={null}
import { Sandbox } from "tensorlake";
const ephemeral = await Sandbox.create({
cpus: 1.0,
memoryMb: 1024,
diskMb: 10240,
timeoutSecs: 300,
});
const named = await Sandbox.create({
name: "my-env",
cpus: 1.0,
memoryMb: 1024,
diskMb: 10240,
timeoutSecs: 300,
});
console.log(named.sandboxId, named.status);
# Get the full metadata (image, resources, timeouts, etc.)
const info = await sandbox.info();
console.log(info.image, info.resources.cpus, info.resources.memoryMb);
const sandboxes = await Sandbox.list();
for (const sandbox of sandboxes) {
console.log(`${sandbox.sandboxId}: ${sandbox.status}`);
}
await sandbox.terminate();
```
```bash theme={null}
# Create an ephemeral sandbox
curl -X POST https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"resources": {"cpus": 1.0, "memory_mb": 1024},
"timeout_secs": 300
}'
# Create a named sandbox (supports suspend/resume)
curl -X POST https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "my-env",
"resources": {"cpus": 1.0, "memory_mb": 1024},
"timeout_secs": 300
}'
# Get one sandbox
curl https://api.tensorlake.ai/sandboxes/ \
-H "Authorization: Bearer $TL_API_KEY"
# List all sandboxes
curl https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TL_API_KEY"
# Delete
curl -X DELETE https://api.tensorlake.ai/sandboxes/ \
-H "Authorization: Bearer $TL_API_KEY"
```
## Sandbox object reference
### Sandbox
The `Sandbox` object returned by `Sandbox.create()` and `Sandbox.connect()` exposes the following properties. Both are resolved from the server on first access and cached for the lifetime of the object.
| Property (Python) | Property (TypeScript) | Type | Description |
| ----------------- | --------------------- | ------------- | -------------------------------------------------------------------------------------- |
| `sandbox_id` | `sandboxId` | `str` | Server-assigned UUID. Always a UUID, never a name, even if you connected using a name. |
| `name` | `name` | `str \| None` | Human-readable name, or `None` for ephemeral sandboxes. |
```python theme={null}
sandbox = Sandbox.connect(identifier="my-env")
print(sandbox.sandbox_id) # "s7jus08qec4axzgbpq76h" ← UUID
print(sandbox.name) # "my-env" ← name
```
### SandboxInfo
The `SandboxInfo` object returned by `Sandbox.info()` and `Sandbox.list()` contains:
| Field | Type | Description |
| --------------- | ------------------------ | ------------------------------------------------------ |
| `sandbox_id` | `str` | Unique sandbox identifier |
| `name` | `str \| None` | Name of the sandbox, or `None` for ephemeral sandboxes |
| `namespace` | `str` | Namespace the sandbox belongs to |
| `status` | `str` | Current lifecycle state |
| `image` | `str` | Container image used |
| `resources` | `ContainerResourcesInfo` | CPU and memory allocation |
| `timeout_secs` | `int` | Timeout in seconds |
| `entrypoint` | `list[str]` | Custom entrypoint command |
| `created_at` | `datetime \| None` | Creation timestamp |
| `terminated_at` | `datetime \| None` | Termination timestamp |
## Learn more
Save and restore sandbox filesystem, memory, and running processes.
Control internet access and blocked destinations.
# Networking
Source: https://docs.tensorlake.ai/sandboxes/networking
Route internet traffic into sandbox applications and control outbound internet access
Sandboxes support two networking features:
1. Routing internet traffic into services running inside a sandbox through `*.sandbox.tensorlake.ai`
2. Restricting the sandbox's own outbound internet access
## Sandbox Public URL
Every running sandbox is reachable through sandbox-specific ingress.
* `https://.sandbox.tensorlake.ai` routes to the sandbox management API on port `9501`
* `https://-.sandbox.tensorlake.ai` routes to a user service listening on `` inside the sandbox
The proxy preserves the request path and query string, supports WebSocket upgrades, and forwards gRPC over HTTP/2.
The hostname can use either the sandbox ID or a sandbox name. The proxy resolves names to the sandbox's canonical ID before forwarding the request.
These examples use the familiar `*.sandbox.tensorlake.ai` hostname pattern. The returned `sandbox_url` is the management URL on port `9501`.
## Route Traffic Into Sandbox Apps
There are two access modes for internet-facing sandbox traffic:
1. `Authenticated requests`: the caller sends TensorLake auth credentials, and the proxy authorizes the request before forwarding it.
2. `Unauthenticated requests`: the sandbox owner explicitly makes selected user ports public, and the proxy skips auth for those user ports.
### Expose a User Port
Port `9501` is the built-in management API and is always routable through the bare sandbox hostname.
For any other port, the proxy only forwards requests if that port is listed in `exposed_ports`.
`allow_unauthenticated_access` does not expose a port by itself. User ports still have to be present in `exposed_ports`.
#### Authenticated-Only Exposure with the HTTP API
Use this when a port should be routable from the internet but still require TensorLake auth on every request.
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = client.expose_ports(
"my-env",
[8080],
allow_unauthenticated_access=False,
)
print(sandbox.exposed_ports)
sandbox = client.unexpose_ports("my-env", [8080])
print(sandbox.exposed_ports)
```
```typescript theme={null}
const sandbox = await client.exposePorts(
"my-env",
[8080],
{ allowUnauthenticatedAccess: false },
);
console.log(sandbox.exposedPorts);
const updated = await client.unexposePorts("my-env", [8080]);
console.log(updated.exposedPorts);
```
```bash theme={null}
curl -X PATCH https://api.tensorlake.ai/sandboxes/ \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"allow_unauthenticated_access": false,
"exposed_ports": [8080]
}'
curl -X PATCH https://api.tensorlake.ai/sandboxes/ \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"allow_unauthenticated_access": false,
"exposed_ports": []
}'
```
#### Unauthenticated Public Internet Access with the CLI
Use this when you want anyone on the internet to be able to reach a sandbox app without TensorLake credentials. Common cases include webhook receivers, demo apps, public APIs, browser clients, and temporary preview environments.
```bash theme={null}
tl sbx port expose 8080
tl sbx port ls
tl sbx port rm 8080
```
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = client.expose_ports(
"my-public-sandbox",
[8080],
allow_unauthenticated_access=True,
)
print(sandbox.allow_unauthenticated_access, sandbox.exposed_ports)
sandbox = client.unexpose_ports("my-public-sandbox", [8080])
print(sandbox.allow_unauthenticated_access, sandbox.exposed_ports)
```
```typescript theme={null}
const sandbox = await client.exposePorts(
"my-public-sandbox",
[8080],
{ allowUnauthenticatedAccess: true },
);
console.log(sandbox.allowUnauthenticatedAccess, sandbox.exposedPorts);
const updated = await client.unexposePorts("my-public-sandbox", [8080]);
console.log(updated.allowUnauthenticatedAccess, updated.exposedPorts);
```
```bash theme={null}
curl -X PATCH https://api.tensorlake.ai/sandboxes/ \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"allow_unauthenticated_access": true,
"exposed_ports": [8080]
}'
curl -X PATCH https://api.tensorlake.ai/sandboxes/ \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"allow_unauthenticated_access": false,
"exposed_ports": []
}'
```
The CLI `port expose` workflow sets both:
* `exposed_ports`
* `allow_unauthenticated_access=true`
So traffic to that user port becomes publicly reachable from the internet without TensorLake auth.
### Authenticated Requests
Authenticated routing is the default model for sandbox access.
* The management URL on port `9501` always requires auth
* User ports can also require auth when they are exposed but `allow_unauthenticated_access=false`
Verified against `sandbox-proxy`, the proxy accepts these auth modes:
* API key: `Authorization: Bearer `
* Personal access token: `Authorization: Bearer tl_pat...` plus `X-Forwarded-Organization-Id` and `X-Forwarded-Project-Id`
* Session cookie: `tl.session_token` or legacy `tl-session`, plus the same forwarded organization/project context
For browser WebSocket clients that cannot set custom `X-Forwarded-*` headers, the proxy also accepts `organizationId` and `projectId` in the query string.
```bash theme={null}
curl https://8080-.sandbox.tensorlake.ai/health \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
```bash theme={null}
curl https://8080-.sandbox.tensorlake.ai/health \
-H "Authorization: Bearer $TENSORLAKE_PAT" \
-H "X-Forwarded-Organization-Id: $TENSORLAKE_ORGANIZATION_ID" \
-H "X-Forwarded-Project-Id: $TENSORLAKE_PROJECT_ID"
```
```bash theme={null}
curl https://8080-.sandbox.tensorlake.ai/health \
-H "Cookie: tl.session_token=$TENSORLAKE_SESSION_TOKEN" \
-H "X-Forwarded-Organization-Id: $TENSORLAKE_ORGANIZATION_ID" \
-H "X-Forwarded-Project-Id: $TENSORLAKE_PROJECT_ID"
```
You can use the same authenticated routing model for HTTP, gRPC, and WebSocket services:
```bash theme={null}
# HTTP
curl https://8080-.sandbox.tensorlake.ai/health \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
# gRPC
grpcurl \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
50051-.sandbox.tensorlake.ai:443 \
list
# WebSocket
wscat \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-c "wss://3000-.sandbox.tensorlake.ai/socket"
```
```typescript theme={null}
const response = await fetch(
"https://8080-my-env.sandbox.tensorlake.ai/health",
{
headers: {
Authorization: `Bearer ${process.env.TENSORLAKE_API_KEY}`,
},
},
);
console.log(await response.text());
```
### Unauthenticated Requests
To make a user port public on the internet, both of these conditions must be true:
* the port is in `exposed_ports`
* `allow_unauthenticated_access=true`
When those are set, the proxy skips TensorLake auth for that user port.
```bash theme={null}
curl -X PATCH https://api.tensorlake.ai/sandboxes/ \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"allow_unauthenticated_access": true,
"exposed_ports": [8080]
}'
```
After that, requests to the exposed user port can omit auth entirely:
```bash theme={null}
curl https://8080-.sandbox.tensorlake.ai/health
```
```typescript theme={null}
const response = await fetch(
"https://8080-my-public-sandbox.sandbox.tensorlake.ai/health",
);
console.log(await response.text());
```
Unauthenticated access only applies to user ports. The management API on port `9501` never becomes public.
If a named sandbox is suspended, the proxy can auto-resume it when a request arrives for an exposed port.
## Outbound Internet Access
By default, sandboxes have outbound internet access enabled. Disable it for untrusted code:
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create(
allow_internet_access=False
)
```
```typescript theme={null}
const sandbox = await Sandbox.create({
allowInternetAccess: false,
});
console.log(sandbox.sandboxId);
```
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"network": {"allow_internet_access": false}
}'
```
```bash theme={null}
tl sbx create --no-internet
```
In a verified public-cloud test, a sandbox created with `allow_internet_access=False` failed DNS resolution for `https://example.com`, confirming that outbound internet access was disabled.
Setting `allow_internet_access=false` blocks all outbound traffic, including DNS requests. In the CLI, `--no-internet` (or `-N`) cannot be combined with `--network-allow` or `--network-deny`.
## Allow Specific Destinations
Use `allow_out` when you want a sandbox to reach only selected destinations.
* values can be domains, IPv4 addresses, or IPv4 CIDR ranges
* `deny_out` takes precedence: a destination matched by both `allow_out` and `deny_out` is blocked
* hostname rules are followed across DNS changes, so a CDN-backed domain keeps working as its IP addresses rotate
```python theme={null}
sandbox = Sandbox.create(
allow_internet_access=True,
allow_out=["example.com", "203.0.113.10", "10.0.0.0/8"],
)
```
```typescript theme={null}
const sandbox = await Sandbox.create({
allowInternetAccess: true,
allowOut: ["example.com", "203.0.113.10", "10.0.0.0/8"],
});
console.log(sandbox.sandboxId);
```
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"network": {
"allow_internet_access": true,
"allow_out": ["example.com", "203.0.113.10", "10.0.0.0/8"]
}
}'
```
```bash theme={null}
tl sbx create \
--network-allow example.com \
--network-allow 203.0.113.10 \
--network-allow 10.0.0.0/8
```
This allows DNS requests to the sandbox's configured resolvers and traffic to the listed domain, IPv4 address, and IPv4 CIDR range. All other outbound traffic is blocked. The short form of `--network-allow` is `-A`.
## Block Specific Destinations
```python theme={null}
sandbox = Sandbox.create(
deny_out=["example.com"]
)
```
```typescript theme={null}
const sandbox = await Sandbox.create({
denyOut: ["example.com"],
});
console.log(sandbox.sandboxId);
```
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"network": {"deny_out": ["example.com"]}
}'
```
```bash theme={null}
tl sbx create --network-deny example.com
```
In a verified public-cloud request, `deny_out=["example.com"]` blocked `https://example.com` while `https://api.openai.com/v1/models` still returned `401`, confirming outbound connectivity was still available for destinations that were not denied.
The short form of `--network-deny` is `-D`. You can combine `-A` and `-D`; deny rules take precedence when a destination matches both lists.
## Update the Policy on a Running Sandbox
You can change a sandbox's egress policy without recreating or suspending it. The new policy is applied to the running sandbox's firewall as a single atomic swap, so there is no window where egress is unprotected. Already-established connections are not interrupted.
The `network` argument is tri-state:
* **omit it** to leave the current policy unchanged (you can update `name` or exposed ports without touching the network policy),
* **pass a policy** to replace the whole policy, or
* **clear it** to return the sandbox to unrestricted egress.
```python theme={null}
from tensorlake.sandbox import CLEAR_NETWORK_POLICY, NetworkConfig, Sandbox
sandbox = Sandbox.connect("")
# Replace the policy: allow only api.example.com.
sandbox.update(
network=NetworkConfig(
allow_internet_access=True,
allow_out=["api.example.com"],
)
)
# Later, clear the policy (unrestricted egress).
sandbox.update(network=CLEAR_NETWORK_POLICY)
```
```typescript theme={null}
const sandbox = await Sandbox.connect({ sandboxId: "" });
// Replace the policy: allow only api.example.com.
await sandbox.update({
network: {
allowInternetAccess: true,
allowOut: ["api.example.com"],
denyOut: [],
},
});
// Later, clear the policy (unrestricted egress) by passing null.
await sandbox.update({ network: null });
```
```bash theme={null}
# Replace the policy.
curl -X PATCH https://api.tensorlake.ai/sandboxes/ \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"network": {
"allow_internet_access": true,
"allow_out": ["api.example.com"]
}
}'
# Clear the policy by sending an explicit null.
curl -X PATCH https://api.tensorlake.ai/sandboxes/ \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"network": null}'
```
```bash theme={null}
# Replace the policy: allow only api.example.com and resolver-scoped DNS.
tl sbx update -A api.example.com
# Replace the policy and block all outbound traffic, including DNS.
tl sbx update --no-internet
# Clear the policy and restore unrestricted egress.
tl sbx update --clear-network
```
Each `tl sbx update` command replaces or clears the complete network policy. Repeat `-A` or `-D` to add multiple rules to the replacement policy. `--no-internet` is an absolute block-all mode and cannot be combined with either rule flag; `--clear-network` cannot be combined with any replacement-policy flag.
If a hostname in the new policy cannot be resolved, the update is rejected and the previous policy stays fully enforced — the sandbox keeps running under the policy it already had.
This is useful for phase-based agents: start a sandbox with a broad allowlist while it fetches dependencies, then tighten to a minimal policy (or block all egress) before running untrusted work.
## Network Configuration Summary
| Parameter | Type | Default | Description |
| ------------------------------ | ------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `allow_internet_access` | `bool` | `true` | Allows internet access, including DNS requests. If `false`, all outbound traffic is blocked, including DNS requests. If `allow_out` is non-empty and this is `true`, only the listed destinations and resolver-scoped DNS requests are allowed |
| `allow_out` | `list[str]` | `[]` | Allowed domains, IPv4 addresses, or IPv4 CIDRs. A non-empty list allows the listed destinations and resolver-scoped DNS requests when `allow_internet_access` is `true` |
| `deny_out` | `list[str]` | `[]` | Denied domains, IPv4 addresses, or IPv4 CIDRs. Takes precedence over `allow_out`: a destination matched by both is blocked |
| `exposed_ports` | `list[int] \| null` | `null` | User ports that the sandbox proxy is allowed to route to |
| `allow_unauthenticated_access` | `bool` | `false` | Skip TensorLake auth for exposed user ports. Never applies to port `9501` |
# Run OpenCode in Tensorlake Sandboxes
Source: https://docs.tensorlake.ai/sandboxes/opencode
Route OpenCode's file and shell tools into a Tensorlake sandbox with a single plugin. The model edits, runs, and searches inside an isolated environment instead of on your machine.
[OpenCode](https://opencode.ai) is a terminal coding agent. The [`tensorlake-opencode`](https://www.npmjs.com/package/tensorlake-opencode) plugin redirects the agent's hands (its file and shell tools) into a Tensorlake sandbox, so the model's commands and edits run in an isolated environment you control rather than on your laptop.
## The model: brain local, hands in the sandbox
OpenCode keeps running locally: the TUI, the model loop, and your session all stay on your machine. The plugin only intercepts the **tool calls** and routes them to a sandbox:
| OpenCode tool | Runs in the sandbox as |
| ------------- | ------------------------------------------ |
| `bash` | `sandbox.run('sh', { args: ['-c', cmd] })` |
| `read` | `sandbox.readFile(path)` |
| `write` | `sandbox.writeFile(path, content)` |
| `edit` | read + string replace + write |
| `ls` | `sandbox.listDirectory(path)` |
| `glob` | `find … -name "pattern"` via bash |
| `grep` | `grep -rn …` via bash |
`webfetch` and `websearch` are **not** intercepted. They stay local, since they don't touch your filesystem.
The sandbox itself runs on Tensorlake: fast boot, sub-second resume from a suspended snapshot, and state that persists across OpenCode restarts. See [Sandbox lifecycle](/sandboxes/lifecycle) for the suspend/resume and snapshot model underneath.
## Why route tool calls into a sandbox
* **Isolation.** The agent's `bash` and `write` never touch your host. A bad command, a runaway install, or an `rm` lands in a disposable environment, not your working tree.
* **Reproducible environment.** Every session gets the same image, CPU, and memory regardless of what's on the developer's machine. Pin a custom image with the right toolchain once and every session inherits it.
* **State that survives restarts.** Sandboxes are named and persisted to disk, so a session reconnects to its sandbox across OpenCode restarts; a suspended sandbox resumes with `/tmp/workspace`, installed deps, and warm caches intact.
## Prerequisites
* An [OpenCode](https://opencode.ai) installation.
* A Tensorlake account and API key. Sign up at [cloud.tensorlake.ai](https://cloud.tensorlake.ai).
## Setup
Add the package name to `~/.config/opencode/opencode.json` (create the file if it doesn't exist):
```json theme={null}
{
"$schema": "https://opencode.ai/config.json",
"plugin": [
"tensorlake-opencode"
]
}
```
OpenCode treats bare names as npm packages and installs them into its own cache (`~/.cache/opencode/packages/`). You don't run `npm install` yourself.
Export it in the same shell you launch OpenCode from:
```bash theme={null}
export TENSORLAKE_API_KEY=your_api_key_here
```
If you use a Personal Access Token instead of a project-scoped key, also set `TENSORLAKE_ORGANIZATION_ID` and `TENSORLAKE_PROJECT_ID`.
```bash theme={null}
opencode
```
On startup the plugin loads but **no sandbox is created yet**. Confirm it loaded by tailing its log:
```bash theme={null}
tail -f ~/.local/share/opencode/log/tensorlake.log
```
You should see a single line: `OpenCode started with TensorLake plugin`.
## Lazy sandbox creation
The sandbox is created **lazily, on the first intercepted tool call** in a session, not when you launch OpenCode. If you start OpenCode and nothing appears to happen, that's expected. A session that only uses `webfetch`/`websearch` will never spin one up, because neither is intercepted.
To trigger creation, ask the model to run something that uses a file or shell tool:
```
Run: uname -a
```
On that first `bash` call the plugin provisions the sandbox. You'll see a **"Sandbox created"** toast and new log lines:
```
[INFO] Creating new sandbox for session abc123
[INFO] Sandbox created sandbox-xyz in 2300ms
```
`uname -a` will report **Linux** (the sandbox), confirming the command ran remotely rather than on your Mac.
## Verify it's working
Ask the model to write and read a file back:
```
Write the text "Hello Tensorlake" to /tmp/workspace/test.txt, then read it back.
```
The `write` call routes to `sandbox.writeFile()` and the `read` call to `sandbox.readFile()`, both over the SDK. The agent's working directory inside the sandbox is `/tmp/workspace`.
## Configure the sandbox
The plugin reads a set of environment variables **at sandbox-creation time** to decide what the sandbox looks like: its image, CPUs, memory, and disk. There is no config file for this; you set the variables in the shell, then launch OpenCode from that same shell.
The variables are read **once, when the sandbox is created** (the first intercepted tool call of a session). Set them *before* you run `opencode`. Changing a variable in another terminal, or after the sandbox already exists, has no effect on the running session. Start a new session to pick up new values.
### How it fits together
```bash theme={null}
# 1. Authenticate (always required)
export TENSORLAKE_API_KEY=your_api_key_here
# 2. Size the sandbox VM
export TENSORLAKE_CPUS=4
export TENSORLAKE_MEMORY_MB=8192
export TENSORLAKE_DISK_MB=20480
# 3. Choose the toolchain image (optional, omit for the platform default)
export TENSORLAKE_IMAGE=my-custom-image
# 4. Launch: the next sandbox this session creates uses all of the above
opencode
```
Every value above describes the sandbox the plugin spins up for that OpenCode session, not OpenCode itself, and not your local machine.
### All variables
| Variable | Default | What it controls |
| ---------------------------- | ----------------------- | ----------------------------------------------------------------------------------------------- |
| `TENSORLAKE_API_KEY` | (required) | Authentication: which Tensorlake account/project the sandbox is created in |
| `TENSORLAKE_ORGANIZATION_ID` | (required for PAT keys) | Organization ID, needed only when using a Personal Access Token |
| `TENSORLAKE_PROJECT_ID` | (required for PAT keys) | Project ID, needed only when using a Personal Access Token |
| `TENSORLAKE_IMAGE` | (platform default) | Registered image the sandbox boots from. Bake your runtimes, build tools, and repo deps in here |
| `TENSORLAKE_CPUS` | `2` | vCPUs allocated to the sandbox |
| `TENSORLAKE_MEMORY_MB` | `4096` | RAM allocated to the sandbox, in MB |
| `TENSORLAKE_DISK_MB` | `10240` | Ephemeral disk allocated to the sandbox, in MB |
### Making the settings persistent
`export` only lasts for the current shell. To apply the same sandbox config every time, add the exports to your shell profile (`~/.zshrc` or `~/.bashrc`):
```bash theme={null}
echo 'export TENSORLAKE_API_KEY=your_api_key_here' >> ~/.zshrc
echo 'export TENSORLAKE_IMAGE=my-custom-image' >> ~/.zshrc
echo 'export TENSORLAKE_CPUS=4' >> ~/.zshrc
```
Open a new terminal (or `source ~/.zshrc`) and every `opencode` session inherits them.
### Use a custom image
`TENSORLAKE_IMAGE` is the most impactful setting for real work: it lets every OpenCode session start from an environment that already has your language runtimes, system packages, and project dependencies, so the agent isn't reinstalling them on each session. Register an image, then point the variable at its name:
```bash theme={null}
tl sbx image create Dockerfile --registered-name my-custom-image
export TENSORLAKE_IMAGE=my-custom-image
```
See [Build and Import Images](/sandboxes/images) for building and managing images.
## Next steps
The full plugin: tool interceptors, session manager, and lifecycle handling.
The suspend/resume and snapshot model that persists session state.
Build a custom image so every OpenCode session gets the same toolchain.
The general pattern: expose sandboxes as tools to any LLM agent.
# Sandbox Pools
Source: https://docs.tensorlake.ai/sandboxes/pools
Pre-warm sandboxes with pools for near-instant startup
A Sandbox Pool is a sandbox template (image, resources, entrypoint, timeout, network policy) plus a set of pre-booted warm containers. Creating a sandbox from a pool claims a warm container instead of cold-booting one, so the sandbox is ready almost instantly.
How it works:
* The pool keeps `warm_containers` idle containers booted and waiting.
* Creating a sandbox from the pool claims a warm container if one is available; otherwise a new container cold-starts on demand.
* After a claim, the pool boots a replacement to restore the warm buffer.
* `max_containers` caps the pool's total (warm + claimed) containers. At the cap, new sandboxes stay `pending` until a slot frees up.
Sandboxes claimed from a pool inherit the pool's image, resources, entrypoint, timeout, and network policy. You cannot override them per sandbox.
## Creating a Pool
```python theme={null}
from tensorlake.sandbox import SandboxClient
client = SandboxClient.for_cloud()
pool = client.create_pool(
image="tensorlake/ubuntu-minimal",
cpus=1.0,
memory_mb=1024,
warm_containers=2,
max_containers=10,
)
print(pool.pool_id)
```
```typescript theme={null}
import { SandboxClient } from "tensorlake";
const client = new SandboxClient();
const pool = await client.createPool({
image: "tensorlake/ubuntu-minimal",
cpus: 1.0,
memoryMb: 1024,
warmContainers: 2,
maxContainers: 10,
});
console.log(pool.poolId);
```
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandbox-pools \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"image": "tensorlake/ubuntu-minimal",
"resources": {"cpus": 1.0, "memory_mb": 1024},
"warm_containers": 2,
"max_containers": 10
}'
```
Not supported in the CLI.
### Pool configuration
| Field | Type | Default | Description |
| ----------------- | ----------- | ------------------ | ---------------------------------------------------------------------------------------- |
| `image` | `str` | — | Sandbox image for containers in the pool. Required. |
| `cpus` | `float` | `1.0` | CPU cores per container (max 16) |
| `memory_mb` | `int` | `1024` | Memory per container in MB (max 65536; must be 1000–8192 MB per CPU core) |
| `warm_containers` | `int` | `None` | Number of idle, pre-booted containers to keep ready. Set `0` to scale to zero when idle. |
| `max_containers` | `int` | `None` (unbounded) | Cap on total containers, warm and claimed |
| `timeout_secs` | `int` | `0` (no timeout) | Timeout applied to sandboxes created from the pool |
| `entrypoint` | `list[str]` | `None` | Entrypoint command for pool containers |
The HTTP API additionally accepts `exposed_ports`, `network` (egress policy), and `allow_unauthenticated_access`, with the same semantics as [sandbox creation](/sandboxes/lifecycle). Root disk size is server-managed for pool containers; `ephemeral_disk_mb` is accepted but ignored.
## Creating a Sandbox from a Pool
Pass `pool_id` to `Sandbox.create()`. This claims a warm container when one is available.
```python theme={null}
from tensorlake.sandbox import Sandbox
with Sandbox.create(pool_id=pool.pool_id) as sandbox:
result = sandbox.run("echo", ["hello from the pool"])
print(result.stdout)
# Sandbox terminates on exit; the pool boots a replacement warm container.
```
```typescript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.create({ poolId: pool.poolId });
const result = await sandbox.run("echo", {
args: ["hello from the pool"],
});
console.log(result.stdout);
await sandbox.terminate();
```
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandbox-pools//sandboxes \
-H "Authorization: Bearer $TL_API_KEY"
```
Returns `sandbox_id` with status `pending`; poll the sandbox until it is `running`.
Not supported in the CLI.
When claiming from a pool, the sandbox `name` and `file_systems` parameters are ignored because the container is already booted from the pool's template.
## Managing Pools
### Get a Pool
`get_pool` returns the pool configuration plus its current containers. Containers with no `sandbox_id` are warm and unclaimed.
```python theme={null}
info = client.get_pool(pool.pool_id)
print(info.image, info.warm_containers, info.max_containers)
for c in info.containers or []:
print(c.id, c.state, c.sandbox_id or "warm")
```
```typescript theme={null}
const info = await client.getPool(pool.poolId);
console.log(info.image, info.warmContainers, info.maxContainers);
for (const c of info.containers ?? []) {
console.log(c.id, c.state, c.sandboxId ?? "warm");
}
```
```bash theme={null}
curl https://api.tensorlake.ai/sandbox-pools/ \
-H "Authorization: Bearer $TL_API_KEY"
```
### List Pools
```python theme={null}
for p in client.list_pools():
print(p.pool_id, p.image)
```
```typescript theme={null}
const pools = await client.listPools();
for (const p of pools) {
console.log(p.poolId, p.image);
}
```
```bash theme={null}
curl https://api.tensorlake.ai/sandbox-pools \
-H "Authorization: Bearer $TL_API_KEY"
```
### Update a Pool
`update_pool` replaces the pool configuration. `image` is required on update. The warm buffer reconciles to the new settings; already-claimed sandboxes are unaffected.
```python theme={null}
client.update_pool(
pool_id=pool.pool_id,
image="tensorlake/ubuntu-minimal",
cpus=2.0,
memory_mb=4096,
warm_containers=5,
)
```
```typescript theme={null}
await client.updatePool(pool.poolId, {
image: "tensorlake/ubuntu-minimal",
cpus: 2.0,
memoryMb: 4096,
warmContainers: 5,
});
```
```bash theme={null}
curl -X PUT https://api.tensorlake.ai/sandbox-pools/ \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"image": "tensorlake/ubuntu-minimal",
"resources": {"cpus": 2.0, "memory_mb": 4096},
"warm_containers": 5
}'
```
### Delete a Pool
Deleting a pool tears down its warm containers. If sandboxes claimed from the pool are still active, the delete fails with `PoolInUseError` (HTTP `409`). Terminate the sandboxes first, or pass `force=true` over HTTP to terminate them along with the pool.
```python theme={null}
from tensorlake.sandbox import PoolInUseError
try:
client.delete_pool(pool.pool_id)
except PoolInUseError:
# Terminate active sandboxes first, then retry.
raise
```
```typescript theme={null}
await client.deletePool(pool.poolId);
// Throws PoolInUseError if sandboxes from the pool are still active.
```
```bash theme={null}
curl -X DELETE https://api.tensorlake.ai/sandbox-pools/ \
-H "Authorization: Bearer $TL_API_KEY"
# Force-delete: also terminates active sandboxes from the pool
curl -X DELETE "https://api.tensorlake.ai/sandbox-pools/?force=true" \
-H "Authorization: Bearer $TL_API_KEY"
```
## Async
`AsyncSandboxClient` exposes the same pool methods (`create_pool`, `get_pool`, `list_pools`, `update_pool`, `delete_pool`, `claim`) as coroutines, and `AsyncSandbox.create(pool_id=...)` claims from a pool. See [Async](/sandboxes/async).
## Related Guides
Create, suspend, resume, and terminate sandboxes.
Build the custom images your pools boot from.
Capture and restore sandbox state. Snapshot restores cold-boot and don't claim warm containers.
Egress policies you can set on a pool's template.
# Sandbox Process Logs
Source: https://docs.tensorlake.ai/sandboxes/process-logs
View, search, and filter stdout and stderr from sandbox processes in the Tensorlake Console
Sandbox process logs give you a searchable history of the output produced by processes in a sandbox. Use them when you need to debug a long-running agent, inspect a failed setup command, compare logs across process restarts, or share what happened in a sandbox with your team.
Process logs are different from the per-process output APIs in [Commands & Processes](/sandboxes/commands):
* **Process logs** are retained telemetry for the sandbox. They are best for browsing, filtering, searching, and debugging after the fact.
* **Process output** is the buffered stdout, stderr, or combined stream for one tracked process. It is best when your application needs to programmatically wait for or stream a specific process.
## Open Process Logs
1. Open the Tensorlake Console.
2. Go to your project.
3. Open **Sandboxes**.
4. Select the sandbox you want to inspect.
5. Open the **Logs** tab.
The Logs tab shows output from sandbox processes as rows with timestamps, log levels, process context, and the log message. New logs appear automatically while live refresh is enabled.
Logs appear after a process writes to stdout or stderr and the log pipeline has ingested the output. Very recent output can take a short moment to show up in the Console.
## Filter Logs
Use the left sidebar or the search bar to narrow the log stream.
Common filters:
| Filter | How to use it |
| ------------------- | --------------------------------------------------------------------------------------------------------- |
| Log level | Select levels in the sidebar, or search with `level:error`, `level:warn`, `level:info`, or `level:debug`. |
| Process | Select a process in the sidebar, or search with `processId:`. |
| Text in the message | Search with quoted text, such as `"connection refused"` or `"pip install"`. |
The process filter uses the stable `processId` shown by the sandbox process log metadata, not necessarily the operating-system PID. This matters for managed or restarted processes, where a logical process can restart with a new PID while keeping a stable process identifier for log filtering.
## Structured JSON Logs
You can write structured logs by printing one JSON object per line to stdout or stderr. Tensorlake parses JSON log lines, extracts a display message, and stores the remaining JSON fields as structured log attributes.
Use this pattern when you want logs that are still readable in the Console, but also carry machine-readable context such as job IDs, tool names, retry counts, model names, durations, or application-specific status.
```python theme={null}
import json
print(json.dumps({
"timestamp": "2026-06-29T12:00:00Z",
"level": "info",
"message": "started document extraction",
"document_id": "doc_123",
"stage": "extract",
"attempt": 1,
"duration_ms": 42,
"metadata": {
"source": "upload",
"mime_type": "application/pdf",
},
}))
```
For the log above:
* `message` becomes the log message shown in the Console and returned as `body`.
* `level` sets the log severity when it is one of `trace`, `debug`, `info`, `warn`, or `error`.
* `timestamp` sets the log timestamp when it is parseable.
* Other JSON keys are preserved in `logAttributes` on the returned log record.
* Nested objects and arrays are preserved as structured attributes. `null` values are omitted.
If there is no `message`, Tensorlake uses `event` as the display message when it is present. Event names with underscores are shown with spaces, so `"event": "tool_call_started"` appears as `tool call started`.
```python theme={null}
print(json.dumps({
"level": "warn",
"event": "tool_retry",
"tool": "browser",
"attempt": 2,
"reason": "timeout",
}))
```
Malformed JSON is treated as plain text. To make sure a line is parsed as structured data, emit exactly one complete JSON object per line.
You can inspect structured attributes from the SDK responses:
```python theme={null}
logs = sandbox.get_logs(body="document extraction", tail=50).value
for log in logs.logs:
print(log.body)
print(log.log_attributes)
```
```typescript theme={null}
const logs = await sandbox.getLogs({
body: "document extraction",
tail: 50,
});
for (const log of logs.logs) {
console.log(log.body);
console.log(JSON.parse(log.logAttributes));
}
```
Today, sandbox log search filters by log level, stable process ID, and text in the extracted `body`. Structured attributes are returned with each log record for inspection and downstream processing, but they are not yet separate sandbox log query filters.
## CLI
Use `tl sbx logs` to read retained process logs from a sandbox.
```bash theme={null}
# Print the newest 100 retained log lines
tl sbx logs --tail 100
# Filter by severity
tl sbx logs --level error --tail 100
# Search the log message body
tl sbx logs --body "connection refused" --tail 100
```
By default, the CLI prints only each log message body. Add `--json` when you need timestamps, levels, resource attributes, pagination tokens, or other metadata:
```bash theme={null}
tl sbx logs --level warn --tail 25 --json
```
To find stable process IDs for filtering, list the available log streams:
```bash theme={null}
tl sbx logs streams
```
Then pass one or more process IDs back to `tl sbx logs`:
```bash theme={null}
tl sbx logs --process-id --tail 100
```
If the JSON response includes `nextToken`, pass it to read the next page:
```bash theme={null}
tl sbx logs --tail 100 --next-token ""
```
## Python SDK
Use `Sandbox.get_logs()` when you already have a sandbox object.
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create()
sandbox.run("python", ["-c", "import sys; print('ready'); print('warning', file=sys.stderr)"])
logs = sandbox.get_logs(tail=100, levels=["info", "warn"]).value
for log in logs.logs:
print(log.body)
```
Filter by retained process stream:
```python theme={null}
processes = sandbox.list_log_processes().value.processes
for process in processes:
print(process.process_id, process.process_pid, process.process_command)
if processes:
logs = sandbox.get_logs(
process_ids=[processes[0].process_id],
tail=100,
).value
```
Search message text and paginate:
```python theme={null}
page = sandbox.get_logs(body="connection refused", tail=100).value
if page.next_token:
next_page = sandbox.get_logs(
body="connection refused",
tail=100,
next_token=page.next_token,
).value
```
If you are using `SandboxClient` directly, pass the sandbox ID:
```python theme={null}
from tensorlake.sandbox import SandboxClient
client = SandboxClient.for_cloud()
logs = client.get_logs("", levels=["error"], tail=50).value
```
## TypeScript SDK
Use `sandbox.getLogs()` when you already have a sandbox object.
```typescript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.create();
await sandbox.run("python", {
args: ["-c", "import sys; print('ready'); print('warning', file=sys.stderr)"],
});
const logs = await sandbox.getLogs({
levels: ["info", "warn"],
tail: 100,
});
for (const log of logs.logs) {
console.log(log.body);
}
```
Filter by retained process stream:
```typescript theme={null}
const processes = await sandbox.listLogProcesses();
for (const process of processes.processes) {
console.log(process.processId, process.processPid, process.processCommand);
}
if (processes.processes.length > 0) {
const processLogs = await sandbox.getLogs({
processIds: [processes.processes[0].processId],
tail: 100,
});
}
```
Search message text and paginate:
```typescript theme={null}
const page = await sandbox.getLogs({
body: "connection refused",
tail: 100,
});
if (page.nextToken) {
const nextPage = await sandbox.getLogs({
body: "connection refused",
tail: 100,
nextToken: page.nextToken,
});
}
```
## Debug a Failed Process
A typical debugging flow is:
1. Open the sandbox and go to **Processes** to find the command, status, PID, and exit code.
2. Go to **Logs**.
3. Select the matching process in the sidebar.
4. Add a text search such as `"Traceback"`, `"error"`, or the package, file, or command you were investigating.
5. Open a log row to inspect the full message and resource attributes.
For one-off commands, you can also use the SDK or process API to read stdout and stderr directly:
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create()
result = sandbox.run("python", ["-c", "import sys; print('hello'); print('oops', file=sys.stderr)"])
print(result.stdout)
print(result.stderr)
print(result.exit_code)
```
Use the Console Logs tab when you want to keep searching after the command has finished, filter across many processes, or inspect output from a process that was started outside your current SDK call.
## Query Logs with HTTP
The Console uses the sandbox logs endpoint behind the scenes. You can query the same retained logs over HTTP:
```bash theme={null}
curl "https://api.tensorlake.ai/v1/namespaces/$PROJECT_ID/sandboxes/$SANDBOX_ID/logs?tail=100" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
Filter by level:
```bash theme={null}
curl "https://api.tensorlake.ai/v1/namespaces/$PROJECT_ID/sandboxes/$SANDBOX_ID/logs?level=5&tail=100" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
Filter by process:
```bash theme={null}
curl "https://api.tensorlake.ai/v1/namespaces/$PROJECT_ID/sandboxes/$SANDBOX_ID/logs?processId=$PROCESS_ID&tail=100" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
Search message text:
```bash theme={null}
curl "https://api.tensorlake.ai/v1/namespaces/$PROJECT_ID/sandboxes/$SANDBOX_ID/logs?body=connection%20refused&tail=100" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
Log levels are numeric in the HTTP API: `1` trace, `2` debug, `3` info, `4` warn, `5` error, and `6` fatal.
## List Process Filter Values
To build your own process picker, list the processes that have retained logs for a sandbox:
```bash theme={null}
curl "https://api.tensorlake.ai/v1/namespaces/$PROJECT_ID/sandboxes/$SANDBOX_ID/processes" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
The response includes stable process IDs, the last observed PID, command, managed process ID or name when available, first and last observed log times, and the retained log count.
```json theme={null}
{
"processes": [
{
"processId": "proc-a",
"processPid": "101",
"processCommand": "/usr/bin/python worker.py",
"processManagedId": "managed-a",
"processManagedName": "worker-a",
"firstSeen": 1773950042728,
"lastSeen": 1773950049123,
"logCount": 42
}
]
}
```
Use the returned `processId` with the `processId` logs query parameter.
## When Logs Are Empty
If the Logs tab is empty:
* Confirm the sandbox has started a process that writes to stdout or stderr.
* Check that you are viewing the correct project and sandbox ID.
* Wait a short moment for ingestion if the process just wrote output.
* Use the **Processes** tab or the process output API if you need immediate output from a currently running process.
Logs are retained according to the log retention policy for your environment.
# SSH and PTY Sessions
Source: https://docs.tensorlake.ai/sandboxes/pty-sessions
Reach a running sandbox over standard SSH, or open a programmatic PTY session over WebSocket
There are two ways to drive an interactive shell inside a sandbox:
* **[SSH](#ssh)**: connect with `ssh`, `scp`, `sftp`, `rsync`, VS Code Remote-SSH, JetBrains Gateway, and any other tool that speaks SSH. Use this when you want a normal terminal, file transfer, or port forwarding.
* **[PTY sessions](#pty-sessions)**: create a PTY over HTTPS, attach to it over a WebSocket, and drive terminal I/O programmatically. Use this when you're building a UI or browser app that needs a shell, when you need WebSocket-only access, or when you want a session you can disconnect and reattach by token.
## SSH
The Tensorlake sandbox proxy exposes a standard SSH endpoint at `sandbox.tensorlake.ai`. Use your sandbox id as the SSH username. Your laptop's SSH key, registered once with your Tensorlake account, authenticates the connection.
### One-time setup
```bash theme={null}
tl login # if you aren't already logged in
tl sbx ssh keys add --name laptop ~/.ssh/id_ed25519.pub
tl sbx ssh keys ls
```
`tl sbx ssh keys` requires user-level auth and is not supported with API-key auth. If you have `TENSORLAKE_API_KEY` exported (which takes precedence over `tl login`), unset it for the registration step (e.g. `env -u TENSORLAKE_API_KEY tl sbx ssh keys add --name laptop ~/.ssh/id_ed25519.pub`) or open a fresh shell without it. Once your key is registered you can put `TENSORLAKE_API_KEY` back; SSH itself uses the registered key, not the API key.
The key is associated with your user across every project you're a member of. There's no per-sandbox or per-project re-registration.
### Connect
```bash theme={null}
ssh @sandbox.tensorlake.ai
```
You land in `/home/tl-user` as the `tl-user` POSIX account, which is in the `sudo` group. The sandbox's hostname inside the session is `tl-sbx`.
To target a specific port (default is the SSH server on 22), prefix the username with the port:
```bash theme={null}
ssh 8080-@sandbox.tensorlake.ai
```
### File transfer
`scp`, `sftp`, and `rsync` ride the same connection:
```bash theme={null}
# Push a file in
scp ./script.py @sandbox.tensorlake.ai:/workspace/
# Pull a directory out
scp -r @sandbox.tensorlake.ai:/workspace/results ./
# Interactive sftp browser
sftp @sandbox.tensorlake.ai
# Mirror with rsync
rsync -avz ./src/ @sandbox.tensorlake.ai:/workspace/src/
```
### Port forwarding
All four standard forwarding modes are supported: TCP and UNIX-socket, each direction.
**Local forward (`-L`)**: reach a service running inside the sandbox from your laptop:
```bash theme={null}
# Web server on :8000 inside the sandbox → localhost:8888 on your laptop
ssh -L 8888:localhost:8000 @sandbox.tensorlake.ai
```
**Dynamic SOCKS (`-D`)**: route arbitrary traffic through the sandbox's network namespace:
```bash theme={null}
ssh -D 1080 -N -f @sandbox.tensorlake.ai
curl --socks5 localhost:1080 https://example.com
```
**Remote forward (`-R`)**: let processes inside the sandbox reach a service running on your laptop:
```bash theme={null}
# Service on your laptop's :9000 → reachable from inside the sandbox at localhost:9000
ssh -R 9000:localhost:9000 @sandbox.tensorlake.ai
```
**UNIX-socket forwards**: same shapes with socket paths instead of ports:
```bash theme={null}
ssh -L /tmp/local.sock:/tmp/remote.sock @sandbox.tensorlake.ai
ssh -R /tmp/remote.sock:/tmp/local.sock @sandbox.tensorlake.ai
```
### VS Code Remote-SSH
`tl sbx describe ` now prints an `SSH Config:` block you can copy into `~/.ssh/config`:
```bash theme={null}
tl sbx describe my-sandbox
```
You can also write the equivalent entry manually:
```sshconfig theme={null}
Host my-sandbox
HostName sandbox.tensorlake.ai
User
IdentityFile ~/.ssh/id_ed25519
IdentitiesOnly yes
```
Then run **Remote-SSH: Connect to Host…** in VS Code and pick `my-sandbox`. VS Code installs its server inside the sandbox automatically. JetBrains Gateway, Cursor, and any other Remote-SSH client work the same way.
When you open a folder in the connected window, use `/home/tl-user/workspace`. That path is writable by the default `tl-user` account and persisted across filesystem snapshots; `/workspace` is not `tl-user`-writable, and `/tmp/*` is writable but excluded from snapshots.
While VS Code (or any Remote-SSH client) is connected, the open SSH session counts as active proxy traffic and prevents idle-suspend. Once you disconnect, the sandbox suspends after `timeout_secs` of idle (see [Timeout](/sandboxes/lifecycle#timeout)), provided it is a [named sandbox](/sandboxes/lifecycle#overview). Resume it with `tl sbx resume ` and reconnect with the same SSH config entry; the sandbox id and your `Host` entry do not change. For the full development-environment workflow, see [Use a sandbox as your dev environment](/sandboxes/remote-dev).
### Persistent shells
`tmux` and `screen` work normally inside the sandbox, useful if you want a session that survives an `ssh` disconnect:
```bash theme={null}
ssh @sandbox.tensorlake.ai
tmux new -s work
# … run things …
# detach with Ctrl-b d, exit ssh, reconnect later, then:
ssh @sandbox.tensorlake.ai
tmux attach -t work
```
### Troubleshooting
When auth fails, the proxy disconnects with one of three specific messages.
**Key not registered.**
```text theme={null}
your SSH public key is not registered with Tensorlake. Run `tl login` and `tl sbx ssh keys add ~/.ssh/id_ed25519.pub`.
```
The offered key isn't on your Tensorlake account. Run `tl sbx ssh keys add ~/.ssh/id_ed25519.pub`.
**Sandbox not in any of your projects.**
```text theme={null}
sandbox is not present in any of your projects (verify the id with `tl sbx ls -r`).
```
Either a typo in the id, or the sandbox lives in a project you're not a member of. Run `tl sbx ls -r` to see running sandboxes in your active project.
**Sandbox is not running.**
```text theme={null}
sandbox is currently — resume it (`tl sbx resume `) or create a new one.
```
The sandbox exists in your project but isn't `running`. For named sandboxes, `tl sbx resume `; otherwise create a fresh one.
If your client offers multiple keys and one is unregistered, you'll see the static banner followed by `Permission denied (publickey).` because OpenSSH iterates through them. Constrain it to the registered key:
```sshconfig theme={null}
Host *.tensorlake.ai
IdentitiesOnly yes
IdentityFile ~/.ssh/id_ed25519
```
### CLI shortcut
If you don't need standard `ssh` semantics (e.g. you just want a quick shell without setting up keys), `tl sbx ssh` opens an interactive PTY using the WebSocket flow described below:
```bash theme={null}
tl sbx ssh my-sandbox
tl sbx ssh my-sandbox --shell /bin/sh
```
`tl sbx ssh` requires an interactive terminal and doesn't support port forwarding or file transfer. Use `ssh`, `scp`, etc. for that.
## PTY sessions
Use PTY sessions when you need to drive an interactive shell programmatically (for example a browser-based terminal UI, a recorder, or a remote-control tool) without a real SSH client. The session is created over HTTPS and terminal I/O moves over a WebSocket.
The PTY management endpoints live on the sandbox proxy host, derived from the sandbox's `ingress_endpoint`, not `https://api.tensorlake.ai`:
`https://.sandbox.tensorlake.ai`
Create, list, get, resize, and kill requests require `Authorization: Bearer $TENSORLAKE_API_KEY`. The WebSocket attach step also requires the per-session PTY token returned from session creation.
### Happy Path
1. Call `createPty()` or `create_pty()` on a connected sandbox client.
2. Tensorlake creates the PTY session, opens the WebSocket, and sends the initial `READY` frame for you.
3. Use the returned handle to send input, resize the terminal, stream output, wait for exit, disconnect, reconnect, or kill the session.
4. If you need to reattach later, call `connectPty()` or `connect_pty()` with the original `sessionId` and `token`.
### High-Level SDK API
The connected sandbox client now exposes a high-level PTY handle instead of making you manage WebSocket framing yourself.
The handle exposes:
* `sendInput()` / `send_input()` to write terminal input
* `resize()` to change rows and columns
* `wait()` to block until the PTY exits and get the exit code
* `disconnect()` to close the current WebSocket without killing the PTY
* `connect()` to reattach the same handle later
* `kill()` to terminate the PTY session over HTTP
* `onData()` / `on_data()` and `onExit()` / `on_exit()` to subscribe to output and exit events
Use `tl sbx ssh` when you want an interactive terminal immediately and do not need to manage PTY sessions programmatically:
```bash theme={null}
tl sbx ssh my-sandbox
```
```bash theme={null}
tl sbx ssh my-sandbox --shell /bin/sh
```
`tl sbx ssh` uses the PTY API under the hood and requires an interactive terminal. For reconnectable sessions or application-managed PTY control, use the Python or TypeScript SDK.
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox_client = Sandbox.create()
try:
pty = sandbox_client.create_pty(
command="/bin/bash",
args=["-l"],
env={"TERM": "xterm-256color"},
working_dir="/workspace",
cols=80,
rows=24,
)
pty.on_data(lambda data: print(data.decode("utf-8"), end=""))
pty.on_exit(lambda code: print(f"\nExited: {code}"))
pty.send_input("printf 'hello from PTY\\n'; pwd\\n")
pty.resize(120, 40)
pty.send_input("exit\n")
exit_code = pty.wait()
print(f"Final exit code: {exit_code}")
finally:
sandbox_client.terminate()
```
To reconnect later:
```python theme={null}
pty = sandbox_client.connect_pty(session_id, token)
```
```typescript theme={null}
import { Sandbox } from "tensorlake";
const sandboxClient = await Sandbox.create();
try {
const pty = await sandboxClient.createPty({
command: "/bin/bash",
args: ["-l"],
env: { TERM: "xterm-256color" },
workingDir: "/workspace",
cols: 80,
rows: 24,
onData: (data) => process.stdout.write(Buffer.from(data)),
onExit: (exitCode) => console.log("Exited:", exitCode),
});
await pty.sendInput("printf 'hello from PTY\\n'; pwd\\n");
await pty.resize(120, 40);
await pty.sendInput("exit\\n");
const exitCode = await pty.wait();
console.log("Final exit code:", exitCode);
} finally {
await sandboxClient.terminate();
}
```
To reconnect later, keep `pty.sessionId` and `pty.token` and call:
```typescript theme={null}
const pty = await sandboxClient.connectPty(sessionId, token, {
onData: (data) => process.stdout.write(Buffer.from(data)),
});
```
`createPty()` / `create_pty()` already open the WebSocket and send `READY`. Use `connectPty()` / `connect_pty()` only when you are reattaching to an existing session.
### Disconnect or kill
`disconnect()` closes the WebSocket but leaves the PTY running, so you can reattach later with `connectPty()` / `connect_pty()`. `kill()` terminates the session over HTTP.
```python theme={null}
# Detach without killing the shell: reconnect later with sandbox_client.connect_pty(...)
pty.disconnect()
# Terminate the session immediately
pty.kill()
```
```typescript theme={null}
// Detach without killing the shell: reconnect later with sandboxClient.connectPty(...)
pty.disconnect();
// Terminate the session immediately
await pty.kill();
```
### Raw HTTP and WebSocket Flow
The raw protocol is small enough that you can drive it yourself from any HTTP client plus any WebSocket client. These calls assume you already have a running sandbox ID or sandbox name.
#### 1. Create the PTY session
```bash theme={null}
curl -sS -X POST https://.sandbox.tensorlake.ai/api/v1/pty \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"command": "/bin/bash",
"args": ["-l"],
"env": {"TERM": "xterm-256color"},
"working_dir": "/workspace",
"rows": 24,
"cols": 80
}'
```
Response:
```json theme={null}
{
"session_id": "LYtJOrxE9Kz3bphPUDzuX",
"token": ""
}
```
#### 2. Attach the WebSocket
Open this URL:
```text theme={null}
wss://.sandbox.tensorlake.ai/api/v1/pty//ws
```
Send the PTY token on the upgrade request:
```http theme={null}
X-PTY-Token:
```
If your client cannot set headers, append `?token=` to the WebSocket URL instead.
#### 3. Exchange PTY frames
| Direction | Bytes | Meaning |
| ---------------- | ----------------------------------- | ------------------------------------------------------------ |
| Client -> server | `02` | `READY`: flush any buffered output |
| Client -> server | `00` + UTF-8 bytes | Send terminal input |
| Client -> server | `01` + `cols` + `rows` | Resize terminal, with `cols` then `rows` as big-endian `u16` |
| Server -> client | `00` + raw bytes | Terminal output |
| Server -> client | `03` + 4-byte big-endian signed int | Process exit code |
Common examples:
| Action | Bytes |
| ---------------- | ------------------- |
| Send `READY` | `02` |
| Run `pwd\n` | `00 70 77 64 0a` |
| Run `exit\n` | `00 65 78 69 74 0a` |
| Exit code `0` | `03 00 00 00 00` |
| Resize to 120x40 | `01 00 78 00 28` |
#### 4. Close or abort
To close cleanly, write `exit\n` to the shell and wait for the `0x03` exit frame followed by the normal WebSocket close.
To terminate the session immediately:
```bash theme={null}
curl -X DELETE https://.sandbox.tensorlake.ai/api/v1/pty/ \
-H "Authorization: Bearer $TENSORLAKE_API_KEY"
```
### Notes
* `createPty()` / `create_pty()` send `READY` for you immediately after the socket opens.
* Closing the WebSocket does not kill the PTY session. You can reconnect while the shell is still running.
* Persist the original PTY token if you plan to reconnect. [Get PTY Session](/api-reference/v2/pty/get) and [List PTY Sessions](/api-reference/v2/pty/list) do not return it again.
* PTY sessions with no connected clients are killed after 300 seconds of inactivity.
* You can resize either with the `0x01` WebSocket frame or with [Resize PTY Session](/api-reference/v2/pty/resize).
* For the endpoint-by-endpoint API reference, see [PTY Sessions API](/api-reference/v2/pty/introduction).
## Related Guides
Run one-shot commands and manage long-running background processes.
Forward arbitrary TCP ports (Postgres, VNC, custom binary protocols) over an authenticated WebSocket.
How long a sandbox lives, and what happens on suspend, resume, and terminate.
# Sandboxes Quickstart
Source: https://docs.tensorlake.ai/sandboxes/quickstart
Install the SDK, authenticate, and run your first sandbox in under five minutes.
## Setup
```bash theme={null}
curl -fsSL https://tensorlake.ai/install | sh
```
This installs the `tl` CLI, which you can use to manage sandboxes and other resources from the command line.
```bash theme={null}
tl login
```
```bash theme={null}
pip install tensorlake
```
This installs the Python SDK. The `tl` CLI is installed separately; see the **CLI** tab.
Get an API key from the [Tensorlake Dashboard](https://cloud.tensorlake.ai) and set it in your environment:
```bash theme={null}
export TENSORLAKE_API_KEY=your-api-key-here
```
```bash theme={null}
npm install tensorlake
```
This installs the TypeScript SDK. The `tl` CLI is installed separately; see the **CLI** tab.
Get an API key from the [Tensorlake Dashboard](https://cloud.tensorlake.ai) and set it in your environment:
```bash theme={null}
export TENSORLAKE_API_KEY=your-api-key-here
```
After you run `tl login`, you can manage your sandboxes in the [Tensorlake Dashboard](https://cloud.tensorlake.ai). You can also create API keys there for sandbox connections. See [Authentication](/platform/authentication#api-keys) for the full API key setup flow.
## Run your first sandbox
Create a tiny sandbox for a quick task, or provision one with more CPU and memory for heavier workloads.
```bash theme={null}
# Create an ephemeral sandbox (no name: terminates when done, cannot be suspended)
tl sbx create
# Run code inside the sandbox
tl sbx exec python -c 'print("Hello from sandbox")'
# Copy files in or out as the sandbox accumulates state
tl sbx cp local-file.txt :/workspace/local-file.txt
```
```python theme={null}
from tensorlake.sandbox import Sandbox
# Ephemeral sandbox: no name, terminates when done, cannot be suspended
sandbox = Sandbox.create()
# Run code inside the sandbox
result = sandbox.run("python", ["-c", "print('Hello from sandbox')"])
print(result.stdout)
# Copy files in or out as the sandbox accumulates state
sandbox.write_file("/workspace/local-file.txt", b"example content")
file_bytes = bytes(sandbox.read_file("/workspace/local-file.txt"))
print(file_bytes.decode("utf-8"))
```
```typescript theme={null}
import { Sandbox } from "tensorlake";
// Ephemeral sandbox: no name, terminates when done, cannot be suspended
const sandbox = await Sandbox.create();
console.log(sandbox.sandboxId, sandbox.status);
for (const sb of await Sandbox.list()) {
console.log(sb.sandboxId, sb.status);
}
// Run code inside the sandbox
const result = await sandbox.run("python", {
args: ["-c", "print('Hello from sandbox')"],
});
console.log(result.stdout);
// Copy files in or out as the sandbox accumulates state
await sandbox.writeFile(
"/workspace/local-file.txt",
new TextEncoder().encode("example content"),
);
const fileBytes = await sandbox.readFile("/workspace/local-file.txt");
console.log(new TextDecoder().decode(fileBytes));
```
#### Configure CPU, Memory, Disk, and Timeout
You can specify CPU, memory, disk, and timeout parameters when creating sandboxes. The defaults are 1 CPU, 1024 MB memory, 10 GB disk, and 600 seconds timeout.
```bash theme={null}
tl sbx create --cpus 2.0 --memory 2048 --disk_mb 51200 --timeout 600
```
```python theme={null}
sandbox = Sandbox.create(cpus=2.0, memory_mb=2048, disk_mb=12000, timeout_secs=600)
```
```typescript theme={null}
const sandbox = await Sandbox.create({
cpus: 2.0,
memoryMb: 2048,
diskMb: 12000,
timeoutSecs: 600,
});
```
#### Suspend and Resume
Tensorlake sandboxes can be suspended and resumed. A resumed sandbox continues from the exact memory and file system state, it was suspended.
This is useful when you want to preserve the sandbox state without paying for idle compute time.
You have to name a sandbox to make them suspendable after timeout. Sandboxes without a name are ephemeral and thrown away after the timeout.
```bash cli theme={null}
tl sbx suspend
tl sbx resume
```
```python sandbox.py theme={null}
sandbox = Sandbox.create(name="my-agent-env", cpus=2.0, memory_mb=2048)
sandbox.suspend()
sandbox.resume()
```
```typescript sandbox.ts theme={null}
const sandbox = await Sandbox.create({
name: "my-agent-env",
cpus: 2.0,
memoryMb: 2048,
timeoutSecs: 600,
});
await sandbox.suspend();
await sandbox.resume();
```
#### Sandbox Checkpoints
Checkpoints are point in time snapshot of a sandbox that you can use to start new sandboxes from.
```bash cli theme={null}
tl sbx checkpoint
```
```python sandbox.py theme={null}
# Save a checkpoint you can return to later
snapshot = sandbox.checkpoint()
print(snapshot.snapshot_id)
```
```typescript sandbox.ts theme={null}
const snapshot = await sandbox.checkpoint();
console.log(snapshot.snapshotId);
```
#### Terminate Sandboxes
```bash cli theme={null}
tl sbx terminate
```
```python sandbox.py theme={null}
sandbox.terminate()
```
```typescript sandbox.ts theme={null}
await sandbox.terminate();
```
## SSH Access
You can also SSH into your sandboxes for an interactive terminal experience.
```bash theme={null}
tl sbx ssh
```
This uses a WebSocket-backed PTY session to connect you to the sandbox.
For programmatic access, you can create and control PTY session with the [Python and TypeScript SDKs](/sandboxes/pty-sessions).
## Next Steps
Understand the different states and behaviors of sandboxes.
Run shell commands and stream output.
Use and customize sandbox images for your use case.
# Use a sandbox as your dev environment
Source: https://docs.tensorlake.ai/sandboxes/remote-dev
Get a portable cloud development workstation: SSH in from any machine, idle-suspend when you're not using it, resume by name with state intact.
A Tensorlake sandbox is not only an execution surface for agents. It is also a real development environment you can SSH into from any machine. Open it in VS Code or any Remote-SSH client, work normally, walk away when you're done. The sandbox idle-suspends on its own and stops charging. Resume tomorrow under the same name and your shell history, installed packages, in-progress branches, running `tmux` sessions, and `~/.vscode-server` are exactly where you left them.
Compared to a long-running VM or a traditional cloud-dev product:
* **Portable identity, not portable hardware.** You connect to the same sandbox id from any machine. Which region or host the underlying VM lives on is the platform's problem.
* **Suspend-on-idle, not always-on.** Named sandboxes auto-suspend after `timeout_secs` of no proxy traffic, so you only pay while you are actively connected.
* **Resume preserves state.** Filesystem, memory, and running processes all survive suspend. The sandbox id never changes, so your `~/.ssh/config` entry keeps working forever.
## Prerequisites
* The [`tl` CLI](/sandboxes/introduction) installed and logged in (`tl login`).
* An ed25519 (or RSA) SSH keypair on your laptop. Generate one with `ssh-keygen -t ed25519` if you don't have one.
## One-time setup: register your SSH key
```bash theme={null}
tl sbx ssh keys add --name laptop ~/.ssh/id_ed25519.pub
tl sbx ssh keys ls
```
The key is associated with your Tensorlake user across every project, so you only do this once per laptop. See [SSH and PTY Sessions](/sandboxes/pty-sessions#one-time-setup) for the `TENSORLAKE_API_KEY` gotcha and other auth details.
## Create your sandbox
Create a **named** sandbox so it can be suspended and resumed:
```bash theme={null}
tl sbx create my-dev --cpus 2 --memory 4096 --disk_mb 25600
```
Pick CPUs, memory, and disk that fit your workload: language servers, builds, and test runs are usually the limiter on CPU/memory, while toolchains, container images, and dataset checkouts are what fill the disk. `--disk_mb` is the root filesystem size in MiB; the allowed range is 10240–102400 (10–100 GiB). The name (`my-dev`) is how you'll refer to the sandbox in `tl sbx suspend/resume` commands.
Grab the SSH config block:
```bash theme={null}
tl sbx describe my-dev
```
Copy the `SSH Config:` block into `~/.ssh/config`. The sandbox id inside that block is stable across suspend/resume, so you only need to do this once.
Already have a project image with your toolchain pre-installed (Node, Python, CUDA, your repo's `apt` deps)? Pass `--image my-image` so every new sandbox starts pre-configured. See [Build and Import Images](/sandboxes/images).
### Pick a timeout
Named sandboxes idle-suspend after `timeout_secs` of no proxy traffic. The default is `600` seconds (10 minutes). While you are SSH'd in, that clock is paused, so 10 minutes is usually fine. If you want a longer idle window before suspend (e.g. so a brief network blip doesn't suspend the sandbox out from under you), bump it:
```bash theme={null}
tl sbx create my-dev --cpus 2 --memory 4096 --disk_mb 25600 --timeout 3600
```
`--timeout 0` requests the plan maximum (24 hours on On-Demand). See [Timeout](/sandboxes/lifecycle#timeout) for precise semantics and plan limits.
## Add an SSH config entry
`tl sbx describe my-dev` prints an `SSH Config:` block you can paste directly into `~/.ssh/config`.
You can also write the equivalent entry manually:
```sshconfig theme={null}
Host my-dev
HostName sandbox.tensorlake.ai
User
IdentityFile ~/.ssh/id_ed25519
IdentitiesOnly yes
ServerAliveInterval 30
ServerAliveCountMax 3
```
The `Host` alias (`my-dev`) is just a local label. Pick whatever you like. The `User` must be the sandbox id; this is what the gateway routes on.
`ServerAliveInterval` keeps the connection healthy across short network gaps. `IdentitiesOnly yes` makes sure your client only offers the registered key (important if you have several keys in your agent).
Test it:
```bash theme={null}
ssh my-dev
# tl-user@tl-sbx:~$
```
## Open it in VS Code
1. Install the **Remote - SSH** extension (`ms-vscode-remote.remote-ssh`).
2. Run **Remote-SSH: Connect to Host…** and pick `my-dev`.
3. **File → Open Folder** → `/home/tl-user/workspace`. That path is writable by the default `tl-user` account and persisted across snapshots; `/workspace` is not `tl-user`-writable in the default image, and `/tmp/*` is writable but excluded from snapshots.
4. First connect takes \~30 seconds while VS Code installs its server inside the sandbox under `~/.vscode-server`. That directory lives under `/home/tl-user`, so it persists across suspend/resume, and subsequent connects are much faster.
JetBrains Gateway, Cursor, and any other Remote-SSH client work the same way.
## Day to day
Work normally: `git clone`, install deps, run your stack. Two things to keep in mind:
* **Long-running jobs vs. SSH disconnect.** When your SSH session ends and no other proxy traffic is in flight, the idle clock starts and the sandbox eventually suspends. Suspend preserves running processes, so a `tmux` job resumes when you do, but it does *not* make progress while the sandbox is suspended. For unattended work that needs to keep running, raise `--timeout`, keep a client connected, or use [background processes](/sandboxes/commands) (which are designed for fire-and-forget work).
* **Suspend explicitly when you're done.** `tl sbx suspend my-dev` stops the meter immediately, rather than waiting for the idle timeout to fire.
Resume tomorrow:
```bash theme={null}
tl sbx resume my-dev
ssh my-dev
```
The sandbox id never changes across suspend/resume, so your `~/.ssh/config` entry and any VS Code Remote-SSH bookmark keep working. There is nothing to update on your laptop, even after weeks of suspend.
## Next steps
Port forwarding, `scp`/`rsync`, PTY API, and troubleshooting.
Suspend, resume, timeout semantics, and snapshots.
Bake pre-installed dependencies and tools into a reusable image.
Outbound access controls, exposed ports, and tunnels.
# SDK Reference
Source: https://docs.tensorlake.ai/sandboxes/sdk-reference
Sandbox, commands, processes, PTYs, files, snapshots, desktop control, and networking reference
This page is the runtime API surface of the Sandbox SDK in one place. It maps the Python and TypeScript sandbox-management APIs you'll use to create sandboxes, execute work inside them, manage files and processes, and interact with desktop sandboxes. Each detail page linked below expands on the same APIs with longer examples and edge cases.
All method names below use the Python form. The TypeScript SDK mirrors them in camelCase (`start_process` → `startProcess`, `read_file` → `readFile`, `memory_mb` → `memoryMb`, etc.). The same JavaScript runtime API is used from Node.js.
This page focuses on the sandbox runtime SDK surface. Related interfaces documented elsewhere include the CLI and HTTP API, the image-building DSLs in [Build and Import Images](/sandboxes/images), and the browser/VNC integration details in [Computer Use](/sandboxes/computer-use).
Every method below is also available as an async-native variant on `AsyncSandbox` in Python: same names and parameters, just `await`ed. See the [Async SDK](/sandboxes/async) page for usage. The TypeScript SDK is already Promise-based, so the methods shown in the TypeScript tabs *are* the async API.
## Sandbox
`Sandbox` is the top-level entry point for managing sandboxes in your namespace. Use `Sandbox.create()` to start a sandbox and get a handle. Use `Sandbox.connect()` to reconnect to an existing sandbox by ID or name.
```python theme={null}
from tensorlake.sandbox import Sandbox
# Authentication comes from `tl login` or TENSORLAKE_API_KEY env var
```
```typescript theme={null}
import { Sandbox } from "tensorlake";
```
See [Authentication](/platform/authentication) for the full auth flow.
### Create
Create a sandbox. Omit `name` for an ephemeral sandbox (cannot be suspended); pass `name` to create a named sandbox that supports suspend/resume. Returns a connected `Sandbox` handle (blocks until the sandbox is `running`).
| Parameter | Type | Default | Description |
| ------------------------ | ------------------- | ---------------- | --------------------------------------------------------------------------- |
| `name` | `str \| None` | `None` | Human-readable name. Required for suspend/resume. |
| `cpus` | `float` | `1.0` | Number of CPUs to allocate. |
| `memory_mb` | `int` | `1024` | Memory in megabytes. 1024–8192 MB per CPU core. |
| `timeout_secs` | `int` | `600` | Auto-suspend (named) or auto-terminate (ephemeral) after this many seconds. |
| `image` | `str \| None` | platform default | Name or ID of a prebuilt [sandbox image](/sandboxes/images). |
| `snapshot_id` | `str \| None` | `None` | Restore from a snapshot instead of booting a fresh VM. |
| `entrypoint` | `list[str] \| None` | `None` | Custom entrypoint command. |
| `allow_internet_access` | `bool` | `True` | Allow outbound internet traffic (see [Networking](/sandboxes/networking)). |
| `allow_out` / `deny_out` | `list[str] \| None` | `None` | Outbound destination allow/deny lists. |
To expose user ports for inbound traffic, call [`sandbox.update(exposed_ports=..., allow_unauthenticated_access=...)`](#update) after `create()`.
```python theme={null}
sandbox = Sandbox.create(
name="my-env",
cpus=2.0,
memory_mb=4096,
timeout_secs=1800,
)
print(sandbox.sandbox_id, sandbox.status)
```
```typescript theme={null}
const sandbox = await Sandbox.create({
name: "my-env",
cpus: 2.0,
memoryMb: 4096,
timeoutSecs: 1800,
});
```
### Create and connect
`Sandbox.create()` creates a sandbox and returns a live `Sandbox` handle you can immediately run commands against.
```python theme={null}
sandbox = Sandbox.create(cpus=2.0, memory_mb=2048)
result = sandbox.run("python", ["-c", "print('hello')"])
print(result.stdout)
# Context manager terminates the sandbox on exit
```
```typescript theme={null}
const sandbox = await Sandbox.create({ cpus: 2.0, memoryMb: 2048 });
try {
const result = await sandbox.run("python", { args: ["-c", "print('hello')"] });
console.log(result.stdout);
} finally {
await sandbox.terminate();
}
```
### Connect
Get a `Sandbox` handle for an existing sandbox (by ID or name) without creating a new one. Use this to rejoin a named sandbox after `resume`, or to operate on a sandbox a different process created.
```python theme={null}
sandbox = Sandbox.connect("my-env")
print(sandbox.sandbox_id) # always UUID, even if you connected by name
print(sandbox.name) # "my-env"
```
```typescript theme={null}
const sandbox = Sandbox.connect("my-env");
```
### List and get
`Sandbox.connect()` attaches to a single sandbox by ID or name. To enumerate all sandboxes in your namespace, use `Sandbox.list()`.
```python theme={null}
sandbox = Sandbox.connect("my-env")
print(sandbox.sandbox_id, sandbox.name, sandbox.status)
for sb in Sandbox.list():
print(sb.sandbox_id, sb.status)
```
```typescript theme={null}
const sandbox = await Sandbox.connect("my-env");
console.log(sandbox.sandboxId, sandbox.name, sandbox.status);
```
### Update
`sandbox.update()` is the unified instance method for changing a sandbox's name, exposed user ports, unauthenticated-access flag, or egress [network policy](/sandboxes/networking#update-the-policy-on-a-running-sandbox). Renaming and port exposure are the same call. Assigning a name to an ephemeral sandbox converts it to a named sandbox that supports suspend/resume.
| Parameter | Type | Default | Description |
| ------------------------------ | --------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | `str \| None` | `None` | New name for the sandbox. Naming an ephemeral sandbox makes it non-ephemeral and enables suspend/resume. |
| `allow_unauthenticated_access` | `bool \| None` | `None` | Whether exposed user ports accept traffic without TensorLake auth. |
| `exposed_ports` | `list[int] \| None` | `None` | User ports routable through the sandbox proxy. Port `9501` is reserved. |
| `network` | `NetworkConfig \| ClearNetworkPolicy \| None` | `None` | Egress network policy update. Omit to keep the current policy, pass a `NetworkConfig` to replace it, or pass `CLEAR_NETWORK_POLICY` (Python) / `null` (TypeScript) to clear it. Applied to the running sandbox atomically. |
```python theme={null}
from tensorlake.sandbox import CLEAR_NETWORK_POLICY, NetworkConfig
info = sandbox.update(name="my-env")
# Tighten egress on the running sandbox, then later clear it.
sandbox.update(network=NetworkConfig(allow_out=["api.example.com"]))
sandbox.update(network=CLEAR_NETWORK_POLICY)
```
```typescript theme={null}
const info = await sandbox.update({ name: "my-env" });
// Tighten egress on the running sandbox, then later clear it.
await sandbox.update({
network: { allowInternetAccess: false, allowOut: ["api.example.com"], denyOut: [] },
});
await sandbox.update({ network: null });
```
If you only have a sandbox ID (for example, from `Sandbox.list()`), connect first and chain `update()`:
```python theme={null}
info = Sandbox.connect("sbx-123").update(name="my-env", exposed_ports=[8080])
```
```typescript theme={null}
const sandbox = await Sandbox.connect("sbx-123");
const info = await sandbox.update({ name: "my-env", exposedPorts: [8080] });
```
### Suspend and resume
Pause a running named sandbox in place; resume it later under the same ID with its memory, filesystem, and running processes intact. Ephemeral sandboxes return an error on `suspend`.
```python theme={null}
sandbox.suspend()
sandbox.resume()
```
```typescript theme={null}
await sandbox.suspend();
await sandbox.resume();
```
### Terminate
`terminate()` ends the sandbox permanently. `Terminated` is a final state and cannot be reversed.
```python theme={null}
sandbox.terminate()
```
```typescript theme={null}
await sandbox.terminate();
```
See [Lifecycle](/sandboxes/lifecycle) for the full state machine.
### Expose and unexpose ports
Route public internet traffic to services listening on user ports inside the sandbox. Requests arrive at `https://-.sandbox.tensorlake.ai`.
Port exposure is just a `sandbox.update()` call: pass `exposed_ports` and (optionally) `allow_unauthenticated_access`. Pass `exposed_ports=[]` to remove all exposed ports.
```python theme={null}
sandbox.update(exposed_ports=[8080], allow_unauthenticated_access=False)
sandbox.update(exposed_ports=[]) # remove all
```
```typescript theme={null}
await sandbox.update({ exposedPorts: [8080], allowUnauthenticatedAccess: false });
await sandbox.update({ exposedPorts: [] }); // remove all
```
See [Networking](/sandboxes/networking) for authenticated vs. unauthenticated access and how clients reach user ports.
### Snapshot and restore
Capture a reusable artifact of the sandbox (filesystem + memory + running processes). Restore by passing `snapshot_id` to `Sandbox.create()`.
```python theme={null}
snapshot = sandbox.checkpoint()
# Restore into a fresh sandbox
restored = Sandbox.create(snapshot_id=snapshot.snapshot_id)
# Manage snapshots
sandbox.list_snapshots()
Sandbox.get_snapshot(snapshot.snapshot_id)
Sandbox.delete_snapshot(snapshot.snapshot_id)
```
```typescript theme={null}
const snapshot = await sandbox.checkpoint();
const restored = await Sandbox.create({ snapshotId: snapshot.snapshotId });
await sandbox.listSnapshots();
await Sandbox.getSnapshot(snapshot.snapshotId);
await Sandbox.deleteSnapshot(snapshot.snapshotId);
```
Suspend pauses *this* sandbox; snapshot captures a reusable artifact you restore into a *new* sandbox. See [Snapshots](/sandboxes/snapshots).
## Sandbox handle
The `Sandbox` object returned by `Sandbox.create()` or `Sandbox.connect()` is how you execute work inside a running sandbox. All methods below target a single live sandbox.
| Property (Python) | Property (TypeScript) | Type | Description |
| ----------------- | --------------------- | ------------- | --------------------------------------------- |
| `sandbox_id` | `sandboxId` | `str` | Server-assigned UUID. |
| `name` | `name` | `str \| None` | Human-readable name, or `None` for ephemeral. |
### Run a command
`run()` is the short-lived foreground execution primitive: send a command, wait for it to exit, receive captured output. Use it for the common case of "do this one thing and give me the result."
```python theme={null}
result = sandbox.run("python", ["-c", "print('hello')"], timeout=30)
print(result.stdout)
print(result.stderr)
print(result.exit_code)
```
```typescript theme={null}
const result = await sandbox.run("python", {
args: ["-c", "print('hello')"],
timeout: 30,
});
console.log(result.stdout, result.stderr, result.exitCode);
```
Pass `env={"KEY": "value"}` (Python) or `env: { KEY: "value" }` (TypeScript) for per-command environment variables. See [Environment Variables](/sandboxes/environment-variables).
See [Commands & Processes](/sandboxes/commands) for streaming, multi-step shell pipelines, and error handling.
### Background processes
For long-running or concurrent work, start a process and keep the handle so you can monitor, stream output, and signal it.
```python theme={null}
proc = sandbox.start_process("python", ["-m", "http.server", "8080"])
print(proc.pid)
for p in sandbox.list_processes():
print(p.pid, p.command, p.status)
for event in sandbox.follow_output(proc.pid):
print(event.line, end="")
import signal
sandbox.send_signal(proc.pid, signal.SIGTERM)
```
```typescript theme={null}
const proc = await sandbox.startProcess("python", { args: ["-m", "http.server", "8080"] });
for (const p of await sandbox.listProcesses()) {
console.log(p.pid, p.command, p.status);
}
for await (const event of sandbox.followOutput(proc.pid)) {
console.log(event.line);
}
await sandbox.sendSignal(proc.pid, 15); // SIGTERM
await sandbox.killProcess(proc.pid); // SIGKILL
```
### Writing to stdin
Drive a process interactively from code by writing bytes to its stdin, then closing the stream when you're done.
```python theme={null}
proc = sandbox.start_process("python", ["-c", "import sys; print(sys.stdin.read())"])
sandbox.write_stdin(proc.pid, b"hello from stdin\n")
sandbox.close_stdin(proc.pid)
```
```typescript theme={null}
const proc = await sandbox.startProcess("python", {
args: ["-c", "import sys; print(sys.stdin.read())"],
});
await sandbox.writeStdin(proc.pid, new TextEncoder().encode("hello from stdin\n"));
await sandbox.closeStdin(proc.pid);
```
See [Commands & Processes](/sandboxes/commands) for the full API.
### PTY sessions
Open an interactive terminal inside the sandbox. The PTY is created over HTTPS; terminal I/O then moves over a WebSocket attached to the session.
```python theme={null}
pty = sandbox.create_pty(
command="bash",
cols=120,
rows=40,
env={"PS1": "sandbox$ "},
)
# `pty` is a connected Pty handle - see PTY Sessions for send_input / resize / wait.
# Reconnect later from session_id and token:
reattached = sandbox.connect_pty(pty.session_id, pty.token)
```
```typescript theme={null}
const pty = await sandbox.createPty({
command: "bash",
cols: 120,
rows: 40,
});
```
See [PTY Sessions](/sandboxes/pty-sessions) for the wire protocol, reconnect flow, and resize frames.
### File operations
Copy data in and out of the sandbox filesystem without spawning a shell.
```python theme={null}
sandbox.write_file("/workspace/data.csv", b"name,score\nAlice,95\n")
content = bytes(sandbox.read_file("/workspace/data.csv"))
print(content.decode("utf-8"))
for entry in sandbox.list_directory("/workspace"):
print(entry.name, entry.is_dir, entry.size)
sandbox.delete_file("/workspace/data.csv")
```
```typescript theme={null}
await sandbox.writeFile(
"/workspace/data.csv",
new TextEncoder().encode("name,score\nAlice,95\n"),
);
const bytes = await sandbox.readFile("/workspace/data.csv");
console.log(new TextDecoder().decode(bytes));
const entries = await sandbox.listDirectory("/workspace");
await sandbox.deleteFile("/workspace/data.csv");
```
See [File Operations](/sandboxes/file-operations) for binary uploads, recursive listings, and move/copy patterns.
### Desktop sessions
Desktop sandboxes expose a higher-level remote-control handle on top of the normal `Sandbox` APIs. Use this with `tensorlake/ubuntu-vnc` to capture screenshots and drive mouse and keyboard input through the authenticated sandbox proxy.
```python theme={null}
with sandbox.connect_desktop(password="tensorlake") as desktop:
png_bytes = desktop.screenshot()
desktop.move_mouse(640, 400)
desktop.click()
desktop.type_text("hello from desktop")
print(desktop.width, desktop.height)
```
```typescript theme={null}
const desktop = await sandbox.connectDesktop({ password: "tensorlake" });
try {
const pngBytes = await desktop.screenshot();
await desktop.moveMouse(640, 400);
await desktop.click();
await desktop.typeText("hello from desktop");
console.log(desktop.width, desktop.height);
} finally {
await desktop.close();
}
```
Common desktop methods are:
| Python | TypeScript | Description |
| ------------------------------- | ----------------------------- | ------------------------------------------------ |
| `screenshot()` | `screenshot()` | Capture the current desktop as PNG bytes. |
| `move_mouse(x, y)` | `moveMouse(x, y)` | Move the pointer to absolute screen coordinates. |
| `click()` | `click()` | Click the current pointer location. |
| `double_click()` | `doubleClick()` | Double-click the current pointer location. |
| `scroll_up()` / `scroll_down()` | `scrollUp()` / `scrollDown()` | Scroll vertically. |
| `press(keys)` | `press(keys)` | Send a key or key chord. |
| `type_text(text)` | `typeText(text)` | Type text into the active window. |
| `width`, `height` | `width`, `height` | Desktop resolution. |
See [Computer Use](/sandboxes/computer-use) for reconnect patterns, coordinate workflows, and noVNC integration.
### Terminate
Shortcut for `sandbox.terminate()` that uses the handle you already have.
```python theme={null}
sandbox.terminate()
```
```typescript theme={null}
await sandbox.terminate();
```
## Data models
The SDK returns typed objects for every API call. The fields below are the ones you'll read most often. Field names shown in Python `snake_case`; TypeScript uses the `camelCase` equivalent.
### SandboxInfo
Returned by `Sandbox.create()`, `Sandbox.connect()`, `client.list()`, and the suspend/resume/expose calls.
| Field | Type | Description |
| ------------------------------ | ------------------------ | ------------------------------------------------------------------------------------- |
| `sandbox_id` | `str` | Server UUID. |
| `name` | `str \| None` | Name, or `None` for ephemeral. |
| `namespace` | `str` | Namespace owning the sandbox. |
| `status` | `SandboxStatus` | One of `pending`, `running`, `snapshotting`, `suspending`, `suspended`, `terminated`. |
| `image` | `str \| None` | Sandbox image in use. |
| `resources` | `ContainerResourcesInfo` | `.cpus: float`, `.memory_mb: int`. |
| `timeout_secs` | `int \| None` | Auto-suspend/terminate timeout. |
| `exposed_ports` | `list[int] \| None` | Public-routed user ports. |
| `allow_unauthenticated_access` | `bool` | Whether exposed ports accept unauthenticated traffic. |
| `ingress_endpoint` | `str \| None` | Base ingress origin for this sandbox's current placement. |
| `sandbox_url` | `str \| None` | Sandbox-specific management URL derived from `ingress_endpoint`. |
| `entrypoint` | `list[str] \| None` | Custom entrypoint command. |
| `network` | `NetworkConfig \| None` | Outbound network configuration (`allow_internet_access`, `allow_out`, `deny_out`). |
| `created_at` | `datetime \| None` | Creation timestamp. |
| `terminated_at` | `datetime \| None` | Termination timestamp (if terminated). |
### Sandbox
Returned by `Sandbox.create()` and `Sandbox.connect()`. Exposes the runtime methods documented above plus:
| Property | Type | Description |
| -------------------------- | ------------- | -------------------------------- |
| `sandbox_id` / `sandboxId` | `str` | UUID, even if connected by name. |
| `name` | `str \| None` | Human-readable name. |
### ProcessInfo
Returned by `start_process()` and `list_processes()`.
| Field | Type | Description |
| ------------ | ------------------ | -------------------------------------------------------- |
| `pid` | `int` | Process ID inside the sandbox. |
| `command` | `str` | The executed command. |
| `args` | `list[str]` | Command arguments. |
| `status` | `ProcessStatus` | One of `running`, `exited`, `signaled`. |
| `exit_code` | `int \| None` | Exit code once the process has exited. |
| `signal` | `int \| None` | Signal number if the process was terminated by a signal. |
| `started_at` | `datetime` | When the process started. |
| `ended_at` | `datetime \| None` | When the process ended. |
### CommandResult
Returned by `run()`.
| Field | Type | Description |
| ----------- | ----- | ------------------------- |
| `stdout` | `str` | Captured standard output. |
| `stderr` | `str` | Captured standard error. |
| `exit_code` | `int` | Process exit code. |
### SnapshotInfo
Returned by the snapshot APIs.
| Field | Type | Description |
| ------------- | ------------------ | ----------------------------------------------- |
| `snapshot_id` | `str` | Server-assigned ID, use to restore. |
| `sandbox_id` | `str` | Source sandbox this snapshot was captured from. |
| `status` | `SnapshotStatus` | One of `in_progress`, `completed`, `failed`. |
| `size_bytes` | `int \| None` | Size of the snapshot artifact. |
| `created_at` | `datetime \| None` | Capture timestamp. |
## Learn more
State machine, suspend/resume, timeouts.
Run commands, capture output, stream, and manage background processes.
Interactive shells over WebSocket.
Read, write, list, delete.
Capture and restore full VM state.
Prebuild dependencies into reusable images.
Desktop sessions, screenshots, mouse, keyboard.
Expose user ports to the internet.
Per-command and per-PTY environment.
# Skills in Sandboxes
Source: https://docs.tensorlake.ai/sandboxes/skills-in-sandboxes
Pre-load TensorLake skill files inside sandbox images so coding agents auto-discover them at startup.
Coding agents discover skill files by scanning specific directories at startup. By placing TensorLake skill files in the right paths inside a sandbox image, any agent running in the sandbox will automatically pick them up without manual installation.
## How Agents Discover Skills
Each coding agent scans a different directory for skill files:
| Agent | Skill File | Discovery Path |
| -------------- | ----------- | ---------------------------------------------------------------- |
| Claude Code | `SKILL.md` | `~/.claude/skills//SKILL.md` |
| OpenAI Codex | `AGENTS.md` | `~/.agents/skills//SKILL.md` or `AGENTS.md` in working dir |
| Google ADK | `SKILL.md` | Loaded explicitly via `load_skill_from_dir()` |
| Cursor | `.mdc` | `.cursor/rules/*.mdc` |
| Cline | `.md` | `.clinerules/` |
| Windsurf | `.md` | `.windsurf/rules/*.md` |
| GitHub Copilot | `.md` | `.github/copilot-instructions.md` |
To make skills work inside a sandbox, bake the skill files into the image at the paths the agent expects.
## Create a Skills Image
### Any Agent
The simplest way to install skills for any agent is with the [skills](https://skills.sh) CLI. It places skill files in the correct discovery paths for Claude Code, Codex, Cursor, Windsurf, and other supported agents.
```python theme={null}
from tensorlake import Image
image = (
Image(name="with-skills", base_image="tensorlake/ubuntu-systemd")
.run("apt-get update && apt-get install -y nodejs npm python3 python3-pip")
.run("npm install -g skills")
.run("skills add tensorlakeai/tensorlake-skills --all -y --copy")
.run("python3 -m pip install --break-system-packages tensorlake")
)
```
```typescript theme={null}
import { Image } from "tensorlake";
const image = new Image({
name: "with-skills",
baseImage: "tensorlake/ubuntu-systemd",
})
.run("apt-get update && apt-get install -y nodejs npm python3 python3-pip")
.run("npm install -g skills")
.run("skills add tensorlakeai/tensorlake-skills --all -y --copy")
.run("python3 -m pip install --break-system-packages tensorlake");
```
```dockerfile Dockerfile theme={null}
FROM tensorlake/ubuntu-systemd
RUN apt-get update && apt-get install -y nodejs npm python3 python3-pip
RUN npm install -g skills
RUN skills add tensorlakeai/tensorlake-skills --all -y --copy
RUN python3 -m pip install --break-system-packages tensorlake
```
* `--all` installs skills to all detected agents
* `-y` skips confirmation prompts for non-interactive use
* `--copy` copies files instead of symlinking, which is more reliable inside containers
### Claude Code Only
If you only need Claude Code support, copy the skill into `~/.claude/skills/` inside the image:
```python theme={null}
from tensorlake import Image
image = (
Image(name="claude-code-skills", base_image="tensorlake/ubuntu-systemd")
.run("apt-get update && apt-get install -y git python3 python3-pip")
.run("git clone https://github.com/tensorlakeai/tensorlake-skills /tmp/tensorlake-skills")
.run(
"mkdir -p /root/.claude/skills/tensorlake && "
"cp -r /tmp/tensorlake-skills/SKILL.md /tmp/tensorlake-skills/references "
"/root/.claude/skills/tensorlake/"
)
.run("rm -rf /tmp/tensorlake-skills")
.run("python3 -m pip install --break-system-packages tensorlake")
)
```
```typescript theme={null}
import { Image } from "tensorlake";
const image = new Image({
name: "claude-code-skills",
baseImage: "tensorlake/ubuntu-systemd",
})
.run("apt-get update && apt-get install -y git python3 python3-pip")
.run("git clone https://github.com/tensorlakeai/tensorlake-skills /tmp/tensorlake-skills")
.run(
"mkdir -p /root/.claude/skills/tensorlake && " +
"cp -r /tmp/tensorlake-skills/SKILL.md /tmp/tensorlake-skills/references " +
"/root/.claude/skills/tensorlake/",
)
.run("rm -rf /tmp/tensorlake-skills")
.run("python3 -m pip install --break-system-packages tensorlake");
```
```dockerfile Dockerfile theme={null}
FROM tensorlake/ubuntu-systemd
RUN apt-get update && apt-get install -y git python3 python3-pip
RUN git clone https://github.com/tensorlakeai/tensorlake-skills /tmp/tensorlake-skills
RUN mkdir -p /root/.claude/skills/tensorlake \
&& cp -r /tmp/tensorlake-skills/SKILL.md /tmp/tensorlake-skills/references /root/.claude/skills/tensorlake/
RUN rm -rf /tmp/tensorlake-skills
RUN python3 -m pip install --break-system-packages tensorlake
```
Claude Code scans `~/.claude/skills/` at startup. The `SKILL.md` file and `references/` directory at `/root/.claude/skills/tensorlake/` are auto-discovered.
## Create a Reusable Sandbox Image
Register the image once, then launch new sandboxes with the skills already baked in:
```bash theme={null}
tl sbx image create ./Dockerfile --registered-name claude-code-skills
```
```bash theme={null}
npx tl sbx image create ./Dockerfile --registered-name claude-code-skills
```
```typescript theme={null}
import { createSandboxImage, Image } from "tensorlake";
const image = new Image({
name: "claude-code-skills",
baseImage: "tensorlake/ubuntu-systemd",
})
.run("apt-get update && apt-get install -y nodejs npm python3 python3-pip")
.run("npm install -g skills")
.run("skills add tensorlakeai/tensorlake-skills --all -y --copy")
.run("python3 -m pip install --break-system-packages tensorlake");
await createSandboxImage(image, {
contextDir: ".",
});
```
Then launch sandboxes from that image:
```bash theme={null}
tl sbx create --image claude-code-skills
```
## Use with the SDK
You can also install skills programmatically each time you create a sandbox:
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create()
sandbox.run("bash", ["-c", "apt-get update && apt-get install -y nodejs npm"])
sandbox.run("bash", ["-c", "npm install -g skills"])
sandbox.run("bash", ["-c", "skills add tensorlakeai/tensorlake-skills --all -y --copy"])
result = sandbox.run(
"find",
["/", "-name", "SKILL.md", "-type", "f", "-not", "-path", "*/node_modules/*"],
)
print(result.stdout)
```
```typescript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.create();
try {
await sandbox.run("bash", {
args: ["-lc", "apt-get update && apt-get install -y nodejs npm"],
});
await sandbox.run("bash", {
args: ["-lc", "npm install -g skills"],
});
await sandbox.run("bash", {
args: [
"-lc",
"skills add tensorlakeai/tensorlake-skills --all -y --copy",
],
});
const result = await sandbox.run("find", {
args: [
"/",
"-name",
"SKILL.md",
"-type",
"f",
"-not",
"-path",
"*/node_modules/*",
],
});
console.log(result.stdout);
} finally {
await sandbox.terminate();
client.close();
}
```
For sandboxes you create frequently, use the [sandbox image approach](#create-a-reusable-sandbox-image) to avoid reinstalling skills on every launch.
## What Gets Included
The skill repo contains SDK references that the agent uses as context:
```text theme={null}
tensorlake-skills/
├── AGENTS.md # Skill definition (OpenAI Codex)
├── SKILL.md # Skill definition (Claude Code, Google ADK)
└── references/
├── applications_sdk.md # Orchestrate API reference
├── sandbox_sdk.md # Sandbox API reference
└── integrations.md # Integration patterns
```
## See Also
Learn about TensorLake skills and how to install them for your coding agent.
Create reusable sandbox images from Dockerfiles or the TensorLake image DSLs.
# Snapshot and Clone
Source: https://docs.tensorlake.ai/sandboxes/snapshots
Save, restore, and clone sandbox filesystem, memory, and running processes
Snapshots support two snapshot types:
* `filesystem`: captures filesystem state and restores with a cold boot.
* `memory`: captures filesystem, memory, and running process state and restores with a warm start.
When you do not specify a type, Tensorlake uses `filesystem` by default.
Snapshots are independent of sandbox [lifecycle](/sandboxes/lifecycle): once captured, the artifact persists after the source sandbox is terminated. This means you can snapshot an ephemeral sandbox before it ends, then restore that state into a new sandbox much later. If you only need to pause a single sandbox in place rather than produce a reusable artifact, use [suspend/resume](/sandboxes/lifecycle#suspend-and-resume) instead.
## Creating a Snapshot
```bash theme={null}
tl sbx checkpoint
tl sbx checkpoint --checkpoint-type filesystem
tl sbx checkpoint --checkpoint-type memory
tl sbx checkpoint --timeout 600
```
```python theme={null}
from tensorlake.sandbox import CheckpointType, Sandbox
sandbox = Sandbox.create()
sandbox.run("pip", ["install", "numpy", "pandas", "--user", "--break-system-packages"])
sandbox.run("python", ["-c", "import pandas as pd; pd.DataFrame({'a': [1,2,3]}).to_csv('/data/output.csv')"])
# Default (server-side default, currently `filesystem`).
snapshot = sandbox.checkpoint()
# Explicitly request a memory checkpoint (warm-restore VM memory + processes).
snapshot = sandbox.checkpoint(checkpoint_type=CheckpointType.MEMORY)
# Filesystem-only checkpoint (cold-boot from snapshot tarball).
snapshot = sandbox.checkpoint(checkpoint_type=CheckpointType.FILESYSTEM)
print(snapshot.snapshot_id)
```
```typescript theme={null}
import { Sandbox, type CheckpointType } from "tensorlake";
const sandbox = await Sandbox.create();
await sandbox.run("pip", {
args: [
"install",
"numpy",
"pandas",
"--user",
"--break-system-packages",
],
});
await sandbox.run("python", {
args: [
"-c",
"import pandas as pd; pd.DataFrame({'a': [1,2,3]}).to_csv('/data/output.csv')",
],
});
// Default (server-side default, currently `filesystem`).
let snapshot = await sandbox.checkpoint();
// Explicitly request a memory checkpoint (warm-restore VM memory + processes).
snapshot = await sandbox.checkpoint({ checkpointType: "memory" });
// Filesystem-only checkpoint (cold-boot from snapshot tarball).
snapshot = await sandbox.checkpoint({ checkpointType: "filesystem" });
console.log(snapshot?.snapshotId);
```
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandboxes//snapshot \
-H "Authorization: Bearer $TL_API_KEY"
```
## Restoring from a Snapshot
Create a new sandbox from a snapshot.
If the snapshot is filesystem (default), the new sandbox restores the captured filesystem. You can change sandbox resources (CPU, memory, disk) for the new sandbox.
If the snapshot is memory, the new sandbox restores filesystem, memory, and running processes exactly as they were. Image, resources (CPUs, memory), and entrypoint come from the snapshot and cannot be changed at restore time. If you need different resources, create a fresh sandbox instead of restoring.
For filesystem snapshots, you can pass `--disk_mb` / `resources.disk_mb` at restore time to grow root disk size (growth-only).
```bash theme={null}
tl sbx create --snapshot
```
```python theme={null}
restored = Sandbox.create(snapshot_id=snapshot.snapshot_id)
result = restored.run("cat", ["/data/output.csv"])
print(result.stdout)
```
```typescript theme={null}
const restored = await Sandbox.create({
snapshotId: snapshot.snapshotId,
});
const result = await restored.run("cat", {
args: ["/data/output.csv"],
});
console.log(result.stdout);
```
```bash theme={null}
curl -X POST https://api.tensorlake.ai/sandboxes \
-H "Authorization: Bearer $TL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"snapshot_id": ""}'
```
## Copy a Sandbox
Cloning a sandbox is done with `tl sbx copy`. It boots one or more new sandboxes from a running or suspended source, restoring filesystem, memory, and running processes so each copy warm-starts. Use `-n` to create several copies from the same source in one call.
The source must be running or suspended. A running source is copied directly from the executor hosting it, and a suspended source is copied from the snapshot its suspend already produced, so a copy does not leave a new checkpoint behind for you to clean up.
Copies inherit the source's image, resources, entrypoint, network policy, and exposed ports.
Names are unique per namespace, so copies cannot reuse the source's name. A copy of a named sandbox is named `-copy` by default, which keeps it suspendable and resumable like its source; an unnamed sandbox terminates at its idle timeout instead. Pass `name` to choose the name yourself; with more than one copy it is suffixed `-1`..`-N`.
```bash theme={null}
tl sbx copy
tl sbx copy -n 4
tl sbx copy --timeout 600
```
Not supported in the Python SDK.
Not yet exposed in the TypeScript SDK. Use the CLI or the HTTP API, or call `checkpoint()` followed by `Sandbox.create()` explicitly.
```bash theme={null}
curl -X POST "https://api.tensorlake.ai/sandboxes//copy?times=1" \
-H "Authorization: Bearer $TL_API_KEY"
```
`times` defaults to `1`, and `name` sets the copy name. The request takes no body. See the [API reference](/api-reference/v2/sandboxes/copy) for the full naming rules.
A `200` means every requested copy is running:
```json theme={null}
{
"source_sandbox_id": "",
"sandboxes": [
{ "sandbox_id": "", "status": "running", "sandbox_url": "..." }
]
}
```
A `422` means one or more copies failed before becoming ready, and a `504` means they did not become ready within the timeout. Both return the same body shape, so inspect each entry's `status` to see which copies succeeded.
## Managing Snapshots
### List Snapshots
```bash theme={null}
tl sbx checkpoint ls
```
```python theme={null}
snapshots = sandbox.list_snapshots()
for s in snapshots:
print(
f"{s.snapshot_id} | {s.status.value} | {s.snapshot_type.value if s.snapshot_type else '-'} | {s.size_bytes} bytes"
)
```
```typescript theme={null}
const snapshots = await sandbox.listSnapshots();
for (const snapshot of snapshots) {
console.log(
`${snapshot.snapshotId} | ${snapshot.status} | ${snapshot.snapshotType ?? "-"} | ${snapshot.sizeBytes ?? 0} bytes`,
);
}
```
```bash theme={null}
curl https://api.tensorlake.ai/snapshots \
-H "Authorization: Bearer $TL_API_KEY"
```
### Get Snapshot Details
```typescript theme={null}
const info = await Sandbox.getSnapshot("snapshot-id");
console.log(info.status, info.snapshotType, info.baseImage, info.sizeBytes);
```
```python theme={null}
info = Sandbox.get_snapshot("snapshot_id")
print(info.status, info.snapshot_type)
```
```bash theme={null}
curl https://api.tensorlake.ai/snapshots/ \
-H "Authorization: Bearer $TL_API_KEY"
```
Not supported in the CLI.
### Delete a Snapshot
```bash theme={null}
tl sbx checkpoint rm
```
```typescript theme={null}
await Sandbox.deleteSnapshot("snapshot-id");
```
```python theme={null}
Sandbox.delete_snapshot("snapshot_id")
```
```bash theme={null}
curl -X DELETE https://api.tensorlake.ai/snapshots/ \
-H "Authorization: Bearer $TL_API_KEY"
```
## `checkpoint()` Parameters
| Parameter | Type | Default | Description |
| ----------------- | ------------------------------------------------------------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `sandbox_id` | `str` | — | ID of the running sandbox to snapshot |
| `checkpoint_type` | `CheckpointType` (Python) / `CheckpointType` (TypeScript: `"memory" \| "filesystem"`) | server default (currently `filesystem`) | Checkpoint type. `FILESYSTEM` captures filesystem-only state; `MEMORY` captures filesystem + VM memory + running processes. |
| `timeout` | `float` | `300` | Max seconds to wait for completion |
| `poll_interval` | `float` | `1.0` | Seconds between status polls |
`CheckpointType` is exported from `tensorlake.sandbox` (Python) and `tensorlake` (TypeScript). The TypeScript field on `CheckpointOptions` is `checkpointType`.
## Related Guides
Create, suspend, resume, and terminate sandboxes: the operations snapshots build on.
Build reusable images. Pair with snapshots for warm starts on top of pinned dependencies.
Snapshot a warmed-up `ubuntu-vnc` desktop and fork parallel agent sessions.
Snapshot a Chrome profile so parallel browser agents start with cookies and history already in place.
# Tensorlake Images
Source: https://docs.tensorlake.ai/sandboxes/tensorlake-images
The managed tensorlake/* images, what ships in them, and how they behave when you run them.
Tensorlake provides managed images optimized for sandbox workloads. These images are designed for rapid boot and are globally available in every project. By default, creating a sandbox without specifying an image uses `tensorlake/ubuntu-minimal`.
These images are fully self-contained and ready to launch without additional build or registration steps. Custom image builds and imports follow a separate workflow, detailed in [Build and Import Images](/sandboxes/images).
## Available Images
* `tensorlake/ubuntu-minimal` (*default sandbox image*): Minimal Ubuntu, systemd excluded. Recommended for scenarios requiring the lowest cold start latency.
* `tensorlake/ubuntu-systemd`: Ubuntu with systemd included. Required for workloads needing service management, such as Docker or Kubernetes, within the sandbox.
* `tensorlake/debian-minimal`: Base Debian 13, minimal profile.
In environments where desktop automation is enabled, you may also see:
* `tensorlake/ubuntu-vnc`: Desktop-enabled Ubuntu derived from `tensorlake/ubuntu-systemd`, preinstalled with XFCE, TigerVNC, and Firefox. Intended for browser automation and interactive desktop workloads. See [Computer Use](/sandboxes/computer-use) for more details.
Launch a sandbox from any of them by name:
```bash theme={null}
tl sbx create --image tensorlake/ubuntu-systemd
```
```python theme={null}
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create(image="tensorlake/ubuntu-systemd")
```
```typescript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.create({ image: "tensorlake/ubuntu-systemd" });
```
## Default User and Working Directory
By default, `tensorlake/*` images execute commands as `tl-user` (UID `1000`, home `/home/tl-user`), a non-root user with passwordless sudo. Unlike Standard Docker images, which commonly run as root, this configuration enforces user-level isolation. Tools hardcoded to write to root-owned paths such as `/workspace` will encounter `Permission denied` errors. Note: non-interactive commands start from the filesystem root `/`, so relative commands like `touch output.txt` may fail unless run from a writable directory. Interactive [PTY](/sandboxes/pty-sessions) and SSH sessions default to `/home/tl-user`.
To resolve `Permission denied` errors, ensure commands execute from a directory owned by tl-user (e.g., /home/tl-user), or escalate privileges using sudo or --user root. If files are created as root but need to be accessed by tl-user, adjust ownership with chown:
```bash theme={null}
# Run from a writable working directory
# (working_dir in Python, workingDir in TypeScript)
tl sbx exec --workdir /home/tl-user -- touch output.txt
# Escalate with sudo (works in run() too, which always executes as tl-user)
tl sbx exec -- sudo apt-get update
# Or run the whole command as root
tl sbx exec --user root -- mkdir -p /opt/tools
# Make a system path writable for tl-user
tl sbx exec -- bash -c 'sudo mkdir -p /workspace && sudo chown tl-user:tl-user /workspace'
```
If a workload consistently requires a path such as `/workspace`, include its creation and ownership assignment in a custom image build rather than reconfiguring each new sandbox instance:
```dockerfile theme={null}
FROM tensorlake/ubuntu-minimal
RUN mkdir -p /workspace && chown tl-user:tl-user /workspace
WORKDIR /workspace
```
See [Build and Import Images](/sandboxes/images) for how to build and register a custom image like this one.
Many third-party integrations assume root and default their workdir to `/workspace/`. Point them to `/home/tl-user/`, or use a custom image like the one above. For an example, see the [Crabbox guide](/sandboxes/crabbox).
## Python Packages
Tensorlake Ubuntu and Debian images include a system Python installation managed according to PEP 668. Installing packages with `pip` requires `--break-system-packages` flag, unless a virtual environment is used. Omitting this flag results in the externally-managed-environment error.
For ad hoc installation within a running sandbox:
```python theme={null}
sandbox.run(
"python3",
["-m", "pip", "install", "--break-system-packages", "pandas", "pyarrow", "duckdb"],
)
```
```typescript theme={null}
await sandbox.run("python3", {
args: ["-m", "pip", "install", "--break-system-packages", "pandas", "pyarrow", "duckdb"],
});
```
For repeatable installs, put the packages in `requirements.txt` and install them during a custom image build, as shown in [Build and Register an Image](/sandboxes/images#build-and-register-an-image).
Do not sidestep PEP 668 by switching Python versions. `python3.11 -m pip install ...` or another alternate system Python can produce the same `externally-managed-environment` error. Use `--break-system-packages` with the system `python3`, or create an explicit virtual environment.
## Using Tensorlake Images as Build Bases
Any `tensorlake/*` image can be a `FROM` base (`base_image=` in the SDKs). See [Build and Import Images](/sandboxes/images) for the full workflow.
## See Also
Build your own Docker image or import one from a registry and run it in sandboxes.
Drive the XFCE desktop that ships in `tensorlake/ubuntu-vnc`.
Learn which sandbox settings you can override at launch time.
# Tool Calls
Source: https://docs.tensorlake.ai/sandboxes/tool-calls
Expose Tensorlake sandboxes as tools to your LLM agents, giving models a fresh, isolated execution environment for code, shell, and file operations.
## How it works
The pattern is the same regardless of which LLM you use:
1. **Define a `run_code` tool**: tell the LLM it can call a function that accepts a code string and returns stdout/stderr.
2. **Create a sandbox once** and keep it alive across the agent loop. Reusing one sandbox preserves state between tool calls (installed packages, files written to disk). Each `run_code` call is a fresh Python process, so variables and imports must be redefined in each call.
3. **Execute the tool call**: when the LLM invokes `run_code`, pass the code into `sandbox.run()` and return the result.
4. **Clean up**: terminate the sandbox when the agent session ends.
***
## TypeScript SDK starter
If your agent loop already runs in Node.js, keep one connected sandbox alive for the session and wrap it as a tool:
```typescript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.create({
cpus: 1.0,
memoryMb: 1024,
timeoutSecs: 600,
allowInternetAccess: false,
});
async function runCode(code: string): Promise {
const result = await sandbox.run("python", {
args: ["-c", code],
});
const chunks = [result.stdout.trim()];
if (result.stderr.trim()) chunks.push(`[stderr]\n${result.stderr.trim()}`);
if (result.exitCode !== 0) chunks.push(`[exit code: ${result.exitCode}]`);
return chunks.filter(Boolean).join("\n\n") || "(no output)";
}
try {
const output = await runCode(
"import statistics\nnums = [4, 8, 15, 16, 23, 42]\nprint(statistics.mean(nums))",
);
console.log(output);
} finally {
await sandbox.terminate();
client.close();
}
```
Use this `runCode()` helper as the implementation behind your OpenAI or Anthropic tool/function call.
***
## Claude (Anthropic SDK)
### Prerequisites
```bash theme={null}
pip install tensorlake anthropic
```
### Full example
```python theme={null}
import anthropic
from tensorlake.sandbox import Sandbox
SYSTEM_PROMPT = """You are a data analysis assistant. You have access to a Python sandbox.
Use the run_code tool whenever you need to compute something, analyze data, or verify your
reasoning with code. Each run_code call is a fresh Python process: include all imports and
redefine any variables you need. Installed packages and files written to disk persist across calls."""
# Define the tool schema Claude will use
RUN_CODE_TOOL = {
"name": "run_code",
"description": (
"Execute Python code in a secure sandbox. "
"Each call is a fresh Python process: include all imports and redefine any variables you need. "
"Installed packages and files written to disk persist across calls. "
"Returns stdout and stderr."
),
"input_schema": {
"type": "object",
"properties": {
"code": {
"type": "string",
"description": "Python code to execute.",
}
},
"required": ["code"],
},
}
def run_agent(user_message: str) -> str:
anthropic_client = anthropic.Anthropic()
# Create one sandbox for the entire agent session
sandbox = Sandbox.create(
cpus=1.0,
memory_mb=1024,
timeout_secs=600,
allow_internet_access=False, # lock down network for untrusted code
)
messages = [{"role": "user", "content": user_message}]
try:
while True:
response = anthropic_client.messages.create(
model="claude-opus-4-5",
max_tokens=4096,
system=SYSTEM_PROMPT,
tools=[RUN_CODE_TOOL],
messages=messages,
)
# Append assistant's response to history
messages.append({"role": "assistant", "content": response.content})
# If no tool use, we're done
if response.stop_reason == "end_turn":
# Extract the final text response
for block in response.content:
if hasattr(block, "text"):
return block.text
# Process all tool calls in this response
tool_results = []
for block in response.content:
if block.type != "tool_use":
continue
code = block.input["code"]
print(f"\n[sandbox] executing:\n{code}\n")
result = sandbox.run("python", ["-c", code])
output = result.stdout or ""
if result.stderr:
output += f"\n[stderr]\n{result.stderr}"
if result.exit_code != 0:
output += f"\n[exit code: {result.exit_code}]"
print(f"[sandbox] output:\n{output}")
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": output or "(no output)",
})
# Feed all results back to Claude in one message
messages.append({"role": "user", "content": tool_results})
finally:
sandbox.close() # always clean up
if __name__ == "__main__":
answer = run_agent(
"I have a list of numbers: [4, 8, 15, 16, 23, 42]. "
"What is the mean, median, and standard deviation? "
"Also plot a histogram and tell me if the distribution looks normal."
)
print("\n=== Final answer ===")
print(answer)
```
### What happens step by step
| Step | What Claude does | What your code does |
| ---- | --------------------------------------------------------------- | ------------------------------------------------------ |
| 1 | Reads the user question | Sends to Claude with `run_code` tool available |
| 2 | Decides it needs to compute something, emits a `tool_use` block | Detects `stop_reason == "tool_use"` |
| 3 | — | Calls `sandbox.run()` with the generated code |
| 4 | — | Appends result as `tool_result` and calls Claude again |
| 5 | Reads the output, continues reasoning or calls tool again | Loops until `stop_reason == "end_turn"` |
| 6 | Returns final text answer | Returns it to the caller, closes sandbox |
***
## OpenAI (function calling)
### Prerequisites
```bash theme={null}
pip install tensorlake openai
```
### Full example
```python theme={null}
import json
import openai
from tensorlake.sandbox import Sandbox
SYSTEM_PROMPT = """You are a data analysis assistant with access to a Python sandbox.
Always use the run_code function to execute code; never compute or guess answers yourself.
Each run_code call is a fresh Python process: include all imports and redefine any variables
you need. Installed packages and files written to disk are available across calls."""
# Define the function schema OpenAI will use
RUN_CODE_FUNCTION = {
"type": "function",
"function": {
"name": "run_code",
"description": (
"Execute Python code in a secure isolated sandbox. "
"Each call runs in a fresh Python process: include all imports and redefine "
"any variables you need. Installed packages and files written to disk persist across calls. "
"Returns stdout and stderr as a string."
),
"parameters": {
"type": "object",
"properties": {
"code": {
"type": "string",
"description": "Python code to execute.",
}
},
"required": ["code"],
},
},
}
def run_agent(user_message: str) -> str:
openai_client = openai.OpenAI()
# Create one sandbox for the entire agent session
sandbox = Sandbox.create(
cpus=1.0,
memory_mb=1024,
timeout_secs=600,
allow_internet_access=False,
)
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
]
try:
while True:
response = openai_client.chat.completions.create(
model="gpt-5.1",
messages=messages,
tools=[RUN_CODE_FUNCTION],
tool_choice="auto",
)
msg = response.choices[0].message
messages.append(msg)
# No tool calls, agent is done
if not msg.tool_calls:
return msg.content
# Process all tool calls
for tool_call in msg.tool_calls:
args = json.loads(tool_call.function.arguments)
code = args["code"]
print(f"\n[sandbox] executing:\n{code}\n")
result = sandbox.run("python", ["-c", code])
output = result.stdout or ""
if result.stderr:
output += f"\n[stderr]\n{result.stderr}"
if result.exit_code != 0:
output += f"\n[exit code: {result.exit_code}]"
print(f"[sandbox] output:\n{output}")
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": output or "(no output)",
})
finally:
sandbox.close()
if __name__ == "__main__":
answer = run_agent(
"Using only Python stdlib (random, datetime), generate sample monthly revenue and cost "
"data for the last 12 months (seed 42). Print a table showing each month, profit, and "
"profit margin. Then print which month had the best and worst margin."
)
print("\n=== Final answer ===")
print(answer)
```
***
## Using OpenAI Agents SDK
If you are using the newer [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/), you can wrap the sandbox as a `FunctionTool` directly:
### Prerequisites
```bash theme={null}
pip install tensorlake openai-agents
```
### Full example
```python theme={null}
from agents import Agent, ModelSettings, Runner, function_tool
from tensorlake.sandbox import Sandbox
# Keep one sandbox alive for the agent's lifetime
_sandbox = Sandbox.create(
cpus=1.0,
memory_mb=1024,
timeout_secs=600,
allow_internet_access=False,
)
@function_tool
def run_code(code: str) -> str:
"""Execute Python code in a secure sandbox. Each call is a fresh Python process:
include all imports and redefine any variables you need. Installed packages and
files written to disk persist across calls."""
result = _sandbox.run("python", ["-c", code])
output = result.stdout or ""
if result.stderr:
output += f"\n[stderr]\n{result.stderr}"
if result.exit_code != 0:
output += f"\n[exit code: {result.exit_code}]"
return output or "(no output)"
agent = Agent(
name="Data Analyst",
instructions="You are a data analysis assistant. Always use run_code to compute answers; never calculate or guess yourself.",
tools=[run_code],
model_settings=ModelSettings(tool_choice="required"),
)
result = Runner.run_sync(
agent,
"Write and run Python code to calculate the compound annual growth rate "
"if revenue grew from $1M to $3.2M over 5 years. Print the result."
)
print(result.final_output)
_sandbox.close()
```
***
## Production tips
### Reuse one sandbox per session, not per call
Creating a new sandbox on every tool call adds cold-start latency and loses any state (installed packages, files written to disk) from prior calls. Create the sandbox before the agent loop and close it afterward.
```python theme={null}
# ✅ Create once, reuse across all tool calls
sandbox = Sandbox.create(...)
try:
run_agent_loop(sandbox)
finally:
sandbox.close()
# ❌ Don't do this: loses state and adds latency every call
def run_code_tool(code):
sandbox = Sandbox.create() # new sandbox every call
return sandbox.run("python", ["-c", code])
```
### Pre-install dependencies with Snapshots
If your agent always needs the same libraries (pandas, numpy, matplotlib, etc.), install them once, snapshot the sandbox, and boot future sandboxes from that snapshot. This avoids re-running `pip install` on every session.
```python theme={null}
# One-time setup: build a snapshot with dependencies pre-installed
setup_sandbox = Sandbox.create()
setup_sandbox.run("pip", ["install", "pandas", "numpy", "matplotlib", "scipy"])
snapshot = sandbox.checkpoint()
setup_sandbox.close()
print(f"Snapshot ready: {snapshot.snapshot_id}")
# Every future session starts with packages already installed
sandbox = Sandbox.create(snapshot_id=snapshot.snapshot_id)
```
See the [Snapshots guide](/sandboxes/snapshots) for details.
### Let the agent install packages on demand
If your agent may need arbitrary or unknown packages, tell it in the system prompt that it can install them with pip. Because sandbox state persists across tool calls, a package installed in one call is available in all subsequent calls.
```python theme={null}
SYSTEM_PROMPT = """You are a data analysis assistant. You have access to a Python sandbox.
Use the run_code tool whenever you need to compute something, analyze data, or verify your
reasoning with code. Each run_code call is a fresh Python process: include all imports and
redefine any variables you need. Installed packages and files written to disk persist across calls.
If a required package is missing, install it before using it:
import subprocess; subprocess.run(["pip", "install", "--break-system-packages", ""], check=True)"""
```
Use this approach when dependencies are unpredictable. For a known set of dependencies, pre-installing via [Snapshots](#pre-install-dependencies-with-snapshots) is faster since it avoids repeating `pip install` on every session.
### Lock down the network
By default, sandboxes have internet access. For agents executing untrusted or LLM-generated code, disable it:
```python theme={null}
sandbox = Sandbox.create(allow_internet_access=False)
```
If the agent needs outbound access, keep internet enabled or use `deny_out` to block destinations you know should be unreachable. See the [Networking guide](/sandboxes/networking).
***
## What to build next
* **[Data Analysis](/sandboxes/data-analysis)**: spin up sandboxes with data science libraries to analyze complex datasets and stream results back in real time.
* **[Snapshots](/sandboxes/snapshots)**: pre-install dependencies so agent sessions start instantly.
# Local Tunnels
Source: https://docs.tensorlake.ai/sandboxes/tunnels
Forward a local TCP port to a port inside a sandbox over an authenticated WebSocket.
Tunnels give your machine a `localhost:` that maps directly to a port inside a running sandbox. The relay travels over a WebSocket through the sandbox proxy, so your TensorLake credentials authenticate every connection. You do **not** need to add the port to `exposed_ports` or make it public.
**Reach for a tunnel when you need a raw TCP connection into a sandbox.** The sandbox proxy at `*.sandbox.tensorlake.ai` only speaks HTTP, WebSocket, gRPC, and SSH. Anything else (VNC's RFB protocol, the Postgres wire protocol, MySQL, Redis's RESP, MongoDB, custom binary protocols) needs a tunnel because the proxy cannot frame those bytes for you.
You can also use a tunnel for HTTP/WS/gRPC traffic when you would rather keep the port private to your laptop than expose it through the public sandbox URL. Driving Chrome's DevTools Protocol from your laptop is a typical case: CDP is WebSocket, so the proxy could carry it, but a tunnel keeps the debugger reachable only at `127.0.0.1` and skips the per-port `exposed_ports` configuration.
Tunnels and exposed ports are independent. A tunnel works even when the port is not in `exposed_ports`.
## Open a Tunnel
The simplest way is the CLI. Pick any local port (defaults to the same number as the remote port) and leave the command running.
```bash theme={null}
tl sbx tunnel 5901 --listen-port 15901
```
The command keeps running and prints connection events. Press `Ctrl+C` to stop the tunnel; the sandbox keeps running.
Without `--listen-port`, the local port matches the remote port:
```bash theme={null}
tl sbx tunnel 9222
```
```javascript theme={null}
import { Sandbox } from "tensorlake";
const sandbox = await Sandbox.connect({ sandboxId: "" });
const tunnel = await sandbox.createTunnel(5901, { localPort: 15901 });
const { host, port } = tunnel.address();
console.log(`tunnel listening on ${host}:${port}`);
// ... use it ...
await tunnel.close();
```
`createTunnel(remotePort, options)` returns a `TcpTunnel`. Useful options:
* `localHost`: bind interface (defaults to `127.0.0.1`).
* `localPort`: local port number; pass `0` for an ephemeral port and read it back from `tunnel.address()`.
* `connectTimeout`: seconds to wait for each WebSocket connection (defaults to `10`).
The Python SDK does not yet ship a native tunnel helper. Drive the CLI from a subprocess:
```python theme={null}
import subprocess
tunnel = subprocess.Popen(
["tl", "sbx", "tunnel", "", "9222", "-l", "9222"],
)
try:
# Use http://127.0.0.1:9222 from your code.
...
finally:
tunnel.terminate()
tunnel.wait()
```
The local listener is per-process. If you want two clients to share one tunnel, run the CLI once and connect both clients to the same `localhost:`.
## How It Works
The CLI and the TypeScript SDK both speak the same protocol:
1. A WebSocket is opened to the sandbox proxy, carrying your API key, PAT, or session cookie.
2. The proxy authorizes the request, finds the dataplane that owns the sandbox, and pipes bytes to `127.0.0.1:` inside the sandbox.
3. The local TCP listener accepts a connection from your client and relays bytes both ways across the WebSocket.
Because every byte rides on an authenticated WebSocket, the remote port stays private to your account: there is no public hostname for it.
## Common Patterns
| Inside the sandbox | Local port | Client |
| --------------------------------- | ---------- | ------------------------------------------------------------------- |
| `5901` (TigerVNC) | `15901` | macOS Screen Sharing, RealVNC, TigerVNC, Remmina |
| `9222` (Chrome DevTools Protocol) | `9222` | Playwright `connect_over_cdp`, Puppeteer, `chrome-remote-interface` |
| `5432` (Postgres) | `5432` | `psql`, DBeaver, TablePlus |
| `3000` (dev server) | `3000` | Browser at `http://localhost:3000` |
Tunneling is also the easiest way to reach the sandbox's authenticated [Computer Use](/sandboxes/computer-use) VNC port from a desktop client without polling screenshots, or to point Playwright at sandboxed Chrome. See [Drive Chrome over CDP](/sandboxes/chrome-cdp) for the full walkthrough.
## Troubleshooting
* **`Connection refused` from the local end.** The remote service inside the sandbox is not listening on the port yet. Tail its logs (`tl sbx exec -- bash -lc 'ss -ltnp'`) and retry.
* **`502 Bad Gateway` during handshake.** The sandbox has not finished booting the workload. Wait a few seconds and reconnect; the proxy returns 502 when nothing is listening on the remote port.
* **WebSocket auth failures.** Confirm `tl whoami` shows the right organization and project, or that `TENSORLAKE_API_KEY` is set in the shell that runs the CLI.