> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensorlake.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Harbor
> Run Harbor evaluations and RL rollouts on Tensorlake Sandboxes — fresh isolation per trial, pre-warmed snapshots for expensive environments, and independent test verification.
[Harbor](https://github.com/harbor-framework/harbor) is a framework from the creators of [Terminal-Bench](https://www.tbench.ai/) for evaluating and optimizing agents and language models. With Harbor you can evaluate arbitrary agents (Claude Code, OpenHands, Codex CLI, and others) against curated datasets like Terminal-Bench, SWE-Bench, and Aider Polyglot, build and share your own benchmarks, run thousands of trials in parallel across cloud providers, and generate rollouts for RL optimization.
Harbor abstracts the execution backend behind an `--env` flag. Tensorlake plugs in as one of those providers — alongside other sandboxes and local Docker — so the same Harbor commands run on Tensorlake sandboxes without changing your tasks, agents, or evaluators.
This guide focuses on running CLI-agent evaluations against benchmarks like Terminal-Bench. Harbor also supports generating rollouts for RL optimization — we'll cover those workflows in follow-up guides.
New to Tensorlake? Sign up at the [dashboard](https://cloud.tensorlake.ai) — new accounts include free credits, enough to run a full Terminal-Bench sweep before you pay for anything.
## Quick start
Grab one from the [Tensorlake Dashboard](https://cloud.tensorlake.ai). You'll also need an API key for whichever agent provider you want to evaluate (e.g., Anthropic).
The `harbor[tensorlake]` extra installs the `TensorLakeEnvironment` provider alongside Harbor.
```bash theme={null}
uv pip install "harbor[tensorlake]"
```
```bash theme={null}
pip install "harbor[tensorlake]"
```
```bash theme={null}
export TENSORLAKE_API_KEY="tl_..."
export ANTHROPIC_API_KEY="sk-ant-..." # or another agent provider
```
Run a single Terminal-Bench task on Tensorlake with Claude Code as the agent:
```bash theme={null}
harbor run --env tensorlake \
--include-task-name terminal-bench/pytorch-model-cli \
--dataset terminal-bench/terminal-bench-2-1 \
--agent claude-code \
--model anthropic/claude-sonnet-4-6 \
--ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY
```
Drop `--include-task-name` to run the full Terminal-Bench 2.1 suite. `--ae KEY=VALUE` forwards an environment variable from your shell into the sandbox where the agent runs — add more `--ae` flags for any other secrets the agent needs.
## Why Tensorlake for Harbor
Harbor's value comes from running large fleets of environments in parallel and trusting the results. Tensorlake's runtime is designed for exactly that workload:
* **Per-trial sandboxes** — each task starts on a clean machine and is destroyed at the end. No shared kernel state between trials, which matters for both eval reproducibility and RL reward integrity.
* **Full task-environment support** — Tensorlake imports a task's real Docker image and converts it into a sandbox image that boots directly, so every trial runs the exact environment the benchmark defines rather than one approximated by replaying a Dockerfile. That closes the environment gap that otherwise quietly skews results.
* **Pre-warmed snapshots** — environments with heavy `apt`/`pip` installs (PyTorch, CUDA toolchains, full Linux desktops) can be built once, snapshotted, and restored under a second for every subsequent trial or rollout.
* **Independent verification** — Harbor's test script runs inside the sandbox and writes `1.0` or `0.0` to `reward.txt`. The agent never sees or touches the verifier, so "the agent said it worked" is never confused with "the tests pass."
* **Parallel scale** — Tensorlake schedules thousands of sandboxes concurrently, which is what RL rollout generation and full benchmark sweeps need.
## Anatomy of a Harbor task
Harbor expects each task to be laid out like this - take [gcode-to-text](https://github.com/harbor-framework/terminal-bench-2/tree/main/gcode-to-text) as an example:
```
gcode-to-text
├── environment
│ ├── Dockerfile
│ └── text.gcode.gz
├── instruction.md
├── solution
│ └── solve.sh
├── task.toml
└── tests
├── test_outputs.py
└── test.sh
```
* `environment/Dockerfile` defines the base image and any setup steps.
* `instruction.md` is the prompt the agent receives.
* `solution/` is an oracle reference used to validate the environment itself.
* `tests/test.sh` runs after the agent finishes and produces `reward.txt`.
## Tune sandbox resources
Each task's `task.toml` controls the sandbox Harbor provisions on Tensorlake. Set resources in the `[environment]` block:
```toml task.toml theme={null}
[environment]
cpus = 2
memory_mb = 4096
storage_mb = 20480
allow_internet = true
```
| Field | Default | Forwarded to Tensorlake |
| ---------------- | ------- | ----------------------- |
| `cpus` | `1` | `cpus` |
| `memory_mb` | `2048` | `memory_mb` |
| `storage_mb` | `10240` | `ephemeral_disk_mb` |
| `allow_internet` | `true` | `allow_internet_access` |
Tensorlake requires `memory_mb` to be between 1024 and 8192 MB per CPU core.
A few rules of thumb:
* **Large or heavy images** — if your `environment/Dockerfile` pulls in big toolchains (PyTorch, CUDA, full Linux desktops, large datasets), bump `cpus` and `memory_mb` so the build and runtime have headroom, and raise `storage_mb` past the image size plus working-set room. Underprovisioned sandboxes show up as build timeouts or OOMs mid-trial.
* **Lock down `allow_internet`** — set `allow_internet = false` to stop the agent from searching the web for answers. If the verifier needs network access, bake those dependencies into the Dockerfile. Per-host allowlists are coming soon, so you'll be able to block search engines while leaving package mirrors reachable.
## Image build & caching
Each trial boots from an image. Harbor uses a prebuilt image when the task declares one, and otherwise builds the task's Dockerfile:
| Source | How to set it | When to use |
| ------------------ | ----------------------------- | -------------------------------------------------------------------------------------------------------------- |
| **Prebuilt image** | `docker_image` in `task.toml` | Fastest start — boot directly from an image with the environment already baked in. Terminal-Bench ships these. |
| **Dockerfile** | `environment/Dockerfile` | No prebuilt image declared. Built once, cached, and reused across trials. |
Either way the image is built or imported **once** and reused — you only pay the cost on the first trial, then every later trial boots directly from the cached image. If a task sets both, the prebuilt image wins over the Dockerfile if it's exist.
Reusing one heavy environment across many runs — RL rollouts or repeated eval of the same task — can restore in under a second from a pre-warmed snapshot instead of rebuilding. See [Snapshots](/sandboxes/snapshots).
### Prebuilt image
If a task declares a `docker_image` in `task.toml`, Harbor boots directly from that image and skips the Dockerfile entirely:
```toml task.toml theme={null}
[environment]
docker_image = "myorg/my-task-env:2025-06"
```
Harbor looks the image up in Tensorlake by name and boots from it; if it isn't registered yet, it imports the image once and reuses it on every later trial. The registered name is derived from the reference string you put here, so the **first** import of a given reference is what every later run boots.
**Always publish with an immutable tag — never `latest`.** Because the registered name comes from the reference string (not the image contents), a tag like `latest` is captured at its first import and then frozen: if you push new content to `latest`, Harbor keeps booting the old image and never re-pulls. Use an immutable tag or digest (e.g. `myorg/my-task-env:2025-06` or `...@sha256:...`) so a new build means a new reference — which is what triggers a fresh import. This is the convention the published Terminal-Bench images follow.
To refresh an image that's already registered, point `docker_image` at a new immutable tag/digest, or delete the registered Tensorlake image so the next run re-imports it. (`--force-build` does **not** re-import — it builds from the Dockerfile instead.)
**Terminal-Bench 2.1 images are already published.** We've registered every Terminal-Bench 2.1 task image publicly, so anyone with Tensorlake access boots straight from them — no build, no import. Just run the dataset as usual and each task picks up its published image.
**Set your org and project context.** Looking an image up by name requires organization and project context. Without it, Harbor can't find the published image and falls back to importing it fresh — you'll see a line like this in the logs:
```
importing docker_image alexgshaw/pypi-server:20251031 directly: Looking up a sandbox image by name requires organization and project context (TENSORLAKE_ORGANIZATION_ID and TENSORLAKE_PROJECT_ID).
```
Set both before running so the lookup resolves and the boot stays instant:
```bash theme={null}
export TENSORLAKE_ORGANIZATION_ID="..."
export TENSORLAKE_PROJECT_ID="..."
```
Harbor also reads these from `~/.tensorlake/config.toml` (`organization` and `project`) if it's present.
### Dockerfile
If a task has no `docker_image`, Harbor builds its `environment/Dockerfile` once via Tensorlake's image builder, caches it, and boots every later trial directly from the cached image — no per-trial `apt`/`pip` work. The cache is keyed on the Dockerfile **and every file in the build context**, so editing a `requirements.txt` pin or any `COPY`'d file automatically triggers a rebuild.
```bash theme={null}
harbor run --env tensorlake \
--dataset terminal-bench/terminal-bench-2-1 \
--agent claude-code \
--model anthropic/claude-sonnet-4-6
```
If a build ever fails, Harbor automatically falls back to replaying the Dockerfile's `RUN`/`COPY` steps on each trial, so a trial is never blocked — it just runs a little slower. The fallback is also available as an explicit escape hatch while you iterate on a Dockerfile:
```bash theme={null}
harbor run --env tensorlake ... --ek use_oci_image_build=false
```
**Dockerfile requirements**
The image builder is stricter than a local `docker build`, so a few Docker conventions need small adjustments:
* **`COPY` does not auto-create parent directories** — `COPY x /a/b/c` fails if `/a/b` doesn't exist yet. Add `RUN mkdir -p /a/b` before the `COPY`.
* **Don't pin exact apt versions** (`apt-get install curl=8.5.0-2ubuntu10.6`) — drop the pin or pick a version that exists in the target distro.
* **Use a FROM image that ships the Python you need** (e.g. `python:3.10-bookworm`) rather than relying on a non-native version being fetched at build time.
To force a fresh rebuild even when a valid cached image exists, add `--force-build`. This applies to that run only and doesn't disturb the cache used by subsequent normal runs.
```bash theme={null}
harbor run --env tensorlake \
--force-build \
--dataset terminal-bench/terminal-bench-2-1 \
--agent claude-code \
--model anthropic/claude-sonnet-4-6
```
### Sharing images publicly
By default, images Harbor builds or imports are **private to your organization** — only your org can boot from them. Add `--ek is_public=true` to register a freshly built or imported image as public, so any organization with Tensorlake access can reuse it:
```bash theme={null}
harbor run --env tensorlake \
--ek is_public=true \
--dataset terminal-bench/terminal-bench-2-1 \
--agent claude-code \
--model anthropic/claude-sonnet-4-6
```
The flag applies to both Dockerfile-built images and prebuilt `docker_image` imports. Automatic boot-from-public by another organization is wired through the **prebuilt `docker_image`** path — that's how the published Terminal-Bench images are reused — so if your goal is to publish an environment others boot directly, prefer a `docker_image` reference.
Publishing public images is gated to an allow list. If your account isn't on it, the flag is ignored and the image stays private. Reach out to Tensorlake to be added.
`is_public` only takes effect when the image is **newly registered**. If an image with the same name already exists — a private copy from an earlier run, or an existing public one — Harbor boots it as-is and won't republish it. To turn an already-private image public, delete it first (or change the build context so it gets a new name), then rerun with `--ek is_public=true`.
### Ad-hoc native dependencies
If a task just needs a couple of extra apt packages and you don't want to edit the Dockerfile or maintain a snapshot, use `preinstall_packages`:
```bash theme={null}
harbor run --env tensorlake \
--ek 'preinstall_packages=["build-essential","rustc","cargo"]' \
--dataset terminal-bench/terminal-bench-2-1 \
--agent claude-code \
--model anthropic/claude-sonnet-4-6
```
The packages are installed at the start of each trial. Prefer snapshots when the package set is large or reused across many runs so you pay the install cost once.
## Interactive debugging
When a trial fails and you want to poke around the live environment, attach to the session:
```bash theme={null}
harbor env attach
```
Drop directly into the running sandbox to inspect state, rerun tests by hand, and confirm whether the failure was the agent or the environment.
## Structured logs
Each trial produces structured artifacts, e.g.:
```
gcode-to-text__UFALMLv
├── agent/
├── verifier/
├── result.json
└── trial.log
```
So you can trace:
* The agent's actions and outputs
* What the verifier checked
* Why the trial passed or failed
## What to build next
Build an environment once, snapshot it, and restore in seconds for every trial.
Use sandboxes as a deterministic reward oracle for RL training loops.