> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensorlake.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Harbor Evaluations on Tensorlake Sandboxes

> Run Harbor evaluations and RL rollouts on Tensorlake Sandboxes: fresh isolation per trial, pre-warmed snapshots for expensive environments, and independent test verification.

[Harbor](https://github.com/harbor-framework/harbor) is a framework from the creators of [Terminal-Bench](https://www.tbench.ai/) for evaluating and optimizing agents and language models. With Harbor you can:

* Evaluate any agent, such as Claude Code, OpenHands, or Codex CLI, against curated datasets like Terminal-Bench, SWE-Bench, and Aider Polyglot
* Build and share your own benchmarks
* Run thousands of trials in parallel across cloud providers
* Generate rollouts for reinforcement learning (RL) optimization

Harbor abstracts the execution backend behind an `--env` flag. Tensorlake is one of those providers, alongside other sandbox vendors and local Docker. The same Harbor commands run on Tensorlake sandboxes without changes to your tasks, agents, or evaluators.

<Note>
  This guide covers command-line interface (CLI) agent evaluations against benchmarks like Terminal-Bench. Harbor also generates rollouts for RL optimization. We'll cover those workflows in follow-up guides.
</Note>

<Tip>
  If you're new to Tensorlake, sign up at [Tensorlake Cloud](https://cloud.tensorlake.ai). New accounts include free credits, enough to run a full Terminal-Bench sweep before you pay for anything.
</Tip>

## Quickstart

Install Harbor with the Tensorlake extra, set 2 API keys, and run 1 Terminal-Bench task.

<Steps>
  <Step title="Get a Tensorlake API key">
    Create one in the [Tensorlake Dashboard](https://cloud.tensorlake.ai). You also need an API key for the agent provider you want to evaluate, such as Anthropic.
  </Step>

  <Step title="Install Harbor with the Tensorlake provider">
    The `harbor[tensorlake]` extra installs the `TensorLakeEnvironment` provider alongside Harbor.

    <Tabs>
      <Tab title="uv">
        ```bash theme={null}
        uv pip install "harbor[tensorlake]"
        ```
      </Tab>

      <Tab title="pip">
        ```bash theme={null}
        pip install "harbor[tensorlake]"
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Set your environment variables">
    ```bash theme={null}
    export TENSORLAKE_API_KEY="your_tensorlake_api_key_here"
    export ANTHROPIC_API_KEY="your_anthropic_api_key_here"   # or another agent provider
    ```
  </Step>

  <Step title="Run a Terminal-Bench task">
    Run a single Terminal-Bench task on Tensorlake with Claude Code as the agent:

    ```bash theme={null}
    harbor run --env tensorlake \
      --include-task-name terminal-bench/pytorch-model-cli \
      --dataset terminal-bench/terminal-bench-2-1 \
      --agent claude-code \
      --model anthropic/claude-sonnet-4-6 \
      --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY
    ```

    Remove `--include-task-name` to run the full Terminal-Bench 2.1 suite. `--ae` forwards an environment variable from your shell into the sandbox where the agent runs. Add more `--ae` flags for any other secrets the agent needs.
  </Step>
</Steps>

## Why Tensorlake for Harbor

Harbor's value comes from running large fleets of environments in parallel and trusting the results. Tensorlake's runtime targets that workload:

* **Per-trial sandboxes**: Each task starts on a clean sandbox, which Tensorlake destroys at the end. Trials share no kernel state, which matters for both eval reproducibility and RL reward integrity.
* **Full task-environment support**: Tensorlake imports a task's real Docker image and converts it into a sandbox image that boots directly. Every trial runs the exact environment the benchmark defines, not an approximation built by replaying a Dockerfile. That closes the environment gap that otherwise skews results.
* **Pre-warmed snapshots**: Build environments with heavy `apt` or `pip` installs, such as PyTorch, CUDA toolchains, or full Linux desktops, once. Snapshot them, then restore the snapshot for every later trial or rollout.
* **Independent verification**: Harbor's test script runs inside the sandbox and writes `1.0` or `0.0` to `reward.txt`. The agent never sees or touches the verifier, so you never confuse “the agent said it worked” with “the tests pass”.
* **Parallel scale**: Tensorlake schedules thousands of sandboxes concurrently, which is what RL rollout generation and full benchmark sweeps need.

## Anatomy of a Harbor task

Harbor expects each task to use the layout below. [gcode-to-text](https://github.com/harbor-framework/terminal-bench-2/tree/main/gcode-to-text) is an example:

```text theme={null}
gcode-to-text
├── environment
│   ├── Dockerfile
│   └── text.gcode.gz
├── instruction.md
├── solution
│   └── solve.sh
├── task.toml
└── tests
    ├── test_outputs.py
    └── test.sh
```

* `environment/Dockerfile` defines the base image and any setup steps.
* `instruction.md` is the prompt the agent receives.
* `solution/` is a reference solution that validates the environment itself.
* `tests/test.sh` runs after the agent finishes and produces `reward.txt`.

## Tune sandbox resources

Each task's `task.toml` controls the sandbox Harbor provisions on Tensorlake. Set resources in the `[environment]` block:

```toml task.toml theme={null}
[environment]
cpus = 2
memory_mb = 4096
storage_mb = 20480
network_mode = "public"
```

| Field | Harbor default | Tensorlake minimum | Forwarded to Tensorlake |
| - | - | - | - |
| `cpus` | `1` | `2` | `cpus` |
| `memory_mb` | `2048` | `4096` | `memory_mb` |
| `storage_mb` | `10240` | `30720` | `disk_mb` |
| `network_mode` | `public` | — | `allow_internet_access` / `allow_out` |

If a value is below the Tensorlake minimum, Harbor raises it to the minimum. Harbor never lowers a value. Many benchmark tasks ask for 1 CPU and 4 GB, which causes out-of-memory (OOM) failures on sandbox hosts. A sandbox that you restore from a snapshot keeps the disk size of the snapshot.

<Note>
  Tensorlake requires between 1024 MB and 8192 MB of `memory_mb` per CPU core.
</Note>

<Note>
  Harbor deprecated `allow_internet` in favor of `network_mode`. Harbor still accepts it and maps `true` and `false` to `network_mode = "public"` and `"no-network"`. Set `network_mode` directly in new tasks.
</Note>

Two sizing guidelines:

* **Large or heavy images**: If your `environment/Dockerfile` pulls in big toolchains, such as PyTorch, CUDA, full Linux desktops, or large datasets, raise `cpus` and `memory_mb` so the build and runtime have headroom. Raise `storage_mb` past the image size plus working-set room. Underprovisioned sandboxes fail with build timeouts or out-of-memory (OOM) errors mid-trial.
* **Lock down `network_mode`**: Set `network_mode = "no-network"` to stop the agent from searching the web for answers. Or set `"allowlist"` with `allowed_hosts` to permit only specific destinations.

### Dynamic network policy

Different phases of a trial need different network access. Setup may need the open internet to install dependencies. The agent and verifier need tighter limits so the agent can't search the web for answers and nobody can tamper with the verifier's result. Tensorlake sandboxes switch the network policy on a running sandbox, so Harbor can scope `network_mode` per phase in `task.toml`:

```toml task.toml theme={null}
[environment]
network_mode = "public"

[agent] # applies during agent.run(), not during agent.setup()
network_mode = "allowlist"
allowed_hosts = ["pypi.org", "api.openai.com"]

[verifier] # applies during verifier.verify()
network_mode = "no-network"
```

Tensorlake replaces the running sandbox's firewall rules for each phase. The sandbox does not restart between phases. The new rules replace the old rules fully, so a narrower policy never keeps an allowance from the previous one.

The switch is fail-safe, but not instant:

* **The old policy stays active for a few seconds.** Tensorlake resolves each hostname in the new policy before it swaps the rules. A tighter policy usually takes effect in about 5 seconds, and a slow hostname can extend this to about 10 seconds. The sandbox is never unprotected during the swap.
* **Open connections continue.** The firewall is stateful. A connection that opened under the wider policy stays open after the swap.
* **A hostname that does not resolve stops the switch.** Tensorlake rejects the update, the previous policy stays in effect, and the phase fails.

`allowed_hosts` accepts exact hostnames, leading-wildcard hostnames such as `*.example.com`, IPv4 addresses, and IPv4 CIDR ranges. Tensorlake does not support IPv6 entries.

<Note>
  Network policies, including `allowlist` and per-phase overrides, also apply to [Docker Compose tasks](#docker-compose-multi-container-tasks). Tensorlake enforces the policy outside the sandbox VM, so one policy covers all compose containers.
</Note>

<Tip>
  [Dynamic Network Policies in Harbor](https://www.tensorlake.ai/blog/dynamic-network-policies-in-harbor) explains why phase-scoped policies matter. Harbor's [Network Policy](https://docs.harborframework.com/core-concepts/tasks/network-policies) docs have the full field reference.
</Tip>

## Image build and caching

Each trial boots from an image. Harbor uses a prebuilt image when the task declares one, and otherwise builds the task's Dockerfile:

| Source | How to set it | When to use |
| - | - | - |
| **Prebuilt image** | `docker_image` in `task.toml` | Fastest start: boot directly from an image with the environment already baked in. Terminal-Bench ships these. |
| **Dockerfile** | `environment/Dockerfile` | No prebuilt image declared. Built once, cached, and reused across trials. |

Either way, Harbor builds or imports the image once and reuses it. You pay the cost on the first trial only. Every later trial boots directly from the cached image. If a task sets both and the prebuilt image exists, Harbor uses the prebuilt image.

<Tip>
  When you reuse one heavy environment across repeated runs, such as RL rollouts or evals of the same task, restore it from a pre-warmed snapshot instead of rebuilding. See [Snapshots](/sandboxes/snapshots).
</Tip>

### Prebuilt image

If a task declares a `docker_image` in `task.toml`, Harbor boots directly from that image and skips the Dockerfile:

```toml task.toml theme={null}
[environment]
docker_image = "myorg/my-task-env:2025-06"
```

Harbor looks the image up in Tensorlake by name and boots from it. If the image isn't registered yet, Harbor imports it once and reuses it on every later trial. Harbor derives the registered name from the reference string, so every later run boots the first import of that reference.

<Warning>
  **Always publish with an immutable tag, never `latest`.** The registered name comes from the reference string, not the image contents. Harbor captures a tag like `latest` at its first import and freezes it. If you push new content to `latest`, Harbor keeps booting the old image and never re-pulls. Use an immutable tag such as `myorg/my-task-env:2025-06` or a `@sha256` digest.

  A new build then means a new reference, which triggers a fresh import. The published Terminal-Bench images follow this convention.

  To refresh an image that's already registered, point `docker_image` at a new immutable tag or digest, or delete the registered Tensorlake image so the next run re-imports it. `--force-build` does not re-import. See [Force a fresh build](#force-a-fresh-build).
</Warning>

<Tip>
  **Terminal-Bench 2.1 images are already published.** We've registered every Terminal-Bench 2.1 task image publicly, so anyone with Tensorlake access boots straight from them with no build and no import. Run the dataset as usual and each task uses its published image.
</Tip>

Harbor uses `TENSORLAKE_API_KEY` for image lookup and import. The API key selects the project, so you don't need separate organization or project environment variables.

### Dockerfile

If a task has no `docker_image`, Harbor builds its `environment/Dockerfile` once with Tensorlake's image builder and caches the result. Every later trial boots directly from the cached image, with no per-trial `apt` or `pip` work. The cache key covers the Dockerfile and every file in the build context, so editing a `requirements.txt` pin or any `COPY`'d file triggers a rebuild.

```bash theme={null}
harbor run --env tensorlake \
  --dataset terminal-bench/terminal-bench-2-1 \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6
```

If the build fails, the trial fails with the build error. Harbor has no fallback path. Earlier versions replayed the Dockerfile's `RUN` and `COPY` steps on each trial, but Harbor removed that path. If you pass `--ek use_oci_image_build=false`, Harbor stops with an error.

#### Dockerfile requirements

The image builder is stricter than a local `docker build`, so 3 Docker conventions need adjustments:

* **`COPY` does not auto-create parent directories**: `COPY x /a/b/c` fails if `/a/b` doesn't exist yet. Add `RUN mkdir -p /a/b` before the `COPY`.
* **Don't pin exact apt versions** (`apt-get install curl=8.5.0-2ubuntu10.6`): Drop the pin or pick a version that exists in the target distro.
* **Use a FROM image that ships the Python you need**, such as `python:3.10-bookworm`, rather than fetching a non-native version at build time.

### Force a fresh build

To rebuild the Dockerfile even when a valid cached image exists, add `--force-build`. Harbor registers the new image under a unique name for that run only. The cache for later runs does not change.

If the task sets both `docker_image` and a Dockerfile, `--force-build` skips `docker_image` and builds from the Dockerfile. If the task has no Dockerfile, Harbor uses `docker_image` as usual. `--force-build` has no effect when you restore from a snapshot.

```bash theme={null}
harbor run --env tensorlake \
  --force-build \
  --dataset terminal-bench/terminal-bench-2-1 \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6
```

### Sharing images publicly

By default, images Harbor builds or imports are private to your organization. Only your org can boot from them. Add `--ek is_public=true` to register a freshly built or imported image as public, so any organization with Tensorlake access can reuse it:

```bash theme={null}
harbor run --env tensorlake \
  --ek is_public=true \
  --dataset terminal-bench/terminal-bench-2-1 \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6
```

The flag applies to both Dockerfile-built images and prebuilt `docker_image` imports. Another organization boots a public image through the prebuilt `docker_image` path. The published Terminal-Bench images work this way. To publish an environment that others boot directly, use a `docker_image` reference.

<Note>
  Publishing public images requires an allowlist entry. If your account isn't on it, Harbor ignores the flag and the image stays private. Contact Tensorlake to be added.
</Note>

`is_public` takes effect only when Harbor registers a new image. If an image with the same name already exists, either a private copy from an earlier run or an existing public one, Harbor boots it as-is and won't republish it. If the existing image is private, Harbor logs a message that it did not publish the image. To turn an already-private image public, delete it first, then rerun with `--ek is_public=true`. Changing the build context so the image gets a new name also works.

### Ad-hoc native dependencies

If a task needs extra apt packages and you don't want to edit the Dockerfile or maintain a snapshot, use `preinstall_packages`:

```bash theme={null}
harbor run --env tensorlake \
  --ek 'preinstall_packages=["build-essential","rustc","cargo"]' \
  --dataset terminal-bench/terminal-bench-2-1 \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6
```

Harbor installs the packages at the start of each trial. Prefer snapshots when the package set is large or reused across runs, so you pay the install cost once.

## Other sandbox options

Pass these options with `--ek`:

| Option | Default | Effect |
| - | - | - |
| `snapshot_id` | None | Restore the sandbox from a pre-warmed snapshot. Harbor skips image build, import, and `preinstall_packages`. See [Snapshots](/sandboxes/snapshots). |
| `timeout_secs` | `86400` (24 hours) | Hard timeout for the sandbox. Tensorlake deletes the sandbox when the timeout ends. |
| `preinstall_packages` | None | Extra apt packages to install at sandbox start. See [Ad-hoc native dependencies](#ad-hoc-native-dependencies). |
| `is_public` | `false` | Register new images as public. See [Sharing images publicly](#sharing-images-publicly). |
| `dind_image` | Harbor's DinD host image | Custom host image for Compose tasks. See [Docker Compose tasks](#docker-compose-multi-container-tasks). |

Harbor also sets these environment variables in each single-container sandbox, so that agents and task scripts can adapt:

| Variable | Value | Meaning |
| - | - | - |
| `TENSORLAKE_SANDBOX` | `1` | The trial runs on a Tensorlake sandbox. |
| `SANDBOX_KVM_AVAILABLE` | `0` | No nested virtualization. For example, do not use `-enable-kvm` with QEMU. |
| `SANDBOX_DOCKER_AVAILABLE` | `0` | No container runtime. Do not use `docker run`. For containers, use a [Docker Compose task](#docker-compose-multi-container-tasks). |

## Docker Compose (multi-container) tasks

If a task needs more than one container, such as a database or sidecar service alongside the agent's environment, add an `environment/docker-compose.yaml`. Harbor detects it and boots a docker-capable sandbox instead of a single container. You don't need an extra flag:

```bash theme={null}
harbor run --env tensorlake \
  --include-task-name my-org/multi-service-task \
  --dataset my-org/my-dataset \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6
```

Harbor boots a systemd-managed Docker-in-Docker (DinD) host sandbox and runs your compose project inside it. `dockerd` and `docker compose` run on the sandbox. Your `main` service, where the agent and verifier operate, runs alongside any other services you declare.

<Tip>
  We've published Harbor's default DinD host image publicly. It contains the outer sandbox: systemd, dockerd, and docker compose. Harbor never has to build or import it. Your compose services still build or pull as usual once `docker compose` runs inside that host.
</Tip>

Compose tasks differ from single-container tasks in 4 ways:

* **No snapshots**: Compose tasks don't support `snapshot_id`. If you set it, Harbor stops with an error. The compose project builds inside a fresh DinD host on every run.
* **`preinstall_packages` has no effect**: Harbor ignores it and logs a warning. Put packages in the compose service images instead.
* **No stream mode**: `--stream` works on single-container tasks only. The agent runs in a nested container that the stream tunnel cannot reach.
* **Size storage generously**: Compose pulls and builds layers on top of the DinD host's own rootfs, so `storage_mb` needs headroom beyond what a single-container task needs for the same images.

Network policy works the same as for single-container tasks, including `allowlist` and [per-phase overrides](#dynamic-network-policy). Tensorlake enforces the policy outside the VM, so one policy covers all compose containers. Your compose networks do not change. The DinD host starts with internet access so that `docker compose` can build and pull images. Harbor then creates the containers, applies the task's policy, and only then starts the containers. No container runs before the policy is in effect.

To use a custom DinD host image instead of Harbor's default, for example a smaller or Alpine-based one, pass `--ek dind_image=your_dind_image_reference`. It must ship `dockerd`, `docker compose`, and a systemd-managed `docker.service`.

## Interactive debugging

### Open a shell in a task environment

To inspect a task's environment by hand, start it on Tensorlake and open a shell:

```bash theme={null}
harbor task start-env -p path/to/task -e tensorlake -i
```

Harbor starts the sandbox and opens a shell with `tl sbx ssh`. Inspect state, rerun tests by hand, and confirm whether the agent or the environment caused the failure. This command needs the [Tensorlake CLI](/sandboxes/quickstart) (`tl`) on your `PATH`.

For Compose tasks, the shell opens on the DinD host. Run `docker ps` to list the compose containers, and `docker exec -it <container> bash` to enter the `main` service.

<Warning>
  Harbor does not delete the sandbox when you exit the shell. The shell replaces the Harbor process, so Harbor's cleanup step does not run. The sandbox stays until its timeout, which is 24 hours by default. To delete it, find its ID with `tl sbx ls`, then run `tl sbx terminate <sandbox-id>`.
</Warning>

### Stream a running trial

Add `--stream` to follow the agent's steps and browse the sandbox files while the trial runs:

```bash theme={null}
harbor run --env tensorlake --stream \
  --include-task-name terminal-bench/pytorch-model-cli \
  --dataset terminal-bench/terminal-bench-2-1 \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6 \
  --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY
```

In another terminal, run `harbor view jobs`. Harbor's [Stream](https://docs.harborframework.com/core-concepts/jobs/stream) docs describe the viewer.

Harbor runs a root SSH server inside the sandbox and connects to it through a Tensorlake tunnel. Stream mode needs only `TENSORLAKE_API_KEY`. It does not need the `tl` CLI. Stream mode works on single-container tasks only.

If the task image has no SSH server, Harbor installs `openssh-server` at sandbox start. When the task's `network_mode` is not `public`, Harbor opens access to the apt mirrors for the install only. It then restores the task's policy and waits until that policy is in effect before the trial starts. To skip this step, include `openssh-server` in the task image.

Add `--no-delete` to keep the sandbox after the run for debugging. Delete it when you finish.

## Structured logs

Each trial produces structured artifacts:

```text theme={null}
gcode-to-text__UFALMLv
├── agent/
├── verifier/
├── result.json
└── trial.log
```

From these you can trace:

* The agent's actions and outputs
* What the verifier checked
* Why the trial passed or failed

## Next steps

<CardGroup cols={2}>
  <Card title="Snapshots" icon="camera" href="/sandboxes/snapshots">
    Build an environment once, snapshot it, and restore it for every trial.
  </Card>

  <Card title="Reproducible RL Environments" icon="dumbbell" href="/sandboxes/agentic-rl-reproducible-env">
    Use sandboxes as a deterministic reward oracle for RL training loops.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.