Skip to main content
Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. With Harbor you can:
  • Evaluate any agent, such as Claude Code, OpenHands, or Codex CLI, against curated datasets like Terminal-Bench, SWE-Bench, and Aider Polyglot
  • Build and share your own benchmarks
  • Run thousands of trials in parallel across cloud providers
  • Generate rollouts for reinforcement learning (RL) optimization
Harbor abstracts the execution backend behind an --env flag. Tensorlake is one of those providers, alongside other sandbox vendors and local Docker. The same Harbor commands run on Tensorlake sandboxes without changes to your tasks, agents, or evaluators.
This guide covers command-line interface (CLI) agent evaluations against benchmarks like Terminal-Bench. Harbor also generates rollouts for RL optimization. We’ll cover those workflows in follow-up guides.
If you’re new to Tensorlake, sign up at Tensorlake Cloud. New accounts include free credits, enough to run a full Terminal-Bench sweep before you pay for anything.

​
Quickstart

Install Harbor with the Tensorlake extra, set 2 API keys, and run 1 Terminal-Bench task.
1

Get a Tensorlake API key

Create one in the Tensorlake Dashboard. You also need an API key for the agent provider you want to evaluate, such as Anthropic.
2

Install Harbor with the Tensorlake provider

The harbor[tensorlake] extra installs the TensorLakeEnvironment provider alongside Harbor.
3

Set your environment variables

4

Run a Terminal-Bench task

Run a single Terminal-Bench task on Tensorlake with Claude Code as the agent:
Remove --include-task-name to run the full Terminal-Bench 2.1 suite. --ae forwards an environment variable from your shell into the sandbox where the agent runs. Add more --ae flags for any other secrets the agent needs.

​
Why Tensorlake for Harbor

Harbor’s value comes from running large fleets of environments in parallel and trusting the results. Tensorlake’s runtime targets that workload:
  • Per-trial sandboxes: Each task starts on a clean sandbox, which Tensorlake destroys at the end. Trials share no kernel state, which matters for both eval reproducibility and RL reward integrity.
  • Full task-environment support: Tensorlake imports a task’s real Docker image and converts it into a sandbox image that boots directly. Every trial runs the exact environment the benchmark defines, not an approximation built by replaying a Dockerfile. That closes the environment gap that otherwise skews results.
  • Pre-warmed snapshots: Build environments with heavy apt or pip installs, such as PyTorch, CUDA toolchains, or full Linux desktops, once. Snapshot them, then restore the snapshot for every later trial or rollout.
  • Independent verification: Harbor’s test script runs inside the sandbox and writes 1.0 or 0.0 to reward.txt. The agent never sees or touches the verifier, so you never confuse “the agent said it worked” with “the tests pass”.
  • Parallel scale: Tensorlake schedules thousands of sandboxes concurrently, which is what RL rollout generation and full benchmark sweeps need.

​
Anatomy of a Harbor task

Harbor expects each task to use the layout below. gcode-to-text is an example:
  • environment/Dockerfile defines the base image and any setup steps.
  • instruction.md is the prompt the agent receives.
  • solution/ is a reference solution that validates the environment itself.
  • tests/test.sh runs after the agent finishes and produces reward.txt.

​
Tune sandbox resources

Each task’s task.toml controls the sandbox Harbor provisions on Tensorlake. Set resources in the [environment] block:
task.toml
Tensorlake requires between 1024 MB and 8192 MB of memory_mb per CPU core.
Harbor deprecated allow_internet in favor of network_mode. Harbor still accepts it and maps true and false to network_mode = "public" and "no-network". Set network_mode directly in new tasks.
Two sizing guidelines:
  • Large or heavy images: If your environment/Dockerfile pulls in big toolchains, such as PyTorch, CUDA, full Linux desktops, or large datasets, raise cpus and memory_mb so the build and runtime have headroom. Raise storage_mb past the image size plus working-set room. Underprovisioned sandboxes fail with build timeouts or out-of-memory (OOM) errors mid-trial.
  • Lock down network_mode: Set network_mode = "no-network" to stop the agent from searching the web for answers. Or set "allowlist" with allowed_hosts to permit only specific destinations.

​
Dynamic network policy

Different phases of a trial need different network access. Setup may need the open internet to install dependencies. The agent and verifier need tighter limits so the agent can’t search the web for answers and nobody can tamper with the verifier’s result. Tensorlake sandboxes switch the network policy on a running sandbox, so Harbor can scope network_mode per phase in task.toml:
task.toml
Tensorlake applies each phase override to the running sandbox’s firewall as one atomic swap. There is no window where the sandbox is unprotected and no restart between phases. allowed_hosts accepts exact hostnames, leading-wildcard hostnames such as *.example.com, IPv4 literals, and IPv4 CIDR ranges.
allowlist and per-phase overrides apply to single-container tasks only. Docker Compose tasks support public and no-network alone. That section explains why.
Dynamic Network Policies in Harbor explains why phase-scoped policies matter. Harbor’s Network Policy docs have the full field reference.

​
Image build and caching

Each trial boots from an image. Harbor uses a prebuilt image when the task declares one, and otherwise builds the task’s Dockerfile: Either way, Harbor builds or imports the image once and reuses it. You pay the cost on the first trial only. Every later trial boots directly from the cached image. If a task sets both and the prebuilt image exists, Harbor uses the prebuilt image.
When you reuse one heavy environment across repeated runs, such as RL rollouts or evals of the same task, restore it from a pre-warmed snapshot instead of rebuilding. See Snapshots.

​
Prebuilt image

If a task declares a docker_image in task.toml, Harbor boots directly from that image and skips the Dockerfile:
task.toml
Harbor looks the image up in Tensorlake by name and boots from it. If the image isn’t registered yet, Harbor imports it once and reuses it on every later trial. Harbor derives the registered name from the reference string, so every later run boots the first import of that reference.
Always publish with an immutable tag, never latest. The registered name comes from the reference string, not the image contents. Harbor captures a tag like latest at its first import and freezes it. If you push new content to latest, Harbor keeps booting the old image and never re-pulls. Use an immutable tag such as myorg/my-task-env:2025-06 or a @sha256 digest.A new build then means a new reference, which triggers a fresh import. The published Terminal-Bench images follow this convention.To refresh an image that’s already registered, point docker_image at a new immutable tag or digest, or delete the registered Tensorlake image so the next run re-imports it. --force-build does not re-import. It builds from the Dockerfile instead.
Terminal-Bench 2.1 images are already published. We’ve registered every Terminal-Bench 2.1 task image publicly, so anyone with Tensorlake access boots straight from them with no build and no import. Run the dataset as usual and each task uses its published image.
Harbor uses TENSORLAKE_API_KEY for image lookup and import. The API key selects the project, so you don’t need separate organization or project environment variables. If Harbor reports that image lookup requires organization and project context, upgrade its Tensorlake SDK integration. That message comes from an older SDK contract.

​
Dockerfile

If a task has no docker_image, Harbor builds its environment/Dockerfile once with Tensorlake’s image builder and caches the result. Every later trial boots directly from the cached image, with no per-trial apt or pip work. The cache key covers the Dockerfile and every file in the build context, so editing a requirements.txt pin or any COPY’d file triggers a rebuild.
If a build fails, Harbor falls back to replaying the Dockerfile’s RUN and COPY steps on each trial. Every trial still runs, but slower. You can also request the fallback explicitly while you iterate on a Dockerfile:

​
Dockerfile requirements

The image builder is stricter than a local docker build, so 3 Docker conventions need adjustments:
  • COPY does not auto-create parent directories: COPY x /a/b/c fails if /a/b doesn’t exist yet. Add RUN mkdir -p /a/b before the COPY.
  • Don’t pin exact apt versions (apt-get install curl=8.5.0-2ubuntu10.6): Drop the pin or pick a version that exists in the target distro.
  • Use a FROM image that ships the Python you need, such as python:3.10-bookworm, rather than fetching a non-native version at build time.
To force a fresh rebuild even when a valid cached image exists, add --force-build. This applies to that run only and leaves the cache for later runs intact.

​
Sharing images publicly

By default, images Harbor builds or imports are private to your organization. Only your org can boot from them. Add --ek is_public=true to register a freshly built or imported image as public, so any organization with Tensorlake access can reuse it:
The flag applies to both Dockerfile-built images and prebuilt docker_image imports. Another organization boots a public image through the prebuilt docker_image path. The published Terminal-Bench images work this way. To publish an environment that others boot directly, use a docker_image reference.
Publishing public images requires an allowlist entry. If your account isn’t on it, Harbor ignores the flag and the image stays private. Contact Tensorlake to be added.
is_public takes effect only when Harbor registers a new image. If an image with the same name already exists, either a private copy from an earlier run or an existing public one, Harbor boots it as-is and won’t republish it. To turn an already-private image public, delete it first, then rerun with --ek is_public=true. Changing the build context so the image gets a new name also works.

​
Ad-hoc native dependencies

If a task needs extra apt packages and you don’t want to edit the Dockerfile or maintain a snapshot, use preinstall_packages:
Harbor installs the packages at the start of each trial. Prefer snapshots when the package set is large or reused across runs, so you pay the install cost once.

​
Docker Compose (multi-container) tasks

If a task needs more than one container, such as a database or sidecar service alongside the agent’s environment, add an environment/docker-compose.yaml. Harbor detects it and boots a docker-capable sandbox instead of a single container. You don’t need an extra flag:
Harbor boots a systemd-managed Docker-in-Docker (DinD) host sandbox and runs your compose project inside it. dockerd and docker compose run on the sandbox. Your main service, where the agent and verifier operate, runs alongside any other services you declare.
We’ve published Harbor’s default DinD host image publicly. It contains the outer sandbox: systemd, dockerd, and docker compose. Harbor never has to build or import it. Your compose services still build or pull as usual once docker compose runs inside that host.
Compose tasks differ from single-container tasks in 4 ways:
  • No snapshots: Compose tasks don’t support snapshot_id. The compose project builds inside a fresh DinD host on every run.
  • preinstall_packages has no effect: Put packages in the compose service images instead.
  • Network policy is coarser: Only public and no-network are available (see the Dynamic network policy note above). allowlist and per-phase switching require host-level firewall control that a nested DinD host doesn’t have, so Tensorlake enforces no-network isolation inside the virtual machine (VM) instead.
  • Size storage generously: Compose pulls and builds layers on top of the DinD host’s own rootfs, so storage_mb needs headroom beyond what a single-container task needs for the same images.
To use a custom DinD host image instead of Harbor’s default, for example a smaller or Alpine-based one, pass --ek dind_image=your_dind_image_reference. It must ship dockerd, docker compose, and a systemd-managed docker.service.

​
Interactive debugging

When a trial fails and you want to inspect the live environment, attach to the session:
This opens a shell in the running sandbox. Inspect state, rerun tests by hand, and confirm whether the agent or the environment caused the failure.

​
Structured logs

Each trial produces structured artifacts:
From these you can trace:
  • The agent’s actions and outputs
  • What the verifier checked
  • Why the trial passed or failed

​
Next steps

Snapshots

Build an environment once, snapshot it, and restore it for every trial.

Reproducible RL Environments

Use sandboxes as a deterministic reward oracle for RL training loops.