Skip to main content
Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. With Harbor you can:
  • Evaluate any agent, such as Claude Code, OpenHands, or Codex CLI, against curated datasets like Terminal-Bench, SWE-Bench, and Aider Polyglot
  • Build and share your own benchmarks
  • Run thousands of trials in parallel across cloud providers
  • Generate rollouts for reinforcement learning (RL) optimization
Harbor abstracts the execution backend behind an --env flag. Tensorlake is one of those providers, alongside other sandbox vendors and local Docker. The same Harbor commands run on Tensorlake sandboxes without changes to your tasks, agents, or evaluators.
This guide covers command-line interface (CLI) agent evaluations against benchmarks like Terminal-Bench. Harbor also generates rollouts for RL optimization. We’ll cover those workflows in follow-up guides.
If you’re new to Tensorlake, sign up at Tensorlake Cloud. New accounts include free credits, enough to run a full Terminal-Bench sweep before you pay for anything.

​
Quickstart

Install Harbor with the Tensorlake extra, set 2 API keys, and run 1 Terminal-Bench task.
1

Get a Tensorlake API key

Create one in the Tensorlake Dashboard. You also need an API key for the agent provider you want to evaluate, such as Anthropic.
2

Install Harbor with the Tensorlake provider

The harbor[tensorlake] extra installs the TensorLakeEnvironment provider alongside Harbor.
3

Set your environment variables

4

Run a Terminal-Bench task

Run a single Terminal-Bench task on Tensorlake with Claude Code as the agent:
Remove --include-task-name to run the full Terminal-Bench 2.1 suite. --ae forwards an environment variable from your shell into the sandbox where the agent runs. Add more --ae flags for any other secrets the agent needs.

​
Why Tensorlake for Harbor

Harbor’s value comes from running large fleets of environments in parallel and trusting the results. Tensorlake’s runtime targets that workload:
  • Per-trial sandboxes: Each task starts on a clean sandbox, which Tensorlake destroys at the end. Trials share no kernel state, which matters for both eval reproducibility and RL reward integrity.
  • Full task-environment support: Tensorlake imports a task’s real Docker image and converts it into a sandbox image that boots directly. Every trial runs the exact environment the benchmark defines, not an approximation built by replaying a Dockerfile. That closes the environment gap that otherwise skews results.
  • Pre-warmed snapshots: Build environments with heavy apt or pip installs, such as PyTorch, CUDA toolchains, or full Linux desktops, once. Snapshot them, then restore the snapshot for every later trial or rollout.
  • Independent verification: Harbor’s test script runs inside the sandbox and writes 1.0 or 0.0 to reward.txt. The agent never sees or touches the verifier, so you never confuse “the agent said it worked” with “the tests pass”.
  • Parallel scale: Tensorlake schedules thousands of sandboxes concurrently, which is what RL rollout generation and full benchmark sweeps need.

​
Anatomy of a Harbor task

Harbor expects each task to use the layout below. gcode-to-text is an example:
  • environment/Dockerfile defines the base image and any setup steps.
  • instruction.md is the prompt the agent receives.
  • solution/ is a reference solution that validates the environment itself.
  • tests/test.sh runs after the agent finishes and produces reward.txt.

​
Tune sandbox resources

Each task’s task.toml controls the sandbox Harbor provisions on Tensorlake. Set resources in the [environment] block:
task.toml
If a value is below the Tensorlake minimum, Harbor raises it to the minimum. Harbor never lowers a value. Many benchmark tasks ask for 1 CPU and 4 GB, which causes out-of-memory (OOM) failures on sandbox hosts. A sandbox that you restore from a snapshot keeps the disk size of the snapshot.
Tensorlake requires between 1024 MB and 8192 MB of memory_mb per CPU core.
Harbor deprecated allow_internet in favor of network_mode. Harbor still accepts it and maps true and false to network_mode = "public" and "no-network". Set network_mode directly in new tasks.
Two sizing guidelines:
  • Large or heavy images: If your environment/Dockerfile pulls in big toolchains, such as PyTorch, CUDA, full Linux desktops, or large datasets, raise cpus and memory_mb so the build and runtime have headroom. Raise storage_mb past the image size plus working-set room. Underprovisioned sandboxes fail with build timeouts or out-of-memory (OOM) errors mid-trial.
  • Lock down network_mode: Set network_mode = "no-network" to stop the agent from searching the web for answers. Or set "allowlist" with allowed_hosts to permit only specific destinations.

​
Dynamic network policy

Different phases of a trial need different network access. Setup may need the open internet to install dependencies. The agent and verifier need tighter limits so the agent can’t search the web for answers and nobody can tamper with the verifier’s result. Tensorlake sandboxes switch the network policy on a running sandbox, so Harbor can scope network_mode per phase in task.toml:
task.toml
Tensorlake replaces the running sandbox’s firewall rules for each phase. The sandbox does not restart between phases. The new rules replace the old rules fully, so a narrower policy never keeps an allowance from the previous one. The switch is fail-safe, but not instant:
  • The old policy stays active for a few seconds. Tensorlake resolves each hostname in the new policy before it swaps the rules. A tighter policy usually takes effect in about 5 seconds, and a slow hostname can extend this to about 10 seconds. The sandbox is never unprotected during the swap.
  • Open connections continue. The firewall is stateful. A connection that opened under the wider policy stays open after the swap.
  • A hostname that does not resolve stops the switch. Tensorlake rejects the update, the previous policy stays in effect, and the phase fails.
allowed_hosts accepts exact hostnames, leading-wildcard hostnames such as *.example.com, IPv4 addresses, and IPv4 CIDR ranges. Tensorlake does not support IPv6 entries.
Network policies, including allowlist and per-phase overrides, also apply to Docker Compose tasks. Tensorlake enforces the policy outside the sandbox VM, so one policy covers all compose containers.
Dynamic Network Policies in Harbor explains why phase-scoped policies matter. Harbor’s Network Policy docs have the full field reference.

​
Image build and caching

Each trial boots from an image. Harbor uses a prebuilt image when the task declares one, and otherwise builds the task’s Dockerfile: Either way, Harbor builds or imports the image once and reuses it. You pay the cost on the first trial only. Every later trial boots directly from the cached image. If a task sets both and the prebuilt image exists, Harbor uses the prebuilt image.
When you reuse one heavy environment across repeated runs, such as RL rollouts or evals of the same task, restore it from a pre-warmed snapshot instead of rebuilding. See Snapshots.

​
Prebuilt image

If a task declares a docker_image in task.toml, Harbor boots directly from that image and skips the Dockerfile:
task.toml
Harbor looks the image up in Tensorlake by name and boots from it. If the image isn’t registered yet, Harbor imports it once and reuses it on every later trial. Harbor derives the registered name from the reference string, so every later run boots the first import of that reference.
Always publish with an immutable tag, never latest. The registered name comes from the reference string, not the image contents. Harbor captures a tag like latest at its first import and freezes it. If you push new content to latest, Harbor keeps booting the old image and never re-pulls. Use an immutable tag such as myorg/my-task-env:2025-06 or a @sha256 digest.A new build then means a new reference, which triggers a fresh import. The published Terminal-Bench images follow this convention.To refresh an image that’s already registered, point docker_image at a new immutable tag or digest, or delete the registered Tensorlake image so the next run re-imports it. --force-build does not re-import. See Force a fresh build.
Terminal-Bench 2.1 images are already published. We’ve registered every Terminal-Bench 2.1 task image publicly, so anyone with Tensorlake access boots straight from them with no build and no import. Run the dataset as usual and each task uses its published image.
Harbor uses TENSORLAKE_API_KEY for image lookup and import. The API key selects the project, so you don’t need separate organization or project environment variables.

​
Dockerfile

If a task has no docker_image, Harbor builds its environment/Dockerfile once with Tensorlake’s image builder and caches the result. Every later trial boots directly from the cached image, with no per-trial apt or pip work. The cache key covers the Dockerfile and every file in the build context, so editing a requirements.txt pin or any COPY’d file triggers a rebuild.
If the build fails, the trial fails with the build error. Harbor has no fallback path. Earlier versions replayed the Dockerfile’s RUN and COPY steps on each trial, but Harbor removed that path. If you pass --ek use_oci_image_build=false, Harbor stops with an error.

​
Dockerfile requirements

The image builder is stricter than a local docker build, so 3 Docker conventions need adjustments:
  • COPY does not auto-create parent directories: COPY x /a/b/c fails if /a/b doesn’t exist yet. Add RUN mkdir -p /a/b before the COPY.
  • Don’t pin exact apt versions (apt-get install curl=8.5.0-2ubuntu10.6): Drop the pin or pick a version that exists in the target distro.
  • Use a FROM image that ships the Python you need, such as python:3.10-bookworm, rather than fetching a non-native version at build time.

​
Force a fresh build

To rebuild the Dockerfile even when a valid cached image exists, add --force-build. Harbor registers the new image under a unique name for that run only. The cache for later runs does not change. If the task sets both docker_image and a Dockerfile, --force-build skips docker_image and builds from the Dockerfile. If the task has no Dockerfile, Harbor uses docker_image as usual. --force-build has no effect when you restore from a snapshot.

​
Sharing images publicly

By default, images Harbor builds or imports are private to your organization. Only your org can boot from them. Add --ek is_public=true to register a freshly built or imported image as public, so any organization with Tensorlake access can reuse it:
The flag applies to both Dockerfile-built images and prebuilt docker_image imports. Another organization boots a public image through the prebuilt docker_image path. The published Terminal-Bench images work this way. To publish an environment that others boot directly, use a docker_image reference.
Publishing public images requires an allowlist entry. If your account isn’t on it, Harbor ignores the flag and the image stays private. Contact Tensorlake to be added.
is_public takes effect only when Harbor registers a new image. If an image with the same name already exists, either a private copy from an earlier run or an existing public one, Harbor boots it as-is and won’t republish it. If the existing image is private, Harbor logs a message that it did not publish the image. To turn an already-private image public, delete it first, then rerun with --ek is_public=true. Changing the build context so the image gets a new name also works.

​
Ad-hoc native dependencies

If a task needs extra apt packages and you don’t want to edit the Dockerfile or maintain a snapshot, use preinstall_packages:
Harbor installs the packages at the start of each trial. Prefer snapshots when the package set is large or reused across runs, so you pay the install cost once.

​
Other sandbox options

Pass these options with --ek: Harbor also sets these environment variables in each single-container sandbox, so that agents and task scripts can adapt:

​
Docker Compose (multi-container) tasks

If a task needs more than one container, such as a database or sidecar service alongside the agent’s environment, add an environment/docker-compose.yaml. Harbor detects it and boots a docker-capable sandbox instead of a single container. You don’t need an extra flag:
Harbor boots a systemd-managed Docker-in-Docker (DinD) host sandbox and runs your compose project inside it. dockerd and docker compose run on the sandbox. Your main service, where the agent and verifier operate, runs alongside any other services you declare.
We’ve published Harbor’s default DinD host image publicly. It contains the outer sandbox: systemd, dockerd, and docker compose. Harbor never has to build or import it. Your compose services still build or pull as usual once docker compose runs inside that host.
Compose tasks differ from single-container tasks in 4 ways:
  • No snapshots: Compose tasks don’t support snapshot_id. If you set it, Harbor stops with an error. The compose project builds inside a fresh DinD host on every run.
  • preinstall_packages has no effect: Harbor ignores it and logs a warning. Put packages in the compose service images instead.
  • No stream mode: --stream works on single-container tasks only. The agent runs in a nested container that the stream tunnel cannot reach.
  • Size storage generously: Compose pulls and builds layers on top of the DinD host’s own rootfs, so storage_mb needs headroom beyond what a single-container task needs for the same images.
Network policy works the same as for single-container tasks, including allowlist and per-phase overrides. Tensorlake enforces the policy outside the VM, so one policy covers all compose containers. Your compose networks do not change. The DinD host starts with internet access so that docker compose can build and pull images. Harbor then creates the containers, applies the task’s policy, and only then starts the containers. No container runs before the policy is in effect. To use a custom DinD host image instead of Harbor’s default, for example a smaller or Alpine-based one, pass --ek dind_image=your_dind_image_reference. It must ship dockerd, docker compose, and a systemd-managed docker.service.

​
Interactive debugging

​
Open a shell in a task environment

To inspect a task’s environment by hand, start it on Tensorlake and open a shell:
Harbor starts the sandbox and opens a shell with tl sbx ssh. Inspect state, rerun tests by hand, and confirm whether the agent or the environment caused the failure. This command needs the Tensorlake CLI (tl) on your PATH. For Compose tasks, the shell opens on the DinD host. Run docker ps to list the compose containers, and docker exec -it <container> bash to enter the main service.
Harbor does not delete the sandbox when you exit the shell. The shell replaces the Harbor process, so Harbor’s cleanup step does not run. The sandbox stays until its timeout, which is 24 hours by default. To delete it, find its ID with tl sbx ls, then run tl sbx terminate <sandbox-id>.

​
Stream a running trial

Add --stream to follow the agent’s steps and browse the sandbox files while the trial runs:
In another terminal, run harbor view jobs. Harbor’s Stream docs describe the viewer. Harbor runs a root SSH server inside the sandbox and connects to it through a Tensorlake tunnel. Stream mode needs only TENSORLAKE_API_KEY. It does not need the tl CLI. Stream mode works on single-container tasks only. If the task image has no SSH server, Harbor installs openssh-server at sandbox start. When the task’s network_mode is not public, Harbor opens access to the apt mirrors for the install only. It then restores the task’s policy and waits until that policy is in effect before the trial starts. To skip this step, include openssh-server in the task image. Add --no-delete to keep the sandbox after the run for debugging. Delete it when you finish.

​
Structured logs

Each trial produces structured artifacts:
From these you can trace:
  • The agent’s actions and outputs
  • What the verifier checked
  • Why the trial passed or failed

​
Next steps

Snapshots

Build an environment once, snapshot it, and restore it for every trial.

Reproducible RL Environments

Use sandboxes as a deterministic reward oracle for RL training loops.