Tensorlake Documentation

Bring your own cloud (BYOC) lets your team run Tensorlake sandboxes on compute that you own. Your code, sandbox storage, and executor capacity stay in your AWS account. Your data never leaves your AWS account, and Tensorlake has no access to data in your AWS account. Tensorlake Cloud remains the control plane that schedules work and gives your team one API for managing sandboxes. An AWS BYOC deployment has two customer-run components. The Sandbox Proxy runs on Amazon Elastic Kubernetes Service (Amazon EKS) and provides the public entry point for sandbox traffic. Executors run on bare-metal Amazon EC2 instances and host the sandboxes themselves.

Users reach an AWS Network Load Balancer in your AWS account. The load balancer sends traffic to the Sandbox Proxy on Amazon EKS, which reaches executors on bare-metal Amazon EC2 over the private VPC. Executors write runtime data and snapshots to Amazon S3. The Sandbox Proxy and the executors both open outbound HTTPS connections to Tensorlake Cloud.

The Sandbox Proxy does not access Amazon S3. Only the executors need access to the project bucket. Tensorlake can also run the Sandbox Proxy for you and reach your executors over a private VPC peering connection. See Set up BYOC on AWS with VPC peering.

What Tensorlake manages

Tensorlake publishes the Sandbox Proxy image, Kubernetes manifests, executor software, and executor configuration. Tensorlake also runs the control plane. The executor provisioner checks for approved updates and applies them after you enroll the host.

What you manage

You manage the AWS account, network, EKS cluster, EC2 instances, S3 bucket, DNS, and IAM permissions. You also decide which networks can reach the public endpoint and which AWS identities can access project data.

Before you begin

Tensorlake requirements

You need a Tensorlake project with Self-hosted Compute enabled. You also need permission to change the project settings and create a project-admin API key. If the Self-hosted Compute page is not available, contact Tensorlake. The same project-admin API key can prepare the Sandbox Proxy and enroll the first executor. Treat the key as a secret. Do not save it in an EC2 user-data script.

AWS account requirements

Prepare these resources in the AWS Region where you want to run sandboxes:

A VPC for the EKS cluster and the executors.
An EKS cluster.
The AWS Load Balancer Controller, version 2.7.2 or later.
An IAM role that gives the controller permission to create an AWS Network Load Balancer (NLB), target groups, and security-group rules.
A public DNS zone for the sandbox domain.
An AWS Certificate Manager (ACM) certificate for the sandbox domain.
An S3 bucket for runtime data, built images, and durable sandbox snapshots.
An EC2 instance profile that can use the project bucket.
Capacity for at least one supported bare-metal EC2 instance: i7i.metal-24xl or i7i.metal-48xl.

Sandbox base domain

Choose the base domain before you create certificates or manifests. Every sandbox receives a subdomain below this base domain. For example, a base domain of sandboxes.customer.example produces URLs such as https://<sandbox-id>.sandboxes.customer.example. The public endpoint must be an HTTPS origin. Do not add a port or path.

1. Prepare project storage

Use a dedicated bucket for the project. It is the durable home for sandbox state, and executors write built sandbox images and snapshots into it. Keeping the bucket in the executor Region reduces latency and AWS data transfer costs. Block public access to the bucket. Enable encryption. If you use an AWS Key Management Service (AWS KMS) customer-managed key, give the executor instance role permission to use that key. The executor instance role needs these S3 permissions for the bucket:

Scope	Required actions
Bucket	`s3:GetBucketLocation`, `s3:ListBucket`, `s3:ListBucketMultipartUploads`
Objects in the bucket	`s3:GetObject`, `s3:PutObject`, `s3:DeleteObject`, `s3:AbortMultipartUpload`, `s3:ListMultipartUploadParts`

The enrollment preflight writes, reads, and deletes a small object under each derived storage path. This test stops the installation before a sandbox starts if the instance role cannot use the storage. Attach the role through an EC2 instance profile. Do not put static AWS access keys on the host. The Sandbox Proxy does not need this role or any other S3 permission.

2. Prepare the AWS network

Subnet layout

Hold the EKS cluster and the executors in one VPC. That is what lets the Sandbox Proxy reach each executor privately. If using VPC CIDR 10.90.0.0/16, you can set up this subnet layout, one subnet per Availability Zone:

Subnet	Ranges	Purpose
Public	`10.90.0.0/24`, `10.90.1.0/24`, `10.90.2.0/24`	NLB
Private	`10.90.16.0/20`, `10.90.32.0/20`, `10.90.48.0/20`	Kubernetes nodes and pods
Executor	`10.90.64.0/24`, `10.90.65.0/24`, `10.90.66.0/24`	Bare-metal EC2 instances

Tag all three public subnets with kubernetes.io/role/elb=1 so the AWS Load Balancer Controller can select them without relying on route-table discovery. See the Amazon EKS NLB requirements.

Load balancer security group

The public NLB terminates TLS and sends traffic to the Sandbox Proxy pods. Create a security group for the NLB:

Direction	Port	Source or target	Purpose
Inbound	`443`	Internet or approved client networks	HTTPS, WebSocket, and CLI SSH
Inbound	`22`	Approved SSH client networks	Optional native SSH
Outbound	`9080`	Sandbox Proxy pod security group	Proxy HTTP traffic
Outbound	`2222`	Sandbox Proxy pod security group	Proxy SSH traffic

The generated Service asks the AWS Load Balancer Controller to manage the matching backend security-group rules. The controller IAM role must include permission to add and remove those rules.

Executor security group

The Sandbox Proxy reaches each executor over the private AWS network. Do not expose executor ports to the internet. Allow traffic from the Sandbox Proxy pod network or security group to the executor security group:

Port	Purpose
`8096`	Native SSH traffic to a sandbox
`8097`	Dataplane API traffic over mutual TLS (mTLS)
`5565`	Raw sandbox TCP traffic over mTLS

mTLS means mutual Transport Layer Security. Both endpoints use certificates to verify each other.

Outbound access

Allow outbound TCP 443 from the EKS nodes and executor hosts to Tensorlake services and required AWS services. The EKS nodes must pull the public Sandbox Proxy image from public.ecr.aws. The executor must reach EC2 instance metadata through IMDSv2 and must reach the configured S3 bucket.

3. Create the public certificate

Request an ACM certificate in the same AWS account and Region as the NLB. Add both names to the certificate:

sandboxes.customer.example
*.sandboxes.customer.example

Create the DNS validation records. Wait until ACM shows the certificate status as ISSUED.

4. Prepare the Sandbox Proxy

The preparation script creates the customer-specific mTLS certificate, two SSH key pairs, and a local Kustomize overlay. Kustomize is a Kubernetes manifest customization tool. The script creates files only in the output directory. It does not change your cluster or AWS account. Download and run the script:

curl -fsSL \
  https://cloud.tensorlake.ai/byoc/sandbox-proxy/prepare.sh \
  --output prepare.sh

chmod +x prepare.sh

./prepare.sh \
  --api-key <tensorlake-api-key> \
  --endpoint https://sandboxes.customer.example \
  --namespace tensorlake-sandbox-proxy \
  --output-dir ./tensorlake-sandbox-proxy \
  --enable-aws-lb \
  --aws-lb-certificate-arn '<acm-certificate-arn>' \
  --aws-lb-security-group-id '<nlb-security-group-id>'

Keep the output directory secure. It contains private mTLS and SSH keys. Do not commit it to a source repository.

5. Configure the Tensorlake project

Open the project settings in Tensorlake Cloud. Find Node configuration and set the bucket URI:

s3://customer-tensorlake

Set SSH proxy public key to the content of this generated file:

./tensorlake-sandbox-proxy/ssh/outbound.key.pub

The Sandbox Proxy uses the matching private key when it connects to a sandbox SSH server. Leave this setting empty only if you want to disable SSH for the project. Save the node configuration before you enroll an executor. Tensorlake derives the runtime object, image-build, and snapshot paths from the bucket URI.

6. Deploy the Sandbox Proxy

Confirm the active Kubernetes context:

kubectl config current-context

Review the generated manifests and run a server-side dry run:

kubectl kustomize ./tensorlake-sandbox-proxy

kubectl apply \
  --server-side \
  --dry-run=server \
  -k ./tensorlake-sandbox-proxy

Add a Deployment patch if the cluster requires a node selector or toleration. Then apply the resources:

kubectl apply -k ./tensorlake-sandbox-proxy

kubectl \
  --namespace tensorlake-sandbox-proxy \
  rollout status deployment/sandbox-proxy

Wait for the controller to publish the NLB hostname:

kubectl \
  --namespace tensorlake-sandbox-proxy \
  get service sandbox-proxy-public

Confirm that both NLB target groups are healthy. The listener on port 443 uses pod port 9080. The listener on port 22 uses pod port 2222.

7. Create DNS records and activate the endpoint

Create two public DNS alias or CNAME records. Point both records to the NLB hostname:

sandboxes.customer.example
*.sandboxes.customer.example

You can use Amazon Route 53 or another public DNS provider. Open the project settings in Tensorlake Cloud. Set Sandbox ingress endpoint to this value:

https://sandboxes.customer.example

Save the setting. Verify the public health route:

curl -fsS https://sandboxes.customer.example/health

Verify that the certificate also covers sandbox subdomains:

openssl s_client \
  -connect sandboxes.customer.example:443 \
  -servername test.sandboxes.customer.example \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -checkhost test.sandboxes.customer.example

The command prints Hostname test.sandboxes.customer.example does match certificate.

8. Launch the first executor

Launch a bare-metal EC2 instance with Ubuntu 24.04. Attach the instance profile from the storage step. The host must expose /dev/kvm because Tensorlake uses hardware virtualization to isolate sandboxes. The instance must also have local NVMe instance storage. Enrollment formats those disks and mounts them at /mnt/data, which holds container images, sandbox images, and the local snapshot cache. Amazon EBS volumes are never used for this, so an instance type without local NVMe cannot run sandboxes. These instance types meet both requirements:

i7i.metal-24xl
i7i.metal-48xl

Place the instance in a private subnet that can receive the Sandbox Proxy traffic described above. Enable IMDSv2. Give the root volume 50 GB so it holds the operating system and the downloaded components.

9. Enroll the executor

In Tensorlake Cloud, open Self-hosted Compute, select Add node, and copy the bootstrap command. It has this form:

curl -fsSL https://cloud.tensorlake.ai/byoc/bootstrap.sh \
  | sudo bash -s -- --api-key <tensorlake-api-key>

Run the command on the EC2 instance. The bootstrap process checks the host and S3 access. It then creates the executor identity, installs the provisioner, and starts the executor services. The project-admin API key is used only during enrollment. The installed provisioner uses its own mTLS identity after enrollment. It continues to check Tensorlake for approved configuration and software updates. Return to Self-hosted Compute. Confirm that the new node appears and becomes ready before you create a sandbox.

10. Verify the complete path

Install the Tensorlake CLI if the tl command is not available:

curl -fsSL https://tensorlake.ai/install | sh

tl login
tl init

Create a sandbox. Use its ID in the remaining commands:

tl sbx create
tl sbx exec <sandbox-id> sh -lc 'printf BYOC_OK'
tl sbx ssh <sandbox-id>

The sandbox URL must have this form:

https://<sandbox-id>.sandboxes.customer.example

Test native SSH if port 22 is enabled:

tl sbx ssh keys add ~/.ssh/id_ed25519.pub
ssh <sandbox-id>@<sandbox-id>.sandboxes.customer.example

Create a snapshot and confirm that the executor writes data into the bucket:

tl sbx snapshot <sandbox-id>

Add more executors

Each executor needs its own EC2 instance and enrollment identity. Launch another supported host with the same network and instance-profile configuration. Then run a new command from Self-hosted Compute → Add node. The project storage and Sandbox Proxy settings apply to all executors in the project. You do not need another NLB or Sandbox Proxy deployment for each host.

Updates and key rotation

The provisioner runs as a service on every enrolled executor. It checks Tensorlake for new approved profiles and applies complete updates. A failed update keeps the last valid revision active. To rotate the Sandbox Proxy mTLS and SSH keys, run prepare.sh again with a new output directory. Review and apply the new resources. The certificate fingerprint starts a rolling pod update. Then replace SSH proxy public key with the new outbound.key.pub value. Keep the previous output directory in secure storage until the rollout and end-to-end test pass.

Troubleshooting

Symptom	Check
The NLB is not created	Check the AWS Load Balancer Controller, its IAM role, and public subnet discovery.
An NLB target is unhealthy	Check pod readiness and traffic from the NLB security group to pod port `9080`.
The executor enrollment stops during host checks	Confirm Ubuntu `24.04`, root access, IMDSv2, and `/dev/kvm`.
The storage preflight fails	Check the EC2 instance profile, S3 actions, bucket policy, and AWS KMS permissions.
A sandbox remains pending	Confirm that the executor is ready and belongs to the same Tensorlake project.
The proxy cannot reach a sandbox	Check private routes and executor security-group rules for ports `8096`, `8097`, and `5565`.
HTTPS works but native SSH fails	Check the NLB listener on port `22`, its target group, and both SSH security-group paths.

​What Tensorlake manages

​What you manage

​Before you begin

​Tensorlake requirements

​AWS account requirements

​Sandbox base domain

​1. Prepare project storage

​2. Prepare the AWS network

​Subnet layout

​Load balancer security group

​Executor security group

​Outbound access

​3. Create the public certificate

​4. Prepare the Sandbox Proxy

​5. Configure the Tensorlake project

​6. Deploy the Sandbox Proxy

​7. Create DNS records and activate the endpoint

​8. Launch the first executor

​9. Enroll the executor

​10. Verify the complete path

​Add more executors

​Updates and key rotation

​Troubleshooting