Tensorlake Documentation

GPU sandboxes are available by request. Reach out on Slack to get access. They run on gVisor rather than the Firecracker and Cloud Hypervisor VMs that regular sandboxes use, which brings the following constraints:

CPU and memory can’t be live resized
Tensorlake filesystems (TLFS) can’t be mounted
Only work with CAS images

Example: create and use a vLLM sandbox

This example creates a GPU sandbox from Tensorlake’s vLLM image, starts a vLLM server in it, sends one prompt, and prints the reply. You can also build or import your own image.

Create the sandbox

SANDBOX_ID=$(tl sbx create --image tensorlake/cas/vllm:0.26 --gpu H100 \
  --cpus 4 --memory 32768 --disk_mb 102400)

Start the vLLM server

Start vLLM in the foreground. It downloads the model from Hugging Face, loads it onto the GPU, and streams its output to your terminal:

tl sbx exec $SANDBOX_ID -e PYTHONUNBUFFERED=1 \
  python3 -m vllm.entrypoints.openai.api_server \
    --model HuggingFaceTB/SmolLM2-135M-Instruct \
    --served-model-name smollm2 --host 127.0.0.1 --port 8000 \
    --max-model-len 4096 --gpu-memory-utilization 0.85 --enforce-eager

When the output shows Application startup complete, press Ctrl+C. That stops the stream, not the server: vLLM keeps running in the sandbox.

Send a prompt

The server listens on loopback, so send the request from inside the sandbox. Terminate the sandbox afterwards to release its GPU.

tl sbx exec $SANDBOX_ID python3 -c '
import json, urllib.request
request = urllib.request.Request(
    "http://127.0.0.1:8000/v1/chat/completions",
    data=json.dumps({"model": "smollm2", "messages": [
        {"role": "user", "content": "Say hello in exactly three words."}]}).encode(),
    headers={"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(request))["choices"][0]["message"]["content"])
'

tl sbx terminate $SANDBOX_ID