Run Terminal-Bench and other Harbor evals on Vercel Sandbox
Vercel Sandbox now supports running Harbor evaluations, including Terminal-Bench and other benchmarks, with enhanced isolation and credential handling via Firecracker microVMs and AI Gateway integration.
Vercel Sandbox now supports running Harbor evaluations, including Terminal-Bench and other benchmarks such as SWE-bench, tau3-bench, and OSWorld. Users can execute trials in isolated Firecracker microVMs by passing the --env vercel flag to harbor run, enabling parallelization beyond local machine capabilities. Each task's network policy is enforced at the sandbox firewall, ensuring security outside the VM environment.
Optional credential injection allows secrets to be attached to outbound requests at the firewall, preventing them from entering the sandbox. This feature enhances security by isolating sensitive data while enabling benchmarking workflows. The integration supports dynamic model switching, allowing users to target different AI providers with minimal configuration changes.
Paired with AI Gateway, a single AI_GATEWAY_API_KEY grants access to hundreds of models across multiple providers. Benchmarking a new model requires only changing the --model flag, such as vercel_ai_gateway/openai/gpt-5.6-luna for OpenAI models. This simplifies cross-provider evaluation without additional setup overhead.
The feature requires Harbor version 0.22.0 or later and is documented in a step-by-step guide covering setup, configuration, and troubleshooting. Detailed instructions are available in the Sandbox documentation for users seeking to implement these evaluations efficiently.