What building Shippy taught us about building agents
Hugging Face’s Skylight team details the architecture of Shippy, a maritime AI agent designed for high-stakes operational decisions, emphasizing reliability, deterministic tooling, and sandboxed user sessions.
Shippy is a maritime AI agent built to assist analysts in real-time maritime domain awareness, where incorrect outputs could lead to costly operational errors. The system prioritizes reliability by structuring Shippy as a combination of a system prompt (soul), specific skills, and configuration settings. This design ensures predictable behavior within defined boundaries, with skills encoded as versioned markdown files and configuration managed separately for flexibility. The agent relies on a deterministic interface, the Skylight CLI, to interact with live data APIs, reducing errors in API calls and ensuring consistent outputs for tasks such as querying fishing activity or vessel tracks.
The agent’s architecture includes sandboxed hosting through Mothership, a platform that provisions isolated Kubernetes deployments for each user session. This ensures data isolation and prevents cross-user visibility, addressing security and privacy concerns for Skylight’s global users. Each session operates within strict network restrictions, allowing Shippy to perform multi-step analyses while maintaining user-specific data access. The CLI’s self-documenting design and structured output further enhance reliability by providing clear error messages and avoiding common pitfalls like buffer limits in piped data.
Evaluation of Shippy focuses on real-world performance rather than static benchmarks, using a custom framework that scores the agent holistically. Subject-matter experts design scenarios and rubrics tailored to operational tasks, weighting criteria such as data accuracy, boundary resolution, and source attribution. The evaluation pipeline runs tasks through a sandboxed Shippy session, with an LLM judge grading responses against ground truth annotations. Versioned builds are tested against live data, and regressions trigger further development to maintain performance standards.
Recent evaluations revealed areas for improvement, including instances where Shippy overstepped into tactical recommendations, misinterpreted geometry-sensitive queries, and occasionally generated invalid CLI commands. These findings directly inform updates to the agent’s skills and configurations. The team plans to refine guardrails, enhance boundary handling, and strengthen the CLI’s robustness to address these gaps, ensuring Shippy remains a reliable tool for high-stakes maritime operations.