sandbox-probe: AI Agent Sandbox Exposure Matrix & Benchmark Reports

AI coding agents execute arbitrary code on developer workstations, with each harness sandbox defining its own security boundaries. sandbox-probe is ControlPlane’s continuous exposure detection tool for evaluating the default security configuration that these sandboxes ship with, whether intentionally or otherwise.

sandbox-probe is a static binary executed within the sandbox environment to assess host visibility and breakout risk. The companion sandbox-probe-reports runs those probes across the latest version of each sandbox to compare security postures.

The live reports and GitHub repository record what each agent permits, blocks or leaks on Linux, macOS and Windows. The release post introduces the project.

The AI Agent Sandbox Exposure Matrix

Every row is normalised against an unconfined baseline scan from the same run on the same OS: a cell is marked leaked when the baseline could do the same thing, blocked when the sandbox constrained a probe that the baseline had open, and n/a when there was nothing to block. Unmeasured categories are marked unprovable.

Eight Capability Categories

The matrix scores each run across eight capability categories:

1. Filesystem Read: Checks whether security-sensitive files are readable, including SSH keys and config (~/.ssh), cloud credentials (~/.aws, ~/.azure, ~/.gcloud, ~/.config/gcloud), the GPG keyring (~/.gnupg), and system files such as /etc/sudoers and /root/.bash_history.
2. Filesystem Write: Evaluates system and home paths for write access using access checks (the probe never writes to them), and separates paths on persistent storage from those backed by ephemeral filesystems such as tmpfs.
3. Network Egress: Tests whether public hostnames (e.g. google.com) resolve from inside the sandbox, as a signal of outbound network access.
4. Local Services: Reads the kernel’s table of listening sockets, then attempts a TCP or UDP connection to each listener from inside the sandbox, so a port that is visible but unreachable is not counted as exposed.
5. IPC Sockets: Enumerates Unix domain sockets visible under runtime directories (/run, /var/run, /tmp, and their macOS equivalents), and Windows named pipes. Seeded decoys stand in for the Docker daemon socket, an SSH agent, the D-Bus system bus, gpg-agent, and editor IPC sockets, so the result does not depend on what happens to be running on the test host.
6. Process Visibility: Records which host processes are visible from inside the sandbox and which process launched the agent, along with Linux namespace membership.
7. Mounts: Records which filesystems are mounted and visible from inside the sandbox, to show how much of the host filesystem is exposed.
8. Privileged: Scored as leaked when the agent’s process runs with an effective UID of root. Kernel-reported enforcement mechanisms (seccomp, no_new_privs, Landlock, user namespaces) are recorded alongside each run as context.

Agents and Sandboxes Tested

Each agent is the real CLI, with its model replaced by a stub that makes one tool call to run the probe. The results measure the agent’s sandbox, not the model’s behaviour.

• Agents with a built-in sandbox, measured on and off: Claude Code (Bubblewrap on Linux, Seatbelt on macOS), OpenAI Codex CLI (also run under a published nono profile), GitHub Copilot CLI, Gemini CLI (macOS Seatbelt sandbox-exec and Docker modes), and Trae Agent (Docker mode).
• Agents with no built-in sandbox, measured as shipped: Cline, Goose, gptme, OpenCode, and the Pi coding agent.
• Standalone sandbox runtimes: Anthropic’s Sandbox Runtime (srt), nono, and Firejail, each run with its vendor-default policy, widened only enough for the probe to write its report.

Tracking Changes Over Time

Agents, harnesses, sandboxes and kernels may change security posture from one release to the next, so the scan matrix runs on a schedule on GitHub Actions runners for Linux, macOS and Windows, and records the agent and sandbox versions with every result. The reports site plots exposure over time and ties each change in a capability to the tested version.

• Live reports: https://controlplaneio.github.io/sandbox-probe-reports/
• Repository: https://github.com/controlplaneio/sandbox-probe-reports
• Release post: sandbox-probe: Putting AI sandboxing to the test