Benchmark
Benchmark Overview
What the SOMA benchmark measures and how submissions are evaluated.
SOMA Bench
SOMA Bench runs SWE-bench style tasks through agent harness backends (Copilot and Claude Code) and records runtime artifacts, patch output, token usage, and optional SWE-bench evaluation results.
Pick the harness with --agent-name (benchmark-solve) or --runtime-backend (benchmark-run-infer): copilot (the benchmark-run-infer default) or claude_code.
Common options
Both harnesses run on the same runner (harness_base), so these flags work the same for each of them:
| Flag | Env var | Purpose |
|---|---|---|
--run-id-header-value | SOMA_RUN_ID_HEADER_VALUE | Send this run id to the upstream LLM gateway in place of the provider API key; the gateway resolves authorization server-side |
--ignore-api-key | - | Skip local LLM API key validation (use together with --run-id-header-value) |
--compression-script-path | SOMA_COMPRESSION_SCRIPT_PATH | Compressor/miner script mounted into the compression-service sidecar |
--compression-service-autobuild | SOMA_COMPRESSION_SERVICE_AUTOBUILD | Rebuild the compression-service image before the run |
--compression-service-build-context | SOMA_COMPRESSION_SERVICE_BUILD_CONTEXT | Build context for that rebuild |
--use-host-docker-socket | SOMA_USE_HOST_DOCKER_SOCKET | Insecure fallback: reach the SWE sandbox through the host Docker socket instead of an isolated dind daemon |
The matching legacy SOMA_COPILOT_* env vars (for example SOMA_COPILOT_COMPRESSION_SCRIPT_PATH) are
still read as a fallback after the neutral ones.
--benchmark-type selects which task mode the agent runs in. If omitted, it defaults to
swebench_verified. Accepted values:
swebench_verified(default) — standard SWE-bench Verified solve mode: the agent gets the issue's problem statement (and hints) and must produce a patch.swe_explorer_explore— SWE-Explorer file exploration mode: the agent explores the repo and outputs the regions it read, without producing a patch.swe_explorer_edit— SWE-Explorer edit mode: the agent receives the ground-truth modified files as a hint and must produce a patch.
Run Copilot
The Copilot backend runs the official @github/copilot CLI inside a sandboxed Docker Compose
stack (copilot service + an internal proxy sidecar + an optional compression-service
sidecar), pointed at your own LLM endpoint instead of real GitHub Copilot auth
(COPILOT_OFFLINE=true + a custom provider base URL). It works fully standalone — no SOMA
validator or plugin repo required.
Prerequisites
The Compose stack expects these images to already exist locally (they are not pulled from a
registry — pull_policy: never):
cd src/soma_bench/benchmark/backends/copilot/copilot-cli-container
docker compose build copilot # builds local/copilot-cli:latest
cd ../../../../../.. # back to repo root
docker build -t soma-copilot-compression-service:latest src/compression_service(If the images already exist — check with docker images | grep -E "copilot-cli|compression-service"
— you can skip this step.)
Required .env settings
The runner resolves LLM credentials with this precedence (first non-empty wins):
| Purpose | Env vars (in priority order) |
|---|---|
| API key | LLM_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY |
| Model | LLM_MODEL, OPENAI_MODEL, OPENROUTER_MODEL (or --model) |
| Base URL | LLM_BASE_URL, OPENAI_BASE_URL, OPENROUTER_BASE_URL (optional) |
| API version | LLM_API_VERSION (optional) |
At minimum you need one API key var and one model var set. For a standalone OpenRouter run:
cp .env.example .envOPENROUTER_API_KEY=sk-or-...
LLM_MODEL=openrouter/qwen/qwen3-coder
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1HF_TOKEN is only needed if the benchmark dataset is a gated/private Hugging Face dataset.
Standalone run
Run one instance directly, without any pre-built manifest:
source .env && uv run python -m soma_bench benchmark-solve \
--agent-name copilot \
--benchmark SWE-bench/SWE-bench_Verified \
--instance-id INSTANCE_ID \
--execute \
--output-dir outputs/run-copilot-standalone \
--swerebench-evalDrop --swerebench-eval to only collect the patch/trajectory without running the SWE-bench
evaluation harness afterward. Drop --execute to only write the benchmark scaffold
(benchmark-manifest.jsonl, image-plan.json) without actually launching the container.
Pointing at a compression/miner script
The compression-service sidecar is enabled by default on every Copilot run (it sits
between the proxy and the upstream LLM). If you don't pass a script, it silently mounts the
no-op fallback at src/compression_service/app/base_miner.py — an identity function that
returns messages unchanged:
def compress_messages(messages=None, path=None, metadata=None) -> list:
...
return messages if isinstance(messages, list) else []To exercise your own compressor/miner, point at your script explicitly. It must be importable
as a module exposing compress_messages(messages, path, metadata) -> list:
uv run python -m soma_bench benchmark-solve \
--agent-name copilot \
--benchmark SWE-bench/SWE-bench_Verified \
--instance-id INSTANCE_ID \
--execute \
--compression-script-path /absolute/path/to/your_miner.py \
--compression-service-autobuild \
--output-dir outputs/run-copilot-with-compression \
--swerebench-eval--compression-script-path(orSOMA_COMPRESSION_SCRIPT_PATH) is bind-mounted read-only into the sidecar as/app/miner/base_miner.py, overriding the default.--compression-service-autobuildrebuilds thecompression-serviceimage fromsrc/compression_servicebefore the run — use it wheneverrequirements.txtor the service code (not just your mounted miner script) changed.--compression-service-build-contextoverrides the build context directory if your service image lives elsewhere.- To skip the sidecar entirely (bypass compression, talk to the LLM directly through the proxy),
set
SOMA_COPILOT_USE_COMPOSE_COMPRESSION_SERVICE=false. --swerebench-evalruns the actual SWE-bench evaluation harness (swebench.harness.run_evaluation) against the patch Copilot produced — applying it to the instance's test environment and checkingFAIL_TO_PASS/PASS_TO_PASSoutcomes — right after the agent finishes, instead of only capturing the raw diff. Results land inevaluation-summary.json(patch_evaluation.resolved_count, etc.) in the output directory. It's a no-op if the agent produced no patch (patch_capture.has_changesisfalse). Related flags let you point at a different SWE-bench harness fork/Python interpreter
Useful SOMA_COPILOT_* overrides
All of these are optional; the CLI flags shown earlier take precedence over the matching env var, which takes precedence over the built-in default.
| Env var | Default | Purpose |
|---|---|---|
SOMA_COPILOT_COMPOSE_FILE | copilot-cli-container/docker-compose.yml | Alternate Compose file |
SOMA_COPILOT_SERVICE | copilot | Compose service name for the CLI container |
SOMA_COPILOT_ARGS | --allow-all --no-ask-user | Extra args passed to the copilot CLI |
SOMA_COPILOT_OUTPUT_FORMAT | json | json or text CLI output |
SOMA_COPILOT_NETWORK_ISOLATION | true | Run the CLI container without direct internet, routed through the proxy sidecar |
SOMA_COPILOT_KEEP_STACK | false | Keep the Compose stack up after the run instead of tearing it down |
SOMA_COPILOT_SWE_SANDBOX | true | Start a read-only sandbox container exposing the task image's repo path (/testbed for SWE-bench, /repo for SOMA task lists) |
SOMA_USE_HOST_DOCKER_SOCKET (--use-host-docker-socket) | false | By default the SWE sandbox runs inside an isolated dind daemon, unreachable from the host and invisible to it. Setting this bridges the agent to the sandbox via the host Docker socket instead - the agent then sees and can control every container on the host, not just its own sandbox. Legacy/insecure fallback only; use if the isolated dind daemon doesn't work in your environment. |
SOMA_COPILOT_SHARED_PROXY | true | Reuse one proxy/compression stack across a batch instead of one per instance |
Run SOMA task lists (tasks-*.jsonl)
SOMA task lists are not Hugging Face datasets: they are a local JSONL where every row names its
own pair of pre-built images — an env image the agent works in and a test image the run is
graded on. Because those images carry the environment and the graded test command, these
instances need neither the SWE-bench image-name conventions nor its evaluation harness (none of
their repos appear in its spec maps).
Cache the task list once, then run instances from it by name:
uv run python -m soma_bench benchmark-load-tasks \
--tasks /path/to/tasks-1.jsonl \
--benchmark soma-is-taskssource .env && uv run python -m soma_bench benchmark-solve \
--agent-name copilot \
--benchmark soma-is-tasks \
--instance-id NVIDIA__SkillSpector-208 \
--execute \
--output-dir outputs/run-soma-tasks \
--swerebench-eval--benchmark takes whatever name you cached under; benchmark-run-infer works the same way
against a manifest built from it. DOCKERHUB_USERNAME/DOCKERHUB_TOKEN in .env are required
when the task images live in a private repo — they are used for the agent's env image and the
grading image alike, without touching ~/.docker/config.json.
Three things follow the images rather than SWE-bench defaults, automatically:
- Repo path. The agent's working directory, the sandbox mount and
swe-execall take the path the image declares (/repofor these lists) instead of/testbed. Override withSOMA_COPILOT_SWE_SANDBOX_REPO_PATHif you need to. - Grading.
--swerebench-evalruns the task's ownrun_testsscript inside itstestimage with the agent's patch applied, and checksFAIL_TO_PASS/PASS_TO_PASSagainst the pytest JSON report that run produces. Results land inmetadata.patch_evaluationand aggregate intoevaluation-summary.jsonexactly like SWE-bench runs do. A graded test id the report never mentions counts as a failure and is listed underpatch_evaluation.missing_tests. - Grading isolation. The grading container runs on
nonenetworking; setSOMA_TASK_EVAL_NETWORKif a repo's tests genuinely need a network.
Output layout
Per instance, under --output-dir:
instances/<instance_id>/copilot/{command.txt,prompt.txt,stdout.log,stderr.log}— exact CLI invocation and raw outputcopilot-trajectory-<instance_id>.jsonl— structured trajectoryoutput.jsonl— one row per instance, with full run metadata (compose_project,compression_script_path,swe_sandbox_image,exit_code, etc.) undermetadataevaluation-summary.json(only with--swerebench-eval) —patch_capture.rows_with_changestells you whether Copilot actually produced a diff before you look at resolution rate; a run can finish withstatus: completedand still have made zero code changes.
Run Claude Code
The Claude Code backend runs Anthropic's claude CLI through the Claude Agent SDK for
Python on the Copilot backend's
sandbox stack: the same run (run_harness_instance in backends/harness_base.py) with a claude-code
service in place of the copilot one. Checkout, workspace seeding, the isolated dind daemon
and task sandbox, the swe-exec broker, the proxy and compression-service sidecars, patch
capture and evaluation are all shared, and configured by the same SOMA_COPILOT_* env vars /
copilot_* runtime options listed above (network isolation, SWE sandbox, keep stack, run id,
compression script, ...). Only the agent itself is configured separately, below.
The SDK is a library rather than a command, so the container's entrypoint is
claude-code-container/agent_runner.py, which turns a prompt into a JSONL trajectory on stdout.
Build the images
cd src/soma_bench/benchmark/backends/claude_code/claude-code-container
docker compose build claude-code # builds local/claude-code-agent:latest
# the proxy/compression sidecars are the same service the Copilot stack runs, so if you
# already built it for Copilot there is nothing to do here:
docker build -t soma-copilot-compression-service:latest ../../../../../compression_serviceRun it
uv run python -m soma_bench benchmark-solve \
--agent-name claude_code \
--benchmark princeton-nlp/SWE-bench_Verified \
--instance-id <instance_id> \
--execute \
--output-dir outputs/run-claude-codePointing it at your provider
Claude Code speaks the Anthropic Messages API: it posts to /v1/messages, appending that
path to the base URL itself. Two consequences for LLM_BASE_URL:
- The backend hands the proxy the upstream with one trailing
/v1removed, so anLLM_BASE_URLwritten for the OpenAI-shaped backends (https://openrouter.ai/api/v1) reaches the provider's Anthropic endpoint athttps://openrouter.ai/api/v1/messages. SetSOMA_CLAUDE_CODE_UPSTREAM_BASE_URLwhen the provider's Anthropic-format endpoint does not sit at the root of its OpenAI-format one. - The upstream must actually serve an Anthropic-format endpoint. OpenRouter does (its
"Anthropic skin"); a plain OpenAI-compatible endpoint does not, and returns 404. Any model the
provider serves works — the model id goes through as
ANTHROPIC_MODEL.
The credential travels as ANTHROPIC_AUTH_TOKEN, which Claude Code sends as
Authorization: Bearer — the header the proxy and the sticky router already read. Streaming is
not optional for Claude Code, and it aborts a response that goes silent for 300s, so the proxy
relays text/event-stream bodies chunk by chunk instead of buffering them, tallying token usage
off the events as they pass.
Useful SOMA_CLAUDE_CODE_* overrides
| Env var | Default | Purpose |
|---|---|---|
SOMA_CLAUDE_CODE_COMPOSE_FILE | claude-code-container/docker-compose.yml | Alternate Compose file. Its sidecars, networks and volumes must match the Copilot one's. |
SOMA_CLAUDE_CODE_SERVICE | claude-code | Compose service name for the agent container |
SOMA_CLAUDE_CODE_UPSTREAM_BASE_URL | derived from llm.base_url | Anthropic-format upstream the proxy forwards to |
SOMA_CLAUDE_CODE_PERMISSION_MODE (--claude-code-permission-mode) | bypassPermissions | A benchmark run has nobody to answer a permission prompt |
SOMA_CLAUDE_CODE_MAX_TURNS (--claude-code-max-turns) | unlimited | Turn ceiling handed to the SDK. Unset, the agent runs until it finishes or SOMA_HARNESS_AGENT_TIMEOUT_SECONDS fires, as Copilot does - --max-iterations is not used by either harness. |
SOMA_CLAUDE_CODE_DISALLOWED_TOOLS (--claude-code-disallowed-tools) | Workflow, Cron*, and other session-management tools | Replaces the deny list. Workflow in particular can fan out agents that all bill through the same proxy, so the instance's token total would stop describing the trajectory it is scored on. WebSearch/WebFetch are denied on top of this whenever the run is network-isolated. Pass an empty string to allow everything. |
PROXY_OPENAI_TRANSLATION | 1 unless the model is Anthropic's | Have the proxy translate Claude Code's Anthropic Messages requests to OpenAI chat completions upstream (and the responses back) |
SOMA_HARNESS_AGENT_TIMEOUT_SECONDS (runtime option harness_agent_timeout_seconds) is a
wall-clock ceiling shared by every harness - Copilot and Claude Code; the run is
reported as timeout and patch capture is skipped. It takes precedence over
SOMA_COPILOT_AGENT_TIMEOUT_SECONDS.
Output layout
instances/<instance_id>/claude-code/{command.txt,prompt.txt,stdout.log,stderr.log}claude-code-trajectory-<instance_id>.jsonl— one JSON object per SDK message, each tagged with the class it came from (AssistantMessage,ResultMessage, ...) andrunner.*lines for the container entrypoint's own lifecyclemetadata.token_usageis what the proxy counted crossing the compression boundary — the number the benchmark scores.metadata.sdk_token_usageis what the agent itself reported; when the two disagree, that difference is the finding.
The run's exit status distinguishes an agent failure from a harness failure: exit 1 means Claude Code ran and ended in an error result (an API failure, a turn-limit stop), exit 2 means the container could not run it at all.
README.md