Benchmark

Benchmark Overview

What the SOMA benchmark measures and how submissions are evaluated.

SOMA Bench

SOMA Bench runs SWE-bench style tasks through agent harness backends (Copilot and Claude Code) and records runtime artifacts, patch output, token usage, and optional SWE-bench evaluation results.

Pick the harness with --agent-name (benchmark-solve) or --runtime-backend (benchmark-run-infer): copilot (the benchmark-run-infer default) or claude_code.

Common options

Both harnesses run on the same runner (harness_base), so these flags work the same for each of them:

FlagEnv varPurpose
--run-id-header-valueSOMA_RUN_ID_HEADER_VALUESend this run id to the upstream LLM gateway in place of the provider API key; the gateway resolves authorization server-side
--ignore-api-key-Skip local LLM API key validation (use together with --run-id-header-value)
--compression-script-pathSOMA_COMPRESSION_SCRIPT_PATHCompressor/miner script mounted into the compression-service sidecar
--compression-service-autobuildSOMA_COMPRESSION_SERVICE_AUTOBUILDRebuild the compression-service image before the run
--compression-service-build-contextSOMA_COMPRESSION_SERVICE_BUILD_CONTEXTBuild context for that rebuild
--use-host-docker-socketSOMA_USE_HOST_DOCKER_SOCKETInsecure fallback: reach the SWE sandbox through the host Docker socket instead of an isolated dind daemon

The matching legacy SOMA_COPILOT_* env vars (for example SOMA_COPILOT_COMPRESSION_SCRIPT_PATH) are still read as a fallback after the neutral ones.

--benchmark-type selects which task mode the agent runs in. If omitted, it defaults to swebench_verified. Accepted values:

  • swebench_verified (default) — standard SWE-bench Verified solve mode: the agent gets the issue's problem statement (and hints) and must produce a patch.
  • swe_explorer_explore — SWE-Explorer file exploration mode: the agent explores the repo and outputs the regions it read, without producing a patch.
  • swe_explorer_edit — SWE-Explorer edit mode: the agent receives the ground-truth modified files as a hint and must produce a patch.

Run Copilot

The Copilot backend runs the official @github/copilot CLI inside a sandboxed Docker Compose stack (copilot service + an internal proxy sidecar + an optional compression-service sidecar), pointed at your own LLM endpoint instead of real GitHub Copilot auth (COPILOT_OFFLINE=true + a custom provider base URL). It works fully standalone — no SOMA validator or plugin repo required.

Prerequisites

The Compose stack expects these images to already exist locally (they are not pulled from a registry — pull_policy: never):

cd src/soma_bench/benchmark/backends/copilot/copilot-cli-container
docker compose build copilot          # builds local/copilot-cli:latest

cd ../../../../../..                  # back to repo root
docker build -t soma-copilot-compression-service:latest src/compression_service

(If the images already exist — check with docker images | grep -E "copilot-cli|compression-service" — you can skip this step.)

Required .env settings

The runner resolves LLM credentials with this precedence (first non-empty wins):

PurposeEnv vars (in priority order)
API keyLLM_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY
ModelLLM_MODEL, OPENAI_MODEL, OPENROUTER_MODEL (or --model)
Base URLLLM_BASE_URL, OPENAI_BASE_URL, OPENROUTER_BASE_URL (optional)
API versionLLM_API_VERSION (optional)

At minimum you need one API key var and one model var set. For a standalone OpenRouter run:

cp .env.example .env
OPENROUTER_API_KEY=sk-or-...
LLM_MODEL=openrouter/qwen/qwen3-coder
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1

HF_TOKEN is only needed if the benchmark dataset is a gated/private Hugging Face dataset.

Standalone run

Run one instance directly, without any pre-built manifest:

source .env && uv run python -m soma_bench benchmark-solve \
  --agent-name copilot \
  --benchmark SWE-bench/SWE-bench_Verified \
  --instance-id INSTANCE_ID \
  --execute \
  --output-dir outputs/run-copilot-standalone \
  --swerebench-eval

Drop --swerebench-eval to only collect the patch/trajectory without running the SWE-bench evaluation harness afterward. Drop --execute to only write the benchmark scaffold (benchmark-manifest.jsonl, image-plan.json) without actually launching the container.

Pointing at a compression/miner script

The compression-service sidecar is enabled by default on every Copilot run (it sits between the proxy and the upstream LLM). If you don't pass a script, it silently mounts the no-op fallback at src/compression_service/app/base_miner.py — an identity function that returns messages unchanged:

def compress_messages(messages=None, path=None, metadata=None) -> list:
    ...
    return messages if isinstance(messages, list) else []

To exercise your own compressor/miner, point at your script explicitly. It must be importable as a module exposing compress_messages(messages, path, metadata) -> list:

uv run python -m soma_bench benchmark-solve \
  --agent-name copilot \
  --benchmark SWE-bench/SWE-bench_Verified \
  --instance-id INSTANCE_ID \
  --execute \
  --compression-script-path /absolute/path/to/your_miner.py \
  --compression-service-autobuild \
  --output-dir outputs/run-copilot-with-compression \
  --swerebench-eval
  • --compression-script-path (or SOMA_COMPRESSION_SCRIPT_PATH) is bind-mounted read-only into the sidecar as /app/miner/base_miner.py, overriding the default.
  • --compression-service-autobuild rebuilds the compression-service image from src/compression_service before the run — use it whenever requirements.txt or the service code (not just your mounted miner script) changed. --compression-service-build-context overrides the build context directory if your service image lives elsewhere.
  • To skip the sidecar entirely (bypass compression, talk to the LLM directly through the proxy), set SOMA_COPILOT_USE_COMPOSE_COMPRESSION_SERVICE=false.
  • --swerebench-eval runs the actual SWE-bench evaluation harness (swebench.harness.run_evaluation) against the patch Copilot produced — applying it to the instance's test environment and checking FAIL_TO_PASS/PASS_TO_PASS outcomes — right after the agent finishes, instead of only capturing the raw diff. Results land in evaluation-summary.json (patch_evaluation.resolved_count, etc.) in the output directory. It's a no-op if the agent produced no patch (patch_capture.has_changes is false). Related flags let you point at a different SWE-bench harness fork/Python interpreter

Useful SOMA_COPILOT_* overrides

All of these are optional; the CLI flags shown earlier take precedence over the matching env var, which takes precedence over the built-in default.

Env varDefaultPurpose
SOMA_COPILOT_COMPOSE_FILEcopilot-cli-container/docker-compose.ymlAlternate Compose file
SOMA_COPILOT_SERVICEcopilotCompose service name for the CLI container
SOMA_COPILOT_ARGS--allow-all --no-ask-userExtra args passed to the copilot CLI
SOMA_COPILOT_OUTPUT_FORMATjsonjson or text CLI output
SOMA_COPILOT_NETWORK_ISOLATIONtrueRun the CLI container without direct internet, routed through the proxy sidecar
SOMA_COPILOT_KEEP_STACKfalseKeep the Compose stack up after the run instead of tearing it down
SOMA_COPILOT_SWE_SANDBOXtrueStart a read-only sandbox container exposing the task image's repo path (/testbed for SWE-bench, /repo for SOMA task lists)
SOMA_USE_HOST_DOCKER_SOCKET (--use-host-docker-socket)falseBy default the SWE sandbox runs inside an isolated dind daemon, unreachable from the host and invisible to it. Setting this bridges the agent to the sandbox via the host Docker socket instead - the agent then sees and can control every container on the host, not just its own sandbox. Legacy/insecure fallback only; use if the isolated dind daemon doesn't work in your environment.
SOMA_COPILOT_SHARED_PROXYtrueReuse one proxy/compression stack across a batch instead of one per instance

Run SOMA task lists (tasks-*.jsonl)

SOMA task lists are not Hugging Face datasets: they are a local JSONL where every row names its own pair of pre-built images — an env image the agent works in and a test image the run is graded on. Because those images carry the environment and the graded test command, these instances need neither the SWE-bench image-name conventions nor its evaluation harness (none of their repos appear in its spec maps).

Cache the task list once, then run instances from it by name:

uv run python -m soma_bench benchmark-load-tasks \
  --tasks /path/to/tasks-1.jsonl \
  --benchmark soma-is-tasks
source .env && uv run python -m soma_bench benchmark-solve \
  --agent-name copilot \
  --benchmark soma-is-tasks \
  --instance-id NVIDIA__SkillSpector-208 \
  --execute \
  --output-dir outputs/run-soma-tasks \
  --swerebench-eval

--benchmark takes whatever name you cached under; benchmark-run-infer works the same way against a manifest built from it. DOCKERHUB_USERNAME/DOCKERHUB_TOKEN in .env are required when the task images live in a private repo — they are used for the agent's env image and the grading image alike, without touching ~/.docker/config.json.

Three things follow the images rather than SWE-bench defaults, automatically:

  • Repo path. The agent's working directory, the sandbox mount and swe-exec all take the path the image declares (/repo for these lists) instead of /testbed. Override with SOMA_COPILOT_SWE_SANDBOX_REPO_PATH if you need to.
  • Grading. --swerebench-eval runs the task's own run_tests script inside its test image with the agent's patch applied, and checks FAIL_TO_PASS/PASS_TO_PASS against the pytest JSON report that run produces. Results land in metadata.patch_evaluation and aggregate into evaluation-summary.json exactly like SWE-bench runs do. A graded test id the report never mentions counts as a failure and is listed under patch_evaluation.missing_tests.
  • Grading isolation. The grading container runs on none networking; set SOMA_TASK_EVAL_NETWORK if a repo's tests genuinely need a network.

Output layout

Per instance, under --output-dir:

  • instances/<instance_id>/copilot/{command.txt,prompt.txt,stdout.log,stderr.log} — exact CLI invocation and raw output
  • copilot-trajectory-<instance_id>.jsonl — structured trajectory
  • output.jsonl — one row per instance, with full run metadata (compose_project, compression_script_path, swe_sandbox_image, exit_code, etc.) under metadata
  • evaluation-summary.json (only with --swerebench-eval) — patch_capture.rows_with_changes tells you whether Copilot actually produced a diff before you look at resolution rate; a run can finish with status: completed and still have made zero code changes.

Run Claude Code

The Claude Code backend runs Anthropic's claude CLI through the Claude Agent SDK for Python on the Copilot backend's sandbox stack: the same run (run_harness_instance in backends/harness_base.py) with a claude-code service in place of the copilot one. Checkout, workspace seeding, the isolated dind daemon and task sandbox, the swe-exec broker, the proxy and compression-service sidecars, patch capture and evaluation are all shared, and configured by the same SOMA_COPILOT_* env vars / copilot_* runtime options listed above (network isolation, SWE sandbox, keep stack, run id, compression script, ...). Only the agent itself is configured separately, below.

The SDK is a library rather than a command, so the container's entrypoint is claude-code-container/agent_runner.py, which turns a prompt into a JSONL trajectory on stdout.

Build the images

cd src/soma_bench/benchmark/backends/claude_code/claude-code-container
docker compose build claude-code      # builds local/claude-code-agent:latest
# the proxy/compression sidecars are the same service the Copilot stack runs, so if you
# already built it for Copilot there is nothing to do here:
docker build -t soma-copilot-compression-service:latest ../../../../../compression_service

Run it

uv run python -m soma_bench benchmark-solve \
  --agent-name claude_code \
  --benchmark princeton-nlp/SWE-bench_Verified \
  --instance-id <instance_id> \
  --execute \
  --output-dir outputs/run-claude-code

Pointing it at your provider

Claude Code speaks the Anthropic Messages API: it posts to /v1/messages, appending that path to the base URL itself. Two consequences for LLM_BASE_URL:

  • The backend hands the proxy the upstream with one trailing /v1 removed, so an LLM_BASE_URL written for the OpenAI-shaped backends (https://openrouter.ai/api/v1) reaches the provider's Anthropic endpoint at https://openrouter.ai/api/v1/messages. Set SOMA_CLAUDE_CODE_UPSTREAM_BASE_URL when the provider's Anthropic-format endpoint does not sit at the root of its OpenAI-format one.
  • The upstream must actually serve an Anthropic-format endpoint. OpenRouter does (its "Anthropic skin"); a plain OpenAI-compatible endpoint does not, and returns 404. Any model the provider serves works — the model id goes through as ANTHROPIC_MODEL.

The credential travels as ANTHROPIC_AUTH_TOKEN, which Claude Code sends as Authorization: Bearer — the header the proxy and the sticky router already read. Streaming is not optional for Claude Code, and it aborts a response that goes silent for 300s, so the proxy relays text/event-stream bodies chunk by chunk instead of buffering them, tallying token usage off the events as they pass.

Useful SOMA_CLAUDE_CODE_* overrides

Env varDefaultPurpose
SOMA_CLAUDE_CODE_COMPOSE_FILEclaude-code-container/docker-compose.ymlAlternate Compose file. Its sidecars, networks and volumes must match the Copilot one's.
SOMA_CLAUDE_CODE_SERVICEclaude-codeCompose service name for the agent container
SOMA_CLAUDE_CODE_UPSTREAM_BASE_URLderived from llm.base_urlAnthropic-format upstream the proxy forwards to
SOMA_CLAUDE_CODE_PERMISSION_MODE (--claude-code-permission-mode)bypassPermissionsA benchmark run has nobody to answer a permission prompt
SOMA_CLAUDE_CODE_MAX_TURNS (--claude-code-max-turns)unlimitedTurn ceiling handed to the SDK. Unset, the agent runs until it finishes or SOMA_HARNESS_AGENT_TIMEOUT_SECONDS fires, as Copilot does - --max-iterations is not used by either harness.
SOMA_CLAUDE_CODE_DISALLOWED_TOOLS (--claude-code-disallowed-tools)Workflow, Cron*, and other session-management toolsReplaces the deny list. Workflow in particular can fan out agents that all bill through the same proxy, so the instance's token total would stop describing the trajectory it is scored on. WebSearch/WebFetch are denied on top of this whenever the run is network-isolated. Pass an empty string to allow everything.
PROXY_OPENAI_TRANSLATION1 unless the model is Anthropic'sHave the proxy translate Claude Code's Anthropic Messages requests to OpenAI chat completions upstream (and the responses back)

SOMA_HARNESS_AGENT_TIMEOUT_SECONDS (runtime option harness_agent_timeout_seconds) is a wall-clock ceiling shared by every harness - Copilot and Claude Code; the run is reported as timeout and patch capture is skipped. It takes precedence over SOMA_COPILOT_AGENT_TIMEOUT_SECONDS.

Output layout

  • instances/<instance_id>/claude-code/{command.txt,prompt.txt,stdout.log,stderr.log}
  • claude-code-trajectory-<instance_id>.jsonl — one JSON object per SDK message, each tagged with the class it came from (AssistantMessage, ResultMessage, ...) and runner.* lines for the container entrypoint's own lifecycle
  • metadata.token_usage is what the proxy counted crossing the compression boundary — the number the benchmark scores. metadata.sdk_token_usage is what the agent itself reported; when the two disagree, that difference is the finding.

The run's exit status distinguishes an agent failure from a harness failure: exit 1 means Claude Code ran and ended in an error result (an API failure, a turn-limit stop), exit 2 means the container could not run it at all.

Synced from DendriteHQ/SOMA-benchmark/README.md
Last updated 9 Nov 2026 by MattEdit this page on GitHub