Miner
SWE Scoring
This document describes the current SWE scoring logic implemented in mcpplatformservices/common/app/api/routes/scoring.py.
Task types and scoring paths
There is one benchmark task type, swebench_verified: the agent is given an issue and
must produce a patch, which is graded against the task's own tests.
A competition mixes two kinds of task under it, which differ only in where the instance
and its images come from — screener stage 1 uses the public SWE-bench Verified dataset,
while screener stage 2 and full evaluation use SOMA task lists whose rows ship their own
env/test images. Both score through the same path below:
compute_swe_task_score, build_swe_miner_scores, and build_swe_miner_total_score.
Where the graded result comes from differs per kind, and is the validator's concern rather than scoring's:
- SWE-bench Verified instances are graded by the SWE-bench harness
(
validator/evaluation/swebench_evaluator.py). - SOMA tasks are graded by running the task's own test image
(
validator/evaluation/soma_task_evaluator.py).
Either way the validator reports one resolved boolean per run, which is what this
document's x and y counts are built from.
Task complexity and layered incentives
Each competition task is classified into one of three complexity groups:
| Group | Meaning |
|---|---|
short | Least expected agent effort |
medium | Intermediate expected agent effort |
long | Greatest expected agent effort |
Complexity does not alter the per-task formula or a miner's overall SWE score. Every task continues to use the scoring path described below. Complexity instead controls the incentive contests used to allocate competition weight.
Layers
When all three groups are present, winners are selected over these layers:
| Layer | Groups compared | Layer weight | Weight per group set |
|---|---|---|---|
| All groups | {short, medium, long} | 0.25 | 0.25 |
| Group pairs | {short, medium}, {short, long}, {medium, long} | 0.45 | 0.15 |
| Individual groups | {short}, {medium}, {long} | 0.30 | 0.10 |
For a group set, a miner's contest score is the plain average of its score in each
included group. The short, medium, and long groups have equal weight.
The highest-scoring miner or tied miners win each group set. Tied winners split that
set's weight equally. A miner needs a score in every group in a set to compete for
that set. For example, strong performance only on short tasks can win the short
individual-group contest, while strong and consistent performance across all three
groups is required to compete for the all-groups contest.
If a competition contains fewer than three groups, only the available group sets are used and their layer weights are renormalized. Tasks without a complexity label still contribute to the miner's SWE score, but do not participate in a complexity contest.
See Incentive Mechanism for the complete allocation and emission calculation.
Token counting
Token totals are computed from a weighted token-type breakdown.
Let:
T_ibe non-cached input tokens,T_cbe cached input tokens,T_obe output tokens,T_jbe Jev input tokens spent by the miner's compressor (see below).
Default weights:
| Token type | Weight |
|---|---|
| Input, non-cached | 1.0 |
| Cached input | 0.1 |
| Output | 3.0 |
| Jev input (compressor services) | 0.2 |
Compressor services (Jev)
A compressor may call Jev, the TypeSafe decision model, through the benchmark proxy
(the compressor itself has no network access). What those calls cost is recorded per
run apart from the agent's tokens - T_i, T_c and T_o never include it - and
enters the run's weighted total as T_j. Jev bills input only.
Only miner runs have T_j: a baseline run uses no compressor, so its weighted total is
the agent's tokens alone. A compressor that calls Jev therefore has to save more agent
tokens than its Jev calls cost. The same total is used by the screening gates and the
score.
Current behavior of compute_weighted_tokens:
input_tokensandoutput_tokensare required; Jev input tokens are added on top and never stand in for them.- Missing
cached_input_tokensis treated as0. - The function returns
Noneif a required value is missing or any supplied token count is negative.
SWE path
Per-task scoring inputs
For each task:
xis the number of resolved baseline runs. Only resolved baselines count.yis the number of resolved miner runs.T_Bis the average weighted token count across baseline runs with valid weighted token counts.T_Ais the average weighted token count across miner runs with valid, positive weighted token counts.nis the total run count for the task: the larger of the baseline run count and the miner run count (and never less thanx,yor1).
Compression ratio
The compression-ratio term is the base-2 logarithm of the baseline-to-miner
weighted-token ratio, clamped to [-2, 2]:
If the token inputs are invalid:
Quality threshold
The threshold is the smallest resolved-run count that reaches 80% of baseline quality:
Per-task score
The per-task score is calculated by compute_swe_task_score. Every per-task
score lies in [-2, 2].
| Constant | Value |
|---|---|
| Score range | [-2, 2] |
| Penalty-only quality ratio | 0.5 |
| Full token-score quality ratio | 0.8 |
| Bonus maximum | 0.1 |
| Bonus minimum extra resolved runs | 2 |
Bonus component
The bonus is a small, capped tie-break reward for clearly beating the baseline's quality. It is zero unless the miner resolves at least two more runs than the baseline:
Otherwise, progress is measured against the remaining headroom above the
baseline, with a headroom of at least 2:
The bonus only starts once the miner clears half of that headroom, and grows
quadratically up to 0.1:
A task is reported in the bonus zone only when b > 0; otherwise it is
reported as maintain.
Hard tasks
A task is hard when x <= 1.
Excluded task
If y == 0:
score=Nonepool=excluded
The task does not contribute to the miner aggregate.
Maintain zone
If x == 1 and y == 1:
Bonus zone
All other non-excluded hard tasks use:
Hard tasks are assigned to pool=hard_boost.
Their hard-boost contribution is:
Only the positive part of the hard-task score contributes to the boost.
Standard tasks
A task is standard when x >= 2.
When y <= x, the score blends a quality penalty with the compression ratio
according to the quality ratio:
Penalty zone
If q <= 0.5, only the quality penalty applies. The score rises linearly from
-2 at zero resolved runs to -1 at 50% of baseline quality:
If 0.5 < q < 0.8 (equivalently y < t), the score moves linearly from -1
toward the compression ratio:
Both cases are reported as zone=penalty.
Maintain zone
If t <= y <= x (80% of baseline quality or better), quality contributes no
penalty and the compression ratio applies in full:
Bonus zone
If y > x:
Standard tasks are assigned to pool=main.
SWE miner aggregation
Miner-level aggregation is performed by build_swe_miner_scores.
Main score
For every task in pool=main, the aggregation weight is:
The main_score is the weighted average of main-task scores:
If the miner has no tasks in pool=main:
Hard boost
The hard_boost is the sum of positive hard-task contributions divided by the
number of scored main and hard tasks:
Where:
N_Mis the number of tasks inpool=main.N_His the number of tasks inpool=hard_boost.h_iis the hard-boost contribution of hard taski.
If there are no hard-boost contributions:
Raw miner total
Final normalized SWE score
Final normalization is performed by build_swe_miner_total_score.
First, the raw total is clamped to [-2, 2]:
The clamped value is then linearly normalized from [-2, 2] to [-1, 1]:
Equivalently:
This normalized value is consumed by downstream category and leaderboard scoring.
docs/miner/scoring.md