Miner

SWE Scoring

This document describes the current SWE scoring logic implemented in mcpplatformservices/common/app/api/routes/scoring.py.

Task types and scoring paths

There is one benchmark task type, swebench_verified: the agent is given an issue and must produce a patch, which is graded against the task's own tests.

A competition mixes two kinds of task under it, which differ only in where the instance and its images come from — screener stage 1 uses the public SWE-bench Verified dataset, while screener stage 2 and full evaluation use SOMA task lists whose rows ship their own env/test images. Both score through the same path below: compute_swe_task_score, build_swe_miner_scores, and build_swe_miner_total_score.

Where the graded result comes from differs per kind, and is the validator's concern rather than scoring's:

  • SWE-bench Verified instances are graded by the SWE-bench harness (validator/evaluation/swebench_evaluator.py).
  • SOMA tasks are graded by running the task's own test image (validator/evaluation/soma_task_evaluator.py).

Either way the validator reports one resolved boolean per run, which is what this document's x and y counts are built from.

Task complexity and layered incentives

Each competition task is classified into one of three complexity groups:

GroupMeaning
shortLeast expected agent effort
mediumIntermediate expected agent effort
longGreatest expected agent effort

Complexity does not alter the per-task formula or a miner's overall SWE score. Every task continues to use the scoring path described below. Complexity instead controls the incentive contests used to allocate competition weight.

Layers

When all three groups are present, winners are selected over these layers:

LayerGroups comparedLayer weightWeight per group set
All groups{short, medium, long}0.250.25
Group pairs{short, medium}, {short, long}, {medium, long}0.450.15
Individual groups{short}, {medium}, {long}0.300.10

For a group set, a miner's contest score is the plain average of its score in each included group. The short, medium, and long groups have equal weight.

The highest-scoring miner or tied miners win each group set. Tied winners split that set's weight equally. A miner needs a score in every group in a set to compete for that set. For example, strong performance only on short tasks can win the short individual-group contest, while strong and consistent performance across all three groups is required to compete for the all-groups contest.

If a competition contains fewer than three groups, only the available group sets are used and their layer weights are renormalized. Tasks without a complexity label still contribute to the miner's SWE score, but do not participate in a complexity contest.

See Incentive Mechanism for the complete allocation and emission calculation.

Token counting

Token totals are computed from a weighted token-type breakdown.

Let:

  • T_i be non-cached input tokens,
  • T_c be cached input tokens,
  • T_o be output tokens,
  • T_j be Jev input tokens spent by the miner's compressor (see below).
T=wiTi+wcTc+woTo+wjTjT = w_i T_i + w_c T_c + w_o T_o + w_j T_j

Default weights:

Token typeWeight
Input, non-cached1.0
Cached input0.1
Output3.0
Jev input (compressor services)0.2

Compressor services (Jev)

A compressor may call Jev, the TypeSafe decision model, through the benchmark proxy (the compressor itself has no network access). What those calls cost is recorded per run apart from the agent's tokens - T_i, T_c and T_o never include it - and enters the run's weighted total as T_j. Jev bills input only.

Only miner runs have T_j: a baseline run uses no compressor, so its weighted total is the agent's tokens alone. A compressor that calls Jev therefore has to save more agent tokens than its Jev calls cost. The same total is used by the screening gates and the score.

Current behavior of compute_weighted_tokens:

  • input_tokens and output_tokens are required; Jev input tokens are added on top and never stand in for them.
  • Missing cached_input_tokens is treated as 0.
  • The function returns None if a required value is missing or any supplied token count is negative.

SWE path

Per-task scoring inputs

For each task:

  • x is the number of resolved baseline runs. Only resolved baselines count.
  • y is the number of resolved miner runs.
  • T_B is the average weighted token count across baseline runs with valid weighted token counts.
  • T_A is the average weighted token count across miner runs with valid, positive weighted token counts.
  • n is the total run count for the task: the larger of the baseline run count and the miner run count (and never less than x, y or 1).

Compression ratio

The compression-ratio term is the base-2 logarithm of the baseline-to-miner weighted-token ratio, clamped to [-2, 2]:

r=max⁡(−2,min⁡(log⁡2(TBTA),2))r = \max\left(-2,\min\left(\log_2\left(\frac{T_B}{T_A}\right),2\right)\right)

If the token inputs are invalid:

r=0r = 0

Quality threshold

The threshold is the smallest resolved-run count that reaches 80% of baseline quality:

t=⌈0.8x⌉t = \left\lceil 0.8x \right\rceil

Per-task score

The per-task score is calculated by compute_swe_task_score. Every per-task score lies in [-2, 2].

ConstantValue
Score range[-2, 2]
Penalty-only quality ratio0.5
Full token-score quality ratio0.8
Bonus maximum0.1
Bonus minimum extra resolved runs2

Bonus component

The bonus is a small, capped tie-break reward for clearly beating the baseline's quality. It is zero unless the miner resolves at least two more runs than the baseline:

b=0if y−x<2b = 0 \quad \text{if } y - x < 2

Otherwise, progress is measured against the remaining headroom above the baseline, with a headroom of at least 2:

p=y−xmax⁡(2, n−x)p = \frac{y-x}{\max(2,\,n-x)}

The bonus only starts once the miner clears half of that headroom, and grows quadratically up to 0.1:

b={0p≤0.50.1(p−0.50.5)2p>0.5b = \begin{cases} 0 & p \le 0.5 \\ 0.1\left(\dfrac{p-0.5}{0.5}\right)^2 & p > 0.5 \end{cases}

A task is reported in the bonus zone only when b > 0; otherwise it is reported as maintain.

Hard tasks

A task is hard when x <= 1.

Excluded task

If y == 0:

  • score=None
  • pool=excluded

The task does not contribute to the miner aggregate.

Maintain zone

If x == 1 and y == 1:

s=rs = r

Bonus zone

All other non-excluded hard tasks use:

s=max⁡(−2,min⁡(r+b,2))s = \max\left(-2,\min\left(r+b,2\right)\right)

Hard tasks are assigned to pool=hard_boost.

Their hard-boost contribution is:

h=max⁡(0,s)h = \max(0,s)

Only the positive part of the hard-task score contributes to the boost.

Standard tasks

A task is standard when x >= 2.

When y <= x, the score blends a quality penalty with the compression ratio according to the quality ratio:

q=yxq = \frac{y}{x}

Penalty zone

If q <= 0.5, only the quality penalty applies. The score rises linearly from -2 at zero resolved runs to -1 at 50% of baseline quality:

s=−2+2qs = -2 + 2q

If 0.5 < q < 0.8 (equivalently y < t), the score moves linearly from -1 toward the compression ratio:

β=q−0.50.8−0.5,s=(1−β)(−1)+βr\beta = \frac{q-0.5}{0.8-0.5}, \qquad s = (1-\beta)(-1) + \beta r

Both cases are reported as zone=penalty.

Maintain zone

If t <= y <= x (80% of baseline quality or better), quality contributes no penalty and the compression ratio applies in full:

s=rs = r

Bonus zone

If y > x:

s=max⁡(−2,min⁡(r+b,2))s = \max\left(-2,\min\left(r+b,2\right)\right)

Standard tasks are assigned to pool=main.

SWE miner aggregation

Miner-level aggregation is performed by build_swe_miner_scores.

Main score

For every task in pool=main, the aggregation weight is:

wi=xi1/3w_i = x_i^{1/3}

The main_score is the weighted average of main-task scores:

SM=∑isixi1/3∑ixi1/3S_M = \frac{\sum_i s_i x_i^{1/3}}{\sum_i x_i^{1/3}}

If the miner has no tasks in pool=main:

SM=0S_M = 0

Hard boost

The hard_boost is the sum of positive hard-task contributions divided by the number of scored main and hard tasks:

BH=∑ihiNM+NHB_H = \frac{\sum_i h_i}{N_M+N_H}

Where:

  • N_M is the number of tasks in pool=main.
  • N_H is the number of tasks in pool=hard_boost.
  • h_i is the hard-boost contribution of hard task i.

If there are no hard-boost contributions:

BH=0B_H = 0

Raw miner total

SR=SM+BHS_R = S_M + B_H

Final normalized SWE score

Final normalization is performed by build_swe_miner_total_score.

First, the raw total is clamped to [-2, 2]:

SC=max⁡(−2,min⁡(SR,2))S_C = \max\left(-2,\min\left(S_R,2\right)\right)

The clamped value is then linearly normalized from [-2, 2] to [-1, 1]:

SN=2(SC+24)−1S_N = 2\left(\frac{S_C+2}{4}\right)-1

Equivalently:

SN=SC2S_N = \frac{S_C}{2}

This normalized value is consumed by downstream category and leaderboard scoring.

Synced from DendriteHQ/SOMA/docs/miner/scoring.md
Last updated 26 Oct 2026 by bahamajohnEdit this page on GitHub