Experiment methodology

Each task is executed in two conditions. In the reference condition the agent receives the original conversation and context. In the candidate condition it receives the transformed context. The task definition, repository revision, environment, evaluator, model, provider, generation settings, and repeat count are held constant between the two conditions.

This isolates the effect of the transformation. The experiments measure both efficiency and quality: lower token use is only meaningful when task quality is not materially reduced.

Every attempt is retained as an individual observation. The collected data includes the task and dataset version, condition, model and provider, input, cached-input and output tokens, weighted token use, agent steps, duration, patch/evaluation outcome, and diagnostic artifacts such as trajectories, request/response records, and evaluator output.

Infrastructure failures, upstream timeouts, missing images, interrupted runs, empty responses, and runs without token accounting are classified separately from task results. They are invalid measurements rather than evidence that a task passed or failed.

The principal efficiency measure is weighted token use rather than a simple count of all tokens. It combines the three token categories using the configured weights.

Input tokens represent uncached prompt/context tokens. Cached input tokens have a lower weight because their marginal cost is lower than uncached input under the pricing model represented by this metric. Output tokens have the highest weight because generation is comparatively more expensive. The same formula is applied to both conditions, so the comparison reflects the relative mix of input, cache reuse, and generated output as well as the raw total.

The component counts remain in the collected data. A lower weighted value can come from reducing uncached input, increasing cache reuse, reducing output, or a combination of these effects. Reporting the components alongside the weighted total makes that distinction visible and avoids interpreting a change in cache behaviour as compression alone.

Agent trajectories are stochastic. On the same task, one attempt can take a short path while another may explore a dead end, loop for many more steps, or encounter a transient provider failure. Token use consequently has substantial run-to-run variance and is commonly right-skewed: a few unusually long trajectories can shift the mean considerably.

Multiple repetitions estimate the typical result and its uncertainty rather than reporting a single incidental outcome. They reveal outliers, reduce the effect of transient failures or lucky trajectories, and make it possible to distinguish a stable improvement from random variation.

Repetitions are spread across tasks. This matters because a candidate can look strong on one repository or issue type while not generalising to others. Per-task repeats show whether a pooled result is consistent across the workload or dominated by a small number of tasks. The selected repeat counts are based on the expected size of the difference and variance observed in pilot data: smaller expected effects require more observations.

Analysis begins with the usable sample. For each condition, the results show the planned attempt count, usable count, and exclusions by reason. Missing records are not treated as zero tokens, a pass, or a task failure. A large difference in invalid-run rate is itself diagnostic, since reliability affects whether an apparent efficiency result can be trusted.

Quality is considered before efficiency. The evaluator pass rate, or other predefined task-quality outcome, is compared between conditions for every task and across the dataset. A token reduction accompanied by a material decline in quality, malformed patches, or changed evaluation behaviour is not counted as a useful improvement.

Token use is treated as a distribution rather than a single headline value. The analysis contains mean and median weighted tokens, variance or standard deviation, percentile ranges, absolute change, and percentage change from the reference condition. The mean estimates expected aggregate cost, while the median describes a typical attempt and is less affected by long trajectories. Task-level tables and distributions are retained beside pooled values.

Uncertainty is represented with confidence intervals or bootstrap estimates over attempts, preferably preserving the task structure. A result is inconclusive when its uncertainty includes both a practically meaningful saving and a practically meaningful regression. In that case, the data does not yet support selecting either condition with confidence.

The interpretation also examines task-level consistency. A result supported by small, consistent improvements across task types is stronger than a larger pooled value driven by one atypical task. Regressions, unusually variable tasks, and substantial behavioural changes remain visible in the report instead of being hidden by the aggregate.

Conclusions are bounded by the sampled workload and configuration. They take a form such as: “On this held-out task set, with this model and provider, the candidate changed mean weighted token use by X% and median token use by Y%, with an observed quality difference of Z percentage points.” They do not claim that the result applies universally beyond the tested task distribution.

The data records the task-set version, code revisions, environment images, model/provider settings, repeat count, exclusions, and analysis method. This makes later comparisons reproducible and gives a concrete explanation when a result changes.

Public software-engineering benchmarks remain useful for comparability, but they do not represent every repository, dependency stack, language, issue type, or context pattern relevant to the target workload. Self-generated tasks extend coverage to those areas.

The tasks use a SWE-bench-style contract: a pinned repository revision, a concrete issue statement, an isolated environment, a deterministic evaluator, and a known expected behaviour. This preserves the important properties of a benchmark task - reproducibility and independently checkable evaluation - without reducing the workload to an informal prompt or hand-judged example.

These tasks complement rather than replace public benchmark tasks. Their definitions and evaluators are reviewed for reproducible images, evaluator correctness, meaningful issues, and separation between the implementation being tested and the tests that determine the outcome.

Mainnet data is part of development. It supplies realistic operating conditions, feedback, and a source of improvements before release. Since it is seen during development, it is not the final evidence used to promote a candidate to production.

The separate held-out evaluation occurs after mainnet testing and before production release. It measures whether a candidate that appeared promising during development generalises to tasks it has not been tuned against. This prevents adaptive overfitting, where repeated development against known tasks gradually optimises for their specific prompts, repositories, tool outputs, and quirks rather than unseen work.

The split also supports fairness: knowledge of earlier mainnet tasks, trajectories, or score feedback cannot become an advantage in the final evaluation. The final score then measures general performance rather than familiarity with a known set.

Where practical, separation occurs by repository and issue family rather than by random attempts alone. Closely related tasks can share solutions and trajectory patterns, so a purely random split can leak information. Dataset versions are retained with the experiment results.