Researchers at ServiceNow present evidence that evolving an agent’s harness — while leaving model weights completely fixed — delivers 20–35 percentage-point performance gains across three enterprise benchmarks, outperforming prompt optimization by 13–22 points in a descriptive comparison.

Before continuing: what to know about this paper

This is an arXiv preprint (v1) submitted 25 August 2026 by a single research collaboration at ServiceNow Research. It has not been peer reviewed. No independent replication exists as of this writing. The primary agent under test is GPT-5.4 at medium reasoning, which is not universally accessible. AutomationBench Finance results use a non-default Finance-100 configuration with the Stirrup harness — scores are not directly comparable to the benchmark paper’s published numbers. EnterpriseOps-Gym results cover 103 tasks drawn from a larger 1,150-task benchmark. The Stirrup harness is publicly available; the StarHarness optimizer is available at github.com/ServiceNow/StarHarness. Treat these as strong preliminary results from one research group, not settled engineering benchmarks.

What StarHarness does

StarHarness is a harness-evolution framework: model weights stay frozen while an optimizer reshapes everything around them. The optimizer — a coding agent running inside a Pi-variant harness — reads the current harness and traces of recent task failures, then proposes a bounded change: a tool schema update, a prompt addition, a skill, a subagent configuration edit, or an operational procedure. Each proposal passes scope and validation checks and a proposer-selected test flip before evaluation on a proposer-hidden selection task set. Accepted changes become the new search frontier.

Before evolution begins, StarHarness partitions tasks by failure mode (wrong tool, context loss, missing evidence, premature conclusion), baseline score, and verifier pass rate. The proposer sees only search-task traces — never selection or held-out task contents, verifier feedback, or per-task outcomes. This separation provides a direct measure of generalization.

Three benchmarks, three kinds of model–environment friction

ITBench SRE (40 tasks — root-cause analysis over Kubernetes operational telemetry): Stirrup baseline: 40.0%. After 4 accepted patches: 75.0% (+35 percentage points). Estimated GPT-5.4 inference cost per task: −17%. The most consequential patch was a forensics overview that ranks candidate upstream causes from observed evidence, moving repeatable diagnostic reasoning into structured pre-processing. The evolved agent reached more accurate conclusions with fewer turns while inspecting a comparable volume of raw telemetry.

EnterpriseOps-Gym ITSM (103 tasks from a 1,150-task benchmark — state-changing ITSM workflows, SQL-verified against database state): 12 patches accepted — 8 from a tree-search phase exposing interacting schema, prompt, and tool failures, then 4 from a hill-climbing phase adding linkage and grounding fixes. The evolved harness reaches 43.7% task success, near Codex’s 41.7% on the same environment. Estimated cost per task fell 53%. Gains versus GEPA prompt optimization: +22.3 percentage points — though the paper explicitly notes this comparison is descriptive: StarHarness and GEPA differ in prompts, tools, execution policies, and harness architecture, so the delta does not isolate a single causal component. Repairs addressed MCP argument handling, compound schema encoding, null-stripping to avoid strict backend validation rejections, and explicit operational linkage cues. (Exact absolute baseline and evolved task-success figures are in Table 4 of the paper; the main text prose reports relative gains.)

AutomationBench Finance (100 tasks, Finance-100 non-default config — multi-application finance workflows, programmatically verified): Baseline: 57.1% domain objective score. After 5 patches: 83.2% (+26.1 pp). Mean partial credit: 67.3% → 86.1% (+18.8 pp). Turns per task: 16.35 → 11.98 (−4.37). Estimated cost per task: $0.14 → $0.10 (−29%). Tasks with guardrail violations: 20 → 4 (−16 tasks). Total guardrail violations: 33 → 4. Zero-score tasks: 24 → 6. Changes included structured row operations replacing fragile spreadsheet edits, date anchoring to the sandbox clock, a pre-mutation triage step, and dedicated finance calculators for deterministic temporal and numerical operations.

Transfer: one harness, multiple models

The evolved harnesses were evaluated on Qwen model families without re-running the optimization. Performance gains persisted across both GPT and Qwen families — addressing a direct enterprise concern about whether harness engineering investment is reusable as the underlying model changes. Gains on models weaker than GPT-5.4, or model families beyond GPT and Qwen, are not reported.

What the patches actually contain

Accepted edits across all three environments fell into three categories. Interface repairs addressed concrete tool-use failures: schema fields that needed narrowing, argument preprocessing that avoided strict validator rejections. Environment conventions made implicit operational rules explicit: coupled field updates, finish contracts, preservation requirements the model couldn’t reliably infer. Operational knowledge compressed search: forensics overviews, deterministic calculators, triage steps moving predictable reasoning into structured pre-processing.

These are engineering decisions about the interaction layer between a fixed model and a specific environment. They can be tested, versioned, and reverted as ordinary code changes.

What this means for enterprise evaluation

For teams evaluating AI agents in enterprise settings, these results support a practical diagnostic step: before paying for a stronger model, test whether a material share of the agent’s failure comes from the interface between model and environment. The evidence supports treating model capability and model–environment fit as separate optimization surfaces — not as grounds for dismissing model choice altogether.

The 20–35-point range covers environment-specific harness optimization on three enterprise task types — ITSM, SRE, and finance automation. It will not automatically generalize across all enterprise settings or model families beyond those tested. GPT-5.4 at medium reasoning is a capable baseline; how harness evolution performs on smaller or less capable models is not addressed here.

Primary source: arXiv:2608.24804. Code: github.com/ServiceNow/StarHarness. Authors: Esakkivel Esakkiraja, Denis Akhiyarov, Vikas Yadav, Sai Rajeswar, Patrice Bechard, Sridhar Nemala, Sagar Davasam (ServiceNow Research). Submitted 25 August 2026.