max_turns counts tool calls on codex/antigravity but agent-loop turns on claude-code — same cap, different budget #138

What happened?

run_limits.max_turns is counted in a different unit per backend. The same number in
the same task file caps a different amount of work depending on --type.

• codex / antigravity — every resolved tool call increments the counter. max_turns: 10 = 10 tool calls, hard stop.
• claude-code — the counter increments per agent-loop turn (one user message + the assistant response to it). A single such turn contains any number of tool calls, so max_turns: 10 = 10 turns = unbounded tool calls.

Expected: max_turns: N caps the same amount of agent work on every backend.
Actual: it caps N tool calls on two backends and N × (however much the model batched) on the third.

This is not a doc gap — the units are genuinely different at runtime, and nothing at
config-resolution time, in the test suite, or in the run record surfaces the difference.

Root cause

Two independent counters, neither aware of the other:

The Codex/Antigravity side has no choice: both SDKs deliver exactly one SDK turn per
communicate(), so a native turn counter would clamp at 1. Counting tool calls is the
only unit available at that layer (codex_agent.py::max_turns_reached docstring says so
explicitly). claude-code keeps its native cap, which counts something else.

Introduced in #110, which fixed the prior bug (max_turns accepted and ignored on Codex
and Antigravity) but did not reconcile the units. #110's own commit message contains the
measurement:

Under a batching prompt with max_turns 2, claude-code permitted all 12 writes across 14
assistant messages, because one SDK agent-loop turn absorbs however many parallel calls
the model emits. Codex and Antigravity stop at 2. Same number, very different budget.

Why it wasn't caught

tasks/run_limits/max_turns_cap.yaml is the cross-harness fixture and it passes on all
three backends — because its prompt says "do not batch multiple tool calls together" and
chains each step onto the previous file, so batching is impossible. Under those conditions
the two units coincide. The fixture proves the cap binds; nothing asserts the caps are
equivalent.

orchestration/run_limits.py::validate_run_limits is the only post-merge cross-field
check. It compares task_timeout against turn_timeout and nothing else — it never reads
max_turns or the resolved agent.type.

Secondary: the enforced counter ≠ the reported counter

• Enforced: EventCollector.visible_turn_count = len(commands).
• Reported: reports_stats.visible_turn_count = commands + 1 if has_final_reply.
• Plus a force-closed in-flight call and recovered sub-agent calls, both documented as landing beyond the cap.

So a run capped at N can report > N visible turns, which makes "was this run capped?"
un-answerable from the count alone.

Boundary inclusivity also differs: Codex breaks on visible_turn_count >= max_turns;
claude_code_agent flags exhaustion on num_turns > max_turns.

Impact

Any task with a binding max_turns produces a backend-dependent verdict. Turn-budget gate
tasks — the ones whose entire purpose is to fail an inefficient agent — fail on
codex/antigravity and pass on claude-code for reasons that have nothing to do with
the agent or the skill under test.

The failure is quiet: a capped run that still satisfies its criteria finishes SUCCESS, so
pooled pass rates mix capped and uncapped populations unless the reader segments on
max_turns_exhausted.

Suggested fixes

1. Warn at resolution. Extend validate_run_limits() to fire whenever max_turns is
   set, naming the unit for the resolved agent.type. It already runs post-merge, has an
   orchestrator call site and a warnings-tuple return.
2. Make the unit selectable — max_turns_unit: agent_loop | visible, or a separate
   max_visible_turns. EventCollector already counts visible turns on every driver
   including claude-code; only the enforcement hook is missing there. That's what makes
   one unit achievable across all three.
3. Add a fixture that permits batching. Keep max_turns_cap.yaml; add a sibling whose
   prompt neither forbids loops nor chains its steps, asserting the resolved tool-call
   count. It should fail today.
4. Report the ratio — tool calls per SDK turn, alongside visible_turns and
   total_turns, so a cross-backend comparison shows whether the budgets were comparable.

References

• feat(agents)!: honor run_limits.max_turns on codex and antigravity #110 (12c5031) — introduced the visible-turn cap, HARNESS_PARITY.md and tasks/run_limits/max_turns_cap.yaml
• docs/agents/HARNESS_PARITY.md § "max_turns counts visible turns on Codex and Antigravity" — already states "holding max_turns constant across harnesses does not hold the budget constant"
• Code, as of dd918e6: agents/codex_agent.py::max_turns_reached + pump break · agents/antigravity_agent.py (same rule) · agents/claude_code_agent.py (options build, exhaustion detect) · streaming/collector.py::visible_turn_count · reports_stats.py::visible_turn_count · orchestration/run_limits.py::validate_run_limits

Longer write-up with a diagram of the two enforcement paths:
turn-accounting.html

Steps to reproduce

Any task whose prompt permits batching, run under a cap tight enough to bind, on two
backends. tasks/run_limits/max_turns_cap.yaml will not show it — its prompt forbids
batching by design.

task_id: max-turns-unit-divergence
tags: [run-limits]
initial_prompt: |
  Create 12 files named step-01.txt through step-12.txt in the current directory.
  Each file contains just its own name. Batch your tool calls freely.
run_limits:
  max_turns: 2
  turn_timeout: 300

coder-eval run <task> --type codex         # stops at 2 tool calls, step-12.txt absent
coder-eval run <task> --type claude-code   # 12 files written; the cap never binds

Compare actual_commands / visible_turns and max_turns_exhausted in each task.json.

Originally observed on the skill-rpa-execution-map-greenfield task in the UiPath skills
repo: identical task config (max_turns: 10), MAX_TURNS_EXHAUSTED at 10 tool calls on
codex, SUCCESS after 20 tool calls on claude-code.

Environment

0.11.2

Logs / output