feat(runtime): immutable ResolvedRuntime + central client factory; fixes extra_headers loss in fallback paths - #97780
Conversation
- agent/runtime_bundle.py: immutable ResolvedRuntime + central client factory - resolve_runtime_provider now returns ResolvedRuntime (legacy mapping kept as _resolve_runtime_provider_mapping) - anthropic adapter + auxiliary client consume the bundle - Bedrock/MoA metadata preserved, migration deferred Design credit: Codex gpt-5.6-sol architecture proposal.
…_bundle (phase 2) - resolve_provider_client named-custom branch now consumes ResolvedRuntime/build_client_bundle - entry extra_headers reach client.default_headers on all wire paths (OpenAI + anthropic rewrite) - fixes bug class where auxiliary/fallback clients dropped entry headers (btcapp Cloudflare 403; cf NousResearch#88463) - contract tests: headers reach client, no-headers unchanged, user-level override, anthropic path - adjacent suites: 25+247 passed
- install_runtime builds bundle off-agent, swaps state under existing lock - agent_init build-introspect-rebuild anti-pattern removed - switch_model 14-field manual rollback replaced by runtime install - old client retired only after commit; prompt cache untouched - adjacent suites green (3+182 passed)
…erride application _make_agent() treated the resolved runtime as a mutable dict to layer persisted session overrides (base_url/api_key/api_mode) on top. With the ResolvedRuntime refactor those assignments raise TypeError on any resume that carries overrides. Apply them via with_updates() instead. Contract tests: tests/tui_gateway/test_custom_provider_session_persistence.py 11/11.
andrexibiza
left a comment
There was a problem hiding this comment.
Reviewed exact head 42fe0f3a51242e7bc4b9ba51c4d108961abd5f58 against the PR base aff5125f8edf5095aef5d3d79bbbb101c95b9413, then reconciled it with current main@a2e19d484cb5591df8dafe667c93345b62d9bf06. I read the full behavior-critical runtime surfaces (runtime_bundle, auxiliary resolution, startup/install, switch/restore, fallback activation, Anthropic construction, TUI resume), the new contract tests, #88463/#88474, the open switch/fallback interlocks, and the exact-head workflow state. There were no prior reviews or unresolved review threads on this head.
There is a lot worth keeping here. ResolvedRuntime is actually immutable down into nested mappings, the bundle factory gives extra_headers the right precedence over defaults, the named-custom auxiliary path now carries header/TLS state through one typed value, and the TUI follow-up correctly repaired persisted override application with with_updates(). Exact-head CI/Docker/Nix are green (CI run 33244907628, Docker 33244907378, Nix 33244907376), and the 27-job CI run has no failed/cancelled job. This is a meaningful improvement over the current duplicated resolver shape.
I cannot clear the current head yet. I found two concrete merge blockers, one of which is the exact other side of the header-loss shape this PR is intended to eliminate.
P1 — anthropic_messages fallback still reconstructs the client outside the bundle and drops the required entry headers
The new named-custom resolver is correct in isolation: agent/auxiliary_client.py builds the ResolvedRuntime, build_client_bundle() passes extra_headers as default_headers to build_anthropic_client(), and resolve_provider_client() wraps bundle.anthropic_client in AnthropicAuxiliaryClient. The new test_anthropic_wire_bundle_merges_runtime_headers_through_builder pins exactly that seam.
But the actual agent fallback activation does not install that resolved client/bundle. At this exact head, agent/chat_completion_helpers.py::try_activate_fallback() reaches fb_api_mode == "anthropic_messages", takes only fb_client.api_key / fb_base_url, and then does a second construction:
agent._anthropic_client = build_anthropic_client(
effective_key, agent._anthropic_base_url, timeout=_fb_timeout,
)No default_headers are supplied. build_anthropic_client() explicitly supports and merges default_headers, so a named custom Anthropic-compatible fallback whose extra_headers are required for WAF/auth/routing gets the header-correct client from resolve_provider_client(), discards it, and recreates a header-less native client at activation. That is the same defect class as the bug this PR is closing, on the fallback-to-Anthropic projection rather than the auxiliary-resolution projection.
This also falsifies the PR's current structural claim that “no construction path exists outside the factory.” There are still live direct constructors/mutation owners in try_activate_fallback(), switch_model(), and restore_primary_runtime(); in particular, the exact-head switch_model() still has its manual old-state snapshot/rollback path and direct Anthropic client construction even though the phase-3 commit message says the 14-field switch rollback was replaced by runtime installation.
Required fix: make fallback activation consume/install the already-resolved runtime/bundle rather than reconstructing the Anthropic client from ambient fields, or otherwise prove that the complete header/TLS/auth contract is preserved at that second constructor. Add a real regression that configures a named custom anthropic_messages fallback with a required extra_headers value, activates it through try_activate_fallback(), and verifies the installed request client carries that header. Then exercise startup → switch → fallback → restore against the same runtime contract so the remaining construction owners cannot silently diverge. If this PR intentionally stops short of migrating switch/restore, narrow the “single factory / all paths” claims accordingly rather than using a stronger invariant than the code establishes.
This interlocks directly with #94036 (native-compaction capability across switch/fallback/restore), #92905 (transactional context ceiling across the same state transitions), and #96051 (turn-scoped routing through switch/restore). Those are adjacent, not duplicates. If this PR centralizes those mutation boundaries now, their capability/policy state has to become part of the installed runtime contract rather than being overwritten by a mechanically narrower swap.
P1 — Fixes #88463 would close an issue whose second reported path this PR does not repair
#88463 is explicitly a two-path bug:
- named-custom auxiliary auto-resolution drops
extra_headers; and hermes_cli/model_setup_flows.py::_model_flow_named_custom()probes/modelswithout those headers, so header-gated providers still 401 during interactive model discovery.
This PR fixes/generalizes (1), but its 14-file diff does not touch hermes_cli/model_setup_flows.py or add a model-probe regression. #88474 by liuhao1024 does contain that second fix: it passes provider_info["extra_headers"] into fetch_api_models(..., headers=...) and has focused coverage for the present/absent-header cases.
As written, merging #97780 with Fixes #88463 can close the canonical issue while the CLI probe defect remains live on main. The contributor/supersession relationship is also more precise than “this replaces #88474”: #97780 structurally supersedes/generalizes #88474's auxiliary construction half, while #88474 remains complementary and unsuperseded for the interactive model-probe half.
Required fix: either carry the model-probe fix + regression into this PR while preserving explicit credit to liuhao1024 / #88474, or change the issue relationship so #97780 does not close #88463 and leave #88474 (or a clean successor) owning path (2). Please do not collapse the two contributions into a duplicate/superseded label when only one half is actually superseded.
Current-main / release proof
Current main@a2e19d484cb5591df8dafe667c93345b62d9bf06 is one commit beyond this PR's base; that commit changes agent/prompt_builder.py and tests/agent/test_prompt_builder.py, so I found no direct FILE-LIST collision with #97780. The current head is mergeable and its exact-head CI/Docker/Nix receipts are green. After the blockers above are repaired, the final candidate still needs to be refreshed/recomposed on current main and re-run through those exact-object gates; the present green receipts prove 42fe0f3a... on its current base, not the eventual merge object.
The central-runtime direction is the right kind of fix for this class: carry the whole route contract once, then make later boundaries consume that typed object rather than reconstruct configuration from ambient fields. The current patch gets that right at resolution and startup; closing the two remaining gaps above would make the claim match the mechanism. Strong work getting this much of a very drift-prone surface under one value contract.
What does this PR do?
The named-custom provider branch of
resolve_provider_client()builds auxiliary/fallback clients from only a subset of the provider entry (base_url,api_key,key_env,api_mode,model). Per-entryextra_headers— required by providers behind Cloudflare/WAF that reject the OpenAI SDK's defaultUser-Agent— never reach the constructed client, so fallback activation fails with403 Your request was blockedeven though the primary path works.This is the same symptom class as #88463 and #88474, but the root cause is structural: client construction is duplicated across six call sites, and only one of them applies the entry headers. Rather than patching individual sites, this PR separates the three concerns that are currently entangled:
ResolvedRuntimevalue (provider, model, api_mode, credentials, endpoint, extra headers, Bedrock/MoA metadata). No SDK client is built here.build_client_bundle(runtime)is the single factory that constructs a complete wire client (OpenAI-wire or Anthropic-Messages) from the runtime value.AIAgent.install_runtime(bundle)installs the finished bundle onto the agent in one atomic step under the existing lock: build fully off-agent, swap state, retire the old client only after commit. Startup, model switching, and fallback activation all route through this same path.With this in place, entry
extra_headerscannot be dropped by any construction path, because no construction path exists outside the factory.Related Issue
Fixes #88463. Builds on the targeted fix in #88474 (independent reproduction posted there: #88474 (comment)).
Type of Change
Changes Made
agent/runtime_bundle.py(new): immutableResolvedRuntime+build_client_bundle()central factory (OpenAI-wire and Anthropic-Messages)agent/auxiliary_client.py: named-custom branch ofresolve_provider_client()now consumes the factory; entryextra_headersreachclient.default_headerson both wire pathsagent/agent_init.py+run_agent.py: startup usesinstall_runtime()(removes the build→introspect→rebuild anti-pattern)agent/agent_runtime_helpers.py:switch_modelinstalls the new runtime instead of 14-field manual rollbackhermes_cli/runtime_provider.py:resolve_runtime_provider()returnsResolvedRuntime; legacy mapping kept as_resolve_runtime_provider_mappingagent/anthropic_adapter.py,hermes_cli/cli_agent_setup_mixin.py: consume the bundleResolvedRuntimeso they can migrate behind the factory without changing the public value contractHow to Test
extra_headers: {User-Agent: <browser UA>}on a Cloudflare-fronted OpenAI-compatible endpoint, plus a rate-limited primary so fallback activates.main: every fallback activation fails with403 Your request was blocked(CF error 1010); the chain falls through to the last-resort provider.Production evidence (same machine, same config, only code differs): after deploying this branch, 591 consecutive fallback calls over ~8 hours, 0× 403, 0× degrade to the last-resort provider; the fallback has remained healthy since.
Unit level: contract tests assert entry headers reach
client.default_headerson both wire paths, absence of headers leaves behavior unchanged, and user-levelmodel.default_headerstakes precedence;install_runtimeatomicity (factory failure leaves agent state untouched); startup → switch → fallback E2E against a tempHERMES_HOME.Checklist
Code
feat(runtime):)pytest tests/ -q— targeted suites: 274 passed (test_runtime_bundle,test_runtime_install,test_auxiliary_client,test_auxiliary_named_custom_providers,test_runtime_provider_resolution); 6 unrelated failures on Windows are pre-existing onmain(os.geteuidPOSIX-only tests, optional SDK imports)scripts/check-windows-footguns.pyclean on all 8 changed filesDocumentation & Housekeeping
cli-config.yaml.example— N/A (no new config keys;extra_headersis already documented there)CONTRIBUTING.mdorAGENTS.md— N/A