Instruments chatd with OpenTelemetry spans and a Prometheus histogram covering the full lifecycle of an Agents chat turn, and adds a Grafana dashboard that renders the aggregate stage profile as a flamegraph with selectable summary statistics (mean, p50, p90, p95, p99) plus summary panels for the whole pipeline.

What's included

• chatloop.StageTracer: a shared helper that emits each lifecycle stage as both an OTel span and an observation on a new coderd_chatd_stage_duration_seconds{stage, scope, model, effort} histogram from a single code path, so traces and metrics cannot drift. Stages: chat_turn, queue_wait, capacity_wait, acquisition, generation_step, prepare, mcp_connect, provider_attempt, time_to_first_token, stream, thinking, tool_call, commit, compaction.
• Scope separation: detached background work (title/summary quickgen) is excluded from turn traces and labeled scope="background" so it cannot skew turn-level aggregates.
• Model and effort dimensions: the resolved model and the effective reasoning effort (resolved once in resolveModelCall, the same value sent to the provider) are recorded as span attributes and histogram labels, threaded through to provider_attempt via the transport.
• Grafana dashboard at examples/monitoring/dashboards/grafana/chatd-lifecycle/ (dashboard JSON + README with per-panel explanations): stage flamegraph and hierarchy bar chart, stage duration trends, time share of chat_turn, scheduling waits p99, TTFT, turn/stage rates, and background provider calls, filterable by $model/$effort with a $stat statistic selector. All queries are NaN-safe so rare stages render as 0 rather than breaking the flamegraph's nested-set ordering.
• Spans export through the existing OTLP path (--trace + standard OTel env vars); per-session drill-down renders each chat_turn as a root span tree in any tracing UI (verified with Tempo).

Verification

Verified end to end against a live dev deployment with Tempo + Prometheus + Grafana: 100+ real chat turns across two models, exact count agreement between trace spans and histogram observations for every stage, clean span nesting (no children outliving parents, no negative offsets), HTTP >= 400 provider attempts marked as span errors, and the dashboard rendering in Grafana 11.4 (including a headless render check of the flamegraph panel).

Not sampled in the test environment (code paths exist, environment never triggers them): capacity_wait (the premium license disables the concurrent-agent cap via unlimited agent_runtime_hours), mcp_connect (no MCP servers configured), compaction (context never neared the threshold).

Notes and known limitations

• chat_turn spans are standalone trace roots: the turn executes asynchronously on a worker (possibly another replica) and no trace context is persisted, so linking to the originating HTTP request span is not possible without persisting a traceparent.
• chat_turn is runner-scoped (same boundary as the debug turn): a queued follow-up folds into the preceding turn span rather than starting a new one.
• Stages that occur before model resolution (chat_turn, queue_wait, capacity_wait, acquisition) carry empty model/effort labels; filtering the dashboard by model narrows the view to the generation stages. Noted in panel descriptions.
• Found during verification, not addressed here: quickgen title generation reliably gets a 400 ("temperature may only be set to 1 when thinking is enabled") from Anthropic models and succeeds on retry; now cleanly visible as scope="background" error spans. Worth a separate issue.

chat_turn
├── queue_wait          chat_queued_messages insert -> promotion
├── capacity_wait       capacity.go limiter acquire
├── acquisition         trigger message insert -> Acquire() applied
└── generation_step (per step, repeats)
    ├── prepare         prompt build, model resolution, context hydration
    ├── mcp_connect     mcpclient connect (when it occurs in the step)
    ├── provider_attempt  one HTTP round trip incl. retries (per attempt)
    │   └── time_to_first_token   stream open -> first part
    ├── stream          stream open -> stream close
    ├── thinking        reasoning part created_at -> completed_at
    ├── tool_call       per local tool call start -> completion
    ├── commit          CommitStep DB txn
    └── compaction      auxiliary compaction LLM call (when triggered)

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.