perf(voice): start CO conversation lookup on setup, not first prompt - #103

Summary

• On every voice call's first turn, the Conversation Orchestrator (CO) conversation lookup (list_conversations poll + list_participants) was only triggered once the first "prompt" WebSocket message arrived — i.e. after the caller's speech had already been transcribed. That latency (0.8s–4.1s observed locally, see data below) was paid serially, in front of memory retrieval and the LLM call, on every call.
• ConversationRelay creates the CO conversation as soon as the call connects, not when the caller speaks — there's no reason to wait for the first prompt before starting the lookup.
• This PR kicks off _initialize_conversation in the background as soon as the "setup" WS message arrives (call_sid is known immediately), so CO's polling overlaps with the wait for the caller's first utterance (speech + ASR) instead of adding to it. The first "prompt" handler now just awaits the already-running (often already-finished) task.

Reason

Investigated first-utterance latency on a local ConversationRelay + Conversation Orchestrator + Memory setup. Breaking down the first turn end-to-end showed CO's conversation lookup was a fixed serial cost on every call, unrelated to what the caller said, entirely avoidable by starting it earlier.

Solution

In VoiceChannel.handle_websocket:

• On "setup", if Conversation Orchestrator is enabled, start _initialize_conversation(...) via asyncio.create_task instead of waiting.
• On the first "prompt", await that task (instead of calling _initialize_conversation fresh) to get conv_id/session_state.
• In the finally cleanup path, handle the case where the call disconnects before any prompt arrives: cancel the task if it's still running, or — if it already completed and registered the session/websocket as a side effect — adopt its conv_id so cleanup still runs instead of leaking the websocket registration. (The conversation entry itself intentionally stays in _conversations until CO's CLOSED webhook, same as any other orchestrator-mode call — unrelated to this change.)
• The finally block's CancelledError handling only swallows the cancellation it caused itself (tracked via an explicit we_cancelled_it flag, set when we call init_task.cancel()); a real external cancellation (e.g. server shutdown) runs the same best-effort cleanup first, then propagates instead of being silently absorbed. (init_task.cancelled() looked like it could answer this without an extra flag, but it can't — cancelling this coroutine's own enclosing task while suspended on await init_task also cancels init_task as a side effect, making external cancellation indistinguishable from our own.)
• Raised _POLL_ATTEMPTS from 5 to 10 and capped the exponential backoff via _POLL_MAX_DELAY (was unbounded — 10 attempts uncapped would balloon to ~2 minutes of sleep). The CO poll window used to be measured from the first prompt (i.e. after the caller had already started speaking, giving CO plenty of time); starting it at "setup" instead measures the same window from call-connect, which is earlier and leaves less slack — an early transient miss could exhaust all attempts before the caller even finishes speaking. The longer, capped window on the single background poll comfortably covers the wait for the first prompt too, so await init_task at the first prompt just waits out whatever's left rather than needing a second poll-and-give-up phase.

Behavior change (flagged in review, addressed via docs)

In orchestrator mode, _initialize_conversation's side effects (registering the session in self._conversations and the websocket in WebSocketManager) can now happen before the caller has said anything, since they run as soon as the background CO lookup finishes rather than being gated on the first prompt. Concretely: get_conversation_session_by_call_sid may now return a session before on_amd fires (under machine_detection="Enable"), where it previously always returned None at that point. end_call is unaffected — it already handles both cases. Updated the docstrings on both methods to reflect this.

Given this changes a documented public-API guarantee, this should ship with a version bump.

Tests

• Rewrote test_setup_message_does_not_initialize_conversation (which asserted the old behavior) as test_setup_message_starts_background_conversation_init, which verifies:
  • the CO lookup runs to completion starting from "setup", not "prompt"
  • if the call disconnects before any prompt arrives, the websocket registration is still cleaned up (not leaked), while the conversation entry legitimately stays tracked until CO's CLOSED webhook, matching existing _cleanup_connection semantics
• test_error_when_no_conversations_found / test_error_when_multiple_conversations_found updated for the _POLL_ATTEMPTS bump (now assert 10 poll calls instead of 5). test_subsequent_prompts_reuse_conversation and other existing tests continue to pass unchanged.
• Full suite: make check (ruff + mypy strict + pytest) — all green, no regressions.

Data

Measured via a temporary local logging harness (not included in this PR) across real calls against a ConversationRelay + CO + Memory + streaming OpenAI Agents SDK setup.

Before (CO init triggered on first prompt, fully serial):

After (CO init triggered on setup, in background), across 4 separate real calls, including one where the caller only said "hi" (shortest realistic case, smallest overlap margin):

In every case, by the time the first "prompt" (i.e. transcribed speech) arrived, the background CO lookup had already finished — confirmed via a direct measurement of the wait at that await point, not inferred from totals. Even the shortest realistic utterance ("hi") didn't erode the overlap margin enough to leak latency back onto the critical path in these tests, though the margin does shrink as the caller's first utterance gets shorter — worth keeping an eye on in production telemetry.

Type of Change

• Bug fix
• New feature
• Breaking change (see "Behavior change" above — a documented get_conversation_session_by_call_sid guarantee no longer holds in orchestrator mode)
• Documentation update (docstrings for get_conversation_session_by_call_sid / end_call)
• Refactoring (CO lookup scheduling)
• Release / version bump

Checklist

• Tests added/updated
• Documentation updated
• Tested E2E (real phone calls against ConversationRelay + CO + Memory)

SDK Parity

This is the Python SDK. If this change affects shared functionality, ensure the TypeScript SDK is updated as well.

• Change is Python-specific (no TypeScript update needed) — asyncio task-scheduling change local to VoiceChannel; the behavior change noted above is Python-SDK-internal and doesn't correspond to shared cross-SDK semantics
• TypeScript SDK PR created:

🤖 Generated with Claude Code