Add extensible chat client routing by joshuajyue · Pull Request #7662 · dotnet/extensions

joshuajyue · 2026-07-27T22:31:27Z

Resolves #7647

Summary

Adds experimental chat-routing primitives for selecting configured IChatClient instances without imposing a particular routing policy.

RoutingChatClient supports one-shot client selection.
FailoverChatClient retries uncanceled failures that occur before output is exposed.
OrderedFailoverChatClient provides ordered fallback.
SemanticRoutingChatClient selects clients using cached profile embeddings.
RoutingContext exposes mutable request messages and options.
FailoverChatClientAttempt reports invocation outcome, duration, time to first update, completion, and output commitment.

API shape

Routing policies have two seams:

protected override ValueTask<IChatClient> SelectClientAsync(
    RoutingContext context,
    CancellationToken cancellationToken);

protected override ValueTask OnRoutingUpdateAsync(
    RoutingContext context,
    FailoverChatClientAttempt? attempt,
    bool isTerminal,
    CancellationToken cancellationToken);

SelectClientAsync runs before every invocation. OnRoutingUpdateAsync reports each attempt so policy state can influence the next selection.

isTerminal means the base will not select another client after the callback completes. A null attempt represents selection terminating before a client was invoked.

Failover behavior

Cancellation never triggers reselection.
Streaming fallback occurs only before the first update is exposed.
Mid-stream failures and caller abandonment are terminal.
Enumerator disposal failures are included in the attempt.
Attempt limits are request-local.
Null selector results are treated as terminal selection failures.
Ordered failover retains request state only between a nonterminal update and the immediately following selection; it holds no state while a provider invocation or stream is active.
Streaming callers must dispose active enumerators for terminal observation and inner-resource cleanup.

Semantic routing

SemanticRoutingChatClient:

Lazily embeds and caches profile utterances.
Embeds the final user message for selection.
Supports global top-K profile matching.
Aggregates matches per client using mean or sum.
Preserves the existing single-best-match behavior with topK: 1.
Uses a required default client when no profile clears the threshold.
Supports caller-controlled ownership through leaveOpen.

Tests

Coverage includes selection failures, retries, cancellation, attempt limits, streaming commitment, early disposal, enumerator failures, concurrent requests, state cleanup, semantic thresholds, top-K aggregation, caching, and ownership.

753 tests passed on net10.0
net472 test project builds successfully

Microsoft Reviewers: Open in CodeFlow

Introduce one-shot and failover routing primitives, ordered and semantic routing implementations, lifecycle reporting, streaming safeguards, and focused routing tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Treat provider-thrown cancellation as terminal, materialize semantic routing messages before inspection, and enforce non-null selection consistently for streaming. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Capture Current access failures before committing output so pre-output failover and attempt reporting remain accurate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Apply ConfigureAwait(false) consistently across selection, invocation, streaming, disposal, and routing update awaits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

joshuajyue

Addressed in commit 0abbfd1, along with some cancellation races

Clarify RoutingContext input semantics, add optional message buffering, centralize validation, simplify timing and disposal, and separate caller cancellation from provider failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Clone request options for selector shaping and restrict routing updates to concrete client attempts, leaving selection-failure cleanup to selectors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Follow MEAI's repeatable-enumerable convention and propagate selector failures without cancellation rewriting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Use ConcurrentDictionary atomic operations instead of explicit lock blocks while preserving the narrow request-state lifetime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Keep client selection simple while preventing selector and failed-client option mutations from affecting downstream attempts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Document client-reference identity and keep semantic scoring parameters together in the constructor signature. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

jozkee

Looking good.

Align routing code with MEAI formatting and documentation conventions, simplify disposal, and split routing tests by component and assembly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

joshuajyue · 2026-07-31T19:53:48Z

Thank you David for the feedback, addressed in 8b79ae4

Consolidate shared streaming and non-streaming cases, reuse routing test helpers, and simplify semantic router disposal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

joshuajyue · 2026-08-01T00:34:14Z

Sorry y'all, I had a change of thought -- just one last thing I felt was a bit unresolved:

I actually think the small per-attempt selection result mentioned earlier is more ergonomic than requiring every option variation to be represented by a configured client:

RoutingSelection(IChatClient client, ChatOptions? options)

ConfigureOptionsChatClient remains the only way to retain a stable identity for the client with varying options. The returned options are simply complete, ephemeral options for that attempt, while RoutingContext.ChatOptions remains the clean request baseline. No merging of options necessary -- the user shapes on top of a clone of caller options. This makes dynamic model/reasoning/temperature selection accessible through the selector without leaking options across failover attempts or treating context mutation as a hidden output.

@jozkee @PranavSenthilnathan let me know what you guys think when you have time.

I kicked off a Copilot session to write it out and push it here -- if you guys prefer the old way then I’ll revert it. I do really like the simplicity of returning a singular IChatClient -- the contract is a lot cleaner -- and I understand if this new approach 'muddies the waters', or so to speak.

jozkee · 2026-08-03T19:46:33Z

@joshuajyue I don’t believe selection-specific options are a first-class scenario. Neither OrderedFailoverChatClient nor SemanticRoutingChatClient needs selection to produce different options; both forward the request baseline unchanged. The parallel Agent Framework routing PR also returns only a destination key and forwards the request options unchanged.

It may be a scenario we should support, but it does not necessarily need to be part of the initial API. I also think it would be better to align with ConfigureOptionsChatClient and configure options similarly after selecting the client:

protected RoutingChatClient(
    Action<RoutingContext, IChatClient, ChatOptions> configureOptions);

public static RoutingChatClient Create(
    Func<RoutingContext, CancellationToken, ValueTask<IChatClient>> selector,
    Action<RoutingContext, IChatClient, ChatOptions> configureOptions);

joshuajyue · 2026-08-03T21:44:17Z

Reverting to the prev architecture. I found some clean up in ordered failover client, so that'll be what the commit is -- but it's still on top of the pre-change-of-thought commit.

Allow repeated client instances in ordered failover. Selection now reads a stored index instead of reverse-resolving the attempted client by reference, so the constructor no longer requires unique instances and a client may appear more than once, being invoked once per position. The per-request state holds only the next index. Exhaustion is detected when the index advances past the last client, so the final failure is rethrown from the routing update rather than stored and rethrown from the next selection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/Libraries/Microsoft.Extensions.AI/ChatRouting/FailoverChatClient.cs:292

In the streaming path, disposing/abandoning the outer async enumeration before any update is exposed (and without cancellation/exception) leaves responseCompleted as false and terminalException as null, which currently results in isTerminalAttempt being reported as false. That implies another selection will follow, but disposal means routing is actually terminal and policy implementations may leak per-request state (e.g., ordered failover stores the next index on nonterminal updates). Consider treating this "no exception + not completed" condition as terminal.

                isTerminalAttempt =
                    attempt.ResponseCompleted ||
                    outputCommitted ||
                    cancellationRequested ||
                    reachedAttemptLimit;

src/Libraries/Microsoft.Extensions.AI.Abstractions/ChatRouting/RoutingContext.cs:49

PR metadata/issue proposal describe RoutingContext as exposing mutable request messages/options (settable properties). In this implementation both properties are get-only (and ChatOptions is a cloned snapshot), which makes it impossible for routing/failover policies to replace the message sequence or swap options between attempts via the context. Either update the published API shape/docs to match (and adjust API baselines/tests accordingly), or revise the PR description to reflect that only the contents of the returned ChatOptions (and any mutable underlying message collection) can be changed.

    /// <summary>Gets the messages supplied to client selection and the selected client.</summary>
    /// <remarks>
    /// Selection and failover may enumerate this sequence multiple times. Callers should supply a repeatable sequence.
    /// </remarks>
    public IEnumerable<ChatMessage> Messages { get; }

    /// <summary>Gets a snapshot of the request options supplied to client selection.</summary>
    /// <remarks>
    /// Changes do not affect the caller's instance or the options passed to the selected client. Client-specific
    /// behavior should generally be attached to the returned client.
    /// </remarks>
    public ChatOptions? ChatOptions { get; }

Document RoutingChatClient as an experimental capability in docs/ai/ichatclient.md, mirroring the existing 'Chat reduction (experimental)' section: an IMPORTANT experimental note plus a concise description with xref links to RoutingChatClient, SemanticRoutingChatClient, and OrderedFailoverChatClient, matching the API shape merged in dotnet/extensions#7662. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7f6512d2-cfbc-4df3-8401-acc729c949e6

Document RoutingChatClient and FailoverChatClient as experimental capabilities in docs/ai/ichatclient.md, mirroring the existing 'Chat reduction (experimental)' section: an IMPORTANT experimental note plus a concise description with xref links to RoutingChatClient, FailoverChatClient, OrderedFailoverChatClient, and SemanticRoutingChatClient, matching the API shape merged in dotnet/extensions#7662. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7f6512d2-cfbc-4df3-8401-acc729c949e6

Copilot AI review requested due to automatic review settings July 27, 2026 22:31

joshuajyue requested review from a team as code owners July 27, 2026 22:31

Copilot started reviewing on behalf of joshuajyue July 27, 2026 22:31 View session

This comment was marked as resolved.

Sign in to view

dotnet-policy-service Bot assigned joshuajyue Jul 27, 2026

This comment was marked as resolved.

Sign in to view

joshuajyue and others added 3 commits July 27, 2026 15:43

Potential fix for pull request finding

665abe4

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Credit semantic-router inspiration

26d7a5c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

joshuajyue requested a review from Copilot July 28, 2026 05:15

Copilot started reviewing on behalf of joshuajyue July 28, 2026 05:15 View session

This comment was marked as resolved.

Sign in to view

Handle streaming Current getter failures

bcff6d3

Capture Current access failures before committing output so pre-output failover and attempt reporting remain accurate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

joshuajyue requested a review from Copilot July 28, 2026 20:13

Copilot started reviewing on behalf of joshuajyue July 28, 2026 20:13 View session

This comment was marked as resolved.

Sign in to view

joshuajyue and others added 2 commits July 28, 2026 14:22

Avoid context capture in failover routing

acc450e

Apply ConfigureAwait(false) consistently across selection, invocation, streaming, disposal, and routing update awaits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Merge branch 'main' into routing-behaviors

d4b3a90

PranavSenthilnathan requested review from PranavSenthilnathan, jeffhandley and jozkee July 28, 2026 22:09

PranavSenthilnathan reviewed Jul 29, 2026

View reviewed changes

joshuajyue commented Jul 29, 2026

View reviewed changes

joshuajyue force-pushed the routing-behaviors branch from a19b74b to 0abbfd1 Compare July 29, 2026 20:44

jozkee reviewed Jul 29, 2026

View reviewed changes

Comment thread src/Libraries/Microsoft.Extensions.AI.Abstractions/ChatRouting/RoutingChatClient.cs Outdated

Comment thread src/Libraries/Microsoft.Extensions.AI/ChatRouting/FailoverChatClient.cs Outdated

jozkee reviewed Jul 30, 2026

View reviewed changes

Comment thread src/Libraries/Microsoft.Extensions.AI/ChatRouting/OrderedFailoverChatClient.cs Outdated

Align routing message and selection semantics

53f6639

Follow MEAI's repeatable-enumerable convention and propagate selector failures without cancellation rewriting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

jozkee reviewed Jul 30, 2026

View reviewed changes

Comment thread src/Libraries/Microsoft.Extensions.AI/ChatRouting/OrderedFailoverChatClient.cs Outdated

Simplify ordered failover state

a576773

Use ConcurrentDictionary atomic operations instead of explicit lock blocks while preserving the narrow request-state lifetime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

joshuajyue force-pushed the routing-behaviors branch from 896e51d to a576773 Compare July 31, 2026 01:12

joshuajyue and others added 2 commits July 30, 2026 19:42

Isolate routing attempt options

87d382d

Keep client selection simple while preventing selector and failed-client option mutations from affecting downstream attempts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Clarify routing identity semantics

72598e1

Document client-reference identity and keep semantic scoring parameters together in the constructor signature. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

jozkee reviewed Jul 31, 2026

View reviewed changes

jozkee approved these changes Jul 31, 2026

View reviewed changes

joshuajyue and others added 2 commits July 31, 2026 16:36

Refine routing tests

938e0b9

Consolidate shared streaming and non-streaming cases, reuse routing test helpers, and simplify semantic router disposal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 74d04840-2379-4615-93f7-84f2299ada74

Merge branch 'main' into routing-behaviors

b8839c1

joshuajyue requested a review from jozkee July 31, 2026 23:40

joshuajyue mentioned this pull request Aug 1, 2026

.NET: Add RoutingChatClient for routing requests across multiple chat clients microsoft/agent-framework#6932

Closed

5 tasks

joshuajyue force-pushed the routing-behaviors branch from 626a79d to 2a2e284 Compare August 3, 2026 22:20

jozkee requested a review from Copilot August 4, 2026 01:51

Copilot started reviewing on behalf of jozkee August 4, 2026 01:51 View session

Copilot AI reviewed Aug 4, 2026

View reviewed changes

jozkee approved these changes Aug 4, 2026

View reviewed changes

joshuajyue merged commit 9885b14 into dotnet:main Aug 4, 2026
5 checks passed

This was referenced Aug 6, 2026

Add a routing context factory for request-scoped policy state #7684

Closed

Pass the request's options to the selected client #7685

Merged

joshuajyue mentioned this pull request Aug 10, 2026

Add experimental chat routing section to IChatClient article dotnet/docs#55354

Draft

westey-m mentioned this pull request Aug 13, 2026

.NET: Add session-persisted chat client routing microsoft/agent-framework#7641

Open

5 tasks

Uh oh!

Conversation

joshuajyue commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

API shape

Failover behavior

Semantic routing

Tests

Microsoft Reviewers: Open in CodeFlow

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshuajyue left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jozkee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshuajyue commented Jul 31, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshuajyue commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jozkee commented Aug 3, 2026

Uh oh!

joshuajyue commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

joshuajyue commented Jul 27, 2026 •

edited

Loading

joshuajyue left a comment •

edited

Loading

joshuajyue commented Aug 1, 2026 •

edited

Loading

joshuajyue commented Aug 3, 2026 •

edited

Loading