Studio: show sidebar context usage and live chat TPS by Etherll · Pull Request #9946 · unslothai/unsloth

Etherll · 2026-08-29T01:45:54Z

Summary

Shows the latest saved request token total under each chat title in the sidebar. Adds a request-scoped TPS widget to the chat header while a generation is running.

Motivation

Chat history did not expose how much context the latest request used, and live generation speed was only visible in the API monitor. Both values can now be checked from the chat surface without estimating tokens or matching an unrelated monitor row.

Changes

Read the newest saved assistant message's validated metadata.contextUsage.totalTokens through the existing batched sidebar message load. Legacy, partial, invalid, and compare rows stay unavailable instead of falling back to an older or aggregated value.
Attach an optional X-Unsloth-Monitor-Id response header to row-owning chat streams and expose only that header through Studio CORS. Durable chat runs forward the same volatile correlation on their event response.
Poll the authenticated exact monitor entry while its request is running. The widget accepts only finite, non-negative running samples and clears on completion, cancellation, replacement, fetch failure, or thread ownership changes.
Keep SSE bodies, saved chat schemas, monitor persistence, authentication, and tunnel origin policy unchanged.

How to test

cd studio/backend
python -m pytest -q tests/test_chat_generation_supervisor.py tests/test_chat_stream_monitor_header.py tests/test_secure_tunnel_gate.py tests/test_sse_streaming_headers.py

The backend suite passed with 93 tests.

cd studio/frontend
node --experimental-strip-types --test tests/chat-generation-reconnect.test.ts tests/legacy-chat-store-timeout.test.ts tests/live-chat-tps.test.ts tests/sidebar-last-request-usage.test.ts
node_modules\.bin\tsc.cmd -b --pretty false
node_modules\.bin\tsc.cmd -p tsconfig.test.json --pretty false

The focused frontend tests and both TypeScript projects passed. A broader local npm test reused an older dependency tree and retained three unchanged remend failures: the install has 1.3.0 while the lock pins 1.3.1. The four feature-attributable failures found by that run were fixed and their focused tests pass.

Screenshots

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-08-29T09:45:46.925979Z	`1ad5ca1`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

for more information, see https://pre-commit.ci

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 86438e3192

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 434b2e0d67

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

for more information, see https://pre-commit.ci

mahiatlinux · 2026-08-29T08:29:32Z

Review findings and fixes

The feature solves a real gap: Studio persisted per-message context usage but did not expose the newest saved assistant request in the sidebar, and the existing API monitor had no request-owned path to a live chat TPS value.

I found and fixed these concrete defects:

Durable run reads, cancellation, active-run lookup, and SSE replay were keyed only by caller-supplied run or thread IDs. Another authenticated subject could reach a run if it learned an ID. Creation now persists owner_subject, and every public lookup filters by it in chat_generation_runs.py and chat_generation_runs_db.py.
The durable SSE route could wait for monitor correlation before constructing the response. A slow or absent monitor kept the client from receiving the stream. The route now constructs the stream immediately, keeps settled reconnects responsive, and attaches the header only when the supervisor owns a monitor ID in chat_generation_runs.py.
CUDA safetensors generations never advanced the live monitor decode counter, so the new widget stayed unavailable for a supported backend. The safetensors callback now records decoded tokens and uses the decode span instead of estimating from wall time in inference.py and api_monitor.py.
Delayed response headers, transient monitor failures, replacements, and shared first-turn keys could retain or replace another request's TPS. TPS state is now owned by the exact run closure, keyed to the routed thread, and removed only for that owner. Monitor errors clear the visible sample before retry in use-live-chat-tps.ts and chat-runtime-store.ts.
Sidebar usage could fall back to an older assistant or an aggregated thread total when the newest saved assistant had missing or malformed metadata. The selector now inspects only the newest chronological assistant and accepts only a complete, finite, non-negative server usage record in sidebar-last-request-usage.ts.
Sidebar loading fetched full message bodies for every thread, and a newly streaming assistant could leave the prior request total visible. The history summary now returns only the latest assistant metadata, while streaming updates publish an immediate clear. I also removed a second dead sidebar metrics implementation.
Stop on a reconnected recovery follower only called assistant-ui's local cancel path. With another chat generating, the recovered server run continued and kept its TPS. The visible Stop action now dispatches the active thread's registered server cancel in thread.tsx.
Focused CI exposed two test defects: an unreferenced timer let the monitor timeout test exit without exercising its assertion, and the import verifier could not resolve the callback module. Both tests now execute the intended paths.

The fixes are split across 13 focused commits.

Verification

Focused local backend: 197 passed.
Focused local frontend: 24 passed, plus the recovered Stop regression. TypeScript, Ruff, import-hoist, and diff checks passed.
Real model: unsloth/Qwen3-1.7B-GGUF, UD-Q4_K_XL, 4096-token context, parallel 2.
Playwright drove the installed Studio UI and asserted rendered values against persisted message metadata and the exact monitor request being polled.
Covered normal completion, long context, cancellation, replacement, reconnect/resume, thread switching, concurrent chats, three rapid start/stop cycles, monitor fetch failure, stale monitor IDs, failed overflow, deleted/reloaded chats, compare rows, no-assistant chats, multiple assistants, legacy/partial/malformed metadata, and exact newest-assistant selection.
Verified the monitor header on direct and durable chat flows, ownership isolation across subjects and threads, SSE replay bodies, persisted schemas, auth, secure-tunnel restrictions, and unchanged behavior outside the intended chat flow.
Focused real-model CI: https://github.com/mahiatlinux/unsloth/actions/runs/33240787747
Full fork CI: https://github.com/mahiatlinux/unsloth/pull/32/checks
The full fork suite ran all 21 pull-request workflows. PR-owned chat metrics, auth/API, lint, TypeScript, focused UI, wheel, Tauri, llama.cpp, and Python 3.11 paths passed. The remaining failures are unrelated baseline or external nondeterminism: stale source-regex tests, Windows pydantic file locking, temperature-zero GGUF response variation, an MCP test using a non-tool-capable model, Python 3.13 environment/test drift, and documented unsloth-zoo xdist isolation failures.

UI evidence

Before: active generation without live TPS

After: exact live TPS and cleared sidebar usage while the next request runs

After completion: newest saved request total and cleared live TPS

mahiatlinux · 2026-08-29T08:32:32Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d975003c58

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mahiatlinux · 2026-08-29T08:46:50Z

@codex review

for more information, see https://pre-commit.ci

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e241779899

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mahiatlinux · 2026-08-29T09:00:59Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 659851dcca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-29T09:08:49Z

+          hasContent: summary.hasMessages,
+        };
      }
      const legacy = await listStoredChatMessages(thread.id).catch(() => null);


Keep the sidebar fallback batched

When the new sidebar-summary endpoint is unavailable or returns a transient error, the fallback thread records lack hasMessages, so this branch calls listStoredChatMessages for every thread concurrently. That helper performs both getChatThread and listChatMessages, meaning an installation with hundreds of chats can issue roughly twice as many HTTP requests on every sidebar refresh; this regresses the previous fallback's single batchListChatMessages request and can overload Studio precisely while the backend is already unhealthy. Reuse a batched message read for backend threads and reserve per-thread Dexie reads for actual legacy gaps.

Useful? React with 👍 / 👎.

No change: this is a compatibility-path performance concern after the summary endpoint fails, not a response correctness defect. The fallback preserves the same thread and message results, and speculative backend load does not meet the fix bar.

mahiatlinux · 2026-08-29T09:26:13Z

@codex review

for more information, see https://pre-commit.ci

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3cc8f9f229

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mahiatlinux · 2026-08-29T09:37:46Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ad5ca181c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-29T09:45:44Z

+      onStopClick?.();
+      return;
+    }
+    stopChatThread(activeThreadId);


Preserve cancellation before the first thread ID resolves

When generation starts on a brand-new chat before the remote thread ID has been assigned, activeThreadId is still null and the adapter deliberately stores the run under "__default". Calling stopChatThread(activeThreadId) therefore returns without dispatching anything, while the previous onStopClick cancelled the visible assistant-ui run; during this first-turn window, the Stop button does nothing. Fall back to the unresolved run key or invoke onStopClick when no active thread ID exists.

Useful? React with 👍 / 👎.

Add chat sidebar usage and live TPS

86438e3

Etherll requested a review from danielhanchen as a code owner August 29, 2026 01:45

[pre-commit.ci] auto fixes from pre-commit.com hooks

23d6885

for more information, see https://pre-commit.ci

chatgpt-codex-connector Bot reviewed Aug 29, 2026

View reviewed changes

Complete chat metrics recovery and sidebar integration

434b2e0

chatgpt-codex-connector Bot reviewed Aug 29, 2026

View reviewed changes

Comment thread studio/frontend/src/components/app-sidebar.tsx Outdated

mahiatlinux added 9 commits August 29, 2026 16:41

scope durable chat runs to their owner

1703919

keep durable event streams responsive

af667ad

report live safetensors throughput

b877a08

preserve live tps request ordering

fcb0f3a

clear stale tps monitor samples

2ba5856

load sidebar usage without message bodies

e2d65f8

publish sidebar usage during streaming

15e3aba

remove duplicate dead sidebar metrics

afdb60b

merge upstream main into chat metrics

40b4f40

mahiatlinux mentioned this pull request Aug 29, 2026

PR #9946 repro CI mahiatlinux/unsloth#28

Closed

pre-commit-ci Bot and others added 7 commits August 29, 2026 05:10

[pre-commit.ci] auto fixes from pre-commit.com hooks

3f299ff

for more information, see https://pre-commit.ci

keep sidebar storage test stub complete

f8f6993

scope live tps to the routed chat

df2bcff

keep the monitor timeout test alive

f72599d

make the monitor callback import verifiable

79b13f7

[pre-commit.ci] auto fixes from pre-commit.com hooks

e4f68d6

for more information, see https://pre-commit.ci

stop recovered generations from the composer

d975003

mahiatlinux mentioned this pull request Aug 29, 2026

run full ci for pr 9946 mahiatlinux/unsloth#32

Open

chatgpt-codex-connector Bot reviewed Aug 29, 2026

View reviewed changes

Comment thread studio/backend/routes/inference.py

Comment thread studio/frontend/src/features/chat/api/chat-generation-api.ts Outdated

mahiatlinux added 2 commits August 29, 2026 20:43

forward live tps through tool turns

50e9aca

stop reconnecting when monitor is disabled

e241779

[pre-commit.ci] auto fixes from pre-commit.com hooks

3c4e4a2

for more information, see https://pre-commit.ci

chatgpt-codex-connector Bot reviewed Aug 29, 2026

View reviewed changes

Comment thread studio/backend/core/inference/api_monitor.py Outdated

Comment thread studio/frontend/src/features/chat/utils/chat-history-storage.ts Outdated

mahiatlinux added 2 commits August 29, 2026 21:00

exclude tool pauses from live tps

7ed06ee

select sidebar usage chronologically

659851d

chatgpt-codex-connector Bot reviewed Aug 29, 2026

View reviewed changes

mahiatlinux added 2 commits August 29, 2026 21:25

count live tps from generated tokens

fc324b2

preserve newest sidebar assistant update

3cc8f9f

[pre-commit.ci] auto fixes from pre-commit.com hooks

e793e5f

for more information, see https://pre-commit.ci

chatgpt-codex-connector Bot reviewed Aug 29, 2026

View reviewed changes

Comment thread studio/backend/routes/inference.py

wire live tps into audio input

1ad5ca1

chatgpt-codex-connector Bot reviewed Aug 29, 2026

View reviewed changes

Uh oh!

Uh oh!

Conversation

Etherll commented Aug 29, 2026

Summary

Motivation

Changes

How to test

Screenshots

Related

Uh oh!

chatgpt-codex-connector Bot commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

mahiatlinux commented Aug 29, 2026

Review findings and fixes

Verification

UI evidence

Before: active generation without live TPS

After: exact live TPS and cleared sidebar usage while the next request runs

After completion: newest saved request total and cleared live TPS

Uh oh!

mahiatlinux commented Aug 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

mahiatlinux commented Aug 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

mahiatlinux commented Aug 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

mahiatlinux Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

mahiatlinux commented Aug 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

mahiatlinux commented Aug 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

chatgpt-codex-connector Bot commented Aug 29, 2026 •

edited

Loading