Skip to content

daemon: memory watchdog decides on current RSS, not lifetime peak - #2249

Draft
scouredimage wants to merge 5 commits into
git-ai-project:mainfrom
scouredimage:fix/2244-5-watchdog-current-rss
Draft

daemon: memory watchdog decides on current RSS, not lifetime peak#2249
scouredimage wants to merge 5 commits into
git-ai-project:mainfrom
scouredimage:fix/2244-5-watchdog-current-rss

Conversation

@scouredimage

@scouredimage scouredimage commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Part 5/7 of the #2244 stack.

The watchdog sampled getrusage ru_maxrss - a monotonic lifetime high-water mark - so one transient spike condemned the process on every later poll even after the memory was freed; the daemon could only ever abort, never recover. Decisions now use current RSS (proc_pidinfo on macOS, /proc/self/statm elsewhere on unix, working set on Windows); the emergency log carries both rss_bytes and peak_rss_bytes.

Stated tradeoff: a spike that allocates and frees entirely within the 1 s poll is no longer detected; the ingestion caps earlier in the stack bound those transients at the source.

Stacked on #2248 - review the last commit only for this part's diff.

Stack

  1. streams: cap JSONL line size and transcript batch bytes #2245 - streams: cap JSONL line size and transcript batch bytes
  2. daemon: serialize transcript metric events incrementally #2246 - daemon: serialize transcript metric events incrementally
  3. daemon: cap control/trace socket line size #2247 - daemon: cap control/trace socket line size
  4. daemon: advance stream watermark before batch processing #2248 - daemon: advance stream watermark before batch processing
  5. daemon: memory watchdog decides on current RSS, not lifetime peak #2249 - daemon: memory watchdog decides on current RSS, not lifetime peak
  6. daemon: never defer self-restart while processing is stalled #2250 - daemon: never defer self-restart while processing is stalled
  7. streams: cap the initial backfill of first-seen stream files #2251 - streams: cap the initial backfill of first-seen stream files

Each PR targets main; the incremental diff of part N is its last commit. Root-cause analysis, production evidence, and the repro live in #2244.

Verification

  • Deterministic repro from Daemon RSS balloons on large agent transcript sweeps; watchdog abort loops from watermark 0 and blocks traced git (v1.6.24) #2244 (315 MB synthetic Cursor transcript + one commit, 384 MB
    daemon memory limit): unpatched v1.6.24 aborts in ~1 s and re-aborts after respawn
    (watermark 0); with the full stack the daemon survives, peak RSS 76 MB vs 589 MB,
    zero memory emergencies, traced-git probes at 0.05-0.07 s throughout.
  • cargo fmt --check clean; cargo test --lib: 2,401 passed. Three failures
    (commands::upgrade::...pending_update, daemon::...conflict_resolution_note_read_errors...,
    git::authorship_traversal::...ai_touched_files...) reproduce identically on clean main
    in this environment - pre-existing, unrelated.

Transcript batches were bounded by event count only (1000), and
read_jsonl_line buffered lines of unbounded length. A transcript whose
events embed file contents (normal for agent tool results) could put
hundreds of MB into a single batch, which downstream redaction and
metrics conversion amplify several times over -- ballooning daemon RSS
past the memory watchdog within seconds (git-ai-project#2244).

- read_jsonl_line: cap a single line at MAX_JSONL_LINE_BYTES (8 MiB).
  The read is byte-based (read_until), NOT read_line: the cap can
  slice a multi-byte character, and read_line's UTF-8 validation would
  return InvalidData instead of classifying the line as Oversized --
  wedging the stream at a fixed watermark. Oversized lines are skipped
  without being buffered (skip_until); callers advance their watermark
  past them via the new JsonlLineState::Oversized state. Non-UTF-8
  content within the cap keeps read_line's InvalidData contract.
- All JSONL byte-offset stream parsers (claude, codex, copilot,
  cursor, droid, gemini, pi, windsurf): stop a batch early once
  MAX_BATCH_BYTES (8 MiB) of raw JSON has been accepted; remaining
  events arrive in later batches.

Not covered here: amp, continue_cli, and opencode parse whole files
into a DOM (serde_json::from_reader) and need a separate treatment --
called out in git-ai-project#2244 as follow-up.

Part 1/7 of the git-ai-project#2244 fix stack.
store_metrics_in_db materialized a Vec<String> of every re-serialized
event while the full Vec<MetricEvent> (each holding the redacted JSON
tree) was still alive -- two complete copies of a transcript batch
resident at once (git-ai-project#2244).

The stream worker now serializes each MetricEvent as it is built and
drops the tree immediately; persistence takes the pre-serialized rows
via the new persist_metric_jsons_blocking. Existing callers of
persist_metrics_blocking are unchanged (it now delegates to the same
insert path).

Behavior note: previously one unserializable event failed the whole
batch (all-or-nothing insert); the stream-worker path now drops the
failing event with a warning and persists the rest. These are
best-effort diagnostics, and partial persistence beats losing the
batch.

Part 2/7 of the git-ai-project#2244 fix stack (stacked on 1/7).
read_json_line buffered socket lines of unbounded length. Both daemon
sockets speak line-delimited JSON with bounded frames (trace2 events,
control request headers; checkpoint bodies travel separately after the
header line), so a runaway line can balloon daemon RSS past the memory
watchdog (git-ai-project#2244).

- Lines beyond MAX_SOCKET_LINE_BYTES (4 MiB) are discarded without
  being buffered (byte-based read_until + skip_until; read_line would
  turn a cap that slices a multi-byte character into InvalidData and
  drop the connection).
- The control loop ANSWERS an oversized request with an error response:
  silently eating it would stall the client until its socket timeout,
  after which it reconnects and resends the same request forever.
- The trace loop skips oversized frames and keeps reading.

Part 3/7 of the git-ai-project#2244 fix stack (stacked on 2/7).
The stream watermark was persisted only after a batch was converted
and stored. When batch processing OOM-aborts the daemon (memory
watchdog), the watermark is still at the batch start, so the respawned
daemon re-reads the same bytes and aborts again -- the abort loop
observed in production, 22 aborts in 5 days on one machine (git-ai-project#2244).

Transcript metrics are best-effort telemetry; losing one batch of
diagnostics on a crash is a safer failure mode than a crash loop that
blocks traced git commands. The watermark now advances immediately
after a batch is read.

Part 4/7 of the git-ai-project#2244 fix stack (stacked on 3/7).
The watchdog sampled getrusage ru_maxrss, a monotonic lifetime
high-water mark. One transient allocation spike therefore condemned
the process on every subsequent poll even after the memory was freed:
the daemon could only ever abort, never recover (git-ai-project#2244).

Decide on the CURRENT resident set size instead (proc_pidinfo on
macOS, /proc/self/statm on other unix, working set on Windows). The
emergency log carries both rss_bytes (decision input) and
peak_rss_bytes (context). Test-support env var names are unchanged;
the sample sequence now represents current-RSS values.

Tradeoff, stated explicitly: with 1 s polls, a workload that spikes
above the limit and frees within the poll interval is no longer
detected (the peak-based check caught the first spike ever). The
watchdog exists to stop sustained balloons, and an undetected
transient spike that the allocator survives is strictly better than a
guaranteed abort loop; the ingestion caps earlier in this stack bound
those transients at the source.

Part 5/7 of the git-ai-project#2244 fix stack (stacked on 4/7).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant