[POC — do not merge] Video sources: connector agent + warm processor + local media plane - #2616

… + batch docs

- batch jobs fetch the signed URL to a local temp file before starting the
  pipeline: a URL fails VideoSource's is_file check, which not only applied
  stream buffer semantics but made the pipeline treat EOF as a dropped
  stream and reconnect — replaying the file forever, so batch jobs never
  completed; a local path gets true file semantics (every frame, natural
  end); temp file cleaned up on completion/stop
- HANDOFF: document the media plane — annotated mediamtx.yml (TCP-only
  RTSP, local WHEP, disabled protocols, debug API, and the no-auth
  catch-all path as the flagged production gap), stream naming
  conventions, unpinned-version note; update batch mode description to
  the download-first implementation

Detailed plan for deploying mediamtx + processors to ck8s-stg: repo
decision (roboflow-infra chart over async-serverless, with rationale and
migration path), target topology + hostnames, chart contents, processor
image, the full control-protocol delta table (8 arrows + 3 security
items), platform-side reachability, ordered execution steps with the
L4-LB risk called out and a GCP-VM fallback, and day-2 gaps.

… lease semantics)

…ignal

- Pub/Sub keeps cells self-contained (no dependency on async-serverless's
  RabbitMQ being co-deployed) and removes the Firestore-to-queue bridge:
  functions publish natively, processors StreamingPull outbound from any
  cluster; per-cell subscriptions via attribute filters
- primary autoscaling signal is the warm-pool invariant (busy-gauge +
  KEDA: replicas = busy + MIN_IDLE), composed with Pub/Sub backlog for
  batch bursts; pod-deletion-cost guards scale-down from killing
  monitoring jobs

…ignaling-only ingress is not enough)

…b wake-ups, busy metric, Dockerfile

- pipelines run with the api key carried on the job (workspace identity
  follows the job); worker key stays for platform calls only
- sim replays publish to the credentialed simPublishUrl from claim
- on batch completion, results upload to GCS via platform-signed URLs
  before the job reports completed; local files remain as fallback
- optional Pub/Sub subscriber (--pubsub-subscription): messages are
  wake-ups only, claim stays the source of truth; busy workers nack so
  backlog stays visible to autoscaling; claim serialized via a lock
  shared with the poll loop; --public-url reported as processorUrl
- /metrics: video_processor_busy gauge (KEDA warm-floor signal)
- Dockerfile: inference GPU image + ffmpeg CLI + pubsub client
- docs: deploy plan marked implemented; HANDOFF gaps updated

- HANDOFF §6: 'Consuming results' — the four consumption cases; SSE via
  the gateway in-cluster (proxy_buffering off is load-bearing), the
  worker-vs-job coupling limitation, and the job-addressed production
  contract (/video-jobs/{id}/events; smart proxy vs published-stream
  fan-out); bandwidth analysis (JSON is never the problem, redaction
  keeps it that way); hour-partitioned bucket storage for streams
- HANDOFF §8: processor auth plan (per-job tokens), reap-to-requeue as
  a named prerequisite, multi-stream-per-GPU + bulkiness gap
- HANDOFF §9 (new): team-alignment direction — ready-pool scaling
  (label-detach + self-delete, replaces replica scaling), Dedicated
  Deployments as workspace-pinned placement of lean processor processes,
  per-block local/remote execution in the EE (bulkiness, externalizable
  state, selective VLM externalization), application-level stream
  control (fps/substream/WebRTC signaling), rtsp-bridge-poc relationship
- DEPLOY_PLAN §8: scaling section revised — ready-pool model replaces
  the KEDA warm-floor-on-StatefulSet design (deletion-cost doesn't
  exist for StatefulSets); chart swap flagged as in progress

…lete on finish)

- PodSelf: raw k8s API self-management with mounted SA credentials —
  detaches the pod from the ready pool the moment a job is claimed (the
  ReplicaSet refills instantly, before the slow pipeline start) and
  deletes the pod when the job ends (completed/cancelled/failed);
  workers are single-use, so the only pods that ever terminate chose to
- pool mode gates on PROCESSOR_POOL_MODE + in-cluster credentials;
  local dev is unchanged (long-lived worker, poll loop)
- start failures retire without posting terminal error: often transient
  (relay stream not up yet) — the platform requeues via the reaper and
  the attempts cap handles genuinely poison jobs
- public URL now derives from GATEWAY_PUBLIC_BASE + POD_IP (/ip-a-b-c-d
  gateway routing) since pool pods have random names
- docs: HANDOFF + deploy plan flipped from 'designed' to implemented

…vior

…ath detection

- /status no longer serializes the claim payload (workspace api key,
  signed/credentialed URLs) — public endpoint gets job metadata only
- claims start the pipeline on its own thread: the poll loop keeps
  heartbeating through source download + model load, so slow starts no
  longer get reaped at 30s and double-processed; local claim state is
  set synchronously under the claim lock so no second claim slips in
- lifecycle lock: stop_job waits for an in-flight start, so cancel
  during 'starting' can't leak a running pipeline with no job (which
  also crashed the poll loop via unguarded self.job['id'])
- pipeline-end watcher now covers stream jobs too: a dying stream
  releases the job (reaper requeues) instead of hanging 'running'
- cancelled jobs no longer upload partial results / report completed
  (cancelling flag consulted by the watcher)
- pool mode: single-use enforced at claim (had_job), download-failure
  path retires like its siblings, SA token re-read per request (bound
  tokens expire), PROCESSOR_POOL_MODE parsed strictly, crash-restarted
  containers inside detached pods self-delete at startup instead of
  claiming as hidden workers
- recorder: per-event videoFrame index (conditional image outputs no
  longer break mp4/JSONL alignment), wedged ffmpeg killed on finalize
  timeout, broken mp4 excluded from upload; download temp tracked from
  creation; start-failure path cleans pipeline + download
- poll loop survives any exception (it is the heartbeat + cancel
  channel); status posts carry processorId for platform identity binding
- docs: mediamtx auth location clarified, claim scope step-up documented,
  video_jobs field list completed, stale plan sections updated

Chrome on an https app page refuses to load <img> subresources from
IP-literal insecure hosts ('not upgraded to HTTPS because its URL's
host is an IP address') while allowing hostname loopbacks — so the
MJPEG preview silently never painted when the app runs on
https://localapp.roboflow.one. fetch/SSE take a different code path
and worked, which made this maximally confusing.

…ching

OutputPublisher: re-encodes the designated image output's JPEGs to H.264
(ultrafast/zerolatency/baseline, constant 12fps tick duplicating the
latest frame) and publishes RTSP to the relay at the credentialed
outPublishUrl from the claim payload. Driven entirely by the watch
signal riding the status-poll response: starts when a viewer's TTL is
fresh, restarts when the viewer switches outputs, stops within a poll
interval of the TTL lapsing — result video never streams unwatched.
One publish serves every WHEP viewer through the relay; MJPEG endpoint
remains for debugging.

…ied e2e)

…names, all-Terraform deploy note

Code dir stays development/video_poc (source tree rename would churn the
whole PR); all deploy-facing names — env prefix, namespace, chart, AR
repo, Spacelift stack — are now video-proc, matching roboflow-infra.

Latency quick wins for the relay-published output:
- pump pushes each frame the moment the pipeline produces it (native
  fps) instead of resampling on a 12fps tick — removes up to ~83ms of
  staleness and duplicate-frame cadence
- ffmpeg stamps PTS from wallclock arrival (-use_wallclock_as_timestamps)
  and encodes VFR, so stream time tracks reality exactly
- keyframe forced every second of stream time (WHEP viewers can't render
  until an IDR): join latency bounded at ~1s, was up to 2s

When VIDEO_PROC_SERVICE_SECRET is set (managed pool), platform calls
authenticate with x-video-proc-service-access-token instead of a
workspace api_key; results calls carry processorId for the fleet
ownership check. Unset = self-hosted behavior, unchanged.

… gap

…t + watchdog hot-swap

- AiortcWhipPublisher: same three-method contract as the ffmpeg/RTSP
  publisher, WebRTC transport — frames stamped with actual arrival time
  (no CFR faking), keyframes on relay PLI instead of a fixed GOP, no
  RTSP layer; JPEG decode happens on the publisher's own thread, never
  the pipeline sink's (raw-frame slot is a later increment)
- transport selection: VIDEO_PROC_PUBLISHER env (per-worker default,
  rtsp), publisherTransport on the claim payload (per-job override),
  graceful rtsp fallback when aiortc is missing
- watchdog enforces the invariant that customer inference latency beats
  preview transport: if decode→result latency degrades >80%/40ms for 3
  poll ticks while the in-process publisher runs, hot-swap to the
  subprocess rtsp transport mid-job — viewers reconnect through the
  same relay stream in ~1s, the pipeline never notices
- Dockerfile: aiortc

…th the decision trigger

Not yet because rolling updates ignore deletion cost (every deploy would
rotate every long-running stream) and the warm floor becomes autoscaler
arithmetic; switch when requeue + relay reattach + externalized block
state make stream re-placement verifiably seamless.

- OPENCV_FFMPEG_CAPTURE_OPTIONS (nobuffer/low_delay, before any capture
  opens): without these the FFmpeg demuxer/decoder holds frames AHEAD of
  the first place the pipeline can drop, invisible to every downstream
  latency measure (frame_timestamp is stamped at grab, after this buffer)
- stream jobs cap the capture queue via video_source_properties
  buffersize=1 (CAP_PROP_BUFFERSIZE; ignored harmlessly where unsupported)

…ion; raw frames to the publisher

Latency: the constant 16-frame (~533ms) source→output delta was H.264
FRAME-THREADED decoding — OpenCV's FFmpeg capture opens the decoder with
threads=auto, which holds (threads-1) frames INSIDE the decoder, ahead
of frame_timestamp and invisible to every latency stat. Stream jobs now
set OPENCV_FFMPEG_CAPTURE_OPTIONS threads;1 (the connector's zerolatency
streams are sliced, so single-thread 720p30 decode is cheap) plus
nobuffer/low_delay and CAP_PROP_BUFFERSIZE=1; batch keeps throughput
defaults. Both knobs are per-job overridable (captureOptions,
captureBufferSize on the claim payload).

Serialization takeover: serialize_results=False; the sink serializes
non-image outputs through inference's own serialize_wildcard_kind
(event content identical to before), JPEG-encodes image outputs itself
for the MJPEG/recorder store, and keeps raw ndarrays in a parallel
store that feeds the whip publisher directly — the JPEG round trip on
the publish path is gone (no decode, no generation loss).

…bserved: avfoundation reshuffles on lid close/Continuity)

…w_delay

The constant ~16-frame source->output lag was ffmpeg's h264 decoder
holding a DPB-sized frame-reorder buffer. Disabling it requires
AV_CODEC_FLAG_LOW_DELAY on the CODEC context, which cv2's
OPENCV_FFMPEG_CAPTURE_OPTIONS (format-level only) cannot set - no cv2
configuration can fix this. LowLatencyRtspProducer (PyAV, already a
dependency via aiortc) sets the flag and single-threaded decode
directly and plugs into VideoSource's producer-factory path; stream
jobs now ingest through it. captureOptions still apply (libavformat
open options for stream mode, cv2 env for batch).

Measured on the new pixel-clock latency harness (32-bar wallclock
pattern; publish/probe/probe-ffmpeg modes plus a WHEP-side probe):
cv2 baseline 586ms -> PyAV producer 40ms glass-to-glass; full pipeline
source->annotated-output now ~20-60ms (was ~600ms).

Claude-Session: https://claude.ai/code/session_01DwQ66z8rDMMh6A6hacqLuN