Keep ModelManager lock generations and fixed-size cache bookkeeping consistent - #2821

Why

ModelManager could remove a per-model lock entry while the same lock
generation was still held or referenced by waiting callers.

A subsequent add_model() for the same resolved model identifier could then
create a new generation. A caller waiting on the old lock and a newcomer using
the new lock could initialize the same model concurrently.

The same mutation lifecycle exposed related consistency problems in manager
decorators:

• an unlocked warm-model fast path could bypass an already-established
  remove/add order;
• failed removal or eviction could leave an identifier registered in the
  underlying manager but absent from the LRU queue;
• concurrent cold additions could create duplicate queue entries;
• aliases were not consistently used as the resolved lifecycle and cache key;
• failed direct removal could change the LRU position even though the mutation
  did not complete;
• a failed multi-model eviction could drop skipped pinned entries and the
  failed eviction candidate from the queue.

Fixes #2819.

What changed

• Store each per-model lock and its active reservation count in one lifecycle
  entry.
• Reserve the exact entry before waiting for its lock.
• Retain that generation through holder, waiter, timeout, and exception paths.
• Retire an entry only when:
  • it is still the current generation;
  • it has no active reservations; and
  • the resolved model identifier is absent.
• Treat decorator-level existence checks as hints and make the final warm-model
  decision under the stable per-model lifecycle lock.
• Use the resolved model identifier consistently for alias-sensitive lifecycle
  and cache bookkeeping.
• Reconcile fixed-size cache entries after both successful and failed
  add/remove attempts.
• Preserve the exact existing LRU position when a direct removal fails.
• Restore skipped pinned entries and failed eviction candidates in their
  original relative order.
• Do not restore entries whose removal completed successfully before a later
  eviction failure.
• Ensure completed additions and removals leave at most one queue entry per
  resolved identifier.
• Add deterministic concurrency and rollback coverage using explicit
  synchronization points rather than scheduling sleeps.

Public method signatures and configured per-model lock-acquisition timeout
values are unchanged. The shared lifecycle-entry guard is not held while
waiting for a per-model lock, constructing a model, or clearing model
resources.

This change addresses the demonstrated process-local load/remove/reload races
and fixed-size cache bookkeeping inconsistencies within one ModelManager
instance. It does not introduce active inference-use leases or cross-process
coordination.

Verification

The parent behavior was verified from:

aa9306bd8e3c674c3c0a324635307b7272eb54ef

Parent regression proof

The original lock-generation regression failed on the unchanged parent:

1 failed, 8 deselected

It observed two live lock generations and two concurrent constructors for the
same resolved model identifier.

The expanded mutation regression selection produced:

7 failed, 4 deselected

It reproduced:

• split lock generations;
• decorator warm-path bypass of an already-established remove/add order;
• failed direct-removal rollback;
• failed eviction rollback;
• duplicate LRU entries;
• alias lifecycle inconsistency;
• eviction/reload inconsistency.

The additional exact-order rollback tests all failed on the unchanged parent:

3 failed

The tests cover:

• test_fixed_size_cache_failed_direct_remove_preserves_exact_lru_order
• test_fixed_size_cache_failed_eviction_restores_skipped_pinned_order
• test_fixed_size_cache_failed_multi_eviction_keeps_only_registered_models

Candidate

The expanded mutation regression selection passed:

7 passed, 4 deselected

The additional exact-order rollback selection passed:

3 passed

Targeted manager tests:

python -m pytest \
  tests/inference/unit_tests/core/managers/test_model_lock_lifecycle.py \
  tests/inference/unit_tests/core/managers/test_decorators.py -q

23 passed

Manager and cache suites:

python -m pytest \
  tests/inference/unit_tests/core/managers \
  tests/inference/unit_tests/core/cache -q

300 passed

Full applicable CPU unit suite with a fresh model cache:

CACHE_ROOT="$(mktemp -d)"
MODEL_CACHE_DIR="$CACHE_ROOT/models" \
INFERENCE_HOME="$CACHE_ROOT/home" \
python -m pytest tests/inference/unit_tests -q

2580 passed, 10 skipped, 3 failed

The same three failures reproduce on the unchanged parent:

• test_factory_logs_construction_failures_before_falling_back
• test_cached_model_aggregation_excludes_conflicting_metadata
• test_index_list_parameters_by_frame_id_warns_once_per_key

They are existing logging-capture failures where the warning is emitted but is
not available through caplog.

Incomplete implementations rejected

The regression coverage rejects intentionally incomplete alternatives:

• retiring an entry immediately after the holder releases it;
• protecting only lock creation without retaining the generation;
• keeping the unlocked decorator warm return;
• omitting failed-candidate eviction rollback;
• omitting skipped-pinned rollback;
• moving a registered model to the end after failed direct removal.

Quality gates

make check_code_quality

Passed:

• Black: 830 files unchanged
• isort
• fatal Flake8 checks
• scoped formatting and lint checks
• git diff --check

The informational Flake8 pass still reports the existing upstream warning set.

Not verified

• Live GPU or production-server behavior.
• A measured RAM/VRAM spike or production OOM.
• Upstream Python/platform CI has not run on this branch yet.
• Removing or evicting a model while inference is actively using it.
• Queue reconciliation after failed inference, preprocess, or other
  non-mutation access paths.
• Cross-process, cross-worker, cross-pod, or independent-manager coordination.
• Lock fairness or starvation guarantees.
• A strict global max_size guarantee during simultaneous cold loads of
  different identifiers.

Risks / rollback

The lifecycle-entry guard briefly serializes entry reservation and retirement,
but it is never held during per-model lock waiting, model construction, or
resource cleanup.

Post-mutation queue reconciliation uses a blocking _queue_lock acquisition so
a secondary bookkeeping timeout cannot replace the original add/remove result
or exception. It can therefore wait behind an ongoing eviction.

The existing eviction path already holds _queue_lock while removing an
evicted model and running clear_cache(). This patch does not acquire
_queue_lock while holding a per-model lifecycle lock, so it does not
introduce the opposite lock order. Reducing the existing eviction critical
section is outside this change.

Different resolved model identifiers continue to use independent per-model
locks.

Reverting this PR restores the previous mutation and cache-bookkeeping
behavior.

Migrations / external effects

No migrations or public API changes.

A warm add_model() call that overlaps an already-started removal may now wait
for that removal to complete instead of returning from an unlocked decorator
fast path. Configured per-lock acquisition timeout values are unchanged, although this path may perform two sequential lock acquisitions.

Screenshots / preview

Not applicable.