Priority holdback alongside eviction

1Setup

• One vLLM 0.30 replica on an H200 with --max-num-seqs 32, behind llm-d-router v0.10.0 with flow control and eviction enabled.
• Batch requests (priority −1) go through a queue processor that resumes an evicted stream from its saved tokens. Interactive requests (priority 10) go straight to the gateway.
• Holdback is priority-holdback-policy. A batch ceiling of 0.9 or 0.8 caps batch at 29 or 26 of 32 slots; interactive keeps ceiling 1.0 and evicts only once the pool is full.
• Batch: 3 jobs × 64 requests × 1,500 output tokens. 3 runs per arm; charts show the mean with min–max whiskers.

2Request-count saturation

With eviction alone, batch refills every freed slot, so about 9 in 10 interactive requests evict a batch request. A 20% reserve absorbs steady arrivals (0 evictions at 1 and 2 req/s, 18 vs 219 at 4 req/s). Bursts of 32 overwhelm any reserve.

Qwen2.5-1.5B, short prompts. Holdback lowers TTFT p95 to the 0.13 s floor at 1 req/s, where waiting on an eviction showed up, and costs up to ~3 s of batch time. Holdback without eviction never gets TTFT near the floor (40% reserve: p50 1.05 s vs 0.22 s under bursts).

3Token and hybrid saturation

Counting tokens, an interactive request weighs about a ninth of a batch request, so one eviction lets ~8 interactive requests in while vLLM has one free sequence. The rest queue inside vLLM, out of flow control's reach. Holdback keeps real slots free and fixes steady load; under bursts it does not help, and with hybrid counting it hurts.

concurrency-detector can count requests, estimated tokens, or both (hybrid: per endpoint, the larger ratio). Output tokens are estimated as min(1.5 × prompt, max_tokens), so these runs use long prompts: ~4.9k-token batch prompts and 500-token interactive prompts × 128 tokens on Qwen2.5-1.5B, with maxTokenConcurrency set to 32 batch requests' worth (204,800). Load is driven in-cluster with a per-run cache_salt, so TTFT is lower here than in section 2; compare only within this section.

4Without resume

Turning resume off (evicted requests restart from scratch after backoff) barely matters under steady load but nearly doubles batch work under bursts: ~288k tokens redone against 288k needed, and the batch takes 74% longer. Interactive TTFT does not change. Holdback does not rescue it: under bursts it trims redone work ~8% and still finishes later.

Request-count saturation, same long-prompt mix and in-cluster driver as section 3. Without a render_url the processor cannot resume, so an evicted request is retried from the start with exponential backoff; resume and immediate requeue cannot be switched off separately.

5A larger model: Qwen2.5-14B

Holdback removes evictions here too (166 → 11 → 2) but TTFT p95 stays at ~0.92 s in every arm, and batch takes 4–5% longer. Interactive latency comes from sharing the GPU with long-context batch work, not from waiting on evictions.

Request-count saturation, steady 1 req/s, ~4.8k-token batch prompts (96 requests) and 500-token interactive prompts × 128 tokens.

6Recommendations

• Keep resume on. Without it, each burst eviction throws away work and the batch takes 74% longer; holdback does not win that back.
• Keep request-count saturation with eviction as the baseline. It keeps interactive TTFT near the floor under both steady load and bursts.
• Add holdback, sized to steady interactive concurrency, when evictions themselves cost something (lost prefix cache, churn). Expect 1–6% more batch time.
• If saturation is counted in tokens on a server with a sequence cap, run holdback: token counts alone admit more requests than the server can run.
• Neither setting absorbs bursts larger than the reserve. That needs request-count backpressure or a larger reserve.

7Caveats and open questions

• Saturation is counted in requests except in section 3. The KV-cache saturation detector is untested. A different token cap would move the section 3 results.
• The 45 s batch job misses deadlines only under eviction (100% met without it on the same build). Leading suspect, not yet confirmed: all 192 batch requests reach the gateway at once, so the processor's earliest-deadline-first order never applies and a resumed request rejoins the gateway's FCFS queue behind later-deadline work.
• Why hybrid counting with holdback gets worse under bursts is not yet explained.
• The 14B runs did not use a per-run cache_salt, so prefill comparisons are left out; its 20% holdback arm has 2 runs.
• One interactive request out of 1,440 (tokens mode, bursts, eviction only) got a 503.
• Every completed batch output was token-identical to a direct vLLM run (101 runs).