Flow Control Interactive

Home

Settings

Scroll horizontally to explore the diagram →

Swipe to explore
Mechanism examples use illustrative values. Measured results are labeled on the evidence pages. premium interactive clients standard bursty app traffic batch offline jobs Inference gateway Envoy proxy ext_proc gRPC x-llm-d-inference-objective x-llm-d-inference-fairness-id Endpoint Picker Objective lookup InferenceObjective → priority objective found → priority 100 missing or unbound → default 0 Admission FlowKey = tenant × priority Router queues priority 100 priority 0 priority −10 Pool saturation 0.35 Dispatch ceiling · 1.0 Model-serverendpoints Endpoint A waiting KV cache Endpoint B waiting KV cache
selected priority waits at its ceiling

Flow control
Per-tenant load
Seed 42 · reset to compare
How to compare modes Use the same loads and tick for each mode. Switching modes restarts seed 42. Repeat any burst or load change at the same tick.

*Illustrative model, not a recording or latency prediction. The unprotected baseline omits legacy shedding of negative-priority requests at saturation. Queued requests expire after 60 simulated seconds.

Keyboard shortcuts
Next step
→ or Space
Previous step
←
Next topic
↓ or PgDn
Previous topic
↑ or PgUp

Details