Choose and validate flow-control settings

Before the benchmark

1Define repeatable traffic and the test question.
2Verify endpoint, runner, versions and active controls.
3Validate a test request. Check component metrics.

Benchmark

1Define traffic and criteria
2Sweep each workload alone
3Add competing workloads
4Calibrate detector and limits
5Test interactive protection
6Check lower-priority progress
7Test fairness within a band
8Validate full mix and recovery

Objectives

Protect interactive latency from competing work.
Keep lower-priority work progressing.
Share service within each priority band.

View

Swipe sideways to explore the connected map. With a keyboard, focus the map and use the left and right arrow keys.

Where did time increase?

Investigate a baseline failure

Did the runner deliver the intended demand? Check actual send times and attempted/completed/failed counts. If not, fix the runner and repeat the same load.
Does the same workload miss the latency criteria at low load? If yes, inspect request sizes, cache preparation and engine processing time. Lower router ceilings cannot remove standalone processing cost.
Does delay appear only as demand rises? Use the component measurements below to distinguish router waiting, engine waiting and processing. Several changing components require matched traces.
Change one identified cause, then repeat the failed load. Compare against the original result. If the cause is still unknown, collect the missing timing before changing policy.

Return to the baseline test →

Compare component timing

Evidence to compare	What it suggests	Next test
`llm_d_epp_flow_control_request_queue_duration_seconds` `llm_d_epp_flow_control_queue_size`	More time waiting for router admission.	Compare the detector and band ceiling. Check the loaded policy and saturation at the same time.
`vllm:request_queue_time_seconds` `vllm:num_requests_waiting`	More time queued inside the engine.	Test whether controlling dispatched load reduces this wait without losing required completions.
Per-request prefill/decode timing; prefix-cache queries and hits	Processing or cache reuse changed.	Identify processing/cache interference. If contention is the cause, compare reduced dispatched load at matched offered demand and cache preparation; check latency, completions, failures and required progress. Standalone processing cost calls for workload or engine investigation.
Filter logs/traces showing removed candidates; scheduling errors	Endpoint selection was restricted.	Check the named filter and its thresholds. Queue metrics alone cannot identify this cause.
Missing timing, unmatched requests or several components changing	The cause is not established.	Collect matching request traces and repeat. Compare stage durations; never subtract unrelated p95 values.

Metric names and version references →

Control roles and configuration

Role	llm-d configuration	Job
Detector	`flowControl.saturationDetector.pluginRef`	Calculates the load signal.
Admission ceiling	`flowControl.usageLimitPolicyPluginRef`	Sets the saturation limit at which a priority band waits.
Priority and fairness	`flowControl.priorityBands` `fairnessPolicyRef` / `orderingPolicyRef`	Select queued work for dispatch; sharing within a band is separate from priority between bands.
Endpoint filter	`schedulingProfiles[].plugins[].pluginRef`	Filters endpoint candidates before selection.
vLLM	Engine scheduler configuration	Processes dispatched work; router priority alone does not instantly interrupt it.

Version boundary: This guide retains router 30f06d9c. At newer 8bd261f1, concurrency filtering returns all candidates if every candidate would be filtered. The older pin does not. These behaviors must not be combined in a test interpretation. Newer implementation.

Admission and filtering are different comparisons

Match flowControl.saturationDetector.pluginRef and schedulingProfiles[].plugins[].pluginRef to plugins[].name. Inspect the loaded profiles, not only the input YAML.

At router revision 30f06d9c, the loader adds the selected detector to each scheduling profile when it implements filtering. A separately named filter does not remove that injected filter. The detector comparisons here therefore test admission and filtering together under this wiring.

For request filtering, the threshold is floor(maxConcurrency × (1 + headroom)); token filtering uses maxTokenConcurrency in the same formula. Hybrid excludes an endpoint when either threshold is reached. Holding headroom fixed does not keep these thresholds fixed when their caps change.

To isolate admission, first establish a supported deployment where the swept detector is absent from the filter chain and the separate filter configuration stays unchanged. That isolation is not established by this guide. An admission-ceiling test can instead hold detector and filter settings fixed and change only the usage-limit policy.

Loader wiring · Filter calculation

featureGates chooses the admission path. Disabling flowControl can leave legacy admission active. Verify the deployed version and loaded filters.