[V0][Metrics] Deprecate some KV/prefix cache metrics #14136
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
vllm:num_requests_swapped, vllm:cpu_cache_usage_perc and vllm:cpu_prefix_cache_hit_rate will no longer be relevant in V1 since we no longer implement KV cache offloading. So these metrics should be considered deprecated. And as agreed in vllm-project#12592, we have added prefix_cache_queries and prefix_cache_hits counters to replace the prefix_cache_hit_rate gauge as it allows the interval over which the hit rate is calculated to be controlled in a Prometheus query like: ``` rate(prefix_cache_queries[5m]) / rate(prefix_cache_hits[5m]) ``` In theory, we could ease the transition be implementing the old hit rate metric in V1 and the new queries/hits metrics in V0, but it's probably not worthwhile unless we learn the hit rate metric is heavily used by V0 users. Signed-off-by: Mark McLoughlin <[email protected]>
markmc
force-pushed
the
metrics-v1-deprecate-prefix-cache-hit-rate
branch
from
e89b9fe to
a2ec1f5
Compare
robertgshaw2-redhat
added
the
ready
) Signed-off-by: Mark McLoughlin <[email protected]> Signed-off-by: Linkun Chen <[email protected]>
) Signed-off-by: Mark McLoughlin <[email protected]> Signed-off-by: Johnny <[email protected]>
vllm:num_requests_swapped,vllm:cpu_cache_usage_percandvllm:cpu_prefix_cache_hit_ratewill no longer be relevant in V1 since we no longer implement KV cache offloading. So these metrics should be considered deprecated.And as agreed in #12592, we have added
prefix_cache_queriesandprefix_cache_hits countersto replace theprefix_cache_hit_rategauge as it allows the interval over which the hit rate is calculated to be controlled in a Prometheus query like:In theory, we could ease the transition be implementing the old hit rate metric in V1 and the new queries/hits metrics in V0, but it's probably not worthwhile unless we learn the hit rate metric is heavily used by V0 users.