8000 R-KV/vLLM at main · Zefan-Cai/R-KV · GitHub
[go: up one dir, main page]

Skip to content

Latest commit

 

History

History

vLLM

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

R-KV on vLLM (v0.25.1)

Decoding-time, redundancy-aware KV-cache compression for vLLM v1.

While a model generates a long output, R-KV periodically evicts the unimportant and redundant past tokens, keeping only a fixed budget of KV entries per request — freeing GPU memory while preserving generation quality. This directory ports R-KV onto a pinned vLLM v0.25.1 baseline, using the same patch-not-fork layout as the SGLang port.

  • Algorithm — joint scoring of importance (attention over a recent observation window) and redundancy (key cosine-similarity); keep the top budget tokens per request.
  • Integration — true physical eviction in vLLM's paged KV cache: the attention backend overwrites each request's surviving KV in place, and the model runner keeps rotary positions logical while writing new KV to the physical (shrunken) slots.

Status: re-ported against v0.25.1 and validated on an NVIDIA H100. The previous proof-of-concept lived on a much older vLLM V1. This port re-implements the runtime wiring against v0.25.1's rewritten v1 stack. The rkv/ algorithm is CPU-tested for bit-parity with the reference; the wiring patch applies cleanly to a pristine v0.25.1 tree; and the end-to-end serving path is validated on GPU (Qwen2.5-0.5B/Math-7B): compaction fires, output is coherent, and with a budget large enough that no eviction occurs the output is byte-identical to Full-KV (proving the position/slot wiring is transparent). See docs/OPTIMIZATIONS.md for the validation record, known limitations, and roadmap.

R-KV runs on vLLM's V1 GPU model runner. v0.25.1 defaults to a newer V2 runner; the port auto-selects V1 whenever R-KV is enabled so compression is never silently skipped (V2 support is on the roadmap).

Why a patch, not a fork?

R-KV touches vLLM in a small, purely additive way: one self-contained package (rkv/) plus ~849 lines of wiring across 13 existing files. Instead of vendoring the entire vLLM tree, this directory ships:

  • rkv/ — the R-KV code (browsable, the source of truth);
  • patch/rkv-vllm-0.25.1.patch — the 13-file wiring diff;
  • scripts/apply_rkv.sh — clones the exact pinned vLLM commit, drops in rkv/, and applies the patch.

So you always see exactly what R-KV changes, and you build against a known-good upstream commit.

vLLM/
├── README.md                      # you are here
├── requirements-rkv.txt           # dependency notes (R-KV adds none)
├── rkv/                           # R-KV package (algo, integration)
├── patch/rkv-vllm-0.25.1.patch    # wiring diff (13 upstream files)
├── scripts/apply_rkv.sh           # clone pinned vLLM + drop in rkv/ + apply patch
├── benchmark/                     # GSM8K accuracy/throughput harness + H100 results
├── docs/                          # DESIGN, IMPLEMENTATION, OPTIMIZATIONS, REPRODUCE
└── tests/                         # GPU-free CPU unit tests

Pinned upstream

Component Pin
vLLM release tag v0.25.1 (commit 752a3a504485790a2e8491cacbb35c137339ad34)
Runtime path vLLM v1 engine, FlashAttention backend
Torch / CUDA / kernels resolved by vLLM's own build for your platform

R-KV itself adds no new runtime dependencies — the algorithm uses only torch, which vLLM already requires.

Step-by-step

# 1. Build the patched vLLM tree (clones v0.25.1, copies rkv/, applies patch).
scripts/apply_rkv.sh

# 2. Install it (source build — needs CUDA + a GPU).
pip install -e vllm-src

# 3. Serve a model with R-KV at **best throughput**. Compression activates
#    automatically once BUDGET and BUFFER are both > 0. Do NOT pass
#    --enforce-eager: R-KV auto-selects PIECEWISE cudagraph (attention stays
#    eager so its hooks fire; the rest of the layer is graphed). ASYNC=1 turns
#    on async scheduling (+16.7% decode tok/s). See benchmark/launch_server.sh.
VLLM_V1_R_KV_BUDGET=256 VLLM_V1_R_KV_BUFFER=64 VLLM_V1_R_KV_ASYNC=1 \
  vllm serve Qwen/Qwen2.5-Math-7B-Instruct

#    ...or use the wrapper (best-throughput flags baked in):
#    benchmark/launch_server.sh rkv 256

Configuration (environment variables)

Variable Default Meaning
VLLM_V1_R_KV_BUDGET 0 (off) KV entries kept per request after compression
VLLM_V1_R_KV_BUFFER 0 (off) compress every N generated tokens
VLLM_V1_R_KV_WINDOW 8 trailing observation window (always retained)
VLLM_V1_R_KV_KERNEL 7 max-pool kernel size for the importance term
VLLM_V1_R_KV_ASYNC 0 (off) allow async scheduling for best throughput (+16.7%); the runner applies its evicted-token count itself so async stays correct
VLLM_V1_R_KV_FREE_BLOCKS 0 (off) opt-in. Free the KV blocks R-KV evicts so the footprint is bounded at budget+buffer (accuracy is bit-identical either way). Off by default because the block manager frees below the runner's cap without a two-way version handshake; enable it for the memory/throughput win under KV pressure.

When BUDGET or BUFFER is 0, every R-KV code path is inert and vLLM behaves exactly as upstream.

Multi-GPU (tensor & data parallelism)

R-KV works with tensor parallelism and data parallelism — add --tensor-parallel-size / --data-parallel-size as usual (or pass them through EXTRA to launch_server.sh). Under TP each rank holds only a shard of the KV heads, so R-KV all-reduces its per-token eviction scores across the TP group, guaranteeing every rank evicts the identical set (TP=2 few-shot GSM8K accuracy matches single-GPU within noise; see benchmark/RESULTS_tp.md). DP replicas are independent, and DP+TP reduces within each replica's TP sub-group. Pipeline parallelism is not yet supported.

Tests

# GPU-free CPU unit tests: algorithm bit-parity + integration/compaction logic.
python -m unittest tests.test_rkv_algo tests.test_rkv_integration

Benchmark

Accuracy + throughput on GSM8K (Qwen2.5-Math-7B-Instruct), regenerated on 8× H100:

  • benchmark/RESULTS.md — budget × buffer sweep vs two Full-KV baselines. budget=512 is lossless, budget=256 near-lossless at buffer ≥ 128; every row reports its physical compaction count.
  • benchmark/RESULTS_tp.md — tensor-parallel scaling (accuracy flat across TP=1/2/4; every rank evicts the identical set).
  • benchmark/RESULTS_dp.md — data-parallel scaling (near-linear, 7.6× at DP=8).
cd benchmark
./prepare_data.sh                                    # verify the bundled dataset
python eval.py --n 200 --label fullkv                 # Full-KV baseline
VLLM_V1_R_KV_BUDGET=256 VLLM_V1_R_KV_BUFFER=128 \
  python eval.py --n 200 --label rkv_b256_buf128      # R-KV

eval.py reports accuracy, offline throughput, and the physical compaction count read back from every worker — so it confirms compression is active (a broken build silently runs Full-KV at a nearly identical accuracy). Set VLLM_V1_R_KV_TRACE=1 to log each [RKV-COMPACT] / [RKV-SKIP] event.

Documentation

0