8000 Qualify Rollout Runtime v2 for v0.11.0 by DaoyuanLi2816 · Pull Request #98 · DaoyuanLi2816/mini-verl · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,16 @@ All notable changes to miniVERL are recorded here. The format follows
- Published the exact-wheel 24-cell RTX 4080 comparison for `hf_reference`,
`hf_cached` and managed vLLM across prompt lengths 128/512, response bounds
64/256/512, `n=1/4`, greedy and seeded sampling.
- Selected vLLM 0.28.0 for the direct-GKD development path after a measured
1.65–4.55× advantage over `hf_cached` at 256/512 tokens, eight confirmed
policy refreshes, 11,820 MiB peak total GPU memory and clean teardown.
- Recorded the blocking negative result: `hf_cached` did not reach the
preregistered 2× speedup over `hf_reference`. v0.11.0 remains unpublished
until the local backend is optimized and the frozen workload is rerun.
- Added an owned BF16 rollout mirror for NF4 training actors, exact live-LoRA
synchronization and StaticCache compilation. The final exact-wheel candidate
passed the `hf_cached >= 2× hf_reference` gate in every required cell.
- Selected CUDA Graph vLLM 0.28.0 for the direct-GKD development path after a
measured 3.08–5.97× advantage over `hf_cached` at 256/512 tokens, eight
confirmed policy refreshes, 11,931 MiB peak total GPU memory and clean
teardown.
- Preserved the first failed candidate as negative evidence and published the
qualified rerun as a separate identity-bound aggregate. PG-k1 remains on
`hf_cached` because vLLM did not pass log-probability conformance.

### Documentation narrative

Expand Down
28 changes: 14 additions & 14 deletions PYPI.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,20 +140,20 @@ rewards and explicit group advantage composition; it has no task-quality claim.

## Measured systems evidence

Rollout Runtime v2 now has a complete 24-cell RTX 4080 measurement across
64/256/512-token responses, `n=1/4`, greedy and seeded sampling. Managed vLLM
0.28.0 delivered 108.5–117.7 output tokens/s and was 1.65–4.55× faster than
`hf_cached` in the 256/512-token cells, with 11.54 GiB peak total GPU memory,
eight confirmed policy refreshes and clean teardown. It is the measured
direct-GKD engine; PG-k1 stays on `hf_cached` because the engine log-probability
probe exceeded its numerical threshold.

The same evidence found the next optimization target: `hf_cached` reached
about 1.8–1.9× on most seeded 256/512-token cells but did not meet the
preregistered 2× gate, while the greedy path was usually slower than the legacy
oracle. The development line therefore remains `0.11.0.dev0` rather than a
published v0.11.0 release. See the [full runtime result](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/benchmarks/rollout-runtime-v2.md)
for every cell and artifact hash.
Rollout Runtime v2 has a complete 24-cell RTX 4080 measurement across
64/256/512-token responses, `n=1/4`, greedy and seeded sampling. The compiled
`hf_cached` path reached 124.2–248.7 output tokens/s and passed the
preregistered 2× gate in every required cell. It keeps the NF4 actor for
training and synchronizes each policy version into an owned BF16 rollout
mirror.

Managed vLLM 0.28.0 with CUDA Graph execution reached 626.6–836.4 output
tokens/s and was 3.08–5.97× faster than `hf_cached` in the 256/512-token cells.
Both backends passed memory, eight-refresh and teardown gates. vLLM is the
measured direct-GKD engine; PG-k1 stays on `hf_cached` because its engine
log-probability probe exceeded the numerical threshold. See the
[full runtime result](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/benchmarks/rollout-runtime-v2.md) for every cell,
the preserved first candidate and artifact hashes.

The Qwen3-0.6B/1.7B developer workload consumed **32 distinct prompts**, used a
64-token response bound, and completed **8 current-policy updates** at
Expand Down
28 changes: 14 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,20 +140,20 @@ rewards and explicit group advantage composition; it has no task-quality claim.

## Measured systems evidence

Rollout Runtime v2 now has a complete 24-cell RTX 4080 measurement across
64/256/512-token responses, `n=1/4`, greedy and seeded sampling. Managed vLLM
0.28.0 delivered 108.5–117.7 output tokens/s and was 1.65–4.55× faster than
`hf_cached` in the 256/512-token cells, with 11.54 GiB peak total GPU memory,
eight confirmed policy refreshes and clean teardown. It is the measured
direct-GKD engine; PG-k1 stays on `hf_cached` because the engine log-probability
probe exceeded its numerical threshold.

The same evidence found the next optimization target: `hf_cached` reached
about 1.8–1.9× on most seeded 256/512-token cells but did not meet the
preregistered 2× gate, while the greedy path was usually slower than the legacy
oracle. The development line therefore remains `0.11.0.dev0` rather than a
published v0.11.0 release. See the [full runtime result](docs/benchmarks/rollout-runtime-v2.md)
for every cell and artifact hash.
Rollout Runtime v2 has a complete 24-cell RTX 4080 measurement across
64/256/512-token responses, `n=1/4`, greedy and seeded sampling. The compiled
`hf_cached` path reached 124.2–248.7 output tokens/s and passed the
preregistered 2× gate in every required cell. It keeps the NF4 actor for
training and synchronizes each policy version into an owned BF16 rollout
mirror.

Managed vLLM 0.28.0 with CUDA Graph execution reached 626.6–836.4 output
tokens/s and was 3.08–5.97× faster than `hf_cached` in the 256/512-token cells.
Both backends passed memory, eight-refresh and teardown gates. vLLM is the
measured direct-GKD engine; PG-k1 stays on `hf_cached` because its engine
log-probability probe exceeded the numerical threshold. See the
[full runtime result](docs/benchmarks/rollout-runtime-v2.md) for every cell,
the preserved first candidate and artifact hashes.

The Qwen3-0.6B/1.7B developer workload consumed **32 distinct prompts**, used a
64-token response bound, and completed **8 current-policy updates** at
Expand Down
18 changes: 9 additions & 9 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,15 +130,15 @@ group advantage 组合;目前不对任务质量作结论。
## 实测系统证据

Rollout Runtime v2 已在 RTX 4080 上完成 24-cell 测量,覆盖 64/256/512-token
response、`n=1/4`、greedy 与 seeded sampling。Managed vLLM 0.28.0 达到
108.5–117.7 output tokens/s;在 256/512-token cell 中比 `hf_cached` 快
1.65–4.55×,peak total GPU memory 为 11.54 GiB,并通过 8 次 policy refresh 与
完整 teardown。它是实测的 direct-GKD engine;engine log-probability probe 超出数值
门槛,因此 PG-k1 继续使用 `hf_cached`。

同一份证据也给出了下一步优化目标:`hf_cached` 在多数 seeded 256/512-token cell 中
约为旧路径的 1.8–1.9×,没有达到预注册的 2× gate;greedy 路径通常比旧 oracle 更慢。
因此开发线保持 `0.11.0.dev0`,尚未发布 v0.11.0。每个 cell 与 artifact hash 见
response、`n=1/4`、greedy 与 seeded sampling。编译后的 `hf_cached` 达到
124.2–248.7 output tokens/s,并在所有必需 cell 中通过预注册的 2× gate。训练仍以
NF4 actor 为准,每个 policy version 都会精确同步到独立的 BF16 rollout mirror。

启用 CUDA Graph 的 managed vLLM 0.28.0 达到 626.6–836.4 output tokens/s;在
256/512-token cell 中比 `hf_cached` 快 3.08–5.97×。两个 backend 都通过显存、8 次
policy refresh 与 teardown gate。vLLM 是实测的 direct-GKD engine;其 engine
log-probability probe 超出数值门槛,因此 PG-k1 继续使用 `hf_cached`。每个 cell、保留的
第一版失败候选与 artifact hash 见
[完整 runtime 结果](docs/benchmarks/rollout-runtime-v2.md)。

Qwen3-0.6B/1.7B developer workload 消费 **32 个不同 prompt**,response 上限为 64 token,
Expand Down
1 change: 1 addition & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ Listed so you can check that this repository does not.

| File | Hardware | What it shows |
| --- | --- | --- |
| `results/rollout-runtime-v2-v0.11.0-candidate-rtx4080.json` | WSL2 RTX 4080 16 GB | Exact-wheel qualified candidate: `hf_cached` passed the 2× local gate in every required cell; CUDA Graph vLLM passed the direct-GKD value gate. Both passed refresh, memory and teardown checks. [Result, method and hashes.](../docs/benchmarks/rollout-runtime-v2.md) |
| `results/rollout-runtime-v2-rtx4080.json` | WSL2 RTX 4080 16 GB | Exact-wheel 24-cell `hf_reference` / `hf_cached` / vLLM comparison. vLLM passed the direct-GKD engine gates; `hf_cached` missed the preregistered 2× speed gate, so v0.11.0 remains blocked. [Result, method and hashes.](../docs/benchmarks/rollout-runtime-v2.md) |
| `results/rollout-runtime-v2-hf-reference.json` | WSL2 RTX 4080 16 GB | Pre-v0.11 actor-rollout systems baseline: all 24 preregistered prompt/response/n/sampling cells completed. It freezes throughput and memory for the future `hf_cached` comparison; teacher scoring, updates, full-cycle timing and typed grouped semantics are not measured here. [Method, table and boundaries.](../docs/benchmarks/rollout-runtime-v2.md) |
| `results/alignment-external-v1.json` | RTX 4080 16 GB | **Preregistered early stop, not a method comparison.** Two declared lineages and eight candidates all measured 0/64 retained JSONNav utility against an unchanged 20% floor. No checkpoint, teacher, continuation arm or final-test result exists. [Study, correction provenance and evaluator boundary.](../docs/alignment-external/alignment-external-v1.md) |
Expand Down
Loading
0