8000 Publish RTX 4080 rollout and competitor evidence by DaoyuanLi2816 · Pull Request #96 · DaoyuanLi2816/mini-verl · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,20 @@ All notable changes to miniVERL are recorded here. The format follows
- Kept PG-k1 on `hf_cached`: the BF16 external engine and NF4 actor agreed on
greedy tokens in the qualification probe, but their sampled-token
log-probabilities did not satisfy the preregistered numerical threshold.
- Preserved the initial managed-server startup timing across later adapter
refreshes instead of overwriting it with no-op start checks.

### Measured rollout evidence

- Published the exact-wheel 24-cell RTX 4080 comparison for `hf_reference`,
`hf_cached` and managed vLLM across prompt lengths 128/512, response bounds
64/256/512, `n=1/4`, greedy and seeded sampling.
- Selected vLLM 0.28.0 for the direct-GKD development path after a measured
1.65–4.55× advantage over `hf_cached` at 256/512 tokens, eight confirmed
policy refreshes, 11,820 MiB peak total GPU memory and clean teardown.
- Recorded the blocking negative result: `hf_cached` did not reach the
preregistered 2× speedup over `hf_reference`. v0.11.0 remains unpublished
until the local backend is optimized and the frozen workload is rerun.

### Documentation narrative

Expand Down
15 changes: 15 additions & 0 deletions PYPI.md
8000
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,21 @@ rewards and explicit group advantage composition; it has no task-quality claim.

## Measured systems evidence

Rollout Runtime v2 now has a complete 24-cell RTX 4080 measurement across
64/256/512-token responses, `n=1/4`, greedy and seeded sampling. Managed vLLM
0.28.0 delivered 108.5–117.7 output tokens/s and was 1.65–4.55× faster than
`hf_cached` in the 256/512-token cells, with 11.54 GiB peak total GPU memory,
eight confirmed policy refreshes and clean teardown. It is the measured
direct-GKD engine; PG-k1 stays on `hf_cached` because the engine log-probability
probe exceeded its numerical threshold.

The same evidence found the next optimization target: `hf_cached` reached
about 1.8–1.9× on most seeded 256/512-token cells but did not meet the
preregistered 2× gate, while the greedy path was usually slower than the legacy
oracle. The development line therefore remains `0.11.0.dev0` rather than a
published v0.11.0 release. See the [full runtime result](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/benchmarks/rollout-runtime-v2.md)
for every cell and artifact hash.

The Qwen3-0.6B/1.7B developer workload consumed **32 distinct prompts**, used a
64-token response bound, and completed **8 current-policy updates** at
**3.1914 GiB peak reserved VRAM** on an RTX 4080. Median steady-state rollout,
Expand Down
15 changes: 15 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,21 @@ rewards and explicit group advantage composition; it has no task-quality claim.

## Measured systems evidence

Rollout Runtime v2 now has a complete 24-cell RTX 4080 measurement across
64/256/512-token responses, `n=1/4`, greedy and seeded sampling. Managed vLLM
0.28.0 delivered 108.5–117.7 output tokens/s and was 1.65–4.55× faster than
`hf_cached` in the 256/512-token cells, with 11.54 GiB peak total GPU memory,
eight confirmed policy refreshes and clean teardown. It is the measured
direct-GKD engine; PG-k1 stays on `hf_cached` because the engine log-probability
probe exceeded its numerical threshold.

The same evidence found the next optimization target: `hf_cached` reached
about 1.8–1.9× on most seeded 256/512-token cells but did not meet the
preregistered 2× gate, while the greedy path was usually slower than the legacy
oracle. The development line therefore remains `0.11.0.dev0` rather than a
published v0.11.0 release. See the [full runtime result](docs/benchmarks/rollout-runtime-v2.md)
for every cell and artifact hash.

The Qwen3-0.6B/1.7B developer workload consumed **32 distinct prompts**, used a
64-token response bound, and completed **8 current-policy updates** at
**3.1914 GiB peak reserved VRAM** on an RTX 4080. Median steady-state rollout,
Expand Down
12 changes: 12 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,18 @@ group advantage 组合;目前不对任务质量作结论。

## 实测系统证据

Rollout Runtime v2 已在 RTX 4080 上完成 24-cell 测量,覆盖 64/256/512-token
response、`n=1/4`、greedy 与 seeded sampling。Managed vLLM 0.28.0 达到
108.5–117.7 output tokens/s;在 256/512-token cell 中比 `hf_cached` 快
1.65–4.55×,peak total GPU memory 为 11.54 GiB,并通过 8 次 policy refresh 与
完整 teardown。它是实测的 direct-GKD engine;engine log-probability probe 超出数值
门槛,因此 PG-k1 继续使用 `hf_cached`。

同一份证据也给出了下一步优化目标:`hf_cached` 在多数 seeded 256/512-token cell 中
约为旧路径的 1.8–1.9×,没有达到预注册的 2× gate;greedy 路径通常比旧 oracle 更慢。
因此开发线保持 `0.11.0.dev0`,尚未发布 v0.11.0。每个 cell 与 artifact hash 见
[完整 runtime 结果](docs/benchmarks/rollout-runtime-v2.md)。

Qwen3-0.6B/1.7B developer workload 消费 **32 个不同 prompt**,response 上限为 64 token,
在 RTX 4080 上完成 **8 次 current-policy update**,**peak reserved VRAM 为 3.1914 GiB**。
稳态 rollout、teacher scoring 与 update 的中位耗时分别为 9.7200、0.4864 与 2.3260 秒。
Expand Down
1 change: 1 addition & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ Listed so you can check that this repository does not.

| File | Hardware | What it shows |
| --- | --- | --- |
| `results/rollout-runtime-v2-rtx4080.json` | WSL2 RTX 4080 16 GB | Exact-wheel 24-cell `hf_reference` / `hf_cached` / vLLM comparison. vLLM passed the direct-GKD engine gates; `hf_cached` missed the preregistered 2× speed gate, so v0.11.0 remains blocked. [Result, method and hashes.](../docs/benchmarks/rollout-runtime-v2.md) |
| `results/rollout-runtime-v2-hf-reference.json` | WSL2 RTX 4080 16 GB | Pre-v0.11 actor-rollout systems baseline: all 24 preregistered prompt/response/n/sampling cells completed. It freezes throughput and memory for the future `hf_cached` comparison; teacher scoring, updates, full-cycle timing and typed grouped semantics are not measured here. [Method, table and boundaries.](../docs/benchmarks/rollout-runtime-v2.md) |
| `results/alignment-external-v1.json` | RTX 4080 16 GB | **Preregistered early stop, not a method comparison.** Two declared lineages and eight candidates all measured 0/64 retained JSONNav utility against an unchanged 20% floor. No checkpoint, teacher, continuation arm or final-test result exists. [Study, correction provenance and evaluator boundary.](../docs/alignment-external/alignment-external-v1.md) |
| `results/alignment-lab-v1.json` | RTX 4080 16 GB | Alignment Lab v1: six methods from one Qwen3-0.6B SFT checkpoint, three seeds and 48 paired policy tasks per arm. The SFT start saturated alignment and utility; no continuation method improved it, so the pilot recommends no online teacher querying. [Report, figures and caveats.](../docs/alignment-lab/alignment-lab-v1.md) |
Expand Down
Loading
0