8000 Productize local PPO and GRPO workflows for v0.14.0 by DaoyuanLi2816 · Pull Request #118 · DaoyuanLi2816/mini-verl · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations 8000
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion .github/workflows/gpu.yml
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,13 @@ jobs:
--output "$full_dir/v013-ppo.json" \
--work "$full_work/v013-ppo" \
"${offline[@]}"
"$QUALIFICATION_PYTHON" scripts/run_v014_product_qualification.py \
--commit "$GITHUB_SHA" \
--wheel "$QUALIFICATION_WHEEL" \
--wheel-sha256 "$QUALIFICATION_WHEEL_SHA256" \
--output "$full_dir/v014-product.json" \
--work "$full_work/v014-product" \
"${offline[@]}"
test -z "$(git status --porcelain)"
"$QUALIFICATION_PYTHON" scripts/benchmark_rollout_runtime_v2_selected.py \
--backend hf_cached \ 10BC8
Expand All @@ -223,7 +230,8 @@ jobs:
--vllm-runtime "$full_dir/vllm-runtime.json" \
--hf-reference benchmarks/results/rollout-runtime-v2-hf-reference.json \
--v012-rl "$full_dir/v012-rl.json" \
--v013-ppo "$full_dir/v013-ppo.json"
--v013-ppo "$full_dir/v013-ppo.json" \
--v014-product "$full_dir/v014-product.json"
- name: Upload full qualification records
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: ${{ success() && inputs.qualification_level == 'full' }}
Expand Down
11 changes: 8 additions & 3 deletions .github/workflows/verl-bridge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
changed=$(git diff --name-only "origin/$BASE_REF"...HEAD)
echo "changed files:"
echo "$changed"
if echo "$changed" | grep -Eq '^(\.github/workflows/verl-bridge\.yml|pyproject\.toml|scripts/[^/]*verl_bridge[^/]*|src/miniverl/(algorithms|bridge)/.*|src/miniverl/losses/verl_(topk|pg)\.py|tests/.*verl_(bridge|opd|pg).*|tests/conformance/test_verl_v0(8_(loss|pg_k1)|9_rl)\.py)$'; then
if echo "$changed" | grep -Eq '^(\.github/workflows/verl-bridge\.yml|pyproject\.toml|scripts/[^/]*verl_bridge[^/]*|scripts/publish_(verl_compatibility_corpus|rl_product_examples)\.py|benchmarks/compatibility/.*|src/miniverl/(algorithms|bridge|resources)/.*|src/miniverl/losses/verl_(topk|pg)\.py|tests/.*verl_(bridge|opd|pg|rl|compatibility).*|tests/conformance/test_verl_v0(8_(loss|pg_k1)|9_rl)\.py)$'; then
echo "bridge=true" >> "$GITHUB_OUTPUT"
else
echo "bridge=false" >> "$GITHUB_OUTPUT"
Expand All @@ -68,7 +68,7 @@ jobs:
- name: Install miniVERL bridge environment
run: >-
python -m pip install --upgrade pip &&
python -m pip install ".[bridge,train]" "hydra-core>=1.3,<2" pytest
python -m pip install ".[bridge,train]" "hydra-core==1.3.2" pytest
- name: Install the exact official verl source without its distributed stack
run: >-
python -m pip install --no-deps
Expand All @@ -85,7 +85,12 @@ jobs:
env:
MINIVERL_VERL_V09_SOURCE: _verl-v0.9.0
run: python -m pytest -q tests/conformance/test_verl_v09_rl.py -m verl_conformance
- name: Materialize and recheck a tiny launchable pure-OPD bundle
- name: Regenerate the real upstream compatibility corpus
run: >-
python scripts/publish_verl_compatibility_corpus.py --upstream _verl-v0.9.0 --check
- name: Verify packaged PPO and GRPO workflows
run: python scripts/publish_rl_product_examples.py --check
- name: Materialize and recheck the pure-OPD bundle
run: >-
python -m pytest -q tests/conformance/test_verl_v08_materialize.py
-m verl_conformance
Expand Down
30 changes: 29 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
10BC8 Expand Up @@ -6,6 +6,34 @@ All notable changes to miniVERL are recorded here. The format follows

## [Unreleased]

## [0.14.0] - 2026-09-08

### Installed local RL workflows

- Package upstream-derived PPO and GRPO examples with exact source/adaptation
ledgers, import reports, native recipes and a complete inspect/resume/handoff
walkthrough. No checkout is required by their user commands.
- Add run-directory inspection and RL reports with actor/critic counters,
rewards, advantages, losses, memory, local lowerings and checkpoint history.
- Gate releases on both installed-candidate workflows, exact tensor replay and
checked v0.9 handoff, alongside the existing full RTX 4080 qualification.

### Compatibility and recovery

- Add the immutable v3 compiler profile: upstream prompt minibatches expand by
rollout count; physical offload hints are recorded; unimplemented reductions
and repeated non-PPO epochs reject explicitly. Published v1/v2 rules remain
unchanged.
- Add six upstream-owned scripts and seven Hydra-resolved configuration cases
with CI-gated field accounting, pinned source hashes and precise rejections.
- Bind PPO old values and RL behavior log-probabilities to their source roles;
restore RNG on critic OOM retry and reject nonfinite critic gradients.
- Bind RL checkpoint log prefixes, archive post-checkpoint tails on resume and
reject corrupt/inconsistent inspection artifacts. Add interruption coverage
across rollout, reward, old-value, actor and critic phases.
- Foreground the current local RL workflow in English/Chinese documentation;
retain frozen scientific results and evidence-scoped hardware claims.

## [0.13.0] - 2026-09-08

### PPO and full local RL
Expand Down Expand Up @@ -1206,7 +1234,7 @@ Same-tokenizer only; one trajectory per forward pass; `swap` unavailable for
quantized models; only Qwen3 and Qwen2 architectures tested; single-seed GPU
results. The full list is in `docs/limitations.md`.

[Unreleased]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.13.0...HEAD
[Unreleased]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.14.0...HEAD
[0.13.0]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.12.0...v0.13.0
[0.12.0]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.11.0...v0.12.0
[0.11.0]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.10.1...v0.11.0
Expand Down
48 changes: 13 additions & 35 deletions CITATION.cff
Original file line number Diff line number Diff line change
@@ -1,46 +1,24 @@
cff-version: 1.2.0
title: "miniVERL: A bounded single-GPU runtime for verl-style OPD"
title: "miniVERL: Run verl experiment semantics on one GPU"
message: "If you use miniVERL in your work, please cite it as below."
type: software
version: 0.13.0
version: 0.14.0
date-released: 2026-09-08
license: Apache-2.0
repository-code: "https://github.com/DaoyuanLi2816/mini-verl"
url: "https://github.com/DaoyuanLi2816/mini-verl"
abstract: >-
miniVERL runs a documented subset of verl-style on-policy distillation on one
consumer GPU. It is a local single-GPU runtime for a documented subset of verl v0.8
on-policy distillation. It consumes typed verl-shaped configuration and
Parquet prompts, executes actor rollout, teacher scoring and actor update in
one inspectable process, and exports standard PEFT, Parquet and configuration
artifacts through a fail-closed pinned bridge. Its native core also supports
multi-turn, tool-aware on-policy distillation: a student language
model samples its own tool-using trajectories against deterministic local
environments, a teacher scores exactly the states the student visited, and
the student is updated with token-level distributional supervision on its own
generated tokens only. It implements exact full-vocabulary forward KL,
reverse KL and Jensen-Shannon divergences as well as a compressed top-k plus
tail coarse-graining, enforces per-token provenance so that tool output can
never become a training label, and stores teacher targets in a versioned,
checksummed, pickle-free cache with policy-version enforcement.
Around that core it provides supervised fine-tuning, recorded-provenance
preference optimization and knowledge distillation; shared-backbone
execution that switches student, teacher and reference
roles across adapters on one set of base weights to fit low-memory hardware;
padded update batching; deterministic replay; transactional checkpoints and
cross-process run locks; and mechanism studies such as RecoveryBench that are
published with their negative and mixed results intact. A fail-closed
artifact bridge exchanges configs, LoRA adapters, tokenizers and Parquet
datasets with one pinned verl profile, verifying what it can and refusing to
imply the rest: it does not launch distributed jobs or establish broad
algorithmic parity. miniVERL is designed for one personal CUDA GPU,
automatically selects bf16 or fp16, and requires neither Ray nor a cluster.
Published performance is measured on one RTX 4080; other GPU models use the
same code path but remain unmeasured. The v0.7 external-alignment study
terminated at its preregistered checkpoint-selection gate: two declared
lineages and eight candidates all scored 0/64 retained JSONNav utility, so
no teacher qualification, continuation method comparison or reserved final
test was run.
miniVERL compiles supported verl experiment semantics for one NVIDIA GPU.
Its versioned profiles consume resolved configuration and Parquet prompts,
run PPO with an independent critic or critic-free reward-driven algorithms,
and support on-policy distillation with explicit teachers. Typed trajectories,
role identities, transactional checkpoints and exact replay make local runs
inspectable and resumable. Portable PEFT, critic, Parquet and provenance
bundles support handoff toward pinned upstream verl. Installed PPO and GRPO
walkthroughs exercise the complete workflow. Runtime qualification is measured
on one RTX 4080 and kept separate from task-quality research; distributed
execution is outside the local runtime. Frozen studies preserve their original
negative and mixed results.
authors:
- family-names: Li
given-names: Daoyuan
Expand Down
26 changes: 22 additions & 4 deletions PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,39 @@ current product and evidence state rather than repeating release history.

Last updated: 2026-09-08.

Canonical release state: stable `v0.13.0` (`8e7bc3b5ff1145ae502cca893b7fffca3e12b6a8`), development `0.13.1.dev0`.
Canonical release state: releasing `v0.14.0`.

## Release state

- Active work: `v0.14.0-p0-productization`, based on verified main `cdc608f`.
The next release focuses on upstream configuration coverage, packaged PPO/GRPO
walkthroughs, run inspection and system-level recovery. Implementation is
settled; release metadata is finalized for exact-candidate qualification.
- Upstream drift check: GitHub's latest stable release is still `v0.9.0`
(published 2026-08-14); existing compiler targets remain appropriate.
- Implemented: v3 prompt-minibatch units and explicit physical lowerings;
six upstream scripts/seven resolved cases with complete field accounting;
installed PPO/GRPO templates and adaptation ledgers; strict run inspection;
PPO value/behavior identity guards and checkpoint-bound log recovery.
- Development rehearsal: both wheel-installed workflows completed on RTX 4080;
PPO 4 actor/4 critic updates and GRPO 2 actor updates, with exact tensor resume
and inspected handoff. This dirty-tree rehearsal is not release evidence.
- Pending: final exact-commit qualification, publication and state sync.
- Local regression: 2,655 non-GPU/non-network tests passed at 83% branch
coverage; 10 RTX 4080 GPU and 16 network tests passed. Four pinned upstream
conformance tests and four-viewport documentation browser checks passed.

- Stable release: `v0.13.0` at
`8e7bc3b5ff1145ae502cca893b7fffca3e12b6a8`.
- Current development line: `0.13.1.dev0`.
- Release candidate line: `0.14.0`.
- Stable docs: <https://daoyuanli2816.github.io/mini-verl/>.
- Development docs: <https://daoyuanli2816.github.io/mini-verl/dev/>.
- Historical build log: [v0.1-v0.9 archive](docs/history/project-state-v0.1-v0.9.md).

## Current product boundary

Development `0.13.1.dev0` has a closed typed profile compiler: it retains the v1 critic-free contract and adds
`verl-rl-v0.9-single-gpu-v2`, a typed resolved PPO subset of official verl
Release `0.14.0` has a closed typed profile compiler: it retains v1/v2 and adds
`verl-rl-v0.9-single-gpu-v3`, a typed resolved RL subset of official verl
`v0.9.0` at commit `483b8a009ba3a97563edee3a19887e4862b8094a`.
It compiles PPO/GAE into phased one-GPU actor, critic, reference and reward
roles. The critic is independently trainable and owns its optimizer, schedule,
Expand Down
Loading
0