8000 Reframe documentation around product workflows by DaoyuanLi2816 · Pull Request #90 · DaoyuanLi2816/mini-verl · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,18 @@ All notable changes to miniVERL are recorded here. The format follows

## [Unreleased]

### Documentation narrative

- Rebuilt the English, Chinese and PyPI landing narratives around the local
workflow, concrete artifacts, user paths and measured systems evidence.
- Reframed the documentation home, stack-selection guide, verl migration,
hardware planning and OPD guides around supported workflows, with complete
execution and scientific boundaries consolidated in compatibility and
limitations.
- Updated the main runtime diagram and contributor guidance so product pages
lead with capabilities while research, security and operational caveats stay
next to the decisions they qualify.

### Release evidence polish

- Corrected the v1-readiness record after the completed v0.10.1 attempt-1 RTX
Expand Down
25 changes: 17 additions & 8 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,14 @@
# Contributing to miniVERL

Thank you for considering it. This document is short and specific, because the
project has a small number of rules that matter a lot and very few that do not.
Thank you for helping make one-GPU distillation easier to inspect, reproduce
and extend. This guide covers the checks that keep code and published evidence
aligned.

## The one rule
## Measurement provenance

**Never state a number you did not measure.** Not in code, not in a docstring,
not in the README, not in a pull-request description. If something was not run,
write "not run" and leave the exact command that would run it. A missing number
is fine; an invented one is not, and it is the fastest way to make a
distillation project worthless.
**Every reported number needs a measurement artifact or command.** When a check
was unavailable, write "not run" and include the exact command for a future
run. This convention keeps reviews useful across different hardware.

## Setup

Expand Down Expand Up @@ -77,6 +76,16 @@ required hardware or credentials are unavailable.
* Silent behaviour: truncating a sequence, changing a model, reusing a stale
cache, or swallowing an exception. Fail loudly with a hint instead.

## Documentation voice

Lead with what the workflow does, who it helps and which artifact it produces.
State a local constraint where it changes the reader's next action; link to
the [compatibility policy](docs/compatibility.md) or
[limitations](docs/limitations.md) for complete exclusion lists and scientific
boundaries. Avoid repeating a catalogue of absent features after every product
capability. Research reports and security contracts should keep their precise
caveats close to the evidence or decision they qualify.

## Adding an environment

See the "New environment" issue template for the checklist. In short: seeded and
Expand Down
351 changes: 134 additions & 217 deletions PYPI.md

Large diffs are not rendered by default.

351 changes: 134 additions & 217 deletions README.md

Large diffs are not rendered by default.

270 changes: 104 additions & 166 deletions README.zh-CN.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ GitHub issues are the active roadmap:
- [#64 — single-GPU verl-style OPD roadmap](https://github.com/DaoyuanLi2816/mini-verl/issues/64)

Completed work and evidence are indexed in [PROJECT_STATE.md](PROJECT_STATE.md)
and [CHANGELOG.md](CHANGELOG.md). Multi-GPU training, Ray, FSDP, DeepSpeed,
vLLM and PPO/GRPO remain outside miniVERL's local runtime.
and [CHANGELOG.md](CHANGELOG.md). Current execution and research boundaries are
maintained in [docs/limitations.md](docs/limitations.md).
5 changes: 2 additions & 3 deletions docs/alignment-external/benchmark-governance.md
8000
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
# External benchmark governance

This page records where every external alignment endpoint comes from, what
licence it carries, what this project redistributes, and what each measurement
does *not* establish. It is the auditable half of
This page records the source, licence, redistribution policy and interpretation
scope for every external alignment endpoint. It is the auditable half of
[`benchmarks/external-alignment/registry.yaml`](https://github.com/DaoyuanLi2816/mini-verl/blob/main/benchmarks/external-alignment/registry.yaml),
which is the machine-readable half.

Expand Down
7 changes: 3 additions & 4 deletions docs/alignment-lab/demo.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
# Alignment Lab demo recording script

This is a reproducible recording plan, not a fabricated video. The short run
uses the CPU toy backend and proves the workflow and artifact surfaces; its 0%
task score is not capability evidence. The measured v0.5 result remains the
separate three-seed Qwen3 artifact.
This reproducible recording plan uses the CPU toy backend to show the workflow
and artifact surfaces. Its 0% task score belongs to the demo fixture; the
measured v0.5 result is the separate three-seed Qwen3 artifact.

## Install

Expand Down
4 changes: 2 additions & 2 deletions docs/alignment-lab/launch-copy.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# miniVERL v0.5.0 launch copy

All text below is evidence-bounded. Update the release URL only after the exact
tag is public.
This archived launch copy is tied to the v0.5.0 evidence bundle. Update its
release URL only when republishing against the exact tag.

## GitHub Release summary

Expand Down
2 changes: 1 addition & 1 deletion docs/banner.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
161 changes: 71 additions & 90 deletions docs/comparisons.md
Original file line number Diff line number Diff line change
@@ -1,95 +1,76 @@
# How miniVERL compares
# Choose the right distillation stack

This page exists so that you can decide quickly whether miniVERL is the wrong
tool for your problem. For most serious distillation work it is.
miniVERL, verl, TRL GKD, KDFlow and research OPD harnesses serve different
workflows. Start from the execution environment and the artifact contract you
need, then choose the smallest stack that covers them.

Every cell below is either something read out of the relevant source tree, a
value returned by the GitHub or arXiv API, or the literal string `not verified`.
Nothing here is inferred from a project's marketing copy. Repository/API and
official-documentation checks were refreshed on 2026-07-29; upstream projects
move, so re-check the primary sources in [references](references.md) before
relying on a row.
The source snapshot behind this page was checked on 2026-07-29. Upstream
projects evolve, so follow the primary links in [references](references.md)
before making a long-lived infrastructure decision.

## Feature table
## Quick choice

| dimension | miniVERL | verl | TRL GKD | KDFlow | OPSD | plain SFT |
| Your priority | Best starting point |
| --- | --- |
| Read, modify and audit OPD on one NVIDIA GPU | **miniVERL** |
| Scale training across accelerators and nodes | **verl** |
| Add generalized-JSD distillation to a Transformers training workflow | **TRL GKD** |
| Explore cross-tokenizer or multimodal KD | **KDFlow** |
| Reproduce one paper's OPD experiments | the paper's **research harness** |
| Train a fixed supervised dataset | **plain SFT** |

## Capability snapshot

| Dimension | miniVERL | verl | TRL GKD | KDFlow | OPSD | plain SFT |
| --- | --- | --- | --- | --- | --- | --- |
| **Primary scope** | Single-accelerator on-policy distillation of a tool-using agent, plus SFT and offline KD, over three built-in deterministic environments | General RL post-training framework (HybridFlow). PPO/GRPO family, plus a first-class distillation trainer in `verl/trainer/distillation/` | One trainer, `trl.experimental.gkd.GKDTrainer`, doing generalized-JSD distillation on chat datasets | Knowledge-distillation framework: off-policy, on-policy, cross-tokenizer and multimodal, per the repository description | Research harness for on-policy distillation experiments, built on verl | Next-token cross-entropy on a fixed dataset. miniVERL implements this itself as `run.mode: sft`, for use as a baseline arm |
| **Mandatory infrastructure** | None beyond a Python process. `doctor`, `validate`, `inspect`, `report`, `cache`, `schema` and `export-benchmark` run on the base install; torch, transformers and peft arrive only with the `train` extra | Ray. `ray[default]` is an unconditional line in `requirements.txt` | The `transformers` Trainer plus `accelerate`. No cluster runtime | Ray and SGLang, both unconditional in `requirements.txt` | Whatever verl needs, therefore Ray. Setup is a shell script (`scripts/opd/setup_opd.sh`), not a package install | None |
| **Target hardware** | One accelerator, or CPU for the toy backend. Every GPU number in this repository comes from a single RTX 4080 (16 GB). There is no multi-GPU code path | The published device-tuning table starts at 1xH100 and runs through 8xH100, 8xH800 and 32xH800 / 32xH20 | Not stated in the trainer documentation; scales with whatever `accelerate` is configured for | Examples assume 8 GPUs per node | not verified; inherits verl's requirements | Any |
| **On-policy rollouts** | Yes. `run.mode: opd` samples from the current student every cycle, and the config validator forces `cache.strict_policy_version: true`, so a target produced at policy version *v* cannot be consumed at *v+1* | Yes | Configurable, not the default. `lmbda` defaults to `0.5`, so on average half the batches use student-generated sequences | Yes | Yes | No. The trajectory set is fixed before training starts |
| **Multi-turn tool use with environment execution** | Yes. `RolloutRunner` interleaves real tool execution (calculator, JSON navigation, read-only SQLite) with generation, bounded by `max_turns`, `max_new_tokens_per_turn`, `max_total_tokens`, `max_parse_errors` and `max_repeated_calls` | Yes. `verl/experimental/agent_loop/` contains `tool_agent_loop.py` and `tool_parser.py` | No tool environment in the distillation trainers | None found. A code search over the repository for `tool_call` and for `agent` returns 0 results | Yes. The README describes multi-turn agent-loop rollouts with tool and environment tokens excluded from the loss | No |
| **Teacher-target compression / cache** | `bucketed_topk_tail` (teacher top-k log-probs plus one aggregated tail bucket) or `exact_full_vocab`. Bucketed targets persist to a sharded safetensors cache with a SHA-256 per shard and a recorded teacher revision, tokenizer fingerprint, temperature and `top_k` | not verified | `GKDTrainer` recomputes full-vocabulary teacher logits under `no_grad` every step and keeps no cache. Separately, `ServerDistillationTrainer` exposes `loss_top_k` (default `1`) and `loss_add_tail` (default `True`) | not verified | Chunked divergence computation rather than a persistent cache; the README states it processes tokens in chunks instead of materializing full-vocabulary tensors for the whole batch | Not applicable; the targets are the tokens |
| **Packaging and tests** | Published hatchling wheel plus self-testing sdist, `miniverl` console script, 1,000+ tests and 86%+ branch coverage; exact release evidence is in the [quality record](generated/quality.json) | Published on PyPI as `verl`; workflow inventory checked on the date above | Published on PyPI as `trl`; tests in-repo, including `tests/experimental/test_server_distillation_trainer.py` | Has a `pyproject.toml`; package/repository surfaces were checked on the date above | Repository package/test/workflow surfaces were checked on the date above | Not applicable |
| **License** | Apache-2.0 (`LICENSE` in the repository root) | Apache-2.0 | Apache-2.0 | MIT | No `LICENSE` file; the GitHub API reports no license, so the code is all-rights-reserved and cannot be reused | Not applicable |

One more research codebase is worth naming even though it does not get a column:
**thunlp/OPD**, the official code for arXiv:2604.13016, also builds on verl
(v0.7.0) with LlamaFactory for the SFT stage, runs its experiments on 8xA800
80 GB, and likewise ships **no LICENSE file**.

## What is not novel here

Two claims that a reader might expect this project to make, and which would be
false:

- **Top-k plus a tail bucket is not a new idea.** TRL's
`ServerDistillationTrainer` already has `loss_top_k` (default `1`) and
`loss_add_tail` (default `True`), which is the same coarse-graining. miniVERL's
contribution on this axis is bookkeeping, not the objective: the bucket
parameters are recorded in the cache index and re-checked on load, and the
documentation is explicit that the result is a lower bound on the exact
divergence rather than the divergence itself.
- **verl is not missing on-policy distillation, and it is not missing tool
use.** verl has `verl/trainer/distillation/` with FSDP and Megatron backends
and a `DistillationConfig` in `verl.workers.config`, and it has an agent loop
with a tool parser in `verl/experimental/agent_loop/`. Any comparison that
presents miniVERL as filling those gaps is wrong.

## What miniVERL is not

miniVERL is not a scaling framework, and it is not a faster or better
implementation of anything listed above. It has no Ray integration, no FSDP or
Megatron backend, no vLLM or SGLang rollout engine, no tensor or pipeline
parallelism, and no multi-node story of any kind. Rollouts come from a custom
sampling loop that decodes one sequence at a time with a KV cache, projecting a
single position through the LM head per step. The update path supports local
mask-isolated padded batches, but has no continuous rollout batching or remote
workers. `train.gradient_accumulation_steps` is the optimizer-group size and
`train.trajectory_batch_size` is only the physical update-forward size. There is no
PPO, no GRPO and no reward model. The three environments are synthetic and
generated in-process; there is no dataset loader, no containerized or networked
tool sandbox, and no support for vision-language models. Capability results in
this repository come from one calculator task family on one consumer GPU at two
prespecified seeds, which is enough to show the pipeline runs and is not enough to establish
that any objective beats any other.

It is a single-GPU teaching and experimentation lab: a readable implementation
of the exact and bucketed divergences with the orientation written into every
function name, a rollout loop that records token provenance as it generates
rather than reconstructing it afterwards, a teacher cache whose policy version
is enforced rather than assumed, and a CLI that refuses contradictory recipes
before anything is downloaded.

## When you should use verl instead

Use verl, not miniVERL, if any of the following is true:

- You have more than one GPU, or more than one node. verl was built for this;
miniVERL has no code path for it at all.
- Your chosen student/teacher pair and token budget do not fit on one device.
miniVERL's `swap` strategy can help for unquantized pairs, but is unavailable
when either model is quantized.
- You need rollout throughput. verl drives vLLM or SGLang; miniVERL decodes one
sequence at a time, and on the machine used here that is kernel-launch bound
rather than compute bound.
- You want on-policy distillation combined with RL objectives, a reward model,
or an advantage estimator. verl has the PPO and GRPO machinery and a
distillation trainer that plugs into it.
- You need the broader maintainer community and release surface of an upstream
scaling framework.

miniVERL is a reasonable choice when you have exactly one personal NVIDIA GPU, you want
to read and modify the loss, and you care more about being able to audit which
token was supervised by which teacher distribution than about throughput.
Reading `docs/limitations.md` before starting is strongly recommended.
| **Design center** | Inspectable single-GPU OPD, plus local SFT/DPO/KD baselines | General RL and distillation post-training at scale | A distillation trainer inside the Transformers ecosystem | Distributed KD across policy, tokenizer and modality choices | Paper-oriented OPD experiments built on verl | Next-token learning on a fixed dataset |
| **Runtime shape** | One Python process; optional CUDA training stack | Ray with distributed model and rollout backends | Transformers Trainer + Accelerate | Ray + SGLang | verl-based | Framework-dependent |
| **On-policy path** | Strict current-policy rollout → teacher score → actor update | First-class distributed distillation and agent-loop paths | Configurable student-generated sequences | Available | Available | Fixed dataset |
| **Tool trajectories** | Calculator, JSON navigation, read-only SQLite and custom typed environments | Agent loop and tool parser | Chat-dataset training | Project-dependent | Tool-oriented experiments | Dataset-defined |
| **Teacher artifacts** | Exact or top-k targets, sampled-k1 signals and sharded safetensors cache | Distributed trainer state | Teacher forward pass in the trainer; server distillation supports top-k + tail controls | Chunked/distributed target handling | Chunked divergence path | Target tokens |
| **Primary output** | PEFT adapter, trajectories, cache, plan and portable provenance bundle | Distributed checkpoints and rollout/training artifacts | Transformers model or adapter | Project-defined model artifacts | Experiment artifacts | Model or adapter |
| **Best fit** | One-GPU experiments where semantic traceability matters | Throughput, scale and RL integration | Existing Transformers/TRL workflows | Broader KD research space | Reproducing its published setup | A known supervised target dataset |

The table summarizes each project's documented design center rather than
ranking quality or speed. Hardware numbers and algorithm outcomes are meaningful
only within their original model, data and runtime setup.

## miniVERL's design center

miniVERL concentrates on the parts of a local distillation run that benefit
from explicit evidence:

- a typed compiler that records how each source field affects local execution;
- strict policy-version binding between rollouts and teacher targets;
- token-span provenance created during generation;
- exact, top-k and sampled-k1 teacher signals with checksummed cache identity;
- transactional plans, checkpoints and export publication;
- standard PEFT, safetensors and Parquet interchange.

That design is especially useful when you have one personal NVIDIA GPU, want
to inspect or change the loss, and value semantic traceability over rollout
throughput.

## When scale is the main requirement

verl is the natural continuation when the workload needs multi-GPU execution,
high-throughput rollout engines, distributed checkpointing or RL objectives.
miniVERL's export path prepares a pinned bundle of local artifacts for that
workflow and reports artifact completeness, materialization and launchability
as separate states.

Use the [scale-out contract](verl-opd-scaleout.md) for that handoff. The full
list of miniVERL's algorithm, architecture and evidence boundaries lives in
[limitations](limitations.md), while [compatibility](compatibility.md) defines
the exact versioned profile contract.

## Evidence notes

The public miniVERL results cover one-GPU systems behavior and several scoped
task studies. They include negative outcomes and preregistered early stops.
Read the detailed reports before treating a method result as portable to a new
teacher, task, model pair or budget.

The external project names and descriptions above are attributed to their
primary repositories and papers. miniVERL is an independent project; the table
is an interoperability and workflow guide.
Loading
0