8000 Align verl OPD documentation with runtime truth by DaoyuanLi2816 · Pull Request #76 · DaoyuanLi2816/mini-verl · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,342 changes: 40 additions & 1,302 deletions PROJECT_STATE.md

Large diffs are not rendered by default.

16 changes: 9 additions & 7 deletions PYPI.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ launch distributed jobs, or claim full algorithmic compatibility.
Install the matching CUDA-enabled PyTorch build for your machine first, then:

```bash
python -m pip install "miniverl[train]"
python -m pip install "miniverl[train,cuda]"
miniverl data sample --format verl-parquet --out prompts.parquet
miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
--config builtin:qwen3-0.6b-1.7b-opd \
Expand Down Expand Up @@ -78,7 +78,8 @@ The actor generates from the current adapter; the teacher scores exactly those
visited token positions; the actor then receives a padded, token-mean update.
The teacher is never treated as a reward model, tool output never becomes a
training label, and a stale actor-policy version cannot enter an on-policy
batch. Memory planning chooses resident, swap, or compatible shared-backbone
batch. Memory planning chooses resident phased roles for quantized models,
swap only for movable unquantized roles, or compatible shared-backbone
placement while keeping actor, teacher and reference identities distinct.

Each phase writes evidence before the next boundary: structured trajectories
Expand Down Expand Up @@ -131,7 +132,8 @@ narrow:
- one trainable actor and one teacher;
- one generated response per prompt (`n=1`);
- reward-free generalized knowledge distillation;
- `forward_kl_topk` with top-k plus tail targets and token-mean aggregation;
- `forward_kl_topk` with teacher top-k IDs/log-probabilities, mass/overlap
diagnostics and token-mean aggregation;
- LoRA or QLoRA adapter updates on one CUDA GPU;
- immutable model revisions and verl-style structured prompt Parquet.

Expand Down Expand Up @@ -164,7 +166,7 @@ fit depends on VRAM, context length, quantization and installed kernels.
| Situation | Recommended starting point | What remains constant |
| --- | --- | --- |
| inspect on CPU or a laptop | `plan` and `run --dry-run` | full config classification |
| one CUDA GPU with limited VRAM | QLoRA plus role swapping | logical batch and loss semantics |
| one CUDA GPU with limited VRAM | QLoRA with resident local phases, or unquantized LoRA with swap | logical batch and loss semantics |
| same-base actor and teacher adapters | shared-backbone mode | explicit role provenance |
| more VRAM available | larger physical phase batches | source data and optimizer intent |

Expand Down Expand Up @@ -196,12 +198,12 @@ miniverl bridge materialize scaleout --download --offline
miniverl bridge doctor scaleout --json
```

The v0.8.1 export preserves student/teacher identities, Parquet bytes and pure
The v0.9 export preserves student/teacher identities, Parquet bytes and pure
OPD overrides, but reports `launchable: false` until exact base snapshots are
materialized and validated against the installed pinned verl commit. Only then
does `bridge materialize` publish a checksummed `launch.sh`; distributed
execution remains untested. Review the [materialization contract](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/scaleout-materialization.md),
[bridge contract](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/verl-bridge.md) and [compatibility policy](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/compatibility.md).
execution remains untested. Review the [current scale-out contract](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/verl-opd-scaleout.md),
[legacy bridge](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/legacy-verl-bridge.md) and [compatibility policy](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/compatibility.md).

The intended operating loop is **plan → inspect → run → inspect → export**.
`plan --out` byte-binds the YAML, ordered overrides and scanned Parquet inputs
Expand Down
16 changes: 9 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ launch distributed jobs, or claim full algorithmic compatibility.
Install the matching CUDA-enabled PyTorch build for your machine first, then:

```bash
python -m pip install "miniverl[train]"
python -m pip install "miniverl[train,cuda]"
miniverl data sample --format verl-parquet --out prompts.parquet
miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
--config builtin:qwen3-0.6b-1.7b-opd \
Expand Down Expand Up @@ -78,7 +78,8 @@ The actor generates from the current adapter; the teacher scores exactly those
visited token positions; the actor then receives a padded, token-mean update.
The teacher is never treated as a reward model, tool output never becomes a
training label, and a stale actor-policy version cannot enter an on-policy
batch. Memory planning chooses resident, swap, or compatible shared-backbone
batch. Memory planning chooses resident phased roles for quantized models,
swap only for movable unquantized roles, or compatible shared-backbone
placement while keeping actor, teacher and reference identities distinct.

Each phase writes evidence before the next boundary: structured trajectories
Expand Down Expand Up @@ -131,7 +132,8 @@ narrow:
- one trainable actor and one teacher;
- one generated response per prompt (`n=1`);
- reward-free generalized knowledge distillation;
- `forward_kl_topk` with top-k plus tail targets and token-mean aggregation;
- `forward_kl_topk` with teacher top-k IDs/log-probabilities, mass/overlap
diagnostics and token-mean aggregation;
- LoRA or QLoRA adapter updates on one CUDA GPU;
- immutable model revisions and verl-style structured prompt Parquet.

Expand Down Expand Up @@ -164,7 +166,7 @@ fit depends on VRAM, context length, quantization and installed kernels.
| Situation | Recommended starting point | What remains constant |
| --- | --- | --- |
| inspect on CPU or a laptop | `plan` and `run --dry-run` | full config classification |
| one CUDA GPU with limited VRAM | QLoRA plus role swapping | logical batch and loss semantics |
| one CUDA GPU with limited VRAM | QLoRA with resident local phases, or unquantized LoRA with swap | logical batch and loss semantics |
| same-base actor and teacher adapters | shared-backbone mode | explicit role provenance |
| more VRAM available | larger physical phase batches | source data and optimizer intent |

Expand Down Expand Up @@ -196,12 +198,12 @@ miniverl bridge materialize scaleout --download --offline
miniverl bridge doctor scaleout --json
```

The v0.8.1 export preserves student/teacher identities, Parquet bytes and pure
The v0.9 export preserves student/teacher identities, Parquet bytes and pure
OPD overrides, but reports `launchable: false` until exact base snapshots are
materialized and validated against the installed pinned verl commit. Only then
does `bridge materialize` publish a checksummed `launch.sh`; distributed
execution remains untested. Review the [materialization contract](docs/scaleout-materialization.md),
[bridge contract](docs/verl-bridge.md) and [compatibility policy](docs/compatibility.md).
execution remains untested. Review the [current scale-out contract](docs/verl-opd-scaleout.md),
[legacy bridge](docs/legacy-verl-bridge.md) and [compatibility policy](docs/compatibility.md).

The intended operating loop is **plan → inspect → run → inspect → export**.
`plan --out` byte-binds the YAML, ordered overrides and scanned Parquet inputs
Expand Down
10 changes: 5 additions & 5 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ PyPI `v0.9.0` 是稳定版;`main` 是开发版。miniVERL 是独立项目,
请先安装与本机 CUDA 匹配的 PyTorch wheel,然后运行:

```bash
python -m pip install "miniverl[train]"
python -m pip install "miniverl[train,cuda]"
miniverl data sample --format verl-parquet --out prompts.parquet
miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
--config builtin:qwen3-0.6b-1.7b-opd \
Expand Down Expand Up @@ -110,7 +110,7 @@ config、data、role 与错误映射见[面向 verl 用户的指南](docs/for-ve
- 一个可训练 actor 与一个 teacher;
- 每个 prompt 只生成一个 response(`n=1`);
- 无 reward 的 generalized knowledge distillation;
- `forward_kl_topk`、top-k + tail target 与 token-mean 聚合;
- `forward_kl_topk` 使用 teacher top-k token ID/log-probability、top-k mass/overlap 诊断与 token-mean 聚合;
- 一张 CUDA GPU 上的 LoRA/QLoRA adapter 更新;
- 不可变模型 revision 与 verl 风格结构化 prompt Parquet。

Expand Down Expand Up @@ -138,7 +138,7 @@ rollout/scoring/update 循环;它不是第二套完整实测 recipe。
| 情况 | 建议起点 | 保持不变的内容 |
| --- | --- | --- |
| CPU 或笔记本上先检查 | `plan` 与 `run --dry-run` | 完整配置分类 |
| 单张小显存 CUDA GPU | QLoRA 与角色 swap | 逻辑 batch 与 loss 语义 |
| 单张小显存 CUDA GPU | QLoRA 使用 resident 本地分阶段;或 unquantized LoRA 使用 swap | 逻辑 batch 与 loss 语义 |
| 同 base 的 actor/teacher adapter | shared-backbone | 明确角色 provenance |
| 显存更充足 | 增大各阶段 physical batch | 数据与 optimizer 意图 |

Expand Down Expand Up @@ -166,10 +166,10 @@ miniverl bridge materialize scaleout --download --offline
miniverl bridge doctor scaleout --json
```

v0.8.1 export 保留 student/teacher 身份、Parquet 原始字节与纯 OPD override;在精确 base
v0.9 export 保留 student/teacher 身份、Parquet 原始字节与纯 OPD override;在精确 base
snapshot 被物化并通过已安装的固定 verl commit 验证前,仍报告 `launchable: false`。只有此后
`bridge materialize` 才会发布带校验和的 `launch.sh`;分布式执行仍是未测试。详见
[物化契约](docs/scaleout-materialization.md)、[桥接契约](docs/verl-bridge.md)与
[物化契约](docs/scaleout-materialization.md)、[当前 scale-out 契约](docs/verl-opd-scaleout.md)与
[兼容性政策](docs/compatibility.md)。

建议操作闭环是 **plan → inspect → run → inspect → export**。`plan --out` 把 YAML、按顺序
Expand Down
35 changes: 6 additions & 29 deletions TODO.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,10 @@
# TODO

This is the post-v0.6 roadmap. Completed RecoveryBench, Consumer Runtime,
Alignment Lab and pinned verl-bridge work lives in `PROJECT_STATE.md` and
`CHANGELOG.md`, not in this active list.
GitHub issues are the active roadmap:

## Scientific follow-up
- [#39 — external alignment benchmark follow-up](https://github.com/DaoyuanLi2816/mini-verl/issues/39)
- [#64 — single-GPU verl-style OPD roadmap](https://github.com/DaoyuanLi2816/mini-verl/issues/64)

- [ ] [External alignment issue #39](https://github.com/DaoyuanLi2816/mini-verl/issues/39):
v0.7.0 preregistered and executed the starting-checkpoint gate, which
failed before teacher/method training. Method-level coverage, evaluator
qualification and the reserved final test remain unexecuted; keep the
issue open without treating a follow-up study as committed work.
- [ ] Evaluate a less saturated task family with at least three prespecified
seeds, eval-only selection and a reserved one-read test split.
- [ ] Extend matched-budget JSON-navigation and SQLite evidence only when the
design adds information beyond the published RecoveryBench study.
- [ ] Repeat the measured GPU recipes on Linux; do not treat the expected
throughput improvement as measured until those artifacts exist.

## Runtime scope

- [ ] Cross-tokenizer distillation with an explicit alignment contract.
- [ ] Engine-backed decoding beyond the current padded local-HF prompt batches;
tool-environment multi-turn generation remains deliberately sequential.
- [ ] Additional tested model families beyond Qwen2/Qwen3.
- [ ] Entropy-aware divergence mixing after a prespecified experiment; current
code records teacher entropy but does not implement the method.
- [ ] A maintained GPU CI runner. GPU tests remain opt-in until real CUDA
infrastructure is configured.

Multi-GPU training, Ray, FSDP, DeepSpeed, vLLM and PPO/GRPO remain intentionally
out of scope; use verl or another distributed training system for those needs.
Completed work and evidence are indexed in [PROJECT_STATE.md](PROJECT_STATE.md)
and [CHANGELOG.md](CHANGELOG.md). Multi-GPU training, Ray, FSDP, DeepSpeed,
vLLM and PPO/GRPO remain outside miniVERL's local runtime.
10 changes: 7 additions & 3 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ artifact. Release notes identify migrations, and the current reader either
loads an older artifact with its historical semantics or rejects it with a
regeneration hint.

## Stable surfaces through v0.6
## Stable surfaces through v0.9

- Package name `miniverl`, command name `miniverl`, documented command names,
and the small Python entry points `RunConfig` and `OPDTrainer`.
Expand Down Expand Up @@ -34,7 +34,10 @@ labelled as a reinterpretation of upstream resource fields.

The [machine-readable field matrix](generated/verl-opd-v0.8-compatibility.json)
is regenerated directly from the typed compiler and its pinned resolved
fixture; CI compares the committed bytes with the generator output.
fixture; CI compares the committed bytes with the generator output. The
[field-effect record](generated/verl-opd-v0.8-field-effects.json) independently
mutates every executable non-informational field and records its observed plan
or native-runtime effect.

The older `single-gpu-online-distillation-v1` import/export contract remains
available for migration and retains its historical non-launchable PPO/reward
Expand All @@ -53,7 +56,8 @@ PPO-reference-cache conversion are unsupported. The release smoke validates
artifacts and configuration; distributed execution is recorded as not tested.
Current bundles contain a fail-closed reward scaffold and an absent base
snapshot, so they use `launch.template.sh` and report `launchable: false`.
See the [bridge contract](verl-bridge.md).
See the [current local runtime](verl-opd-runtime.md), [current scale-out
contract](verl-opd-scaleout.md), and [legacy bridge](legacy-verl-bridge.md).

## Versioned but extensible

Expand Down
15 changes: 8 additions & 7 deletions docs/for-verl-users.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
```

Planning is weight-free and offline. Use `--json` to retain the complete field
matrix. The development CLI records repeated `--set`, override files and
matrix. The stable v0.9 CLI records repeated `--set`, override files and
trailing tokens with deterministic precedence; it does not execute `${...}`
interpolations or shell text. See [Config overrides](config-overrides.md).

Expand Down Expand Up @@ -89,11 +89,12 @@ bytes and schema are copied into export provenance.

## Actor, teacher and reference roles

The actor is the trainable PEFT policy. The teacher produces top-k plus tail
targets on actor-generated tokens. Pure GKD in this profile has no reference
The actor is the trainable PEFT policy. The teacher produces top-k token IDs
and log-probabilities on actor-generated tokens; top-k mass and overlap are
diagnostics, not an explicit tail bucket. Pure GKD in this profile has no reference
policy, critic, task reward or policy-gradient term. Local runtime strategies
may keep roles resident, swap them, or share a compatible backbone, but role
identities never collapse in provenance.
may keep quantized roles resident, swap movable unquantized roles, or share a
compatible backbone, but role identities never collapse in provenance.

## Export boundary

Expand All @@ -103,8 +104,8 @@ miniverl bridge materialize scaleout --download --offline
miniverl bridge doctor scaleout --json
```

The v0.8.1 bundle carries PEFT, Parquet, config and provenance artifacts, but is
reported as `launchable: false` until exact base snapshots are materialized and
The stable v0.9 bundle carries PEFT, Parquet, config and provenance artifacts,
but is reported as `launchable: false` until exact base snapshots are materialized and
the pinned upstream checks pass. Read the [materialization workflow](scaleout-materialization.md).
Upstream parse/load smoke, artifact completeness, launchability and distributed
execution are separate statuses. miniVERL never reports a distributed job as
Expand Down
Loading
0