8000 Add verl-style override workflow by DaoyuanLi2816 · Pull Request #68 · DaoyuanLi2816/mini-verl · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ jobs:
run: |
python scripts/publish_alignment_lab_artifacts.py --check
python scripts/publish_verl_bridge_diagrams.py --check
python scripts/publish_verl_opd_compatibility.py --check
- name: Build complete development site strictly
run: mkdocs build --strict
- name: Install deterministic Chromium
Expand Down
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,22 @@ All notable changes to miniVERL are recorded here. The format follows

## [Unreleased]

### Verl config UX

- Added safe trailing Hydra-style overrides and repeatable plain/JSON
`--overrides-file` inputs with deterministic base → files → `--set` →
trailing precedence. The compiled report preserves every duplicate, source,
prior value and final effective value without evaluating shell text or
interpolation.
- Added value-bound approval metadata for the packaged profile. External
configs, or packaged profiles whose high-risk values drift, require explicit
`--accept-local-reinterpretations` before `run`; unsupported algorithm and
distributed fields still fail closed.
- Classified all 82 leaves in an Apache-2.0-provenanced fixture derived from
the pinned official verl v0.8.0 Qwen3 OPD example. Harmless logging and
compile hints are informational, while policy-gradient and enabled FSDP
offload semantics remain unsupported.

### Fixed

- Made the PyPI release verifier require the durable product, project and
Expand Down
10 changes: 10 additions & 0 deletions PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,16 @@ path easier to configure, inspect, reproduce and materialize. The v0.8.1
release remains the stable product surface; no frozen research result is being
reinterpreted as v0.9 evidence.

The first v0.9 slice accepts safe trailing dotted overrides, repeatable plain
files and JSON argv arrays with base → files → `--set` → trailing precedence.
Every duplicate retains its source, prior value and final effective value.
External configs require explicit acceptance of the displayed high-risk local
reinterpretations; the packaged Qwen profile's approval is value-bound and
fails closed if one of those values drifts. A generated report classifies all
82 leaves in the pinned official-example field fixture: 23 exact, 16
semantically conformant, 22 locally reinterpreted, 18 informational and 3
unsupported. No algorithm or distributed boundary widened.

## v0.8.1 product surface

The landing pages now lead with the documented one-GPU verl-style OPD journey,
Expand Down
7 changes: 6 additions & 1 deletion PYPI.md
8000
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ without reconstructing intent from console text.

| What you do in verl | miniVERL equivalent |
| --- | --- |
| pass Hydra overrides | repeat `--set key=value` in v0.8.1 |
| pass Hydra-style overrides | repeat `--set`; v0.9 development also accepts `--overrides-file` and tokens after `--` |
| inspect the resolved config | `miniverl plan --json` |
| execute pure OPD | `miniverl run` |
| reuse prompt Parquet | point `data.train_files` at it directly |
Expand All @@ -112,6 +112,11 @@ miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml \
--set actor_rollout_ref.actor.optim.lr=1e-5
```

External YAML must explicitly accept the high-risk local mappings printed by
`plan` before `run`; the packaged profile carries a value-bound reviewed
manifest. [Override precedence and safe input forms](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/config-overrides.md)
are documented without executing Hydra interpolation or shell text.

The public built-in profile deliberately uses upstream-shaped `name: vllm`
values. miniVERL classifies both rollout and teacher engine names as local
reinterpretations and executes them with sequential local HF phases; this is
Expand Down
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ without reconstructing intent from console text.

| What you do in verl | miniVERL equivalent |
| --- | --- |
| pass Hydra overrides | repeat `--set key=value` in v0.8.1 |
| pass Hydra-style overrides | repeat `--set`; v0.9 development also accepts `--overrides-file` and tokens after `--` |
| inspect the resolved config | `miniverl plan --json` |
| execute pure OPD | `miniverl run` |
| reuse prompt Parquet | point `data.train_files` at it directly |
Expand All @@ -112,6 +112,11 @@ miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml \
--set actor_rollout_ref.actor.optim.lr=1e-5
```

External YAML must explicitly accept the high-risk local mappings printed by
`plan` before `run`; the packaged profile carries a value-bound reviewed
manifest. [Override precedence and safe input forms](docs/config-overrides.md)
are documented without executing Hydra interpolation or shell text.

The public built-in profile deliberately uses upstream-shaped `name: vllm`
values. miniVERL classifies both rollout and teacher engine names as local
reinterpretations and executes them with sequential local HF phases; this is
Expand Down
6 changes: 5 additions & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ target 以无 pickle 格式校验,checkpoint 事务化发布,最终 adapter

| verl 中的动作 | miniVERL 对应方式 |
| --- | --- |
| 传入 Hydra override | v0.8.1 中重复使用 `--set key=value` |
| 传入 Hydra 风格 override | 重复使用 `--set`;v0.9 开发版也接受 `--overrides-file` 与 `--` 后的参数 |
| 检查 resolved config | `miniverl plan --json` |
| 执行纯 OPD | `miniverl run` |
| 复用 prompt Parquet | 直接设置 `data.train_files` |
Expand All @@ -95,6 +95,10 @@ miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml \
--set actor_rollout_ref.actor.optim.lr=1e-5
```

外部 YAML 在 `run` 前必须显式接受 `plan` 列出的高风险本地重解释;内置
profile 使用与具体值绑定的审核清单。详见[覆盖优先级与安全输入格式](docs/config-overrides.md),
miniVERL 不执行 Hydra interpolation 或 shell 文本。

公开内置 profile 刻意使用上游形状的 `name: vllm`。miniVERL 会把 rollout 与 teacher
engine 名分类为本地重解释,再通过顺序本地 HF 阶段执行;这不等于 vLLM 语义。完整
config、data、role 与错误映射见[面向 verl 用户的指南](docs/for-verl-users.md)。
Expand Down
12 changes: 7 additions & 5 deletions THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,13 @@ No source code from another repository has been copied, vendored or adapted.
This matters for two projects in particular, because miniVERL is positioned next
to them:

* **verl** (`https://github.com/verl-project/verl`, Apache-2.0). No verl code,
configuration schema, documentation wording, logo or visual identity is used
here. miniVERL is an independent implementation of a much narrower scope and
is not compatible with, derived from, or endorsed by verl. The name is a nod to
the problem space only.
* **verl** (`https://github.com/volcengine/verl`, Apache-2.0). No verl runtime
code, documentation wording, logo or visual identity is used here. The
reduced test fixture
`tests/fixtures/verl/opd-v0.8-official-example-fields.yaml` is derived from
the field surface of the pinned v0.8.0 Qwen3 OPD example and retains source,
commit and license provenance. miniVERL is an independent implementation of
a much narrower scope and is not endorsed by verl.
* **OPSD** (`https://github.com/HJSang/OPSD_OnPolicyDistillation`). This
repository has no LICENSE file and is therefore all-rights-reserved; nothing
from it has been copied. It is cited in `docs/comparisons.md` and
Expand Down
76 changes: 76 additions & 0 deletions docs/config-overrides.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Config overrides

miniVERL accepts one resolved YAML mapping plus familiar dotted `key=value`
tokens. It does not execute a verl launcher, shell command, Hydra resolver or
OmegaConf interpolation. Resolve composition inside the trusted verl workflow,
then pass either the YAML, an argv JSON array, or a plain override file.

```bash
miniverl plan \
--config verl-opd.yaml \
--overrides-file site.overrides \
--set actor_rollout_ref.actor.optim.lr=1e-5 \
-- \
'data.train_files=["data/train.parquet"]' \
distillation.distillation_loss.topk=64
```

Precedence is deterministic:

1. base YAML;
2. each `--overrides-file`, in command order;
3. each `--set`, in command order;
4. tokens after `--`, in command order.

Every occurrence remains in the compiled report with its source, order,
previous value, previous source, final value and whether it is the effective
occurrence. Duplicate fields are therefore visible rather than silently
collapsed.

## Safe input forms

A plain file contains one expression per line. Empty lines and lines beginning
with `#` are ignored:

```text
data.train_batch_size=8
actor_rollout_ref.rollout.name=vllm
```

A `.json` file must contain an array of strings, suitable for argv captured by
a trusted launcher:

```json
[
"data.train_batch_size=8",
"actor_rollout_ref.rollout.name=vllm"
]
```

Values use safe YAML scalar/list/dict parsing. `${...}`, non-finite numbers,
unknown fields, YAML object constructors, Hydra `+`/`~` operators and shell
scripts fail closed. No input string is evaluated as code.

## High-risk local reinterpretations

`miniverl plan` always lists fields whose upstream meaning is replaced by a
materially different one-GPU meaning—for example a PPO mini-batch becoming a
logical direct-GKD update batch, or `vllm` selecting the sequential local HF
runtime. The packaged profile carries a reviewed approval manifest. Running an
external YAML requires:

```bash
miniverl run --config verl-opd.yaml --accept-local-reinterpretations
```

This flag accepts only the high-risk mappings printed in that compiled report.
It never enables an unsupported algorithm, multi-GPU field, FSDP behavior or
policy-gradient objective.

## Official-example field coverage

The generated [coverage report](generated/verl-opd-v08-official-fields.json)
classifies every leaf in a reduced field-surface fixture derived from verl
v0.8.0's Apache-2.0 Qwen3 FSDP OPD example. It intentionally retains fields
that are rejected, so coverage means “explained deterministically,” not
“supported.”
12 changes: 8 additions & 4 deletions docs/for-verl-users.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ the mapping is explicit and does not imply endorsement or full compatibility.
`distillation.teacher_models.teacher_model.model_path`, response bounds,
learning rate and LoRA configuration.

What is not reusable: arbitrary Hydra composition, shell launch scripts,
What is not reusable: arbitrary Hydra composition inside miniVERL, shell launch scripts,
resource pools, Ray actors, FSDP/Megatron checkpoints, PPO/GRPO, critics,
policy-gradient OPD, task-reward mixtures, multiple teachers and multimodal
workers. Unsupported semantics fail closed instead of falling back silently.
Expand All @@ -30,7 +30,7 @@ workers. Unsupported semantics fail closed instead of falling back silently.
| verl action | miniVERL action |
| --- | --- |
| compose or capture a resolved config | provide resolved YAML to `--config` |
| add a Hydra override | repeat `--set key=value` |
| add a Hydra-style override | repeat `--set`, use `--overrides-file`, or place tokens after `--` |
| inspect resolved intent | `miniverl plan --json` |
| launch OPD | `miniverl run` |
| read prompt Parquet | use the file directly |
Expand All @@ -46,8 +46,9 @@ miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
```

Planning is weight-free and offline. Use `--json` to retain the complete field
matrix. v0.8.1 accepts only repeated `--set`; it does not execute `${...}`
interpolations or shell text.
matrix. The development CLI records repeated `--set`, override files and
trailing tokens with deterministic precedence; it does not execute `${...}`
interpolations or shell text. See [Config overrides](config-overrides.md).

## Same fields, different placement

Expand Down Expand Up @@ -111,6 +112,9 @@ tested.
teachers and distributed counts are intentionally unsupported.
- **Interpolation rejected:** resolve Hydra/OmegaConf in your trusted verl
environment first. miniVERL will not execute `${...}`.
- **High-risk reinterpretation not accepted:** inspect `miniverl plan`, then
pass `--accept-local-reinterpretations` for an external config. Packaged
profiles carry reviewed acceptance metadata.
- **Prompt schema mismatch:** validate the Parquet `prompt` column as structured
role/content messages, or convert it explicitly.
- **Tokenizer mismatch:** actor and teacher scoring require structural identity
Expand Down
Loading
0