@@ -8,7 +8,7 @@
[](https://github.com/DaoyuanLi2816/mini-verl/actions/workflows/build.yml)
[](https://pypi.org/project/miniverl/)
[](https://www.python.org)
-[](https://github.com/DaoyuanLi2816/mini-verl/blob/main/LICENSE)
+[](https://github.com/DaoyuanLi2816/mini-verl/blob/v0.12.0/LICENSE)
@@ -16,16 +16,20 @@
@@ -19,13 +19,17 @@
中文
-**Run verl-style on-policy distillation on one NVIDIA GPU, with every config
-mapping, teacher target and training artifact available for inspection.**
-miniVERL turns typed YAML and structured Parquet prompts into a local actor
-rollout → teacher scoring → actor update loop, then exports standard PEFT,
-Parquet and config artifacts for scale-out work.
+**miniVERL runs a validated subset of verl experiment semantics on one NVIDIA
+GPU.** Give it a resolved verl-shaped config and Parquet prompts; its versioned
+compiler produces a reviewable local plan, executes actor/reference/teacher/
+reward roles in phases, and publishes portable PEFT and data artifacts.
-PyPI `v0.11.0` is stable; `main` is development.
+The current development line adds critic-free RL against official verl
+`v0.9.0` (`483b8a00`): GRPO, Dr.GRPO, RLOO and REINFORCE++, grouped rollouts,
+task rewards and fixed reference-policy KL. The established verl `v0.8.0` OPD
+profiles remain available for direct GKD and sampled-k1 distillation.
+
+PyPI `v0.12.0` is stable; `main` is development.
## Start in 60 seconds
@@ -33,186 +37,124 @@ Install the CUDA-enabled PyTorch build that matches your machine, then:
```bash
python -m pip install "miniverl[train,cuda]"
-miniverl data sample --format verl-parquet --out prompts.parquet
-miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
- --config builtin:qwen3-0.6b-1.7b-opd \
- --set 'data.train_files=["prompts.parquet"]' --out plan.json
-miniverl run --profile verl-opd-v0.8-single-gpu-v1 \
- --plan plan.json --dry-run
+miniverl data sample --task-rewards --rows 8 --out data/rl-prompts.parquet
+miniverl import-verl --profile verl-rl-v0.9-single-gpu-v1 \
+ --config examples/verl-rl-v0.9-single-gpu.yaml --out local-grpo.yaml
+miniverl validate local-grpo.yaml
+miniverl train local-grpo.yaml --dry-run
```
-This path works from the published wheel and loads no model weights while
-planning. The generated `plan.json` binds the source config, ordered overrides,
-profile version and input Parquet bytes. On a CUDA GPU, remove `--dry-run` to
-run the pinned Qwen3-0.6B actor and Qwen3-1.7B teacher recipe.
+The importer writes `local-grpo.import-report.json` beside the native recipe.
+It accounts for every accepted source field and records how distributed
+resource settings lower to one process and one device. Remove `--dry-run` to
+load the pinned Qwen3-0.6B actor and execute the two-iteration example.
-The `[train,cuda]` extra installs the ML and quantization dependencies. Select
-the matching CUDA PyTorch wheel separately with the
-[PyTorch installer](https://pytorch.org/get-started/locally/). The
-[single-GPU guide](docs/single-gpu-guide.md) covers memory planning from 8 GiB
-cards upward and includes the maintainer-measured RTX 4080 stack.
+The `[train,cuda]` extra installs the ML and quantization stack; select PyTorch's
+CUDA build separately with the [PyTorch installer](https://pytorch.org/get-started/locally/).
+The [single-GPU guide](docs/single-gpu-guide.md) covers memory planning and the
+maintainer-measured RTX 4080 environment.
## What a run gives you
-- **A reviewable plan.** Every accepted verl field has a local effect,
- classification and risk level before weights are loaded.
-- **Strict current-policy trajectories.** The actor policy version, token
- spans and teacher-supervised positions travel together.
-- **Compact teacher targets.** Top-k targets and sampled-k1 signals use
- checksummed, pickle-free caches.
-- **Recoverable training.** Transactional manifests, checkpoints and cache
- indexes support inspection and exact resume.
-- **Standard outputs.** PEFT adapters, safetensors, structured Parquet,
- resolved config and typed provenance remain portable.
-
-```bash
-miniverl run --profile verl-opd-v0.8-single-gpu-v1 --plan plan.json \
- --output runs --run-id my-opd
-miniverl inspect runs/my-opd/trajectories.jsonl
-miniverl cache stats runs/my-opd/teacher-cache
-miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout
-miniverl bridge doctor scaleout --json
-```
+- **A field-by-field compiler report.** Experiment fields retain their meaning;
+ physical distribution fields receive an explicit one-GPU lowering.
+- **Policy-bound trajectories.** Group/sample identity, generated-token spans,
+ behavior log-probabilities and policy version travel together.
+- **Auditable rewards and objectives.** Reward components, group advantages,
+ reference KL, clipping, entropy and update metrics are structured data.
+- **Exact recovery.** Transactional manifests and checkpoints restore policy,
+ optimizer, cursor, RNG and reward identities before another rollout.
+- **Portable outputs.** PEFT adapters, safetensors, Parquet, resolved config and
+ typed provenance can move into a larger workflow.
## How it works
-
-miniVERL schedules actor, teacher and optional reference roles in phases inside
-one ordinary process. The actor generates with its current adapter, the teacher
-scores the visited token positions, and the actor receives a padded token-mean
-update. Resident, swap and shared-backbone placement strategies keep those role
-identities explicit while adapting to available memory.
-
-Each phase publishes evidence before the next boundary: structured
-trajectories carry per-token provenance, teacher targets are checksummed,
-checkpoints publish transactionally, and the final adapter is reloaded through
-standard PEFT. The result is a training run you can inspect without
-reconstructing intent from console logs.
+One GPU is treated as a temporal scheduler for logical roles. Critic-free RL
+generates complete prompt groups, scores outcomes, computes the pinned
+advantage estimator, optionally evaluates a frozen reference adapter, and then
+updates the actor. OPD uses the same trajectory and checkpoint foundations,
+with a teacher-scoring phase in place of outcome-only rewards.
## Choose your path
| Goal | First command | Primary artifact | Next step |
| --- | --- | --- | --- |
-| **Run local OPD** | `miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml` | immutable execution plan | [OPD quickstart](docs/opd-quickstart.md) |
-| **Bring a verl profile** | `miniverl compat check --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml` | field-by-field compatibility report | [For verl users](docs/for-verl-users.md) |
+| **Run local RL** | `miniverl import-verl --profile verl-rl-v0.9-single-gpu-v1 --config verl-rl.yaml --out local.yaml` | native recipe + compatibility report | [RL quickstart](docs/verl-rl-runtime.md) |
+| **Run local OPD** | `miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml --out plan.json` | immutable execution plan | [OPD quickstart](docs/opd-quickstart.md) |
| **Fit your GPU** | `miniverl plan --config verl-opd.yaml --probe` | measured placement plan | [Hardware planning](docs/hardware-planning.md) |
-| **Hand off for scale-out** | `miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout` | PEFT + Parquet + config bundle | [Scale-out contract](docs/verl-opd-scaleout.md) |
-
-The native recipe system also supports SFT, DPO, offline KD and tool-aware OPD
-over calculator, JSON navigation, read-only SQLite and custom environments.
+| **Hand off artifacts** | `miniverl export-verl --run runs/my-run --target-verl v0.8.0 --out scaleout` | PEFT + Parquet + config bundle | [Scale-out contract](docs/verl-opd-scaleout.md) |
-## Familiar verl inputs, local execution
+Native recipes also support SFT, DPO and offline KD, plus calculator, JSON
+navigation, read-only SQLite and custom tool environments.
-The current profiles pin official verl `v0.8.0` at
-`7aed6b230776f963fa09509c10d9c3a767d1102c` and preserve recognizable fields:
+## Current capability matrix
-```yaml
-actor_rollout_ref:
- model: {path: Qwen/Qwen3-0.6B}
- rollout: {name: vllm, n: 1}
-distillation:
- teacher_models:
- teacher_model:
- model_path: Qwen/Qwen3-1.7B
- inference: {name: vllm}
-```
-
-The compiler translates distributed resource intent into sequential local
-Hugging Face phases and records that translation in the plan. Two measured
-profiles are available:
-
-| Profile | Objective | Teacher target |
+| Experiment surface | Local status | Contract |
| --- | --- | --- |
-| `verl-opd-v0.8-single-gpu-v1` | direct GKD `forward_kl_topk` | top-k token IDs and log-probabilities |
-| `verl-opd-v0.8-single-gpu-pg-k1-v1` | sampled `k1` + vanilla policy loss | sampled-token teacher log-probability |
-
-Use `miniverl profiles show`, `compat explain` and `compat check` to inspect
-the complete mapping. [Compatibility profiles](docs/profiles/index.md) explains
-how profile identity follows plans, caches, checkpoints and exports. Separate
-conformance-only grouped profiles add transactional Parquet `n>1` samples
-without changing either measured `n=1` profile or introducing GRPO semantics.
-A separate conformance-only rewarded profile adds deterministic exact-answer
-rewards and explicit group advantage composition; it has no task-quality claim.
+| GRPO / Dr.GRPO | semantically conformant | verl v0.9 group statistics and vanilla clipped policy loss |
+| RLOO / REINFORCE++ | semantically conformant | verl v0.9 advantage and masking rules |
+| Grouped `n > 1` rollouts | supported | complete groups, stable sample seeds and behavior-policy identity |
+| Task rewards | supported | exact-answer/target-length built-ins, environment verifier, or trusted Python API |
+| Fixed reference KL | locally lowered | frozen adapter role on the shared actor backbone |
+| Direct GKD / sampled-k1 OPD | supported | pinned verl v0.8 profiles with teacher targets |
+| PPO / GAE execution | not implemented | math is conformance-tested; a trainable critic lifecycle is not yet present |
+| Ray, FSDP/FSDP2, Megatron, TP/PP/DP > 1 | distributed-only | these change physical scale, not the local objective |
+
+The generated [v0.9 compatibility record](docs/generated/verl-rl-v0.9-compatibility.json)
+binds every example field to its source value, local target, classification and
+compiler-rule digest. `miniverl import-verl` accepts a resolved documented
+subset, rejects unknown algorithm-changing fields, and never substitutes a
+dataset or reward implementation.
## Measured systems evidence
-Rollout Runtime v2 has a complete 24-cell RTX 4080 measurement across
-64/256/512-token responses, `n=1/4`, greedy and seeded sampling. The compiled
-`hf_cached` path reached 124.2–248.7 output tokens/s and passed the
-preregistered 2× gate in every required cell. It keeps the NF4 actor for
-training and synchronizes each policy version into an owned BF16 rollout
-mirror.
-
-Managed vLLM 0.28.0 with CUDA Graph execution reached 626.6–836.4 output
-tokens/s and was 3.08–5.97× faster than `hf_cached` in the 256/512-token cells.
-Both backends passed memory, eight-refresh and teardown gates. vLLM is the
-measured direct-GKD engine; PG-k1 stays on `hf_cached` because its engine
-log-probability probe exceeded the numerical threshold. See the
-[full runtime result](docs/benchmarks/rollout-runtime-v2.md) for every cell,
-the preserved first candidate and artifact hashes.
-
-The Qwen3-0.6B/1.7B developer workload consumed **32 distinct prompts**, used a
-64-token response bound, and completed **8 current-policy updates** at
-**3.1914 GiB peak reserved VRAM** on an RTX 4080. Median steady-state rollout,
-teacher-scoring and update times were 9.7200, 0.4864 and 2.3260 seconds. A
-matched interruption after update four resumed to byte-identical trajectories,
-adapter and optimizer tensors.
-
-A second SmolLM2-360M/1.7B workload completed the same 32-prompt, 8-update
-shape at 1.4961 GiB peak reserved VRAM, including exact resume, PEFT reload and
-scale-out materialization. Read the
-[Qwen3](docs/verl-opd-reference-workload.md) and
-[SmolLM2](docs/smollm2-opd-workload.md) systems records for configs, hashes and
-phase-level measurements.
-
-Other NVIDIA GPUs use the same device-name-agnostic CUDA path. Model fit and
-speed vary with VRAM, context, quantization, kernels and software versions;
-`miniverl doctor` and `plan --probe` expose those machine-specific choices.
+The published Qwen3-0.6B/1.7B OPD workload consumed 32 distinct prompts and
+completed eight current-policy updates at 3.1914 GiB peak reserved VRAM on an
+RTX 4080. A matched interruption after update four reproduced byte-identical
+trajectories, adapter and optimizer tensors. The SmolLM2-360M/1.7B workload
+completed the same shape at 1.4961 GiB. [System records](docs/verl-opd-reference-workload.md)
+contain configs, hashes and phase timings.
+
+Rollout Runtime v2 measured `hf_cached` and managed vLLM across 24 RTX 4080
+cells spanning response length, sampling mode and `n=1/4`. See the
+[runtime report](docs/benchmarks/rollout-runtime-v2.md). Evidence for the new
+RL family is published through the exact-wheel release qualification rather
+than presented as a task-quality comparison.
## Research record
-The repository publishes positive, mixed and negative results with the same
-resolved configs and source hashes. The calculator study found that a
-protocol-qualified teacher prevented collapse but tied supervised
-continuation. RecoveryBench found no fresh-state advantage in its scoped
-SQLite setting. Alignment Lab began at a saturated SFT checkpoint and exposed
-utility regressions that two sandbox safety checks missed. The preregistered
-External Alignment Gate stopped before continuation because every candidate
-failed the retained-utility threshold.
-
-- [Calculator protocol study](docs/benchmarking.md)
-- [RecoveryBench](docs/recoverybench/recoverybench-v1.md)
-- [Alignment Lab](docs/alignment-lab/alignment-lab-v1.md)
-- [External Alignment Gate](docs/alignment-external/alignment-external-v1.md)
-
-These reports answer scoped experimental questions; the
-[limitations](docs/limitations.md) page collects the measurement, architecture,
-security and generalization boundaries in one place.
-
-## Scope
-
-miniVERL is designed for one local process, one NVIDIA CUDA GPU and the
-documented OPD profiles above. Scale-out support produces and validates a
-portable bundle against the pinned upstream source. The
-[compatibility policy](docs/compatibility.md) lists supported fields, profile
-semantics and handoff readiness states; [limitations](docs/limitations.md)
-covers the broader execution and scientific boundaries. miniVERL is an
-independent Apache-2.0 project.
+The scientific reports preserve their original outcomes and frozen inputs:
+[calculator protocol](docs/benchmarking.md),
+[RecoveryBench](docs/recoverybench/recoverybench-v1.md),
+[Alignment Lab](docs/alignment-lab/alignment-lab-v1.md), and the
+[External Alignment Gate](docs/alignment-external/alignment-external-v1.md).
+They are scoped studies, separate from runtime qualification.
+
+## Compatibility boundary
+
+miniVERL targets one local process and one NVIDIA CUDA GPU. The compiler
+distinguishes exact or conformant semantics, local physical lowering,
+distributed-only settings, feasible work not yet implemented, and technically
+unsupported inputs. The [compatibility policy](docs/compatibility.md) is the
+complete matrix; [limitations](docs/limitations.md) collects architecture,
+measurement, security and generalization boundaries in one place. miniVERL is
+an independent Apache-2.0 project.
## Development
```bash
git clone https://github.com/DaoyuanLi2816/mini-verl.git
cd mini-verl
-python -m pip install -e ".[dev]"
+python -m pip install -e ".[dev,train]"
pytest -q -m "not gpu and not network"
```
See [CONTRIBUTING.md](CONTRIBUTING.md), [CHANGELOG.md](CHANGELOG.md),
-[CITATION.cff](CITATION.cff), the
-[reproducibility guide](docs/reproducibility.md), [SECURITY.md](SECURITY.md)
-and the [Apache-2.0 license](LICENSE).
+[CITATION.cff](CITATION.cff), the [guide for verl users](docs/for-verl-users.md),
+[SECURITY.md](SECURITY.md) and the
+[Apache-2.0 license](LICENSE).
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 095e4d7..96b3327 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -1,5 +1,5 @@
-
@@ -18,12 +18,15 @@
English
-**在一张 NVIDIA GPU 上运行 verl 风格在线策略蒸馏，并能检查每一项配置映射、
-teacher target 与训练产物。** miniVERL 把类型化 YAML 和结构化 Parquet prompt
-编译成本地 actor rollout → teacher scoring → actor update 循环，再导出标准 PEFT、
-Parquet 与配置产物，便于继续扩展。
+**miniVERL 在一张 NVIDIA GPU 上运行经过验证的 verl 实验语义子集。** 输入 resolved
+verl 风格配置与 Parquet prompt，版本化编译器会生成可审阅的本地计划；actor、reference、
+teacher 与 reward 角色按阶段执行，最终发布可携带的 PEFT 与数据产物。
-PyPI `v0.11.0` 是稳定版；`main` 是开发版。
+当前开发线针对官方 verl `v0.9.0`（`483b8a00`）加入 critic-free RL：GRPO、
+Dr.GRPO、RLOO、REINFORCE++、grouped rollout、task reward 与固定 reference-policy
+KL。已有 verl `v0.8.0` OPD profile 继续提供 direct GKD 与 sampled-k1 蒸馏。
+
+PyPI `v0.12.0` 是稳定版；`main` 是开发版。
## 60 秒开始
@@ -31,165 +34,112 @@ PyPI `v0.11.0` 是稳定版；`main` 是开发版。
```bash
python -m pip install "miniverl[train,cuda]"
-miniverl data sample --format verl-parquet --out prompts.parquet
-miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
- --config builtin:qwen3-0.6b-1.7b-opd \
- --set 'data.train_files=["prompts.parquet"]' --out plan.json
-miniverl run --profile verl-opd-v0.8-single-gpu-v1 \
- --plan plan.json --dry-run
+miniverl data sample --task-rewards --rows 8 --out data/rl-prompts.parquet
+miniverl import-verl --profile verl-rl-v0.9-single-gpu-v1 \
+ --config examples/verl-rl-v0.9-single-gpu.yaml --out local-grpo.yaml
+miniverl validate local-grpo.yaml
+miniverl train local-grpo.yaml --dry-run
```
-这条路径直接使用 PyPI wheel。规划阶段无需加载模型权重；生成的 `plan.json` 会绑定
-源配置、顺序 override、profile 版本与输入 Parquet 字节。准备好 CUDA GPU 后移除
-`--dry-run`，即可运行固定版本的 Qwen3-0.6B actor 与 Qwen3-1.7B teacher recipe。
+Importer 会在原生 recipe 旁写入 `local-grpo.import-report.json`，逐项说明源字段及其
+本地效果，并记录分布式资源设置如何转换为一个进程、一张 GPU。移除 `--dry-run` 后，
+示例会加载固定 revision 的 Qwen3-0.6B actor 并执行两次 rollout iteration。
-`[train,cuda]` extra 安装 ML 与量化依赖。CUDA PyTorch wheel 请通过
+`[train,cuda]` extra 安装训练与量化依赖；CUDA PyTorch build 仍需通过
[PyTorch 安装器](https://pytorch.org/get-started/locally/)单独选择。
-[单卡指南](docs/single-gpu-guide.md)介绍从 8 GiB 显卡开始的显存规划，并附有维护者实测的
-RTX 4080 环境。
+[单卡指南](docs/single-gpu-guide.md)包含显存规划与维护者实测的 RTX 4080 环境。
## 一次运行会得到什么
-- **可审阅的 plan。** 每个接受的 verl 字段在加载权重前就有本地作用、分类与风险等级。
-- **严格的 current-policy trajectory。** actor policy version、token span 与 teacher
- 监督位置始终绑定在一起。
-- **紧凑的 teacher target。** top-k target 与 sampled-k1 signal 写入带校验和、无 pickle
- 的 cache。
-- **可恢复训练。** 事务化 manifest、checkpoint 与 cache index 支持检查和精确续跑。
-- **标准产物。** PEFT adapter、safetensors、结构化 Parquet、resolved config 与类型化
- provenance 可直接携带。
-
-```bash
-miniverl run --profile verl-opd-v0.8-single-gpu-v1 --plan plan.json \
- --output runs --run-id my-opd
-miniverl inspect runs/my-opd/trajectories.jsonl
-miniverl cache stats runs/my-opd/teacher-cache
-miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout
-miniverl bridge doctor scaleout --json
-```
+- **逐字段编译报告。** 实验字段保留语义，物理分布字段得到明确的单卡 lowering。
+- **与策略绑定的 trajectory。** group/sample 身份、模型生成 token span、behavior
+ log-probability 与 policy version 始终一起保存。
+- **可审计的 reward 与 objective。** reward component、group advantage、reference
+ KL、clip fraction、entropy 与 update metric 都是结构化数据。
+- **精确恢复。** 事务化 manifest 与 checkpoint 在下一次 rollout 前恢复 policy、
+ optimizer、cursor、RNG 与 reward identity。
+- **可携带产物。** PEFT adapter、safetensors、Parquet、resolved config 与类型化
+ provenance 可以进入更大的工作流。
## 工作方式
-
-miniVERL 在一个普通进程中分阶段调度 actor、teacher 与可选 reference。actor 使用当前
-adapter 生成，teacher 对实际访问的 token 位置评分，随后 actor 接收 padded token-mean
-update。resident、swap 与 shared-backbone placement 会适配可用显存，同时保持各角色身份
-清晰可追踪。
-
-每个阶段都会先发布证据，再跨越下一边界：trajectory 携带逐 token provenance，teacher
-target 带校验和，checkpoint 事务化发布，最终 adapter 通过标准 PEFT 重新加载。检查一次
-运行时不必再从终端日志反推原始意图。
+miniVERL 把一张 GPU 当作逻辑角色的时间调度器。Critic-free RL 先生成完整 prompt
+group，计算 outcome reward 和固定版本的 advantage estimator，按需运行冻结的 reference
+adapter，最后更新 actor。OPD 复用相同的 trajectory 与 checkpoint 基础设施，并以
+teacher-scoring 阶段提供训练信号。
## 选择你的路径
| 目标 | 第一个命令 | 主要产物 | 下一步 |
| --- | --- | --- | --- |
-| **在本地运行 OPD** | `miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml` | 不可变执行 plan | [OPD quickstart](docs/opd-quickstart.md) |
-| **迁移 verl profile** | `miniverl compat check --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml` | 逐字段兼容报告 | [面向 verl 用户](docs/for-verl-users.md) |
-| **适配你的显卡** | `miniverl plan --config verl-opd.yaml --probe` | 实测 placement plan | [硬件规划](docs/hardware-planning.md) |
-| **准备 scale-out 交接** | `miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout` | PEFT + Parquet + config bundle | [Scale-out 契约](docs/verl-opd-scaleout.md) |
-
-原生 recipe 系统还支持 SFT、DPO、offline KD，以及 calculator、JSON navigation、只读
-SQLite 和自定义环境上的 tool-aware OPD。
-
-## 熟悉的 verl 输入，本地执行
-
-当前 profile 固定官方 verl `v0.8.0` commit
-`7aed6b230776f963fa09509c10d9c3a767d1102c`，并保留熟悉的字段：
-
-```yaml
-actor_rollout_ref:
- model: {path: Qwen/Qwen3-0.6B}
- rollout: {name: vllm, n: 1}
-distillation:
- teacher_models:
- teacher_model:
- model_path: Qwen/Qwen3-1.7B
- inference: {name: vllm}
-```
+| **本地 RL** | `miniverl import-verl --profile verl-rl-v0.9-single-gpu-v1 --config verl-rl.yaml --out local.yaml` | 原生 recipe + 兼容报告 | [RL quickstart](docs/verl-rl-runtime.md) |
+| **本地 OPD** | `miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml --out plan.json` | 不可变执行计划 | [OPD quickstart](docs/opd-quickstart.md) |
+| **适配显卡** | `miniverl plan --config verl-opd.yaml --probe` | 实测 placement plan | [硬件规划](docs/hardware-planning.md) |
+| **交接产物** | `miniverl export-verl --run runs/my-run --target-verl v0.8.0 --out scaleout` | PEFT + Parquet + config bundle | [Scale-out 契约](docs/verl-opd-scaleout.md) |
-编译器把分布式 resource intent 转换为顺序本地 Hugging Face 阶段，并在 plan 中记录这次
-转换。目前有两个实测 profile：
+原生 recipe 还支持 SFT、DPO、offline KD，以及 calculator、JSON navigation、只读
+SQLite 和自定义 tool environment。
-| Profile | 目标 | Teacher target |
-| --- | --- | --- |
-| `verl-opd-v0.8-single-gpu-v1` | direct GKD `forward_kl_topk` | top-k token ID 与 log-probability |
-| `verl-opd-v0.8-single-gpu-pg-k1-v1` | sampled `k1` + vanilla policy loss | sampled-token teacher log-probability |
+## 当前能力矩阵
-使用 `miniverl profiles show`、`compat explain` 与 `compat check` 查看完整映射。
-[兼容 profile](docs/profiles/index.md)说明 profile identity 如何跟随 plan、cache、checkpoint
-与 export。另有两个仅完成一致性验证的 grouped profile，为 Parquet prompt 提供事务化
-`n>1` 独立样本；它们不改变两个实测 `n=1` profile，也不引入 GRPO 语义。
-另有一个仅完成一致性验证的 rewarded profile，加入确定性的 exact-answer reward 与显式
-group advantage 组合；目前不对任务质量作结论。
+| 实验能力 | 本地状态 | 契约 |
+| --- | --- | --- |
+| GRPO / Dr.GRPO | 语义一致 | verl v0.9 group statistic 与 vanilla clipped policy loss |
+| RLOO / REINFORCE++ | 语义一致 | verl v0.9 advantage 与 masking 规则 |
+| Grouped `n > 1` rollout | 已支持 | 完整 group、稳定 sample seed 与 behavior-policy identity |
+| Task reward | 已支持 | exact-answer/target-length 内置奖励、environment verifier 或可信 Python API |
+| 固定 reference KL | 本地 lowering | 与 actor 共用 backbone 的冻结 adapter 角色 |
+| Direct GKD / sampled-k1 OPD | 已支持 | 固定 verl v0.8 profile 与 teacher target |
+| PPO / GAE 执行 | 尚未实现 | 数学原语已做一致性验证；还没有可训练 critic 生命周期 |
+| Ray、FSDP/FSDP2、Megatron、TP/PP/DP > 1 | 仅分布式 | 这些能力改变物理规模，不改变本地 objective |
+
+生成的 [v0.9 兼容记录](docs/generated/verl-rl-v0.9-compatibility.json)绑定示例中每个
+字段的源值、本地目标、分类与编译规则哈希。`miniverl import-verl` 接收 documented
+resolved subset；未知且会改变算法的字段会被拒绝，dataset 与 reward 实现不会被替换。
## 实测系统证据
-Rollout Runtime v2 已在 RTX 4080 上完成 24-cell 测量，覆盖 64/256/512-token
-response、`n=1/4`、greedy 与 seeded sampling。编译后的 `hf_cached` 达到
-124.2–248.7 output tokens/s，并在所有必需 cell 中通过预注册的 2× gate。训练仍以
-NF4 actor 为准，每个 policy version 都会精确同步到独立的 BF16 rollout mirror。
-
-启用 CUDA Graph 的 managed vLLM 0.28.0 达到 626.6–836.4 output tokens/s；在
-256/512-token cell 中比 `hf_cached` 快 3.08–5.97×。两个 backend 都通过显存、8 次
-policy refresh 与 teardown gate。vLLM 是实测的 direct-GKD engine；其 engine
-log-probability probe 超出数值门槛，因此 PG-k1 继续使用 `hf_cached`。每个 cell、保留的
-第一版失败候选与 artifact hash 见
-[完整 runtime 结果](docs/benchmarks/rollout-runtime-v2.md)。
-
-Qwen3-0.6B/1.7B developer workload 消费 **32 个不同 prompt**，response 上限为 64 token，
-在 RTX 4080 上完成 **8 次 current-policy update**，**peak reserved VRAM 为 3.1914 GiB**。
-稳态 rollout、teacher scoring 与 update 的中位耗时分别为 9.7200、0.4864 与 2.3260 秒。
-匹配运行在第 4 次 update 后中断并续跑，最终 trajectory、adapter 与 optimizer tensor
-字节一致。
-
-第二套 SmolLM2-360M/1.7B workload 以 1.4961 GiB peak reserved VRAM 完成相同的
-32-prompt、8-update 形状，并通过精确续跑、PEFT 重载与 scale-out 物化。完整配置、哈希和
-分阶段测量见 [Qwen3](docs/verl-opd-reference-workload.md) 与
-[SmolLM2](docs/smollm2-opd-workload.md) 系统记录。
-
-其他 NVIDIA GPU 使用同一条 device-name-agnostic CUDA 路径。模型能否装下以及实际速度
-取决于显存、上下文、量化、kernel 与软件版本；`miniverl doctor` 和 `plan --probe` 会把
-这些机器相关选择显示出来。
+已发布的 Qwen3-0.6B/1.7B OPD workload 在 RTX 4080 上消费 32 个不同 prompt，完成
+8 次 current-policy update，peak reserved VRAM 为 3.1914 GiB。第 4 次 update 后的
+匹配中断/续跑复现了字节一致的 trajectory、adapter 与 optimizer tensor。配套
+SmolLM2-360M/1.7B workload 以 1.4961 GiB 完成同一形状；完整配置、哈希和阶段耗时见
+[系统记录](docs/verl-opd-reference-workload.md)。
-## 研究记录
+Rollout Runtime v2 在 RTX 4080 上测量了 `hf_cached` 与 managed vLLM 的 24 个 cell，
+覆盖不同 response length、sampling mode 与 `n=1/4`。详见
+[runtime 报告](docs/benchmarks/rollout-runtime-v2.md)。新 RL 家族的证据由 exact-wheel
+release qualification 发布，不作为任务质量对比。
-仓库以相同的 resolved config 与源哈希发布正面、混合和负面结果。Calculator study 中，
-protocol-qualified teacher 避免了 collapse，但只追平 supervised continuation。
-RecoveryBench 在限定的 SQLite 设置中没有观察到 fresh-state 优势。Alignment Lab 从饱和
-SFT checkpoint 出发，发现了两个 sandbox safety check 没有捕捉到的 utility regression。
-预注册 External Alignment Gate 则因所有候选都未通过 retained-utility 阈值，在 continuation
-之前结束。
-
-- [Calculator protocol study](docs/benchmarking.md)
-- [RecoveryBench](docs/recoverybench/recoverybench-v1.md)
-- [Alignment Lab](docs/alignment-lab/alignment-lab-v1.md)
-- [External Alignment Gate](docs/alignment-external/alignment-external-v1.md)
+## 研究记录
-这些报告回答的是各自限定的实验问题；[限制页面](docs/limitations.md)集中列出测量、架构、
-安全与泛化边界。
+科学报告保留原始结论与 frozen 输入：
+[calculator protocol](docs/benchmarking.md)、
+[RecoveryBench](docs/recoverybench/recoverybench-v1.md)、
+[Alignment Lab](docs/alignment-lab/alignment-lab-v1.md) 与
+[External Alignment Gate](docs/alignment-external/alignment-external-v1.md)。这些是范围明确的
+研究记录，与 runtime qualification 分开。
-## 适用范围
+## 兼容边界
-miniVERL 面向一个本地进程、一张 NVIDIA CUDA GPU 和上面的文档化 OPD profile。
-Scale-out 支持会生成 portable bundle，并对固定版本的上游源码执行验证。
-[兼容性政策](docs/compatibility.md)列出支持字段、profile 语义与 handoff readiness state；
-[限制页面](docs/limitations.md)集中说明更广的执行和科学边界。miniVERL 是独立的
-Apache-2.0 项目。
+miniVERL 面向一个本地进程、一张 NVIDIA CUDA GPU。编译器分别标记精确或语义一致、
+本地物理 lowering、仅分布式、概念上可行但尚未实现，以及因具体原因不支持的输入。
+[兼容性政策](docs/compatibility.md)给出完整矩阵；[限制页面](docs/limitations.md)集中说明
+架构、测量、安全与泛化边界。miniVERL 是独立的 Apache-2.0 项目。
## 开发
```bash
git clone https://github.com/DaoyuanLi2816/mini-verl.git
cd mini-verl
-python -m pip install -e ".[dev]"
+python -m pip install -e ".[dev,train]"
pytest -q -m "not gpu and not network"
```
另见 [CONTRIBUTING.md](CONTRIBUTING.md)、[CHANGELOG.md](CHANGELOG.md)、
-[CITATION.cff](CITATION.cff)、[复现指南](docs/reproducibility.md)、
-[SECURITY.md](SECURITY.md) 与 [Apache-2.0 license](LICENSE)。
+[CITATION.cff](CITATION.cff)、[SECURITY.md](SECURITY.md) 与
+[Apache-2.0 license](LICENSE)。
diff --git a/SECURITY.md b/SECURITY.md
index 8b60535..ab37f7f 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -2,7 +2,7 @@
## Supported versions
-The current supported stable line is `0.11.0`; security fixes land on `main` and
+The current supported stable line is `0.12.0`; security fixes land on `main` and
in the next patch release. There are no separately maintained older branches.
## Reporting a vulnerability
diff --git a/docs/banner.svg b/docs/banner.svg
index f058b63..e2eed5f 100644
--- a/docs/banner.svg
+++ b/docs/banner.svg
@@ -1,7 +1,7 @@
miniVERL
- A local runtime for inspectable verl-style on-policy distillation on one CUDA GPU.
+ A local runtime that compiles supported verl experiment semantics for one CUDA GPU.
VERL
- Run verl-style OPD on one consumer GPU.
+ Run verl experiment semantics on one consumer GPU.
- Typed config, Parquet prompts, local role scheduling,
+ Versioned config lowering, grouped rollouts, local roles,
- inspectable targets and standard PEFT artifacts.
+ inspectable rewards, targets and standard PEFT artifacts.
@@ -100,7 +100,7 @@
Run locally
- actor rollout · teacher score · actor update
+ actor rollout · reward / ref / teacher · update
diff --git a/docs/comparisons.md b/docs/comparisons.md
index 6a588ef..50e2205 100644
--- a/docs/comparisons.md
+++ b/docs/comparisons.md
@@ -12,7 +12,7 @@ before making a long-lived infrastructure decision.
| Your priority | Best starting point |
| --- | --- |
-| Read, modify and audit OPD on one NVIDIA GPU | **miniVERL** |
+| Prototype and inspect supported verl RL or OPD on one NVIDIA GPU | **miniVERL** |
| Scale training across accelerators and nodes | **verl** |
| Add generalized-JSD distillation to a Transformers training workflow | **TRL GKD** |
| Explore cross-tokenizer or multimodal KD | **KDFlow** |
@@ -23,9 +23,9 @@ before making a long-lived infrastructure decision.
| Dimension | miniVERL | verl | TRL GKD | KDFlow | OPSD | plain SFT |
| --- | --- | --- | --- | --- | --- | --- |
-| **Design center** | Inspectable single-GPU OPD, plus local SFT/DPO/KD baselines | General RL and distillation post-training at scale | A distillation trainer inside the Transformers ecosystem | Distributed KD across policy, tokenizer and modality choices | Paper-oriented OPD experiments built on verl | Next-token learning on a fixed dataset |
+| **Design center** | Verl experiment semantics lowered to one GPU, plus native SFT/DPO/KD | General RL and distillation post-training at scale | A distillation trainer inside the Transformers ecosystem | Distributed KD across policy, tokenizer and modality choices | Paper-oriented OPD experiments built on verl | Next-token learning on a fixed dataset |
| **Runtime shape** | One Python process; optional CUDA training stack | Ray with distributed model and rollout backends | Transformers Trainer + Accelerate | Ray + SGLang | verl-based | Framework-dependent |
-| **On-policy path** | Strict current-policy rollout → teacher score → actor update | First-class distributed distillation and agent-loop paths | Configurable student-generated sequences | Available | Available | Fixed dataset |
+| **On-policy path** | Grouped rollout → reward/reference or teacher → actor update | First-class distributed RL, distillation and agent-loop paths | Configurable student-generated sequences | Available | Available | Fixed dataset |
| **Tool trajectories** | Calculator, JSON navigation, read-only SQLite and custom typed environments | Agent loop and tool parser | Chat-dataset training | Project-dependent | Tool-oriented experiments | Dataset-defined |
| **Teacher artifacts** | Exact or top-k targets, sampled-k1 signals and sharded safetensors cache | Distributed trainer state | Teacher forward pass in the trainer; server distillation supports top-k + tail controls | Chunked/distributed target handling | Chunked divergence path | Target tokens |
| **Primary output** | PEFT adapter, trajectories, cache, plan and portable provenance bundle | Distributed checkpoints and rollout/training artifacts | Transformers model or adapter | Project-defined model artifacts | Experiment artifacts | Model or adapter |
@@ -37,11 +37,12 @@ only within their original model, data and runtime setup.
## miniVERL's design center
-miniVERL concentrates on the parts of a local distillation run that benefit
+miniVERL concentrates on the parts of a local post-training run that benefit
from explicit evidence:
- a typed compiler that records how each source field affects local execution;
-- strict policy-version binding between rollouts and teacher targets;
+- group/sample and policy-version binding across rollout, rewards and targets;
+- verl v0.9-conformant GRPO, Dr.GRPO, RLOO and REINFORCE++ primitives;
- token-span provenance created during generation;
- exact, top-k and sampled-k1 teacher signals with checksummed cache identity;
- transactional plans, checkpoints and export publication;
@@ -54,10 +55,10 @@ throughput.
## When scale is the main requirement
verl is the natural continuation when the workload needs multi-GPU execution,
-high-throughput rollout engines, distributed checkpointing or RL objectives.
-miniVERL's export path prepares a pinned bundle of local artifacts for that
-workflow and reports artifact completeness, materialization and launchability
-as separate states.
+distributed checkpointing or the broader algorithm/runtime surface. miniVERL's
+export path prepares a pinned bundle of local artifacts for that workflow and
+reports artifact completeness, materialization and launchability as separate
+states.
Use the [scale-out contract](verl-opd-scaleout.md) for that handoff. The full
list of miniVERL's algorithm, architecture and evidence boundaries lives in
diff --git a/docs/compatibility.md b/docs/compatibility.md
index c5dcb0e..78fbf9f 100644
--- a/docs/compatibility.md
+++ b/docs/compatibility.md
@@ -1,80 +1,107 @@
# Compatibility policy
-miniVERL versions its executable profiles and artifact schemas independently.
-A minor release may add validated fields or tighten input validation while
-preserving the historical meaning of existing artifacts. Release notes identify
-migrations, and readers retain the schema-specific behavior needed to load them.
-
-## Stable surfaces through v0.9
-
-- Package name `miniverl`, command name `miniverl`, documented command names,
- and the small Python entry points `RunConfig` and `OPDTrainer`.
-- Existing run artifact names. The v0.2.4 config files are additive;
- `config.original.yaml` remains the checkpoint-resume compatibility layer.
-- Trajectory schema 1, cache schema 2 plus the explicitly constrained legacy
- schema-1 reader, checkpoint manifest schema 1, and benchmark schemas 1/2.
-- Historical protocol-v1 and verifier-v1 semantics. New runs default to the
- stricter v2 semantics rather than rewriting old evidence.
-- The v0.6 verl bridge is separately versioned as profile
- `single-gpu-online-distillation-v1`. It targets official verl `v0.8.0` at
- commit `7aed6b230776f963fa09509c10d9c3a767d1102c`; every bundle carries that
- exact contract in `REQUIRED_VERL.txt`.
-
-## Pinned OPD profiles
-
-v0.8 introduced `verl-opd-v0.8-single-gpu-v1`: one local actor, one teacher,
-pure GKD `forward_kl_topk`, token-mean aggregation, LoRA/QLoRA and Parquet
-prompts. The registry now also contains sampled-k1, grouped-independent and a
-separate rewarded sampled-k1 contract.
-`import-verl --config` publishes a canonical profile plus field report;
-`export-verl` recognizes a completed compatible run and emits its profile-bound
-OPD overrides. The rewarded profile accepts only a closed deterministic
-exact-answer provider and remains critic/value/GRPO-free. Multiple teachers,
-arbitrary reward code, multimodal and all distributed semantics fail closed.
-Local physical phase scheduling is always
-labelled as a reinterpretation of upstream resource fields.
-
-The [machine-readable field matrix](generated/verl-opd-v0.8-compatibility.json)
-is regenerated directly from the typed compiler and its pinned resolved
-fixture; CI compares the committed bytes with the generator output. The
-[field-effect record](generated/verl-opd-v0.8-field-effects.json) independently
-mutates every executable non-informational field and records its observed plan
-or native-runtime effect.
-
-The older `single-gpu-online-distillation-v1` import/export contract remains
-available for migration and retains its historical non-launchable PPO/reward
-scaffold semantics.
-
-Compatibility Level 1 covers standard Hugging Face, PEFT, safetensors,
-tokenizer and Parquet artifacts. Level 2 is a fail-closed 14-field config
-whitelist. **miniVERL-defined compatibility Level 3** adds the generated
-bundle, reward scaffold, hashes and an exact-source smoke. Unknown,
-algorithm-changing or distributed-only verl fields are rejected instead of
-guessed.
-
-These miniVERL-defined levels describe artifact and configuration exchange.
-Optimizer state, distributed RNG, FSDP/Megatron native checkpoints, Ray runtime
-state and teacher-cache conversion sit beyond that exchange contract. The
-release smoke records artifact/config validation and distributed execution as
-separate evidence fields. Legacy reward-scaffold bundles use
-`launch.template.sh`; current profile exports remain explicit about their
-individual launch blockers and semantic-parity status.
-See the [current local runtime](verl-opd-runtime.md), [current scale-out
-contract](verl-opd-scaleout.md), and [legacy bridge](legacy-verl-bridge.md).
-
-## Versioned but extensible
-
-Run manifests are descriptive records rather than executable inputs. New
-v0.2.4 runs use manifest schema 3 for explicit config-layer provenance; report,
-inspect and export continue to read older v0.2 manifests by feature detection.
-Unknown fields are preserved where a schema allows them, while an unknown
-schema version in a training-critical artifact is rejected.
-
-Deprecations emit a warning in at least one minor release before removal and
-name the replacement. Security fixes may reject input that an older permissive
-parser accepted. Private modules and undocumented CLI output formatting are not
-compatibility promises.
-
-When exact resumption matters, use the package version recorded in the run and
-keep the full run directory. See [reproducibility](reproducibility.md) for the
-submitted/validated/runtime config layers.
+miniVERL versions experiment compilers separately from artifact schemas. A
+compiler profile names an exact upstream tag and commit, a closed field-rule
+set, the native objective implementation and its export contract. That identity
+travels through recipes, plans, trajectories, caches, checkpoints and reports.
+
+## Status vocabulary
+
+| Status | Meaning |
+| --- | --- |
+| exact | the same value and unit drive the same experiment meaning |
+| semantically conformant | the representation differs, while tested algorithm semantics agree |
+| locally lowered | physical execution changes without changing the objective |
+| informational only | retained for provenance and inactive for this selection |
+| distributed only | cluster placement or communication has no one-GPU counterpart |
+| not implemented | meaningful on one GPU but absent from the current runtime |
+| unsupported | a concrete field/value has no safe versioned rule |
+
+These states deliberately separate algorithm support from physical scale. A
+source config may request eight GPUs and two nodes while its GRPO experiment
+still lowers to one device; those resource values are reported as
+`distributed_only`, not treated as an unknown algorithm.
+
+## verl v0.9 RL compiler
+
+Profile `verl-rl-v0.9-single-gpu-v1` targets official verl `v0.9.0` at
+`483b8a009ba3a97563edee3a19887e4862b8094a`.
+
+| Surface | Status | Local contract |
+| --- | --- | --- |
+| GRPO | semantically conformant | sample-std group normalization, epsilon placement and masking match v0.9 |
+| Dr.GRPO | semantically conformant | GRPO centering without std normalization |
+| RLOO | semantically conformant | leave-one-out prompt-group baseline |
+| REINFORCE++ | semantically conformant | discounted token returns plus masked whitening |
+| vanilla dual-clipped policy objective | semantically conformant | token-mean reduction, clipping and diagnostics match v0.9 |
+| grouped `rollout.n` | exact | complete prompt groups and stable sample identities |
+| behavior log-probability | semantically conformant | current rollout actor, temperature-scaled, recomputed current policy at update |
+| Parquet data and token bounds | exact | source files, key, shuffle, seed and limits drive local loading |
+| exact-answer / target-length rewards | exact | reward-bearing Parquet metadata and fail-closed deterministic scorers |
+| environment/Python reward providers | supported local API | trusted injection with ordered input and implementation identity |
+| fixed reference reward KL | locally lowered | frozen adapter is scheduled on the compatible actor backbone |
+| actor logical mini-batch | semantically conformant | one logical mini-batch, physically microbatched on the GPU |
+| TP/PP/DP, nodes, resource pools | distributed only | original values retained; execution uses one process/device |
+| PPO/GAE runtime | not implemented | conformance-tested math exists; trainable critic lifecycle does not |
+| nonzero entropy objective / actor-loss KL | not implemented | metrics/reference reward KL exist, but these objective terms are not connected |
+
+The compiler accepts a resolved documented subset rather than arbitrary Hydra
+YAML. Missing reward or parameterization choices produce a non-executable
+template. Unknown fields, unresolved interpolation, NaN/infinity, unsupported
+estimators and objective-changing values are rejected before publication. Every
+accepted runnable recipe is validated with `RunConfig` and published
+transactionally.
+
+The [machine-readable v0.9 report](generated/verl-rl-v0.9-compatibility.json)
+is generated from the shipped example and checked byte-for-byte in CI. The
+[RL runtime guide](verl-rl-runtime.md) covers execution and resume.
+
+## verl v0.8 OPD profiles
+
+The OPD family targets official verl `v0.8.0` at
+`7aed6b230776f963fa09509c10d9c3a767d1102c`:
+
+| Profile family | Objective | Status |
+| --- | --- | --- |
+| direct GKD | `forward_kl_topk`, token mean | measured local runtime |
+| sampled-k1 | sampled `k1` plus vanilla policy loss | measured local runtime |
+| grouped variants | independent `n > 1` samples | conformance only |
+| rewarded sampled-k1 | exact-answer advantage plus teacher advantage | conformance only |
+
+The [generated OPD matrix](generated/verl-opd-v0.8-compatibility.json) and
+[field-effect record](generated/verl-opd-v0.8-field-effects.json) bind compiler
+rules to observed native plan effects. Existing profile identities and
+published evidence retain their historical meaning.
+
+## Artifact and scale-out bridge
+
+OPD export produces standard PEFT, safetensors, tokenizer, Parquet, config and
+provenance artifacts. The report keeps these states independent:
+
+- artifact bundle complete;
+- upstream config parse passed;
+- model/data load smoke passed;
+- reward implementation complete;
+- launchable;
+- distributed execution tested;
+- algorithm semantic parity.
+
+Where the legacy label remains, **miniVERL-defined compatibility Level 3**
+means the generated bundle and exact-source smoke, not a distributed training
+claim. Read the [scale-out contract](verl-opd-scaleout.md) and
+[legacy bridge](legacy-verl-bridge.md).
+
+## Stable surfaces
+
+- Package `miniverl`, CLI `miniverl`, documented command names, `RunConfig` and
+ the `OPDTrainer` compatibility facade.
+- Existing run artifact names and historical trajectory/cache/checkpoint schema
+ readers, including the constrained legacy cache reader.
+- Historical protocol-v1 and verifier-v1 behavior; new protocol versions do
+ not rewrite frozen evidence.
+- Existing compiler profile identities and upstream pins.
+
+Minor releases may add a new versioned profile or tighten unsafe input
+validation. Deprecations warn for at least one minor release. For exact resume,
+retain the package version and complete run directory; see
+[reproducibility](reproducibility.md).
diff --git a/docs/design.md b/docs/design.md
index 75c1a74..71eaaa6 100644
--- a/docs/design.md
+++ b/docs/design.md
@@ -1,8 +1,9 @@
# miniVERL design
-This document explains miniVERL's layers, one on-policy distillation (OPD)
-cycle, and the invariants checked at each boundary. Module, function and field
-names follow the source tree; measured numbers link to their result artifacts.
+This document explains miniVERL's layers, critic-free RL and on-policy
+distillation (OPD) cycles, and the invariants checked at each boundary. Module,
+function and field names follow the source tree; measured numbers link to their
+result artifacts.
For the mathematics of the objectives see [math.md](math.md).
@@ -10,6 +11,12 @@ For the mathematics of the objectives see [math.md](math.md).
## 1. The problem
+verl expresses useful experiment semantics together with a distributed runtime.
+miniVERL preserves a tested subset of the experiment—data, rollout groups,
+rewards or teacher targets, advantage/objective and schedule—while lowering its
+logical roles onto one GPU. This makes a cluster-shaped experiment easier to
+prototype, inspect and resume locally.
+
On-policy distillation trains a small student on **its own** rollouts, scored by
a larger teacher at exactly the states the student visited. Compared with
supervised fine-tuning on teacher text, it removes the train/inference
@@ -24,7 +31,7 @@ that the model must condition on but must never be trained to reproduce. Getting
that wrong does not crash anything — it silently trains the policy to hallucinate
tool results.
-miniVERL is a single-GPU implementation of that loop with three properties it
+The OPD path is a single-GPU implementation of that loop with three properties it
tries to make checkable rather than aspirational:
1. **Token provenance is a validated data structure, not a convention.** Every
@@ -37,11 +44,10 @@ tries to make checkable rather than aspirational:
out-of-memory retry ever changes is the projection chunk size, which does not
change the loss or the gradient.
-### 1.1 What this is not competing with
+### 1.1 Position in the ecosystem
-miniVERL is a small, readable, single-GPU lab. It is not a replacement for the
-production frameworks, and the following statements about them are accurate as
-of 2026-07:
+miniVERL is a small, readable, single-GPU lab. Production frameworks cover
+larger execution surfaces; the following comparison was checked in 2026-07:
- **verl** (`verl-project/verl`, Apache-2.0) already has first-class on-policy
distillation in core (`verl/trainer/distillation/`, config namespace
@@ -621,40 +627,27 @@ quietly did no work.
---
-## 7. Deliberately not here
-
-Every omission below is a scope decision, not an oversight. Each one is a thing a
-production framework does that a single-process one-GPU lab should not pretend to
-do.
-
-**Ray.** verl depends on `ray[default]` unconditionally because it schedules
-heterogeneous actor pools across nodes. miniVERL runs one process and one GPU,
-so a cluster scheduler would add a large dependency, a second failure mode and a
-second mental model in exchange for nothing. Concurrency here is a `for` loop.
-
-**FSDP / FSDP2 / Megatron-LM.** Sharding exists to train a model that does not
-fit on one device. The target configuration — a 0.6B NF4-QLoRA student plus a
-bf16 1.7B teacher — fits: the one-cycle GPU smoke test on the RTX 4080 measured
-peak CUDA allocated 4.251 GiB and peak reserved 4.762 GiB. Adding a sharding
-runtime would make the memory accounting in this document unverifiable by a
-reader with one GPU.
-
-**vLLM / SGLang for rollouts.** A dedicated inference server is the right answer
-when rollout throughput dominates and you are batching hundreds of sequences. At
-this scale it does not: the decode-throughput probe on the RTX 4080 measured a
-14-token prefill at 37.0 ms against a cached single-token step at 30.9 ms, which
-means single-sequence decoding here is kernel-launch bound rather than compute
-bound. A separate server process would add weight-sync complexity and a second
-sampling implementation without changing that. `HFBackend.generate` and
-`ToyBackend.generate` share one loop (`models/sampling.py::run_generation`) so
-stop-string handling, seeding and token accounting are implemented once.
-
-**GRPO / PPO / any RL algorithm.** miniVERL has no advantage estimator, no value
-head, no reference-policy KL penalty and no reward model. The objective is a
-token-level divergence against a teacher, full stop. Mixing an RL objective in
-would make it impossible to attribute a result to the distillation term. The
-environments do return an exact `reward` in `VerificationRecord`, but nothing
-optimizes it — it is used for reporting and for evaluation success rates only.
+## 7. Runtime scope
+
+**Critic-free RL.** `algorithms` contains pure GRPO, Dr.GRPO, RLOO,
+REINFORCE++, reference-KL and clipped-policy primitives pinned to official verl
+v0.9. `rewards` owns deterministic scalar/batch provider contracts and trusted
+composition. The trainer publishes complete prompt groups, attaches rewards
+and token-aligned advantages, then updates the current actor.
+
+**PPO critic.** GAE and clipped value-loss mathematics are conformance-tested,
+but no executable PPO plan is published. A trainable value role still needs its
+own model/head, optimizer, checkpoint identity and temporal phase ownership.
+
+**Distributed placement.** Ray, FSDP/FSDP2, Megatron-LM and parallel degrees
+above one remain upstream scale-out machinery. miniVERL compiles safe experiment
+fields and records those placement values as distributed-only while running one
+process and one device.
+
+**Rollout backends.** `hf_reference` is the compatibility path; `hf_cached`
+adds grouped prefill, per-row RNG and KV-cache decode. Direct-GKD OPD also has a
+measured managed-vLLM path on Linux/WSL2. The RL policy-loss path uses HF
+behavior log-probabilities and has no asynchronous actor pool.
**Cross-tokenizer distillation.** `build_tokenizer` loads the teacher tokenizer
when its id differs and compares behavioural fingerprints, raising
@@ -671,8 +664,8 @@ transcript codec, in the alignment map and in the cache. None of that exists.
but rollout decoding remains one sequence at a time. There is no continuous
batching scheduler or external inference engine.
-**Telemetry.** There is none. `utils/logging.py` writes to the console and to
-`events.jsonl` and nowhere else.
+**Telemetry.** Structured JSON/JSONL is the primary observability surface.
+There is no required external tracking service.
---
diff --git a/docs/for-verl-users.md b/docs/for-verl-users.md
index f1040b2..1114e9b 100644
--- a/docs/for-verl-users.md
+++ b/docs/for-verl-users.md
@@ -1,114 +1,94 @@
# For verl users
-miniVERL brings five closed verl v0.8 OPD profiles into a local, inspectable
-workflow; two have measured systems records and three are conformance-only.
-Familiar field names and Parquet data flow through a typed compiler,
-which turns resource intent into sequential phases on one CUDA GPU and records
-every local mapping in the resulting plan.
-
-Inspect the closed, versioned registry with `miniverl profiles list`; use
-`miniverl compat explain` before assuming that an accepted field is effective.
-The [profile registry guide](profiles/index.md) documents the identity carried
-by plans, caches, checkpoints and exports.
+miniVERL turns supported experiment semantics from a resolved verl-shaped
+config into a validated one-GPU plan. Field names, datasets, logical batches,
+algorithms and artifacts remain recognizable; cluster placement is replaced by
+sequential local phases and recorded as a lowering decision.
-
-## Supported input shape
+## Start with the v0.9 RL profile
+
+```bash
+miniverl data sample --task-rewards --rows 8 --out data/rl-prompts.parquet
+miniverl import-verl --profile verl-rl-v0.9-single-gpu-v1 \
+ --config examples/verl-rl-v0.9-single-gpu.yaml --out local-grpo.yaml
+miniverl validate local-grpo.yaml --json
+miniverl train local-grpo.yaml --dry-run
+```
-- A resolved YAML using the `verl-opd-v0.8-single-gpu-v1` field subset.
-- Verl-style Parquet prompts with structured chat messages; the rewarded
- profile additionally requires bound deterministic exact-answer metadata.
-- One actor, one teacher, token-mean aggregation, and either direct
- forward-top-k GKD or sampled-k1 vanilla policy loss. The measured profiles
- remain `n=1`; their grouped counterparts accept independent `n>1` samples.
-- Immutable Hugging Face revisions, PEFT adapters and tokenizer snapshots.
-- Familiar fields such as `actor_rollout_ref.model.path`,
- `distillation.teacher_models.teacher_model.model_path`, response bounds,
- learning rate and LoRA configuration.
+This profile pins official verl `v0.9.0` at
+`483b8a009ba3a97563edee3a19887e4862b8094a`. It accepts a resolved documented
+subset for GRPO, Dr.GRPO, RLOO or REINFORCE++, then writes both a native recipe
+and `*.import-report.json`. Scientific-notation strings such as `1e-5` are
+accepted when finite; `${...}`, NaN, infinity and unknown fields fail before a
+runnable recipe is published.
-The profile compiler accepts a resolved, documented subset and classifies every
-field before execution. [Compatibility profiles](profiles/index.md) contains
-the field contract; [limitations](limitations.md) collects the algorithms and
-distributed runtime surfaces covered by other tools.
+The [single-GPU RL guide](verl-rl-runtime.md) documents every algorithm and
+lowering. The committed [example report](generated/verl-rl-v0.9-compatibility.json)
+shows the exact machine-readable output.
## Command mapping
| verl action | miniVERL action |
| --- | --- |
-| compose or capture a resolved config | provide resolved YAML to `--config` |
-| add a Hydra-style override | repeat `--set`, use `--overrides-file`, or place tokens after `--` |
-| inspect resolved intent | `miniverl plan --json` |
-| launch OPD | `miniverl run` |
-| read prompt Parquet | use the file directly |
-| allocate resource pools | compile to sequential local phases |
-| hand artifacts back | `miniverl export-verl` |
+| capture a resolved Hydra config | provide it to `import-verl --config` |
+| inspect field semantics | read `*.import-report.json` |
+| validate the local plan | `miniverl validate local.yaml --json` |
+| run actor/reward/reference phases | `miniverl train local.yaml` |
+| read prompt Parquet | retain `data.train_files` and `data.prompt_key` directly |
+| lower resource pools | record one process/device plus original source intent |
+| inspect trajectories | `miniverl inspect runs//trajectories.jsonl` |
+
+## What maps into the RL runtime
+
+- `data.train_files`, `val_files`, `prompt_key`, prompt/response limits,
+ shuffle and seed drive the local Parquet source.
+- `data.train_batch_size × rollout.n` is the logical trajectory count per
+ rollout iteration.
+- `actor.ppo_mini_batch_size` controls how that logical batch is divided into
+ actor updates; physical trajectory batching remains a separate `miniverl`
+ execution control.
+- Actor model, revision, LoRA, optimizer, sampling, clipping and schedule
+ fields feed the native recipe with their source units recorded.
+- `trainer.total_training_steps` is the rollout-iteration cap. Epoch-only
+ scheduling is not guessed because it depends on dataset traversal semantics.
+- Task rewards are explicit. The portable compiler accepts the built-in
+ exact-answer and target-length providers; trusted Python and environment
+ providers are injected through the local API.
+- Fixed reference KL requires an explicit frozen reference adapter. The
+ compiler never creates an unqualified same-base reference policy.
+
+Distributed resource counts can be larger than one in the source config. They
+are classified `distributed_only`, preserved in the report, and lowered to one
+local process/device. Unknown algorithm or objective fields remain rejected.
+
+## Existing v0.8 OPD profiles
+
+The OPD compiler targets official verl `v0.8.0` at
+`7aed6b230776f963fa09509c10d9c3a767d1102c`. It supports direct forward-top-k
+GKD and sampled-k1 policy-gradient distillation, including measured profiles
+and grouped/rewarded conformance variants.
```bash
miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
--config verl-opd.yaml \
--set 'data.train_files=["data/train.parquet"]' \
- --set actor_rollout_ref.actor.optim.lr=1e-5
-```
-
-Planning is weight-free and offline. Use `--json` to retain the complete field
-matrix. The CLI records repeated `--set`, override files and
-trailing tokens with deterministic precedence. Resolve `${...}` composition in
-the trusted source workflow first; see [Config overrides](config-overrides.md).
-
-For a serious run, add `--accept-local-reinterpretations --out plan.json`,
-inspect the immutable artifact, then execute `miniverl run --plan plan.json`.
-This binds the source YAML, overrides and scanned Parquet bytes to the exact
-native config; see [Immutable execution plans](immutable-plans.md).
-
-## Same fields, different placement
-
-An upstream-shaped fragment can stay recognizable:
-
-```yaml
-actor_rollout_ref:
- model: {path: Qwen/Qwen3-0.6B}
- rollout: {name: vllm, n: 1}
-distillation:
- teacher_models:
- teacher_model:
- model_path: Qwen/Qwen3-1.7B
- inference: {name: vllm}
+ --set actor_rollout_ref.actor.optim.lr=1e-5 \
+ --accept-local-reinterpretations --out plan.json
+miniverl run --profile verl-opd-v0.8-single-gpu-v1 --plan plan.json --dry-run
```
-The `vllm` values remain visible as source intent. The compiler classifies them
-as local reinterpretations, runs Hugging Face generation and teacher scoring
-sequentially, and preserves the source value, local meaning and risk in the
-compatibility report.
+`plan.json` binds the source YAML, ordered overrides, scanned Parquet bytes,
+profile identity and native config. Use [immutable plans](immutable-plans.md)
+and the [OPD quickstart](opd-quickstart.md) for that workflow.
-## Data mapping
+## Artifact handoff
-`data.train_files`, `data.val_files`, `data.prompt_key`, prompt and response
-bounds, shuffle and seed feed the native Parquet source directly. Each prompt
-row preserves its structured messages and source metadata. Run
-`miniverl data sample --out prompts.parquet` for a valid small file, or use
-`miniverl convert-dataset` when crossing the native trajectory boundary.
-
-Parquet paths are direct execution inputs, so a missing file produces an
-actionable error when execution begins. Dataset bytes and schema are copied
-into export provenance.
-
-## Actor, teacher and reference roles
-
-The actor is the trainable PEFT policy. In the direct-GKD profile the teacher
-produces top-k token IDs and log-probabilities on actor-generated tokens;
-top-k mass and overlap are diagnostics, not an explicit tail bucket. In the PG
-k1 profile the target is only the sampled token's teacher log-probability,
-bound to the rollout actor log-probability and policy version; current actor
-log-probability is recomputed at update time. The rewarded PG profile adds a
-separately logged task advantage from the closed exact-answer provider; it has
-no critic, value model or GRPO semantics. Local runtime strategies
-may keep quantized roles resident, swap movable unquantized roles, or share a
-compatible backbone while preserving distinct role identities in provenance.
-
-## Export boundary
+Completed OPD runs can produce a pinned scale-out bundle:
```bash
miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout
@@ -116,35 +96,17 @@ miniverl bridge materialize scaleout --download --offline
miniverl bridge doctor scaleout --json
```
-The export bundle carries PEFT, Parquet, config and provenance artifacts,
-but is reported as `launchable: false` until exact base snapshots are materialized and
-the pinned upstream checks pass. Read the [materialization workflow](scaleout-materialization.md).
-Upstream parse/load smoke, artifact completeness, launchability and distributed
-execution are separate statuses, so the report shows exactly how far a bundle
-has progressed.
-
-## Common errors
-
-- **Unknown field:** capture a resolved config and remove fields outside the
- documented profile; inspect-only compilation can still explain known
- unsupported values.
-- **Algorithm field rejected:** select the explicit PG-k1 profile for its
- narrow sampled policy-loss path, or a grouped profile for Parquet `n>1`.
- Arbitrary reward code, critics, value models, KL penalties, multiple teachers
- and distributed counts remain unsupported.
-- **Interpolation rejected:** resolve Hydra/OmegaConf in your trusted verl
- environment first. miniVERL will not execute `${...}`.
-- **High-risk reinterpretation not accepted:** inspect `miniverl plan`, then
- pass `--accept-local-reinterpretations` for an external config. Packaged
- profiles carry reviewed acceptance metadata.
-- **Prompt schema mismatch:** validate the Parquet `prompt` column as structured
- role/content messages, or convert it explicitly.
-- **Tokenizer mismatch:** actor and teacher scoring require structural identity
- for the shared token space; a legacy behavioral fingerprint is not proof.
-- **CUDA out of memory:** reduce context, response length or physical batches;
- keep logical update semantics unchanged. See [single-GPU planning](single-gpu-guide.md).
-- **Bundle not launchable:** materialize exact snapshots and install the pinned
- verl commit; inspect the reported blocker. Read [scale-out materialization](scaleout-materialization.md).
-
-Next: follow the [OPD quickstart](opd-quickstart.md), inspect the
-[compatibility policy](compatibility.md), or review [all limitations](limitations.md).
+The report separates artifact completeness, upstream config parse, model/data
+load smoke, reward implementation, launchability, distributed execution and
+algorithm parity. Read the [scale-out contract](verl-opd-scaleout.md).
+
+## Practical diagnostics
+
+- Use `miniverl doctor` to inspect the CUDA stack.
+- Resolve Hydra interpolation in the trusted upstream environment before
+ importing.
+- Keep immutable model and tokenizer revisions in portable configs.
+- Reduce context or physical trajectory batch size on OOM; logical group and
+ objective semantics remain unchanged.
+- Read [compatibility](compatibility.md) for field status and
+ [limitations](limitations.md) for the consolidated boundaries.
diff --git a/docs/generated/quality.json b/docs/generated/quality.json
index 9ce72f3..ff4213b 100644
--- a/docs/generated/quality.json
+++ b/docs/generated/quality.json
@@ -1,22 +1,22 @@
{
"schema_version": 2,
- "release": "0.11.0",
- "status": "released",
- "quality_floor": "2,000+ tests and 80%+ branch coverage at v0.11.0",
+ "release": "0.12.0",
+ "status": "candidate",
+ "quality_floor": "2,000+ tests and 80%+ branch coverage at v0.12.0",
"local_validation": {
- "scope": "the maintainer's workstation, including the Windows CUDA and network paths",
- "commit": "e9271794ae0aa4cc6adfd3c2bf125864c6f08b78",
- "commit_relationship": "PR 99 head; merge commit b1befb5 carries the same tree and passed the complete hosted CI, build, docs and pinned-bridge matrix",
- "measured_at": "2026-08-29T23:30:00Z",
+ "scope": "the maintainer's workstation, including Windows CUDA and network paths plus the exact official verl v0.9 checkout",
+ "commit": "feed761514c3925dd2290d855baac6d38c417e50",
+ "commit_relationship": "implementation commit; the following release commit changes versioned release metadata and this candidate record only",
+ "measured_at": "2026-09-08T04:20:14Z",
"platform": "Windows 11",
"python": "CPython 3.10 for coverage, local CUDA tests and network tests",
"coverage_mode": "branch",
"cpu_non_gpu_non_network": {
- "passed": 2500,
+ "passed": 2566,
"skipped": 10,
"deselected": 24,
"branch_coverage_percent": 83.0,
- "skip_reason": "ten Windows platform or privilege skips; package and pinned-verl gates ran separately"
+ "skip_reason": "ten Windows platform or privilege skips; pinned verl v0.9 conformance executed from the exact checkout"
},
"gpu": {
"passed": 10,
@@ -27,26 +27,26 @@
}
},
"release_validation": {
- "scope": "the exact published v0.11.0 release commit",
- "commit": "479695c4980150bc6f7a51bddec2b90c08c8bb98",
+ "scope": "the exact v0.12.0 release commit",
+ "commit": "pending",
"workflows": {
- "ci": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/33288026959",
- "build": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/33288026935",
- "docs": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/33290669271",
- "pinned_verl_bridge": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/33290669272",
- "gpu_full_qualification": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/33288192471",
- "release_dry_run": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/33290350620",
- "release": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/33290669280"
+ "ci": "pending",
+ "build": "pending",
+ "docs": "pending",
+ "pinned_verl_bridge": "pending",
+ "gpu_full_qualification": "pending",
+ "release_dry_run": "pending",
+ "release": "pending"
},
- "conclusion": "passed",
- "gpu_coverage": "exact-SHA attempt-1 full qualification passed on the ephemeral WSL2 RTX 4080 runner; runtime correctness only",
+ "conclusion": "pending",
+ "gpu_coverage": "pending exact-SHA attempt-1 full qualification on the WSL2 RTX 4080 runner",
"publication": {
- "pypi": "https://pypi.org/project/miniverl/0.11.0/",
- "github_release": "https://github.com/DaoyuanLi2816/mini-verl/releases/tag/v0.11.0",
- "documentation": "https://daoyuanli2816.github.io/mini-verl/",
- "immutable_documentation_build": "https://github.com/DaoyuanLi2816/mini-verl/actions/runs/33290669271",
- "wheel_sha256": "d2fb981987b34bc53dfdefa505ded96d019614c69c4d0509360a414e8ab69aca",
- "sdist_sha256": "5ee1de64b2eaf82d7076f11add1f897d99668b245ba3635a8f9bb7260f970b13"
+ "pypi": "pending",
+ "github_release": "pending",
+ "documentation": "pending",
+ "immutable_documentation_build": "pending",
+ "wheel_sha256": "pending",
+ "sdist_sha256": "pending"
}
}
}
diff --git a/docs/generated/verl-rl-v0.9-compatibility.json b/docs/generated/verl-rl-v0.9-compatibility.json
new file mode 100644
index 0000000..4d389d2
--- /dev/null
+++ b/docs/generated/verl-rl-v0.9-compatibility.json
@@ -0,0 +1,510 @@
+{
+ "algorithm": "grpo",
+ "executable": true,
+ "field_classification": [
+ {
+ "classification": "exact",
+ "local_target": "loss.clip_ratio",
+ "reason": "same vanilla policy clip",
+ "resolution_status": "resolved",
+ "source_value": 0.2,
+ "upstream_field": "actor_rollout_ref.actor.clip_ratio"
+ },
+ {
+ "classification": "exact",
+ "local_target": "loss.clip_ratio_c",
+ "reason": "same dual-clip coefficient",
+ "resolution_status": "resolved",
+ "source_value": 3.0,
+ "upstream_field": "actor_rollout_ref.actor.clip_ratio_c"
+ },
+ {
+ "classification": "exact",
+ "local_target": "loss.clip_ratio_high",
+ "reason": "same upper clip",
+ "resolution_status": "resolved",
+ "source_value": 0.2,
+ "upstream_field": "actor_rollout_ref.actor.clip_ratio_high"
+ },
+ {
+ "classification": "exact",
+ "local_target": "loss.clip_ratio_low",
+ "reason": "same lower clip",
+ "resolution_status": "resolved",
+ "source_value": 0.2,
+ "upstream_field": "actor_rollout_ref.actor.clip_ratio_low"
+ },
+ {
+ "classification": "exact",
+ "local_target": null,
+ "reason": "entropy regularization is disabled in both runtimes",
+ "resolution_status": "resolved",
+ "source_value": 0.0,
+ "upstream_field": "actor_rollout_ref.actor.entropy_coeff"
+ },
+ {
+ "classification": "exact",
+ "local_target": "loss.aggregation",
+ "reason": "token-mean aggregation",
+ "resolution_status": "resolved",
+ "source_value": "token-mean",
+ "upstream_field": "actor_rollout_ref.actor.loss_agg_mode"
+ },
+ {
+ "classification": "exact",
+ "local_target": "train.learning_rate",
+ "reason": "same optimizer learning rate",
+ "resolution_status": "resolved",
+ "source_value": 1e-05,
+ "upstream_field": "actor_rollout_ref.actor.optim.lr"
+ },
+ {
+ "classification": "exact",
+ "local_target": "train.warmup_steps",
+ "reason": "same optimizer-step count",
+ "resolution_status": "resolved",
+ "source_value": 0,
+ "upstream_field": "actor_rollout_ref.actor.optim.lr_warmup_steps"
+ },
+ {
+ "classification": "exact",
+ "local_target": "train.weight_decay",
+ "reason": "same optimizer coefficient",
+ "resolution_status": "resolved",
+ "source_value": 0.01,
+ "upstream_field": "actor_rollout_ref.actor.optim.weight_decay"
+ },
+ {
+ "classification": "exact",
+ "local_target": "train.opd_freshness",
+ "reason": "one fresh-policy update pass is supported",
+ "resolution_status": "resolved",
+ "source_value": 1,
+ "upstream_field": "actor_rollout_ref.actor.ppo_epochs"
+ },
+ {
+ "classification": "semantically_conformant",
+ "local_target": "train.gradient_accumulation_steps",
+ "reason": "one logical upstream mini-batch is physically microbatched on one GPU",
+ "resolution_status": "resolved",
+ "source_value": 8,
+ "upstream_field": "actor_rollout_ref.actor.ppo_mini_batch_size"
+ },
+ {
+ "classification": "exact",
+ "local_target": null,
+ "reason": "actor-loss KL is disabled in both runtimes",
+ "resolution_status": "resolved",
+ "source_value": false,
+ "upstream_field": "actor_rollout_ref.actor.use_kl_loss"
+ },
+ {
+ "classification": "semantically_conformant",
+ "local_target": "models.student.gradient_checkpointing",
+ "reason": "same activation-memory technique",
+ "resolution_status": "resolved",
+ "source_value": true,
+ "upstream_field": "actor_rollout_ref.model.enable_gradient_checkpointing"
+ },
+ {
+ "classification": "semantically_conformant",
+ "local_target": "models.student.lora.alpha",
+ "reason": "same PEFT scale when enabled",
+ "resolution_status": "resolved",
+ "source_value": 32,
+ "upstream_field": "actor_rollout_ref.model.lora_alpha"
+ },
+ {
+ "classification": "semantically_conformant",
+ "local_target": "models.student.lora.r",
+ "reason": "same PEFT rank when enabled",
+ "resolution_status": "resolved",
+ "source_value": 16,
+ "upstream_field": "actor_rollout_ref.model.lora_rank"
+ },
+ {
+ "classification": "exact",
+ "local_target": "models.student.model_id",
+ "reason": "same actor checkpoint identity",
+ "resolution_status": "resolved",
+ "source_value": "Qwen/Qwen3-0.6B",
+ "upstream_field": "actor_rollout_ref.model.path"
+ },
+ {
+ "classification": "semantically_conformant",
+ "local_target": "models.student.lora.target_modules",
+ "reason": "same PEFT target names when enabled",
+ "resolution_status": "resolved",
+ "source_value": [
+ "q_proj",
+ "k_proj",
+ "v_proj",
+ "o_proj",
+ "gate_proj",
+ "up_proj",
+ "down_proj"
+ ],
+ "upstream_field": "actor_rollout_ref.model.target_modules"
+ },
+ {
+ "classification": "locally_lowered",
+ "local_target": "memory planner",
+ "reason": "recorded as an upstream engine hint; local placement is planned separately",
+ "resolution_status": "resolved",
+ "source_value": 0.6,
+ "upstream_field": "actor_rollout_ref.rollout.gpu_memory_utilization"
+ },
+ {
+ "classification": "exact",
+ "local_target": "rollout.samples_per_prompt",
+ "reason": "same grouped sample count",
+ "resolution_status": "resolved",
+ "source_value": 4,
+ "upstream_field": "actor_rollout_ref.rollout.n"
+ },
+ {
+ "classification": "locally_lowered",
+ "local_target": "rollout.backend",
+ "reason": "distributed engine selection becomes a validated local HF backend",
+ "resolution_status": "resolved",
+ "source_value": "vllm",
+ "upstream_field": "actor_rollout_ref.rollout.name"
+ },
+ {
+ "classification": "exact",
+ "local_target": "rollout.temperature",
+ "reason": "same behavior-policy temperature",
+ "resolution_status": "resolved",
+ "source_value": 1.0,
+ "upstream_field": "actor_rollout_ref.rollout.temperature"
+ },
+ {
+ "classification": "distributed_only",
+ "local_target": null,
+ "reason": "tensor parallel placement is not part of one-GPU semantics",
+ "resolution_status": "resolved",
+ "source_value": 8,
+ "upstream_field": "actor_rollout_ref.rollout.tensor_model_parallel_size"
+ },
+ {
+ "classification": "exact",
+ "local_target": "rollout.top_k",
+ "reason": "same top-k bound",
+ "resolution_status": "resolved",
+ "source_value": 0,
+ "upstream_field": "actor_rollout_ref.rollout.top_k"
+ },
+ {
+ "classification": "exact",
+ "local_target": "rollout.top_p",
+ "reason": "same nucleus threshold",
+ "resolution_status": "resolved",
+ "source_value": 1.0,
+ "upstream_field": "actor_rollout_ref.rollout.top_p"
+ },
+ {
+ "classification": "semantically_conformant",
+ "local_target": "algorithm.name",
+ "reason": "pinned estimator implementation",
+ "resolution_status": "resolved",
+ "source_value": "grpo",
+ "upstream_field": "algorithm.adv_estimator"
+ },
+ {
+ "classification": "informational_only",
+ "local_target": null,
+ "reason": "not consumed by the grpo estimator",
+ "resolution_status": "resolved",
+ "source_value": 1.0,
+ "upstream_field": "algorithm.gamma"
+ },
+ {
+ "classification": "informational_only",
+ "local_target": null,
+ "reason": "not consumed by the grpo estimator",
+ "resolution_status": "resolved",
+ "source_value": 1.0,
+ "upstream_field": "algorithm.lam"
+ },
+ {
+ "classification": "semantically_conformant",
+ "local_target": "algorithm.name",
+ "reason": "false selects Dr.GRPO centering",
+ "resolution_status": "resolved",
+ "source_value": true,
+ "upstream_field": "algorithm.norm_adv_by_std_in_grpo"
+ },
+ {
+ "classification": "exact",
+ "local_target": "algorithm.kl_coef",
+ "reason": "reference-policy reward KL is disabled in both runtimes",
+ "resolution_status": "resolved",
+ "source_value": false,
+ "upstream_field": "algorithm.use_kl_in_reward"
+ },
+ {
+ "classification": "exact",
+ "local_target": "source.max_prompt_length",
+ "reason": "same token bound",
+ "resolution_status": "resolved",
+ "source_value": 128,
+ "upstream_field": "data.max_prompt_length"
+ },
+ {
+ "classification": "exact",
+ "local_target": "source.max_response_length",
+ "reason": "same token bound",
+ "resolution_status": "resolved",
+ "source_value": 24,
+ "upstream_field": "data.max_response_length"
+ },
+ {
+ "classification": "exact",
+ "local_target": "source.prompt_key",
+ "reason": "same Parquet column",
+ "resolution_status": "resolved",
+ "source_value": "prompt",
+ "upstream_field": "data.prompt_key"
+ },
+ {
+ "classification": "exact",
+ "local_target": "source.seed",
+ "reason": "same seed",
+ "resolution_status": "resolved",
+ "source_value": 17,
+ "upstream_field": "data.seed"
+ },
+ {
+ "classification": "semantically_conformant",
+ "local_target": "source.shuffle",
+ "reason": "same data policy",
+ "resolution_status": "resolved",
+ "source_value": true,
+ "upstream_field": "data.shuffle"
+ },
+ {
+ "classification": "semantically_conformant",
+ "local_target": "train.rollouts_per_cycle",
+ "reason": "same logical prompts per rollout",
+ "resolution_status": "resolved",
+ "source_value": 2,
+ "upstream_field": "data.train_batch_size"
+ },
+ {
+ "classification": "exact",
+ "local_target": "source.train_files",
+ "reason": "same Parquet inputs",
+ "resolution_status": "resolved",
+ "source_value": [
+ "data/rl-prompts.parquet"
+ ],
+ "upstream_field": "data.train_files"
+ },
+ {
+ "classification": "semantically_conformant",
+ "local_target": "source.truncation",
+ "reason": "same policy",
+ "resolution_status": "resolved",
+ "source_value": "error",
+ "upstream_field": "data.truncation"
+ },
+ {
+ "classification": "exact",
+ "local_target": "source.val_files",
+ "reason": "same Parquet inputs",
+ "resolution_status": "resolved",
+ "source_value": [],
+ "upstream_field": "data.val_files"
+ },
+ {
+ "classification": "exact",
+ "local_target": "models.student.dtype",
+ "reason": "local runtime choice",
+ "resolution_status": "resolved",
+ "source_value": "auto",
+ "upstream_field": "miniverl.actor.dtype"
+ },
+ {
+ "classification": "exact",
+ "local_target": "models.student.lora.enabled",
+ "reason": "explicit local parameterization choice",
+ "resolution_status": "resolved",
+ "source_value": true,
+ "upstream_field": "miniverl.actor.lora_enabled"
+ },
+ {
+ "classification": "exact",
+ "local_target": "models.student.quantization",
+ "reason": "local runtime choice",
+ "resolution_status": "resolved",
+ "source_value": "nf4",
+ "upstream_field": "miniverl.actor.quantization"
+ },
+ {
+ "classification": "exact",
+ "local_target": "models.student.revision",
+ "reason": "immutable actor snapshot revision",
+ "resolution_status": "resolved",
+ "source_value": "c1899de289a04d12100db370d81485cdf75e47ca",
+ "upstream_field": "miniverl.actor.revision"
+ },
+ {
+ "classification": "exact",
+ "local_target": "models.student.tokenizer_revision",
+ "reason": "immutable tokenizer snapshot revision",
+ "resolution_status": "resolved",
+ "source_value": "c1899de289a04d12100db370d81485cdf75e47ca",
+ "upstream_field": "miniverl.actor.tokenizer_revision"
+ },
+ {
+ "classification": "exact",
+ "local_target": "memory.strategy",
+ "reason": "local placement choice",
+ "resolution_status": "resolved",
+ "source_value": "resident",
+ "upstream_field": "miniverl.memory.strategy"
+ },
+ {
+ "classification": "exact",
+ "local_target": "reward.provider",
+ "reason": "explicit trusted local reward implementation",
+ "resolution_status": "resolved",
+ "source_value": "exact_answer",
+ "upstream_field": "miniverl.reward.provider"
+ },
+ {
+ "classification": "exact",
+ "local_target": "rollout.backend",
+ "reason": "local runtime choice",
+ "resolution_status": "resolved",
+ "source_value": "hf_cached",
+ "upstream_field": "miniverl.rollout.backend"
+ },
+ {
+ "classification": "exact",
+ "local_target": "train.trajectory_batch_size",
+ "reason": "physical-only one-GPU batch size",
+ "resolution_status": "resolved",
+ "source_value": 1,
+ "upstream_field": "miniverl.update.trajectory_batch_size"
+ },
+ {
+ "classification": "exact",
+ "local_target": "run.name",
+ "reason": "same run label",
+ "resolution_status": "resolved",
+ "source_value": "qwen3-grpo-local",
+ "upstream_field": "trainer.experiment_name"
+ },
+ {
+ "classification": "informational_only",
+ "local_target": null,
+ "reason": "external logging is not activated",
+ "resolution_status": "resolved",
+ "source_value": [
+ "console"
+ ],
+ "upstream_field": "trainer.logger"
+ },
+ {
+ "classification": "distributed_only",
+ "local_target": null,
+ "reason": "resource placement lowers to one local CUDA device",
+ "resolution_status": "resolved",
+ "source_value": 8,
+ "upstream_field": "trainer.n_gpus_per_node"
+ },
+ {
+ "classification": "distributed_only",
+ "local_target": null,
+ "reason": "resource placement lowers to one local process",
+ "resolution_status": "resolved",
+ "source_value": 2,
+ "upstream_field": "trainer.nnodes"
+ },
+ {
+ "classification": "semantically_conformant",
+ "local_target": "run.tags",
+ "reason": "same provenance label",
+ "resolution_status": "resolved",
+ "source_value": "mini-verl",
+ "upstream_field": "trainer.project_name"
+ },
+ {
+ "classification": "exact",
+ "local_target": "train.save_every_cycles",
+ "reason": "same rollout-iteration interval",
+ "resolution_status": "resolved",
+ "source_value": 1,
+ "upstream_field": "trainer.save_freq"
+ },
+ {
+ "classification": "exact",
+ "local_target": "train.eval_every_cycles",
+ "reason": "same rollout-iteration interval",
+ "resolution_status": "resolved",
+ "source_value": -1,
+ "upstream_field": "trainer.test_freq"
+ },
+ {
+ "classification": "informational_only",
+ "local_target": null,
+ "reason": "recorded but not used when total_training_steps supplies the exact optimizer-step cap",
+ "resolution_status": "resolved",
+ "source_value": 1,
+ "upstream_field": "trainer.total_epochs"
+ },
+ {
+ "classification": "exact",
+ "local_target": "train.cycles",
+ "reason": "one local cycle is one upstream rollout iteration",
+ "resolution_status": "resolved",
+ "source_value": 2,
+ "upstream_field": "trainer.total_training_steps"
+ }
+ ],
+ "generated_path": "verl-rl-v0.9-single-gpu.yaml",
+ "generated_recipe_sha256": "8a47fa6edf8679a8c82e41e8c5e0fc0aec30898567bc6f5ce6904510c8b2545a",
+ "generated_recipe_validated": true,
+ "input_contract": "resolved documented verl v0.9 RL subset; not arbitrary Hydra YAML",
+ "local_lowering": [
+ {
+ "kind": "physical_runtime",
+ "local_value": "hf_cached",
+ "source_field": "actor_rollout_ref.rollout.name",
+ "source_value": "vllm"
+ },
+ {
+ "kind": "distributed_only",
+ "local_value": 1,
+ "source_field": "actor_rollout_ref.rollout.tensor_model_parallel_size",
+ "source_value": 8
+ },
+ {
+ "kind": "distributed_only",
+ "local_value": {
+ "devices": 1,
+ "processes": 1
+ },
+ "source_field": "trainer.n_gpus_per_node/trainer.nnodes",
+ "source_value": {
+ "n_gpus_per_node": 8,
+ "nnodes": 2
+ }
+ }
+ ],
+ "profile": "verl-rl-v0.9-single-gpu-v1",
+ "profile_field_rules_sha256": "d37d10c946993d1da840eda7157e4577fcd7c77ae238852c84b03298097e660f",
+ "report_path": "verl-rl-v0.9-single-gpu.import-report.json",
+ "required_user_input": [],
+ "schema_version": 1,
+ "source_config_sha256": "52fb94c28c6f4a878f584c1bcc9997ca35d4ad979e91c02071618f5a4a0f0a07",
+ "source_verl": {
+ "commit": "483b8a009ba3a97563edee3a19887e4862b8094a",
+ "repository": "https://github.com/verl-project/verl",
+ "tag": "v0.9.0"
+ },
+ "status": "accepted",
+ "unresolved_fields": [],
+ "unsupported_fields": []
+}
diff --git a/docs/index.md b/docs/index.md
index 22ebd80..424d5f8 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,45 +1,44 @@
# miniVERL
-Run verl-style on-policy distillation on one NVIDIA GPU, inspect every local
-mapping and teacher target, and carry standard artifacts into a scale-out
-workflow.
-
-[Install and run locally](single-gpu-guide.md){ .md-button .md-button--primary }
-[Start with a verl profile](for-verl-users.md){ .md-button }
+Compile supported verl experiment semantics into a validated execution plan for
+one NVIDIA GPU. The same local runtime produces policy-bound trajectories,
+transactional checkpoints and portable PEFT/Parquet artifacts.
+
+[Run verl-shaped RL locally](verl-rl-runtime.md){ .md-button .md-button--primary }
+[Choose a workflow](comparisons.md){ .md-button }
+
+## From a verl config to one GPU
+
+```text
+resolved verl-shaped config
+ ↓ versioned compatibility compiler
+field report + validated native recipe
+ ↓ temporal role scheduler
+actor rollout → reward / reference / teacher → actor update
+ ↓
+PEFT + Parquet + config + typed provenance
+```
-## From profile to adapter
+The current RL profile targets official verl `v0.9.0` at commit
+`483b8a009ba3a97563edee3a19887e4862b8094a`. It runs GRPO, Dr.GRPO,
+RLOO and REINFORCE++ with grouped samples, task rewards and optional fixed
+reference-policy KL. The existing verl `v0.8.0` profiles run direct GKD and
+sampled-k1 OPD with explicit teacher targets.
-miniVERL compiles typed YAML and structured Parquet prompts into three local
-phases: actor rollout, teacher scoring and actor update. The same execution plan
-binds the profile version, overrides and input bytes to trajectories,
-checkpoints, teacher caches and the final PEFT adapter.
+## Start with GRPO
```bash
python -m pip install torch --index-url https://download.pytorch.org/whl/cu130
python -m pip install "miniverl[train,cuda]"
-miniverl data sample --format verl-parquet --out prompts.parquet
-miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
- --config builtin:qwen3-0.6b-1.7b-opd \
- --set 'data.train_files=["prompts.parquet"]' --out plan.json
-miniverl run --profile verl-opd-v0.8-single-gpu-v1 \
- --plan plan.json --dry-run
-```
-
-Planning is weight-free. Review `plan.json`, remove `--dry-run` on a CUDA GPU,
-then inspect and export the result:
-
-```bash
-miniverl run --profile verl-opd-v0.8-single-gpu-v1 --plan plan.json \
- --output runs --run-id my-opd
-miniverl inspect runs/my-opd/trajectories.jsonl
-miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout
-miniverl bridge doctor scaleout --json
+miniverl data sample --task-rewards --rows 8 --out data/rl-prompts.parquet
+miniverl import-verl --profile verl-rl-v0.9-single-gpu-v1 \
+ --config examples/verl-rl-v0.9-single-gpu.yaml --out local-grpo.yaml
+miniverl train local-grpo.yaml --dry-run
```
-The `[train,cuda]` extra supplies the training and quantization stack. Choose
-the matching CUDA PyTorch wheel separately; the
-[single-GPU installation guide](single-gpu-guide.md) includes flexible and
-maintainer-measured setups.
+The import report shows the value and disposition of every source field. The
+generated recipe has already passed `RunConfig` validation and can be inspected
+with `miniverl validate local-grpo.yaml --json` before model weights are loaded.
## Three ways to use miniVERL
@@ -47,36 +46,37 @@ maintainer-measured setups.
-### Run local OPD
+### Prototype verl RL
-Compile a pinned direct-GKD or sampled-k1 profile, execute it in local phases,
-and inspect strict current-policy trajectories.
+Bring a resolved v0.9-shaped config and run a critic-free algorithm with true
+grouped rollouts on one CUDA device.
```bash
-miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml
+miniverl import-verl --profile verl-rl-v0.9-single-gpu-v1 \
+ --config verl-rl.yaml --out local.yaml
```
-**Artifact:** immutable plan, trajectories, teacher cache and PEFT adapter.
+**Artifact:** native recipe plus field-by-field compatibility report.
-**Next:** [OPD quickstart](opd-quickstart.md)
+**Next:** [Single-GPU verl RL](verl-rl-runtime.md)
-### Bring verl-shaped inputs
+### Distill locally
-Keep familiar field names and structured Parquet while receiving a
-field-by-field account of each local effect.
+Compile a direct-GKD or sampled-k1 OPD profile, then inspect teacher targets,
+trajectory provenance and the final adapter.
```bash
-miniverl compat check --profile verl-opd-v0.8-single-gpu-v1 \
- --config verl-opd.yaml
+miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
+ --config verl-opd.yaml --out plan.json
```
-**Artifact:** resolved compatibility matrix and local execution plan.
+**Artifact:** immutable plan, teacher cache, trajectories and PEFT adapter.
-**Next:** [For verl users](for-verl-users.md)
+**Next:** [OPD quickstart](opd-quickstart.md)
@@ -84,14 +84,14 @@ miniverl compat check --profile verl-opd-v0.8-single-gpu-v1 \
### Prepare scale-out artifacts
-Package the local adapter, source Parquet, resolved config and provenance, then
-materialize exact model snapshots against the pinned upstream source.
+Package a local run into standard PEFT, safetensors, Parquet and config
+artifacts with a separate readiness report.
```bash
-miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout
+miniverl export-verl --run runs/my-run --target-verl v0.8.0 --out scaleout
```
-**Artifact:** checksummed PEFT + Parquet + config bundle with readiness states.
+**Artifact:** checksummed portable bundle and compatibility states.
**Next:** [Scale-out contract](verl-opd-scaleout.md)
@@ -99,51 +99,29 @@ miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout
-## Runtime design
+## Capability map
-The local scheduler keeps actor, teacher and optional reference roles explicit
-while choosing resident, swap or shared-backbone placement. Structured token
-provenance, pickle-free caches and transactional publication make a run
-inspectable and resumable across phase boundaries.
+| Surface | Local status |
+| --- | --- |
+| GRPO, Dr.GRPO, RLOO, REINFORCE++ | verl v0.9-conformant estimator and policy-loss semantics |
+| Grouped rollout, task rewards, fixed reference KL | supported and provenance-bound |
+| Direct GKD and sampled-k1 OPD | supported through pinned verl v0.8 profiles |
+| SFT, DPO, offline KD, tool environments | native recipe workflows |
+| PPO/GAE execution | value math tested; trainable critic lifecycle not yet implemented |
+| Ray, FSDP/FSDP2, Megatron, TP/PP/DP > 1 | distributed-only |
-Two measured profiles pin official verl `v0.8.0` at `7aed6b23`:
-
-| Profile | Objective | Teacher signal |
-| --- | --- | --- |
-| direct GKD | `forward_kl_topk` | top-k token IDs and log-probabilities |
-| sampled-k1 PG | sampled `k1` + vanilla policy loss | sampled-token teacher log-probability |
-
-[Compatibility profiles](profiles/index.md) describes the exact field and
-semantic contract. [Hardware planning](hardware-planning.md) explains how the
-same device-name-agnostic CUDA path adapts to different VRAM budgets.
-
-## Measured runtime evidence
-
-The RTX 4080 Qwen3 developer workload consumed 32 distinct prompts and
-completed eight current-policy updates at 3.1914 GiB peak reserved VRAM. A
-matched interruption/resume reproduced byte-identical trajectories, adapter
-and optimizer tensors. The companion SmolLM2 workload completed the same run
-shape at 1.4961 GiB peak reserved VRAM.
-
-[Qwen3 workload](verl-opd-reference-workload.md){ .md-button }
-[SmolLM2 workload](smollm2-opd-workload.md){ .md-button }
-
-## Research record
-
-The studies section preserves the actual outcome of each scoped experiment.
-It includes a protocol-qualified teacher that tied supervised continuation, a
-negative RecoveryBench result, regressions from a saturated Alignment Lab
-starting point, and a preregistered external-alignment early stop.
+Read [compatibility](compatibility.md) for the complete field and semantic
+contract. [Limitations](limitations.md) collects measurement, architecture,
+security and generalization boundaries.
-- [Calculator protocol study](benchmarking.md)
-- [RecoveryBench](recoverybench/recoverybench-v1.md)
-- [Alignment Lab](alignment-lab/alignment-lab-v1.md)
-- [External Alignment Gate](alignment-external/alignment-external-v1.md)
+## Evidence and research
-## Scope and boundaries
+The RTX 4080 systems record covers current-policy OPD updates, exact resume and
+rollout backends. The v0.9 RL path adds upstream numerical/gradient conformance,
+CPU integration and exact-wheel GPU qualification. Runtime qualification is
+kept separate from task-quality claims.
-miniVERL's execution scope is one local process, one NVIDIA CUDA GPU and its
-versioned profiles. Its scale-out path ends in a validated artifact handoff.
-Read [compatibility](compatibility.md) for field and handoff semantics, and
-[limitations](limitations.md) for the consolidated architecture, measurement,
-security and generalization boundaries.
+- [RTX 4080 workload](verl-opd-reference-workload.md)
+- [Rollout Runtime v2](benchmarks/rollout-runtime-v2.md)
+- [Scientific studies](alignment-lab/alignment-lab-v1.md)
+- [For verl users](for-verl-users.md)
diff --git a/docs/limitations.md b/docs/limitations.md
index 9620319..9432215 100644
--- a/docs/limitations.md
+++ b/docs/limitations.md
@@ -138,9 +138,11 @@ share one physical backbone forward. `auto` pads the whole group and can be
slower or use more memory when lengths differ. Sequential physical batching
remains the compatibility default.
-Rollout decoding is still one sequence at a time. There is no continuous
-batching scheduler, vLLM/SGLang engine or asynchronous actor pool, so update
-speedups must not be described as rollout-throughput improvements.
+The `hf_reference` compatibility path decodes one sequence at a time.
+`hf_cached` adds grouped prefill, per-row RNG and KV-cache decode; the measured
+direct-GKD path can also use miniVERL-managed vLLM 0.28.0 on Linux/WSL2.
+There is no asynchronous actor pool or general-purpose continuous-batching
+scheduler, and the RL policy-loss path uses HF behavior log-probabilities.
If you set `gradient_accumulation_steps < rollouts_per_cycle` in `opd` mode,
the default `train.opd_freshness: strict` rejects the config: later optimizer
@@ -150,6 +152,29 @@ updated. Explicit `opd_freshness: replay` allows the schedule but labels it
shipped 16 GB recipe sets `gradient_accumulation_steps: 6` equal to
`rollouts_per_cycle: 6` for strict freshness.
+For `run.mode: rl`, one cycle is one rollout iteration. The logical trajectory
+count is `rollouts_per_cycle × rollout.samples_per_prompt`.
+`gradient_accumulation_steps` is the logical actor mini-batch; a smaller value
+intentionally performs multiple replay-labelled updates from one complete
+rollout batch. Physical trajectory batching remains separate. Changing a
+physical batch or OOM chunk must not change group membership, sample seed,
+reward or advantage.
+
+### PPO needs a complete critic lifecycle
+
+The repository includes upstream-conformance tests for v0.9 GAE and clipped
+value-loss primitives, but does not expose an executable PPO recipe. A correct
+runtime needs an independently trainable critic/value head, its optimizer and
+checkpoint state, old/current value provenance, plus an
+actor/reference/critic phase schedule that survives interruption and OOM.
+Those pieces are not yet implemented. This is feasible single-GPU work, not a
+distributed-only feature.
+
+GRPO, Dr.GRPO, RLOO and REINFORCE++ do execute locally. Their portable profile
+supports exact-answer and target-length rewards; environment and trusted Python reward providers
+are available through explicit local API injection. There is no trained reward
+model inference role in this release.
+
### `swap` is unavailable whenever anything is quantized
`memory.strategy: swap` moves the student and its optimizer state to host memory
@@ -427,20 +452,27 @@ projection have not been exercised beyond the pinned Qwen3 pair (which does have
### Execution boundary
miniVERL has no distributed training, multi-GPU or multi-node execution,
-high-throughput rollout engine, PPO/GRPO objective, trained reward model,
-vision-language support, cross-tokenizer distillation, or containerized or
-networked tool sandbox. The three built-in environments still generate their
-tasks in-process; the v0.6 Parquet converter only exchanges validated prompt
-datasets and does not turn arbitrary verl datasets into those environments.
-Tools execute in-process. Where that is risky the environment restricts the
-input rather than the process: the calculator walks a parsed `ast` with a closed
-node whitelist instead of calling `eval`, and the SQLite environment uses a
-`sqlite3` authorizer plus a function whitelist and permits one statement per
-call.
+trainable PPO critic, trained reward-model role, vision-language path,
+cross-tokenizer distillation, or containerized/networked tool sandbox. It does
+run critic-free GRPO, Dr.GRPO, RLOO and REINFORCE++ objectives on one GPU. Ray,
+FSDP/FSDP2, Megatron and TP/PP/DP above one are distributed-only placement
+systems.
+
+The built-in environments generate tasks in-process; Parquet conversion does
+not turn an arbitrary verl dataset into an environment. Tools also execute
+in-process. The calculator walks a parsed `ast` with a closed node whitelist
+instead of calling `eval`, and SQLite uses an authorizer, function whitelist and
+one-statement limit. Use an external sandbox when the tool itself is untrusted.
### Verl bridge boundary
-The v0.6 **miniVERL-defined compatibility Level 3** bridge targets official verl
+The RL compiler targets official verl `v0.9.0` at commit
+`483b8a009ba3a97563edee3a19887e4862b8094a` and accepts a closed resolved
+subset. It preserves data, actor, rollout, algorithm and schedule fields with
+tested local semantics; distributed resource fields lower to one process and
+device. It is not a generic Hydra composition engine.
+
+The historical **miniVERL-defined compatibility Level 3** bridge targets official verl
`v0.8.0` at one exact commit and one named profile. It validates standard
PEFT/safetensors/tokenizer artifacts, Parquet prompt data, 14 whitelisted
config fields, a safe reward scaffold and bundle hashes. The recorded smoke
@@ -521,5 +553,6 @@ These are future design directions rather than current runtime surfaces.
- **Native multi-GPU or multi-node execution.** Not implemented. The v0.6
bridge can export one documented profile to pinned verl, where scale-out
remains the user's separately reviewed and executed operation.
-- **A batched or engine-backed rollout path.** Not implemented.
+- **PPO critic execution.** GAE/value math is pinned and tested; critic model,
+ optimizer, checkpoint and temporal phase ownership remain to be built.
- **Additional tested architectures.** Only Qwen3 and Qwen2 are tested today.
diff --git a/docs/maintainer-architecture.md b/docs/maintainer-architecture.md
index 19e76b8..49d4a87 100644
--- a/docs/maintainer-architecture.md
+++ b/docs/maintainer-architecture.md
@@ -4,12 +4,13 @@ This page maps the change boundaries maintainers use for review.
## Config to runtime
-`bridge.profiles` selects one closed profile and dispatches to its typed source
-compiler. The compiler classifies every source field and produces a resolved
-compatibility plan. `bridge.opd_runtime` derives legal one-device placement and
-the native `RunConfig`; `bridge.opd_plan` binds data files, revisions, profile
-identity and native config into an immutable plan. `OPDTrainer.from_config`
-then constructs the local runtime. No generic dynamic profile loader exists.
+`bridge.profiles` selects a closed OPD profile, while `bridge.rl_v09` owns the
+resolved verl v0.9 RL subset. Both classify source fields before publishing a
+native `RunConfig`. `bridge.opd_runtime` and `bridge.opd_plan` add placement,
+data and immutable-plan identity for OPD; the RL compiler emits its native
+recipe and field report transactionally. `OPDTrainer.from_config` remains the
+public runtime facade for native modes. No generic dynamic profile loader
+exists.
The profile identity enters immutable plans and is copied into run manifests,
teacher-cache metadata, checkpoint identity and export/materialization reports.
@@ -18,18 +19,27 @@ using their contents.
## Logical roles and physical placement
-Actor, teacher and optional reference are distinct logical identities. Rollout
-generates with the current actor policy; teacher scoring observes only the
-visited positions; update consumes provenance-bound targets. Physical
-placement may be resident phased models, an allowed unquantized swap, or a
-shared backbone with separate adapters. Quantized swap remains illegal.
+Actor, teacher, reference, reward and future critic are distinct logical
+identities. Rollout generates with the current actor policy. OPD teacher
+scoring observes only visited positions; RL reward providers consume ordered
+trajectory identities; reference KL evaluates a frozen adapter before the
+actor is restored for update. Physical placement may be resident phased
+models, an allowed unquantized swap, or a shared backbone with separate
+adapters. Quantized swap remains illegal.
`training.trainer.OPDTrainer` is the compatibility facade and state-machine
owner. Batching, memory planning, optimizer construction, checkpoint I/O and
offline-dataset persistence live in dedicated `training` modules; model and
tokenizer construction live under `models`; cache transactions live under
-`cache`. Public methods remain on the facade so internal extraction does not
-change user imports.
+`cache`. Pure upstream-conformant advantage, policy, value and KL mathematics
+live under `algorithms`; trusted reward protocols live under `rewards`. Public
+methods remain on the facade so internal extraction does not change user
+imports.
+
+RL trajectories are schema v3 records with prompt-group, sample, seed,
+generated-span and policy identity. Tool/environment content remains context;
+only model-generated positions enter behavior log-probability and loss masks.
+The trainer publishes complete groups before scoring or updating them.
## State and transaction boundaries
@@ -58,6 +68,12 @@ effect, add pinned upstream scalar/gradient/optimizer conformance, exercise
plan/cache/checkpoint/export identity mismatch failures, then complete the
[upstream lifecycle](upstream-support-policy.md) and GPU qualification.
+For an RL compiler field, choose one of `exact`, `semantically_conformant`,
+`locally_lowered`, `informational_only`, `distributed_only`,
+`not_implemented` or `unsupported`. An accepted runnable source must have no
+unknown fields and must validate as a complete `RunConfig` before atomic
+publication. Keep local physical controls under `miniverl.*` where practical.
+
## Validation entry points
```bash
diff --git a/docs/math.md b/docs/math.md
index 5afcf62..9a9f982 100644
--- a/docs/math.md
+++ b/docs/math.md
@@ -1,8 +1,9 @@
# The miniVERL objective
-This document defines exactly what miniVERL optimizes, and states which
-property is asserted by which test. It covers the five modules under
-`src/miniverl/losses/`:
+This document defines miniVERL's distillation objectives and states which
+property is asserted by which test. Critic-free RL uses the separate pure
+primitives under `src/miniverl/algorithms/`, summarized below; the remaining
+numbered sections cover the modules under `src/miniverl/losses/`:
| module | responsibility |
| --- | --- |
@@ -15,6 +16,32 @@ property is asserted by which test. It covers the five modules under
Nothing here depends on a model, a config object or a device. That is what makes
the brute-force reference tests possible.
+## Critic-free RL objectives
+
+For a prompt group with scalar trajectory outcomes $R_i$, GRPO assigns every
+generated token in sample $i$ the sequence advantage
+
+$$
+A_i = \frac{R_i - \bar R}{s_R + \epsilon},
+$$
+
+where $s_R$ is the sample standard deviation used by verl v0.9. Dr.GRPO uses
+$R_i - \bar R$ without the denominator. RLOO uses
+$A_i = R_i - \sum_{j \ne i} R_j/(n-1)$. REINFORCE++ forms discounted token
+returns and whitens them over the selected response mask.
+
+The actor objective uses the recorded behavior-policy log-probability
+$\log \pi_{old}$, the recomputed current value $\log \pi_\theta$, ratio
+$r=\exp(\log \pi_\theta-\log \pi_{old})$, and verl's vanilla asymmetric plus
+dual clipping before token-mean reduction. A fixed reference reward penalty is
+subtracted token-wise before advantage estimation when configured.
+
+`tests/conformance/test_verl_v09_rl.py` extracts the official functions from
+verl `v0.9.0` commit `483b8a009ba3a97563edee3a19887e4862b8094a` and compares
+estimator values, returns, policy/value losses, KL penalties, gradients and an
+optimizer step. GAE and clipped value loss are included in that mathematical
+conformance suite, but the executable runtime has no trainable critic yet.
+
---
## 1. Notation
diff --git a/docs/overrides/main.html b/docs/overrides/main.html
index 8a99330..b71da5b 100644
--- a/docs/overrides/main.html
+++ b/docs/overrides/main.html
@@ -1,12 +1,12 @@
{% extends "base.html" %}
{% block announce %}
-
+
Stable documentation
Version
- Stable 0.11.0
- Development 0.11.1.dev0
+ Stable 0.12.0
+ Development 0.12.0
{% endblock %}
diff --git a/docs/profiles/index.md b/docs/profiles/index.md
index 06c4273..83999d3 100644
--- a/docs/profiles/index.md
+++ b/docs/profiles/index.md
@@ -4,6 +4,11 @@ A miniVERL compatibility profile is a closed, versioned contract that binds an
accepted schema and its field rules to one upstream repository, tag and commit.
Native-compiler, loss-conformance and export versions travel with that identity.
+The v0.8 OPD family is exposed through the `profiles` registry. The newer
+`verl-rl-v0.9-single-gpu-v1` compiler is invoked through `import-verl`; its
+machine-readable field report is the profile inspection surface while the
+registry API is generalized in a future compatible change.
+
```bash
miniverl profiles list
miniverl profiles show verl-opd-v0.8-single-gpu-v1
@@ -39,6 +44,16 @@ name.
## Which profile should I use?
+For critic-free RL, start with:
+
+```bash
+miniverl import-verl --profile verl-rl-v0.9-single-gpu-v1 \
+ --config examples/verl-rl-v0.9-single-gpu.yaml --out local-grpo.yaml
+```
+
+It supports GRPO, Dr.GRPO, RLOO and REINFORCE++ against the pinned verl v0.9
+contract; read [Single-GPU verl RL](../verl-rl-runtime.md).
+
| Profile | Objective | Teacher target | Trade-off | Status |
| --- | --- | --- | --- | --- |
| `verl-opd-v0.8-single-gpu-v1` | direct GKD `forward_kl_topk` | top-k token IDs and log-probabilities | fuller distributional signal; larger target artifact | measured |
@@ -55,7 +70,8 @@ policy-loss form. The measured records compare runtime and semantic
conformance; the rewarded profile has no task-quality result yet.
Library callers may inject an object implementing `RewardProvider` with
`reward.provider: python_api`; YAML and run artifacts never name or load Python
-modules. The built-in verl-shaped profile remains fixed to `exact_answer`.
+modules. The built-in verl-shaped RL profile accepts the deterministic
+`exact_answer` and `target_length` providers.
The group-capable profiles use transactional Parquet prompt groups while
leaving the published `n=1` identities unchanged.
See [For verl users](../for-verl-users.md), the
diff --git a/docs/release-checklist.md b/docs/release-checklist.md
index 13f013a..a000b3b 100644
--- a/docs/release-checklist.md
+++ b/docs/release-checklist.md
@@ -4,7 +4,27 @@ This is the release gate and publication record for miniVERL. A checked item
names an invariant exercised on the stated source. Publication begins only
after the exact release commit and its remote checks are green.
-## v0.11.1 development
+## v0.12.0 development
+
+- [x] Pin the critic-free RL compiler and conformance fixtures to upstream verl
+ v0.9.0 at `483b8a009ba3a97563edee3a19887e4862b8094a`.
+- [x] Implement grouped GRPO, Dr.GRPO, RLOO and REINFORCE++ with typed rewards,
+ behavior-policy log-probs, optional reference KL and exact resume.
+- [x] Add deterministic exact-answer, target-length, environment-verifier,
+ weighted-composite and explicit trusted Python API reward paths.
+- [x] Add a v0.12 exact-wheel RTX 4080 qualification record to the existing
+ full-release promotion and canonical evidence archive.
+- [x] Validate implementation commit
+ `feed761514c3925dd2290d855baac6d38c417e50`: 2,566 non-GPU/non-network
+ tests at 83% branch coverage, 10 local RTX 4080 GPU tests, 16 network
+ tests, pinned verl v0.9 conformance, strict docs and four browser widths.
+- [ ] Merge the reviewed release PR after all hosted checks pass.
+- [ ] Run the exact-release-SHA WSL2 full qualification on attempt 1.
+- [ ] Complete the non-publishing release dry run and OIDC publication.
+- [ ] Verify PyPI, GitHub Release, stable/dev docs and clean installation.
+- [ ] Advance main to `0.12.1.dev0` in a state-sync PR.
+
+## v0.11.1 development (superseded by v0.12 scope)
- [x] Begin from the verified v0.11.0 release and advance only the canonical
development state to `0.11.1.dev0`.
diff --git a/docs/release-qualification.md b/docs/release-qualification.md
index 58b74c8..ccc57fe 100644
--- a/docs/release-qualification.md
+++ b/docs/release-qualification.md
@@ -10,7 +10,7 @@ workflow is manual, not continuous GPU CI and not a pull-request required check.
| level | cadence | executed scope |
| --- | --- | --- |
| release smoke | diagnostic use | install the hosted-runner candidate wheel, verify import/CLI origin, run CLI doctor/plan/dry-run, pinned Qwen actor and teacher, one rollout/score/update, PEFT export/reload and CUDA teardown |
-| full qualification | every formal release | release smoke, the three canonical resume workloads, and the v0.11 direct n=1, grouped PG n=4, rewarded PG n=4, `hf_cached` and vLLM runtime gates |
+| full qualification | every formal release | release smoke, the three canonical resume workloads, the v0.11 rollout-backend matrix, and the v0.12 pinned-v0.9 GRPO workload |
The smoke budget is intentionally small and is never substituted for a frozen
full workload or a scientific benchmark. Both levels record runtime
@@ -42,6 +42,13 @@ trajectory identity; reward/advantage completion; and vLLM's
policy-gradient fail-closed boundary. Loose result files cannot claim the
higher level.
+For v0.12 and later, promotion also requires a compiled
+`verl-rl-v0.9-single-gpu-v1` plan and two Qwen3-0.6B NF4 LoRA GRPO cycles on
+the exact candidate wheel. The deterministic target-length formatting reward
+must vary within at least one prompt group, produce nonzero advantages, commit
+two parameter-changing updates, remain below 14.5 GiB peak reserved memory and
+retain the runtime-only scientific scope.
+
Every dispatch is single-use. Both jobs reject `GITHUB_RUN_ATTEMPT` values
other than `1`; if infrastructure fails, start a new `workflow_dispatch`
instead of using GitHub's rerun button. This prevents a hosted candidate from
@@ -58,7 +65,8 @@ authorization, proxy authorization and cookies. It publishes the accepted
wheel and sdist without rebuilding them. Future release runs retain the full
qualification record, four principal workload JSON files and a deterministic
subordinate-evidence archive. For v0.11, that archive also contains the three
-profile/backend qualification records. A committed
+profile/backend qualification records; v0.12 adds the RL qualification record.
+A committed
JSON file, manual upload, fork run, different workflow or cross-run artifact
pair cannot satisfy this gate.
@@ -132,9 +140,9 @@ portable bounded artifacts, and checks cleanup targets remain under
`GITHUB_WORKSPACE`. Model caches remain runner-local and are not uploaded.
Rotate the runner token after suspected exposure.
-The v0.11 release runner is Ubuntu under WSL2 with Python 3.12.13 and the
+The current release runner is Ubuntu under WSL2 with Python 3.12.13 and the
checked `known-good-rtx4080-wsl2-cu130` stack. The older Windows qualification
-record remains historical and cannot authorize v0.11.
+record remains historical and cannot authorize a current release.
## Measured release state
diff --git a/docs/reproducibility.md b/docs/reproducibility.md
index 3616b8a..cee1362 100644
--- a/docs/reproducibility.md
+++ b/docs/reproducibility.md
@@ -277,10 +277,11 @@ $ curl -s https://huggingface.co/api/models/Qwen/Qwen3-0.6B \
apache-2.0
```
-Finally, verify the tokenizers behaviourally, which is what miniVERL actually
-checks. `HFTokenizerAdapter` computes a fingerprint from the tokenizer class,
-`len(tokenizer)`, the EOS and PAD ids, the added special tokens, and the token
-ids produced for a fixed probe string:
+Finally, verify tokenizer structural identity, which is miniVERL's primary
+new-run check. `HFTokenizerAdapter` canonicalizes and hashes the vocabulary,
+added vocabulary, special-token map, backend tokenizer and behavior-relevant
+configuration. It also retains the historical behavioral fingerprint for
+legacy artifacts that have no structural digest:
```console
$ python -c "
@@ -288,12 +289,12 @@ from miniverl.models.tokenizers import HFTokenizerAdapter
a = HFTokenizerAdapter.load('Qwen/Qwen3-0.6B', revision='c1899de289a04d12100db370d81485cdf75e47ca')
b = HFTokenizerAdapter.load('Qwen/Qwen3-1.7B', revision='70d244cc86ccca08cf5af4e1e306ecf908b1ad5e')
print('vocab_size', a.vocab_size, b.vocab_size)
-print('identical', a.fingerprint == b.fingerprint)
-print(a.fingerprint)
+print('structurally identical', a.identity.digest == b.identity.digest)
+print('legacy fingerprint equal', a.fingerprint == b.fingerprint)
"
vocab_size 151669 151669
-identical True
-f2f5e826dddc3ff1e2481111075f2ed6eced4e553168222d67650931d25be035
+structurally identical True
+legacy fingerprint equal True
```
Note the two vocabulary numbers. `config.json` declares `vocab_size: 151936`
@@ -301,11 +302,10 @@ Note the two vocabulary numbers. `config.json` declares `vocab_size: 151936`
records the tokenizer number as `models.tokenizer_vocab_size` in the manifest
and uses the model's own `vocab_size` for the LM-head projection.
-The fingerprint value above was measured with transformers 5.14.1. It depends
-on the tokenizer class name and the added-token list, so a transformers upgrade
-can change it even when `tokenizer.json` does not. Compare fingerprints between
-student and teacher within one environment; do not compare a fingerprint across
-library versions.
+Structural identity intentionally removes local path spellings such as
+`name_or_path`, so identical snapshots loaded from different directories hash
+the same way. The legacy fingerprint depends on the tokenizer class and probe
+encoding; agreement is useful for old artifacts but is not an identity proof.
## What is not deterministic, and why
diff --git a/docs/rollout-runtime-v2.md b/docs/rollout-runtime-v2.md
index 7d50279..9ae495c 100644
--- a/docs/rollout-runtime-v2.md
+++ b/docs/rollout-runtime-v2.md
@@ -63,6 +63,12 @@ remain fixed at `n=1`; the grouped profiles are conformance-only until a later
evidence stage measures them. Grouped rollouts currently require a Parquet
prompt source. Environment-backed recipes remain `n=1`.
+The newer `verl-rl-v0.9-single-gpu-v1` profile consumes the same schema-v3
+group identity as actual algorithm input: GRPO, Dr.GRPO and RLOO compute their
+prompt-group baselines across the complete `n > 1` samples. Environment-backed
+RL also supports grouped episodes through the recorded verifier path. See
+[Single-GPU verl RL](verl-rl-runtime.md).
+
## Managed vLLM for direct GKD
Install the optional engine in the same Linux or WSL2 environment as the
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
index 0e0669e..b7e4fdd 100644
--- a/docs/troubleshooting.md
+++ b/docs/troubleshooting.md
@@ -325,13 +325,12 @@ error student and teacher tokenizers differ (<12 hex chars>... vs <12 hex chars>
```
**Cause.** `build_tokenizer` loads one tokenizer for both sides. When the
-teacher declares its own `tokenizer_id`, that tokenizer is loaded and
-fingerprint-compared rather than trusted. The fingerprint is behavioural: it
-hashes the tokenizer class name, `len(tokenizer)`, the EOS and PAD ids, the
-added special tokens, and the token ids produced for a fixed probe string that
-includes the chat template markers and the tool-call tags. Two tokenizers with
-the same declared vocabulary size but different merge behaviour therefore do
-not match.
+teacher declares its own `tokenizer_id`, that tokenizer is loaded and compared
+by structural identity: full vocabulary, added vocabulary, special-token map,
+backend tokenizer and behavior-relevant configuration. Local path spellings are
+canonicalized away. Old artifacts without that digest use the legacy behavioral
+probe fingerprint, which is a compatibility signal rather than an identity
+proof.
**Fix.** Use a teacher from the same family as the student. The pinned pair in
`recipes/qwen_consumer_gpu_calc.yaml` is verified byte-identical on
@@ -528,31 +527,16 @@ loads no models.
`metrics.jsonl` and in the `rollouts_collected` events is around ten tokens per
second on a modern GPU.
-**Cause, measured.** Single-sequence decoding in miniVERL is kernel-launch
-bound, not compute bound. On this machine a 14-token prefill costs 37.0 ms
-while a cached 1-token step costs 30.9 ms: almost all of the per-step cost is
-fixed overhead rather than work proportional to the token count. Supporting
-measurements from the same probe, at 64 new tokens with a 36-token prefix:
+**Cause, measured.** `hf_reference` decodes each sequence through the simple
+compatibility loop and can be kernel-launch bound. Rollout Runtime v2 measured
+the compiled `hf_cached` backend at 124.2–248.7 output tokens/s over its RTX
+4080 matrix. Managed vLLM reached 626.6–836.4 tokens/s on the qualified
+direct-GKD cells. See the [runtime result](benchmarks/rollout-runtime-v2.md)
+before comparing numbers across models, lengths or sampling modes.
-| configuration | tok/s | peak allocated |
-| --- | --- | --- |
-| NF4 + bf16, deterministic | 11.19 | 0.862 GiB |
-| NF4 + bf16, non-deterministic | 11.29 | not recorded |
-| bf16 LoRA, deterministic | 12.84 | 1.170 GiB |
-| bf16 LoRA, non-deterministic | 14.12 | not recorded |
-
-For reference, applying the LM head at one position costs 0.48 ms, decoding 64
-token ids costs 0.02 ms, and copying a 151936-float vector to the host costs
-0.11 ms. None of those is the bottleneck.
-
-Reproduce the probe on your own machine:
-
-```bash
-python scripts/gpu_probe_throughput.py
-```
-
-**Fix.** There is no batched rollout in this version, so the levers are budget
-levers rather than throughput levers:
+**Fix.** Use `hf_cached` for HF RL and sampled-k1 paths. The direct-GKD OPD
+profile can opt into managed vLLM on the qualified Linux/WSL2 stack. Context and
+workload bounds remain useful controls:
```yaml
rollout:
@@ -570,11 +554,6 @@ models:
attn_implementation: sdpa # sdpa or eager
```
-Turning off deterministic mode with `run: {deterministic: false}` recovered
-about 10 percent on bf16 LoRA in the table above and almost nothing on NF4. It
-also gives up the guarantees in `docs/reproducibility.md`; the default is
-`true` for that reason.
-
Note that `models.student.gradient_checkpointing: true` sets
`model.config.use_cache = False` for training, but `HFBackend.generate`
temporarily re-enables `use_cache` for the duration of a generation call and
diff --git a/docs/verl-local-runtime-mobile.svg b/docs/verl-local-runtime-mobile.svg
index 200cbe3..913f3da 100644
--- a/docs/verl-local-runtime-mobile.svg
+++ b/docs/verl-local-runtime-mobile.svg
@@ -1,19 +1,19 @@
- miniVERL local OPD product flow, mobile layout
- A vertical view of typed verl-shaped inputs, local one-GPU OPD phases, inspectable artifacts and a portable export bundle with explicit readiness states.
+ miniVERL semantic lowering and local execution flow, mobile layout
+ A vertical view of resolved verl inputs, versioned compilation, local actor, reward, reference or teacher phases, and portable artifacts with explicit readiness states.
- One-GPU OPD workflow typed · inspectable · resumable
+ verl semantics → one GPU typed · inspectable · resumable
INPUTS verl-shaped YAML + overrides Parquet prompt data
TYPED COMPILER Compatibility report exact · reinterpreted · rejected
+ VERSIONED COMPILER Compatibility report conformant · lowered · rejected
LOCAL SCHEDULER · 1× CUDA GPU
actor rollout teacher scoring actor update
- INSPECTABLE OUTPUTS checkpoint · PEFT · targets trajectories · metrics · provenance
- materialized export bundle → pinned verl v0.8.0
+ actor rollout reward / ref / teacher actor update
+ INSPECTABLE OUTPUTS checkpoint · PEFT · rewards / targets trajectories · metrics · provenance
+ portable artifact bundle PEFT · Parquet · config
READINESS REPORT bundle · launch · execution
diff --git a/docs/verl-local-runtime.svg b/docs/verl-local-runtime.svg
index 56787c9..564f4da 100644
--- a/docs/verl-local-runtime.svg
+++ b/docs/verl-local-runtime.svg
@@ -1,6 +1,6 @@
- miniVERL local OPD product flow
- verl-shaped configuration and Parquet prompts enter a typed compiler, then one GPU runs actor rollout, teacher scoring, and actor update. Inspectable artifacts form a portable bundle with explicit readiness states.
+ miniVERL semantic lowering and local execution flow
+ A resolved verl configuration and Parquet prompts enter a versioned compiler, then one GPU runs actor rollout, reward, reference or teacher phases, and actor update. Inspectable artifacts retain explicit readiness states.
- One-GPU OPD, from familiar inputs to portable artifacts
+ verl experiment semantics, lowered onto one GPU
Every boundary is typed, recorded, and resumable.
@@ -22,13 +22,13 @@
overrides · Parquet prompts
COMPILER
Typed compatibility
- exact · reinterpreted · rejected
+ conformant · lowered · rejected
LOCAL ROLE SCHEDULER · 1× CUDA GPU
actor rollout
- teacher scoring
+ reward / ref or teacher
actor update
INSPECTABLE OUTPUTS
- checkpoint · PEFT adapter · teacher targets · trajectories
+ checkpoint · PEFT adapter · rewards / targets · trajectories
metrics · provenance · source and compatibility records
materialized export bundle → pinned verl v0.8.0
+ portable PEFT + Parquet + config artifacts
READINESS REPORT
bundle · launch · execution
diff --git a/docs/verl-rl-runtime.md b/docs/verl-rl-runtime.md
new file mode 100644
index 0000000..baa290b
--- /dev/null
+++ b/docs/verl-rl-runtime.md
@@ -0,0 +1,116 @@
+# Single-GPU verl RL
+
+The `verl-rl-v0.9-single-gpu-v1` compiler translates a resolved, documented
+verl v0.9 RL subset into a native miniVERL recipe. Algorithm semantics stay
+recognizable while distributed placement becomes an explicit temporal schedule
+for one process and one CUDA GPU.
+
+## Quickstart
+
+Create a reward-bearing Parquet file and compile the shipped GRPO example:
+
+```bash
+miniverl data sample --task-rewards --rows 8 --out data/rl-prompts.parquet
+miniverl import-verl --profile verl-rl-v0.9-single-gpu-v1 \
+ --config examples/verl-rl-v0.9-single-gpu.yaml --out local-grpo.yaml
+miniverl validate local-grpo.yaml --json
+miniverl train local-grpo.yaml --dry-run
+```
+
+The importer publishes two files:
+
+- `local-grpo.yaml`: a validated native recipe;
+- `local-grpo.import-report.json`: every source field, source value, local
+ target, classification, reason and lowering decision.
+
+Run without `--dry-run` after reviewing both artifacts. A source with missing
+reward or parameterization choices receives a non-executable
+`*.template.yaml`; unknown fields, unresolved interpolation and objective
+changes are rejected.
+
+## Supported algorithms
+
+| verl estimator | Local algorithm | Group requirement | Main statistic |
+| --- | --- | --- | --- |
+| `grpo` + standard normalization | `grpo` | `n > 1` | group-centered, sample-std normalized outcome |
+| `grpo` + `norm_adv_by_std_in_grpo: false` | `dr_grpo` | `n > 1` | group-centered outcome without std division |
+| `rloo` | `rloo` | `n > 1` | leave-one-out group baseline |
+| `reinforce_plus_plus` | `reinforce_plus_plus` | any complete batch | discounted token returns with masked whitening |
+
+All four use the pinned verl v0.9 vanilla dual-clipped actor objective and
+token-mean aggregation. Conformance tests compare values, masks, diagnostics,
+gradients and optimizer steps against official source at tag `v0.9.0`, commit
+`483b8a009ba3a97563edee3a19887e4862b8094a`.
+
+## Behavior policy and grouped rollouts
+
+Each trajectory records the actor policy version, prompt-group ID, sample ID,
+sample seed, generated token span and sampled-token behavior log-probability.
+Top-p and top-k choose tokens; the recorded old log-probability follows
+upstream's temperature-scaled actor distribution. Tool results and environment
+observations remain context and are never selected as model-generated targets.
+
+`data.train_batch_size × actor_rollout_ref.rollout.n` is the logical rollout
+batch. `actor.ppo_mini_batch_size` determines the logical actor mini-batch and
+lowers to gradient accumulation over the configured physical trajectory batch.
+A full logical mini-batch performs one strict fresh-policy update. A smaller
+mini-batch is labelled `replay`, matching multiple actor updates from one
+upstream rollout rather than pretending every update was freshly sampled.
+
+## Reward and reference roles
+
+The portable profile supports `miniverl.reward.provider: exact_answer` and the
+deterministic `target_length` formatting reward, both driven by explicit
+Parquet metadata. The Python API can inject a trusted weighted batch provider
+or an environment verifier without importing code from YAML.
+Provider identity, ordered inputs, component outputs, failures and composition
+are bound into run and checkpoint provenance.
+
+The exact-wheel RTX 4080 release qualification uses `target_length` only to
+prove nonconstant grouped rewards, advantages and parameter-changing GRPO
+updates. It is a systems workload, not evidence of task-quality improvement.
+
+Set `algorithm.use_kl_in_reward: true`, a fixed positive `kl_coef`, and an
+explicit frozen `miniverl.reference.adapter_path` to enable reference-policy
+reward KL. The scheduler switches adapters on a shared compatible backbone,
+then restores the actor before the update. Supported verl penalties are `kl`,
+`abs`, `mse` and `low_var_kl`.
+
+## Field classifications
+
+The compiler uses seven dispositions:
+
+| Classification | Meaning |
+| --- | --- |
+| `exact` | value and unit feed the same experiment meaning |
+| `semantically_conformant` | local representation differs, tested meaning is the same |
+| `locally_lowered` | physical implementation changes while the objective is preserved |
+| `informational_only` | retained in the report but inactive for the selected algorithm |
+| `distributed_only` | placement machinery has no one-GPU execution counterpart |
+| `not_implemented` | meaningful on one GPU but absent from this runtime |
+| `unsupported` | no safe versioned rule exists for the supplied field/value |
+
+The committed [machine-readable example report](generated/verl-rl-v0.9-compatibility.json)
+is regenerated from the shipped YAML in CI.
+
+## Resume and artifacts
+
+Checkpoints store algorithm identity, reward identity, policy version, dataset
+cursor, group/sample counters, optimizer state and RNG state. Resume verifies
+those identities before generating another group. The CPU integration suite
+compares uninterrupted and interrupted GRPO runs tensor-for-tensor and rejects
+a changed reward provider.
+
+Run artifacts include schema-v3 trajectories, JSONL reward and metric records,
+transactional checkpoints, the resolved recipe and final PEFT adapter. Use
+`miniverl inspect` for trajectories and metrics.
+
+## Current boundary
+
+PPO/GAE value and clipped-value primitives are upstream-conformance tested, but
+the executable runtime does not yet have an independently trainable critic,
+critic optimizer/checkpoint state or actor/reference/critic phase lifecycle.
+Ray, multi-node scheduling, FSDP/FSDP2, Megatron and parallel degrees above one
+are distributed placement systems and remain outside the local runtime. See
+[compatibility](compatibility.md) and [limitations](limitations.md) for the
+complete boundary.
diff --git a/examples/README.md b/examples/README.md
index 4a37396..165b56e 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -1,7 +1,29 @@
# Examples
-Two runnable examples, both CPU-only and network-free. Each one is executed as
-part of the release check, so if it is here it works.
+The directory contains a verl v0.9-shaped single-GPU RL config plus two
+runnable, CPU-only Python extension examples.
+
+## `verl-rl-v0.9-single-gpu.yaml`
+
+A resolved GRPO source config for the versioned compatibility compiler. It
+keeps upstream data, actor, rollout, algorithm and trainer fields recognizable,
+and places local reward, revision and physical batching choices under
+`miniverl.*`.
+
+```bash
+miniverl data sample --task-rewards --rows 8 --out data/rl-prompts.parquet
+miniverl import-verl --profile verl-rl-v0.9-single-gpu-v1 \
+ --config examples/verl-rl-v0.9-single-gpu.yaml --out local-grpo.yaml
+miniverl train local-grpo.yaml --dry-run
+```
+
+The generated report records how the example's eight-GPU/two-node source
+resource intent lowers to one local process/device. The logical GRPO group and
+objective remain unchanged.
+
+## Python extension examples
+
+Both scripts are network-free and execute as part of the release check.
```bash
pip install "miniverl[train]" # CPU torch is enough
diff --git a/examples/verl-rl-v0.9-single-gpu.yaml b/examples/verl-rl-v0.9-single-gpu.yaml
new file mode 100644
index 0000000..eed018a
--- /dev/null
+++ b/examples/verl-rl-v0.9-single-gpu.yaml
@@ -0,0 +1,87 @@
+# Resolved verl v0.9-shaped GRPO example for the
+# verl-rl-v0.9-single-gpu-v1 compatibility compiler.
+#
+# Create its deterministic reward-bearing input with:
+# miniverl data sample --task-rewards --rows 8 --out data/rl-prompts.parquet
+#
+# Compile and review it with:
+# miniverl import-verl --config examples/verl-rl-v0.9-single-gpu.yaml \
+# --profile verl-rl-v0.9-single-gpu-v1 --out recipes/qwen3_grpo_local.yaml
+#
+# The distributed resource fields are retained as source intent and classified
+# as distributed-only. The compiler lowers execution to one local CUDA process.
+
+data:
+ train_files: [data/rl-prompts.parquet]
+ val_files: []
+ prompt_key: prompt
+ train_batch_size: 2
+ max_prompt_length: 128
+ max_response_length: 24
+ truncation: error
+ shuffle: true
+ seed: 17
+
+actor_rollout_ref:
+ model:
+ path: Qwen/Qwen3-0.6B
+ enable_gradient_checkpointing: true
+ lora_rank: 16
+ lora_alpha: 32
+ target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]
+ actor:
+ optim:
+ lr: 1.0e-5
+ weight_decay: 0.01
+ lr_warmup_steps: 0
+ ppo_mini_batch_size: 8
+ ppo_epochs: 1
+ loss_agg_mode: token-mean
+ clip_ratio: 0.2
+ clip_ratio_low: 0.2
+ clip_ratio_high: 0.2
+ clip_ratio_c: 3.0
+ use_kl_loss: false
+ entropy_coeff: 0.0
+ rollout:
+ name: vllm
+ n: 4
+ temperature: 1.0
+ top_p: 1.0
+ top_k: 0
+ tensor_model_parallel_size: 8
+ gpu_memory_utilization: 0.6
+
+algorithm:
+ adv_estimator: grpo
+ norm_adv_by_std_in_grpo: true
+ gamma: 1.0
+ lam: 1.0
+ use_kl_in_reward: false
+
+trainer:
+ total_training_steps: 2
+ total_epochs: 1
+ project_name: mini-verl
+ experiment_name: qwen3-grpo-local
+ save_freq: 1
+ test_freq: -1
+ logger: [console]
+ n_gpus_per_node: 8
+ nnodes: 2
+
+miniverl:
+ reward:
+ provider: exact_answer
+ actor:
+ lora_enabled: true
+ revision: c1899de289a04d12100db370d81485cdf75e47ca
+ tokenizer_revision: c1899de289a04d12100db370d81485cdf75e47ca
+ dtype: auto
+ quantization: nf4
+ rollout:
+ backend: hf_cached
+ memory:
+ strategy: resident
+ update:
+ trajectory_batch_size: 1
diff --git a/mkdocs.yml b/mkdocs.yml
index bea2873..a8f16a4 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -1,5 +1,5 @@
site_name: miniVERL
-site_description: Inspectable verl-style on-policy distillation on one NVIDIA GPU
+site_description: Verl experiment semantics compiled for one NVIDIA GPU
site_url: https://daoyuanli2816.github.io/mini-verl/
repo_url: https://github.com/DaoyuanLi2816/mini-verl
repo_name: DaoyuanLi2816/mini-verl
@@ -43,6 +43,7 @@ nav:
- Home: index.md
- Choose a stack: comparisons.md
- For verl users: for-verl-users.md
+ - Single-GPU verl RL: verl-rl-runtime.md
- Compatibility profiles: profiles/index.md
- PG-k1 semantic ADR: adr/0010-verl-v0.8-pg-k1-contract.md
- Config overrides: config-overrides.md
@@ -101,4 +102,4 @@ extra_css:
- assets/stylesheets/extra.css
extra_javascript:
- assets/javascripts/versioning.js
-copyright: miniVERL · inspectable single-GPU post-training · Apache-2.0
+copyright: miniVERL · verl semantics on one GPU · Apache-2.0
diff --git a/pyproject.toml b/pyproject.toml
index ff9b548..628e20a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -5,7 +5,7 @@ build-backend = "hatchling.build"
[project]
name = "miniverl"
dynamic = ["version"]
-description = "Inspectable verl-style on-policy distillation on one NVIDIA GPU."
+description = "Verl experiment semantics compiled for one NVIDIA GPU."
readme = "PYPI.md"
requires-python = ">=3.10"
license = { text = "Apache-2.0" }
@@ -15,6 +15,8 @@ keywords = [
"on-policy-distillation",
"knowledge-distillation",
"agentic-rl",
+ "grpo",
+ "rloo",
"tool-use",
"qlora",
"consumer-gpu",
diff --git a/release-state.yaml b/release-state.yaml
index 04fedf3..1d66ef6 100644
--- a/release-state.yaml
+++ b/release-state.yaml
@@ -17,13 +17,13 @@
# such distinction, which is how its tag shipped a docs selector still
# advertising "Stable 0.6.1 / Development 0.6.2.dev0".
schema_version: 1
-phase: development
+phase: release
stable:
- version: "0.11.0"
- tag: "v0.11.0"
- release_commit: "479695c4980150bc6f7a51bddec2b90c08c8bb98"
- released_at: "2026-08-29"
+ version: "0.12.0"
+ tag: "v0.12.0"
+ release_commit: pending
+ released_at: "2026-09-07"
development:
- version: "0.11.1.dev0"
+ version: "0.12.0"
diff --git a/scripts/promote_full_gpu_qualification.py b/scripts/promote_full_gpu_qualification.py
index 58bbd2f..df861e7 100644
--- a/scripts/promote_full_gpu_qualification.py
+++ b/scripts/promote_full_gpu_qualification.py
@@ -31,6 +31,11 @@
"v011_policy_refresh_cache_invalidation",
"v011_external_engine_teardown",
)
+_V012_CHECKS = (
+ "v012_pinned_verl_v09_compiler",
+ "v012_grpo_nonconstant_reward_update",
+ "v012_exact_wheel_rl_runtime",
+)
_MAX_GPU_MEMORY_MIB = 14.5 * 1024
_PREREGISTRATION_SHA256 = "8cc3ba738c69b59ed19c22c1de874fd00249404198a3e05983477dc8899bb7e5"
_FROZEN_CALCULATOR_SHA256 = "53fc1d4d5b7adee09618d77ad62d4086ba56b78569832d6fc7c3bcd5c2695bbc"
@@ -92,6 +97,96 @@ def _is_v011(version: str) -> bool:
return False
+def _is_v012(version: str) -> bool:
+ match = version.split(".", 2)
+ try:
+ return (int(match[0]), int(match[1])) >= (0, 12)
+ except (IndexError, ValueError):
+ return False
+
+
+def _validate_v012_rl(payload: dict[str, Any], qualification: GPUQualification) -> None:
+ if payload.get("schema_version") != 1 or payload.get("status") != "passed":
+ raise ValueError("v0.12 RL: unsupported schema or failed status")
+ if payload.get("kind") != "miniverl_v012_rl_qualification":
+ raise ValueError("v0.12 RL: unexpected evidence kind")
+ if payload.get("source_commit") != qualification.source_commit:
+ raise ValueError("v0.12 RL: source commit does not match release smoke")
+ if payload.get("miniverl_version") != qualification.miniverl_version:
+ raise ValueError("v0.12 RL: miniVERL version does not match release smoke")
+ if payload.get("wheel_sha256") != qualification.wheel.sha256:
+ raise ValueError("v0.12 RL: wheel binding does not match release smoke")
+ hardware = payload.get("hardware") or {}
+ if hardware.get("gpu") != qualification.environment.gpu_name or hardware.get("gpu_count") != 1:
+ raise ValueError("v0.12 RL: hardware does not match release smoke")
+ if "microsoft" not in str(hardware.get("platform", "")).lower():
+ raise ValueError("v0.12 RL: execution was not measured under WSL2")
+ for field, expected in (
+ ("python", qualification.environment.python),
+ ("cuda_runtime", qualification.environment.cuda_runtime),
+ ("driver", qualification.environment.driver),
+ ("packages", qualification.environment.packages),
+ ):
+ if hardware.get(field) != expected:
+ raise ValueError(f"v0.12 RL: environment {field} does not match release smoke")
+ upstream = payload.get("upstream") or {}
+ if (
+ upstream.get("tag") != "v0.9.0"
+ or upstream.get("commit") != "483b8a009ba3a97563edee3a19887e4862b8094a"
+ or upstream.get("profile") != "verl-rl-v0.9-single-gpu-v1"
+ or upstream.get("compiler_status") != "accepted"
+ or upstream.get("generated_recipe_validated") is not True
+ ):
+ raise ValueError("v0.12 RL: pinned v0.9 compiler evidence did not pass")
+ workload = payload.get("workload") or {}
+ if (
+ workload.get("model_id") != "Qwen/Qwen3-0.6B"
+ or workload.get("algorithm") != "grpo"
+ or workload.get("reward_provider") != "builtin_target_length"
+ or workload.get("rollout_backend") != "hf_cached"
+ or workload.get("cycles") != 2
+ or workload.get("prompts_per_cycle") != 2
+ or workload.get("samples_per_prompt") != 4
+ or workload.get("trajectories") != 16
+ or workload.get("optimizer_updates") != 2
+ or workload.get("policy_version") != 2
+ or int(workload.get("selected_positions", 0)) < 1
+ ):
+ raise ValueError("v0.12 RL: workload shape or optimizer updates are incomplete")
+ if float(workload.get("max_within_prompt_reward_variance", 0.0)) <= 0.0 or float(
+ workload.get("reward_max", 0.0)
+ ) <= float(workload.get("reward_min", 0.0)):
+ raise ValueError("v0.12 RL: nonconstant reward variation was not measured")
+ if float(workload.get("max_absolute_advantage", 0.0)) <= 0.0:
+ raise ValueError("v0.12 RL: nonzero GRPO advantages were not measured")
+ losses = workload.get("losses") or []
+ if len(losses) != 2 or any(not math.isfinite(float(value)) for value in losses):
+ raise ValueError("v0.12 RL: optimizer losses are incomplete or non-finite")
+ initial = workload.get("initial_trainable_state_sha256")
+ final = workload.get("final_trainable_state_sha256")
+ if (
+ not isinstance(initial, str)
+ or not isinstance(final, str)
+ or len(initial) != 64
+ or len(final) != 64
+ or initial == final
+ ):
+ raise ValueError("v0.12 RL: actor parameters did not change")
+ resource = payload.get("resource_contract") or {}
+ if (
+ resource.get("peak_reserved_within_limit") is not True
+ or float(resource.get("peak_reserved_gib", math.inf)) > 14.5
+ ):
+ raise ValueError("v0.12 RL: VRAM resource contract did not pass")
+ scope = payload.get("scientific_scope") or {}
+ if (
+ scope.get("runtime_correctness_only") is not True
+ or scope.get("task_quality_evaluated") is not False
+ or scope.get("distributed_execution_tested") is not False
+ ):
+ raise ValueError("v0.12 RL: scientific scope is not fail-closed")
+
+
def _validate_v011_profiles(payload: dict[str, Any], qualification: GPUQualification) -> None:
if payload.get("schema_version") != 1 or payload.get("status") != "passed":
raise ValueError("v0.11 profiles: unsupported schema or failed status")
@@ -295,6 +390,7 @@ def promote(
hf_cached_runtime: Path | None = None,
vllm_runtime: Path | None = None,
hf_reference: Path | None = None,
+ v012_rl: Path | None = None,
) -> GPUQualification:
problems = validate_qualification_file(qualification_path)
if problems:
@@ -325,6 +421,12 @@ def promote(
"hf_cached_runtime": hf_cached_runtime,
"vllm_runtime": vllm_runtime,
}
+ v012_sources: dict[str, Path] = {}
+ if _is_v012(qualification.miniverl_version):
+ if v012_rl is None:
+ raise ValueError("v0.12 full qualification requires RL evidence")
+ _validate_v012_rl(_load(v012_rl), qualification)
+ v012_sources = {"v012_rl": v012_rl}
root = qualification_path.parent
destination = root / "full"
@@ -340,6 +442,10 @@ def promote(
target = destination / f"{name.replace('_', '-')}.json"
shutil.copy2(source, target)
additions.append((f"full_{name}_result", target))
+ for name, source in v012_sources.items():
+ target = destination / f"{name.replace('_', '-')}.json"
+ shutil.copy2(source, target)
+ additions.append((f"full_{name}_result", target))
for name, path in additions:
payload["artifacts"].append(
{
@@ -360,6 +466,8 @@ def promote(
)
if v011_sources:
payload["checks"]["executed"].extend(_V011_CHECKS)
+ if v012_sources:
+ payload["checks"]["executed"].extend(_V012_CHECKS)
promoted = GPUQualification.model_validate(payload)
write_json_atomic(qualification_path, promoted.model_dump(mode="json"))
final_problems = validate_qualification_file(qualification_path)
@@ -378,6 +486,7 @@ def main() -> int:
parser.add_argument("--hf-cached-runtime", type=Path)
parser.add_argument("--vllm-runtime", type=Path)
parser.add_argument("--hf-reference", type=Path)
+ parser.add_argument("--v012-rl", type=Path)
args = parser.parse_args()
promoted = promote(
args.qualification,
@@ -388,6 +497,7 @@ def main() -> int:
hf_cached_runtime=args.hf_cached_runtime,
vllm_runtime=args.vllm_runtime,
hf_reference=args.hf_reference,
+ v012_rl=args.v012_rl,
)
print(json.dumps(promoted.model_dump(mode="json"), sort_keys=True, allow_nan=False))
return 0
diff --git a/scripts/publish_verl_rl_v09_compatibility.py b/scripts/publish_verl_rl_v09_compatibility.py
new file mode 100644
index 0000000..226e2aa
--- /dev/null
+++ b/scripts/publish_verl_rl_v09_compatibility.py
@@ -0,0 +1,46 @@
+"""Publish the deterministic compatibility report for the verl v0.9 RL example."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import tempfile
+from pathlib import Path
+
+from miniverl.bridge.rl_v09 import publish_imported_verl_rl_v09
+
+ROOT = Path(__file__).resolve().parents[1]
+SOURCE = ROOT / "examples/verl-rl-v0.9-single-gpu.yaml"
+TARGET = ROOT / "docs/generated/verl-rl-v0.9-compatibility.json"
+
+
+def render() -> str:
+ with tempfile.TemporaryDirectory(prefix="miniverl-rl-v09-") as directory:
+ output = Path(directory) / "verl-rl-v0.9-single-gpu.yaml"
+ report = publish_imported_verl_rl_v09(
+ SOURCE,
+ out=output,
+ target_verl="v0.9.0",
+ )
+ if report["status"] != "accepted" or report["executable"] is not True:
+ raise RuntimeError("the committed verl v0.9 RL example no longer compiles")
+ return json.dumps(report, indent=2, sort_keys=True, ensure_ascii=False) + "\n"
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--check", action="store_true")
+ args = parser.parse_args()
+ expected = render()
+ if args.check:
+ if not TARGET.is_file() or TARGET.read_text(encoding="utf-8") != expected:
+ parser.error(f"generated compatibility report is stale: {TARGET.relative_to(ROOT)}")
+ return 0
+ TARGET.parent.mkdir(parents=True, exist_ok=True)
+ TARGET.write_text(expected, encoding="utf-8", newline="\n")
+ print(TARGET.relative_to(ROOT).as_posix())
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/release_gate.py b/scripts/release_gate.py
index cb3e289..f5a5056 100644
--- a/scripts/release_gate.py
+++ b/scripts/release_gate.py
@@ -79,6 +79,10 @@ def gate_plan(
"generated_compatibility",
(python, "scripts/publish_verl_opd_compatibility.py", "--check"),
),
+ Gate(
+ "generated_rl_v09_compatibility",
+ (python, "scripts/publish_verl_rl_v09_compatibility.py", "--check"),
+ ),
Gate(
"generated_hardware_schema",
(python, "scripts/publish_hardware_record_schema.py", "--check"),
diff --git a/scripts/run_v012_rl_qualification.py b/scripts/run_v012_rl_qualification.py
new file mode 100644
index 0000000..b6ae129
--- /dev/null
+++ b/scripts/run_v012_rl_qualification.py
@@ -0,0 +1,389 @@
+"""Run the exact-wheel v0.12 GRPO qualification on one CUDA GPU.
+
+The workload compiles a pinned verl v0.9-shaped profile, executes two real
+Qwen3-0.6B NF4 LoRA rollout/update cycles, and records systems evidence. Its
+target-length reward is a deterministic formatting objective, not a task-
+quality benchmark.
+"""
+
+from __future__ import annotations
+
+import argparse
+import gc
+import hashlib
+import importlib.metadata
+import json
+import math
+import platform
+import subprocess
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+from miniverl import __version__
+from miniverl.algorithms.contract import UPSTREAM_VERL_COMMIT, UPSTREAM_VERL_TAG
+from miniverl.bridge.rl_v09 import VERL_RL_V09_PROFILE, publish_imported_verl_rl_v09
+from miniverl.config import RunConfig
+from miniverl.trainer import OPDTrainer
+from miniverl.trajectory.io import read_trajectories
+from miniverl.utils.runs import read_jsonl, write_json_atomic
+
+MODEL_ID = "Qwen/Qwen3-0.6B"
+MODEL_REVISION = "c1899de289a04d12100db370d81485cdf75e47ca"
+PROMPTS_PER_CYCLE = 2
+SAMPLES_PER_PROMPT = 4
+CYCLES = 2
+MAX_PEAK_RESERVED_GIB = 14.5
+
+
+def _sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def _state_digest(trainer: OPDTrainer) -> str:
+ import torch
+
+ digest = hashlib.sha256()
+ for name, tensor in sorted(trainer.student.trainable_state_dict().items()):
+ value = tensor.detach().cpu().contiguous()
+ digest.update(name.encode("utf-8"))
+ digest.update(str(tuple(value.shape)).encode("ascii"))
+ digest.update(str(value.dtype).encode("ascii"))
+ digest.update(value.view(dtype=torch.uint8).numpy().tobytes())
+ return digest.hexdigest()
+
+
+def _write_dataset(path: Path) -> str:
+ import pyarrow as pa
+ import pyarrow.parquet as pq
+
+ rows = []
+ for index, target in enumerate((80, 100, 120, 140), start=1):
+ rows.append(
+ {
+ "prompt": [
+ {
+ "role": "system",
+ "content": "Follow the requested response-length constraint. /no_think",
+ },
+ {
+ "role": "user",
+ "content": (
+ f"Write one plain-text sentence about local experiment {index}. "
+ f"Aim for exactly {target} characters. /no_think"
+ ),
+ },
+ ],
+ "data_source": "miniverl_v012_systems_qualification",
+ "ability": "target_length",
+ "reward_model": {"style": "target_length", "characters": target},
+ "extra_info": {"qualification_item": index, "target_characters": target},
+ }
+ )
+ path.parent.mkdir(parents=True, exist_ok=True)
+ pq.write_table(pa.Table.from_pylist(rows), path, row_group_size=PROMPTS_PER_CYCLE)
+ return _sha256(path)
+
+
+def _source_payload(dataset: Path) -> dict[str, Any]:
+ return {
+ "data": {
+ "train_files": [dataset.resolve().as_posix()],
+ "val_files": [],
+ "prompt_key": "prompt",
+ "train_batch_size": PROMPTS_PER_CYCLE,
+ "max_prompt_length": 128,
+ "max_response_length": 32,
+ "truncation": "error",
+ "shuffle": False,
+ "seed": 1207,
+ },
+ "actor_rollout_ref": {
+ "model": {
+ "path": MODEL_ID,
+ "enable_gradient_checkpointing": True,
+ "lora_rank": 16,
+ "lora_alpha": 32,
+ "target_modules": [
+ "q_proj",
+ "k_proj",
+ "v_proj",
+ "o_proj",
+ "gate_proj",
+ "up_proj",
+ "down_proj",
+ ],
+ },
+ "actor": {
+ "optim": {"lr": 1.0e-5, "weight_decay": 0.01, "lr_warmup_steps": 0},
+ "ppo_mini_batch_size": PROMPTS_PER_CYCLE * SAMPLES_PER_PROMPT,
+ "ppo_epochs": 1,
+ "loss_agg_mode": "token-mean",
+ "clip_ratio": 0.2,
+ "clip_ratio_low": 0.2,
+ "clip_ratio_high": 0.2,
+ "clip_ratio_c": 3.0,
+ "use_kl_loss": False,
+ "entropy_coeff": 0.0,
+ },
+ "rollout": {
+ "name": "vllm",
+ "n": SAMPLES_PER_PROMPT,
+ "temperature": 1.0,
+ "top_p": 1.0,
+ "top_k": 0,
+ "tensor_model_parallel_size": 8,
+ "gpu_memory_utilization": 0.6,
+ },
+ },
+ "algorithm": {
+ "adv_estimator": "grpo",
+ "norm_adv_by_std_in_grpo": True,
+ "gamma": 1.0,
+ "lam": 1.0,
+ "use_kl_in_reward": False,
+ },
+ "trainer": {
+ "total_training_steps": CYCLES,
+ "total_epochs": 1,
+ "project_name": "mini-verl",
+ "experiment_name": "v012-grpo-qualification",
+ "save_freq": 1,
+ "test_freq": -1,
+ "logger": ["console"],
+ "n_gpus_per_node": 8,
+ "nnodes": 2,
+ },
+ "miniverl": {
+ "reward": {"provider": "target_length"},
+ "actor": {
+ "lora_enabled": True,
+ "revision": MODEL_REVISION,
+ "tokenizer_revision": MODEL_REVISION,
+ "dtype": "auto",
+ "quantization": "nf4",
+ },
+ "rollout": {"backend": "hf_cached"},
+ "memory": {"strategy": "resident"},
+ "update": {"trajectory_batch_size": 1},
+ },
+ }
+
+
+def _driver_version() -> str:
+ return subprocess.run(
+ ["nvidia-smi", "--query-gpu=driver_version", "--format=csv,noheader"],
+ check=True,
+ capture_output=True,
+ text=True,
+ ).stdout.splitlines()[0]
+
+
+def run(args: argparse.Namespace) -> dict[str, Any]:
+ import bitsandbytes.functional as bnb_functional
+ import torch
+
+ if not torch.cuda.is_available() or torch.cuda.device_count() != 1:
+ raise RuntimeError("v0.12 RL qualification requires exactly one CUDA GPU")
+ if _sha256(args.wheel) != args.wheel_sha256:
+ raise RuntimeError("candidate wheel checksum does not match the declared binding")
+ head = subprocess.run(
+ ["git", "rev-parse", "HEAD"], check=True, capture_output=True, text=True
+ ).stdout.strip()
+ if head != args.commit:
+ raise RuntimeError("checkout HEAD does not match the candidate source commit")
+ if subprocess.run(
+ ["git", "status", "--porcelain"], check=True, capture_output=True, text=True
+ ).stdout:
+ raise RuntimeError("v0.12 RL qualification requires a clean source checkout")
+
+ args.work.mkdir(parents=True, exist_ok=False)
+ dataset = args.work / "qualification.parquet"
+ dataset_sha256 = _write_dataset(dataset)
+ source = args.work / "resolved-verl-v0.9.yaml"
+ source.write_text(
+ yaml.safe_dump(_source_payload(dataset), sort_keys=False, allow_unicode=True),
+ encoding="utf-8",
+ newline="\n",
+ )
+ recipe = args.work / "native.yaml"
+ report = publish_imported_verl_rl_v09(
+ source,
+ out=recipe,
+ target_verl=UPSTREAM_VERL_COMMIT,
+ )
+ if report.get("status") != "accepted" or report.get("executable") is not True:
+ raise RuntimeError("pinned verl v0.9 profile did not compile to an executable recipe")
+ config = RunConfig.from_yaml(recipe)
+ trainer = OPDTrainer.from_config(
+ config,
+ output_dir=args.work / "runs",
+ run_id="qualification",
+ local_files_only=args.offline,
+ )
+ try:
+ initial_state_sha256 = _state_digest(trainer)
+ torch.cuda.reset_peak_memory_stats()
+ result = trainer.train()
+ final_state_sha256 = _state_digest(trainer)
+ peak_reserved_gib = torch.cuda.max_memory_reserved() / 1024**3
+ finally:
+ trainer.close()
+
+ if result.global_step != CYCLES or result.policy_version != CYCLES:
+ raise RuntimeError("GRPO qualification did not commit two policy updates")
+ if initial_state_sha256 == final_state_sha256:
+ raise RuntimeError("GRPO qualification did not change the trainable actor parameters")
+ if peak_reserved_gib > MAX_PEAK_RESERVED_GIB:
+ raise RuntimeError("GRPO qualification exceeded the 14.5 GiB release limit")
+
+ trajectories = read_trajectories(result.run_dir / "trajectories.jsonl")
+ rewards = read_jsonl(result.run_dir / "rewards.jsonl")
+ advantages = read_jsonl(result.run_dir / "advantages.jsonl")
+ metrics = read_jsonl(result.run_dir / "metrics.jsonl")
+ updates = [row for row in metrics if row.get("phase") == "rl"]
+ cycles = [row for row in metrics if row.get("phase") == "rl_cycle"]
+ expected_trajectories = CYCLES * PROMPTS_PER_CYCLE * SAMPLES_PER_PROMPT
+ if not (len(trajectories) == len(rewards) == len(advantages) == expected_trajectories):
+ raise RuntimeError("GRPO qualification produced an incomplete reward/trajectory set")
+ if len(updates) != CYCLES or len(cycles) != CYCLES:
+ raise RuntimeError("GRPO qualification produced an incomplete optimizer/cycle record")
+ reward_values = [float(row["raw_reward"]) for row in rewards]
+ advantage_values = [
+ value for row in advantages for value in row.get("advantages", []) if math.isfinite(value)
+ ]
+ variances = [
+ float(value)
+ for cycle in cycles
+ for value in (
+ ((cycle.get("grouped_rollouts") or {}).get("per_group_reward_variance") or {}).get(
+ "values", {}
+ )
+ ).values()
+ ]
+ if not variances or max(variances) <= 0.0:
+ raise RuntimeError("GRPO qualification observed no within-prompt reward variation")
+ if not advantage_values or max(map(abs, advantage_values)) <= 0.0:
+ raise RuntimeError("GRPO qualification observed no nonzero policy advantage")
+ if any(
+ not math.isfinite(float(row["loss"]))
+ or int(row["selected_positions"]) < 1
+ or float((row.get("verl_rl") or {}).get("advantage_std", 0.0)) <= 0.0
+ for row in updates
+ ):
+ raise RuntimeError("GRPO optimizer update was empty, constant, or non-finite")
+ checkpoint = result.run_dir / "checkpoints" / "final"
+ checkpoint_files = ("adapter.safetensors", "optimizer.safetensors", "state.json")
+ if any(not (checkpoint / name).is_file() for name in checkpoint_files):
+ raise RuntimeError("GRPO qualification final checkpoint is incomplete")
+
+ bnb_functional.name2qmap.clear()
+ torch._C._cuda_clearCublasWorkspaces()
+ gc.collect()
+ torch.cuda.empty_cache()
+ torch.cuda.synchronize()
+ payload = {
+ "schema_version": 1,
+ "kind": "miniverl_v012_rl_qualification",
+ "status": "passed",
+ "measured_at": datetime.now(timezone.utc)
+ .replace(microsecond=0)
+ .isoformat()
+ .replace("+00:00", "Z"),
+ "source_commit": args.commit,
+ "miniverl_version": __version__,
+ "wheel_sha256": args.wheel_sha256,
+ "hardware": {
+ "gpu": torch.cuda.get_device_name(0),
+ "gpu_count": 1,
+ "platform": platform.platform(),
+ "python": platform.python_version(),
+ "cuda_runtime": str(torch.version.cuda),
+ "driver": _driver_version(),
+ "packages": {
+ name: importlib.metadata.version(name)
+ for name in (
+ "torch",
+ "transformers",
+ "peft",
+ "accelerate",
+ "bitsandbytes",
+ "numpy",
+ "pyarrow",
+ "safetensors",
+ )
+ },
+ },
+ "upstream": {
+ "repository": "https://github.com/verl-project/verl",
+ "tag": UPSTREAM_VERL_TAG,
+ "commit": UPSTREAM_VERL_COMMIT,
+ "profile": VERL_RL_V09_PROFILE,
+ "compiler_status": report["status"],
+ "generated_recipe_validated": report["generated_recipe_validated"],
+ "source_config_sha256": report["source_config_sha256"],
+ "generated_recipe_sha256": report["generated_recipe_sha256"],
+ },
+ "workload": {
+ "model_id": MODEL_ID,
+ "model_revision": MODEL_REVISION,
+ "algorithm": "grpo",
+ "reward_provider": "builtin_target_length",
+ "reward_scope": "deterministic response-format systems qualification",
+ "rollout_backend": "hf_cached",
+ "cycles": CYCLES,
+ "prompts_per_cycle": PROMPTS_PER_CYCLE,
+ "samples_per_prompt": SAMPLES_PER_PROMPT,
+ "trajectories": len(trajectories),
+ "generated_tokens": sum(item.generated_token_count for item in trajectories),
+ "optimizer_updates": result.global_step,
+ "policy_version": result.policy_version,
+ "selected_positions": sum(int(row["selected_positions"]) for row in updates),
+ "losses": [float(row["loss"]) for row in updates],
+ "reward_min": min(reward_values),
+ "reward_max": max(reward_values),
+ "max_within_prompt_reward_variance": max(variances),
+ "max_absolute_advantage": max(map(abs, advantage_values)),
+ "initial_trainable_state_sha256": initial_state_sha256,
+ "final_trainable_state_sha256": final_state_sha256,
+ "checkpoint_sha256": {name: _sha256(checkpoint / name) for name in checkpoint_files},
+ },
+ "resource_contract": {
+ "peak_reserved_gib": peak_reserved_gib,
+ "limit_gib": MAX_PEAK_RESERVED_GIB,
+ "peak_reserved_within_limit": True,
+ },
+ "inputs": {"dataset_sha256": dataset_sha256},
+ "scientific_scope": {
+ "runtime_correctness_only": True,
+ "task_quality_evaluated": False,
+ "distributed_execution_tested": False,
+ "other_hardware_measured": False,
+ },
+ }
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ write_json_atomic(args.output, payload)
+ return payload
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--commit", required=True)
+ parser.add_argument("--wheel", required=True, type=Path)
+ parser.add_argument("--wheel-sha256", required=True)
+ parser.add_argument("--output", required=True, type=Path)
+ parser.add_argument("--work", required=True, type=Path)
+ parser.add_argument("--offline", action="store_true")
+ args = parser.parse_args()
+ print(json.dumps(run(args), sort_keys=True, allow_nan=False))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/src/miniverl/__init__.py b/src/miniverl/__init__.py
index 9d37dba..81903cd 100644
--- a/src/miniverl/__init__.py
+++ b/src/miniverl/__init__.py
@@ -14,6 +14,6 @@
from __future__ import annotations
-__version__ = "0.11.1.dev0"
+__version__ = "0.12.0"
__all__ = ["__version__"]
diff --git a/src/miniverl/agent/loop.py b/src/miniverl/agent/loop.py
index 5e60190..10e551e 100644
--- a/src/miniverl/agent/loop.py
+++ b/src/miniverl/agent/loop.py
@@ -432,6 +432,7 @@ def rollout(
final_answers_format_valid = 0
final_answers_verified = 0
generated_tokens = 0
+ generated_logprobs: list[float] = []
verification: VerificationRecord | None = None
metadata_error: str | None = None
termination = TerminationReason.MAX_TURNS
@@ -448,8 +449,19 @@ def rollout(
temperature=sample_temperature,
top_p=cfg.top_p,
top_k=cfg.top_k,
- seed=seed * 1_000_003 + turn_id,
+ # ``derive_sample_seed`` already occupies the full signed
+ # 63-bit range. Keep the historical turn-mixing formula for
+ # ordinary seeds while bounding it for torch.Generator.
+ seed=(seed * 1_000_003 + turn_id) & ((1 << 63) - 1),
+ record_logprobs=cfg.record_logprobs,
)
+ if cfg.record_logprobs:
+ if len(generation.logprobs) != len(generation.token_ids):
+ raise ToolEnvironmentError(
+ "agent rollout requested behavior log-probabilities but the backend "
+ "returned a misaligned result"
+ )
+ generated_logprobs.extend(generation.logprobs)
if not generation.token_ids:
termination = TerminationReason.EOS_WITHOUT_FINAL
break
@@ -699,6 +711,7 @@ def rollout(
"split": task.split,
"initial_observation_state_id": initial_observation.state_id,
**self.environment.trajectory_metadata(task),
+ **({"actor_rollout_log_probs": generated_logprobs} if cfg.record_logprobs else {}),
**({"environment_error": metadata_error} if metadata_error else {}),
},
)
diff --git a/src/miniverl/algorithms/__init__.py b/src/miniverl/algorithms/__init__.py
new file mode 100644
index 0000000..6857118
--- /dev/null
+++ b/src/miniverl/algorithms/__init__.py
@@ -0,0 +1,57 @@
+"""Composable verl-conformant single-GPU RL mathematical primitives.
+
+The implementations use :mod:`torch`, but importing this package remains safe
+for the dependency-light core install. Individual symbols are resolved only
+when used, following the same contract as :mod:`miniverl.losses`.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from miniverl.errors import MissingDependencyError
+from miniverl.utils.lazy import have_module
+
+__all__ = [
+ "AdvantageOutput",
+ "KLPenalty",
+ "PolicyLossOutput",
+ "ValueLossOutput",
+ "clipped_policy_loss",
+ "clipped_value_loss",
+ "gae_advantage_return",
+ "grpo_outcome_advantage",
+ "reinforce_plus_plus_advantage",
+ "reference_kl_penalty",
+ "rloo_outcome_advantage",
+]
+
+_MODULE_OF = {
+ "AdvantageOutput": "advantages",
+ "gae_advantage_return": "advantages",
+ "grpo_outcome_advantage": "advantages",
+ "reinforce_plus_plus_advantage": "advantages",
+ "rloo_outcome_advantage": "advantages",
+ "KLPenalty": "kl",
+ "reference_kl_penalty": "kl",
+ "PolicyLossOutput": "policy",
+ "clipped_policy_loss": "policy",
+ "ValueLossOutput": "value",
+ "clipped_value_loss": "value",
+}
+
+
+def __getattr__(name: str) -> Any:
+ submodule = _MODULE_OF.get(name)
+ if submodule is None:
+ raise AttributeError(f"module 'miniverl.algorithms' has no attribute {name!r}")
+ if not have_module("torch"):
+ raise MissingDependencyError("torch", "train", f"miniverl.algorithms.{name}")
+ import importlib
+
+ module = importlib.import_module(f"miniverl.algorithms.{submodule}")
+ return getattr(module, name)
+
+
+def __dir__() -> list[str]:
+ return sorted(__all__)
diff --git a/src/miniverl/algorithms/advantages.py b/src/miniverl/algorithms/advantages.py
new file mode 100644
index 0000000..f7bf4fd
--- /dev/null
+++ b/src/miniverl/algorithms/advantages.py
@@ -0,0 +1,177 @@
+"""Advantage and return estimators matching pinned verl v0.9.0 semantics."""
+
+from __future__ import annotations
+
+from collections import Counter, defaultdict
+from collections.abc import Sequence
+from dataclasses import dataclass
+
+import torch
+
+
+@dataclass(frozen=True)
+class AdvantageOutput:
+ advantages: torch.Tensor
+ returns: torch.Tensor
+ sequence_advantages: torch.Tensor | None = None
+ zero_variance_groups: int = 0
+
+
+def _validate_batch(
+ token_level_rewards: torch.Tensor,
+ response_mask: torch.Tensor,
+) -> None:
+ if token_level_rewards.ndim != 2 or response_mask.shape != token_level_rewards.shape:
+ raise ValueError(
+ "token_level_rewards and response_mask must have identical [batch, time] shape"
+ )
+ if not torch.isfinite(token_level_rewards).all() or not torch.isfinite(response_mask).all():
+ raise ValueError("rewards and masks must be finite")
+ if bool((response_mask < 0).any()) or bool((response_mask > 1).any()):
+ raise ValueError("response_mask must contain values in [0, 1]")
+ if bool((response_mask.sum(dim=-1) == 0).any()):
+ raise ValueError("every response must contain at least one model token")
+
+
+def _validate_groups(groups: Sequence[str], batch_size: int) -> dict[str, list[int]]:
+ if len(groups) != batch_size:
+ raise ValueError("group identities must align with the response batch")
+ counts = Counter(groups)
+ if any(count < 2 for count in counts.values()):
+ raise ValueError("group estimators require at least two complete samples per prompt")
+ indices: dict[str, list[int]] = defaultdict(list)
+ for index, group in enumerate(groups):
+ if not group:
+ raise ValueError("prompt group identity cannot be empty")
+ indices[group].append(index)
+ return dict(indices)
+
+
+def _masked_mean(values: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
+ return torch.where(mask.bool(), values, 0.0).sum() / (mask.sum() + 1e-8)
+
+
+def _masked_whiten(values: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
+ mean = _masked_mean(values, mask)
+ centered = values - mean
+ variance = _masked_mean(centered.square(), mask)
+ count = mask.sum()
+ if bool(count <= 1):
+ raise ValueError("advantage whitening requires at least two selected tokens")
+ variance = variance * count / (count - 1)
+ return centered * torch.rsqrt(variance + 1e-8)
+
+
+def grpo_outcome_advantage(
+ token_level_rewards: torch.Tensor,
+ response_mask: torch.Tensor,
+ groups: Sequence[str],
+ *,
+ epsilon: float = 1e-6,
+ normalize_by_std: bool = True,
+) -> AdvantageOutput:
+ """Compute upstream GRPO, or Dr.GRPO when ``normalize_by_std`` is false.
+
+ verl v0.9.0 uses ``torch.std``'s sample standard deviation within each
+ prompt group and adds epsilon after the standard deviation.
+ """
+ _validate_batch(token_level_rewards, response_mask)
+ if epsilon <= 0:
+ raise ValueError("epsilon must be positive")
+ grouped = _validate_groups(groups, token_level_rewards.shape[0])
+ scores = token_level_rewards.sum(dim=-1).detach().clone()
+ sequence = torch.empty_like(scores)
+ zero_variance = 0
+ for member_indices in grouped.values():
+ index = torch.tensor(member_indices, device=scores.device)
+ values = scores[index]
+ centered = values - values.mean()
+ std = values.std()
+ if bool(std == 0):
+ zero_variance += 1
+ sequence[index] = centered / (std + epsilon) if normalize_by_std else centered
+ advantages = sequence.unsqueeze(-1) * response_mask
+ return AdvantageOutput(
+ advantages=advantages,
+ returns=advantages,
+ sequence_advantages=sequence,
+ zero_variance_groups=zero_variance,
+ )
+
+
+def rloo_outcome_advantage(
+ token_level_rewards: torch.Tensor,
+ response_mask: torch.Tensor,
+ groups: Sequence[str],
+) -> AdvantageOutput:
+ """Compute verl v0.9.0 leave-one-out outcome advantages."""
+ _validate_batch(token_level_rewards, response_mask)
+ grouped = _validate_groups(groups, token_level_rewards.shape[0])
+ scores = token_level_rewards.sum(dim=-1).detach().clone()
+ sequence = torch.empty_like(scores)
+ for member_indices in grouped.values():
+ index = torch.tensor(member_indices, device=scores.device)
+ values = scores[index]
+ sequence[index] = values - (values.sum() - values) / (len(member_indices) - 1)
+ advantages = sequence.unsqueeze(-1) * response_mask
+ return AdvantageOutput(advantages, advantages, sequence)
+
+
+def reinforce_plus_plus_advantage(
+ token_level_rewards: torch.Tensor,
+ response_mask: torch.Tensor,
+ *,
+ gamma: float = 1.0,
+) -> AdvantageOutput:
+ """Compute discounted REINFORCE++ returns and globally whiten advantages."""
+ _validate_batch(token_level_rewards, response_mask)
+ if not 0 <= gamma <= 1:
+ raise ValueError("gamma must be in [0, 1]")
+ with torch.no_grad():
+ returns = torch.zeros_like(token_level_rewards)
+ running_return: torch.Tensor | int = 0
+ for position in reversed(range(token_level_rewards.shape[1])):
+ candidate = token_level_rewards[:, position] + gamma * running_return
+ returns[:, position] = candidate * response_mask[:, position]
+ running_return = candidate * response_mask[:, position] + running_return * (
+ 1 - response_mask[:, position]
+ )
+ advantages = _masked_whiten(returns, response_mask) * response_mask
+ return AdvantageOutput(advantages=advantages, returns=returns)
+
+
+def gae_advantage_return(
+ token_level_rewards: torch.Tensor,
+ values: torch.Tensor,
+ response_mask: torch.Tensor,
+ *,
+ gamma: float = 1.0,
+ lam: float = 1.0,
+) -> AdvantageOutput:
+ """Compute verl v0.9.0 GAE, including observation-token carry semantics."""
+ _validate_batch(token_level_rewards, response_mask)
+ if values.shape != token_level_rewards.shape or not torch.isfinite(values).all():
+ raise ValueError("values must be a finite tensor aligned with rewards")
+ if not 0 <= gamma <= 1 or not 0 <= lam <= 1:
+ raise ValueError("gamma and lam must be in [0, 1]")
+ with torch.no_grad():
+ next_values: torch.Tensor | int = 0
+ last_gae: torch.Tensor | int = 0
+ reversed_advantages: list[torch.Tensor] = []
+ for position in reversed(range(token_level_rewards.shape[1])):
+ delta = token_level_rewards[:, position] + gamma * next_values - values[:, position]
+ candidate = delta + gamma * lam * last_gae
+ next_values = (
+ values[:, position] * response_mask[:, position]
+ + (1 - response_mask[:, position]) * next_values
+ )
+ last_gae = (
+ candidate * response_mask[:, position] + (1 - response_mask[:, position]) * last_gae
+ )
+ reversed_advantages.append(last_gae)
+ raw_advantages = torch.stack(reversed_advantages[::-1], dim=1)
+ returns = raw_advantages + values
+ # Keep upstream's values outside the mask byte-for-byte; consumers mask
+ # them in policy/value aggregation.
+ advantages = _masked_whiten(raw_advantages, response_mask)
+ return AdvantageOutput(advantages=advantages, returns=returns)
diff --git a/src/miniverl/algorithms/contract.py b/src/miniverl/algorithms/contract.py
new file mode 100644
index 0000000..44699ba
--- /dev/null
+++ b/src/miniverl/algorithms/contract.py
@@ -0,0 +1,13 @@
+"""Torch-free identity for the pinned verl RL semantics."""
+
+from __future__ import annotations
+
+UPSTREAM_VERL_TAG = "v0.9.0"
+UPSTREAM_VERL_COMMIT = "483b8a009ba3a97563edee3a19887e4862b8094a"
+ADVANTAGE_IMPLEMENTATION_VERSION = "verl-v0.9-advantages-v1"
+
+__all__ = [
+ "ADVANTAGE_IMPLEMENTATION_VERSION",
+ "UPSTREAM_VERL_COMMIT",
+ "UPSTREAM_VERL_TAG",
+]
diff --git a/src/miniverl/algorithms/kl.py b/src/miniverl/algorithms/kl.py
new file mode 100644
index 0000000..d539538
--- /dev/null
+++ b/src/miniverl/algorithms/kl.py
@@ -0,0 +1,33 @@
+"""Reference-policy KL estimators matching pinned verl v0.9.0."""
+
+from __future__ import annotations
+
+from typing import Literal
+
+import torch
+
+KLPenalty = Literal["kl", "abs", "mse", "low_var_kl"]
+
+
+def reference_kl_penalty(
+ log_probs: torch.Tensor,
+ reference_log_probs: torch.Tensor,
+ *,
+ penalty: KLPenalty = "kl",
+) -> torch.Tensor:
+ """Return verl's sampled-token KL estimator for a fixed reference policy."""
+ if log_probs.shape != reference_log_probs.shape:
+ raise ValueError("policy and reference log-probabilities must have identical shape")
+ if not torch.isfinite(log_probs).all() or not torch.isfinite(reference_log_probs).all():
+ raise ValueError("policy and reference log-probabilities must be finite")
+ difference = log_probs - reference_log_probs
+ if penalty == "kl":
+ return difference
+ if penalty == "abs":
+ return difference.abs()
+ if penalty == "mse":
+ return 0.5 * difference.square()
+ if penalty == "low_var_kl":
+ reverse = (-difference).clamp(-20.0, 20.0)
+ return (torch.exp(reverse) - reverse - 1.0).clamp(-10.0, 10.0)
+ raise ValueError(f"unsupported reference KL penalty {penalty!r}")
diff --git a/src/miniverl/algorithms/policy.py b/src/miniverl/algorithms/policy.py
new file mode 100644
index 0000000..6d105c1
--- /dev/null
+++ b/src/miniverl/algorithms/policy.py
@@ -0,0 +1,65 @@
+"""Clipped policy objective matching verl v0.9.0 vanilla policy loss."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import torch
+
+
+@dataclass(frozen=True)
+class PolicyLossOutput:
+ loss: torch.Tensor
+ per_token_loss: torch.Tensor
+ ratio: torch.Tensor
+ metrics: dict[str, float]
+
+
+def _masked_mean(values: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
+ return torch.where(mask.bool(), values, 0.0).sum() / (mask.sum() + 1e-8)
+
+
+def clipped_policy_loss(
+ *,
+ current_log_probs: torch.Tensor,
+ old_log_probs: torch.Tensor,
+ advantages: torch.Tensor,
+ response_mask: torch.Tensor,
+ clip_ratio: float = 0.2,
+ clip_ratio_low: float | None = None,
+ clip_ratio_high: float | None = None,
+ clip_ratio_c: float = 3.0,
+) -> PolicyLossOutput:
+ """Return token-mean dual-clipped PPO/GRPO policy loss."""
+ shape = current_log_probs.shape
+ if any(value.shape != shape for value in (old_log_probs, advantages, response_mask)):
+ raise ValueError("policy tensors must have identical shape")
+ if not all(
+ torch.isfinite(value).all() for value in (current_log_probs, old_log_probs, advantages)
+ ):
+ raise ValueError("policy log-probabilities and advantages must be finite")
+ if not bool(response_mask.sum() > 0):
+ raise ValueError("response_mask must select at least one token")
+ if clip_ratio_c <= 1:
+ raise ValueError("clip_ratio_c must be greater than one")
+ low = clip_ratio if clip_ratio_low is None else clip_ratio_low
+ high = clip_ratio if clip_ratio_high is None else clip_ratio_high
+ negative_approx_kl = (current_log_probs - old_log_probs).clamp(-20.0, 20.0)
+ ratio = torch.exp(negative_approx_kl)
+ losses1 = -advantages * ratio
+ losses2 = -advantages * ratio.clamp(1 - low, 1 + high)
+ clipped = torch.maximum(losses1, losses2)
+ dual = torch.minimum(-advantages * clip_ratio_c, clipped)
+ per_token = torch.where(advantages < 0, dual, clipped)
+ loss = _masked_mean(per_token, response_mask)
+ metrics = {
+ "pg_clipfrac": float(_masked_mean((losses2 > losses1).float(), response_mask).detach()),
+ "ppo_kl": float(_masked_mean(-negative_approx_kl, response_mask).detach()),
+ "pg_clipfrac_lower": float(
+ _masked_mean(
+ ((clipped > -advantages * clip_ratio_c) & (advantages < 0)).float(), response_mask
+ ).detach()
+ ),
+ "ratio_mean": float(_masked_mean(ratio, response_mask).detach()),
+ }
+ return PolicyLossOutput(loss=loss, per_token_loss=per_token, ratio=ratio, metrics=metrics)
diff --git a/src/miniverl/algorithms/value.py b/src/miniverl/algorithms/value.py
new file mode 100644
index 0000000..c672efe
--- /dev/null
+++ b/src/miniverl/algorithms/value.py
@@ -0,0 +1,54 @@
+"""Clipped critic objective matching verl's PPO value loss."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import torch
+
+
+@dataclass(frozen=True)
+class ValueLossOutput:
+ loss: torch.Tensor
+ per_token_loss: torch.Tensor
+ metrics: dict[str, float]
+
+
+def clipped_value_loss(
+ current_values: torch.Tensor,
+ old_values: torch.Tensor,
+ returns: torch.Tensor,
+ response_mask: torch.Tensor,
+ *,
+ cliprange_value: float = 0.5,
+) -> ValueLossOutput:
+ """Compute the token-mean maximum of unclipped and clipped value errors."""
+ shape = current_values.shape
+ if any(value.shape != shape for value in (old_values, returns, response_mask)):
+ raise ValueError("value tensors must have identical shape")
+ if not all(torch.isfinite(value).all() for value in (current_values, old_values, returns)):
+ raise ValueError("values and returns must be finite")
+ count = response_mask.sum()
+ if not bool(count > 0):
+ raise ValueError("response_mask must select at least one token")
+ clipped = old_values + (current_values - old_values).clamp(-cliprange_value, cliprange_value)
+ loss_unclipped = (current_values - returns).square()
+ loss_clipped = (clipped - returns).square()
+ per_token = 0.5 * torch.maximum(loss_unclipped, loss_clipped)
+ loss = torch.where(response_mask.bool(), per_token, 0.0).sum() / (count + 1e-8)
+ clipfrac = torch.where(
+ response_mask.bool(), (loss_clipped > loss_unclipped).float(), 0.0
+ ).sum() / (count + 1e-8)
+ return ValueLossOutput(
+ loss=loss,
+ per_token_loss=per_token,
+ metrics={
+ "value_loss": float(loss.detach()),
+ "value_clipfrac": float(clipfrac.detach()),
+ "value_mean": float(
+ (
+ torch.where(response_mask.bool(), current_values, 0.0).sum() / (count + 1e-8)
+ ).detach()
+ ),
+ },
+ )
diff --git a/src/miniverl/alignment/workflow.py b/src/miniverl/alignment/workflow.py
index 355cd2e..7220fe0 100644
--- a/src/miniverl/alignment/workflow.py
+++ b/src/miniverl/alignment/workflow.py
@@ -238,10 +238,10 @@ def kl(values: Mapping[str, float]) -> float:
def _teacher_identity(config: RunConfig) -> dict[str, Any] | None:
if config.alignment is None or config.alignment.teacher_mode is None:
return None
- adapter = config.models.teacher.adapter
+ adapter = config.models.required_teacher.adapter
return {
- "id": config.models.teacher.model_id,
- "revision": config.models.teacher.revision,
+ "id": config.models.required_teacher.model_id,
+ "revision": config.models.required_teacher.revision,
"mode": config.alignment.teacher_mode.value,
"adapter_revision": adapter.revision if adapter is not None else None,
}
diff --git a/src/miniverl/bridge/opd_plan.py b/src/miniverl/bridge/opd_plan.py
index dbfaa1d..9c6af11 100644
--- a/src/miniverl/bridge/opd_plan.py
+++ b/src/miniverl/bridge/opd_plan.py
@@ -181,8 +181,8 @@ def build_immutable_opd_plan(
role="student",
)
teacher = _revision(
- native.models.teacher.model_id,
- native.models.teacher.revision,
+ native.models.required_teacher.model_id,
+ native.models.required_teacher.revision,
role="teacher",
)
from miniverl.bridge.profiles import get_profile
diff --git a/src/miniverl/bridge/opd_probe.py b/src/miniverl/bridge/opd_probe.py
index 2af0f50..c6515d6 100644
--- a/src/miniverl/bridge/opd_probe.py
+++ b/src/miniverl/bridge/opd_probe.py
@@ -74,15 +74,15 @@ def _identity(native: RunConfig, *, plan_digest: str, device: dict[str, Any]) ->
"lora": native.models.student.lora.model_dump(mode="json"),
},
"teacher": {
- "model_id": native.models.teacher.model_id,
- "revision": native.models.teacher.revision,
- "tokenizer_id": native.models.teacher.tokenizer_id,
- "tokenizer_revision": native.models.teacher.tokenizer_revision,
- "quantization": native.models.teacher.quantization.value,
- "dtype": native.models.teacher.dtype.value,
+ "model_id": native.models.required_teacher.model_id,
+ "revision": native.models.required_teacher.revision,
+ "tokenizer_id": native.models.required_teacher.tokenizer_id,
+ "tokenizer_revision": native.models.required_teacher.tokenizer_revision,
+ "quantization": native.models.required_teacher.quantization.value,
+ "dtype": native.models.required_teacher.dtype.value,
"adapter": (
- native.models.teacher.adapter.model_dump(mode="json")
- if native.models.teacher.adapter is not None
+ native.models.required_teacher.adapter.model_dump(mode="json")
+ if native.models.required_teacher.adapter is not None
else None
),
},
diff --git a/src/miniverl/bridge/rl_v09.py b/src/miniverl/bridge/rl_v09.py
new file mode 100644
index 0000000..edaa75f
--- /dev/null
+++ b/src/miniverl/bridge/rl_v09.py
@@ -0,0 +1,831 @@
+"""Versioned compiler from a resolved verl v0.9 RL subset to one-GPU miniVERL.
+
+The compiler preserves experiment semantics that miniVERL implements and
+classifies physical distributed settings separately. It never imports Python
+objects named by an input config and never guesses a reward implementation.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import math
+from collections.abc import Mapping
+from pathlib import Path
+from typing import Any, Final, Literal
+
+import yaml
+
+from miniverl.algorithms.contract import (
+ ADVANTAGE_IMPLEMENTATION_VERSION,
+ UPSTREAM_VERL_COMMIT,
+ UPSTREAM_VERL_TAG,
+)
+from miniverl.bridge.interpolation import audit_interpolation
+from miniverl.bridge.publish import (
+ DEFAULT_LOCK_TIMEOUT,
+ OutputTransaction,
+ import_output_targets,
+ reject_source_output_alias,
+)
+from miniverl.errors import ConfigError
+from miniverl.utils.privacy import portable_payload
+
+__all__ = [
+ "VERL_RL_V09_PROFILE",
+ "publish_imported_verl_rl_v09",
+ "rl_v09_field_rules_digest",
+]
+
+VERL_RL_V09_PROFILE: Final = "verl-rl-v0.9-single-gpu-v1"
+VERL_REPOSITORY: Final = "https://github.com/verl-project/verl"
+
+FieldClassification = Literal[
+ "exact",
+ "semantically_conformant",
+ "locally_lowered",
+ "informational_only",
+ "distributed_only",
+ "not_implemented",
+ "unsupported",
+]
+
+
+def _rule(
+ target: str | None,
+ classification: FieldClassification,
+ reason: str,
+) -> tuple[str | None, FieldClassification, str]:
+ return target, classification, reason
+
+
+_RULES: dict[str, tuple[str | None, FieldClassification, str]] = {
+ "data.train_files": _rule("source.train_files", "exact", "same Parquet inputs"),
+ "data.val_files": _rule("source.val_files", "exact", "same Parquet inputs"),
+ "data.prompt_key": _rule("source.prompt_key", "exact", "same Parquet column"),
+ "data.train_batch_size": _rule(
+ "train.rollouts_per_cycle", "semantically_conformant", "same logical prompts per rollout"
+ ),
+ "data.max_prompt_length": _rule("source.max_prompt_length", "exact", "same token bound"),
+ "data.max_response_length": _rule("source.max_response_length", "exact", "same token bound"),
+ "data.truncation": _rule("source.truncation", "semantically_conformant", "same policy"),
+ "data.shuffle": _rule("source.shuffle", "semantically_conformant", "same data policy"),
+ "data.seed": _rule("source.seed", "exact", "same seed"),
+ "actor_rollout_ref.model.path": _rule(
+ "models.student.model_id", "exact", "same actor checkpoint identity"
+ ),
+ "actor_rollout_ref.model.enable_gradient_checkpointing": _rule(
+ "models.student.gradient_checkpointing",
+ "semantically_conformant",
+ "same activation-memory technique",
+ ),
+ "actor_rollout_ref.model.lora_rank": _rule(
+ "models.student.lora.r", "semantically_conformant", "same PEFT rank when enabled"
+ ),
+ "actor_rollout_ref.model.lora_alpha": _rule(
+ "models.student.lora.alpha", "semantically_conformant", "same PEFT scale when enabled"
+ ),
+ "actor_rollout_ref.model.target_modules": _rule(
+ "models.student.lora.target_modules",
+ "semantically_conformant",
+ "same PEFT target names when enabled",
+ ),
+ "actor_rollout_ref.actor.optim.lr": _rule(
+ "train.learning_rate", "exact", "same optimizer learning rate"
+ ),
+ "actor_rollout_ref.actor.optim.weight_decay": _rule(
+ "train.weight_decay", "exact", "same optimizer coefficient"
+ ),
+ "actor_rollout_ref.actor.optim.lr_warmup_steps": _rule(
+ "train.warmup_steps", "exact", "same optimizer-step count"
+ ),
+ "actor_rollout_ref.actor.ppo_mini_batch_size": _rule(
+ "train.gradient_accumulation_steps",
+ "semantically_conformant",
+ "one logical upstream mini-batch is physically microbatched on one GPU",
+ ),
+ "actor_rollout_ref.actor.ppo_epochs": _rule(
+ "train.opd_freshness", "exact", "one fresh-policy update pass is supported"
+ ),
+ "actor_rollout_ref.actor.loss_agg_mode": _rule(
+ "loss.aggregation", "exact", "token-mean aggregation"
+ ),
+ "actor_rollout_ref.actor.clip_ratio": _rule(
+ "loss.clip_ratio", "exact", "same vanilla policy clip"
+ ),
+ "actor_rollout_ref.actor.clip_ratio_low": _rule(
+ "loss.clip_ratio_low", "exact", "same lower clip"
+ ),
+ "actor_rollout_ref.actor.clip_ratio_high": _rule(
+ "loss.clip_ratio_high", "exact", "same upper clip"
+ ),
+ "actor_rollout_ref.actor.clip_ratio_c": _rule(
+ "loss.clip_ratio_c", "exact", "same dual-clip coefficient"
+ ),
+ "actor_rollout_ref.actor.use_kl_loss": _rule(
+ None, "not_implemented", "RL actor/reference KL is not connected"
+ ),
+ "actor_rollout_ref.actor.entropy_coeff": _rule(
+ None, "not_implemented", "entropy regularization is not connected to the local objective"
+ ),
+ "actor_rollout_ref.rollout.name": _rule(
+ "rollout.backend",
+ "locally_lowered",
+ "distributed engine selection becomes a validated local HF backend",
+ ),
+ "actor_rollout_ref.rollout.n": _rule(
+ "rollout.samples_per_prompt", "exact", "same grouped sample count"
+ ),
+ "actor_rollout_ref.rollout.temperature": _rule(
+ "rollout.temperature", "exact", "same behavior-policy temperature"
+ ),
+ "actor_rollout_ref.rollout.top_p": _rule("rollout.top_p", "exact", "same nucleus threshold"),
+ "actor_rollout_ref.rollout.top_k": _rule("rollout.top_k", "exact", "same top-k bound"),
+ "actor_rollout_ref.rollout.tensor_model_parallel_size": _rule(
+ None, "distributed_only", "tensor parallel placement is not part of one-GPU semantics"
+ ),
+ "actor_rollout_ref.rollout.gpu_memory_utilization": _rule(
+ "memory planner",
+ "locally_lowered",
+ "recorded as an upstream engine hint; local placement is planned separately",
+ ),
+ "algorithm.adv_estimator": _rule(
+ "algorithm.name", "semantically_conformant", "pinned estimator implementation"
+ ),
+ "algorithm.norm_adv_by_std_in_grpo": _rule(
+ "algorithm.name", "semantically_conformant", "false selects Dr.GRPO centering"
+ ),
+ "algorithm.gamma": _rule("algorithm.gamma", "exact", "same discount"),
+ "algorithm.lam": _rule("algorithm.lam", "exact", "same GAE coefficient where applicable"),
+ "algorithm.use_kl_in_reward": _rule(
+ "algorithm.kl_coef", "semantically_conformant", "same sampled-token KL reward penalty"
+ ),
+ "algorithm.kl_penalty": _rule(
+ "algorithm.kl_penalty", "semantically_conformant", "same pinned estimator"
+ ),
+ "algorithm.kl_ctrl.type": _rule(
+ "algorithm.kl_coef", "semantically_conformant", "fixed coefficient is supported"
+ ),
+ "algorithm.kl_ctrl.kl_coef": _rule(
+ "algorithm.kl_coef", "exact", "same fixed reference-KL coefficient"
+ ),
+ "trainer.total_training_steps": _rule(
+ "train.cycles", "exact", "one local cycle is one upstream rollout iteration"
+ ),
+ "trainer.total_epochs": _rule(
+ None,
+ "informational_only",
+ "recorded but not used when total_training_steps supplies the exact optimizer-step cap",
+ ),
+ "trainer.project_name": _rule("run.tags", "semantically_conformant", "same provenance label"),
+ "trainer.experiment_name": _rule("run.name", "exact", "same run label"),
+ "trainer.save_freq": _rule(
+ "train.save_every_cycles", "exact", "same rollout-iteration interval"
+ ),
+ "trainer.test_freq": _rule(
+ "train.eval_every_cycles", "exact", "same rollout-iteration interval"
+ ),
+ "trainer.logger": _rule(None, "informational_only", "external logging is not activated"),
+ "trainer.n_gpus_per_node": _rule(
+ None, "distributed_only", "resource placement lowers to one local CUDA device"
+ ),
+ "trainer.nnodes": _rule(
+ None, "distributed_only", "resource placement lowers to one local process"
+ ),
+ "miniverl.reward.provider": _rule(
+ "reward.provider", "exact", "explicit trusted local reward implementation"
+ ),
+ "miniverl.actor.lora_enabled": _rule(
+ "models.student.lora.enabled", "exact", "explicit local parameterization choice"
+ ),
+ "miniverl.actor.revision": _rule(
+ "models.student.revision", "exact", "immutable actor snapshot revision"
+ ),
+ "miniverl.actor.tokenizer_revision": _rule(
+ "models.student.tokenizer_revision", "exact", "immutable tokenizer snapshot revision"
+ ),
+ "miniverl.actor.dtype": _rule("models.student.dtype", "exact", "local runtime choice"),
+ "miniverl.actor.quantization": _rule(
+ "models.student.quantization", "exact", "local runtime choice"
+ ),
+ "miniverl.rollout.backend": _rule("rollout.backend", "exact", "local runtime choice"),
+ "miniverl.memory.strategy": _rule("memory.strategy", "exact", "local placement choice"),
+ "miniverl.reference.adapter_path": _rule(
+ "models.reference.adapter.path", "exact", "explicit frozen reference adapter"
+ ),
+ "miniverl.reference.adapter_source": _rule(
+ "models.reference.adapter.source", "exact", "local or Hub artifact source"
+ ),
+ "miniverl.reference.adapter_revision": _rule(
+ "models.reference.adapter.revision", "exact", "immutable Hub artifact revision"
+ ),
+ "miniverl.update.trajectory_batch_size": _rule(
+ "train.trajectory_batch_size", "exact", "physical-only one-GPU batch size"
+ ),
+}
+
+
+def _canonical_digest(value: Any) -> str:
+ import json
+
+ encoded = json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
+ return hashlib.sha256(encoded).hexdigest()
+
+
+def rl_v09_field_rules_digest() -> str:
+ return _canonical_digest(
+ [
+ {"field": field, "target": target, "classification": kind, "reason": reason}
+ for field, (target, kind, reason) in sorted(_RULES.items())
+ ]
+ )
+
+
+def _flatten(value: Mapping[str, Any], prefix: str = "") -> dict[str, Any]:
+ flattened: dict[str, Any] = {}
+ for key, item in value.items():
+ path = f"{prefix}.{key}" if prefix else str(key)
+ if isinstance(item, Mapping):
+ flattened.update(_flatten(item, path))
+ else:
+ flattened[path] = item
+ return flattened
+
+
+_MISSING = object()
+
+
+def _get(value: Mapping[str, Any], path: str, default: Any = _MISSING) -> Any:
+ current: Any = value
+ for part in path.split("."):
+ if not isinstance(current, Mapping) or part not in current:
+ if default is _MISSING:
+ raise ConfigError(f"resolved verl field {path} is required")
+ return default
+ current = current[part]
+ return current
+
+
+def _integer(value: Any, field: str, *, minimum: int = 0) -> int:
+ if isinstance(value, bool):
+ raise ConfigError(f"verl field {field} must be an integer >= {minimum}")
+ if isinstance(value, str):
+ try:
+ parsed = int(value)
+ except ValueError as exc:
+ raise ConfigError(f"verl field {field} must be an integer >= {minimum}") from exc
+ if str(parsed) != value.strip():
+ raise ConfigError(f"verl field {field} must be an integer >= {minimum}")
+ elif isinstance(value, int):
+ parsed = value
+ else:
+ raise ConfigError(f"verl field {field} must be an integer >= {minimum}")
+ if parsed < minimum:
+ raise ConfigError(f"verl field {field} must be an integer >= {minimum}")
+ return parsed
+
+
+def _number(value: Any, field: str, *, minimum: float | None = None) -> float:
+ if isinstance(value, bool):
+ raise ConfigError(f"verl field {field} must be finite")
+ try:
+ parsed = float(value)
+ except (TypeError, ValueError) as exc:
+ raise ConfigError(f"verl field {field} must be finite") from exc
+ if not math.isfinite(parsed) or (minimum is not None and parsed < minimum):
+ raise ConfigError(f"verl field {field} must be finite and >= {minimum}")
+ return parsed
+
+
+def _bool(value: Any, field: str) -> bool:
+ if not isinstance(value, bool):
+ raise ConfigError(f"verl field {field} must be boolean")
+ return value
+
+
+def _list(value: Any, field: str) -> list[str]:
+ values = [value] if isinstance(value, str) else value
+ if not isinstance(values, list) or any(
+ not isinstance(item, str) or not item for item in values
+ ):
+ raise ConfigError(f"verl field {field} must contain non-empty strings")
+ return values
+
+
+def _classify(flat: Mapping[str, Any]) -> tuple[list[dict[str, Any]], list[str], list[str]]:
+ rows: list[dict[str, Any]] = []
+ unsupported: list[str] = []
+ unresolved: list[str] = []
+ estimator = flat.get("algorithm.adv_estimator")
+ estimator_specific = {
+ "algorithm.gamma": "reinforce_plus_plus",
+ "algorithm.lam": "gae",
+ "algorithm.norm_adv_by_std_in_grpo": "grpo",
+ }
+ for field, value in sorted(flat.items()):
+ rule = _RULES.get(field)
+ if rule is None:
+ target, kind, reason = None, "unsupported", "outside the versioned resolved subset"
+ unsupported.append(field)
+ else:
+ target, kind, reason = rule
+ if field == "actor_rollout_ref.actor.use_kl_loss" and value is False:
+ kind = "exact"
+ reason = "actor-loss KL is disabled in both runtimes"
+ elif field == "actor_rollout_ref.actor.entropy_coeff":
+ try:
+ is_zero = float(value) == 0.0 and math.isfinite(float(value))
+ except (TypeError, ValueError):
+ is_zero = False
+ if is_zero:
+ kind = "exact"
+ reason = "entropy regularization is disabled in both runtimes"
+ elif field in estimator_specific and estimator != estimator_specific[field]:
+ target = None
+ kind = "informational_only"
+ reason = f"not consumed by the {estimator!s} estimator"
+ elif field == "algorithm.use_kl_in_reward" and value is False:
+ kind = "exact"
+ reason = "reference-policy reward KL is disabled in both runtimes"
+ findings = audit_interpolation(value, label=field)
+ if findings:
+ unresolved.append(field)
+ rows.append(
+ {
+ "upstream_field": field,
+ "source_value": portable_payload(value),
+ "local_target": target,
+ "classification": kind,
+ "reason": reason,
+ "resolution_status": "unresolved" if findings else "resolved",
+ }
+ )
+ return rows, unsupported, unresolved
+
+
+def _algorithm(source: Mapping[str, Any]) -> str:
+ estimator = str(_get(source, "algorithm.adv_estimator"))
+ if estimator == "grpo":
+ normalize = _bool(
+ _get(source, "algorithm.norm_adv_by_std_in_grpo", True),
+ "algorithm.norm_adv_by_std_in_grpo",
+ )
+ return "grpo" if normalize else "dr_grpo"
+ if estimator in {"rloo", "reinforce_plus_plus"}:
+ return estimator
+ reason = (
+ "PPO/GAE needs a critic lifecycle that miniVERL does not yet implement"
+ if estimator == "gae"
+ else f"advantage estimator {estimator!r} is not implemented by this profile"
+ )
+ raise ConfigError(reason)
+
+
+def _required_inputs(source: Mapping[str, Any]) -> list[dict[str, str]]:
+ required: list[dict[str, str]] = []
+ if _get(source, "trainer.total_training_steps", None) is None:
+ required.append(
+ {
+ "field": "trainer.total_training_steps",
+ "reason": "an exact rollout-iteration cap is required; epochs are dataset-dependent",
+ }
+ )
+ if _get(source, "miniverl.reward.provider", None) is None:
+ required.append(
+ {
+ "field": "miniverl.reward.provider",
+ "reason": "the source must name a trusted local reward implementation",
+ }
+ )
+ if _get(source, "miniverl.actor.lora_enabled", None) is None:
+ required.append(
+ {
+ "field": "miniverl.actor.lora_enabled",
+ "reason": "PEFT versus full-parameter training is an explicit experiment choice",
+ }
+ )
+ if (
+ _get(source, "algorithm.use_kl_in_reward", False) is True
+ and _get(source, "miniverl.reference.adapter_path", None) is None
+ ):
+ required.append(
+ {
+ "field": "miniverl.reference.adapter_path",
+ "reason": "reference KL needs an explicit frozen reference-policy adapter",
+ }
+ )
+ return required
+
+
+def _recipe(source: Mapping[str, Any]) -> tuple[dict[str, Any], list[dict[str, Any]]]:
+ algorithm = _algorithm(source)
+ use_kl_in_reward = _bool(
+ _get(source, "algorithm.use_kl_in_reward", False), "algorithm.use_kl_in_reward"
+ )
+ if not use_kl_in_reward and _get(source, "miniverl.reference.adapter_path", None) is not None:
+ raise ConfigError("miniverl.reference is inactive unless algorithm.use_kl_in_reward=true")
+ if _bool(
+ _get(source, "actor_rollout_ref.actor.use_kl_loss", False),
+ "actor_rollout_ref.actor.use_kl_loss",
+ ):
+ raise ConfigError("actor.use_kl_loss=true is not implemented by the local RL path")
+ entropy = _number(
+ _get(source, "actor_rollout_ref.actor.entropy_coeff", 0.0),
+ "actor_rollout_ref.actor.entropy_coeff",
+ minimum=0.0,
+ )
+ if entropy != 0.0:
+ raise ConfigError("nonzero actor entropy_coeff is not implemented by the local RL path")
+ ppo_epochs = _integer(
+ _get(source, "actor_rollout_ref.actor.ppo_epochs", 1),
+ "actor_rollout_ref.actor.ppo_epochs",
+ minimum=1,
+ )
+ if ppo_epochs != 1:
+ raise ConfigError("actor.ppo_epochs must be 1 for strict current-policy local RL")
+ aggregation = str(_get(source, "actor_rollout_ref.actor.loss_agg_mode", "token-mean"))
+ if aggregation != "token-mean":
+ raise ConfigError("only actor.loss_agg_mode=token-mean is semantically conformant locally")
+
+ prompt_batch = _integer(
+ _get(source, "data.train_batch_size"), "data.train_batch_size", minimum=1
+ )
+ n = _integer(_get(source, "actor_rollout_ref.rollout.n", 1), "rollout.n", minimum=1)
+ if algorithm in {"grpo", "dr_grpo", "rloo"} and n < 2:
+ raise ConfigError(f"algorithm {algorithm} requires actor_rollout_ref.rollout.n > 1")
+ mini_batch = _integer(
+ _get(source, "actor_rollout_ref.actor.ppo_mini_batch_size", prompt_batch),
+ "actor_rollout_ref.actor.ppo_mini_batch_size",
+ minimum=1,
+ )
+ trajectory_count = prompt_batch * n
+ if mini_batch > trajectory_count:
+ raise ConfigError(
+ "actor.ppo_mini_batch_size cannot exceed data.train_batch_size * rollout.n"
+ )
+ prompt_length = _integer(
+ _get(source, "data.max_prompt_length"), "data.max_prompt_length", minimum=1
+ )
+ response_length = _integer(
+ _get(source, "data.max_response_length"), "data.max_response_length", minimum=1
+ )
+ provider = str(_get(source, "miniverl.reward.provider"))
+ if provider not in {"exact_answer", "target_length"}:
+ raise ConfigError(
+ "the portable v0.9 profile accepts miniverl.reward.provider=exact_answer or "
+ "target_length; Python reward objects must be injected explicitly by the local API"
+ )
+ local_backend = str(_get(source, "miniverl.rollout.backend", "hf_cached"))
+ if local_backend not in {"hf_cached", "hf_reference"}:
+ raise ConfigError("miniverl.rollout.backend must be hf_cached or hf_reference")
+ source_engine = str(_get(source, "actor_rollout_ref.rollout.name", "unknown"))
+ if local_backend == "hf_reference" and n > 1:
+ raise ConfigError("grouped RL requires miniverl.rollout.backend=hf_cached")
+ tensor_parallel = _integer(
+ _get(source, "actor_rollout_ref.rollout.tensor_model_parallel_size", 1),
+ "actor_rollout_ref.rollout.tensor_model_parallel_size",
+ minimum=1,
+ )
+ gpus = _integer(
+ _get(source, "trainer.n_gpus_per_node", 1), "trainer.n_gpus_per_node", minimum=1
+ )
+ nodes = _integer(_get(source, "trainer.nnodes", 1), "trainer.nnodes", minimum=1)
+ lora_enabled = _bool(_get(source, "miniverl.actor.lora_enabled"), "miniverl.actor.lora_enabled")
+ if use_kl_in_reward and not lora_enabled:
+ raise ConfigError(
+ "reference-KL temporal lowering requires miniverl.actor.lora_enabled=true"
+ )
+ kl_penalty = str(_get(source, "algorithm.kl_penalty", "kl"))
+ if kl_penalty not in {"kl", "abs", "mse", "low_var_kl"}:
+ raise ConfigError(f"algorithm.kl_penalty {kl_penalty!r} is not supported locally")
+ kl_type = str(_get(source, "algorithm.kl_ctrl.type", "fixed"))
+ if use_kl_in_reward and kl_type != "fixed":
+ raise ConfigError("reference KL currently requires algorithm.kl_ctrl.type=fixed")
+ kl_coef = (
+ _number(
+ _get(source, "algorithm.kl_ctrl.kl_coef"),
+ "algorithm.kl_ctrl.kl_coef",
+ minimum=0.0,
+ )
+ if use_kl_in_reward
+ else 0.0
+ )
+ if use_kl_in_reward and kl_coef <= 0.0:
+ raise ConfigError("reference KL requires a positive algorithm.kl_ctrl.kl_coef")
+ target_modules = _list(
+ _get(source, "actor_rollout_ref.model.target_modules", ["q_proj", "v_proj"]),
+ "actor_rollout_ref.model.target_modules",
+ )
+ steps = _integer(
+ _get(source, "trainer.total_training_steps"), "trainer.total_training_steps", minimum=1
+ )
+ save_freq = _integer(_get(source, "trainer.save_freq", -1), "trainer.save_freq", minimum=-1)
+ test_freq = _integer(_get(source, "trainer.test_freq", -1), "trainer.test_freq", minimum=-1)
+ recipe: dict[str, Any] = {
+ "schema_version": 1,
+ "run": {
+ "name": str(_get(source, "trainer.experiment_name", "verl-local-rl")),
+ "mode": "rl",
+ "seed": _integer(_get(source, "data.seed", 0), "data.seed"),
+ "output_dir": "runs",
+ "deterministic": True,
+ "tags": [
+ str(_get(source, "trainer.project_name", "verl")),
+ VERL_RL_V09_PROFILE,
+ ],
+ "profile_identity": {
+ "profile_name": VERL_RL_V09_PROFILE,
+ "upstream_tag": UPSTREAM_VERL_TAG,
+ "upstream_commit": UPSTREAM_VERL_COMMIT,
+ "field_rules_sha256": rl_v09_field_rules_digest(),
+ },
+ },
+ "models": {
+ "backend": "hf",
+ "runtime": "shared_backbone" if use_kl_in_reward else "dual_model",
+ "device": "cuda",
+ "student": {
+ "model_id": str(_get(source, "actor_rollout_ref.model.path")),
+ "revision": _get(source, "miniverl.actor.revision", None),
+ "tokenizer_revision": _get(
+ source,
+ "miniverl.actor.tokenizer_revision",
+ _get(source, "miniverl.actor.revision", None),
+ ),
+ "dtype": str(_get(source, "miniverl.actor.dtype", "auto")),
+ "quantization": str(_get(source, "miniverl.actor.quantization", "none")),
+ "gradient_checkpointing": _bool(
+ _get(source, "actor_rollout_ref.model.enable_gradient_checkpointing", False),
+ "actor_rollout_ref.model.enable_gradient_checkpointing",
+ ),
+ "lora": {
+ "enabled": lora_enabled,
+ "r": _integer(
+ _get(source, "actor_rollout_ref.model.lora_rank", 16),
+ "actor_rollout_ref.model.lora_rank",
+ minimum=1,
+ ),
+ "alpha": _integer(
+ _get(source, "actor_rollout_ref.model.lora_alpha", 32),
+ "actor_rollout_ref.model.lora_alpha",
+ minimum=1,
+ ),
+ "target_modules": target_modules,
+ },
+ },
+ },
+ "source": {
+ "kind": "verl_parquet",
+ "train_files": _list(_get(source, "data.train_files"), "data.train_files"),
+ "val_files": _list(_get(source, "data.val_files", []), "data.val_files"),
+ "prompt_key": str(_get(source, "data.prompt_key", "prompt")),
+ "use_task_rewards": True,
+ "max_prompt_length": prompt_length,
+ "max_response_length": response_length,
+ "truncation": str(_get(source, "data.truncation", "error")),
+ "shuffle": _bool(_get(source, "data.shuffle", True), "data.shuffle"),
+ "seed": _integer(_get(source, "data.seed", 0), "data.seed"),
+ },
+ "rollout": {
+ "backend": local_backend,
+ "samples_per_prompt": n,
+ "prompt_batch_size": prompt_batch,
+ "max_turns": 1,
+ "max_new_tokens_per_turn": response_length,
+ "max_total_tokens": prompt_length + response_length,
+ "temperature": _number(
+ _get(source, "actor_rollout_ref.rollout.temperature", 1.0),
+ "actor_rollout_ref.rollout.temperature",
+ minimum=0.0,
+ ),
+ "top_p": _number(
+ _get(source, "actor_rollout_ref.rollout.top_p", 1.0),
+ "actor_rollout_ref.rollout.top_p",
+ minimum=0.0,
+ ),
+ "top_k": _integer(
+ _get(source, "actor_rollout_ref.rollout.top_k", 0),
+ "actor_rollout_ref.rollout.top_k",
+ ),
+ "record_logprobs": True,
+ },
+ "selection": {"selector": "all_model_tokens"},
+ "loss": {
+ "mode": "verl_rl_policy",
+ "aggregation": "token-mean",
+ "scale_by_temperature_squared": False,
+ "clip_ratio": _number(
+ _get(source, "actor_rollout_ref.actor.clip_ratio", 0.2),
+ "actor_rollout_ref.actor.clip_ratio",
+ minimum=0.0,
+ ),
+ "clip_ratio_low": _number(
+ _get(source, "actor_rollout_ref.actor.clip_ratio_low", 0.2),
+ "actor_rollout_ref.actor.clip_ratio_low",
+ minimum=0.0,
+ ),
+ "clip_ratio_high": _number(
+ _get(source, "actor_rollout_ref.actor.clip_ratio_high", 0.2),
+ "actor_rollout_ref.actor.clip_ratio_high",
+ minimum=0.0,
+ ),
+ "clip_ratio_c": _number(
+ _get(source, "actor_rollout_ref.actor.clip_ratio_c", 3.0),
+ "actor_rollout_ref.actor.clip_ratio_c",
+ minimum=0.0,
+ ),
+ },
+ "algorithm": {
+ "name": algorithm,
+ "implementation_version": ADVANTAGE_IMPLEMENTATION_VERSION,
+ "gamma": _number(_get(source, "algorithm.gamma", 1.0), "algorithm.gamma", minimum=0.0),
+ "lam": _number(_get(source, "algorithm.lam", 1.0), "algorithm.lam", minimum=0.0),
+ "kl_coef": kl_coef,
+ "kl_penalty": kl_penalty,
+ },
+ "reward": {"enabled": True, "provider": provider, "error_policy": "fail"},
+ "train": {
+ "cycles": steps,
+ "rollouts_per_cycle": prompt_batch,
+ "gradient_accumulation_steps": mini_batch,
+ "trajectory_batch_size": _get(source, "miniverl.update.trajectory_batch_size", 1),
+ "opd_freshness": "strict" if mini_batch == trajectory_count else "replay",
+ "learning_rate": _number(
+ _get(source, "actor_rollout_ref.actor.optim.lr"),
+ "actor_rollout_ref.actor.optim.lr",
+ minimum=0.0,
+ ),
+ "weight_decay": _number(
+ _get(source, "actor_rollout_ref.actor.optim.weight_decay", 0.0),
+ "actor_rollout_ref.actor.optim.weight_decay",
+ minimum=0.0,
+ ),
+ "warmup_steps": _integer(
+ _get(source, "actor_rollout_ref.actor.optim.lr_warmup_steps", 0),
+ "actor_rollout_ref.actor.optim.lr_warmup_steps",
+ ),
+ "save_every_cycles": 0 if save_freq < 0 else save_freq,
+ "eval_every_cycles": 0 if test_freq < 0 else test_freq,
+ },
+ "memory": {"strategy": str(_get(source, "miniverl.memory.strategy", "auto"))},
+ "eval": {"enabled": False},
+ }
+ if use_kl_in_reward:
+ adapter_source = str(_get(source, "miniverl.reference.adapter_source", "local"))
+ adapter: dict[str, Any] = {
+ "path": str(_get(source, "miniverl.reference.adapter_path")),
+ "source": adapter_source,
+ }
+ adapter_revision = _get(source, "miniverl.reference.adapter_revision", None)
+ if adapter_revision is not None:
+ adapter["revision"] = str(adapter_revision)
+ recipe["models"]["reference"] = {
+ "model_id": str(_get(source, "actor_rollout_ref.model.path")),
+ "revision": _get(source, "miniverl.actor.revision", None),
+ "tokenizer_revision": _get(
+ source,
+ "miniverl.actor.tokenizer_revision",
+ _get(source, "miniverl.actor.revision", None),
+ ),
+ "dtype": str(_get(source, "miniverl.actor.dtype", "auto")),
+ "quantization": str(_get(source, "miniverl.actor.quantization", "none")),
+ "adapter": adapter,
+ }
+ lowering: list[dict[str, Any]] = [
+ {
+ "source_field": "actor_rollout_ref.rollout.name",
+ "source_value": source_engine,
+ "local_value": local_backend,
+ "kind": "physical_runtime",
+ },
+ {
+ "source_field": "actor_rollout_ref.rollout.tensor_model_parallel_size",
+ "source_value": tensor_parallel,
+ "local_value": 1,
+ "kind": "distributed_only",
+ },
+ {
+ "source_field": "trainer.n_gpus_per_node/trainer.nnodes",
+ "source_value": {"n_gpus_per_node": gpus, "nnodes": nodes},
+ "local_value": {"devices": 1, "processes": 1},
+ "kind": "distributed_only",
+ },
+ ]
+ return recipe, lowering
+
+
+def publish_imported_verl_rl_v09(
+ source: str | Path,
+ *,
+ out: str | Path,
+ target_verl: str,
+ overwrite: bool = False,
+ lock_timeout: float = DEFAULT_LOCK_TIMEOUT,
+) -> dict[str, Any]:
+ """Publish a native recipe, a user-input template, or a rejection report."""
+ if target_verl not in {UPSTREAM_VERL_TAG, UPSTREAM_VERL_COMMIT}:
+ raise ConfigError(
+ f"profile {VERL_RL_V09_PROFILE} targets {UPSTREAM_VERL_TAG} or "
+ f"{UPSTREAM_VERL_COMMIT}, not {target_verl!r}"
+ )
+ source_path = Path(source)
+ targets = import_output_targets(out)
+ reject_source_output_alias({"source config": source_path}, targets)
+ try:
+ source_bytes = source_path.read_bytes()
+ payload = yaml.safe_load(source_bytes)
+ except (OSError, yaml.YAMLError) as exc:
+ raise ConfigError(f"cannot read verl config {source_path}: {exc}") from exc
+ if not isinstance(payload, Mapping):
+ raise ConfigError("verl config must contain one YAML mapping")
+ flat = _flatten(payload)
+ fields, unsupported, unresolved = _classify(flat)
+ required = _required_inputs(payload)
+ common: dict[str, Any] = {
+ "schema_version": 1,
+ "profile": VERL_RL_V09_PROFILE,
+ "profile_field_rules_sha256": rl_v09_field_rules_digest(),
+ "source_verl": {
+ "repository": VERL_REPOSITORY,
+ "tag": UPSTREAM_VERL_TAG,
+ "commit": UPSTREAM_VERL_COMMIT,
+ },
+ "source_config_sha256": hashlib.sha256(source_bytes).hexdigest(),
+ "input_contract": "resolved documented verl v0.9 RL subset; not arbitrary Hydra YAML",
+ "field_classification": fields,
+ "unsupported_fields": unsupported,
+ "unresolved_fields": unresolved,
+ "required_user_input": required,
+ "report_path": targets["report"].name,
+ }
+ transaction = OutputTransaction(
+ targets=targets,
+ stem=targets["recipe"].stem,
+ lock_root=targets["recipe"].parent,
+ overwrite=overwrite,
+ lock_timeout=lock_timeout,
+ ).begin()
+ try:
+ if unsupported or unresolved:
+ reason = "unsupported fields" if unsupported else "unresolved interpolation"
+ report = {
+ **common,
+ "status": "rejected",
+ "executable": False,
+ "reason": reason,
+ "generated_path": targets["report"].name,
+ }
+ transaction.write_json("report", report)
+ transaction.discard("recipe")
+ transaction.discard("template")
+ transaction.commit()
+ return report
+ if required:
+ template = {
+ "schema_version": 1,
+ "status": "needs_user_input",
+ "note": "Non-executable source template; supply every required field.",
+ "profile": VERL_RL_V09_PROFILE,
+ "source_values": portable_payload(dict(payload)),
+ "required_user_input": required,
+ }
+ report = {
+ **common,
+ "status": "needs_user_input",
+ "executable": False,
+ "generated_path": targets["template"].name,
+ }
+ transaction.write_bytes(
+ "template",
+ yaml.safe_dump(template, sort_keys=False, allow_unicode=True).encode("utf-8"),
+ )
+ transaction.write_json("report", report)
+ transaction.discard("recipe")
+ transaction.commit()
+ return report
+ try:
+ recipe, lowering = _recipe(payload)
+ from miniverl.config import RunConfig
+
+ validated = RunConfig.from_mapping(recipe)
+ except ConfigError:
+ raise
+ except Exception as exc:
+ raise ConfigError(f"generated local RL recipe failed validation: {exc}") from exc
+ rendered = yaml.safe_dump(recipe, sort_keys=False, allow_unicode=True, width=100).encode()
+ report = {
+ **common,
+ "status": "accepted",
+ "executable": True,
+ "generated_path": targets["recipe"].name,
+ "generated_recipe_sha256": hashlib.sha256(rendered).hexdigest(),
+ "generated_recipe_validated": True,
+ "algorithm": validated.algorithm.name.value,
+ "local_lowering": lowering,
+ }
+ transaction.write_bytes("recipe", rendered)
+ transaction.write_json("report", report)
+ transaction.discard("template")
+ transaction.commit()
+ return report
+ finally:
+ transaction.close()
diff --git a/src/miniverl/cli.py b/src/miniverl/cli.py
index 41edb09..ede7d31 100644
--- a/src/miniverl/cli.py
+++ b/src/miniverl/cli.py
@@ -33,9 +33,9 @@
app = typer.Typer(
name="miniverl",
help=(
- "Run a documented subset of verl-style OPD on one consumer GPU.\n\n"
- "Compile typed verl-shaped config, reuse Parquet prompts, execute actor rollout, "
- "teacher scoring and actor update locally, then export inspectable PEFT artifacts."
+ "Compile supported verl experiment semantics for one consumer GPU.\n\n"
+ "Turn resolved YAML and Parquet prompts into actor rollout, reward/reference/teacher "
+ "phases and actor updates, then export inspectable PEFT artifacts."
),
add_completion=False,
no_args_is_help=True,
@@ -146,7 +146,7 @@ def main(
envvar="MINIVERL_LOG_LEVEL",
),
) -> None:
- """miniVERL: the bounded local runtime for verl-style OPD on one GPU."""
+ """miniVERL: verl experiment semantics compiled for one GPU."""
from miniverl.utils.logging import configure_logging
configure_logging(log_level)
@@ -159,8 +159,13 @@ def data_sample_command(
"verl-parquet", "--format", help="Portable output format (verl-parquet only)."
),
rows: int = typer.Option(4, "--rows", min=1, max=1024, help="Number of sample prompts."),
+ task_rewards: bool = typer.Option(
+ False,
+ "--task-rewards",
+ help="Add deterministic exact-answer reward metadata for local RL examples.",
+ ),
) -> None:
- """Create a small reward-free verl-style Parquet prompt dataset."""
+ """Create a small verl-style Parquet prompt dataset."""
if format_name != "verl-parquet":
_fail(ConfigError("--format must be verl-parquet"))
return
@@ -178,22 +183,42 @@ def data_sample_command(
"What does token-mean loss aggregation mean?",
"State one limitation of a single-GPU training runtime.",
]
+ rewarded = [
+ ("Reply with exactly the word: provenance", "provenance"),
+ ("Reply with exactly the word: checkpoint", "checkpoint"),
+ ("Reply with exactly the word: adapter", "adapter"),
+ ("Reply with exactly the word: reward", "reward"),
+ ]
records = []
for index in range(rows):
- prompt = prompts[index % len(prompts)]
- if index >= len(prompts):
- prompt = f"Sample {index + 1}: {prompt}"
- records.append(
- {
- "prompt": [
- {"role": "system", "content": "Answer clearly and briefly."},
- {"role": "user", "content": prompt},
- ],
- "data_source": "miniverl_quickstart",
- "ability": "short_answer",
- "extra_info": {"sample_index": index},
- }
+ prompt, ground_truth = (
+ rewarded[index % len(rewarded)]
+ if task_rewards
+ else (prompts[index % len(prompts)], None)
)
+ if index >= len(prompts) and not task_rewards:
+ prompt = f"Sample {index + 1}: {prompt}"
+ extra_info: dict[str, Any] = {"sample_index": index}
+ record: dict[str, Any] = {
+ "prompt": [
+ {
+ "role": "system",
+ "content": (
+ "Follow the exact response instruction."
+ if task_rewards
+ else "Answer clearly and briefly."
+ ),
+ },
+ {"role": "user", "content": prompt},
+ ],
+ "data_source": "miniverl_quickstart_rl" if task_rewards else "miniverl_quickstart",
+ "ability": "exact_answer" if task_rewards else "short_answer",
+ "extra_info": extra_info,
+ }
+ if ground_truth is not None:
+ record["reward_model"] = {"style": "exact", "ground_truth": ground_truth}
+ extra_info["ground_truth"] = ground_truth
+ records.append(record)
out.parent.mkdir(parents=True, exist_ok=True)
pq.write_table(pa.Table.from_pylist(records), out)
digest = hashlib.sha256(out.read_bytes()).hexdigest()
@@ -303,8 +328,10 @@ def validate(
if environment_config is not None:
try:
environment = make_environment(environment_config.name, **environment_config.params)
- if config.models.teacher.mode.value == "privileged_context" and not hasattr(
- environment, "privileged_context"
+ if (
+ config.models.teacher is not None
+ and config.models.teacher.mode.value == "privileged_context"
+ and not hasattr(environment, "privileged_context")
):
warnings.append("environment provides no privileged context")
except MiniVerlError as exc:
@@ -326,7 +353,11 @@ def validate(
)
if config.models.backend.value == "hf" and not config.models.student.revision:
warnings.append("models.student.revision is unpinned; the manifest will record 'unpinned'")
- if config.models.backend.value == "hf" and not config.models.teacher.revision:
+ if (
+ config.models.backend.value == "hf"
+ and config.models.teacher is not None
+ and not config.models.teacher.revision
+ ):
warnings.append("models.teacher.revision is unpinned")
payload = {
@@ -340,7 +371,7 @@ def validate(
),
"backend": config.models.backend.value,
"student": config.models.student.model_id,
- "teacher": config.models.teacher.model_id,
+ "teacher": config.models.teacher.model_id if config.models.teacher is not None else None,
"source_kind": config.source.kind.value,
"environment": environment_config.name if environment_config is not None else None,
"difficulty": environment_config.difficulty if environment_config is not None else None,
@@ -588,8 +619,10 @@ def prepare_offline_kd_command(
"collection_tasks": (config.offline_kd.collection_tasks or config.train.rollouts_per_cycle),
"student": config.models.student.model_id,
"student_revision": config.models.student.revision,
- "teacher": config.models.teacher.model_id,
- "teacher_revision": config.models.teacher.revision,
+ "teacher": config.models.teacher.model_id if config.models.teacher is not None else None,
+ "teacher_revision": (
+ config.models.teacher.revision if config.models.teacher is not None else None
+ ),
}
if dry_run:
if as_json:
@@ -1180,7 +1213,8 @@ def verl_run_command(
f"n={native.rollout.samples_per_prompt}, max response "
f"{_esc(native.source.max_response_length)}"
)
- console.print(f" teacher {_esc(native.models.teacher.model_id)}")
+ if native.models.teacher is not None:
+ console.print(f" teacher {_esc(native.models.teacher.model_id)}")
console.print(" distill forward_kl_topk / token-mean / no reward")
console.print(f" trainer {_esc(native.train.cycles)} optimizer update(s)")
console.print(f" placement {_esc(system_plan.local_execution['strategy'])}")
@@ -1390,9 +1424,12 @@ def train(
return
if dry_run:
+ trajectories_per_cycle = config.train.rollouts_per_cycle * (
+ config.rollout.samples_per_prompt if config.run.mode.value == "rl" else 1
+ )
steps_per_cycle = max(
1,
- (config.train.rollouts_per_cycle + config.train.gradient_accumulation_steps - 1)
+ (trajectories_per_cycle + config.train.gradient_accumulation_steps - 1)
// config.train.gradient_accumulation_steps,
)
plan = {
@@ -1403,12 +1440,17 @@ def train(
"backend": config.models.backend.value,
"student": config.models.student.model_id,
"student_revision": config.models.student.revision,
- "teacher": config.models.teacher.model_id,
- "teacher_revision": config.models.teacher.revision,
+ "teacher": config.models.teacher.model_id
+ if config.models.teacher is not None
+ else None,
+ "teacher_revision": (
+ config.models.teacher.revision if config.models.teacher is not None else None
+ ),
"downloads_required": config.models.backend.value == "hf",
"planned_optimizer_steps": steps_per_cycle
* (config.train.cycles + config.train.sft_warmup_cycles),
- "planned_rollouts": config.train.rollouts_per_cycle * config.train.cycles,
+ "planned_prompt_groups": config.train.rollouts_per_cycle * config.train.cycles,
+ "planned_rollouts": trajectories_per_cycle * config.train.cycles,
"output_dir": str(output or config.run.output_dir),
"resume": str(resume) if resume is not None else None,
"resume_from": str(resume_from) if resume_from is not None else None,
@@ -1567,7 +1609,11 @@ def import_verl_command(
None, "--config", help="Resolved verl YAML configuration (v2 spelling)."
),
profile: str = typer.Option(..., "--profile", help="Documented bridge profile."),
- target_verl: str = typer.Option("v0.8.0", "--target-verl", help="Pinned verl tag or commit."),
+ target_verl: Optional[str] = typer.Option(
+ None,
+ "--target-verl",
+ help="Pinned verl tag or commit (defaults to the selected profile's pin).",
+ ),
out: Path = typer.Option(..., "--out", help="New miniVERL recipe path."),
overrides: list[str] = typer.Option([], "--set", help="Repeatable dotted key=value override."),
environment: Optional[str] = typer.Option(
@@ -1600,12 +1646,29 @@ def import_verl_command(
if selected_source is None:
raise ConfigError("a resolved verl YAML path is required", hint="pass --config FILE")
from miniverl.bridge.opd_v08 import VERL_OPD_V08_PROFILE
+ from miniverl.bridge.rl_v09 import VERL_RL_V09_PROFILE
+
+ if profile == VERL_RL_V09_PROFILE:
+ resolved_target_verl = target_verl or "v0.9.0"
+ if overrides:
+ raise ConfigError(
+ "--set is not supported by the fail-closed verl v0.9 RL importer; "
+ "pass a fully resolved YAML"
+ )
+ from miniverl.bridge.rl_v09 import publish_imported_verl_rl_v09
- if profile == VERL_OPD_V08_PROFILE:
+ report = publish_imported_verl_rl_v09(
+ selected_source,
+ out=out,
+ target_verl=resolved_target_verl,
+ overwrite=overwrite,
+ )
+ elif profile == VERL_OPD_V08_PROFILE:
from miniverl.bridge.contract import validate_target_verl
from miniverl.bridge.opd_v08 import publish_imported_verl_opd_v08
- validate_target_verl(target_verl)
+ resolved_target_verl = target_verl or "v0.8.0"
+ validate_target_verl(resolved_target_verl)
report = publish_imported_verl_opd_v08(
selected_source,
out=out,
@@ -1613,6 +1676,7 @@ def import_verl_command(
overwrite=overwrite,
)
else:
+ resolved_target_verl = target_verl or "v0.8.0"
if overrides:
raise ConfigError(
"--set is supported by the verl OPD v2 profile only",
@@ -1623,7 +1687,7 @@ def import_verl_command(
report = import_verl_config(
selected_source,
profile=profile,
- target_verl=target_verl,
+ target_verl=resolved_target_verl,
out=out,
environment=environment,
teacher_model=teacher_model,
diff --git a/src/miniverl/config/__init__.py b/src/miniverl/config/__init__.py
index 867db97..aca3d15 100644
--- a/src/miniverl/config/__init__.py
+++ b/src/miniverl/config/__init__.py
@@ -4,6 +4,7 @@
from miniverl.config.models import (
AdapterSource,
+ AlgorithmConfig,
AlignmentConfig,
AlignmentMethod,
CacheConfig,
@@ -33,6 +34,7 @@
ReportConfig,
RewardConfig,
RewardProviderKind,
+ RLAlgorithm,
RolloutConfig,
RunConfig,
RunMeta,
@@ -53,6 +55,7 @@
__all__ = [
"AdapterSource",
+ "AlgorithmConfig",
"AlignmentConfig",
"AlignmentMethod",
"CacheConfig",
@@ -82,6 +85,7 @@
"RewardConfig",
"RewardProviderKind",
"ReportConfig",
+ "RLAlgorithm",
"RolloutConfig",
"RunConfig",
"RunMeta",
diff --git a/src/miniverl/config/models.py b/src/miniverl/config/models.py
index a4542e0..b666ef6 100644
--- a/src/miniverl/config/models.py
+++ b/src/miniverl/config/models.py
@@ -33,6 +33,8 @@
__all__ = [
"TrainingMode",
+ "RLAlgorithm",
+ "AlgorithmConfig",
"AlignmentConfig",
"AlignmentMethod",
"GateConfig",
@@ -92,6 +94,18 @@ class TrainingMode(str, Enum):
SFT = "sft"
OFFLINE_KD = "offline_kd"
OPD = "opd"
+ RL = "rl"
+
+
+class RLAlgorithm(str, Enum):
+ """Pinned upstream advantage estimator executed by the local RL path."""
+
+ NONE = "none"
+ GRPO = "grpo"
+ DR_GRPO = "dr_grpo"
+ RLOO = "rloo"
+ REINFORCE_PLUS_PLUS = "reinforce_plus_plus"
+ PPO = "ppo"
class OfflineKDTrajectorySource(str, Enum):
@@ -162,6 +176,7 @@ class LossMode(str, Enum):
VERL_FORWARD_KL_TOPK = "forward_kl_topk"
VERL_PG_K1 = "verl_pg_k1"
VERL_PG_K1_REWARDED = "verl_pg_k1_rewarded"
+ VERL_RL_POLICY = "verl_rl_policy"
class LossAggregation(str, Enum):
@@ -385,9 +400,16 @@ class ModelsConfig(_Base):
runtime: ModelRuntime = ModelRuntime.DUAL_MODEL
device: str = Field(default="auto", pattern="^(auto|cpu|cuda)$")
student: StudentModelConfig
- teacher: TeacherModelConfig
+ teacher: TeacherModelConfig | None = None
reference: ReferenceModelConfig | None = None
+ @property
+ def required_teacher(self) -> TeacherModelConfig:
+ """Return the configured teacher for a teacher-dependent execution path."""
+ if self.teacher is None:
+ raise ConfigError("this execution path requires models.teacher")
+ return self.teacher
+
@model_validator(mode="after")
def _validate_runtime(self) -> ModelsConfig:
if self.runtime is ModelRuntime.DUAL_MODEL:
@@ -400,42 +422,45 @@ def _validate_runtime(self) -> ModelsConfig:
raise ValueError("shared_backbone requires a trainable student LoRA adapter")
if self.student.prepare_kbit_training:
self.student = self.student.model_copy(update={"prepare_kbit_training": False})
- if self.teacher.adapter is None:
- raise ValueError("shared_backbone requires a frozen teacher adapter")
- if (self.student.model_id, self.student.revision) != (
- self.teacher.model_id,
- self.teacher.revision,
- ):
- raise ValueError(
- "shared_backbone student and teacher must use the same model_id and revision"
- )
student_settings = (
self.student.dtype,
self.student.quantization,
self.student.attn_implementation,
self.student.trust_remote_code,
)
- teacher_settings = (
- self.teacher.dtype,
- self.teacher.quantization,
- self.teacher.attn_implementation,
- self.teacher.trust_remote_code,
- )
- if student_settings != teacher_settings:
- raise ValueError(
- "shared_backbone student and teacher must use identical precision, "
- "quantization, attention and trust_remote_code settings"
- )
student_tokenizer = (
self.student.tokenizer_id or self.student.model_id,
self.student.tokenizer_revision or self.student.revision,
)
- teacher_tokenizer = (
- self.teacher.tokenizer_id or self.teacher.model_id,
- self.teacher.tokenizer_revision or self.teacher.revision,
- )
- if student_tokenizer != teacher_tokenizer:
- raise ValueError("shared_backbone roles must use one tokenizer identity")
+ if self.teacher is None and self.reference is None:
+ raise ValueError("shared_backbone requires a frozen teacher or reference role")
+ if self.teacher is not None:
+ if self.teacher.adapter is None:
+ raise ValueError("shared_backbone requires a frozen teacher adapter")
+ if (self.student.model_id, self.student.revision) != (
+ self.teacher.model_id,
+ self.teacher.revision,
+ ):
+ raise ValueError(
+ "shared_backbone student and teacher must use the same model_id and revision"
+ )
+ teacher_settings = (
+ self.teacher.dtype,
+ self.teacher.quantization,
+ self.teacher.attn_implementation,
+ self.teacher.trust_remote_code,
+ )
+ if student_settings != teacher_settings:
+ raise ValueError(
+ "shared_backbone student and teacher must use identical precision, "
+ "quantization, attention and trust_remote_code settings"
+ )
+ teacher_tokenizer = (
+ self.teacher.tokenizer_id or self.teacher.model_id,
+ self.teacher.tokenizer_revision or self.teacher.revision,
+ )
+ if student_tokenizer != teacher_tokenizer:
+ raise ValueError("shared_backbone roles must use one tokenizer identity")
if self.reference is not None:
if (self.reference.model_id, self.reference.revision) != (
self.student.model_id,
@@ -452,6 +477,12 @@ def _validate_runtime(self) -> ModelsConfig:
raise ValueError(
"shared_backbone reference must use the shared base runtime settings"
)
+ reference_tokenizer = (
+ self.reference.tokenizer_id or self.reference.model_id,
+ self.reference.tokenizer_revision or self.reference.revision,
+ )
+ if reference_tokenizer != student_tokenizer:
+ raise ValueError("shared_backbone reference must use the shared tokenizer")
return self
@@ -799,6 +830,7 @@ class RewardProviderKind(str, Enum):
ENVIRONMENT_VERIFIER = "environment_verifier"
EXACT_ANSWER = "exact_answer"
+ TARGET_LENGTH = "target_length"
PYTHON_API = "python_api"
@@ -810,6 +842,20 @@ class RewardConfig(_Base):
error_policy: Literal["fail"] = "fail"
+class AlgorithmConfig(_Base):
+ """Algorithm semantics kept separate from physical single-GPU controls."""
+
+ name: RLAlgorithm = RLAlgorithm.NONE
+ implementation_version: str | None = None
+ gamma: float = Field(default=1.0, ge=0.0, le=1.0)
+ lam: float = Field(default=1.0, ge=0.0, le=1.0)
+ epsilon: float = Field(default=1e-6, gt=0.0, le=1e-2)
+ kl_coef: float = Field(default=0.0, ge=0.0)
+ kl_penalty: Literal["kl", "abs", "mse", "low_var_kl"] = "kl"
+ value_loss_coef: float = Field(default=0.5, ge=0.0)
+ cliprange_value: float = Field(default=0.5, gt=0.0)
+
+
class RunMeta(_Base):
"""Identity and reproducibility settings for the run."""
@@ -849,6 +895,7 @@ class RunConfig(_Base):
eval: EvalConfig = Field(default_factory=EvalConfig)
report: ReportConfig = Field(default_factory=ReportConfig)
reward: RewardConfig = Field(default_factory=RewardConfig)
+ algorithm: AlgorithmConfig = Field(default_factory=AlgorithmConfig)
alignment: AlignmentConfig | None = None
# -- cross-field validation ----------------------------------------
@@ -895,38 +942,52 @@ def _validate_combination(self) -> RunConfig:
)
if self.source.kind is SourceKind.VERL_PARQUET and self.environment is not None:
raise ValueError("source.kind=verl_parquet must not define environment")
- if self.source.kind is SourceKind.ENVIRONMENT and self.rollout.samples_per_prompt != 1:
+ if (
+ self.source.kind is SourceKind.ENVIRONMENT
+ and self.rollout.samples_per_prompt != 1
+ and mode is not TrainingMode.RL
+ ):
raise ValueError(
"rollout.samples_per_prompt > 1 currently requires source.kind=verl_parquet; "
"multi-turn tool episodes retain one trajectory per task"
)
+ if mode in {TrainingMode.OPD, TrainingMode.OFFLINE_KD} and self.models.teacher is None:
+ raise ValueError(f"run.mode={mode.value} requires models.teacher")
+ if mode is TrainingMode.RL and self.models.teacher is not None:
+ raise ValueError("run.mode=rl is teacher-free; remove models.teacher")
if self.source.kind is SourceKind.VERL_PARQUET:
- if mode is not TrainingMode.OPD:
+ if mode not in {TrainingMode.OPD, TrainingMode.RL}:
raise ValueError(
- "source.kind=verl_parquet supports pure OPD in v0.8; it has no oracle "
+ "source.kind=verl_parquet supports OPD or RL; it has no oracle "
"labels for sft or offline_kd"
)
if self.train.sft_warmup_cycles:
raise ValueError(
"source.kind=verl_parquet cannot run sft_warmup_cycles without oracle labels"
)
- if self.models.teacher.mode is not TeacherContextMode.STANDARD:
+ if (
+ self.models.teacher is not None
+ and self.models.teacher.mode is not TeacherContextMode.STANDARD
+ ):
raise ValueError(
"source.kind=verl_parquet requires the actor and teacher to score the "
"same rendered prompt; privileged_context is outside the v0.8 profile"
)
- if self.source.use_task_rewards and self.loss.mode is not LossMode.VERL_PG_K1_REWARDED:
+ if self.source.use_task_rewards and self.loss.mode not in {
+ LossMode.VERL_PG_K1_REWARDED,
+ LossMode.VERL_RL_POLICY,
+ }:
raise ValueError(
"existing PG-k1 profile is reward-free; source.use_task_rewards=true "
"requires the explicit rewarded PG profile"
)
- if mode is TrainingMode.OPD and self.cache.reuse_across_policy_versions:
+ if mode in {TrainingMode.OPD, TrainingMode.RL} and self.cache.reuse_across_policy_versions:
raise ValueError(
"cache.reuse_across_policy_versions=true contradicts run.mode=opd: "
"reusing one teacher cache across policy versions is offline KD. "
"Set run.mode: offline_kd, or keep the cache strictly per-cycle."
)
- if mode is TrainingMode.OPD and not self.cache.strict_policy_version:
+ if mode in {TrainingMode.OPD, TrainingMode.RL} and not self.cache.strict_policy_version:
raise ValueError(
"run.mode=opd requires cache.strict_policy_version=true so teacher "
"targets can never be consumed by a different policy version"
@@ -989,6 +1050,75 @@ def _validate_combination(self) -> RunConfig:
"supported verl v0.8 profile"
)
+ if self.loss.mode is LossMode.VERL_RL_POLICY:
+ from miniverl.algorithms.contract import ADVANTAGE_IMPLEMENTATION_VERSION
+
+ if mode is not TrainingMode.RL:
+ raise ValueError("loss.mode=verl_rl_policy requires run.mode=rl")
+ if not self.reward.enabled:
+ raise ValueError("run.mode=rl requires task rewards and reward.enabled=true")
+ if self.source.kind is SourceKind.VERL_PARQUET:
+ if not self.source.use_task_rewards:
+ raise ValueError("Parquet RL requires source.use_task_rewards=true")
+ if self.reward.provider not in {
+ RewardProviderKind.EXACT_ANSWER,
+ RewardProviderKind.TARGET_LENGTH,
+ RewardProviderKind.PYTHON_API,
+ }:
+ raise ValueError(
+ "Parquet RL requires exact_answer, target_length, or an injected "
+ "python_api provider"
+ )
+ elif self.reward.provider is not RewardProviderKind.ENVIRONMENT_VERIFIER:
+ raise ValueError("environment RL requires reward.provider=environment_verifier")
+ if self.algorithm.name is RLAlgorithm.NONE:
+ raise ValueError("run.mode=rl requires an explicit algorithm.name")
+ if self.algorithm.implementation_version != ADVANTAGE_IMPLEMENTATION_VERSION:
+ raise ValueError(
+ "run.mode=rl requires algorithm.implementation_version="
+ f"{ADVANTAGE_IMPLEMENTATION_VERSION}"
+ )
+ if self.algorithm.name is RLAlgorithm.PPO:
+ raise ValueError(
+ "algorithm.name=ppo is not executable yet; GAE and value-loss primitives are "
+ "conformant, but the critic checkpoint/runtime contract is not complete"
+ )
+ if self.algorithm.kl_coef > 0.0:
+ if self.models.reference is None:
+ raise ValueError("algorithm.kl_coef > 0 requires models.reference")
+ if self.models.runtime is not ModelRuntime.SHARED_BACKBONE:
+ raise ValueError("RL reference KL requires runtime=shared_backbone")
+ elif self.models.reference is not None:
+ raise ValueError("models.reference requires algorithm.kl_coef > 0 in run.mode=rl")
+ if self.algorithm.value_loss_coef != 0.5 or self.algorithm.cliprange_value != 0.5:
+ raise ValueError(
+ "algorithm.value_loss_coef and cliprange_value apply only to the "
+ "not-yet-executable PPO critic path; keep their defaults"
+ )
+ if (
+ self.algorithm.name in {RLAlgorithm.GRPO, RLAlgorithm.DR_GRPO, RLAlgorithm.RLOO}
+ and self.rollout.samples_per_prompt < 2
+ ):
+ raise ValueError(
+ f"algorithm.name={self.algorithm.name.value} requires samples_per_prompt > 1"
+ )
+ if not self.rollout.record_logprobs:
+ raise ValueError("run.mode=rl requires rollout.record_logprobs=true")
+ if self.rollout.temperature <= 0:
+ raise ValueError("run.mode=rl requires stochastic rollout.temperature > 0")
+ if self.selection.selector is not SelectorName.ALL_MODEL_TOKENS:
+ raise ValueError("run.mode=rl requires selection.selector=all_model_tokens")
+ if self.loss.aggregation is not LossAggregation.TOKEN_MEAN:
+ raise ValueError("run.mode=rl requires loss.aggregation=token-mean")
+ if self.loss.sampled_token_nll_weight != 0.0:
+ raise ValueError("run.mode=rl cannot mix sampled-token NLL")
+ if self.loss.policy_loss_mode != "vanilla":
+ raise ValueError("run.mode=rl currently supports upstream vanilla policy loss")
+ elif mode is TrainingMode.RL:
+ raise ValueError("run.mode=rl requires loss.mode=verl_rl_policy")
+ elif self.algorithm != AlgorithmConfig():
+ raise ValueError("algorithm settings apply only to run.mode=rl")
+
pg_modes = {LossMode.VERL_PG_K1, LossMode.VERL_PG_K1_REWARDED}
if self.loss.mode in pg_modes:
from miniverl.bridge.opd_pg_contract import VERL_PG_K1_IMPLEMENTATION_VERSION
@@ -997,7 +1127,7 @@ def _validate_combination(self) -> RunConfig:
raise ValueError("loss.mode=verl_pg_k1 requires run.mode=opd")
if self.source.kind is not SourceKind.VERL_PARQUET:
raise ValueError("loss.mode=verl_pg_k1 requires source.kind=verl_parquet")
- if self.models.teacher.mode is not TeacherContextMode.STANDARD:
+ if self.models.required_teacher.mode is not TeacherContextMode.STANDARD:
raise ValueError("loss.mode=verl_pg_k1 requires a standard same-prompt teacher")
if self.loss.divergence is not Divergence.REVERSE_KL:
raise ValueError("loss.mode=verl_pg_k1 requires divergence=reverse_kl")
@@ -1064,10 +1194,12 @@ def _validate_combination(self) -> RunConfig:
raise ValueError("loss.mode=verl_pg_k1_rewarded requires reward.enabled=true")
if self.reward.provider not in {
RewardProviderKind.EXACT_ANSWER,
+ RewardProviderKind.TARGET_LENGTH,
RewardProviderKind.PYTHON_API,
}:
raise ValueError(
- "rewarded PG requires exact_answer or an injected python_api provider"
+ "rewarded PG requires exact_answer, target_length, or an injected "
+ "python_api provider"
)
if self.loss.advantage_composer_version != ADVANTAGE_COMPOSER_VERSION:
raise ValueError(
@@ -1091,7 +1223,7 @@ def _validate_combination(self) -> RunConfig:
f"advantage_mode={self.loss.advantage_mode.value} requires "
"samples_per_prompt > 1"
)
- elif (
+ elif mode is not TrainingMode.RL and (
self.reward != RewardConfig()
or self.loss.advantage_composer_version is not None
or self.loss.advantage_mode is not AdvantageMode.NONE
@@ -1108,13 +1240,16 @@ def _validate_combination(self) -> RunConfig:
"(explicit)"
)
+ trajectories_per_rollout_batch = self.train.rollouts_per_cycle * (
+ self.rollout.samples_per_prompt if mode is TrainingMode.RL else 1
+ )
steps_per_rollout_batch = max(
1,
- (self.train.rollouts_per_cycle + self.train.gradient_accumulation_steps - 1)
+ (trajectories_per_rollout_batch + self.train.gradient_accumulation_steps - 1)
// self.train.gradient_accumulation_steps,
)
if (
- mode is TrainingMode.OPD
+ mode in {TrainingMode.OPD, TrainingMode.RL}
and self.train.opd_freshness is OPDFreshness.STRICT
and steps_per_rollout_batch != 1
):
@@ -1137,12 +1272,15 @@ def _validate_combination(self) -> RunConfig:
"the toy backend does not support quantization "
"(models.student.quantization must be 'none')"
)
- if self.models.teacher.quantization is not Quantization.NONE:
+ if (
+ self.models.teacher is not None
+ and self.models.teacher.quantization is not Quantization.NONE
+ ):
raise ValueError(
"the toy backend does not support quantization "
"(models.teacher.quantization must be 'none')"
)
- if self.models.teacher.adapter is not None:
+ if self.models.teacher is not None and self.models.teacher.adapter is not None:
raise ValueError(
"the toy backend cannot load a PEFT teacher adapter; use models.backend: hf"
)
@@ -1202,14 +1340,14 @@ def _validate_combination(self) -> RunConfig:
)
if (
alignment.teacher_mode is TeacherMode.POLICY_CONDITIONED
- and self.models.teacher.mode is not TeacherContextMode.PRIVILEGED_CONTEXT
+ and self.models.required_teacher.mode is not TeacherContextMode.PRIVILEGED_CONTEXT
):
raise ValueError(
"policy_conditioned alignment requires models.teacher.mode=privileged_context"
)
if (
alignment.teacher_mode is TeacherMode.ALIGNED_ADAPTER
- and self.models.teacher.adapter is None
+ and self.models.required_teacher.adapter is None
):
raise ValueError(
"aligned_adapter alignment requires a frozen models.teacher.adapter"
@@ -1228,14 +1366,17 @@ def _validate_combination(self) -> RunConfig:
@property
def is_on_policy(self) -> bool:
- """``True`` only when the full strict OPD freshness contract holds."""
+ """``True`` when one fresh rollout batch feeds exactly one update."""
+ trajectories = self.train.rollouts_per_cycle * (
+ self.rollout.samples_per_prompt if self.run.mode is TrainingMode.RL else 1
+ )
steps_per_batch = max(
1,
- (self.train.rollouts_per_cycle + self.train.gradient_accumulation_steps - 1)
+ (trajectories + self.train.gradient_accumulation_steps - 1)
// self.train.gradient_accumulation_steps,
)
return (
- self.run.mode is TrainingMode.OPD
+ self.run.mode in {TrainingMode.OPD, TrainingMode.RL}
and self.train.opd_freshness is OPDFreshness.STRICT
and steps_per_batch == 1
and self.cache.strict_policy_version
@@ -1316,7 +1457,11 @@ def submitted_bytes(self) -> bytes | None:
def resolved_for_runtime(self) -> RunConfig:
"""Return a deep copy with local paths resolved, without mutating provenance."""
runtime = self.model_copy(deep=True)
- for adapter in (runtime.models.student.adapter, runtime.models.teacher.adapter):
+ for adapter in (
+ runtime.models.student.adapter,
+ runtime.models.teacher.adapter if runtime.models.teacher is not None else None,
+ runtime.models.reference.adapter if runtime.models.reference is not None else None,
+ ):
if adapter is not None and adapter.source is AdapterSource.LOCAL:
adapter_path = Path(adapter.path)
if not adapter_path.is_absolute():
diff --git a/src/miniverl/evaluation/teacher_gate.py b/src/miniverl/evaluation/teacher_gate.py
index 473f71a..7d914a9 100644
--- a/src/miniverl/evaluation/teacher_gate.py
+++ b/src/miniverl/evaluation/teacher_gate.py
@@ -155,8 +155,8 @@ def evaluate_teacher_candidate(
"test_tasks_generated": 0,
"student_policy_used": False,
"teacher": {
- "model_id": config.models.teacher.model_id,
- "revision": config.models.teacher.revision,
+ "model_id": config.models.required_teacher.model_id,
+ "revision": config.models.required_teacher.revision,
"adapter": getattr(teacher, "adapter_provenance", None),
},
"metrics": metrics,
diff --git a/src/miniverl/losses/bucketed.py b/src/miniverl/losses/bucketed.py
index 80515e9..dc60e37 100644
--- a/src/miniverl/losses/bucketed.py
+++ b/src/miniverl/losses/bucketed.py
@@ -286,10 +286,16 @@ def bucketed_teacher_entropy(
discards its internal spread. Reports label it accordingly.
"""
log_eps = math.log(tail_epsilon)
+ tail = to_float32(teacher_tail_log_prob)
+ # ``teacher_topk_targets`` uses -inf to represent the exactly empty tail
+ # when K covers the vocabulary. Preserve that identity element instead of
+ # turning it into a phantom epsilon-probability bucket. Apply the rule per
+ # row so mixed-vocabulary batches remain correct as well.
+ tail = torch.where(torch.isneginf(tail), tail, tail.clamp_min(log_eps))
buckets = torch.cat(
[
to_float32(teacher_topk_log_probs),
- to_float32(teacher_tail_log_prob).clamp_min(log_eps).unsqueeze(-1),
+ tail.unsqueeze(-1),
],
dim=-1,
)
diff --git a/src/miniverl/losses/chunked.py b/src/miniverl/losses/chunked.py
index aca2662..cf75a07 100644
--- a/src/miniverl/losses/chunked.py
+++ b/src/miniverl/losses/chunked.py
@@ -47,6 +47,7 @@
"BucketedTargetProvider",
"VerlTopKTargetProvider",
"VerlPGK1TargetProvider",
+ "VerlRLTargetProvider",
"LossOutput",
"chunked_selected_position_loss",
]
@@ -256,6 +257,62 @@ def teacher_entropy(self, start: int, end: int) -> torch.Tensor:
return torch.full((end - start,), float("nan"), dtype=torch.float32)
+@dataclass
+class VerlRLTargetProvider:
+ """Reward-derived token advantages and behavior-policy anchors for local RL."""
+
+ target_token_ids: torch.Tensor
+ old_actor_log_probs: torch.Tensor
+ advantages: torch.Tensor
+ temperature: float = 1.0
+ clip_ratio: float = 0.2
+ clip_ratio_low: float = 0.2
+ clip_ratio_high: float = 0.2
+ clip_ratio_c: float = 3.0
+ algorithm: str = "grpo"
+ reward_kl_mean: float = 0.0
+ reward_kl_coef: float = 0.0
+ kind: str = "verl_rl_policy"
+ diagnostics: list[dict[str, torch.Tensor | float | str]] = field(default_factory=list)
+
+ def divergence(self, start: int, end: int, student_logits: torch.Tensor) -> torch.Tensor:
+ from miniverl.algorithms.policy import clipped_policy_loss
+
+ targets = self.target_token_ids[start:end].to(student_logits.device)
+ log_probs = torch.log_softmax(student_logits.to(torch.float32) / self.temperature, dim=-1)
+ current = log_probs.gather(-1, targets.unsqueeze(-1)).squeeze(-1)
+ actor_entropy = -(log_probs.exp() * log_probs).sum(dim=-1)
+ old = self.old_actor_log_probs[start:end].to(student_logits.device)
+ advantages = self.advantages[start:end].to(student_logits.device)
+ response_mask = torch.ones(end - start, device=student_logits.device)
+ output = clipped_policy_loss(
+ current_log_probs=current,
+ old_log_probs=old,
+ advantages=advantages,
+ response_mask=response_mask,
+ clip_ratio=self.clip_ratio,
+ clip_ratio_low=self.clip_ratio_low,
+ clip_ratio_high=self.clip_ratio_high,
+ clip_ratio_c=self.clip_ratio_c,
+ )
+ self.diagnostics.append(
+ {
+ "algorithm": self.algorithm,
+ "reward_kl_mean": self.reward_kl_mean,
+ "reward_kl_coef": self.reward_kl_coef,
+ "advantages": advantages.detach().cpu(),
+ "ratio": output.ratio.detach().cpu(),
+ "actor_entropy": float(actor_entropy.mean().detach()),
+ **output.metrics,
+ }
+ )
+ return output.per_token_loss
+
+ def teacher_entropy(self, start: int, end: int) -> torch.Tensor:
+ """Teacher entropy is not applicable to teacher-free RL."""
+ return torch.full((end - start,), float("nan"), dtype=torch.float32)
+
+
@dataclass
class LossOutput:
"""Result of one chunked objective evaluation."""
diff --git a/src/miniverl/models/factory.py b/src/miniverl/models/factory.py
index a7f611f..2ebaff3 100644
--- a/src/miniverl/models/factory.py
+++ b/src/miniverl/models/factory.py
@@ -81,9 +81,11 @@ def build_tokenizer(config: RunConfig, *, local_files_only: bool = False) -> Tok
trust_remote_code=student.trust_remote_code,
local_files_only=local_files_only,
)
- teacher_id = teacher.tokenizer_id or teacher.model_id
student_id = student.tokenizer_id or student.model_id
student_revision = student.tokenizer_revision or student.revision
+ if teacher is None:
+ return student_tok
+ teacher_id = teacher.tokenizer_id or teacher.model_id
teacher_revision = teacher.tokenizer_revision or teacher.revision
if (teacher_id, teacher_revision) != (student_id, student_revision):
teacher_tok = HFTokenizerAdapter.load(
@@ -156,6 +158,8 @@ def build_teacher(
) -> CausalLMBackend:
"""Load the frozen scoring model."""
models = config.models
+ if models.teacher is None:
+ raise ConfigError("this training mode has no teacher role")
if models.backend is ModelBackend.TOY:
from miniverl.models.toy import ToyBackend
diff --git a/src/miniverl/models/sampling.py b/src/miniverl/models/sampling.py
index d7c40e8..63620a7 100644
--- a/src/miniverl/models/sampling.py
+++ b/src/miniverl/models/sampling.py
@@ -107,7 +107,10 @@ def run_generation(
generator=generator,
)
if record_logprobs:
- log_probs = torch.log_softmax(logits.detach().to(torch.float32).flatten(), dim=-1)
+ behavior_logits = logits.detach().to(torch.float32).flatten()
+ if temperature > 0.0:
+ behavior_logits = behavior_logits / temperature
+ log_probs = torch.log_softmax(behavior_logits, dim=-1)
logprobs.append(float(log_probs[token].item()))
generated.append(token)
pending = [token]
@@ -277,9 +280,10 @@ def run_cached_padded_generation(
if greedy_logprobs is not None:
logprobs[index].append(greedy_logprobs[index])
else:
- row_log_probs = torch.log_softmax(
- row_logits.detach().to(torch.float32).flatten(), dim=-1
- )
+ behavior_logits = row_logits.detach().to(torch.float32).flatten()
+ if temperature > 0.0:
+ behavior_logits = behavior_logits / temperature
+ row_log_probs = torch.log_softmax(behavior_logits, dim=-1)
logprobs[index].append(float(row_log_probs[token].item()))
generated[index].append(token)
if token == eos_token_id:
diff --git a/src/miniverl/models/shared.py b/src/miniverl/models/shared.py
index e4dd42e..8c91366 100644
--- a/src/miniverl/models/shared.py
+++ b/src/miniverl/models/shared.py
@@ -274,7 +274,11 @@ def load_shared_adapter_backends(
from miniverl.models.adapter_io import validate_teacher_adapter
from miniverl.models.hf import HFBackend
- protocol_version = str(config.environment.params.get("protocol_version", "v1"))
+ protocol_version = (
+ str(config.environment.params.get("protocol_version", "v1"))
+ if config.environment is not None
+ else None
+ )
owner = HFBackend.load(
config.models.student,
device=device,
diff --git a/src/miniverl/qualification.py b/src/miniverl/qualification.py
index 2e58aeb..22da7d7 100644
--- a/src/miniverl/qualification.py
+++ b/src/miniverl/qualification.py
@@ -259,6 +259,25 @@ def _roles_and_gpu_are_exact(self) -> GPUQualification:
raise ValueError(
"v0.11 full qualification evidence is missing: " + ", ".join(details)
)
+ try:
+ requires_v012 = (int(version_parts[0]), int(version_parts[1])) >= (0, 12)
+ except (IndexError, ValueError):
+ requires_v012 = False
+ if requires_v012:
+ v012_checks = {
+ "v012_pinned_verl_v09_compiler",
+ "v012_grpo_nonconstant_reward_update",
+ "v012_exact_wheel_rl_runtime",
+ }
+ missing_v012_checks = sorted(v012_checks - set(self.checks.executed))
+ missing_v012_artifacts = sorted(
+ {"full_v012_rl_result"} - {artifact.name for artifact in self.artifacts}
+ )
+ if missing_v012_checks or missing_v012_artifacts:
+ details = missing_v012_checks + missing_v012_artifacts
+ raise ValueError(
+ "v0.12 full qualification evidence is missing: " + ", ".join(details)
+ )
return self
diff --git a/src/miniverl/release_assets.py b/src/miniverl/release_assets.py
index 18cdbed..3bcde2e 100644
--- a/src/miniverl/release_assets.py
+++ b/src/miniverl/release_assets.py
@@ -54,6 +54,9 @@
),
"full_vllm_runtime_result": ("v011_vllm_runtime", "v011/vllm-runtime.json"),
}
+V012_ARCHIVE_EVIDENCE = {
+ "full_v012_rl_result": ("v012_rl", "v012/rl-qualification.json"),
+}
_SPECIAL_EVIDENCE = {"candidate-manifest.json"}
QUALIFICATION_SUM_FILES = (
"candidate-manifest.json",
@@ -134,6 +137,7 @@ def _validate_archive_mapping() -> None:
for original_name, (semantic_role, archive_path) in {
**ARCHIVE_EVIDENCE,
**V011_ARCHIVE_EVIDENCE,
+ **V012_ARCHIVE_EVIDENCE,
}.items():
if (
not original_name
@@ -162,6 +166,12 @@ def _archive_evidence_for_version(version: str) -> dict[str, tuple[str, str]]:
is_v011 = False
if is_v011:
mapping.update(V011_ARCHIVE_EVIDENCE)
+ try:
+ is_v012 = (int(version_parts[0]), int(version_parts[1])) >= (0, 12)
+ except (IndexError, ValueError):
+ is_v012 = False
+ if is_v012:
+ mapping.update(V012_ARCHIVE_EVIDENCE)
return mapping
diff --git a/src/miniverl/rewards/__init__.py b/src/miniverl/rewards/__init__.py
index 18f14b0..4b91900 100644
--- a/src/miniverl/rewards/__init__.py
+++ b/src/miniverl/rewards/__init__.py
@@ -14,9 +14,13 @@
RewardStatus,
)
from miniverl.rewards.providers import (
+ BatchRewardProvider,
EnvironmentVerifierRewardProvider,
ExactAnswerRewardProvider,
RewardProvider,
+ TargetLengthRewardProvider,
+ WeightedRewardProvider,
+ score_reward_requests,
)
__all__ = [
@@ -24,12 +28,16 @@
"AdvantageComposition",
"AdvantageComposer",
"AdvantageMode",
+ "BatchRewardProvider",
"EnvironmentVerifierRewardProvider",
"ExactAnswerRewardProvider",
+ "TargetLengthRewardProvider",
"RewardComponent",
"RewardProvider",
"RewardProviderIdentity",
"RewardRequest",
"RewardResult",
"RewardStatus",
+ "WeightedRewardProvider",
+ "score_reward_requests",
]
diff --git a/src/miniverl/rewards/providers.py b/src/miniverl/rewards/providers.py
index 5a7c621..eac8496 100644
--- a/src/miniverl/rewards/providers.py
+++ b/src/miniverl/rewards/providers.py
@@ -4,9 +4,12 @@
import hashlib
import json
+import math
import time
+from collections.abc import Sequence
from typing import Any, Protocol, runtime_checkable
+from miniverl.errors import ConfigError
from miniverl.rewards.models import (
RewardComponent,
RewardProviderIdentity,
@@ -16,9 +19,13 @@
)
__all__ = [
+ "BatchRewardProvider",
"EnvironmentVerifierRewardProvider",
"ExactAnswerRewardProvider",
+ "TargetLengthRewardProvider",
"RewardProvider",
+ "WeightedRewardProvider",
+ "score_reward_requests",
]
@@ -36,6 +43,134 @@ def identity(self) -> RewardProviderIdentity: ...
def score(self, request: RewardRequest) -> RewardResult: ...
+@runtime_checkable
+class BatchRewardProvider(Protocol):
+ @property
+ def identity(self) -> RewardProviderIdentity: ...
+
+ def score_batch(self, requests: Sequence[RewardRequest]) -> Sequence[RewardResult]: ...
+
+
+def score_reward_requests(
+ provider: RewardProvider,
+ requests: Sequence[RewardRequest],
+) -> list[RewardResult]:
+ """Score in order and verify that each result is bound to its request."""
+ try:
+ if isinstance(provider, BatchRewardProvider):
+ results = list(provider.score_batch(requests))
+ else:
+ results = [provider.score(request) for request in requests]
+ except Exception as exc:
+ raise ConfigError(f"reward provider {provider.identity.name!r} raised: {exc}") from exc
+ if len(results) != len(requests):
+ raise ConfigError(
+ f"reward provider returned {len(results)} results for {len(requests)} requests"
+ )
+ for request, result in zip(requests, results, strict=True):
+ expected = (
+ request.trajectory_id,
+ request.prompt_group_id,
+ request.sample_index,
+ request.samples_per_prompt,
+ request.input_digest,
+ provider.identity,
+ )
+ actual = (
+ result.trajectory_id,
+ result.prompt_group_id,
+ result.sample_index,
+ result.samples_per_prompt,
+ result.input_digest,
+ result.provider,
+ )
+ if actual != expected:
+ raise ConfigError(
+ "reward result identity does not match its request/provider; "
+ f"trajectory={request.trajectory_id!r}"
+ )
+ return results
+
+
+class WeightedRewardProvider:
+ """Deterministically combine trusted providers with explicit scalar weights."""
+
+ def __init__(self, components: Sequence[tuple[str, float, RewardProvider]]) -> None:
+ if not components:
+ raise ValueError("weighted reward requires at least one component")
+ names = [name for name, _, _ in components]
+ if len(set(names)) != len(names) or any(not name for name in names):
+ raise ValueError("weighted reward component names must be unique and non-empty")
+ if any(not math.isfinite(weight) for _, weight, _ in components):
+ raise ValueError("weighted reward component weights must be finite")
+ self.components = tuple(components)
+ config = [
+ {
+ "name": name,
+ "weight": weight,
+ "provider_identity": provider.identity.model_dump(mode="json"),
+ }
+ for name, weight, provider in self.components
+ ]
+ self._identity = RewardProviderIdentity(
+ name="weighted_composite",
+ version="miniverl-weighted-reward-v1",
+ config_digest=_config_digest(config),
+ package_name="miniverl",
+ deterministic=all(provider.identity.deterministic for _, _, provider in components),
+ )
+
+ @property
+ def identity(self) -> RewardProviderIdentity:
+ return self._identity
+
+ def score(self, request: RewardRequest) -> RewardResult:
+ started = time.perf_counter()
+ values: list[tuple[str, float, float, str]] = []
+ for name, weight, provider in self.components:
+ child = score_reward_requests(provider, [request])[0]
+ if child.status is not RewardStatus.OK or child.raw_reward is None:
+ return RewardResult(
+ trajectory_id=request.trajectory_id,
+ prompt_group_id=request.prompt_group_id,
+ sample_index=request.sample_index,
+ samples_per_prompt=request.samples_per_prompt,
+ provider=self.identity,
+ input_digest=request.input_digest,
+ raw_reward=None,
+ status=RewardStatus.ERROR,
+ failure_category=f"component:{name}:{child.failure_category or child.status.value}",
+ detail=child.detail,
+ duration_ms=max((time.perf_counter() - started) * 1000.0, 0.0),
+ deterministic=self.identity.deterministic,
+ )
+ values.append((name, weight, float(child.raw_reward), child.provider.name))
+ total = sum(weight * value for _, weight, value, _ in values)
+ return RewardResult(
+ trajectory_id=request.trajectory_id,
+ prompt_group_id=request.prompt_group_id,
+ sample_index=request.sample_index,
+ samples_per_prompt=request.samples_per_prompt,
+ provider=self.identity,
+ input_digest=request.input_digest,
+ raw_reward=total,
+ components=tuple(
+ RewardComponent(
+ name=name,
+ value=weight * value,
+ detail=f"weight={weight}; provider={provider_name}",
+ )
+ for name, weight, value, provider_name in values
+ ),
+ status=RewardStatus.OK,
+ duration_ms=max((time.perf_counter() - started) * 1000.0, 0.0),
+ deterministic=self.identity.deterministic,
+ )
+
+ def score_batch(self, requests: Sequence[RewardRequest]) -> Sequence[RewardResult]:
+ return [self.score(request) for request in requests]
+
+
class ExactAnswerRewardProvider:
"""Strict string equality over declared Parquet ground truth."""
@@ -122,6 +257,77 @@ def _result(
)
+class TargetLengthRewardProvider:
+ """Score closeness to an explicit response-length target without loading code."""
+
+ def __init__(self, *, strip_whitespace: bool = True) -> None:
+ self.strip_whitespace = strip_whitespace
+ self._identity = RewardProviderIdentity(
+ name="builtin_target_length",
+ version="miniverl-target-length-v1",
+ config_digest=_config_digest({"strip_whitespace": strip_whitespace}),
+ package_name="miniverl",
+ deterministic=True,
+ )
+
+ @property
+ def identity(self) -> RewardProviderIdentity:
+ return self._identity
+
+ def score(self, request: RewardRequest) -> RewardResult:
+ started = time.perf_counter()
+ metadata = request.reward_model
+ target = metadata.get("characters") if isinstance(metadata, dict) else None
+ valid = (
+ isinstance(metadata, dict)
+ and metadata.get("style") == "target_length"
+ and isinstance(target, int)
+ and not isinstance(target, bool)
+ and 1 <= target <= 1_000_000
+ )
+ if not valid:
+ return RewardResult(
+ trajectory_id=request.trajectory_id,
+ prompt_group_id=request.prompt_group_id,
+ sample_index=request.sample_index,
+ samples_per_prompt=request.samples_per_prompt,
+ provider=self.identity,
+ input_digest=request.input_digest,
+ raw_reward=None,
+ status=RewardStatus.ERROR,
+ failure_category="invalid_reward_metadata",
+ detail=(
+ "reward_model must declare style target_length and an integer "
+ "characters target in [1, 1000000]"
+ ),
+ duration_ms=max((time.perf_counter() - started) * 1000.0, 0.0),
+ deterministic=True,
+ )
+ assert isinstance(target, int)
+ response = request.response_text.strip() if self.strip_whitespace else request.response_text
+ distance = abs(len(response) - target)
+ score = max(0.0, 1.0 - distance / target)
+ return RewardResult(
+ trajectory_id=request.trajectory_id,
+ prompt_group_id=request.prompt_group_id,
+ sample_index=request.sample_index,
+ samples_per_prompt=request.samples_per_prompt,
+ provider=self.identity,
+ input_digest=request.input_digest,
+ raw_reward=score,
+ components=(
+ RewardComponent(
+ name="target_length",
+ value=score,
+ detail=f"observed_characters={len(response)}; target_characters={target}",
+ ),
+ ),
+ status=RewardStatus.OK,
+ duration_ms=max((time.perf_counter() - started) * 1000.0, 0.0),
+ deterministic=True,
+ )
+
+
class EnvironmentVerifierRewardProvider:
"""Adapter for the active deterministic ToolEnvironment verifier."""
@@ -143,6 +349,53 @@ def identity(self) -> RewardProviderIdentity:
def score(self, request: RewardRequest) -> RewardResult:
started = time.perf_counter()
+ recorded = request.reward_model
+ if isinstance(recorded, dict) and recorded.get("style") == "recorded_environment_verifier":
+ reward = recorded.get("reward")
+ if (
+ not isinstance(reward, (int, float))
+ or isinstance(reward, bool)
+ or not math.isfinite(float(reward))
+ ):
+ return RewardResult(
+ trajectory_id=request.trajectory_id,
+ prompt_group_id=request.prompt_group_id,
+ sample_index=request.sample_index,
+ samples_per_prompt=request.samples_per_prompt,
+ provider=self.identity,
+ input_digest=request.input_digest,
+ raw_reward=None,
+ status=RewardStatus.ERROR,
+ failure_category="invalid_recorded_verification",
+ detail="recorded environment verification lacks a finite numeric reward",
+ duration_ms=max((time.perf_counter() - started) * 1000.0, 0.0),
+ deterministic=True,
+ )
+ return RewardResult(
+ trajectory_id=request.trajectory_id,
+ prompt_group_id=request.prompt_group_id,
+ sample_index=request.sample_index,
+ samples_per_prompt=request.samples_per_prompt,
+ provider=self.identity,
+ input_digest=request.input_digest,
+ raw_reward=float(reward),
+ components=(
+ RewardComponent(
+ name="environment_verifier",
+ value=float(reward),
+ detail=str(recorded.get("detail") or "recorded during agent rollout"),
+ ),
+ ),
+ status=RewardStatus.OK,
+ failure_category=(
+ str(recorded["failure_category"])
+ if recorded.get("failure_category") is not None
+ else None
+ ),
+ detail=str(recorded.get("detail") or "") or None,
+ duration_ms=max((time.perf_counter() - started) * 1000.0, 0.0),
+ deterministic=True,
+ )
try:
verification = self.environment.verify(request.response_text)
except Exception as exc:
diff --git a/src/miniverl/training/checkpoint.py b/src/miniverl/training/checkpoint.py
index 1e2b5bb..a151dc5 100644
--- a/src/miniverl/training/checkpoint.py
+++ b/src/miniverl/training/checkpoint.py
@@ -65,6 +65,7 @@ class CheckpointState:
backend_sync_identity: str = ""
reward_provider_identity_digest: str | None = None
advantage_composer_version: str | None = None
+ algorithm_identity: dict[str, Any] | None = None
scheduler: dict[str, Any] = field(default_factory=dict)
scaler: dict[str, Any] | None = None
optimizer_param_groups: list[dict[str, Any]] = field(default_factory=list)
@@ -110,6 +111,7 @@ def to_dict(self) -> dict[str, Any]:
"backend_sync_identity": self.backend_sync_identity,
"reward_provider_identity_digest": self.reward_provider_identity_digest,
"advantage_composer_version": self.advantage_composer_version,
+ "algorithm_identity": self.algorithm_identity,
"scheduler": self.scheduler,
"scaler": self.scaler,
"optimizer_param_groups": self.optimizer_param_groups,
diff --git a/src/miniverl/training/trainer.py b/src/miniverl/training/trainer.py
index af61384..ede0fd3 100644
--- a/src/miniverl/training/trainer.py
+++ b/src/miniverl/training/trainer.py
@@ -46,6 +46,7 @@
OPDFreshness,
Quantization,
RewardProviderKind,
+ RLAlgorithm,
RunConfig,
SourceKind,
TeacherContextMode,
@@ -75,6 +76,7 @@
SpanType,
Trajectory,
VerificationRecord,
+ derive_grouped_trajectory_id,
)
from miniverl.selection.selectors import (
SelectionResult,
@@ -138,6 +140,7 @@ class TrainSample:
alignment: AlignmentMap
selection: SelectionResult
teacher: TeacherScoreResult | None = None
+ objective_provider: Any | None = None
@dataclass
@@ -261,23 +264,35 @@ def __init__(
self.reference = reference
self.reward_provider: Any | None = reward_provider
- if config.loss.mode is LossMode.VERL_PG_K1_REWARDED:
+ if config.loss.mode in {LossMode.VERL_PG_K1_REWARDED, LossMode.VERL_RL_POLICY}:
from miniverl.rewards.providers import (
+ EnvironmentVerifierRewardProvider,
ExactAnswerRewardProvider,
RewardProvider,
+ TargetLengthRewardProvider,
)
if config.reward.provider is RewardProviderKind.EXACT_ANSWER:
if reward_provider is not None:
raise ConfigError("an injected provider requires reward.provider=python_api")
self.reward_provider = ExactAnswerRewardProvider()
+ elif config.reward.provider is RewardProviderKind.TARGET_LENGTH:
+ if reward_provider is not None:
+ raise ConfigError("an injected provider requires reward.provider=python_api")
+ self.reward_provider = TargetLengthRewardProvider()
+ elif config.reward.provider is RewardProviderKind.ENVIRONMENT_VERIFIER:
+ if reward_provider is not None:
+ raise ConfigError("an injected provider requires reward.provider=python_api")
+ if environment is None:
+ raise ConfigError("environment_verifier requires a ToolEnvironment")
+ self.reward_provider = EnvironmentVerifierRewardProvider(environment)
elif reward_provider is None or not isinstance(reward_provider, RewardProvider):
raise ConfigError(
"reward.provider=python_api requires a RewardProvider object passed to "
"OPDTrainer.from_config(..., reward_provider=...)"
)
elif reward_provider is not None:
- raise ConfigError("an injected reward provider requires the rewarded PG profile")
+ raise ConfigError("an injected reward provider requires a reward-driven policy profile")
self.reward_log = JsonlWriter(paths.root / "rewards.jsonl")
self.advantage_log = JsonlWriter(paths.root / "advantages.jsonl")
self.artifact_bridge = LocalArtifactBridge(paths.root)
@@ -344,7 +359,7 @@ def optimizer_steps_per_cycle(self) -> int:
rollouts = self.config.train.rollouts_per_cycle
trajectories = (
rollouts * self.config.rollout.samples_per_prompt
- if self.config.source.kind is SourceKind.VERL_PARQUET
+ if self.config.run.mode is TrainingMode.RL
else rollouts
)
return max(1, (trajectories + accum - 1) // accum)
@@ -452,9 +467,9 @@ def from_config(
# bitsandbytes 4-bit parameters are pinned to the device they were
# quantized on, so a quantized model cannot be moved off the GPU and
# back. `swap` is therefore only available for unquantized pairs.
- quantized = (
- config.models.student.quantization is not Quantization.NONE
- or config.models.teacher.quantization is not Quantization.NONE
+ quantized = config.models.student.quantization is not Quantization.NONE or (
+ config.models.teacher is not None
+ and config.models.teacher.quantization is not Quantization.NONE
)
memory_config = config.memory
from miniverl.bridge.opd_capabilities import assert_runtime_placement_legal
@@ -462,7 +477,11 @@ def from_config(
assert_runtime_placement_legal(
strategy=memory_config.strategy.value,
student_quantization=config.models.student.quantization.value,
- teacher_quantization=config.models.teacher.quantization.value,
+ teacher_quantization=(
+ config.models.teacher.quantization.value
+ if config.models.teacher is not None
+ else Quantization.NONE.value
+ ),
)
if (
config.models.runtime is ModelRuntime.SHARED_BACKBONE
@@ -580,7 +599,10 @@ def from_config(
raise ConfigError(
"eval task range exceeds the number of Parquet validation rows"
)
- if config.models.teacher.mode is TeacherContextMode.PRIVILEGED_CONTEXT:
+ if (
+ config.models.teacher is not None
+ and config.models.teacher.mode is TeacherContextMode.PRIVILEGED_CONTEXT
+ ):
assert environment is not None
probe = environment.privileged_context(splits["train"][0])
if not probe:
@@ -601,7 +623,8 @@ def from_config(
device=device,
local_files_only=local_files_only,
include_teacher=(
- config.run.mode is not TrainingMode.SFT and not for_evaluation
+ config.run.mode in {TrainingMode.OPD, TrainingMode.OFFLINE_KD}
+ and not for_evaluation
),
)
plan.reason = "shared_backbone -> resident: all policy roles share one base"
@@ -616,7 +639,7 @@ def from_config(
)
if (
config.models.runtime is ModelRuntime.DUAL_MODEL
- and config.run.mode is not TrainingMode.SFT
+ and config.run.mode in {TrainingMode.OPD, TrainingMode.OFFLINE_KD}
and not for_evaluation
):
if memory_config.strategy is MemoryStrategy.AUTO and device.startswith("cuda"):
@@ -899,6 +922,8 @@ def build_manifest(self) -> dict[str, Any]:
if config.is_on_policy
else "online_distillation_with_replay"
)
+ elif config.run.mode is TrainingMode.RL:
+ name = config.algorithm.name.value
objective = {
"name": name,
"loss_mode": config.loss.mode.value,
@@ -919,6 +944,11 @@ def build_manifest(self) -> dict[str, Any]:
if config.run.mode is TrainingMode.OPD
else None
),
+ "algorithm": (
+ config.algorithm.model_dump(mode="json")
+ if config.run.mode is TrainingMode.RL
+ else None
+ ),
}
if self.environment is not None and config.environment is not None:
environment_info: dict[str, Any] | None = {
@@ -984,12 +1014,12 @@ def build_manifest(self) -> dict[str, Any]:
},
"teacher": (
{
- "model_id": config.models.teacher.model_id,
- "revision": config.models.teacher.revision,
- "tokenizer_revision": config.models.teacher.tokenizer_revision,
- "quantization": config.models.teacher.quantization.value,
- "precision": config.models.teacher.dtype.value,
- "context_mode": config.models.teacher.mode.value,
+ "model_id": config.models.required_teacher.model_id,
+ "revision": config.models.required_teacher.revision,
+ "tokenizer_revision": config.models.required_teacher.tokenizer_revision,
+ "quantization": config.models.required_teacher.quantization.value,
+ "precision": config.models.required_teacher.dtype.value,
+ "context_mode": config.models.required_teacher.mode.value,
"adapter": getattr(self.teacher, "adapter_provenance", None),
"capabilities": self.teacher.capabilities.to_dict(),
}
@@ -1158,7 +1188,7 @@ def _prepare_toy_teacher(self) -> None:
if (
config.models.backend is not ModelBackend.TOY
or self.teacher is None
- or config.models.teacher.toy_pretrain_steps <= 0
+ or config.models.required_teacher.toy_pretrain_steps <= 0
or self.environment is None
):
return
@@ -1187,9 +1217,9 @@ def _prepare_toy_teacher(self) -> None:
losses = fit_toy_model(
self.teacher,
batches,
- steps=config.models.teacher.toy_pretrain_steps,
- lr=config.models.teacher.toy_pretrain_lr,
- seed=config.models.teacher.toy_teacher_seed,
+ steps=config.models.required_teacher.toy_pretrain_steps,
+ lr=config.models.required_teacher.toy_pretrain_lr,
+ seed=config.models.required_teacher.toy_teacher_seed,
chunk_size=self.plan.chunk_size,
batch_size=config.train.gradient_accumulation_steps,
)
@@ -1297,6 +1327,131 @@ def _collect(
stats.observe(trajectory)
return trajectories, stats
assert self.runner is not None
+ if self.config.run.mode is TrainingMode.RL:
+ from miniverl.runtime.generation import RolloutBackendKind, derive_sample_seed
+ from miniverl.runtime.policy_sync import build_rollout_policy_identity
+
+ task_cursor_before = self.task_cursor - len(tasks)
+ group_cursor_before = self._rollout_group_cursor
+ identity = build_rollout_policy_identity(
+ backend=self.student,
+ parameter_version=self.policy_version,
+ generation_backend=RolloutBackendKind.HF_REFERENCE,
+ backend_version="agent-hf-reference-v1",
+ profile_identity=self.config.run.profile_identity,
+ execution_plan_digest=self.config.run.execution_plan_digest,
+ )
+ self._backend_sync_identity = identity.digest
+ self._pending_group_identity = []
+ try:
+ for offset, task in enumerate(tasks):
+ prompt_digest = hashlib.sha256(
+ (
+ f"{self.environment.name if self.environment else ''}\0"
+ f"{task.task_id}\0{self.environment.system_prompt() if self.environment else ''}"
+ ).encode()
+ ).hexdigest()
+ group_id = f"g{group_cursor_before + offset:012d}-{prompt_digest[:12]}"
+ self._pending_group_identity.append(group_id)
+ for sample_index in range(self.config.rollout.samples_per_prompt):
+ generation_seed = derive_sample_seed(
+ run_seed=(
+ rollout_seed_base
+ if rollout_seed_base is not None
+ else self.config.run.seed + self.global_step * 1013
+ ),
+ prompt_digest=prompt_digest,
+ policy_version=self.policy_version,
+ sample_index=sample_index,
+ )
+ legacy = self.runner.rollout(
+ task,
+ policy_version=self.policy_version,
+ seed=generation_seed,
+ )
+ verification = legacy.verification
+ reward_metadata = {
+ "style": "recorded_environment_verifier",
+ "reward": float(verification.reward)
+ if verification is not None
+ else 0.0,
+ "failure_category": (
+ verification.failure_category
+ if verification is not None
+ else "no_final_answer"
+ ),
+ "detail": (
+ verification.detail
+ if verification is not None
+ else legacy.termination_reason.value
+ ),
+ }
+ trajectory_id = derive_grouped_trajectory_id(
+ prompt_group_id=group_id,
+ sample_index=sample_index,
+ rollout_policy_identity_digest=identity.digest,
+ generation_seed=generation_seed,
+ )
+ trajectory = Trajectory.model_validate(
+ {
+ **legacy.model_dump(mode="python"),
+ "schema_version": TRAJECTORY_SCHEMA_VERSION,
+ "trajectory_id": trajectory_id,
+ "prompt_group_id": group_id,
+ "prompt_digest": prompt_digest,
+ "sample_index": sample_index,
+ "samples_per_prompt": self.config.rollout.samples_per_prompt,
+ "generation_seed": generation_seed,
+ "rollout_backend": "agent_hf_reference",
+ "rollout_policy_identity_digest": identity.digest,
+ "metadata": {
+ **legacy.metadata,
+ "row_digest": prompt_digest,
+ "data_source": f"environment:{legacy.environment}",
+ "reward_model": reward_metadata,
+ },
+ }
+ )
+ if not any(trajectory.model_generated_mask):
+ raise ConfigError(
+ "RL environment rollout produced no model-generated tokens; "
+ "increase rollout.max_total_tokens beyond the rendered system "
+ "prompt and task context"
+ )
+ trajectories.append(trajectory)
+ transaction_id = hashlib.sha256(
+ f"{self.run_id}:{self.cycle}:{group_cursor_before}:{identity.digest}".encode()
+ ).hexdigest()
+ append_trajectory_groups(
+ self.paths.trajectories,
+ trajectories,
+ transaction_id=transaction_id,
+ )
+ except BaseException as exc:
+ self.task_cursor = task_cursor_before
+ self._pending_group_identity = []
+ if isinstance(exc, KeyboardInterrupt):
+ try:
+ self._save_checkpoint_impl(name="interrupted-group")
+ except BaseException as checkpoint_error:
+ logger.warning(
+ "could not save the rolled-back interruption checkpoint: %s",
+ checkpoint_error,
+ )
+ raise
+ self._committed_group_identity = list(self._pending_group_identity)
+ self._pending_group_identity = []
+ self._rollout_group_cursor += len(tasks)
+ self._trajectory_count += len(trajectories)
+ self._committed_task_cursor = self.task_cursor
+ for trajectory in trajectories:
+ stats.observe(trajectory)
+ self._last_rollout_execution = {
+ "physical_batch_sizes": [1] * len(trajectories),
+ "physical_generation_batches": len(trajectories),
+ "oom_downshifts": 0,
+ }
+ return trajectories, stats
for offset, task in enumerate(tasks):
if oracle:
traj = self.runner.oracle_rollout(
@@ -1344,8 +1499,8 @@ def _open_cache(self) -> TeacherCache:
)
path = Path(self.config.cache.dir or self.paths.teacher_cache)
identity = {
- "teacher_model_id": self.config.models.teacher.model_id,
- "teacher_model_revision": self.config.models.teacher.revision,
+ "teacher_model_id": self.config.models.required_teacher.model_id,
+ "teacher_model_revision": self.config.models.required_teacher.revision,
"tokenizer_fingerprint": self.tokenizer.fingerprint,
"tokenizer_identity": getattr(self.tokenizer, "identity", {}),
"teacher_adapter_provenance": getattr(self.teacher, "adapter_provenance", None),
@@ -1403,7 +1558,7 @@ def _open_cache(self) -> TeacherCache:
return self._cache
def _teacher_identity(self) -> dict[str, Any]:
- teacher = self.config.models.teacher
+ teacher = self.config.models.required_teacher
return {
"model_id": teacher.model_id,
"revision": teacher.revision,
@@ -1578,7 +1733,10 @@ def _load_offline_dataset(self, *, expected_digest: str) -> None:
if expected_schedule and manifest.get("task_schedule_digest") != expected_schedule:
raise CheckpointError("offline dataset task schedule digest changed")
- privileged = self.config.models.teacher.mode is TeacherContextMode.PRIVILEGED_CONTEXT
+ privileged = bool(
+ self.config.models.teacher is not None
+ and self.config.models.teacher.mode is TeacherContextMode.PRIVILEGED_CONTEXT
+ )
task_by_id = {task.task_id: task for split in self.splits.values() for task in split}
samples: list[TrainSample] = []
for trajectory in trajectories:
@@ -1683,7 +1841,10 @@ def _build_samples(self, trajectories: list[Trajectory]) -> list[TrainSample]:
config = self.config
samples: list[TrainSample] = []
selections: list[SelectionStats] = []
- privileged = config.models.teacher.mode is TeacherContextMode.PRIVILEGED_CONTEXT
+ privileged = bool(
+ config.models.teacher is not None
+ and config.models.teacher.mode is TeacherContextMode.PRIVILEGED_CONTEXT
+ )
task_by_id = {t.task_id: t for split in self.splits.values() for t in split}
for traj in trajectories:
@@ -1756,20 +1917,30 @@ def _build_samples(self, trajectories: list[Trajectory]) -> list[TrainSample]:
def _score_task_rewards(self, trajectories: list[Trajectory]) -> None:
"""Score one complete rollout group and attach explicit task advantages."""
- if self.config.loss.mode is not LossMode.VERL_PG_K1_REWARDED:
+ if self.config.loss.mode not in {
+ LossMode.VERL_PG_K1_REWARDED,
+ LossMode.VERL_RL_POLICY,
+ }:
return
if self.reward_provider is None:
- raise ConfigError("rewarded PG requires a configured reward provider")
- from miniverl.rewards import AdvantageComposer, AdvantageMode, RewardRequest, RewardStatus
+ raise ConfigError("reward-driven policy training requires a configured reward provider")
+ from miniverl.rewards import (
+ AdvantageComposer,
+ AdvantageMode,
+ RewardRequest,
+ RewardStatus,
+ score_reward_requests,
+ )
groups: dict[str, list[tuple[Trajectory, Any]]] = {}
+ pending: list[tuple[Trajectory, Any]] = []
for trajectory in trajectories:
if (
trajectory.prompt_group_id is None
or trajectory.sample_index is None
or trajectory.samples_per_prompt is None
):
- raise ConfigError("rewarded PG requires schema-v3 grouped trajectories")
+ raise ConfigError("reward-driven policy training requires grouped trajectories")
metadata = trajectory.metadata
row_digest = metadata.get("row_digest")
data_source = metadata.get("data_source")
@@ -1778,7 +1949,7 @@ def _score_task_rewards(self, trajectories: list[Trajectory]) -> None:
or not isinstance(data_source, str)
or not data_source
):
- raise ConfigError("rewarded PG trajectory is missing bound row/data provenance")
+ raise ConfigError("reward trajectory is missing bound row/data provenance")
extra = metadata.get("extra_info")
ground_truth = extra.get("ground_truth") if isinstance(extra, dict) else None
response = "".join(
@@ -1797,14 +1968,25 @@ def _score_task_rewards(self, trajectories: list[Trajectory]) -> None:
ground_truth=ground_truth,
data_source=data_source,
)
- result = self.reward_provider.score(request)
+ pending.append((trajectory, request))
+ results = score_reward_requests(
+ self.reward_provider,
+ [request for _, request in pending],
+ )
+ for (trajectory, _), result in zip(pending, results, strict=True):
self.reward_log.write(result.model_dump(mode="json"))
if result.status is not RewardStatus.OK or result.raw_reward is None:
raise ConfigError(
f"reward provider failed closed for {trajectory.trajectory_id}: "
f"{result.failure_category or result.status.value}: {result.detail or ''}"
)
- groups.setdefault(trajectory.prompt_group_id, []).append((trajectory, result))
+ group_id = trajectory.prompt_group_id
+ assert group_id is not None
+ groups.setdefault(group_id, []).append((trajectory, result))
+
+ if self.config.loss.mode is LossMode.VERL_RL_POLICY:
+ self._attach_rl_advantages(groups)
+ return
composer = AdvantageComposer(
mode=AdvantageMode(self.config.loss.advantage_mode),
@@ -1848,6 +2030,150 @@ def _score_task_rewards(self, trajectories: list[Trajectory]) -> None:
detail=result.detail,
)
+ def _attach_rl_advantages(self, groups: dict[str, list[tuple[Trajectory, Any]]]) -> None:
+ """Compute one pinned verl estimator over complete reward groups."""
+ import torch
+
+ from miniverl.algorithms.advantages import (
+ gae_advantage_return,
+ grpo_outcome_advantage,
+ reinforce_plus_plus_advantage,
+ rloo_outcome_advantage,
+ )
+
+ ordered: list[tuple[Trajectory, Any]] = []
+ group_ids: list[str] = []
+ for group_id, members in groups.items():
+ members.sort(key=lambda item: int(item[0].sample_index or 0))
+ expected = int(members[0][0].samples_per_prompt or 0)
+ if len(members) != expected:
+ raise ConfigError(f"reward group {group_id!r} is incomplete")
+ ordered.extend(members)
+ group_ids.extend([group_id] * len(members))
+ lengths = [sum(trajectory.model_generated_mask) for trajectory, _ in ordered]
+ if not lengths or any(length < 1 for length in lengths):
+ raise ConfigError("RL requires a non-empty generated response for every sample")
+ width = max(lengths)
+ rewards = torch.zeros((len(ordered), width), dtype=torch.float32)
+ mask = torch.zeros_like(rewards)
+ kl_rows = torch.zeros_like(rewards)
+ for row, ((_, result), length) in enumerate(zip(ordered, lengths, strict=True)):
+ mask[row, :length] = 1.0
+ rewards[row, length - 1] = float(result.raw_reward)
+
+ if self.config.algorithm.kl_coef > 0.0:
+ if self.reference is None: # pragma: no cover - RunConfig/build guard
+ raise ConfigError("RL reference KL requires a loaded reference policy")
+ from miniverl.algorithms.kl import reference_kl_penalty
+
+ for row, (trajectory, _) in enumerate(ordered):
+ target_positions = [
+ index
+ for index, generated in enumerate(trajectory.model_generated_mask)
+ if generated
+ ]
+ prediction_positions = [index - 1 for index in target_positions]
+ targets = torch.tensor(
+ [trajectory.token_ids[index] for index in target_positions],
+ dtype=torch.long,
+ )
+ reference_logits = self.reference.logits_at(
+ trajectory.token_ids,
+ prediction_positions,
+ chunk_size=self.config.loss.chunk_size,
+ )
+ reference_log_probs = (
+ torch.log_softmax(
+ reference_logits.to(torch.float32) / self.config.rollout.temperature,
+ dim=-1,
+ )
+ .gather(-1, targets.to(reference_logits.device).unsqueeze(-1))
+ .squeeze(-1)
+ .cpu()
+ )
+ old_values = trajectory.metadata.get("actor_rollout_log_probs")
+ if not isinstance(old_values, list) or len(old_values) != len(target_positions):
+ raise ConfigError("RL trajectory lacks behavior log-probs for reference KL")
+ old = torch.tensor(old_values, dtype=torch.float32)
+ penalty = reference_kl_penalty(
+ old,
+ reference_log_probs,
+ penalty=self.config.algorithm.kl_penalty,
+ )
+ length = lengths[row]
+ kl_rows[row, :length] = penalty
+ rewards[row, :length] -= self.config.algorithm.kl_coef * penalty
+ trajectory.metadata["reference_log_probs"] = reference_log_probs.tolist()
+ trajectory.metadata["reward_kl"] = {
+ "penalty": self.config.algorithm.kl_penalty,
+ "coefficient": self.config.algorithm.kl_coef,
+ "token_values": penalty.tolist(),
+ "mean": float(penalty.mean()),
+ }
+
+ algorithm = self.config.algorithm.name
+ if algorithm in {RLAlgorithm.GRPO, RLAlgorithm.DR_GRPO}:
+ output = grpo_outcome_advantage(
+ rewards,
+ mask,
+ group_ids,
+ epsilon=self.config.algorithm.epsilon,
+ normalize_by_std=algorithm is RLAlgorithm.GRPO,
+ )
+ elif algorithm is RLAlgorithm.RLOO:
+ output = rloo_outcome_advantage(rewards, mask, group_ids)
+ elif algorithm is RLAlgorithm.REINFORCE_PLUS_PLUS:
+ output = reinforce_plus_plus_advantage(
+ rewards,
+ mask,
+ gamma=self.config.algorithm.gamma,
+ )
+ elif algorithm is RLAlgorithm.PPO: # rejected by RunConfig until critic lifecycle ships
+ values = torch.zeros_like(rewards)
+ output = gae_advantage_return(
+ rewards,
+ values,
+ mask,
+ gamma=self.config.algorithm.gamma,
+ lam=self.config.algorithm.lam,
+ )
+ else: # pragma: no cover - RunConfig closes this set
+ raise ConfigError(f"unsupported RL algorithm {algorithm.value!r}")
+
+ for row, (trajectory, result) in enumerate(ordered):
+ length = lengths[row]
+ advantages = [float(value) for value in output.advantages[row, :length].tolist()]
+ trajectory.metadata["task_reward"] = result.model_dump(mode="json")
+ trajectory.metadata["rl_advantages"] = advantages
+ trajectory.metadata["advantage_estimator"] = {
+ "name": algorithm.value,
+ "implementation_version": self.config.algorithm.implementation_version,
+ "gamma": self.config.algorithm.gamma,
+ "lambda": self.config.algorithm.lam,
+ "epsilon": self.config.algorithm.epsilon,
+ "zero_variance_groups": output.zero_variance_groups,
+ "reward_kl_mean": float(kl_rows[row, :length].mean()),
+ "reward_kl_coefficient": self.config.algorithm.kl_coef,
+ }
+ trajectory.verification = VerificationRecord(
+ solved=bool(result.raw_reward > 0.0),
+ reward=float(result.raw_reward),
+ failure_category=result.failure_category,
+ detail=result.detail,
+ )
+ self.advantage_log.write(
+ {
+ "trajectory_id": trajectory.trajectory_id,
+ "prompt_group_id": trajectory.prompt_group_id,
+ "sample_index": trajectory.sample_index,
+ "reward_input_digest": result.input_digest,
+ "reward_provider_identity_digest": result.provider.digest,
+ "raw_task_reward": result.raw_reward,
+ "advantages": advantages,
+ **trajectory.metadata["advantage_estimator"],
+ }
+ )
+
# -- optimization --------------------------------------------------------------
def _loss_by_span_type(
@@ -1910,6 +2236,9 @@ def _compute_group_gradients(
verl_pg_advantages: list[Any] = []
verl_pg_ratios: list[Any] = []
verl_pg_metric_rows: list[dict[str, float]] = []
+ rl_advantages: list[Any] = []
+ rl_ratios: list[Any] = []
+ rl_metric_rows: list[dict[str, float]] = []
group_scale = 1.0 / max(len(group), 1)
token_mean = config.loss.aggregation is LossAggregation.TOKEN_MEAN
group_weight_total = sum(sum(sample.alignment.token_weights) for sample in group)
@@ -1952,7 +2281,11 @@ def _compute_group_gradients(
]
)
providers = [
- sample.teacher.provider if sample.teacher is not None else None
+ sample.objective_provider
+ if sample.objective_provider is not None
+ else sample.teacher.provider
+ if sample.teacher is not None
+ else None
for sample in samples
]
if all(provider is None for provider in providers):
@@ -2005,6 +2338,16 @@ def _compute_group_gradients(
if isinstance(value, (int, float))
}
)
+ elif "advantages" in values and "ratio" in values:
+ rl_advantages.append(values["advantages"])
+ rl_ratios.append(values["ratio"])
+ rl_metric_rows.append(
+ {
+ key: float(value)
+ for key, value in values.items()
+ if isinstance(value, (int, float))
+ }
+ )
diagnostics.clear()
loss_total += float(output.loss) * microbatch_scale
positions_total += output.num_positions
@@ -2047,7 +2390,7 @@ def _compute_group_gradients(
entropy_count += int(sample.teacher.teacher_entropy.numel())
del hidden, output
- result = {
+ result: dict[str, Any] = {
"loss": loss_total,
"selected_positions": positions_total,
"trajectories_in_step": len(group),
@@ -2099,6 +2442,20 @@ def _compute_group_gradients(
for key in metric_names
},
}
+ if rl_advantages:
+ advantages = torch.cat(rl_advantages).float()
+ ratios = torch.cat(rl_ratios).float()
+ metric_names = sorted({key for row in rl_metric_rows for key in row})
+ result["verl_rl"] = {
+ "algorithm": config.algorithm.name.value,
+ "advantage_mean": float(advantages.mean()),
+ "advantage_std": float(advantages.std(unbiased=False)),
+ "ratio_mean": float(ratios.mean()),
+ **{
+ key: sum(row.get(key, 0.0) for row in rl_metric_rows) / len(rl_metric_rows)
+ for key in metric_names
+ },
+ }
return result
def _commit_update(self) -> dict[str, float]:
@@ -2155,12 +2512,12 @@ def _optimize(self, samples: list[TrainSample], *, phase: str) -> list[dict[str,
next(iter(rollout_versions)) if rollout_versions else self.parameter_version
)
if (
- self.config.run.mode is TrainingMode.OPD
+ self.config.run.mode in {TrainingMode.OPD, TrainingMode.RL}
and self.config.train.opd_freshness is OPDFreshness.STRICT
and rollout_policy_version != self.parameter_version
):
raise LifecycleError(
- "strict OPD requires rollout policy version to equal the current parameter "
+ "strict on-policy training requires rollout policy version to equal the current parameter "
f"version before update (rollout={rollout_policy_version}, "
f"parameters={self.parameter_version})"
)
@@ -2534,6 +2891,63 @@ def _build_samples_ce_only(self, trajectories: list[Trajectory]) -> list[TrainSa
self._last_selection_stats = selections
return samples
+ def _build_rl_samples(self, trajectories: list[Trajectory]) -> list[TrainSample]:
+ """Bind reward advantages and rollout-policy log-probs to selected response tokens."""
+ import torch
+
+ from miniverl.losses.chunked import VerlRLTargetProvider
+
+ samples: list[TrainSample] = []
+ selections: list[SelectionStats] = []
+ for trajectory in trajectories:
+ selection = select_positions(
+ trajectory,
+ self.config.selection,
+ run_seed=self.config.run.seed,
+ )
+ selections.append(selection.stats)
+ if not selection.positions:
+ continue
+ alignment = build_alignment_map(trajectory, selection.positions, selection.weights)
+ old = trajectory.metadata.get("actor_rollout_log_probs")
+ advantages = trajectory.metadata.get("rl_advantages")
+ generated_positions = [
+ index
+ for index, generated in enumerate(trajectory.model_generated_mask)
+ if generated
+ ]
+ if selection.positions != generated_positions:
+ raise ConfigError(
+ "RL selection must include every generated response token in order"
+ )
+ if not isinstance(old, list) or len(old) != len(generated_positions):
+ raise ConfigError("RL trajectory lacks aligned behavior-policy log-probabilities")
+ if not isinstance(advantages, list) or len(advantages) != len(generated_positions):
+ raise ConfigError("RL trajectory lacks aligned reward advantages")
+ provider = VerlRLTargetProvider(
+ target_token_ids=torch.tensor(alignment.target_token_ids, dtype=torch.long),
+ old_actor_log_probs=torch.tensor(old, dtype=torch.float32),
+ advantages=torch.tensor(advantages, dtype=torch.float32),
+ temperature=self.config.rollout.temperature,
+ clip_ratio=self.config.loss.clip_ratio,
+ clip_ratio_low=self.config.loss.clip_ratio_low,
+ clip_ratio_high=self.config.loss.clip_ratio_high,
+ clip_ratio_c=self.config.loss.clip_ratio_c,
+ algorithm=self.config.algorithm.name.value,
+ reward_kl_mean=float((trajectory.metadata.get("reward_kl") or {}).get("mean", 0.0)),
+ reward_kl_coef=self.config.algorithm.kl_coef,
+ )
+ samples.append(
+ TrainSample(
+ trajectory=trajectory,
+ alignment=alignment,
+ selection=selection,
+ objective_provider=provider,
+ )
+ )
+ self._last_selection_stats = selections
+ return samples
+
def _run_cycle(self) -> list[dict[str, Any]]:
config = self.config
mode = config.run.mode
@@ -2582,7 +2996,9 @@ def _run_cycle(self) -> list[dict[str, Any]]:
rollout_policy_version = trajectories[0].policy_version
rollout_seconds = max(time.perf_counter() - rollout_started, 1e-9)
- if mode is TrainingMode.SFT or self.teacher is None:
+ if mode is TrainingMode.RL:
+ samples = self._build_rl_samples(trajectories)
+ elif mode is TrainingMode.SFT or self.teacher is None:
samples = self._build_samples_ce_only(trajectories)
else:
teacher_scoring_started = time.perf_counter()
@@ -2617,7 +3033,9 @@ def _run_cycle(self) -> list[dict[str, Any]]:
else round(stats.to_dict()["success_rate"], 4)
),
reward_status=(
- "not_applicable_pure_opd" if self.prompt_dataset is not None else "measured"
+ "measured"
+ if mode is TrainingMode.RL or self.prompt_dataset is None
+ else "not_applicable_pure_opd"
),
generated_tokens=stats.generated_tokens,
rollout_tokens_per_second=round(stats.generated_tokens / rollout_seconds, 2),
@@ -2662,6 +3080,9 @@ def _run_cycle(self) -> list[dict[str, Any]]:
if mode is TrainingMode.OPD:
for sample in samples:
sample.teacher = None
+ if mode is TrainingMode.RL:
+ for sample in samples:
+ sample.objective_provider = None
selection_stats = aggregate_selection_stats(
self._last_selection_stats or [sample.selection.stats for sample in samples]
)
@@ -3268,6 +3689,11 @@ def _save_checkpoint_impl(self, *, name: str | None = None) -> Path:
if self.config.loss.mode is LossMode.VERL_PG_K1_REWARDED
else None
),
+ algorithm_identity=(
+ self.config.algorithm.model_dump(mode="json")
+ if self.config.run.mode is TrainingMode.RL
+ else None
+ ),
scheduler=self.schedule.state_dict(),
config_digest=self._config_digest(),
resolved_config_digest=self._resolved_config_digest(),
@@ -3340,6 +3766,13 @@ def _load_from_checkpoint_impl(self, directory: str | Path) -> CheckpointState:
raise ConfigError(
"checkpoint advantage composer version does not match the current run"
)
+ expected_algorithm = (
+ self.config.algorithm.model_dump(mode="json")
+ if self.config.run.mode is TrainingMode.RL
+ else None
+ )
+ if validated.state.algorithm_identity != expected_algorithm:
+ raise ConfigError("checkpoint RL algorithm identity does not match the current run")
if validated.state.pending_group_identity:
raise ConfigError(
"checkpoint contains a pending partial rollout group",
diff --git a/tests/cli/test_cli.py b/tests/cli/test_cli.py
index 5eac621..7db3001 100644
--- a/tests/cli/test_cli.py
+++ b/tests/cli/test_cli.py
@@ -253,8 +253,8 @@ def test_root_help_describes_the_current_product_without_promising_v08() -> None
assert result.exit_code == 0
collapsed = _collapse(result.stdout)
- assert "documented subset of verl-style OPD on one consumer GPU" in collapsed
- assert "teacher scoring and actor update locally" in collapsed
+ assert "supported verl experiment semantics for one consumer GPU" in collapsed
+ assert "reward/reference/teacher phases and actor updates" in collapsed
assert "inspectable PEFT artifacts" in collapsed
assert "distributed verl execution" not in collapsed
diff --git a/tests/cli/test_opd_product_cli.py b/tests/cli/test_opd_product_cli.py
index 1b9c514..4b70b35 100644
--- a/tests/cli/test_opd_product_cli.py
+++ b/tests/cli/test_opd_product_cli.py
@@ -253,3 +253,28 @@ def test_data_sample_writes_portable_message_rows(tmp_path) -> None:
assert rows[0]["prompt"][1]["role"] == "user"
assert rows[0]["data_source"] == "miniverl_quickstart"
assert "reward_model" not in rows[0]
+
+
+def test_data_sample_can_emit_bound_exact_answer_rewards(tmp_path) -> None:
+ pytest.importorskip("pyarrow")
+ import pyarrow.parquet as pq
+
+ target = tmp_path / "rl-sample.parquet"
+ result = runner.invoke(
+ app,
+ [
+ "data",
+ "sample",
+ "--format",
+ "verl-parquet",
+ "--task-rewards",
+ "--out",
+ str(target),
+ ],
+ )
+
+ assert result.exit_code == 0, result.output
+ rows = pq.read_table(target).to_pylist()
+ assert rows[0]["data_source"] == "miniverl_quickstart_rl"
+ assert rows[0]["reward_model"] == {"style": "exact", "ground_truth": "provenance"}
+ assert rows[0]["extra_info"]["ground_truth"] == "provenance"
diff --git a/tests/cli/test_verl_bridge_cli.py b/tests/cli/test_verl_bridge_cli.py
index 62e2ead..a72d38a 100644
--- a/tests/cli/test_verl_bridge_cli.py
+++ b/tests/cli/test_verl_bridge_cli.py
@@ -8,6 +8,7 @@
from miniverl.bridge.contract import BRIDGE_PROFILE, VERL_TAG
from miniverl.bridge.opd_v08 import VERL_OPD_V08_PROFILE
+from miniverl.bridge.rl_v09 import VERL_RL_V09_PROFILE
from miniverl.cli import app
@@ -289,6 +290,56 @@ def test_import_verl_v2_writes_a_round_trippable_prompt_opd_profile(tmp_path: Pa
assert round_trip.source.data.train_files == ["train.parquet"]
+def test_import_verl_rl_profile_uses_its_v09_pin_by_default(tmp_path: Path) -> None:
+ source = tmp_path / "verl-rl.yaml"
+ source.write_text(
+ yaml.safe_dump(
+ {
+ "data": {
+ "train_files": ["train.parquet"],
+ "train_batch_size": 1,
+ "max_prompt_length": 32,
+ "max_response_length": 8,
+ },
+ "actor_rollout_ref": {
+ "model": {"path": "Qwen/Qwen3-0.6B"},
+ "actor": {"optim": {"lr": "1e-5"}, "ppo_mini_batch_size": 2},
+ "rollout": {"name": "hf", "n": 2, "temperature": 1.0},
+ },
+ "algorithm": {"adv_estimator": "grpo"},
+ "trainer": {"total_training_steps": 1},
+ "miniverl": {
+ "reward": {"provider": "exact_answer"},
+ "actor": {"lora_enabled": True},
+ },
+ },
+ sort_keys=False,
+ ),
+ encoding="utf-8",
+ )
+ out = tmp_path / "local-rl.yaml"
+
+ result = CliRunner().invoke(
+ app,
+ [
+ "import-verl",
+ "--config",
+ str(source),
+ "--profile",
+ VERL_RL_V09_PROFILE,
+ "--out",
+ str(out),
+ "--json",
+ ],
+ )
+
+ assert result.exit_code == 0, result.output
+ report = json.loads(result.stdout)
+ assert report["status"] == "accepted"
+ assert report["source_verl"]["tag"] == "v0.9.0"
+ assert out.is_file()
+
+
def test_benchmark_export_community_exact_command_needs_no_training_stack(
tmp_path: Path,
) -> None:
diff --git a/tests/conformance/test_verl_v09_rl.py b/tests/conformance/test_verl_v09_rl.py
new file mode 100644
index 0000000..23d3f1e
--- /dev/null
+++ b/tests/conformance/test_verl_v09_rl.py
@@ -0,0 +1,221 @@
+# ruff: noqa: E402 - validate upstream source before torch-backed imports
+"""Numerical and gradient conformance against pinned official verl v0.9.0."""
+
+from __future__ import annotations
+
+import ast
+import os
+import subprocess
+from collections import defaultdict
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Optional
+
+import pytest
+
+torch = pytest.importorskip("torch")
+
+from miniverl.algorithms.advantages import (
+ gae_advantage_return,
+ grpo_outcome_advantage,
+ reinforce_plus_plus_advantage,
+ rloo_outcome_advantage,
+)
+from miniverl.algorithms.contract import UPSTREAM_VERL_COMMIT
+from miniverl.algorithms.kl import reference_kl_penalty
+from miniverl.algorithms.policy import clipped_policy_loss
+from miniverl.algorithms.value import clipped_value_loss
+
+pytestmark = [pytest.mark.torch, pytest.mark.verl_conformance]
+
+
+class _Functional:
+ @staticmethod
+ def masked_sum(values, mask, axis=None): # type: ignore[no-untyped-def]
+ return torch.where(mask.bool(), values, 0.0).mul(mask).sum(dim=axis)
+
+ @classmethod
+ def masked_mean(cls, values, mask, axis=None): # type: ignore[no-untyped-def]
+ return cls.masked_sum(values, mask, axis) / (mask.sum(dim=axis) + 1e-8)
+
+ @classmethod
+ def masked_var(cls, values, mask, unbiased=True): # type: ignore[no-untyped-def]
+ mean = cls.masked_mean(values, mask)
+ variance = cls.masked_mean((values - mean) ** 2, mask)
+ if unbiased:
+ count = mask.sum()
+ variance = variance * count / (count - 1)
+ return variance
+
+ @classmethod
+ def masked_whiten(cls, values, mask, shift_mean=True): # type: ignore[no-untyped-def]
+ mean = cls.masked_mean(values, mask)
+ output = (values - mean) * torch.rsqrt(cls.masked_var(values, mask) + 1e-8)
+ return output if shift_mean else output + mean
+
+ @staticmethod
+ def clip_by_value(value, minimum, maximum): # type: ignore[no-untyped-def]
+ return torch.max(torch.min(value, maximum), minimum)
+
+
+def _official(): # type: ignore[no-untyped-def]
+ root = Path(os.environ.get("MINIVERL_VERL_V09_SOURCE", ".audit/upstream-verl-v0.9.0"))
+ if not (root / ".git").exists():
+ pytest.skip("pinned verl v0.9.0 source checkout is unavailable")
+ commit = subprocess.run(
+ ["git", "rev-parse", "HEAD"], cwd=root, check=True, capture_output=True, text=True
+ ).stdout.strip()
+ assert commit == UPSTREAM_VERL_COMMIT
+ source = root / "verl/trainer/ppo/core_algos.py"
+ tree = ast.parse(source.read_text(encoding="utf-8"), filename=str(source))
+ names = (
+ "agg_loss",
+ "compute_gae_advantage_return",
+ "compute_grpo_outcome_advantage",
+ "compute_rloo_outcome_advantage",
+ "compute_reinforce_plus_plus_outcome_advantage",
+ "compute_policy_loss_vanilla",
+ "compute_value_loss",
+ "kl_penalty_forward",
+ )
+ selected = {
+ node.name: node
+ for node in tree.body
+ if isinstance(node, ast.FunctionDef) and node.name in names
+ }
+ assert set(selected) == set(names)
+ module = ast.Module(
+ body=[
+ ast.ImportFrom(module="__future__", names=[ast.alias(name="annotations")], level=0),
+ *[selected[name] for name in names],
+ ],
+ type_ignores=[],
+ )
+ ast.fix_missing_locations(module)
+ namespace = {
+ "torch": torch,
+ "np": SimpleNamespace(ndarray=object),
+ "Optional": Optional,
+ "defaultdict": defaultdict,
+ "verl_F": _Functional,
+ "AlgoConfig": type("AlgoConfig", (), {}),
+ "ActorConfig": type("ActorConfig", (), {}),
+ "AdvantageEstimator": SimpleNamespace(
+ GAE="gae", GRPO="grpo", RLOO="rloo", REINFORCE_PLUS_PLUS="reinforce_plus_plus"
+ ),
+ "register_adv_est": lambda _name: lambda fn: fn,
+ "register_policy_loss": lambda _name: lambda fn: fn,
+ }
+ exec(compile(module, str(source), "exec"), namespace)
+ return SimpleNamespace(**namespace)
+
+
+class _Actor:
+ clip_ratio = 0.2
+ clip_ratio_low = 0.2
+ clip_ratio_high = 0.2
+ global_batch_info = {} # noqa: RUF012
+
+ def get(self, name, default): # type: ignore[no-untyped-def]
+ return getattr(self, name, default)
+
+
+def test_advantage_estimators_match_pinned_upstream() -> None:
+ official = _official()
+ rewards = torch.tensor([[0.0, 1.0], [0.0, 3.0], [2.0, 0.0], [4.0, 0.0]])
+ mask = torch.tensor([[1.0, 1.0], [1.0, 1.0], [1.0, 0.0], [1.0, 1.0]])
+ groups = ("a", "a", "b", "b")
+ for normalize in (True, False):
+ expected, expected_returns = official.compute_grpo_outcome_advantage(
+ rewards.clone(), mask, groups, norm_adv_by_std_in_grpo=normalize
+ )
+ actual = grpo_outcome_advantage(rewards, mask, groups, normalize_by_std=normalize)
+ torch.testing.assert_close(actual.advantages, expected)
+ torch.testing.assert_close(actual.returns, expected_returns)
+ expected, _ = official.compute_rloo_outcome_advantage(rewards.clone(), mask, groups)
+ torch.testing.assert_close(rloo_outcome_advantage(rewards, mask, groups).advantages, expected)
+
+ config = SimpleNamespace(gamma=0.9)
+ expected, expected_returns = official.compute_reinforce_plus_plus_outcome_advantage(
+ rewards.clone(), mask, config=config
+ )
+ actual = reinforce_plus_plus_advantage(rewards, mask, gamma=0.9)
+ torch.testing.assert_close(actual.advantages, expected)
+ torch.testing.assert_close(actual.returns, expected_returns)
+
+
+def test_gae_policy_value_gradients_and_values_match_pinned_upstream() -> None:
+ official = _official()
+ rewards = torch.tensor([[0.0, 1.0, 0.0], [0.0, 2.0, 0.0]])
+ values = torch.tensor([[0.2, 0.3, 0.0], [0.1, 0.4, 0.0]])
+ mask = torch.tensor([[1.0, 1.0, 0.0], [1.0, 1.0, 0.0]])
+ expected_adv, expected_returns = official.compute_gae_advantage_return(
+ rewards, values, mask, torch.tensor(0.9), torch.tensor(0.8)
+ )
+ actual = gae_advantage_return(rewards, values, mask, gamma=0.9, lam=0.8)
+ torch.testing.assert_close(actual.advantages, expected_adv)
+ torch.testing.assert_close(actual.returns, expected_returns)
+
+ old = torch.tensor([[-1.0, -0.7, 0.0], [-1.1, -0.8, 0.0]])
+ local_current = old.clone().add(0.15).requires_grad_(True)
+ upstream_current = local_current.detach().clone().requires_grad_(True)
+ expected_loss, expected_metrics = official.compute_policy_loss_vanilla(
+ old, upstream_current, expected_adv, mask, config=_Actor()
+ )
+ expected_loss.backward()
+ local = clipped_policy_loss(
+ current_log_probs=local_current,
+ old_log_probs=old,
+ advantages=actual.advantages,
+ response_mask=mask,
+ )
+ local.loss.backward()
+ torch.testing.assert_close(local.loss, expected_loss)
+ torch.testing.assert_close(local_current.grad, upstream_current.grad)
+ assert local.metrics["pg_clipfrac"] == pytest.approx(expected_metrics["actor/pg_clipfrac"])
+
+ current_values = values.clone().add(0.3).requires_grad_(True)
+ upstream_values = current_values.detach().clone().requires_grad_(True)
+ expected_v, expected_clip = official.compute_value_loss(
+ upstream_values, expected_returns, values, mask, 0.2
+ )
+ expected_v.backward()
+ local_v = clipped_value_loss(current_values, values, actual.returns, mask, cliprange_value=0.2)
+ local_v.loss.backward()
+ torch.testing.assert_close(local_v.loss, expected_v)
+ torch.testing.assert_close(current_values.grad, upstream_values.grad)
+ assert local_v.metrics["value_clipfrac"] == pytest.approx(float(expected_clip))
+
+ local_parameter = torch.nn.Parameter(old.clone().add(0.15))
+ upstream_parameter = torch.nn.Parameter(old.clone().add(0.15))
+ local_optimizer = torch.optim.AdamW([local_parameter], lr=3e-4, weight_decay=0.01)
+ upstream_optimizer = torch.optim.AdamW([upstream_parameter], lr=3e-4, weight_decay=0.01)
+ upstream_step_loss, _ = official.compute_policy_loss_vanilla(
+ old, upstream_parameter, expected_adv, mask, config=_Actor()
+ )
+ upstream_step_loss.backward()
+ upstream_optimizer.step()
+ local_step_loss = clipped_policy_loss(
+ current_log_probs=local_parameter,
+ old_log_probs=old,
+ advantages=actual.advantages,
+ response_mask=mask,
+ ).loss
+ local_step_loss.backward()
+ local_optimizer.step()
+ torch.testing.assert_close(local_parameter, upstream_parameter)
+
+
+def test_reference_kl_estimators_match_pinned_upstream() -> None:
+ official = _official()
+ old = torch.tensor([[-1.2, -0.3], [-3.0, -0.8]])
+ reference = torch.tensor([[-1.0, -0.7], [-2.4, -1.2]])
+ for local_name, upstream_name in (
+ ("kl", "kl"),
+ ("abs", "abs"),
+ ("mse", "mse"),
+ ("low_var_kl", "low_var_kl"),
+ ):
+ expected = official.kl_penalty_forward(old, reference, upstream_name)
+ actual = reference_kl_penalty(old, reference, penalty=local_name)
+ torch.testing.assert_close(actual, expected)
diff --git a/tests/integration/test_rl_pipeline.py b/tests/integration/test_rl_pipeline.py
new file mode 100644
index 0000000..382374f
--- /dev/null
+++ b/tests/integration/test_rl_pipeline.py
@@ -0,0 +1,403 @@
+# ruff: noqa: E402 - optional torch dependency is checked before torch-backed imports
+"""End-to-end teacher-free grouped RL over a verl-style Parquet prompt."""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+import pytest
+
+pa = pytest.importorskip("pyarrow")
+pq = pytest.importorskip("pyarrow.parquet")
+pytest.importorskip("torch")
+
+from miniverl.algorithms.contract import ADVANTAGE_IMPLEMENTATION_VERSION
+from miniverl.rewards import RewardProviderIdentity, RewardResult, RewardStatus
+
+pytestmark = pytest.mark.torch
+
+
+class _SampleIndexReward:
+ identity = RewardProviderIdentity(
+ name="test_sample_index",
+ version="v1",
+ config_digest="1" * 64,
+ deterministic=True,
+ )
+
+ def score(self, request): # type: ignore[no-untyped-def]
+ return RewardResult(
+ trajectory_id=request.trajectory_id,
+ prompt_group_id=request.prompt_group_id,
+ sample_index=request.sample_index,
+ samples_per_prompt=request.samples_per_prompt,
+ provider=self.identity,
+ input_digest=request.input_digest,
+ raw_reward=float(request.sample_index),
+ status=RewardStatus.OK,
+ duration_ms=0.0,
+ deterministic=True,
+ )
+
+
+def _trainable_state(trainer) -> dict[str, Any]: # type: ignore[no-untyped-def]
+ return {
+ name: tensor.detach().cpu().clone()
+ for name, tensor in trainer.student.trainable_state_dict().items()
+ }
+
+
+@pytest.mark.parametrize("algorithm", ["grpo", "dr_grpo", "rloo", "reinforce_plus_plus"])
+def test_teacher_free_rl_executes_current_policy_rollout_reward_and_update(
+ tmp_path, algorithm: str
+) -> None: # type: ignore[no-untyped-def]
+ from miniverl.config import RunConfig
+ from miniverl.trainer import OPDTrainer
+
+ train = tmp_path / "train.parquet"
+ pq.write_table(
+ pa.Table.from_pylist(
+ [
+ {
+ "prompt": "Choose one token.",
+ "data_source": "unit",
+ "reward_model": {"style": "exact", "ground_truth": "unused"},
+ "extra_info": {"ground_truth": "unused"},
+ }
+ ]
+ ),
+ train,
+ )
+ config = RunConfig.model_validate(
+ {
+ "run": {
+ "name": "teacher-free-grpo",
+ "mode": "rl",
+ "seed": 17,
+ "output_dir": str(tmp_path / "runs"),
+ },
+ "models": {
+ "backend": "toy",
+ "device": "cpu",
+ "student": {
+ "model_id": "toy-actor",
+ "lora": {"enabled": False},
+ "toy": {
+ "hidden_size": 16,
+ "num_layers": 1,
+ "num_heads": 2,
+ "intermediate_size": 32,
+ "max_position_embeddings": 128,
+ },
+ },
+ },
+ "source": {
+ "kind": "verl_parquet",
+ "train_files": [str(train)],
+ "allow_plain_string_prompts": True,
+ "shuffle": False,
+ "use_task_rewards": True,
+ },
+ "rollout": {
+ "backend": "hf_cached",
+ "samples_per_prompt": 2,
+ "temperature": 1.0,
+ "max_turns": 1,
+ "max_new_tokens_per_turn": 3,
+ "max_total_tokens": 64,
+ "record_logprobs": True,
+ },
+ "selection": {"selector": "all_model_tokens"},
+ "loss": {
+ "mode": "verl_rl_policy",
+ "aggregation": "token-mean",
+ "scale_by_temperature_squared": False,
+ "chunk_size": 16,
+ },
+ "algorithm": {
+ "name": algorithm,
+ "implementation_version": ADVANTAGE_IMPLEMENTATION_VERSION,
+ },
+ "reward": {"enabled": True, "provider": "python_api"},
+ "train": {
+ "cycles": 1,
+ "rollouts_per_cycle": 1,
+ "gradient_accumulation_steps": 2,
+ "learning_rate": 0.001,
+ },
+ "memory": {"strategy": "resident"},
+ "eval": {"enabled": False},
+ "report": {"enabled": False},
+ }
+ )
+
+ trainer = OPDTrainer.from_config(
+ config,
+ run_id="grpo-e2e",
+ reward_provider=_SampleIndexReward(),
+ )
+ try:
+ result = trainer.train()
+ finally:
+ trainer.close()
+
+ assert result.mode == "rl"
+ assert result.global_step == 1
+ rows = [
+ json.loads(line) for line in (result.run_dir / "metrics.jsonl").read_text().splitlines()
+ ]
+ update = next(row for row in rows if row.get("phase") == "rl")
+ assert update["verl_rl"]["algorithm"] == algorithm
+ assert update["verl_rl"]["ratio_mean"] == pytest.approx(1.0)
+ assert update["verl_rl"]["actor_entropy"] > 0.0
+ advantages = [
+ json.loads(line) for line in (result.run_dir / "advantages.jsonl").read_text().splitlines()
+ ]
+ assert len(advantages) == 2
+ assert {row["name"] for row in advantages} == {algorithm}
+
+
+def test_grouped_rl_preserves_agent_tool_context_and_uses_recorded_verifier(tmp_path) -> None: # type: ignore[no-untyped-def]
+ from miniverl.config import RunConfig
+ from miniverl.schemas.trajectory import TRAJECTORY_SCHEMA_VERSION, SpanType
+ from miniverl.trainer import OPDTrainer
+ from miniverl.trajectory.io import read_trajectories
+
+ config = RunConfig.model_validate(
+ {
+ "run": {
+ "name": "agentic-grpo",
+ "mode": "rl",
+ "seed": 31,
+ "output_dir": str(tmp_path / "runs"),
+ },
+ "models": {
+ "backend": "toy",
+ "device": "cpu",
+ "student": {
+ "model_id": "toy-agent",
+ "lora": {"enabled": False},
+ "toy": {
+ "hidden_size": 16,
+ "num_layers": 1,
+ "num_heads": 2,
+ "intermediate_size": 32,
+ "max_position_embeddings": 1024,
+ },
+ },
+ },
+ "source": {"kind": "environment"},
+ "environment": {
+ "name": "calculator",
+ "train_tasks": 1,
+ "eval_tasks": 1,
+ "test_tasks": 0,
+ },
+ "rollout": {
+ "backend": "hf_reference",
+ "samples_per_prompt": 2,
+ "temperature": 1.0,
+ "max_turns": 1,
+ "max_new_tokens_per_turn": 4,
+ "max_total_tokens": 768,
+ "record_logprobs": True,
+ },
+ "selection": {"selector": "all_model_tokens"},
+ "loss": {
+ "mode": "verl_rl_policy",
+ "aggregation": "token-mean",
+ "scale_by_temperature_squared": False,
+ "chunk_size": 16,
+ },
+ "algorithm": {
+ "name": "grpo",
+ "implementation_version": ADVANTAGE_IMPLEMENTATION_VERSION,
+ },
+ "reward": {"enabled": True, "provider": "environment_verifier"},
+ "train": {
+ "cycles": 1,
+ "rollouts_per_cycle": 1,
+ "gradient_accumulation_steps": 2,
+ "learning_rate": 0.001,
+ },
+ "memory": {"strategy": "resident"},
+ "eval": {"enabled": False},
+ "report": {"enabled": False},
+ }
+ )
+
+ trainer = OPDTrainer.from_config(config, run_id="agentic-grpo")
+ try:
+ result = trainer.train()
+ finally:
+ trainer.close()
+
+ trajectories = read_trajectories(result.run_dir / "trajectories.jsonl")
+ assert len(trajectories) == 2
+ assert all(item.schema_version == TRAJECTORY_SCHEMA_VERSION for item in trajectories)
+ assert {(item.prompt_group_id, item.sample_index) for item in trajectories} == {
+ (trajectories[0].prompt_group_id, 0),
+ (trajectories[0].prompt_group_id, 1),
+ }
+ for trajectory in trajectories:
+ old = trajectory.metadata["actor_rollout_log_probs"]
+ assert len(old) == sum(trajectory.model_generated_mask)
+ for span in trajectory.spans:
+ if span.span_type in {SpanType.SYSTEM, SpanType.USER, SpanType.TOOL_RESULT}:
+ assert not any(trajectory.model_generated_mask[span.start : span.end])
+ rewards = [
+ json.loads(line)
+ for line in (result.run_dir / "rewards.jsonl").read_text(encoding="utf-8").splitlines()
+ ]
+ assert len(rewards) == 2
+ assert {row["provider"]["name"] for row in rewards} == {"environment:calculator"}
+ assert result.global_step == 1
+
+
+def test_rl_checkpoint_resume_matches_uninterrupted_policy_and_group_progress(tmp_path) -> None: # type: ignore[no-untyped-def]
+ import torch
+
+ from miniverl.config import RunConfig
+ from miniverl.trainer import OPDTrainer
+ from miniverl.trajectory.io import read_trajectories
+ from miniverl.utils.seeding import seed_everything
+
+ train = tmp_path / "resume.parquet"
+ pq.write_table(
+ pa.Table.from_pylist(
+ [
+ {
+ "prompt": f"Choose one token for row {index}.",
+ "data_source": "unit",
+ "reward_model": {"style": "exact", "ground_truth": "unused"},
+ "extra_info": {"ground_truth": "unused"},
+ }
+ for index in range(2)
+ ]
+ ),
+ train,
+ )
+ config = RunConfig.model_validate(
+ {
+ "run": {
+ "name": "rl-resume",
+ "mode": "rl",
+ "seed": 43,
+ "output_dir": str(tmp_path / "runs"),
+ },
+ "models": {
+ "backend": "toy",
+ "device": "cpu",
+ "student": {
+ "model_id": "toy-resume-actor",
+ "lora": {"enabled": False},
+ "toy": {
+ "hidden_size": 16,
+ "num_layers": 1,
+ "num_heads": 2,
+ "intermediate_size": 32,
+ "max_position_embeddings": 128,
+ },
+ },
+ },
+ "source": {
+ "kind": "verl_parquet",
+ "train_files": [str(train)],
+ "allow_plain_string_prompts": True,
+ "shuffle": False,
+ "use_task_rewards": True,
+ },
+ "rollout": {
+ "backend": "hf_cached",
+ "samples_per_prompt": 2,
+ "temperature": 1.0,
+ "max_turns": 1,
+ "max_new_tokens_per_turn": 3,
+ "max_total_tokens": 64,
+ "record_logprobs": True,
+ },
+ "selection": {"selector": "all_model_tokens"},
+ "loss": {
+ "mode": "verl_rl_policy",
+ "aggregation": "token-mean",
+ "scale_by_temperature_squared": False,
+ "chunk_size": 16,
+ },
+ "algorithm": {
+ "name": "grpo",
+ "implementation_version": ADVANTAGE_IMPLEMENTATION_VERSION,
+ },
+ "reward": {"enabled": True, "provider": "python_api"},
+ "train": {
+ "cycles": 2,
+ "rollouts_per_cycle": 1,
+ "gradient_accumulation_steps": 2,
+ "learning_rate": 0.001,
+ },
+ "memory": {"strategy": "resident"},
+ "eval": {"enabled": False},
+ "report": {"enabled": False},
+ }
+ )
+
+ seed_everything(43, deterministic=True)
+ reference = OPDTrainer.from_config(
+ config, run_id="rl-reference", reward_provider=_SampleIndexReward()
+ )
+ try:
+ reference_result = reference.train()
+ reference_state = _trainable_state(reference)
+ finally:
+ reference.close()
+
+ seed_everything(43, deterministic=True)
+ interrupted = OPDTrainer.from_config(
+ config, run_id="rl-interrupted", reward_provider=_SampleIndexReward()
+ )
+ interrupted.cycle = 0
+ interrupted._run_cycle()
+ checkpoint = interrupted.save_checkpoint(name="interrupt")
+ interrupted_root = interrupted.paths.root
+ interrupted.close()
+
+ class _DifferentReward(_SampleIndexReward):
+ identity = RewardProviderIdentity(
+ name="different",
+ version="v1",
+ config_digest="2" * 64,
+ deterministic=True,
+ )
+
+ from miniverl.errors import ConfigError
+
+ with pytest.raises(ConfigError, match="reward provider identity"):
+ OPDTrainer.from_config(
+ config,
+ resume=interrupted_root,
+ reward_provider=_DifferentReward(),
+ )
+
+ resumed = OPDTrainer.from_config(
+ config,
+ resume=interrupted_root,
+ reward_provider=_SampleIndexReward(),
+ )
+ try:
+ resumed_result = resumed.train()
+ resumed_state = _trainable_state(resumed)
+ finally:
+ resumed.close()
+
+ assert checkpoint.is_dir()
+ assert resumed_result.global_step == reference_result.global_step == 2
+ assert resumed_result.policy_version == reference_result.policy_version == 2
+ resumed_rows = read_trajectories(resumed_result.run_dir / "trajectories.jsonl")
+ reference_rows = read_trajectories(reference_result.run_dir / "trajectories.jsonl")
+ assert [(row.prompt_group_id, row.sample_index) for row in resumed_rows] == [
+ (row.prompt_group_id, row.sample_index) for row in reference_rows
+ ]
+ assert len(resumed_rows) == len(reference_rows) == 4
+ assert resumed_state.keys() == reference_state.keys()
+ for name in resumed_state:
+ assert torch.equal(resumed_state[name], reference_state[name]), name
diff --git a/tests/unit/test_agent_event_metrics.py b/tests/unit/test_agent_event_metrics.py
index f2144d3..2baf25d 100644
--- a/tests/unit/test_agent_event_metrics.py
+++ b/tests/unit/test_agent_event_metrics.py
@@ -246,3 +246,28 @@ def user_prompt(self, task: Task) -> str:
oracle_user = next(span for span in oracle.spans if span.span_type is SpanType.USER)
assert oracle_user.env_state_id == "dynamic:7"
assert oracle.metadata["initial_observation_state_id"] == "dynamic:7"
+
+
+def test_full_width_group_seed_is_bounded_before_backend_generation() -> None:
+ class _SeedCapturingBackend(_ScriptedBackend):
+ received_seed: int | None = None
+
+ def generate(self, prefix_token_ids, **kwargs) -> GenerationOutput:
+ self.received_seed = kwargs["seed"]
+ return super().generate(prefix_token_ids, **kwargs)
+
+ backend = _SeedCapturingBackend([(render_final("2"), "stop_sequence")])
+ runner = RolloutRunner(
+ backend=backend,
+ environment=CalculatorEnvironment(prompt_style="compact"),
+ config=RolloutConfig(max_turns=1, max_new_tokens_per_turn=32, max_total_tokens=1600),
+ )
+
+ runner.rollout(
+ Task(task_id="seed", prompt="Compute 1 + 1.", answer="2"),
+ policy_version=0,
+ seed=(1 << 63) - 1,
+ )
+
+ assert backend.received_seed is not None
+ assert 0 <= backend.received_seed <= (1 << 63) - 1
diff --git a/tests/unit/test_cached_sampling.py b/tests/unit/test_cached_sampling.py
index 79795ea..de8d89c 100644
--- a/tests/unit/test_cached_sampling.py
+++ b/tests/unit/test_cached_sampling.py
@@ -139,3 +139,28 @@ def decode(tokens, active, state): # type: ignore[no-untyped-def]
assert observed[0].dtype == torch.long
assert observed[0].tolist() == [1, 0]
assert [row.token_ids for row in outputs] == [[1, 0], [0, 1]]
+
+
+def test_recorded_behavior_logprob_uses_sampling_temperature() -> None:
+ """The PPO anchor is the temperature-scaled policy, as in pinned verl."""
+ from miniverl.models.sampling import run_generation
+
+ logits = torch.tensor([0.0, 2.0, -1.0])
+
+ def step(_pending, state): # type: ignore[no-untyped-def]
+ return logits, state
+
+ output = run_generation(
+ step=step,
+ prefix_token_ids=[1],
+ decode=lambda ids: "x" * len(ids),
+ eos_token_id=99,
+ max_new_tokens=1,
+ temperature=2.0,
+ top_p=0.6,
+ top_k=2,
+ generator=torch.Generator().manual_seed(4),
+ record_logprobs=True,
+ )
+ expected = torch.log_softmax(logits / 2.0, dim=-1)[output.token_ids[0]]
+ assert output.logprobs[0] == pytest.approx(float(expected))
diff --git a/tests/unit/test_documentation_voice.py b/tests/unit/test_documentation_voice.py
index 2a44d37..46f79d3 100644
--- a/tests/unit/test_documentation_voice.py
+++ b/tests/unit/test_documentation_voice.py
@@ -43,7 +43,8 @@ def test_complete_boundaries_have_canonical_destinations() -> None:
limitations = _read("docs/limitations.md")
assert "one machine" in limitations
assert "multi-GPU" in limitations
- assert "PPO/GRPO" in limitations
+ assert "PPO" in limitations
+ assert "GRPO" in limitations
assert "## Evidence scope" in limitations
assert "### Execution boundary" in limitations
diff --git a/tests/unit/test_gpu_qualification.py b/tests/unit/test_gpu_qualification.py
index b91456d..f63b3f9 100644
--- a/tests/unit/test_gpu_qualification.py
+++ b/tests/unit/test_gpu_qualification.py
@@ -300,11 +300,62 @@ def _v011_runtime_result(name: str) -> dict[str, object]:
return payload
-def _promote_v011(tmp_path: Path, *, mutate=None):
+def _v012_rl_result() -> dict[str, object]:
+ return {
+ "schema_version": 1,
+ "kind": "miniverl_v012_rl_qualification",
+ "status": "passed",
+ "source_commit": "a" * 40,
+ "miniverl_version": "0.12.0",
+ "wheel_sha256": "4" * 64,
+ "hardware": _v011_profile_result()["hardware"],
+ "upstream": {
+ "tag": "v0.9.0",
+ "commit": "483b8a009ba3a97563edee3a19887e4862b8094a",
+ "profile": "verl-rl-v0.9-single-gpu-v1",
+ "compiler_status": "accepted",
+ "generated_recipe_validated": True,
+ },
+ "workload": {
+ "model_id": "Qwen/Qwen3-0.6B",
+ "model_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
+ "algorithm": "grpo",
+ "reward_provider": "builtin_target_length",
+ "rollout_backend": "hf_cached",
+ "cycles": 2,
+ "prompts_per_cycle": 2,
+ "samples_per_prompt": 4,
+ "trajectories": 16,
+ "optimizer_updates": 2,
+ "policy_version": 2,
+ "selected_positions": 64,
+ "losses": [0.1, 0.2],
+ "reward_min": 0.1,
+ "reward_max": 1.0,
+ "max_within_prompt_reward_variance": 0.02,
+ "max_absolute_advantage": 1.2,
+ "initial_trainable_state_sha256": "1" * 64,
+ "final_trainable_state_sha256": "2" * 64,
+ },
+ "resource_contract": {
+ "peak_reserved_gib": 8.5,
+ "limit_gib": 14.5,
+ "peak_reserved_within_limit": True,
+ },
+ "scientific_scope": {
+ "runtime_correctness_only": True,
+ "task_quality_evaluated": False,
+ "distributed_execution_tested": False,
+ "other_hardware_measured": False,
+ },
+ }
+
+
+def _promote_v011(tmp_path: Path, *, mutate=None, version: str = "0.11.0"):
from scripts.promote_full_gpu_qualification import promote
payload, root = _payload(tmp_path)
- payload["miniverl_version"] = "0.11.0"
+ payload["miniverl_version"] = version
payload["environment"]["python"] = "3.12.13" # type: ignore[index]
payload["environment"]["packages"].update( # type: ignore[index,union-attr]
{
@@ -320,13 +371,20 @@ def _promote_v011(tmp_path: Path, *, mutate=None):
name: _full_result(name) for name in ("direct", "pg_k1", "smollm2")
}
for result in results.values():
- result["miniverl_version"] = "0.11.0"
+ result["miniverl_version"] = version
results["v011_profiles"] = _v011_profile_result()
results["hf_cached_runtime"] = _v011_runtime_result("hf_cached")
results["vllm_runtime"] = _v011_runtime_result("vllm")
results["v011_profiles"]["wheel_sha256"] = wheel_sha256
+ results["v011_profiles"]["miniverl_version"] = version
results["hf_cached_runtime"]["source"]["wheel_sha256"] = wheel_sha256 # type: ignore[index]
results["vllm_runtime"]["source"]["wheel_sha256"] = wheel_sha256 # type: ignore[index]
+ results["hf_cached_runtime"]["source"]["miniverl_version"] = version # type: ignore[index]
+ results["vllm_runtime"]["source"]["miniverl_version"] = version # type: ignore[index]
+ if version.startswith("0.12."):
+ results["v012_rl"] = _v012_rl_result()
+ results["v012_rl"]["miniverl_version"] = version
+ results["v012_rl"]["wheel_sha256"] = wheel_sha256
if mutate is not None:
mutate(results)
for name, result in results.items():
@@ -342,6 +400,7 @@ def _promote_v011(tmp_path: Path, *, mutate=None):
hf_cached_runtime=paths["hf_cached_runtime"],
vllm_runtime=paths["vllm_runtime"],
hf_reference=Path("benchmarks/results/rollout-runtime-v2-hf-reference.json"),
+ v012_rl=paths.get("v012_rl"),
)
@@ -363,6 +422,43 @@ def test_v011_full_qualification_binds_profile_and_runtime_evidence(tmp_path: Pa
}
+def test_v012_full_qualification_binds_nonconstant_grpo_evidence(tmp_path: Path) -> None:
+ promoted = _promote_v011(tmp_path, version="0.12.0")
+
+ assert {artifact.name for artifact in promoted.artifacts} >= {
+ "full_v012_rl_result",
+ }
+ assert set(promoted.checks.executed) >= {
+ "v012_pinned_verl_v09_compiler",
+ "v012_grpo_nonconstant_reward_update",
+ "v012_exact_wheel_rl_runtime",
+ }
+
+
+@pytest.mark.parametrize(
+ ("mutate", "message"),
+ [
+ (
+ lambda r: r["v012_rl"]["workload"].update( # type: ignore[index,union-attr]
+ max_within_prompt_reward_variance=0.0
+ ),
+ "reward variation",
+ ),
+ (
+ lambda r: r["v012_rl"]["workload"].update( # type: ignore[index,union-attr]
+ final_trainable_state_sha256="1" * 64
+ ),
+ "actor parameters",
+ ),
+ ],
+)
+def test_v012_full_qualification_rejects_empty_rl_evidence(
+ tmp_path: Path, mutate, message: str
+) -> None:
+ with pytest.raises(ValueError, match=message):
+ _promote_v011(tmp_path, version="0.12.0", mutate=mutate)
+
+
@pytest.mark.parametrize(
("mutate", "message"),
[
diff --git a/tests/unit/test_losses_bucketed.py b/tests/unit/test_losses_bucketed.py
index 04d2171..2606f4f 100644
--- a/tests/unit/test_losses_bucketed.py
+++ b/tests/unit/test_losses_bucketed.py
@@ -222,6 +222,25 @@ def test_bucketed_entropy_lower_bounds_the_exact_entropy(logits):
assert bool((coarse <= exact + 1e-5).all())
+@pytest.mark.parametrize("tail_epsilon", [1e-9, 1e-3, 9e-3])
+def test_full_vocabulary_bucketed_entropy_is_exact(tail_epsilon: float) -> None:
+ """An empty K == V tail must not become a synthetic epsilon bucket."""
+ from miniverl.losses.bucketed import bucketed_teacher_entropy, teacher_topk_targets
+ from miniverl.losses.exact import exact_teacher_entropy
+
+ torch.manual_seed(23)
+ teacher = torch.randn(20, 32) * 2
+ _, topk_log_probs, empty_tail = teacher_topk_targets(teacher, top_k=32)
+
+ actual = bucketed_teacher_entropy(
+ topk_log_probs,
+ empty_tail,
+ tail_epsilon=tail_epsilon,
+ )
+
+ torch.testing.assert_close(actual, exact_teacher_entropy(teacher), rtol=1e-6, atol=1e-7)
+
+
def test_half_precision_student_logits_are_upcast(logits):
from miniverl.losses.bucketed import bucketed_divergence, teacher_topk_targets
diff --git a/tests/unit/test_packaging.py b/tests/unit/test_packaging.py
index 3dea44a..e86bbbe 100644
--- a/tests/unit/test_packaging.py
+++ b/tests/unit/test_packaging.py
@@ -33,6 +33,7 @@
#: Every subpackage that must exist in an installed distribution.
REQUIRED_SUBPACKAGES = (
"agent",
+ "algorithms",
"alignment",
"alignment_external",
"bridge",
@@ -67,6 +68,8 @@
"miniverl.bridge.doctor",
"miniverl.bridge.export",
"miniverl.bridge.profiles",
+ "miniverl.bridge.rl_v09",
+ "miniverl.algorithms.contract",
"miniverl.commands",
"miniverl.commands.compat",
"miniverl.commands.qualification",
@@ -420,8 +423,8 @@ def test_single_gpu_visual_identity_and_pypi_link_are_prominent() -> None:
assert "1× CUDA GPU" in banner
assert "BF16 / FP16 auto" in banner
assert "typed provenance" in banner
- assert "Run verl-style OPD on one consumer GPU" in banner
- assert "actor rollout · teacher score · actor update" in banner
+ assert "Run verl experiment semantics on one consumer GPU" in banner
+ assert "actor rollout · reward / ref / teacher · update" in banner
assert "PEFT · Parquet · targets · metrics · provenance" in banner
assert 'pip install "miniverl[train,cuda]"' not in banner
assert "the GPU you have" not in banner
diff --git a/tests/unit/test_release_assets.py b/tests/unit/test_release_assets.py
index 7a4e071..e2d2113 100644
--- a/tests/unit/test_release_assets.py
+++ b/tests/unit/test_release_assets.py
@@ -17,14 +17,13 @@ def _sha(path: Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()
-def _fixture(tmp_path: Path) -> tuple[Path, Path, Path, Path]:
+def _fixture(tmp_path: Path, *, version: str = "0.11.0.dev0") -> tuple[Path, Path, Path, Path]:
from miniverl.release_candidate import PINNED_BUILD_TOOLS
candidate = tmp_path / "candidate"
qualification = tmp_path / "qualification"
candidate.mkdir(parents=True)
qualification.mkdir(parents=True)
- version = "0.11.0.dev0"
commit = "a" * 40
wheel = candidate / f"miniverl-{version}-py3-none-any.whl"
sdist = candidate / f"miniverl-{version}.tar.gz"
@@ -84,6 +83,11 @@ def _fixture(tmp_path: Path) -> tuple[Path, Path, Path, Path]:
b'{"kind":"vllm-runtime"}\n',
),
}
+ if version.startswith("0.12."):
+ evidence["full_v012_rl_result"] = (
+ "full/v012-rl.json",
+ b'{"kind":"v012-rl"}\n',
+ )
artifacts = []
for name, (relative, content) in evidence.items():
path = qualification / relative
@@ -195,6 +199,14 @@ def _fixture(tmp_path: Path) -> tuple[Path, Path, Path, Path]:
"other_hardware_measured": False,
},
}
+ if version.startswith("0.12."):
+ payload["checks"]["executed"].extend(
+ [
+ "v012_pinned_verl_v09_compiler",
+ "v012_grpo_nonconstant_reward_update",
+ "v012_exact_wheel_rl_runtime",
+ ]
+ )
qualification_path = qualification / "qualification.json"
qualification_path.write_text(json.dumps(payload), encoding="utf-8")
verification = tmp_path / "verification.json"
@@ -217,10 +229,10 @@ def _fixture(tmp_path: Path) -> tuple[Path, Path, Path, Path]:
return candidate, manifest_path, qualification, verification
-def _build(tmp_path: Path, *, name: str = "out") -> Path:
+def _build(tmp_path: Path, *, name: str = "out", version: str = "0.11.0.dev0") -> Path:
from miniverl.release_assets import prepare_release_assets
- candidate, manifest, qualification, verification = _fixture(tmp_path)
+ candidate, manifest, qualification, verification = _fixture(tmp_path, version=version)
output = tmp_path / name
prepare_release_assets(
candidate_dir=candidate,
@@ -269,12 +281,24 @@ def test_release_evidence_mapping_is_versioned_for_historical_records() -> None:
historical = _archive_evidence_for_version("0.10.1")
current = _archive_evidence_for_version("0.11.0")
+ v012 = _archive_evidence_for_version("0.12.0")
assert "full_v011_profiles_result" not in historical
assert set(current) - set(historical) == {
"full_v011_profiles_result",
"full_hf_cached_runtime_result",
"full_vllm_runtime_result",
}
+ assert set(v012) - set(current) == {"full_v012_rl_result"}
+
+
+def test_v012_release_archive_includes_rl_qualification(tmp_path: Path) -> None:
+ from miniverl.release_assets import check_release_assets
+
+ output = _build(tmp_path, version="0.12.0")
+ manifest = json.loads((output / "qualification-evidence-manifest.json").read_text())
+
+ assert "v012_rl" in {member["semantic_role"] for member in manifest["members"]}
+ assert check_release_assets(output) == []
def test_archive_and_manifest_are_reproducible_and_explicit(tmp_path: Path) -> None:
diff --git a/tests/unit/test_rewards.py b/tests/unit/test_rewards.py
index bc7fbad..196d6ae 100644
--- a/tests/unit/test_rewards.py
+++ b/tests/unit/test_rewards.py
@@ -4,14 +4,20 @@
import pytest
+from miniverl.errors import ConfigError
from miniverl.rewards import (
ADVANTAGE_COMPOSER_VERSION,
AdvantageComposer,
AdvantageMode,
EnvironmentVerifierRewardProvider,
ExactAnswerRewardProvider,
+ RewardProviderIdentity,
RewardRequest,
+ RewardResult,
RewardStatus,
+ TargetLengthRewardProvider,
+ WeightedRewardProvider,
+ score_reward_requests,
)
@@ -66,6 +72,27 @@ def test_exact_answer_provider_returns_error_not_zero_for_invalid_metadata() ->
assert result.failure_category == "invalid_reward_metadata"
+def test_target_length_provider_scores_declared_character_distance() -> None:
+ provider = TargetLengthRewardProvider()
+ exact = provider.score(
+ _request(response="four", reward_model={"style": "target_length", "characters": 4})
+ )
+ near = provider.score(
+ _request(response="three", reward_model={"style": "target_length", "characters": 4})
+ )
+ invalid = provider.score(
+ _request(response="four", reward_model={"style": "target_length", "characters": 0})
+ )
+
+ assert exact.status is RewardStatus.OK
+ assert exact.raw_reward == 1.0
+ assert exact.components[0].name == "target_length"
+ assert near.raw_reward == pytest.approx(0.75)
+ assert invalid.status is RewardStatus.ERROR
+ assert invalid.raw_reward is None
+ assert invalid.failure_category == "invalid_reward_metadata"
+
+
def test_environment_verifier_adapter_preserves_components() -> None:
from miniverl.environments.base import FailureCategory, VerificationResult
@@ -94,6 +121,30 @@ def verify(self, answer: str) -> VerificationResult:
assert result.provider.name == "environment:unit_env"
+@pytest.mark.parametrize("reward", [float("nan"), float("inf"), float("-inf")])
+def test_recorded_environment_verifier_rejects_non_finite_rewards(reward: float) -> None:
+ class Verifier:
+ name = "unit_env"
+ verifier_version = "v2"
+
+ def __init__(self) -> None:
+ self.params: dict[str, object] = {}
+
+ request = _request().model_copy(
+ update={
+ "reward_model": {
+ "style": "recorded_environment_verifier",
+ "reward": reward,
+ }
+ }
+ )
+ result = EnvironmentVerifierRewardProvider(Verifier()).score(request)
+
+ assert result.status is RewardStatus.ERROR
+ assert result.raw_reward is None
+ assert result.failure_category == "invalid_recorded_verification"
+
+
@pytest.mark.parametrize(
("mode", "expected"),
[
@@ -146,3 +197,58 @@ def test_composer_expands_task_scalar_only_to_selected_assistant_tokens() -> Non
expanded = composer.expand_task_advantage(2.0, [False, True, True, False])
assert expanded == [0.0, 2.0, 2.0, 0.0]
+
+
+def test_weighted_reward_preserves_named_components_and_identity() -> None:
+ provider = WeightedRewardProvider(
+ [
+ ("correctness", 2.0, ExactAnswerRewardProvider()),
+ ("format", -0.25, ExactAnswerRewardProvider(strip_whitespace=False)),
+ ]
+ )
+
+ result = provider.score(_request())
+
+ assert result.status is RewardStatus.OK
+ assert result.raw_reward == pytest.approx(1.75)
+ assert [(item.name, item.value) for item in result.components] == [
+ ("correctness", 2.0),
+ ("format", -0.25),
+ ]
+ assert provider.identity.name == "weighted_composite"
+ assert len(provider.identity.digest) == 64
+
+
+def test_batch_reward_results_must_match_request_order_and_provider() -> None:
+ request = _request()
+
+ class MisboundProvider:
+ identity = RewardProviderIdentity(
+ name="misbound",
+ version="v1",
+ config_digest="b" * 64,
+ deterministic=True,
+ )
+
+ def score_batch(self, requests): # type: ignore[no-untyped-def]
+ return [
+ RewardResult(
+ trajectory_id="another-trajectory",
+ prompt_group_id=item.prompt_group_id,
+ sample_index=item.sample_index,
+ samples_per_prompt=item.samples_per_prompt,
+ provider=self.identity,
+ input_digest=item.input_digest,
+ raw_reward=1.0,
+ status=RewardStatus.OK,
+ duration_ms=0.0,
+ deterministic=True,
+ )
+ for item in requests
+ ]
+
+ def score(self, request): # type: ignore[no-untyped-def]
+ raise AssertionError("batch path expected")
+
+ with pytest.raises(ConfigError, match="identity does not match"):
+ score_reward_requests(MisboundProvider(), [request])
diff --git a/tests/unit/test_rl_algorithms.py b/tests/unit/test_rl_algorithms.py
new file mode 100644
index 0000000..1bc4ac4
--- /dev/null
+++ b/tests/unit/test_rl_algorithms.py
@@ -0,0 +1,140 @@
+from __future__ import annotations
+
+import pytest
+
+torch = pytest.importorskip("torch")
+
+pytestmark = pytest.mark.torch
+
+
+def _fixture(): # type: ignore[no-untyped-def]
+ rewards = torch.tensor([[0.0, 1.0, 0.0], [0.0, 3.0, 0.0], [2.0, 0.0, 0.0], [4.0, 0.0, 0.0]])
+ mask = torch.tensor([[1.0, 1.0, 0.0], [1.0, 1.0, 1.0], [1.0, 0.0, 0.0], [1.0, 1.0, 0.0]])
+ groups = ("a", "a", "b", "b")
+ return rewards, mask, groups
+
+
+def test_grpo_uses_sample_std_and_dr_grpo_does_not_scale() -> None:
+ from miniverl.algorithms.advantages import grpo_outcome_advantage
+
+ rewards, mask, groups = _fixture()
+ grpo = grpo_outcome_advantage(rewards, mask, groups, normalize_by_std=True)
+ dr_grpo = grpo_outcome_advantage(rewards, mask, groups, normalize_by_std=False)
+
+ # torch.std([1, 3]) is sqrt(2): upstream verl uses the sample std.
+ expected_grpo = torch.tensor([-1 / 2**0.5, 1 / 2**0.5, -1 / 2**0.5, 1 / 2**0.5])
+ expected_dr = torch.tensor([-1.0, 1.0, -1.0, 1.0])
+ torch.testing.assert_close(grpo.sequence_advantages, expected_grpo, atol=1e-6, rtol=1e-6)
+ torch.testing.assert_close(dr_grpo.sequence_advantages, expected_dr)
+ torch.testing.assert_close(grpo.advantages, expected_grpo[:, None] * mask)
+ assert grpo.zero_variance_groups == 0
+
+
+def test_grpo_zero_variance_group_is_finite_and_zero() -> None:
+ from miniverl.algorithms.advantages import grpo_outcome_advantage
+
+ rewards = torch.tensor([[1.0], [1.0]])
+ mask = torch.ones_like(rewards)
+ output = grpo_outcome_advantage(rewards, mask, ("same", "same"))
+
+ assert torch.isfinite(output.advantages).all()
+ torch.testing.assert_close(output.advantages, torch.zeros_like(output.advantages))
+ assert output.zero_variance_groups == 1
+
+
+def test_rloo_is_leave_one_out_per_prompt() -> None:
+ from miniverl.algorithms.advantages import rloo_outcome_advantage
+
+ rewards, mask, groups = _fixture()
+ output = rloo_outcome_advantage(rewards, mask, groups)
+
+ torch.testing.assert_close(output.sequence_advantages, torch.tensor([-2.0, 2.0, -2.0, 2.0]))
+ torch.testing.assert_close(output.advantages, output.sequence_advantages[:, None] * mask)
+
+
+def test_group_estimators_reject_partial_or_singleton_groups() -> None:
+ from miniverl.algorithms.advantages import grpo_outcome_advantage, rloo_outcome_advantage
+
+ rewards = torch.ones((3, 2))
+ mask = torch.ones_like(rewards)
+ groups = ("a", "a", "b")
+ with pytest.raises(ValueError, match="at least two complete samples"):
+ grpo_outcome_advantage(rewards, mask, groups)
+ with pytest.raises(ValueError, match="at least two complete samples"):
+ rloo_outcome_advantage(rewards, mask, groups)
+
+
+def test_reinforce_plus_plus_carries_return_across_masked_observation() -> None:
+ from miniverl.algorithms.advantages import reinforce_plus_plus_advantage
+
+ rewards = torch.tensor([[0.0, 5.0, 0.0, 2.0]])
+ mask = torch.tensor([[1.0, 0.0, 1.0, 1.0]])
+ output = reinforce_plus_plus_advantage(rewards, mask, gamma=0.5)
+
+ # The masked observation does not terminate return propagation.
+ torch.testing.assert_close(output.returns, torch.tensor([[0.5, 0.0, 1.0, 2.0]]))
+ assert output.advantages[0, 1] == 0
+ assert torch.isfinite(output.advantages).all()
+
+
+def test_gae_masks_observations_and_returns_are_unwhitened() -> None:
+ from miniverl.algorithms.advantages import gae_advantage_return
+
+ rewards = torch.tensor([[1.0, 99.0, 2.0]])
+ values = torch.tensor([[0.4, 0.7, 0.5]])
+ mask = torch.tensor([[1.0, 0.0, 1.0]])
+ output = gae_advantage_return(rewards, values, mask, gamma=0.9, lam=0.8)
+
+ # Upstream leaves masked positions populated after whitening; downstream
+ # policy/value reductions apply the mask.
+ assert output.advantages.shape == rewards.shape
+ assert output.returns.shape == rewards.shape
+ assert torch.isfinite(output.advantages).all()
+
+
+def test_policy_loss_matches_clipped_dual_clip_equations_and_gradient() -> None:
+ from miniverl.algorithms.policy import clipped_policy_loss
+
+ old = torch.tensor([-1.2, -0.7, -2.0])
+ current = torch.tensor([-0.8, -1.1, -2.0], requires_grad=True)
+ advantages = torch.tensor([1.0, -2.0, 0.5])
+ mask = torch.tensor([1.0, 1.0, 0.0])
+ output = clipped_policy_loss(
+ current_log_probs=current,
+ old_log_probs=old,
+ advantages=advantages,
+ response_mask=mask,
+ )
+ output.loss.backward()
+
+ assert torch.isfinite(output.loss)
+ assert current.grad is not None
+ assert current.grad[2] == 0
+ assert 0.0 <= output.metrics["pg_clipfrac"] <= 1.0
+
+
+def test_value_loss_clips_around_old_values_and_honors_mask() -> None:
+ from miniverl.algorithms.value import clipped_value_loss
+
+ current = torch.tensor([0.8, -0.1, 4.0], requires_grad=True)
+ old = torch.tensor([0.0, 0.0, 0.0])
+ returns = torch.tensor([1.0, -1.0, 9.0])
+ mask = torch.tensor([1.0, 1.0, 0.0])
+ output = clipped_value_loss(current, old, returns, mask, cliprange_value=0.2)
+ output.loss.backward()
+
+ assert current.grad is not None
+ assert current.grad[2] == 0
+ assert output.metrics["value_clipfrac"] == pytest.approx(0.5)
+
+
+@pytest.mark.parametrize("penalty", ["kl", "abs", "mse", "low_var_kl"])
+def test_reference_kl_estimators_are_finite_and_shape_preserving(penalty: str) -> None:
+ from miniverl.algorithms.kl import reference_kl_penalty
+
+ old = torch.tensor([[-1.0, -0.5], [-4.0, -2.0]])
+ reference = torch.tensor([[-1.2, -0.2], [-3.0, -2.5]])
+ result = reference_kl_penalty(old, reference, penalty=penalty)
+
+ assert result.shape == old.shape
+ assert torch.isfinite(result).all()
diff --git a/tests/unit/test_rl_config.py b/tests/unit/test_rl_config.py
new file mode 100644
index 0000000..801fab6
--- /dev/null
+++ b/tests/unit/test_rl_config.py
@@ -0,0 +1,126 @@
+from __future__ import annotations
+
+from copy import deepcopy
+
+import pytest
+from pydantic import ValidationError
+
+from miniverl.algorithms.contract import ADVANTAGE_IMPLEMENTATION_VERSION
+from miniverl.config import RunConfig
+
+
+def rl_config() -> dict: # type: ignore[type-arg]
+ return {
+ "run": {"name": "local-grpo", "mode": "rl", "seed": 7},
+ "models": {
+ "backend": "toy",
+ "device": "cpu",
+ "student": {"model_id": "student", "lora": {"enabled": False}},
+ },
+ "source": {
+ "kind": "verl_parquet",
+ "train_files": ["train.parquet"],
+ "use_task_rewards": True,
+ },
+ "rollout": {
+ "backend": "hf_cached",
+ "samples_per_prompt": 2,
+ "temperature": 1.0,
+ "record_logprobs": True,
+ },
+ "selection": {"selector": "all_model_tokens"},
+ "loss": {
+ "mode": "verl_rl_policy",
+ "aggregation": "token-mean",
+ "scale_by_temperature_squared": False,
+ },
+ "algorithm": {
+ "name": "grpo",
+ "implementation_version": ADVANTAGE_IMPLEMENTATION_VERSION,
+ },
+ "reward": {"enabled": True, "provider": "exact_answer"},
+ "train": {"cycles": 1, "rollouts_per_cycle": 1},
+ "eval": {"enabled": False},
+ "report": {"enabled": False},
+ }
+
+
+def test_teacher_free_grpo_config_round_trips() -> None:
+ config = RunConfig.model_validate(rl_config())
+ assert config.models.teacher is None
+ assert config.algorithm.name.value == "grpo"
+ assert RunConfig.model_validate(config.model_dump()).algorithm == config.algorithm
+
+
+def test_parquet_rl_accepts_the_builtin_target_length_reward() -> None:
+ payload = rl_config()
+ payload["reward"]["provider"] = "target_length"
+
+ assert RunConfig.model_validate(payload).reward.provider.value == "target_length"
+
+
+@pytest.mark.parametrize("algorithm", ["grpo", "dr_grpo", "rloo"])
+def test_group_estimators_require_real_grouped_rollouts(algorithm: str) -> None:
+ payload = rl_config()
+ payload["algorithm"]["name"] = algorithm
+ payload["rollout"]["samples_per_prompt"] = 1
+ with pytest.raises(ValidationError, match="requires samples_per_prompt > 1"):
+ RunConfig.model_validate(payload)
+
+
+def test_rl_rejects_silent_teacher_and_missing_reward() -> None:
+ payload = rl_config()
+ payload["models"]["teacher"] = {"model_id": "unused"}
+ with pytest.raises(ValidationError, match="teacher-free"):
+ RunConfig.model_validate(payload)
+
+ payload = deepcopy(rl_config())
+ payload["reward"]["enabled"] = False
+ with pytest.raises(ValidationError, match="requires task rewards"):
+ RunConfig.model_validate(payload)
+
+
+def test_ppo_fails_closed_until_critic_lifecycle_is_complete() -> None:
+ payload = rl_config()
+ payload["algorithm"]["name"] = "ppo"
+ with pytest.raises(ValidationError, match="critic checkpoint/runtime contract"):
+ RunConfig.model_validate(payload)
+
+
+@pytest.mark.parametrize(
+ ("field", "value", "match"),
+ [
+ ("kl_coef", 0.1, "requires models.reference"),
+ ("value_loss_coef", 0.7, "not-yet-executable PPO critic path"),
+ ("cliprange_value", 0.2, "not-yet-executable PPO critic path"),
+ ],
+)
+def test_rl_rejects_unwired_algorithm_controls(field: str, value: float, match: str) -> None:
+ payload = rl_config()
+ payload["algorithm"][field] = value
+ with pytest.raises(ValidationError, match=match):
+ RunConfig.model_validate(payload)
+
+
+def test_rl_reference_kl_requires_an_explicit_shared_reference_role() -> None:
+ payload = rl_config()
+ payload["models"] = {
+ "backend": "hf",
+ "runtime": "shared_backbone",
+ "device": "cpu",
+ "student": {
+ "model_id": "org/base",
+ "lora": {"enabled": True},
+ },
+ "reference": {
+ "model_id": "org/base",
+ "adapter": {"path": "reference-adapter", "source": "local"},
+ },
+ }
+ payload["algorithm"]["kl_coef"] = 0.01
+
+ config = RunConfig.model_validate(payload)
+
+ assert config.models.teacher is None
+ assert config.models.reference is not None
+ assert config.algorithm.kl_penalty == "kl"
diff --git a/tests/unit/test_v012_rl_qualification.py b/tests/unit/test_v012_rl_qualification.py
new file mode 100644
index 0000000..dfc6ff0
--- /dev/null
+++ b/tests/unit/test_v012_rl_qualification.py
@@ -0,0 +1,34 @@
+from __future__ import annotations
+
+from pathlib import Path
+
+import yaml
+
+
+def test_v012_gpu_workload_source_compiles_to_validated_grpo_recipe(tmp_path: Path) -> None:
+ from miniverl.algorithms.contract import UPSTREAM_VERL_COMMIT
+ from miniverl.bridge.rl_v09 import publish_imported_verl_rl_v09
+ from miniverl.config import RunConfig
+ from scripts.run_v012_rl_qualification import _source_payload
+
+ source = tmp_path / "source.yaml"
+ source.write_text(
+ yaml.safe_dump(_source_payload(tmp_path / "data.parquet"), sort_keys=False),
+ encoding="utf-8",
+ )
+ recipe = tmp_path / "native.yaml"
+
+ report = publish_imported_verl_rl_v09(
+ source,
+ out=recipe,
+ target_verl=UPSTREAM_VERL_COMMIT,
+ )
+ config = RunConfig.from_yaml(recipe)
+
+ assert report["status"] == "accepted"
+ assert report["generated_recipe_validated"] is True
+ assert config.algorithm.name.value == "grpo"
+ assert config.reward.provider.value == "target_length"
+ assert config.rollout.samples_per_prompt == 4
+ assert config.train.cycles == 2
+ assert config.train.gradient_accumulation_steps == 8
diff --git a/tests/unit/test_verl_rl_v09_bridge.py b/tests/unit/test_verl_rl_v09_bridge.py
new file mode 100644
index 0000000..7c73521
--- /dev/null
+++ b/tests/unit/test_verl_rl_v09_bridge.py
@@ -0,0 +1,259 @@
+from __future__ import annotations
+
+import json
+from copy import deepcopy
+from pathlib import Path
+
+import pytest
+import yaml
+from pydantic import ValidationError
+
+from miniverl.algorithms.contract import UPSTREAM_VERL_COMMIT
+from miniverl.bridge.rl_v09 import VERL_RL_V09_PROFILE, publish_imported_verl_rl_v09
+from miniverl.config import RunConfig
+from miniverl.errors import ConfigError
+
+
+def source_payload() -> dict: # type: ignore[type-arg]
+ return {
+ "data": {
+ "train_files": ["train.parquet"],
+ "val_files": ["val.parquet"],
+ "prompt_key": "prompt",
+ "train_batch_size": 2,
+ "max_prompt_length": 128,
+ "max_response_length": 32,
+ "truncation": "error",
+ "shuffle": True,
+ "seed": 17,
+ },
+ "actor_rollout_ref": {
+ "model": {
+ "path": "Qwen/Qwen3-0.6B",
+ "enable_gradient_checkpointing": True,
+ "lora_rank": 8,
+ "lora_alpha": 16,
+ "target_modules": ["q_proj", "v_proj"],
+ },
+ "actor": {
+ "optim": {"lr": "1e-5", "weight_decay": 0.01, "lr_warmup_steps": 1},
+ "ppo_mini_batch_size": 8,
+ "ppo_epochs": 1,
+ "loss_agg_mode": "token-mean",
+ "clip_ratio": 0.2,
+ "clip_ratio_low": 0.2,
+ "clip_ratio_high": 0.2,
+ "clip_ratio_c": 3.0,
+ "use_kl_loss": False,
+ "entropy_coeff": 0.0,
+ },
+ "rollout": {
+ "name": "vllm",
+ "n": 4,
+ "temperature": 1.0,
+ "top_p": 1.0,
+ "top_k": 0,
+ "tensor_model_parallel_size": 8,
+ "gpu_memory_utilization": 0.6,
+ },
+ },
+ "algorithm": {
+ "adv_estimator": "grpo",
+ "norm_adv_by_std_in_grpo": True,
+ "gamma": 1.0,
+ "lam": 1.0,
+ "use_kl_in_reward": False,
+ },
+ "trainer": {
+ "total_training_steps": 3,
+ "total_epochs": 30,
+ "project_name": "verl-home",
+ "experiment_name": "local-grpo",
+ "save_freq": -1,
+ "test_freq": -1,
+ "logger": ["console"],
+ "n_gpus_per_node": 8,
+ "nnodes": 2,
+ },
+ "miniverl": {
+ "reward": {"provider": "exact_answer"},
+ "actor": {"lora_enabled": True, "dtype": "auto", "quantization": "none"},
+ "rollout": {"backend": "hf_cached"},
+ "memory": {"strategy": "auto"},
+ "update": {"trajectory_batch_size": 1},
+ },
+ }
+
+
+def _write(tmp_path: Path, payload: dict) -> Path: # type: ignore[type-arg]
+ path = tmp_path / "verl.yaml"
+ path.write_text(yaml.safe_dump(payload, sort_keys=False), encoding="utf-8")
+ return path
+
+
+def test_v09_grpo_import_preserves_semantics_and_lowers_placement(tmp_path: Path) -> None:
+ out = tmp_path / "local.yaml"
+ report = publish_imported_verl_rl_v09(
+ _write(tmp_path, source_payload()),
+ out=out,
+ target_verl=UPSTREAM_VERL_COMMIT,
+ )
+ config = RunConfig.from_yaml(out)
+
+ assert report["status"] == "accepted"
+ assert report["profile"] == VERL_RL_V09_PROFILE
+ assert config.algorithm.name.value == "grpo"
+ assert config.source.train_files == ["train.parquet"]
+ assert config.resolved_for_runtime().source.train_files == [
+ str((tmp_path / "train.parquet").resolve())
+ ]
+ assert config.rollout.samples_per_prompt == 4
+ assert config.train.cycles == 3
+ assert config.train.rollouts_per_cycle == 2
+ assert config.train.gradient_accumulation_steps == 8
+ assert config.train.opd_freshness.value == "strict"
+ assert config.train.learning_rate == pytest.approx(1e-5)
+ distributed = [
+ row for row in report["field_classification"] if row["classification"] == "distributed_only"
+ ]
+ assert {row["upstream_field"] for row in distributed} == {
+ "actor_rollout_ref.rollout.tensor_model_parallel_size",
+ "trainer.n_gpus_per_node",
+ "trainer.nnodes",
+ }
+ assert report["local_lowering"][2]["local_value"] == {"devices": 1, "processes": 1}
+ assert not [
+ row
+ for row in report["field_classification"]
+ if row["classification"] in {"not_implemented", "unsupported"}
+ ]
+ lam = next(
+ row for row in report["field_classification"] if row["upstream_field"] == "algorithm.lam"
+ )
+ assert lam["classification"] == "informational_only"
+
+
+def test_v09_grpo_false_std_maps_to_dr_grpo(tmp_path: Path) -> None:
+ payload = source_payload()
+ payload["algorithm"]["norm_adv_by_std_in_grpo"] = False
+ out = tmp_path / "local.yaml"
+ publish_imported_verl_rl_v09(_write(tmp_path, payload), out=out, target_verl="v0.9.0")
+ assert RunConfig.from_yaml(out).algorithm.name.value == "dr_grpo"
+
+
+def test_v09_import_accepts_explicit_builtin_target_length_reward(tmp_path: Path) -> None:
+ payload = source_payload()
+ payload["miniverl"]["reward"]["provider"] = "target_length"
+ out = tmp_path / "local.yaml"
+
+ report = publish_imported_verl_rl_v09(_write(tmp_path, payload), out=out, target_verl="v0.9.0")
+
+ assert report["status"] == "accepted"
+ assert RunConfig.from_yaml(out).reward.provider.value == "target_length"
+
+
+def test_v09_actor_minibatch_preserves_multiple_updates_as_explicit_replay(
+ tmp_path: Path,
+) -> None:
+ payload = source_payload()
+ payload["actor_rollout_ref"]["actor"]["ppo_mini_batch_size"] = 4
+ out = tmp_path / "local.yaml"
+
+ publish_imported_verl_rl_v09(_write(tmp_path, payload), out=out, target_verl="v0.9.0")
+ config = RunConfig.from_yaml(out)
+
+ assert config.train.gradient_accumulation_steps == 4
+ assert config.train.opd_freshness.value == "replay"
+ assert config.is_on_policy is False
+
+
+def test_v09_fixed_reference_kl_lowers_to_shared_temporal_role(tmp_path: Path) -> None:
+ payload = source_payload()
+ payload["algorithm"].update(
+ {
+ "use_kl_in_reward": True,
+ "kl_penalty": "low_var_kl",
+ "kl_ctrl": {"type": "fixed", "kl_coef": 0.02},
+ }
+ )
+ payload["miniverl"]["reference"] = {
+ "adapter_path": "reference-adapter",
+ "adapter_source": "local",
+ }
+ out = tmp_path / "local.yaml"
+
+ report = publish_imported_verl_rl_v09(_write(tmp_path, payload), out=out, target_verl="v0.9.0")
+ config = RunConfig.from_yaml(out)
+
+ assert report["status"] == "accepted"
+ assert config.models.runtime.value == "shared_backbone"
+ assert config.models.reference is not None
+ assert config.models.reference.adapter.path == "reference-adapter"
+ assert config.resolved_for_runtime().models.reference.adapter.path == str(
+ (tmp_path / "reference-adapter").resolve()
+ )
+ assert config.algorithm.kl_coef == pytest.approx(0.02)
+ assert config.algorithm.kl_penalty == "low_var_kl"
+
+
+def test_v09_import_emits_non_executable_template_when_choices_are_missing(tmp_path: Path) -> None:
+ payload = source_payload()
+ del payload["trainer"]["total_training_steps"]
+ del payload["miniverl"]["reward"]
+ del payload["miniverl"]["actor"]["lora_enabled"]
+ out = tmp_path / "local.yaml"
+ report = publish_imported_verl_rl_v09(_write(tmp_path, payload), out=out, target_verl="v0.9.0")
+
+ assert report["status"] == "needs_user_input"
+ assert not out.exists()
+ template = tmp_path / "local.template.yaml"
+ assert template.exists()
+ with pytest.raises(ValidationError):
+ RunConfig.from_yaml(template)
+ assert {row["field"] for row in report["required_user_input"]} == {
+ "trainer.total_training_steps",
+ "miniverl.reward.provider",
+ "miniverl.actor.lora_enabled",
+ }
+
+
+@pytest.mark.parametrize(
+ ("mutate", "match"),
+ [
+ (("algorithm", "adv_estimator", "gae"), "critic lifecycle"),
+ (("actor_rollout_ref.actor", "use_kl_loss", True), "not implemented"),
+ (("actor_rollout_ref.actor", "ppo_epochs", 2), "must be 1"),
+ ],
+)
+def test_v09_import_fails_closed_for_unimplemented_semantics(
+ tmp_path: Path, mutate: tuple[str, str, object], match: str
+) -> None:
+ payload = source_payload()
+ parent, key, value = mutate
+ current = payload
+ for part in parent.split("."):
+ current = current[part]
+ current[key] = value
+ with pytest.raises(ConfigError, match=match):
+ publish_imported_verl_rl_v09(
+ _write(tmp_path, payload), out=tmp_path / "local.yaml", target_verl="v0.9.0"
+ )
+
+
+def test_v09_import_rejects_unknown_fields_and_non_finite_numbers(tmp_path: Path) -> None:
+ payload = source_payload()
+ payload["trainer"]["mystery"] = 1
+ out = tmp_path / "unknown.yaml"
+ report = publish_imported_verl_rl_v09(_write(tmp_path, payload), out=out, target_verl="v0.9.0")
+ assert report["status"] == "rejected"
+ assert not out.exists()
+ assert "trainer.mystery" in report["unsupported_fields"]
+ on_disk = json.loads((tmp_path / "unknown.import-report.json").read_text(encoding="utf-8"))
+ assert on_disk == report
+
+ payload = deepcopy(source_payload())
+ payload["actor_rollout_ref"]["actor"]["optim"]["lr"] = ".nan"
+ with pytest.raises(ConfigError, match="must be finite"):
+ publish_imported_verl_rl_v09(
+ _write(tmp_path, payload), out=tmp_path / "nan.yaml", target_verl="v0.9.0"
+ )
diff --git a/tests/unit/test_verl_rl_v09_publish.py b/tests/unit/test_verl_rl_v09_publish.py
new file mode 100644
index 0000000..f8a3c0f
--- /dev/null
+++ b/tests/unit/test_verl_rl_v09_publish.py
@@ -0,0 +1,36 @@
+from __future__ import annotations
+
+import importlib.util
+import json
+from pathlib import Path
+from typing import Any
+
+
+def _publisher() -> Any:
+ path = Path("scripts/publish_verl_rl_v09_compatibility.py")
+ spec = importlib.util.spec_from_file_location("publish_verl_rl_v09_compatibility", path)
+ assert spec is not None and spec.loader is not None
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+def test_verl_v09_compatibility_report_is_exactly_generated() -> None:
+ publisher = _publisher()
+ committed = Path("docs/generated/verl-rl-v0.9-compatibility.json").read_text(encoding="utf-8")
+ assert committed == publisher.render()
+ payload = json.loads(committed)
+ assert payload["status"] == "accepted"
+ assert payload["executable"] is True
+ assert payload["algorithm"] == "grpo"
+ assert payload["source_verl"] == {
+ "repository": "https://github.com/verl-project/verl",
+ "tag": "v0.9.0",
+ "commit": "483b8a009ba3a97563edee3a19887e4862b8094a",
+ }
+ assert not payload["unsupported_fields"]
+ assert not payload["unresolved_fields"]
+ assert not payload["required_user_input"]
+ classifications = {row["classification"] for row in payload["field_classification"]}
+ assert "distributed_only" in classifications
+ assert "locally_lowered" in classifications
+ 
+ 
+