8000 Add transactional grouped rollout semantics by DaoyuanLi2816 · Pull Request #93 · DaoyuanLi2816/mini-verl · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,17 @@ All notable changes to miniVERL are recorded here. The format follows
- Bound prompt-generation manifests and every v2 request to the live actor
parameter version, base/tokenizer/adapter identity, backend and execution plan.

### Transactional grouped rollouts

- Added trajectory schema v3 with typed prompt-group, sample, seed, backend and
rollout-policy identity; v1 and v2 artifacts remain readable.
- Added two conformance-only grouped profiles for Parquet prompt sources. They
train `n>1` current-policy samples independently through the existing direct
GKD or sampled-k1 objective, without a group baseline or GRPO semantics.
- Made complete group publication journaled and replay-idempotent, and bound
prompt/group cursors, trajectory counts, backend synchronization, cache
identity and export provenance to exact interruption/resume state.

### Documentation narrative

- Rebuilt the English, Chinese and PyPI landing narratives around the local
Expand Down
4 changes: 3 additions & 1 deletion PYPI.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,9 @@ profiles are available:

Use `miniverl profiles show`, `compat explain` and `compat check` to inspect
the complete mapping. [Compatibility profiles](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/profiles/index.md) explains
how profile identity follows plans, caches, checkpoints and exports.
how profile identity follows plans, caches, checkpoints and exports. Separate
conformance-only grouped profiles add transactional Parquet `n>1` samples
without changing either measured `n=1` profile or introducing GRPO semantics.

## Measured systems evidence

Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,9 @@ profiles are available:

Use `miniverl profiles show`, `compat explain` and `compat check` to inspect
the complete mapping. [Compatibility profiles](docs/profiles/index.md) explains
how profile identity follows plans, caches, checkpoints and exports.
how profile identity follows plans, caches, checkpoints and exports. Separate
conformance-only grouped profiles add transactional Parquet `n>1` samples
without changing either measured `n=1` profile or introducing GRPO semantics.

## Measured systems evidence

Expand Down
3 changes: 2 additions & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,8 @@ distillation:

使用 `miniverl profiles show`、`compat explain` 与 `compat check` 查看完整映射。
[兼容 profile](docs/profiles/index.md)说明 profile identity 如何跟随 plan、cache、checkpoint
与 export。
与 export。另有两个仅完成一致性验证的 grouped profile,为 Parquet prompt 提供事务化
`n>1` 独立样本;它们不改变两个实测 `n=1` profile,也不引入 GRPO 语义。

## 实测系统证据

Expand Down
10 changes: 6 additions & 4 deletions docs/for-verl-users.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,9 @@ by plans, caches, checkpoints and exports.

- A resolved YAML using the `verl-opd-v0.8-single-gpu-v1` field subset.
- Reward-free verl-style Parquet prompts with structured chat messages.
- One actor, one teacher, `n=1`, token-mean aggregation, and either direct
forward-top-k GKD or the sampled-k1 vanilla policy-loss profile.
- One actor, one teacher, token-mean aggregation, and either direct
forward-top-k GKD or sampled-k1 vanilla policy loss. The measured profiles
remain `n=1`; their grouped counterparts accept independent `n>1` samples.
- Immutable Hugging Face revisions, PEFT adapters and tokenizer snapshots.
- Familiar fields such as `actor_rollout_ref.model.path`,
`distillation.teacher_models.teacher_model.model_path`, response bounds,
Expand Down Expand Up @@ -125,8 +126,9 @@ has progressed.
documented profile; inspect-only compilation can still explain known
unsupported values.
- **Algorithm field rejected:** select the explicit PG-k1 profile for its
narrow sampled policy-loss path. Rewards, critics, external advantages, KL
penalties, `n>1`, multiple teachers and distributed counts remain unsupported.
narrow sampled policy-loss path, or a grouped profile for Parquet `n>1`.
Rewards, critics, external advantages, KL penalties, group-relative
baselines, multiple teachers and distributed counts remain unsupported.
- **Interpolation rejected:** resolve Hydra/OmegaConf in your trusted verl
environment first. miniVERL will not execute `${...}`.
- **High-risk reinterpretation not accepted:** inspect `miniverl plan`, then
Expand Down
16 changes: 16 additions & 0 deletions docs/generated/upstream-support-matrix.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,22 @@
"upstream_commit": "7aed6b230776f963fa09509c10d9c3a767d1102c",
"objective": "sampled_k1_vanilla_policy_loss",
"distributed_execution_tested": false
},
{
"name": "verl-opd-v0.8-single-gpu-grouped-v1",
"status": "active",
"upstream_tag": "v0.8.0",
"upstream_commit": "7aed6b230776f963fa09509c10d9c3a767d1102c",
"objective": "forward_kl_topk_grouped_independent_samples",
"distributed_execution_tested": false
},
{
"name": "verl-opd-v0.8-single-gpu-pg-k1-grouped-v1",
"status": "active",
"upstream_tag": "v0.8.0",
"upstream_commit": "7aed6b230776f963fa09509c10d9c3a767d1102c",
"objective": "sampled_k1_grouped_independent_samples",
"distributed_execution_tested": false
}
],
"unprofiled_versions": "unsupported"
Expand Down
2 changes: 1 addition & 1 deletion docs/generated/verl-opd-v0.8-compatibility.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"locally_reinterpreted": 20,
"semantically_conformant": 14
},
"compiled_digest": "860b25e12c88bc772c9003008aafece2e4a71337a75373441f940ddde65bd008",
"compiled_digest": "0a42776d063064da484ddb47670baa03d11f51a9cf13b8b2e450e7455c66d5eb",
"executable": true,
"field_count": 77,
"fields": [
Expand Down
8 changes: 6 additions & 2 deletions docs/profiles/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,15 @@ name.
| --- | --- | --- | --- | --- |
| `verl-opd-v0.8-single-gpu-v1` | direct GKD `forward_kl_topk` | top-k token IDs and log-probabilities | fuller distributional signal; larger target artifact | measured |
| `verl-opd-v0.8-single-gpu-pg-k1-v1` | sampled `k1` + vanilla policy loss | sampled-token teacher log-probability | sampled-token signal; smaller target artifact | measured |
| `verl-opd-v0.8-single-gpu-grouped-v1` | direct GKD over independent grouped samples | top-k token IDs and log-probabilities | configurable `n>1`; no group baseline | conformance only |
| `verl-opd-v0.8-single-gpu-pg-k1-grouped-v1` | sampled `k1` over independent grouped samples | sampled-token teacher log-probability | configurable `n>1`; no GRPO estimator | conformance only |

Both are reward-free, strict current-policy, one-actor/one-teacher profiles on
one CUDA GPU. The PG profile uses a detached distillation-derived advantage
All four are reward-free, strict current-policy, one-actor/one-teacher profiles
on one CUDA GPU. The PG profiles use a detached distillation-derived advantage
with the pinned vanilla policy-loss form. The measured records compare runtime
and semantic conformance; task-quality questions belong to a declared benchmark.
The two grouped profiles add transactional Parquet prompt groups while leaving
the published `n=1` identities unchanged; each sample is optimized independently.
See [For verl users](../for-verl-users.md), the
[PG-k1 ADR](../adr/0010-verl-v0.8-pg-k1-contract.md), and the runtime evidence
for the measured boundary.
6 changes: 3 additions & 3 deletions docs/release-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ after the exact release commit and its remote checks are green.
`0d6e0070ae73ef35f718aec3624ee5263ac96e3a` and candidate wheel
`0256bd9e63ca6ed52999a5073a3577a008581a8d4418062314750d86f21cd5fe`
before implementing `hf_cached`.
- [ ] Add typed policy identity, lifecycle and strict synchronization contracts.
- [ ] Qualify `hf_cached` at 64/256/512 response bounds and require at least 2x
- [x] Add typed policy identity, lifecycle and strict synchronization contracts.
- [x] Qualify `hf_cached` at 64/256/512 response bounds and require at least 2x
speedup at 256 and 512 without exceeding 14.5 GiB peak reserved VRAM.
- [ ] Add transactional `samples_per_prompt > 1` and exact no-skip/no-duplicate
- [x] Add transactional `samples_per_prompt > 1` and exact no-skip/no-duplicate
resume semantics without changing the published `n=1` profiles.
- [ ] Add the new deterministic task-reward plus distillation-advantage profile;
keep direct GKD and existing PG-k1 identities unchanged.
Expand Down
31 changes: 29 additions & 2 deletions docs/rollout-runtime-v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,33 @@ profile identity and execution plan. A request for any other identity fails
before model execution. Lifecycle transitions are explicit: `new` →
`synchronized` → `quiesced` or `closed`.

## Grouped prompt rollouts

The grouped profiles accept `actor_rollout_ref.rollout.n > 1` for Parquet
prompt sources. Each prompt receives a stable group identity and `n` independent
current-policy trajectories. The direct-GKD and sampled-k1 objectives train
those trajectories independently; there is no group baseline, GRPO estimator
or reward normalization hidden behind `n`.

Trajectory schema v3 binds the prompt digest, group and sample indices,
generation seed, backend and policy identity into every record. A complete
group is journaled and appended as one transaction. Checkpoints retain the
prompt cursor, group cursor, trajectory count, policy version and backend sync
identity, so an interrupted group is regenerated without skipping prompts;
replaying an already committed identical group is a no-op rather than a
duplicate append. Cache and export identities carry the same `n`, schema and
seed-derivation versions.

The runtime reports unique prompts, groups, trajectories, generated tokens,
physical generation batches and per-group reward variance when rewards exist.
Logical `n` stays separate from physical batch partitioning.

Use `verl-opd-v0.8-single-gpu-grouped-v1` or
`verl-opd-v0.8-single-gpu-pg-k1-grouped-v1`. The original measured profiles
remain fixed at `n=1`; the grouped profiles are conformance-only until a later
evidence stage measures them. Grouped rollouts currently require a Parquet
prompt source. Environment-backed recipes remain `n=1`.

The v0.11 baseline result remains the pre-change `hf_reference` measurement.
It is not evidence for `hf_cached` performance. Grouped `n>1` publication and
external-engine support are separate release-chain stages.
It is not evidence for `hf_cached` or grouped-rollout performance. External
engine support is a separate release-chain stage.
7 changes: 6 additions & 1 deletion src/miniverl/alignment/workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,12 @@ def _sha256(path: Path) -> str:

def _tagged(rows: list[Any], tag: str) -> list[Any]:
marker = f":{tag}:v"
return [row for row in rows if marker in row.trajectory_id]
return [
row
for row in rows
if marker in row.trajectory_id
or marker in str(row.metadata.get("legacy_trajectory_id", ""))
]


def _decision_distribution(rows: list[Any]) -> dict[str, float]:
Expand Down
41 changes: 29 additions & 12 deletions src/miniverl/bridge/export.py
8
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@
)
from miniverl.bridge.opd_pg_v08 import VERL_OPD_PG_K1_V08_PROFILE
from miniverl.bridge.opd_v08 import VERL_OPD_V08_PROFILE
from miniverl.bridge.profiles import (
VERL_OPD_GROUPED_V08_PROFILE,
VERL_OPD_PG_K1_GROUPED_V08_PROFILE,
)
from miniverl.errors import ConfigError
from miniverl.utils.privacy import portable_payload
from miniverl.utils.runs import read_json, write_json, write_text
Expand Down Expand Up @@ -428,7 +432,15 @@ def _write_hashes(root: Path) -> None:
write_text(checksum, "\n".join(lines) + "\n")


_OPD_PROFILES = frozenset({VERL_OPD_V08_PROFILE, VERL_OPD_PG_K1_V08_PROFILE})
_PG_PROFILES = frozenset({VERL_OPD_PG_K1_V08_PROFILE, VERL_OPD_PG_K1_GROUPED_V08_PROFILE})
_OPD_PROFILES = frozenset(
{
VERL_OPD_V08_PROFILE,
VERL_OPD_PG_K1_V08_PROFILE,
VERL_OPD_GROUPED_V08_PROFILE,
VERL_OPD_PG_K1_GROUPED_V08_PROFILE,
}
)


def _opd_run_contract(run: Path) -> tuple[dict[str, Any], dict[str, Any]] | None:
Expand Down Expand Up @@ -458,7 +470,7 @@ def _opd_run_contract(run: Path) -> tuple[dict[str, Any], dict[str, Any]] | None
"distillation.distillation_loss.use_policy_gradient": True,
"distillation.distillation_loss.policy_loss_mode": "vanilla",
}
if profile == VERL_OPD_PG_K1_V08_PROFILE
if profile in _PG_PROFILES
else {
"distillation.distillation_loss.loss_mode": "forward_kl_topk",
"distillation.distillation_loss.use_policy_gradient": False,
Expand Down Expand Up @@ -549,7 +561,7 @@ def _opd_overrides(
# no such teacher inference key. It therefore belongs in provenance, not
# in an override file that promises to parse upstream.
teacher.get("inference", {}).pop("pipeline_model_parallel_size", None)
if profile == VERL_OPD_PG_K1_V08_PROFILE:
if profile in _PG_PROFILES:
overrides["distillation"]["distillation_loss"].pop("topk", None)
return overrides

Expand Down Expand Up @@ -581,7 +593,7 @@ def _opd_launch_script(adapter: dict[str, Any], teacher: dict[str, Any]) -> str:
def _opd_bundle_readme(profile: str) -> str:
semantics = (
"sampled k1 teacher log-probabilities and pinned vanilla policy-gradient loss"
if profile == VERL_OPD_PG_K1_V08_PROFILE
if profile in _PG_PROFILES
else "pure GKD `forward_kl_topk` targets"
)
return f"""# miniVERL pure-OPD scale-out bundle
Expand Down Expand Up @@ -610,6 +622,9 @@ def _export_opd_bundle(
destination: Path,
) -> dict[str, Any]:
profile = str(compatibility["profile"])
samples_per_prompt = _get(source, "actor_rollout_ref.rollout.n")
if not isinstance(samples_per_prompt, int) or samples_per_prompt < 1:
raise ConfigError("compiled OPD source has no valid rollout.n identity")
student_id = _get(source, "actor_rollout_ref.model.path")
student_revision = _get(source, "miniverl.student_revision")
if student_id != adapter["base_model"] or student_revision != adapter["revision"]:
Expand Down Expand Up @@ -697,14 +712,19 @@ def _export_opd_bundle(
"miniverl_version": __version__,
"target_semantics": (
"pure OPD sampled k1 policy gradient"
if profile == VERL_OPD_PG_K1_V08_PROFILE
if profile in _PG_PROFILES
else "pure GKD forward_kl_topk OPD"
),
"target_representation": (
"sampled_token_log_probability"
if profile == VERL_OPD_PG_K1_V08_PROFILE
else "topk_distribution"
"sampled_token_log_probability" if profile in _PG_PROFILES else "topk_distribution"
),
"grouped_rollout_semantics": {
"samples_per_prompt": samples_per_prompt,
"trajectory_schema_version": 3,
"sample_semantics": "independent_current_policy_trajectory",
"group_baseline": False,
"grpo": False,
},
"artifact_complete": False,
"artifact_bundle_complete": True,
"config_semantics_supported": True,
Expand All @@ -724,10 +744,7 @@ def _export_opd_bundle(
"unsupported_semantics": [
item
for item in _UNSUPPORTED
if not (
profile == VERL_OPD_PG_K1_V08_PROFILE
and item == "PPO advantage or clipping semantics"
)
if not (profile in _PG_PROFILES and item == "PPO advantage or clipping semantics")
],
"distributed_execution_status": "not tested",
}
Expand Down
31 changes: 27 additions & 4 deletions src/miniverl/bridge/opd_pg_v08.py
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,9 @@ def pg_field_rules_digest() -> str:
return _canonical_digest([pg_compatibility_rule(field) for field in sorted(_PG_FIELD_RULES)])


def _pg_semantic_blocker(path: str, value: Any) -> str | None:
def _pg_semantic_blocker(
path: str, value: Any, *, allow_grouped_samples: bool = False
) -> str | None:
required = {
"distillation.enabled": (True, "distillation must be enabled"),
"distillation.distillation_loss.loss_mode": (
Expand Down Expand Up @@ -175,7 +177,6 @@ def _pg_semantic_blocker(path: str, value: Any) -> str | None:
"token-mean",
"only token-mean aggregation is supported",
),
"actor_rollout_ref.rollout.n": (1, "one generation per prompt is required"),
"actor_rollout_ref.rollout.tensor_model_parallel_size&quo 10BC8 t;: (
1,
"tensor parallelism greater than one is unsupported",
Expand All @@ -194,6 +195,11 @@ def _pg_semantic_blocker(path: str, value: Any) -> str | None:
),
"trainer.nnodes": (1, "multi-node execution is unsupported"),
}
if path == "actor_rollout_ref.rollout.n":
if allow_grouped_samples and isinstance(value, int) and value >= 1:
return None
if value != 1:
return "one generation per prompt is required by this legacy profile"
expected = required.get(path)
if expected is not None and value != expected[0]:
return expected[1]
Expand Down Expand Up @@ -226,6 +232,8 @@ def compile_verl_pg_k1_v08(
reinterpretations_accepted: bool = True,
acceptance_source: str = "library_call",
require_executable: bool = True,
allow_grouped_samples: bool = False,
profile_name: str = VERL_OPD_PG_K1_V08_PROFILE,
) -> CompiledLocalExecutionPlan:
"""Compile the closed sampled-k1 profile into a deterministic local plan."""
merged = copy.deepcopy(dict(payload))
Expand Down Expand Up @@ -268,7 +276,11 @@ def compile_verl_pg_k1_v08(
blockers: list[str] = []
for path, value in sorted(flat.items()):
rule = _PG_FIELD_RULES[path]
blocker = _pg_semantic_blocker(path, value)
blocker = _pg_semantic_blocker(
path,
value,
allow_grouped_samples=allow_grouped_samples,
)
if blocker:
blockers.append(path)
classification: FieldClassification = "unsupported"
Expand Down Expand Up @@ -326,10 +338,13 @@ def compile_verl_pg_k1_v08(
"policy_gradient": True,
"policy_loss_mode": "vanilla",
"teacher_target": "sampled_token_log_probability",
"samples_per_prompt": source.actor_rollout_ref.rollout.n,
"group_semantics": "independent_current_policy_trajectories",
"trajectory_schema_version": 3 if allow_grouped_samples else 2,
}
high_risk = [item.upstream_field for item in compatibility if item.user_confirmation_required]
common = {
"profile": VERL_OPD_PG_K1_V08_PROFILE,
"profile": profile_name,
"upstream": {"repository": VERL_REPOSITORY, "tag": VERL_TAG, "commit": VERL_COMMIT},
"source_digest": _canonical_digest(merged),
"source_leaf_fields": sorted(flat),
Expand Down Expand Up @@ -370,6 +385,8 @@ def load_verl_pg_k1_v08(
trailing_overrides: Sequence[str] = (),
accept_local_reinterpretations: bool = False,
require_executable: bool = True,
allow_grouped_samples: bool = False,
profile_name: str = VERL_OPD_PG_K1_V08_PROFILE,
) -> CompiledLocalExecutionPlan:
"""Load one resolved YAML document without executing interpolation."""
if path.suffix.lower() in {".sh", ".bash", ".ps1", ".cmd", ".bat"}:
Expand All @@ -388,6 +405,8 @@ def load_verl_pg_k1_v08(
reinterpretations_accepted=accept_local_reinterpretations,
acceptance_source="cli_flag" if accept_local_reinterpretations else "not_accepted",
require_executable=require_executable,
allow_grouped_samples=allow_grouped_samples,
profile_name=profile_name,
)


Expand All @@ -399,6 +418,8 @@ def load_verl_pg_k1_v08_source(
trailing_overrides: Sequence[str] = (),
accept_local_reinterpretations: bool = False,
require_executable: bool = True,
allow_grouped_samples: bool = False,
profile_name: str = VERL_OPD_PG_K1_V08_PROFILE,
) -> CompiledLocalExecutionPlan:
"""Load the PG profile from an explicit resolved YAML path."""
return load_verl_pg_k1_v08(
Expand All @@ -408,4 +429,6 @@ def load_verl_pg_k1_v08_source(
trailing_overrides=trailing_overrides,
accept_local_reinterpretations=accept_local_reinterpretations,
require_executable=require_executable,
allow_grouped_samples=allow_grouped_samples,
profile_name=profile_name,
)
Loading
0