8000 Add portable hardware evidence records by DaoyuanLi2816 · Pull Request #82 · DaoyuanLi2816/mini-verl · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions .github/ISSUE_TEMPLATE/benchmark_submission.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,18 @@ body:
Thank you. Measured numbers from hardware we do not own are the most
useful contribution this project can receive.

Produce a submission file first:
Produce and validate a private-by-default hardware record first:

```bash
miniverl train <recipe>
miniverl export-benchmark runs/<run-id> --out my-result.json
miniverl hardware record --run runs/<run-id> --out hardware-record.json
miniverl hardware validate hardware-record.json
```

`export-benchmark` strips absolute paths and keeps only the GPU model,
VRAM, OS family and library versions. Validate it against
`benchmarks/schema/benchmark-result.schema.json`, compute its SHA-256,
and read it before posting. Never include tokens, private paths or
proprietary data.
The record binds profile, plan, models, quantization, token bounds,
batches, updates, memory, timing and retained artifact hashes. The
command never uploads. Read the JSON before posting and never include
tokens, private paths or proprietary data. Schema-valid community
records remain unreviewed until a maintainer validates their evidence.

- type: input
id: hardware
Expand All @@ -40,7 +40,7 @@ body:
- type: textarea
id: result
attributes:
label: Contents of the exported result JSON
label: Contents of the validated hardware-record JSON
render: json
validations:
required: true
Expand All @@ -58,7 +58,7 @@ body:
- type: input
id: artifact_hash
attributes:
label: Exported JSON SHA-256
label: Hardware-record JSON SHA-256
description: Hash the exact JSON attached to this submission.
placeholder: "64 lowercase hexadecimal characters"
validations:
Expand All @@ -81,9 +81,11 @@ body:
required: true
- label: I ran the recipe to completion and did not select the best of several runs without saying so.
required: true
- label: I validated the JSON against the committed schema and recorded the exact artifact SHA-256.
- label: I ran `miniverl hardware validate` and recorded the exact JSON SHA-256.
required: true
- label: I reviewed the export and attachments for tokens, private paths and proprietary data.
required: true
- label: I consent to publishing this sanitized record and its reported hardware/software metadata.
required: true
- label: I am willing to open a pull request adding this file under `benchmarks/results/`.
required: false
1 change: 1 addition & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ jobs:
python scripts/publish_alignment_lab_artifacts.py --check
python scripts/publish_verl_bridge_diagrams.py --check
python scripts/publish_verl_opd_compatibility.py --check
python scripts/publish_hardware_record_schema.py --check
- name: Build complete development site strictly
run: mkdocs build --strict
- name: Install deterministic Chromium
Expand Down
2 changes: 2 additions & 0 deletions PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ Development `0.10.0.dev0` has a closed typed profile registry and torch-free
compatibility introspection. Profile-scoped plans, caches, checkpoints and
exports bind an independent identity. The direct-GKD and sampled-k1 vanilla
policy-loss profiles both have pinned conformance and measured RTX 4080 paths.
Portable hardware records preserve measured, estimated and unknown states;
schema-valid community submissions remain unreviewed until maintainer validation.

Arbitrary verl YAML, other policy-gradient modes, rewards, PPO/GRPO, Ray, FSDP,
Megatron, multi-GPU and distributed execution remain unsupported. The legacy
Expand Down
10 changes: 10 additions & 0 deletions PYPI.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,16 @@ updates. See [hardware planning](https://github.com/DaoyuanLi2816/mini-verl/blob
automatic downgrade to a different model,
teacher, context, top-k or loss when memory is tight.

Share a sanitized result without uploading it automatically:

```bash
miniverl hardware record --run runs/my-opd --out hardware-record.json
miniverl hardware validate hardware-record.json
```

Community records remain unreviewed until their hashes and provenance pass the
[documented maintainer gate](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/community-benchmarks.md).

## Data and artifact interoperability

The profile consumes structured verl-style Parquet without substituting a toy
Expand Down
10 changes: 10 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,16 @@ updates. See [hardware planning](docs/hardware-planning.md). There is no
automatic downgrade to a different model,
teacher, context, top-k or loss when memory is tight.

Share a sanitized result without uploading it automatically:

```bash
miniverl hardware record --run runs/my-opd --out hardware-record.json
miniverl hardware validate hardware-record.json
```

Community records remain unreviewed until their hashes and provenance pass the
[documented maintainer gate](docs/community-benchmarks.md).

## Data and artifact interoperability

The profile consumes structured verl-style Parquet without substituting a toy
Expand Down
10 changes: 10 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,16 @@ BF16/FP16 自动选择取决于设备能力,而不是 3070、4080、5090 或 T
[硬件规划](docs/hardware-planning.md)。
显存紧张时不会静默更换模型、teacher、context、top-k 或 loss。

已完成的 run 可以生成经过清理、且不会自动上传的硬件记录:

```bash
miniverl hardware record --run artifacts/runs/<run-id> --out hardware-record.json
miniverl hardware validate hardware-record.json
```

社区提交即使通过 schema 校验也仍标记为 `unreviewed`;只有维护者复核并确认发布许可后,
才会进入公开的“maintainer-validated measured”矩阵。

## 数据与产物互操作

profile 直接读取结构化 verl 风格 Parquet,不会在缺失数据时静默换成 calculator 环境。
Expand Down
28 changes: 28 additions & 0 deletions docs/community-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,31 @@ To propose a real result, add the JSON plus any new versioned recipe record in
one pull request. Describe unsupported hardware or software explicitly; a
negative or failed run is useful evidence and should not be removed merely
because it does not improve a benchmark.

## OPD hardware records

Completed OPD runs use a separate, stricter version-1 record:

```bash
miniverl hardware record --run runs/my-opd --out hardware-record.json
miniverl hardware validate hardware-record.json
```

The record binds the profile identity, plan and config digests, GPU/VRAM and
software stack, pinned actor/teacher identities, quantization, prompt/response
bounds, teacher-target mode, logical and physical batches, update count,
measured memory and phase timing, resume state, and artifact hashes. Every
numeric evidence item is labelled `measured`, `estimated`, or `unknown`.
Generation and validation are torch-free and perform no upload. The committed
[JSON Schema](generated/hardware-record-v1.schema.json) is generated from the
same Pydantic contract used by the CLI.

| Record origin | Public status | Required gate |
| --- | --- | --- |
| `maintainer_measured` | measured only after repository review | retained run, hashes, schema/privacy validation and maintainer approval |
| `community_submitted` | unreviewed candidate | validated JSON plus explicit publication consent |
| `not_measured` | documented gap | no numeric value may be presented as measured |

`hardware validate` proves schema and privacy conformance only. It does not
change `review_status`, publish a file or add a row to the public matrix. An
issue comment or unreviewed JSON can never become a measured row automatically.
Loading
0