From 6f47935675abe2ea4af6fd6ecd1d3353073782ec Mon Sep 17 00:00:00 2001 From: Daoyuan Li <94409450+DaoyuanLi2816@users.noreply.github.com> Date: Sun, 16 Aug 2026 11:36:52 -0700 Subject: [PATCH 1/3] Fix v1 readiness state after v0.10.1 --- docs/v1-readiness.md | 72 +++++++++++++++++++-------------- tests/unit/test_v1_readiness.py | 46 +++++++++++++++++++++ 2 files changed, 87 insertions(+), 31 deletions(-) create mode 100644 tests/unit/test_v1_readiness.py diff --git a/docs/v1-readiness.md b/docs/v1-readiness.md index e1e659c..d15ab1f 100644 --- a/docs/v1-readiness.md +++ b/docs/v1-readiness.md @@ -1,26 +1,25 @@ # v1 readiness contract -Version 1 is a stability promise, not a feature-count milestone. miniVERL may -be proposed as a v1 candidate only when every required row below is `passed` on -the exact candidate commit. The current development line remains Beta. +Version 1 is a stability promise, not a feature-count milestone. A successful +release qualification is necessary evidence, but it does not by itself freeze +the public surface or make miniVERL a v1 candidate. -## Compatibility commitments +## Stability contract - Stable CLI commands retain their names, option meanings, exit-code class and documented JSON shape. A rename or semantic change requires deprecation for at least one minor release. - `miniverl.config.RunConfig`, `miniverl.trainer.OPDTrainer` and documented - adapter/cache/checkpoint readers remain import-compatible across the v1 + adapter, cache and checkpoint readers remain import-compatible across the v1 line. Internal coordinators are not public APIs. - Published compatibility profile identities never drift. A new upstream verl version, estimator or compiler contract receives a new profile identity. - Plan, trajectory, teacher-cache, checkpoint, run-manifest, hardware-record, - qualification and export-bundle schemas are versioned. Readers either accept - an older version exactly or reject it with a migration path; writers never + qualification and export-bundle schemas are versioned. Readers accept an + older version exactly or reject it with a migration path; writers never silently reuse a version number for new semantics. - Security fixes may tighten acceptance of hostile or ambiguous input without - a deprecation window. They must not reinterpret an accepted training - objective. + a deprecation window, but must not reinterpret an accepted objective. Exact resume means the documented tensors, progress cursor, policy version, optimizer/RNG state and trajectory order match for an eligible same-platform, @@ -28,27 +27,38 @@ same-profile run. It does not promise equivalence across different CUDA, PyTorch, kernels, model revisions, objectives or unsupported distributed runtimes. -## Candidate gate +## Evidence achieved in v0.10.1 -| requirement | required state | +The following evidence **passed in v0.10.1** on release commit +[`2364e9b8e8b550f44d1b66a77fc2d407b76b05b5`](https://github.com/DaoyuanLi2816/mini-verl/commit/2364e9b8e8b550f44d1b66a77fc2d407b76b05b5): + +| evidence | verified record | | --- | --- | -| stable docs and PyPI quickstart agree with `release-state.yaml` | passed | -| base wheel install remains torch-free | passed | -| known-good CUDA environment is machine-readable and installable | passed | -| hosted-built candidate wheel is the exact wheel qualified on one RTX 4080 | passed | -| full qualification covers both profiles, SmolLM2 and exact resume | passed for a v1 candidate | -| public schemas and profile lifecycle policy are documented | passed | -| pre-release gate, build, Twine, sdist, link, visual and privacy checks | passed | -| deprecation and supported-version policy are current | passed | -| candidate manifest, same-run qualification, tag, published bytes and attestations share provenance | passed | - -The following are explicitly outside v1: arbitrary verl YAML, PPO, GRPO, -critics, general reward pipelines, Ray, FSDP, Megatron, vLLM/SGLang execution, -multi-GPU, multi-node, distributed launch, multimodal models, multiple teachers -and broad cross-hardware qualification. Scale-out artifacts do not prove -distributed execution. - -At present miniVERL is **not a v1 candidate**: the byte-identical candidate -chain is defined, but private-runner activation and its first successful -same-run candidate/qualification pair are external pending steps. No version -change follows from this document alone. +| hosted-built candidate wheel and full qualification from the same first-attempt run on one RTX 4080 | [GPU qualification run 31932226695](https://github.com/DaoyuanLi2816/mini-verl/actions/runs/31932226695) | +| direct GKD, sampled-k1, SmolLM2, PEFT reload and exact interruption/resume | the full qualification record from run 31932226695 | +| non-publishing reuse of those exact bytes without rebuilding | [release dry-run 31933844796](https://github.com/DaoyuanLi2816/mini-verl/actions/runs/31933844796) | +| OIDC publication, public file hashes and attestations, clean install and GitHub Release creation | [tag release run 31934196365](https://github.com/DaoyuanLi2816/mini-verl/actions/runs/31934196365) | +| immutable release identity | [`v0.10.1` tag](https://github.com/DaoyuanLi2816/mini-verl/tree/v0.10.1), [GitHub Release](https://github.com/DaoyuanLi2816/mini-verl/releases/tag/v0.10.1), [PyPI](https://pypi.org/project/miniverl/0.10.1/) | +| base wheel remains torch-free; stable docs and PyPI quickstart agree | release gate and clean public-install jobs in run 31934196365 | + +The published wheel SHA-256 is +`fa5cdef1b6d0602ead7e90f6f51a024aa67c2f10d3f6c9d0507c3bbc3f1c82e8`; +the sdist SHA-256 is +`743cf17f365710bf3ddbe8c5599c6a8daf0afa82779344b449f7f1e93cd2d4f92`. +This is a manual maintainer qualification, not continuous GPU CI. It establishes +runtime correctness only on the measured stack, not task quality or broad +consumer-GPU coverage. + +## Remaining v1 gates + +| gate | current state | reviewable completion condition | +| --- | --- | --- | +| v1 public API baseline | **not yet satisfied** | Freeze the intended CLI, Python imports, JSON outputs and artifact schemas in a machine-readable baseline; gate compatibility in CI while excluding internal coordinators. | +| backward-compatibility evidence | **not yet satisfied** | Across at least one later stable release, execute the listed old plan, cache, checkpoint, adapter/export and other promised reader paths; unsupported distributed formats remain out of scope. | +| independent reproduction or explicit v1 hardware scope | **not yet satisfied** | Reproduce on a machine independent of the current workstation before a broad consumer-GPU claim, or approve a dedicated v1 scope decision limited to the maintainer-qualified stack. Untested GPUs are not evidence. | +| dedicated v1 candidate decision | **not yet satisfied** | Review a dedicated PR that updates classifiers, versioning, support policy and migration notes after all chosen gates pass. This documentation repair cannot trigger it. | + +miniVERL is therefore **not a v1 candidate**. PPO, GRPO, critics, general reward +pipelines, arbitrary verl YAML, Ray, FSDP, Megatron, vLLM/SGLang execution, +multi-GPU, multi-node and distributed launch remain outside the current product +boundary. Scale-out artifacts do not prove distributed execution. diff --git a/tests/unit/test_v1_readiness.py b/tests/unit/test_v1_readiness.py new file mode 100644 index 0000000..dc06e2c --- /dev/null +++ b/tests/unit/test_v1_readiness.py @@ -0,0 +1,46 @@ +"""Prevent v1-readiness prose from drifting behind canonical release state.""" + +from __future__ import annotations + +from pathlib import Path + +import yaml + +ROOT = Path(__file__).resolve().parents[2] + + +def test_v1_readiness_matches_canonical_v0101_release_evidence() -> None: + state = yaml.safe_load((ROOT / "release-state.yaml").read_text(encoding="utf-8")) + text = (ROOT / "docs/v1-readiness.md").read_text(encoding="utf-8") + stable = state["stable"] + assert stable["version"] >= "0.10.1" + assert stable["release_commit"] in text + assert "passed in v0.10.1" in text + assert "31932226695" in text + assert "31933844796" in text + assert "31934196365" in text + + +def test_v1_readiness_has_no_obsolete_or_contradictory_gate() -> None: + text = (ROOT / "docs/v1-readiness.md").read_text(encoding="utf-8").lower() + obsolete = ( + "private-runner activation", + "private runner activation", + "first successful same-run", + "first same-run pair", + ) + assert all(phrase not in text for phrase in obsolete) + assert "not a v1 candidate" in text + assert "not yet satisfied" in text + assert "v1 public api baseline" in text + assert "backward-compatibility evidence" in text + assert "independent reproduction or explicit v1 hardware scope" in text + assert "dedicated v1 candidate" in text + + +def test_polish_does_not_change_version_or_beta_classifier() -> None: + state = yaml.safe_load((ROOT / "release-state.yaml").read_text(encoding="utf-8")) + pyproject = (ROOT / "pyproject.toml").read_text(encoding="utf-8") + assert state["stable"]["version"] == "0.10.1" + assert state["development"]["version"] == "0.10.2.dev0" + assert "Development Status :: 4 - Beta" in pyproject From 9f777a076a172d7b654a7ccc11392716f0f07b99 Mon Sep 17 00:00:00 2001 From: Daoyuan Li <94409450+DaoyuanLi2816@users.noreply.github.com> Date: Sun, 16 Aug 2026 11:36:53 -0700 Subject: [PATCH 2/3] Canonicalize future release evidence assets --- .github/workflows/release.yml | 80 ++-- scripts/prepare_release_assets.py | 55 +++ src/miniverl/release_assets.py | 635 ++++++++++++++++++++++++++++ tests/unit/test_release_assets.py | 478 +++++++++++++++++++++ tests/unit/test_release_workflow.py | 22 +- 5 files changed, 1217 insertions(+), 53 deletions(-) create mode 100644 scripts/prepare_release_assets.py create mode 100644 src/miniverl/release_assets.py create mode 100644 tests/unit/test_release_assets.py diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index e80727c..24fdcf6 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -126,57 +126,26 @@ jobs: with: name: verified-qualified-candidate path: accepted - - name: Revalidate and copy the identical candidate bytes + - name: Revalidate and prepare canonical identical assets run: | python -m pip install . python scripts/build_release_candidate.py --output accepted/candidate --commit "$GITHUB_SHA" --check - python - <<'PY' - import hashlib, json, pathlib, shutil - root = pathlib.Path("accepted") - candidate = root / "candidate" - qualification_root = root / "qualification" - qualification = qualification_root / "qualification.json" - manifest_path = candidate / "candidate-manifest.json" - manifest = json.loads(manifest_path.read_text(encoding="utf-8")) - manifest_sha = hashlib.sha256(manifest_path.read_bytes()).hexdigest() - record = json.loads(qualification.read_text(encoding="utf-8")) - if record["candidate"]["manifest_sha256"] != manifest_sha: - raise SystemExit("qualification candidate-manifest binding changed") - if record["wheel"]["sha256"] != manifest["wheel"]["sha256"]: - raise SystemExit("qualification wheel binding changed") - output = pathlib.Path("release-distributions") - (output / "dist").mkdir(parents=True) - for key in ("wheel", "sdist"): - shutil.copy2(candidate / manifest[key]["filename"], output / "dist") - shutil.copy2(candidate / "SHA256SUMS", output / "SHA256SUMS") - shutil.copy2(manifest_path, output / "candidate-manifest.json") - shutil.copy2(qualification, output / "qualification.json") - shutil.copy2(root / "verification.json", output / "verification.json") - evidence_output = output / "qualification-evidence" - evidence_output.mkdir() - version = manifest["miniverl_version"] - evidence_files = [("full-qualification", qualification)] - evidence_files.extend( - (item["name"].replace("_", "-"), qualification_root / item["path"]) - for item in record["artifacts"] - ) - sums = [] - for label, source in evidence_files: - suffix = source.suffix or ".bin" - target = evidence_output / f"miniverl-{version}-{label}{suffix}" - shutil.copy2(source, target) - digest = hashlib.sha256(target.read_bytes()).hexdigest() - sums.append(f"{digest} {target.name}") - (output / "qualification-full-SHA256SUMS").write_text( - "\n".join(sums) + "\n", encoding="utf-8", newline="\n" - ) - for key in ("wheel", "sdist"): - copied = output / "dist" / manifest[key]["filename"] - if hashlib.sha256(copied.read_bytes()).hexdigest() != manifest[key]["sha256"]: - raise SystemExit(f"copy changed {key} bytes") - PY + python scripts/prepare_release_assets.py \ + --candidate-dir accepted/candidate \ + --candidate-manifest accepted/candidate/candidate-manifest.json \ + --qualification-root accepted/qualification \ + --qualification accepted/qualification/qualification.json \ + --verification accepted/verification.json \ + --output release-distributions + python scripts/prepare_release_assets.py \ + --candidate-dir accepted/candidate \ + --candidate-manifest accepted/candidate/candidate-manifest.json \ + --qualification-root accepted/qualification \ + --qualification accepted/qualification/qualification.json \ + --verification accepted/verification.json \ + --output release-distributions --check (cd release-distributions/dist && sha256sum --check ../SHA256SUMS) - (cd release-distributions/qualification-evidence && sha256sum --check ../qualification-full-SHA256SUMS) + (cd release-distributions && sha256sum --check qualification-SHA256SUMS) - name: Upload the identical qualified distributions uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: @@ -208,7 +177,7 @@ jobs: - name: Verify bytes immediately before publication run: | (cd release-artifacts/dist && sha256sum --check ../SHA256SUMS) - (cd release-artifacts/qualification-evidence && sha256sum --check ../qualification-full-SHA256SUMS) + (cd release-artifacts && sha256sum --check qualification-SHA256SUMS) - name: Determine whether exact candidate bytes already exist on PyPI id: pypi-state run: | @@ -286,12 +255,21 @@ jobs: GH_TOKEN: ${{ github.token }} run: | (cd release-artifacts/dist && sha256sum --check ../SHA256SUMS) + WHEEL=$(python -c 'import json; print(json.load(open("release-artifacts/candidate-manifest.json"))["wheel"]["filename"])') + SDIST=$(python -c 'import json; print(json.load(open("release-artifacts/candidate-manifest.json"))["sdist"]["filename"])') gh release create "$GITHUB_REF_NAME" \ - release-artifacts/dist/* \ + "release-artifacts/dist/$WHEEL" \ + "release-artifacts/dist/$SDIST" \ release-artifacts/SHA256SUMS \ - release-artifacts/qualification-full-SHA256SUMS \ - release-artifacts/qualification-evidence/* \ release-artifacts/candidate-manifest.json \ + release-artifacts/release-verification.json \ release-artifacts/qualification.json \ + release-artifacts/qualification-SHA256SUMS \ + release-artifacts/qualification-release-smoke.json \ + release-artifacts/qualification-direct-gkd.json \ + release-artifacts/qualification-pg-k1.json \ + release-artifacts/qualification-smollm2.json \ + release-artifacts/qualification-evidence.tar.gz \ + release-artifacts/qualification-evidence-manifest.json \ --repo "$GITHUB_REPOSITORY" --verify-tag \ --title "miniVERL $GITHUB_REF_NAME" --generate-notes diff --git a/scripts/prepare_release_assets.py b/scripts/prepare_release_assets.py new file mode 100644 index 0000000..5029a6f --- /dev/null +++ b/scripts/prepare_release_assets.py @@ -0,0 +1,55 @@ +"""Build or check the canonical future GitHub Release asset layout.""" + +from __future__ import annotations + +import argparse +from pathlib import Path + +from miniverl.release_assets import check_release_assets, prepare_release_assets + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--candidate-dir", type=Path) + parser.add_argument("--candidate-manifest", type=Path) + parser.add_argument("--qualification-root", type=Path) + parser.add_argument("--qualification", type=Path) + parser.add_argument("--verification", type=Path) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--check", action="store_true") + args = parser.parse_args() + if args.check: + problems = check_release_assets(args.output) + if problems: + raise SystemExit("\n".join(problems)) + print(f"canonical release assets valid: {args.output}") + return 0 + missing = [ + name + for name in ( + "candidate_dir", + "candidate_manifest", + "qualification_root", + "qualification", + "verification", + ) + if getattr(args, name) is None + ] + if missing: + parser.error( + "build mode requires: " + ", ".join(name.replace("_", "-") for name in missing) + ) + prepare_release_assets( + args.candidate_dir, + args.candidate_manifest, + args.qualification_root, + args.qualification, + args.verification, + args.output, + ) + print(f"canonical release assets prepared: {args.output}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/miniverl/release_assets.py b/src/miniverl/release_assets.py new file mode 100644 index 0000000..7beedb4 --- /dev/null +++ b/src/miniverl/release_assets.py @@ -0,0 +1,635 @@ +"""Build and validate the canonical, torch-free GitHub Release asset set.""" + +from __future__ import annotations + +import gzip +import hashlib +import io +import json +import re +import shutil +import stat +import tarfile +import tempfile +import unicodedata +from pathlib import Path, PurePosixPath +from typing import Any + +from miniverl.qualification import GPUQualification, sha256_file, validate_qualification_payload +from miniverl.release_candidate import load_candidate_manifest, validate_candidate_directory +from miniverl.release_chain import validate_release_chain +from miniverl.utils.privacy import portable_text + +__all__ = [ + "QUALIFICATION_SUM_FILES", + "check_release_assets", + "prepare_release_assets", + "validate_canonical_names", +] + +PRIMARY_EVIDENCE = { + "release_smoke_record": "qualification-release-smoke.json", + "full_direct_result": "qualification-direct-gkd.json", + "full_pg_k1_result": "qualification-pg-k1.json", + "full_smollm2_result": "qualification-smollm2.json", +} +ARCHIVE_EVIDENCE = { + "adapter_adapter_config.json": ("adapter_config", "adapter/adapter_config.json"), + "adapter_miniverl_adapter_manifest.json": ( + "adapter_manifest", + "adapter/miniverl_adapter_manifest.json", + ), + "adapter_adapter_model.safetensors": ( + "adapter_weights", + "adapter/adapter_model.safetensors", + ), + "inputs_prompts.parquet": ("input_prompts", "inputs/prompts.parquet"), + "run_summary": ("run_summary", "run-summary.json"), +} +_SPECIAL_EVIDENCE = {"candidate-manifest.json"} +QUALIFICATION_SUM_FILES = ( + "candidate-manifest.json", + "release-verification.json", + "qualification.json", + "qualification-release-smoke.json", + "qualification-direct-gkd.json", + "qualification-pg-k1.json", + "qualification-smollm2.json", + "qualification-evidence.tar.gz", + "qualification-evidence-manifest.json", +) +_FIXED_FILES = { + "SHA256SUMS", + "candidate-manifest.json", + "release-verification.json", + "qualification.json", + "qualification-SHA256SUMS", + *QUALIFICATION_SUM_FILES[3:], +} +_DOUBLE_SUFFIX = re.compile(r"(?i)(\.json\.json|\.parquet\.parquet|\.safetensors\.safetensors)$") +_WINDOWS_RESERVED = { + "CON", + "PRN", + "AUX", + "NUL", + *(f"COM{number}" for number in range(1, 10)), + *(f"LPT{number}" for number in range(1, 10)), +} + + +def _json(path: Path) -> Any: + return json.loads( + path.read_text(encoding="utf-8"), + parse_constant=lambda value: (_ for _ in ()).throw(ValueError(value)), + ) + + +def _json_bytes(payload: Any) -> bytes: + return ( + json.dumps(payload, indent=2, sort_keys=True, ensure_ascii=False, allow_nan=False) + "\n" + ).encode() + + +def _private(payload: Any) -> bool: + serialized = json.dumps(payload, ensure_ascii=False, sort_keys=True, allow_nan=False) + return portable_text(serialized) != serialized + + +def validate_canonical_names(names: list[str] | tuple[str, ...]) -> None: + """Reject names that are ambiguous on Windows or unsafe as release assets.""" + seen: set[str] = set() + for name in names: + if ( + not name + or name != unicodedata.normalize("NFC", name) + or not name.isascii() + or "/" in name + or "\\" in name + or name.startswith(".") + or name.endswith((".", " ")) + or ".." in name + or _DOUBLE_SUFFIX.search(name) + ): + raise ValueError(f"noncanonical release filename: {name!r}") + stem = name.split(".", 1)[0].upper() + if stem in _WINDOWS_RESERVED: + raise ValueError(f"reserved release filename: {name!r}") + folded = unicodedata.normalize("NFC", name).casefold() + if folded in seen: + raise ValueError(f"case-insensitive release filename collision: {name!r}") + seen.add(folded) + + +def _validate_archive_mapping() -> None: + seen_paths: set[str] = set() + seen_roles: set[str] = set() + for original_name, (semantic_role, archive_path) in ARCHIVE_EVIDENCE.items(): + if ( + not original_name + or not semantic_role.isascii() + or not re.fullmatch(r"[a-z][a-z0-9_]*", semantic_role) + ): + raise ValueError(f"noncanonical archive semantic role: {semantic_role!r}") + path = PurePosixPath(archive_path) + if path.is_absolute() or any(part in {"", ".", ".."} for part in path.parts): + raise ValueError(f"unsafe archive mapping path: {archive_path!r}") + for part in path.parts: + validate_canonical_names([part]) + folded_path = unicodedata.normalize("NFC", archive_path).casefold() + if folded_path in seen_paths or semantic_role in seen_roles: + raise ValueError("archive evidence mapping contains a collision") + seen_paths.add(folded_path) + seen_roles.add(semantic_role) + + +def _safe_source(root: Path, relative: str) -> Path: + candidate = root / relative + try: + metadata = candidate.lstat() + resolved = candidate.resolve(strict=True) + resolved.relative_to(root) + except (OSError, ValueError) as exc: + raise ValueError(f"unsafe or missing evidence path: {relative!r}") from exc + attributes = getattr(metadata, "st_file_attributes", 0) + reparse = getattr(stat, "FILE_ATTRIBUTE_REPARSE_POINT", 0) + if stat.S_ISLNK(metadata.st_mode) or (reparse and attributes & reparse): + raise ValueError(f"evidence must be a regular non-symlink file: {relative!r}") + if not stat.S_ISREG(metadata.st_mode): + raise ValueError(f"evidence must be a regular file: {relative!r}") + return resolved + + +def _verification( + path: Path, + *, + manifest_sha: str, + wheel_sha: str, + sdist_sha: str, + qualification_sha: str, + source_commit: str, + run_id: int, + run_attempt: int, +) -> dict[str, Any]: + try: + payload = _json(path) + except (OSError, UnicodeError, json.JSONDecodeError, ValueError) as exc: + raise ValueError(f"invalid release verification: {exc}") from exc + if not isinstance(payload, dict) or _private(payload): + raise ValueError("release verification failed privacy or object validation") + expected = { + "candidate_manifest_sha256": manifest_sha, + "candidate_wheel_sha256": wheel_sha, + "candidate_sdist_sha256": sdist_sha, + "qualification_sha256": qualification_sha, + "source_commit": source_commit, + "workflow_run_id": run_id, + "workflow_run_attempt": run_attempt, + "valid": True, + } + for key, value in expected.items(): + if payload.get(key) != value: + raise ValueError(f"release verification {key} is not bound to the accepted chain") + return payload + + +def _write_archive(destination: Path, members: list[tuple[str, str, str, Path, str, int]]) -> None: + raw = io.BytesIO() + with ( + gzip.GzipFile(filename="", mode="wb", fileobj=raw, mtime=0, compresslevel=9) as zipped, + tarfile.open(fileobj=zipped, mode="w", format=tarfile.PAX_FORMAT) as archive, + ): + for _, archive_path, _, source, _, size in sorted(members, key=lambda item: item[1]): + info = tarfile.TarInfo(archive_path) + info.size = size + info.mode = 0o644 + info.uid = 0 + info.gid = 0 + info.uname = "" + info.gname = "" + info.mtime = 0 + with source.open("rb") as handle: + archive.addfile(info, handle) + destination.write_bytes(raw.getvalue()) + + +def _copy(source: Path, destination: Path) -> None: + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(source, destination) + + +def prepare_release_assets( + candidate_dir: str | Path, + candidate_manifest: str | Path, + qualification_root: str | Path, + qualification: str | Path, + verification: str | Path, + output: str | Path, +) -> None: + """Validate one accepted full chain and atomically publish its canonical assets.""" + candidate_root = Path(candidate_dir).resolve(strict=True) + manifest_path = Path(candidate_manifest).resolve(strict=True) + evidence_root = Path(qualification_root).resolve(strict=True) + qualification_path = Path(qualification).resolve(strict=True) + verification_path = Path(verification).resolve(strict=True) + target = Path(output) + if target.exists() and (not target.is_dir() or any(target.iterdir())): + raise ValueError("release asset output must initially be empty") + if qualification_path != evidence_root / "qualification.json": + raise ValueError("qualification must be qualification-root/qualification.json") + _validate_archive_mapping() + candidate_problems = validate_candidate_directory(candidate_root, manifest_path=manifest_path) + if candidate_problems: + raise ValueError("candidate validation failed: " + "; ".join(candidate_problems)) + candidate = load_candidate_manifest(manifest_path) + try: + qualification_payload = _json(qualification_path) + record = GPUQualification.model_validate(qualification_payload) + except (OSError, UnicodeError, json.JSONDecodeError, ValueError) as exc: + raise ValueError(f"qualification validation failed: {exc}") from exc + if record.level != "full_qualification": + raise ValueError("qualification level must be full_qualification") + chain_problems = validate_release_chain( + candidate_root, + manifest_path, + qualification_path, + expected_commit=record.source_commit, + expected_known_good_sha256=record.environment.known_good_manifest_sha256, + required_gpu_name=record.environment.gpu_name, + ) + if chain_problems: + raise ValueError("release chain validation failed: " + "; ".join(chain_problems)) + artifact_problems = validate_qualification_payload( + qualification_payload, + artifact_root=evidence_root, + ) + if artifact_problems: + raise ValueError( + "qualification evidence validation failed: " + "; ".join(artifact_problems) + ) + + artifacts = {item.name: item for item in record.artifacts} + expected_roles = set(PRIMARY_EVIDENCE) | set(ARCHIVE_EVIDENCE) | _SPECIAL_EVIDENCE + unknown = sorted(set(artifacts) - expected_roles) + missing = sorted(expected_roles - set(artifacts)) + if unknown: + raise ValueError("unknown evidence role: " + ", ".join(unknown)) + if missing: + raise ValueError("required evidence role is missing: " + ", ".join(missing)) + if len(artifacts) != len(record.artifacts): + raise ValueError("duplicate evidence roles are not allowed") + manifest_evidence = artifacts["candidate-manifest.json"] + manifest_evidence_path = _safe_source(evidence_root, manifest_evidence.path) + if manifest_evidence_path.read_bytes() != manifest_path.read_bytes(): + raise ValueError("qualification candidate-manifest evidence differs from the candidate") + if candidate.workflow.run_id is None or candidate.workflow.run_attempt is None: + raise ValueError("release assets require a GitHub Actions run id and attempt") + + qualification_sha = sha256_file(qualification_path) + verification_payload = _verification( + verification_path, + manifest_sha=sha256_file(manifest_path), + wheel_sha=candidate.wheel.sha256, + sdist_sha=candidate.sdist.sha256, + qualification_sha=qualification_sha, + source_commit=candidate.source_commit, + run_id=candidate.workflow.run_id, + run_attempt=candidate.workflow.run_attempt, + ) + validate_canonical_names(list(_FIXED_FILES)) + + target.parent.mkdir(parents=True, exist_ok=True) + staging = Path(tempfile.mkdtemp(prefix=".release-assets-", dir=target.parent)) + try: + for distribution in (candidate.wheel, candidate.sdist): + _copy( + _safe_source(candidate_root, distribution.filename), + staging / "dist" / distribution.filename, + ) + _copy(candidate_root / "SHA256SUMS", staging / "SHA256SUMS") + _copy(manifest_path, staging / "candidate-manifest.json") + (staging / "release-verification.json").write_bytes(_json_bytes(verification_payload)) + _copy(qualification_path, staging / "qualification.json") + for role, destination in PRIMARY_EVIDENCE.items(): + artifact = artifacts[role] + _copy(_safe_source(evidence_root, artifact.path), staging / destination) + + archive_members: list[tuple[str, str, str, Path, str, int]] = [] + for original_name, (semantic_role, member_path) in ARCHIVE_EVIDENCE.items(): + artifact = artifacts[original_name] + source = _safe_source(evidence_root, artifact.path) + archive_members.append( + ( + semantic_role, + member_path, + original_name, + source, + artifact.sha256, + artifact.bytes, + ) + ) + archive_output = staging / "qualification-evidence.tar.gz" + _write_archive(archive_output, archive_members) + archive_manifest = { + "schema_version": 1, + "kind": "miniverl_qualification_evidence_manifest", + "miniverl_version": candidate.miniverl_version, + "source_commit": candidate.source_commit, + "qualification_sha256": qualification_sha, + "archive": { + "filename": archive_output.name, + "sha256": sha256_file(archive_output), + "bytes": archive_output.stat().st_size, + }, + "members": [ + { + "semantic_role": role, + "archive_path": member_path, + "original_evidence_name": original_name, + "sha256": digest, + "bytes": size, + } + for role, member_path, original_name, _, digest, size in sorted( + archive_members, key=lambda item: item[1] + ) + ], + } + (staging / "qualification-evidence-manifest.json").write_bytes( + _json_bytes(archive_manifest) + ) + (staging / "qualification-SHA256SUMS").write_text( + "".join(f"{sha256_file(staging / name)} {name}\n" for name in QUALIFICATION_SUM_FILES), + encoding="utf-8", + newline="\n", + ) + problems = check_release_assets(staging) + if problems: + raise ValueError("built release assets failed validation: " + "; ".join(problems)) + if target.exists(): + target.rmdir() + staging.replace(target) + finally: + if staging.exists(): + shutil.rmtree(staging) + + +def _checksum_problems( + root: Path, sums_name: str, expected_names: tuple[str, ...] | None +) -> list[str]: + problems: list[str] = [] + try: + lines = (root / sums_name).read_text(encoding="utf-8").splitlines() + except (OSError, UnicodeError) as exc: + return [f"cannot read {sums_name}: {exc}"] + names: list[str] = [] + for line in lines: + match = re.fullmatch(r"([0-9a-f]{64}) ([^/\\]+)", line) + if not match: + problems.append(f"invalid {sums_name} line: {line!r}") + continue + digest, name = match.groups() + names.append(name) + path = root / name + if not path.is_file() or path.is_symlink(): + problems.append(f"{sums_name} references missing or unsafe file {name}") + elif sha256_file(path) != digest: + problems.append(f"{sums_name} checksum mismatch for {name}") + if expected_names is not None and names != list(expected_names): + problems.append(f"{sums_name} does not contain the canonical ordered file set") + return problems + + +def check_release_assets(output: str | Path) -> list[str]: + """Purely validate an already-built canonical asset directory.""" + root = Path(output) + if not root.is_dir() or root.is_symlink(): + return ["release asset output is not a regular directory"] + actual_files = { + path.relative_to(root).as_posix() + for path in root.rglob("*") + if path.is_file() or path.is_symlink() + } + actual_directories = { + path.relative_to(root).as_posix() for path in root.rglob("*") if path.is_dir() + } + dist = root / "dist" + dist_names = sorted(path.name for path in dist.iterdir()) if dist.is_dir() else [] + expected_files = _FIXED_FILES | {f"dist/{name}" for name in dist_names} + problems: list[str] = [] + try: + _validate_archive_mapping() + except ValueError as exc: + problems.append(str(exc)) + if actual_directories != {"dist"}: + problems.append( + "unexpected release asset directory set: " + + ", ".join(sorted(actual_directories ^ {"dist"})) + ) + unsafe_files = sorted( + path.relative_to(root).as_posix() for path in root.rglob("*") if path.is_symlink() + ) + if unsafe_files: + problems.append("release assets must not contain symlinks: " + ", ".join(unsafe_files)) + if actual_files != expected_files: + problems.append( + "unexpected release asset file set: " + ", ".join(sorted(actual_files ^ expected_files)) + ) + try: + validate_canonical_names([*(_FIXED_FILES), *dist_names]) + except ValueError as exc: + problems.append(str(exc)) + if ( + len(dist_names) != 2 + or sum(name.endswith(".whl") for name in dist_names) != 1 + or sum(name.endswith(".tar.gz") for name in dist_names) != 1 + ): + problems.append("dist must contain exactly one wheel and one sdist") + problems.extend(_checksum_problems(root, "qualification-SHA256SUMS", QUALIFICATION_SUM_FILES)) + # Distribution sums live one directory above dist and intentionally name only distributions. + if dist.is_dir(): + try: + sums = (root / "SHA256SUMS").read_text(encoding="utf-8").splitlines() + declared = [] + for line in sums: + digest, name = line.split(" ", 1) + declared.append(name) + if not re.fullmatch(r"[0-9a-f]{64}", digest) or sha256_file(dist / name) != digest: + problems.append(f"SHA256SUMS checksum mismatch for {name}") + if sorted(declared) != dist_names: + problems.append("SHA256SUMS does not cover the exact dist file set") + except (OSError, ValueError, IndexError) as exc: + problems.append(f"invalid SHA256SUMS: {exc}") + + qualification_record: GPUQualification | None = None + candidate_record = None + qualification_artifacts: dict[str, Any] = {} + try: + candidate_record = load_candidate_manifest(root / "candidate-manifest.json") + qualification_payload = _json(root / "qualification.json") + qualification_record = GPUQualification.model_validate(qualification_payload) + if qualification_record.level != "full_qualification": + problems.append("qualification level is not full_qualification") + qualification_artifacts = { + artifact.name: artifact for artifact in qualification_record.artifacts + } + expected_roles = set(PRIMARY_EVIDENCE) | set(ARCHIVE_EVIDENCE) | _SPECIAL_EVIDENCE + if set(qualification_artifacts) != expected_roles or len(qualification_artifacts) != len( + qualification_record.artifacts + ): + problems.append("qualification evidence roles are not the canonical exact set") + problems.extend( + validate_qualification_payload( + qualification_payload, + expected_commit=candidate_record.source_commit, + expected_wheel_sha256=candidate_record.wheel.sha256, + expected_candidate_manifest_sha256=sha256_file(root / "candidate-manifest.json"), + expected_known_good_sha256=( + qualification_record.environment.known_good_manifest_sha256 + ), + required_gpu_name=qualification_record.environment.gpu_name, + ) + ) + bindings = ( + ( + candidate_record.miniverl_version, + qualification_record.miniverl_version, + "version", + ), + ( + candidate_record.workflow.run_id, + qualification_record.candidate.workflow_run_id, + "run id", + ), + ( + candidate_record.workflow.run_attempt, + qualification_record.candidate.workflow_run_attempt, + "run attempt", + ), + ) + for candidate_value, qualification_value, label in bindings: + if candidate_value != qualification_value: + problems.append(f"candidate and qualification {label} binding differs") + expected_dist = {candidate_record.wheel.filename, candidate_record.sdist.filename} + if set(dist_names) != expected_dist: + problems.append("dist filenames do not match the candidate manifest") + for distribution in (candidate_record.wheel, candidate_record.sdist): + distribution_path = dist / distribution.filename + if distribution_path.is_file() and ( + distribution_path.stat().st_size != distribution.bytes + or sha256_file(distribution_path) != distribution.sha256 + ): + problems.append( + f"candidate distribution byte identity mismatch: {distribution.filename}" + ) + candidate_evidence = qualification_artifacts.get("candidate-manifest.json") + if candidate_evidence is None or ( + candidate_evidence.bytes != (root / "candidate-manifest.json").stat().st_size + or candidate_evidence.sha256 != sha256_file(root / "candidate-manifest.json") + ): + problems.append("qualification evidence does not bind candidate-manifest.json") + for role, destination in PRIMARY_EVIDENCE.items(): + artifact = qualification_artifacts.get(role) + destination_path = root / destination + if artifact is None or ( + destination_path.stat().st_size != artifact.bytes + or sha256_file(destination_path) != artifact.sha256 + ): + problems.append(f"qualification evidence byte identity mismatch: {role}") + if ( + candidate_record.workflow.run_id is None + or candidate_record.workflow.run_attempt is None + ): + problems.append("candidate has no GitHub Actions run identity") + else: + _verification( + root / "release-verification.json", + manifest_sha=sha256_file(root / "candidate-manifest.json"), + wheel_sha=candidate_record.wheel.sha256, + sdist_sha=candidate_record.sdist.sha256, + qualification_sha=sha256_file(root / "qualification.json"), + source_commit=candidate_record.source_commit, + run_id=candidate_record.workflow.run_id, + run_attempt=candidate_record.workflow.run_attempt, + ) + except (OSError, UnicodeError, json.JSONDecodeError, ValueError, KeyError) as exc: + problems.append(f"invalid candidate, qualification or verification binding: {exc}") + + archive_path = root / "qualification-evidence.tar.gz" + try: + manifest = _json(root / "qualification-evidence-manifest.json") + if _private(manifest): + problems.append("archive manifest contains private data") + declared_archive = manifest["archive"] + if declared_archive["filename"] != archive_path.name: + problems.append("archive manifest filename mismatch") + if declared_archive["sha256"] != sha256_file(archive_path): + problems.append("archive manifest checksum mismatch") + if declared_archive["bytes"] != archive_path.stat().st_size: + problems.append("archive manifest size mismatch") + if qualification_record is not None and candidate_record is not None: + if ( + manifest.get("schema_version") != 1 + or manifest.get("kind") != "miniverl_qualification_evidence_manifest" + or manifest.get("miniverl_version") != candidate_record.miniverl_version + or manifest.get("source_commit") != candidate_record.source_commit + or manifest.get("qualification_sha256") != sha256_file(root / "qualification.json") + ): + problems.append("archive manifest release-chain binding mismatch") + expected_member_rows = [] + for original_name, (semantic_role, member_path) in ARCHIVE_EVIDENCE.items(): + artifact = qualification_artifacts.get(original_name) + if artifact is not None: + expected_member_rows.append( + { + "semantic_role": semantic_role, + "archive_path": member_path, + "original_evidence_name": original_name, + "sha256": artifact.sha256, + "bytes": artifact.bytes, + } + ) + expected_member_rows.sort(key=lambda item: item["archive_path"]) + if manifest.get("members") != expected_member_rows: + problems.append("archive manifest semantic mapping does not match qualification") + declared_members = {item["archive_path"]: item for item in manifest["members"]} + if len(declared_members) != len(manifest["members"]): + problems.append("duplicate archive manifest member") + names: set[str] = set() + with tarfile.open(archive_path, "r:gz") as archive: + for member in archive.getmembers(): + tar_member_path = PurePosixPath(member.name) + if tar_member_path.is_absolute() or ".." in tar_member_path.parts: + problems.append(f"unsafe archive member: {member.name}") + if member.name in names: + problems.append(f"duplicate archive member: {member.name}") + names.add(member.name) + if not member.isfile(): + problems.append(f"non-regular archive member: {member.name}") + if ( + any((member.uid, member.gid, member.mtime)) + or member.uname + or member.gname + or member.mode != 0o644 + ): + problems.append(f"nondeterministic archive metadata: {member.name}") + extracted = archive.extractfile(member) + data = extracted.read() if extracted is not None else b"" + member_declaration = declared_members.get(member.name) + if member_declaration is None: + problems.append(f"undeclared archive member: {member.name}") + elif ( + member_declaration["bytes"] != len(data) + or member_declaration["sha256"] != hashlib.sha256(data).hexdigest() + ): + problems.append(f"archive member byte identity mismatch: {member.name}") + if names != set(declared_members): + problems.append("archive member set does not match its manifest") + except ( + OSError, + KeyError, + TypeError, + ValueError, + tarfile.TarError, + json.JSONDecodeError, + ) as exc: + problems.append(f"invalid evidence archive or manifest: {exc}") + return problems diff --git a/tests/unit/test_release_assets.py b/tests/unit/test_release_assets.py new file mode 100644 index 0000000..75c661a --- /dev/null +++ b/tests/unit/test_release_assets.py @@ -0,0 +1,478 @@ +"""Canonical, deterministic and fail-closed future release assets.""" + +from __future__ import annotations + +import hashlib +import io +import json +import subprocess +import sys +import tarfile +from pathlib import Path + +import pytest + + +def _sha(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _fixture(tmp_path: Path) -> tuple[Path, Path, Path, Path]: + from miniverl.release_candidate import PINNED_BUILD_TOOLS + + candidate = tmp_path / "candidate" + qualification = tmp_path / "qualification" + candidate.mkdir(parents=True) + qualification.mkdir(parents=True) + version = "0.10.2.dev0" + commit = "a" * 40 + wheel = candidate / f"miniverl-{version}-py3-none-any.whl" + sdist = candidate / f"miniverl-{version}.tar.gz" + wheel.write_bytes(b"qualified wheel") + sdist.write_bytes(b"qualified sdist") + manifest = { + "schema_version": 1, + "kind": "miniverl_release_candidate", + "source_commit": commit, + "miniverl_version": version, + "created_at": "2026-08-16T12:00:00Z", + "artifact_name": "candidate-distributions", + "workflow": { + "kind": "github_actions", + "repository": "DaoyuanLi2816/mini-verl", + "workflow_path": ".github/workflows/gpu.yml", + "run_id": 42, + "run_attempt": 1, + }, + "build": {"os": "Linux", "python": "3.12.0", "tools": PINNED_BUILD_TOOLS}, + "wheel": {"filename": wheel.name, "bytes": wheel.stat().st_size, "sha256": _sha(wheel)}, + "sdist": {"filename": sdist.name, "bytes": sdist.stat().st_size, "sha256": _sha(sdist)}, + } + manifest_path = candidate / "candidate-manifest.json" + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + (candidate / "SHA256SUMS").write_text( + f"{_sha(wheel)} {wheel.name}\n{_sha(sdist)} {sdist.name}\n", encoding="utf-8" + ) + + evidence = { + "run_summary": ("run-summary.json", b'{"status":"completed"}\n'), + "inputs_prompts.parquet": ("inputs/prompts.parquet", b"PAR1 prompts"), + "adapter_adapter_model.safetensors": ( + "adapter/adapter_model.safetensors", + b"safe tensors", + ), + "adapter_adapter_config.json": ("adapter/adapter_config.json", b'{"r":8}\n'), + "adapter_miniverl_adapter_manifest.json": ( + "adapter/miniverl_adapter_manifest.json", + b'{"schema_version":1}\n', + ), + "candidate-manifest.json": ("candidate-manifest.json", manifest_path.read_bytes()), + "release_smoke_record": ("full/release-smoke.json", b'{"kind":"smoke"}\n'), + "full_direct_result": ("full/direct.json", b'{"kind":"direct"}\n'), + "full_pg_k1_result": ("full/pg_k1.json", b'{"kind":"pg-k1"}\n'), + "full_smollm2_result": ("full/smollm2.json", b'{"kind":"smollm2"}\n'), + } + artifacts = [] + for name, (relative, content) in evidence.items(): + path = qualification / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(content) + artifacts.append( + {"name": name, "path": relative, "sha256": _sha(path), "bytes": len(content)} + ) + qualification_wheel = qualification / wheel.name + qualification_wheel.write_bytes(wheel.read_bytes()) + payload = { + "schema_version": 1, + "kind": "miniverl_gpu_qualification", + "level": "full_qualification", + "status": "passed", + "measured_at": "2026-08-16T12:01:00Z", + "source_commit": commit, + "miniverl_version": version, + "wheel": {"filename": wheel.name, "sha256": _sha(wheel)}, + "candidate": { + "manifest_sha256": _sha(manifest_path), + "artifact_name": "candidate-distributions", + "workflow_repository": "DaoyuanLi2816/mini-verl", + "workflow_path": ".github/workflows/gpu.yml", + "workflow_run_id": 42, + "workflow_run_attempt": 1, + "installed_from_candidate": True, + "import_origin_verified": True, + "cli_origin_verified": True, + "import_origin": "qualification_venv_site_packages", + }, + "profile": { + "name": "verl-opd-v0.8-single-gpu-v1", + "identity_digest": "b" * 64, + "upstream_tag": "v0.8.0", + "upstream_commit": "c" * 40, + }, + "environment": { + "known_good_manifest_sha256": "d" * 64, + "gpu_name": "NVIDIA GeForce RTX 4080", + "gpu_count": 1, + "vram_gib": 15.99, + "driver": "596.49", + "cuda_runtime": "13.0", + "python": "3.10.11", + "packages": dict.fromkeys( + ( + "torch", + "transformers", + "peft", + "accelerate", + "bitsandbytes", + "numpy", + "pyarrow", + "safetensors", + ), + "1.0", + ), + }, + "models": [ + {"role": "actor", "model_id": "org/actor", "revision": "e" * 40}, + {"role": "teacher", "model_id": "org/teacher", "revision": "f" * 40}, + ], + "execution": { + "rollout_completed": True, + "teacher_scoring_completed": True, + "optimizer_updates": 1, + "peft_adapter_exported": True, + "peft_adapter_reload_verified": True, + "cuda_allocated_before_bytes": 0, + "cuda_allocated_after_teardown_bytes": 0, + "cuda_teardown_tolerance_bytes": 2097152, + }, + "inputs": { + "config_sha256": "1" * 64, + "plan_sha256": "2" * 64, + "parquet_sha256": "3" * 64, + }, + "artifacts": artifacts, + "checks": { + "executed": [ + "wheel_install", + "doctor", + "plan", + "run_dry_run", + "real_actor_teacher_update", + "peft_reload", + "cuda_teardown", + "direct_gkd_resume_equivalence", + "sampled_k1_resume_equivalence", + "smollm2_resume_equivalence", + "export_materialize_doctor", + ], + "skipped": [], + "not_applicable": ["distributed_verl_execution"], + }, + "scientific_scope": { + "runtime_correctness_only": True, + "task_quality_evaluated": False, + "alignment_quality_evaluated": False, + "distributed_execution_tested": False, + "other_hardware_measured": False, + }, + } + qualification_path = qualification / "qualification.json" + qualification_path.write_text(json.dumps(payload), encoding="utf-8") + verification = tmp_path / "verification.json" + verification.write_text( + json.dumps( + { + "candidate_manifest_sha256": _sha(manifest_path), + "candidate_wheel_sha256": _sha(wheel), + "candidate_sdist_sha256": _sha(sdist), + "qualification_sha256": _sha(qualification_path), + "repository": "DaoyuanLi2816/mini-verl", + "source_commit": commit, + "valid": True, + "workflow_run_attempt": 1, + "workflow_run_id": 42, + } + ), + encoding="utf-8", + ) + return candidate, manifest_path, qualification, verification + + +def _build(tmp_path: Path, *, name: str = "out") -> Path: + from miniverl.release_assets import prepare_release_assets + + candidate, manifest, qualification, verification = _fixture(tmp_path) + output = tmp_path / name + prepare_release_assets( + candidate_dir=candidate, + candidate_manifest=manifest, + qualification_root=qualification, + qualification=qualification / "qualification.json", + verification=verification, + output=output, + ) + return output + + +def test_builds_exact_canonical_layout_and_checks_it(tmp_path: Path) -> None: + from miniverl.release_assets import QUALIFICATION_SUM_FILES, check_release_assets + + output = _build(tmp_path) + expected = { + "dist/miniverl-0.10.2.dev0-py3-none-any.whl", + "dist/miniverl-0.10.2.dev0.tar.gz", + "SHA256SUMS", + "candidate-manifest.json", + "release-verification.json", + "qualification.json", + "qualification-SHA256SUMS", + "qualification-release-smoke.json", + "qualification-direct-gkd.json", + "qualification-pg-k1.json", + "qualification-smollm2.json", + "qualification-evidence.tar.gz", + "qualification-evidence-manifest.json", + } + assert { + path.relative_to(output).as_posix() for path in output.rglob("*") if path.is_file() + } == expected + assert check_release_assets(output) == [] + sums = (output / "qualification-SHA256SUMS").read_text(encoding="utf-8").splitlines() + assert [line.split(" ", 1)[1] for line in sums] == list(QUALIFICATION_SUM_FILES) + assert all( + not path.name.endswith((".json.json", ".parquet.parquet", ".safetensors.safetensors")) + for path in output.rglob("*") + ) + + +def test_archive_and_manifest_are_reproducible_and_explicit(tmp_path: Path) -> None: + first = _build(tmp_path / "first") + second = _build(tmp_path / "second") + assert (first / "qualification-evidence.tar.gz").read_bytes() == ( + second / "qualification-evidence.tar.gz" + ).read_bytes() + assert (first / "qualification-evidence-manifest.json").read_bytes() == ( + second / "qualification-evidence-manifest.json" + ).read_bytes() + manifest = json.loads((first / "qualification-evidence-manifest.json").read_text()) + assert [member["semantic_role"] for member in manifest["members"]] == [ + "adapter_config", + "adapter_weights", + "adapter_manifest", + "input_prompts", + "run_summary", + ] + with tarfile.open(first / "qualification-evidence.tar.gz", "r:gz") as archive: + members = archive.getmembers() + assert [member.name for member in members] == sorted(member.name for member in members) + assert all( + member.isfile() and member.uid == member.gid == member.mtime == 0 for member in members + ) + + +@pytest.mark.parametrize( + "names", + [ + ["result.json", "RESULT.JSON"], + ["result.json.json"], + ["bad/name.json"], + ["..tar.gz"], + ["CON.json"], + ["caf\u00e9.json", "cafe\u0301.json"], + ], +) +def test_filename_hygiene_rejects_collisions_and_noncanonical_names(names: list[str]) -> None: + from miniverl.release_assets import validate_canonical_names + + with pytest.raises(ValueError): + validate_canonical_names(names) + + +def test_fails_closed_for_unknown_role_and_evidence_drift(tmp_path: Path) -> None: + from miniverl.release_assets import prepare_release_assets + + candidate, manifest, qualification, verification = _fixture(tmp_path) + qualification_path = qualification / "qualification.json" + payload = json.loads(qualification_path.read_text()) + payload["artifacts"][0]["name"] = "unknown_role" + qualification_path.write_text(json.dumps(payload), encoding="utf-8") + with pytest.raises(ValueError, match="unknown evidence role"): + prepare_release_assets( + candidate, manifest, qualification, qualification_path, verification, tmp_path / "out" + ) + candidate, manifest, qualification, verification = _fixture(tmp_path / "drift") + (qualification / "run-summary.json").write_bytes(b"changed") + with pytest.raises(ValueError, match=r"size mismatch|checksum mismatch"): + prepare_release_assets( + candidate, + manifest, + qualification, + qualification / "qualification.json", + verification, + tmp_path / "drift-out", + ) + + +@pytest.mark.parametrize( + ("mutation", "match"), + [ + ( + lambda payload: payload["artifacts"][0].update(path="../escape.json"), + r"qualification validation|release chain validation", + ), + ( + lambda payload: payload["environment"].update(vram_gib=float("nan")), + r"qualification validation", + ), + ( + lambda payload: payload["artifacts"][0].update(sha256="0" * 64), + r"checksum mismatch|release chain validation", + ), + ( + lambda payload: payload["artifacts"][0].update(bytes=999999), + r"size mismatch|release chain validation", + ), + ], +) +def test_fails_closed_for_traversal_nonfinite_and_declared_identity( + tmp_path: Path, mutation, match: str +) -> None: + from miniverl.release_assets import prepare_release_assets + + candidate, manifest, qualification, verification = _fixture(tmp_path) + qualification_path = qualification / "qualification.json" + payload = json.loads(qualification_path.read_text()) + mutation(payload) + qualification_path.write_text(json.dumps(payload), encoding="utf-8") + with pytest.raises(ValueError, match=match): + prepare_release_assets( + candidate, + manifest, + qualification, + qualification_path, + verification, + tmp_path / "out", + ) + + +def test_archive_mapping_collisions_fail_closed( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + from miniverl import release_assets + + candidate, manifest, qualification, verification = _fixture(tmp_path) + monkeypatch.setitem( + release_assets.ARCHIVE_EVIDENCE, + "run_summary", + ("run_summary", "ADAPTER/adapter_config.json"), + ) + with pytest.raises(ValueError, match="collision"): + release_assets.prepare_release_assets( + candidate, + manifest, + qualification, + qualification / "qualification.json", + verification, + tmp_path / "out", + ) + + +def test_rejects_symlink_private_verification_and_nonempty_output(tmp_path: Path) -> None: + from miniverl.release_assets import prepare_release_assets + + candidate, manifest, qualification, verification = _fixture(tmp_path) + output = tmp_path / "out" + output.mkdir() + (output / "extra").write_text("unexpected") + with pytest.raises(ValueError, match="empty"): + prepare_release_assets( + candidate, + manifest, + qualification, + qualification / "qualification.json", + verification, + output, + ) + + verification.write_text('{"path":"C:\\\\Users\\\\secret\\\\file"}', encoding="utf-8") + with pytest.raises(ValueError, match=r"privacy|verification"): + prepare_release_assets( + candidate, + manifest, + qualification, + qualification / "qualification.json", + verification, + tmp_path / "private", + ) + + target = qualification / "run-summary.json" + target.unlink() + try: + target.symlink_to(qualification / "full/direct.json") + except OSError: + pytest.skip("symlinks are unavailable") + with pytest.raises(ValueError, match=r"regular|checksum|size"): + prepare_release_assets( + candidate, + manifest, + qualification, + qualification / "qualification.json", + verification, + tmp_path / "linked", + ) + + +def test_check_rejects_extra_and_malicious_archive_members(tmp_path: Path) -> None: + from miniverl.release_assets import check_release_assets + + output = _build(tmp_path) + (output / "extra.json").write_text("{}") + assert any("unexpected" in problem for problem in check_release_assets(output)) + (output / "extra.json").unlink() + stream = io.BytesIO() + with tarfile.open(fileobj=stream, mode="w:gz") as archive: + for name in ("../escape", "duplicate", "duplicate"): + info = tarfile.TarInfo(name) + info.size = 1 + archive.addfile(info, io.BytesIO(b"x")) + (output / "qualification-evidence.tar.gz").write_bytes(stream.getvalue()) + problems = check_release_assets(output) + assert any("unsafe archive member" in problem for problem in problems) + assert any("duplicate archive member" in problem for problem in problems) + + +def test_check_rebinds_primary_and_archive_members_to_qualification(tmp_path: Path) -> None: + from miniverl.release_assets import QUALIFICATION_SUM_FILES, check_release_assets + + output = _build(tmp_path) + primary = output / "qualification-direct-gkd.json" + primary.write_text('{"substituted":true}\n', encoding="utf-8") + (output / "qualification-SHA256SUMS").write_text( + "".join(f"{_sha(output / name)} {name}\n" for name in QUALIFICATION_SUM_FILES), + encoding="utf-8", + ) + assert any("qualification evidence" in problem for problem in check_release_assets(output)) + + output = _build(tmp_path / "archive") + manifest_path = output / "qualification-evidence-manifest.json" + manifest = json.loads(manifest_path.read_text()) + manifest["members"][0]["semantic_role"] = "unknown_role" + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + (output / "qualification-SHA256SUMS").write_text( + "".join(f"{_sha(output / name)} {name}\n" for name in QUALIFICATION_SUM_FILES), + encoding="utf-8", + ) + assert any("semantic mapping" in problem for problem in check_release_assets(output)) + + +def test_check_rejects_extra_directories(tmp_path: Path) -> None: + from miniverl.release_assets import check_release_assets + + output = _build(tmp_path) + (output / "unexpected-empty-directory").mkdir() + assert any("directory set" in problem for problem in check_release_assets(output)) + + +def test_module_import_is_torch_free() -> None: + code = "import sys, miniverl.release_assets; assert 'torch' not in sys.modules" + subprocess.run([sys.executable, "-c", code], check=True) diff --git a/tests/unit/test_release_workflow.py b/tests/unit/test_release_workflow.py index 101ffb4..283c490 100644 --- a/tests/unit/test_release_workflow.py +++ b/tests/unit/test_release_workflow.py @@ -55,10 +55,12 @@ def test_release_consumes_qualified_candidate_without_rebuild() -> None: verify = _job(text, "verify-pypi", "create-github-release") release = _job(text, "create-github-release", None) - assert "copy the identical candidate bytes" in prepare + assert "scripts/prepare_release_assets.py" in prepare + assert "--check" in prepare + assert 'item["name"].replace("_", "-")' not in prepare assert "candidate-manifest.json" in prepare assert "qualification.json" in prepare - assert "qualification-full-SHA256SUMS" in prepare + assert "qualification-SHA256SUMS" in prepare assert "actions/upload-artifact@" in prepare assert "actions/download-artifact@" in publish assert "python -m build" not in publish @@ -142,3 +144,19 @@ def test_release_recovery_preserves_full_qualification_evidence() -> None: assert "release-artifacts/dist" in text assert text.count("qualification-full-SHA256SUMS") >= 3 assert "release-artifacts/qualification-evidence/*" in text + + +def test_github_release_uses_only_explicit_canonical_assets() -> None: + release = _job(_workflow_text(), "create-github-release", None) + assert "release-artifacts/dist/*" not in release + assert "release-artifacts/qualification-evidence/*" not in release + for name in ( + "qualification-release-smoke.json", + "qualification-direct-gkd.json", + "qualification-pg-k1.json", + "qualification-smollm2.json", + "qualification-evidence.tar.gz", + "qualification-evidence-manifest.json", + "release-verification.json", + ): + assert f"release-artifacts/{name}" in release From 57e14fa38f1960c02021caacb5416d4e2cca447c Mon Sep 17 00:00:00 2001 From: Daoyuan Li <94409450+DaoyuanLi2816@users.noreply.github.com> Date: Sun, 16 Aug 2026 11:36:53 -0700 Subject: [PATCH 3/3] Document release asset hygiene --- CHANGELOG.md | 12 ++++++++ PROJECT_STATE.md | 9 ++++-- docs/release-checklist.md | 11 ++++++- docs/release-qualification.md | 54 +++++++++++++++++++++++++++++------ docs/troubleshooting.md | 25 +++++++++++++++- 5 files changed, 98 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d952610..9481bdc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,18 @@ All notable changes to miniVERL are recorded here. The format follows ## [Unreleased] +### Release evidence polish + +- Corrected the v1-readiness record after the completed v0.10.1 attempt-1 RTX + 4080 qualification, dry-run and publication while keeping the remaining v1 + stability gates explicitly unsatisfied. +- Added a torch-free canonical future release-asset builder with explicit + principal-result mapping, deterministic subordinate-evidence archive, + separate checksum scopes and fail-closed filename/path validation. +- Replaced future wildcard evidence uploads and inline filename flattening with + one validated, explicit GitHub Release asset list. Existing v0.10.1 assets and + all published bytes remain immutable. + ## [0.10.1] - 2026-08-16 ### Product maturity and release qualification diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index 1cab04b..bbbe23c 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -37,9 +37,12 @@ exports bind an independent identity. The direct-GKD and sampled-k1 vanilla policy-loss profiles both have pinned conformance and measured RTX 4080 paths. Portable hardware records preserve measured, estimated and unknown states; schema-valid community submissions remain unreviewed until maintainer validation. -The development release chain now builds one hosted-runner candidate and binds -the private RTX 4080 qualification to that exact manifest and wheel. Release -jobs may publish only the same verified bytes and never rebuild them. +The release chain builds one hosted-runner candidate and binds the maintainer's +RTX 4080 qualification to that exact manifest and wheel. Release jobs may +publish only the same verified bytes and never rebuild them. Future GitHub +Releases use a canonical, explicit evidence layout with four principal JSON +records and one deterministic subordinate-evidence archive; historical v0.10.1 +asset names remain immutable. Arbitrary verl YAML, other policy-gradient modes, rewards, PPO/GRPO, Ray, FSDP, Megatron, multi-GPU and distributed execution remain unsupported. The legacy diff --git a/docs/release-checklist.md b/docs/release-checklist.md index 9dfde56..fa01131 100644 --- a/docs/release-checklist.md +++ b/docs/release-checklist.md @@ -8,6 +8,13 @@ after the exact release commit and its remote checks are green. - [ ] Define and review the next focused release scope before changing product behavior or scientific evidence. +- [x] Correct v1-readiness prose to record the completed v0.10.1 same-run + attempt-1 qualification while retaining explicit unsatisfied v1 gates. +- [x] Prepare future GitHub Release evidence with the torch-free canonical + builder: four principal JSON files, one deterministic subordinate archive, + one archive manifest and separate distribution/qualification checksums. +- [x] Require the future publisher to enumerate every release asset explicitly; + no raw qualification directory or wildcard evidence upload is accepted. ## v0.10.1 release @@ -29,7 +36,9 @@ after the exact release commit and its remote checks are green. `candidate-distributions` and `gpu-full-qualification`, preserve the complete evidence set and perform no distribution rebuild. - [x] Require release assets to contain versioned full qualification records - and `qualification-full-SHA256SUMS`; release smoke alone is not accepted. + and the historical `qualification-full-SHA256SUMS`; release smoke alone + is not accepted. These v0.10.1 names remain immutable, while future + releases use the canonical layout documented above. ### Maintainer self-review for v0.10.1 diff --git a/docs/release-qualification.md b/docs/release-qualification.md index 70ff714..2b1d07c 100644 --- a/docs/release-qualification.md +++ b/docs/release-qualification.md @@ -52,12 +52,47 @@ artifact digest when provided, wheel byte hash and qualification bindings. It also checks each declared evidence file's regular-file type, byte count and SHA-256. Cross-origin artifact redirects retain ordinary API headers but strip authorization, proxy authorization and cookies. It publishes the accepted -wheel and sdist without rebuilding them. The workflow and GitHub Release retain -the full qualification record, declared evidence files and a separate checksum -inventory under versioned names. A committed +wheel and sdist without rebuilding them. Future release runs retain the full +qualification record, four principal workload JSON files and a deterministic +subordinate-evidence archive in the canonical layout below. A committed JSON file, manual upload, fork run, different workflow or cross-run artifact pair cannot satisfy this gate. +## Canonical future Release assets + +The release-asset builder uses an explicit evidence-role mapping. It never +derives public filenames from internal paths or appends a guessed extension. +The top level is exactly: + +```text +dist/ +SHA256SUMS +candidate-manifest.json +release-verification.json +qualification.json +qualification-SHA256SUMS +qualification-release-smoke.json +qualification-direct-gkd.json +qualification-pg-k1.json +qualification-smollm2.json +qualification-evidence.tar.gz +qualification-evidence-manifest.json +``` + +The four principal workload records remain directly inspectable. Adapter +configuration and safetensors, its miniVERL manifest, input prompts and the run +summary live once in the deterministic archive; its manifest binds every member +to its semantic role, byte count and SHA-256. `SHA256SUMS` covers only the wheel +and sdist. `qualification-SHA256SUMS` covers the nine provenance and evidence +assets in canonical order. Hashes prove byte integrity, not code signing or +third-party endorsement. + +The historical v0.10.1 assets are immutable and retain the names emitted by the +original flattening step, including duplicated suffixes on some subordinate +files. They are not rewritten or re-uploaded. The canonical layout applies to +future releases only and does not prove distributed verl execution or hardware +beyond the one measured RTX 4080. + Local validation is torch-free: ```bash @@ -92,9 +127,12 @@ portable bounded artifacts, and checks cleanup targets remain under `GITHUB_WORKSPACE`. Model caches remain runner-local and are not uploaded. Rotate the runner token after suspected exposure. -## Activation state +## Measured release state -Repository code, schema, validator and workflows can be complete while runner -registration remains incomplete. Until the first successful exact-commit run -exists, report `blocked_external_setup`; do not write “exact-commit -qualification passed” or “continuous GPU CI”. +The first exact-commit full run completed for v0.10.1 on attempt 1: +[GPU qualification 31932226695](https://github.com/DaoyuanLi2816/mini-verl/actions/runs/31932226695), +[dry-run 31933844796](https://github.com/DaoyuanLi2816/mini-verl/actions/runs/31933844796) +and [tag publication 31934196365](https://github.com/DaoyuanLi2816/mini-verl/actions/runs/31934196365). +Future release commits still require their own new exact-SHA, same-run, attempt-1 +full qualification; the v0.10.1 record cannot authorize them. This remains a +manual maintainer process, not continuous GPU CI. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 50b3868..0e0669e 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -39,6 +39,7 @@ Contents: - [Resume refused: config digest mismatch](#resume-refused-config-digest-mismatch) - [Known-good environment mismatch](#known-good-environment-mismatch) - [Release qualification not found](#release-qualification-not-found) +- [Release asset layout check failed](#release-asset-layout-check-failed) ## Known-good environment mismatch @@ -70,18 +71,40 @@ no successful gpu.yml workflow_dispatch run exists for whose `head_sha` is the exact release commit. A successful run for a parent, another branch or a rebuilt local JSON is intentionally insufficient. -**Fix.** Register the private runner with labels `[self-hosted, cuda, +**Fix.** For a future release commit, register the private runner with labels `[self-hosted, cuda, rtx4080]`, review the candidate SHA, dispatch `gpu.yml` at that SHA, and wait for both `candidate-distributions` and `gpu-full-qualification` from that one run. Validate the full record with `miniverl qualification validate`. `gpu-release-smoke` is diagnostic only. Until the full record exists, the correct status is `blocked_external_setup`, not passed GPU CI. +The v0.10.1 release already completed this process in attempt-1 run +`31932226695`; that evidence is immutable and cannot qualify a later commit. + If the run fails, do not use **Re-run jobs**. Qualification requires `GITHUB_RUN_ATTEMPT=1` to prevent cross-attempt artifact reuse. Fix the cause, make a fresh ephemeral runner available, then create a new `workflow_dispatch` run for the same reviewed SHA. +## Release asset layout check failed + +**Symptom** + +```text +unexpected release asset file set +``` + +**Cause.** A future release output contains an extra file, a missing principal +record, a checksum mismatch, an unsafe archive member or a noncanonical name. +The builder deliberately rejects guessed extensions, case-insensitive filename +collisions, symlinks and undeclared evidence. + +**Fix.** Do not rename or flatten evidence manually. Re-run +`scripts/prepare_release_assets.py` from the accepted candidate, qualification +root and verification record, then run the same command with `--check`. Keep +PyPI publication restricted to `dist/` and enumerate the GitHub Release assets +explicitly. Historical v0.10.1 filenames are preserved as-is. + ## Missing extras **Symptom**