The directory contains a verl v0.9-shaped single-GPU RL config plus two runnable, CPU-only Python extension examples.
A resolved GRPO source config for the versioned compatibility compiler. It
keeps upstream data, actor, rollout, algorithm and trainer fields recognizable,
and places local reward, revision and physical batching choices under
miniverl.*.
miniverl data sample --task-rewards --rows 8 --out data/rl-prompts.parquet
miniverl import-verl --profile verl-rl-v0.9-single-gpu-v1 \
--config examples/verl-rl-v0.9-single-gpu.yaml --out local-grpo.yaml
miniverl train local-grpo.yaml --dry-runThe generated report records how the example's eight-GPU/two-node source resource intent lowers to one local process/device. The logical GRPO group and objective remain unchanged.
Both scripts are network-free and execute as part of the release check.
pip install "miniverl[train]" # CPU torch is enough
python examples/custom_environment/reverse_environment.py runs/examples
python examples/custom_teacher/sharpened_teacher.py runs/examplesA complete new ToolEnvironment: two tools, three difficulties, a deterministic
oracle, an exact verifier and a privileged context string for the teacher. It
registers itself with @register, verifies that its own oracle solves 8/8 hard
tasks, and then trains a toy student on it with genuine on-policy distillation.
The important line in its output is the oracle check:
registered environments: ['calculator', 'jsonnav', 'reverse', 'sqlite']
oracle solves 8/8 hard tasks
The final task-success number will usually be low or zero. That is expected: the
toy models are around 200k parameters and the budget is a smoke test. The example
demonstrates the extension point, not a capability. See
docs/limitations.md.
Copy this file as the starting point for your own environment. The checklist for a correct one is in the "New environment" issue template.
A custom TeacherScorer that wraps the built-in local scorer and sharpens its
top-k distribution before handing it to the loss. It shows the whole contract:
score the student's own states, return a provider the chunked loss can slice, and
keep the returned targets a normalized distribution.
It asserts what it claims:
teacher: sharpened sharpness 2.0
mean teacher entropy: 1.1773 -> 0.5559 (nats)
sharpened targets stayed normalized and trained without error
Sharpening measurably reduced the teacher's entropy, the K+1 bucket
distribution still sums to one to within 1e-5, and a real training step ran
against the modified targets.
If you only need to change hyper-parameters, an environment or an objective, you
do not need Python at all — edit a recipe. Start from
recipes/toy_cpu.yaml, which is commented
line by line, and validate it before running:
miniverl validate recipes/toy_cpu.yaml
miniverl train recipes/toy_cpu.yaml --dry-run