PyPI · Stable docs · Development docs · 中文
verl for a single consumer GPU.
Run PPO, GRPO and on-policy distillation on one NVIDIA GPU. Bring your verl configs, train locally, pick up interrupted runs and export your models.
PyPI v0.16.0 is stable; main is development.
Install the matching CUDA-enabled build using the PyTorch installer, then:
python -m pip install "miniverl[train,hydra]"
miniverl data sample --reward-profile target-length --rows 8 --out data/rl-prompts.parquet
miniverl run --example hydra-ppo --bind reward.provider=target_length --dry-run
miniverl run --example hydra-ppo --bind reward.provider=target_length --run-id local-ppoThis Qwen3-0.6B example trains with a simple length reward. Inspect it in
rewards.jsonl, or follow the five-minute workflow
to use GRPO, resume training and export an adapter.
Already have a verl experiment? Bring its config tree and Hydra overrides:
miniverl run --verl-config-path /path/to/verl/trainer/config \
--verl-config-name ppo_trainer --dry-run \
algorithm.adv_estimator=grpo actor_rollout_ref.rollout.n=8 trainer.n_gpus_per_node=8The direct-config guide walks through binding your data and reward function. The single-GPU guide helps you choose a model and memory settings. Install extras add training, Hydra or quantization dependencies; choose the CUDA PyTorch build separately.
| You want to… | Start here | Take away |
|---|---|---|
| Train with rewards | PPO / GRPO walkthrough | A trained adapter, reward logs and resumable checkpoints |
| Distill a teacher | OPD quickstart | A student trained on its own rollouts with teacher feedback |
| Move to a larger setup | Export and hand off | PEFT, Parquet and config files, plus a report of remaining setup |
SFT, DPO and offline KD are also available through native recipes. Hardware planning helps fit a workload to your GPU.
- Familiar configs. Reuse verl fields and overrides; inspect how each maps to local execution.
- Visible training. Follow rollouts, rewards, losses and teacher targets.
- Exact recovery. Resume model, optimizer, data position and random state from a checkpoint.
- Reusable outputs. Export adapters, datasets and the configuration and source identities behind the run.
Actor, critic, reference, reward and teacher roles share the GPU in phases. PPO trains an actor and a separate critic; GRPO uses grouped rewards; OPD learns from teacher targets on the student's current rollouts.
On an RTX 4080, the Qwen3-0.6B/1.7B OPD workload completed eight updates over 32 distinct prompts at 3.1914 GiB peak reserved VRAM. An interruption after update four reproduced byte-identical trajectories, adapter and optimizer tensors on resume. The SmolLM2-360M/1.7B workload used 1.4961 GiB. Workload and measurements · 24-cell rollout-backend study.
For task outcomes, read the calculator study, RecoveryBench, Alignment Lab and External Alignment Gate. These include negative and mixed results, with their original data and scope.
miniVERL is an independent Apache-2.0 project for single-GPU NVIDIA CUDA training. It supports documented verl configuration profiles; model fit depends on GPU memory and workload. See the compatibility matrix for supported upstream versions and semantics, and limitations for hardware coverage, scientific scope and the distributed-execution boundary.
git clone https://github.com/DaoyuanLi2816/mini-verl.git
cd mini-verl
python -m pip install -e ".[dev,train]"
pytest -q -m "not gpu and not network"Contributing · Changelog · Citation · Security · License