Repository navigation
Expand file tree
/
Copy pathCITATION.cff
More file actions
124 lines (124 loc) · 5.05 KB
/
Copy pathCITATION.cff
File metadata and controls
124 lines (124 loc) · 5.05 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
cff-version: 1.2.0
title: "miniVERL: A bounded single-GPU runtime for verl-style OPD"
message: "If you use miniVERL in your work, please cite it as below."
type: software
version: 0.9.0
date-released: 2026-08-13
license: Apache-2.0
repository-code: "https://github.com/DaoyuanLi2816/mini-verl"
url: "https://github.com/DaoyuanLi2816/mini-verl"
abstract: >-
miniVERL runs a documented subset of verl-style on-policy distillation on one
consumer GPU. It is a local single-GPU runtime for a documented subset of verl v0.8
on-policy distillation. It consumes typed verl-shaped configuration and
Parquet prompts, executes actor rollout, teacher scoring and actor update in
one inspectable process, and exports standard PEFT, Parquet and configuration
artifacts through a fail-closed pinned bridge. Its native core also supports
multi-turn, tool-aware on-policy distillation: a student language
model samples its own tool-using trajectories against deterministic local
environments, a teacher scores exactly the states the student visited, and
the student is updated with token-level distributional supervision on its own
generated tokens only. It implements exact full-vocabulary forward KL,
reverse KL and Jensen-Shannon divergences as well as a compressed top-k plus
tail coarse-graining, enforces per-token provenance so that tool output can
never become a training label, and stores teacher targets in a versioned,
checksummed, pickle-free cache with policy-version enforcement.
Around that core it provides supervised fine-tuning, recorded-provenance
preference optimization and knowledge distillation; shared-backbone
execution that switches student, teacher and reference
roles across adapters on one set of base weights to fit low-memory hardware;
padded update batching; deterministic replay; transactional checkpoints and
cross-process run locks; and mechanism studies such as RecoveryBench that are
published with their negative and mixed results intact. A fail-closed
artifact bridge exchanges configs, LoRA adapters, tokenizers and Parquet
datasets with one pinned verl profile, verifying what it can and refusing to
imply the rest: it does not launch distributed jobs or establish broad
algorithmic parity. miniVERL is designed for one personal CUDA GPU,
automatically selects bf16 or fp16, and requires neither Ray nor a cluster.
Published performance is measured on one RTX 4080; other GPU models use the
same code path but remain unmeasured. The v0.7 external-alignment study
terminated at its preregistered checkpoint-selection gate: two declared
lineages and eight candidates all scored 0/64 retained JSONNav utility, so
no teacher qualification, continuation method comparison or reserved final
test was run.
authors:
- family-names: Li
given-names: Daoyuan
email: lidaoyuan2816@gmail.com
keywords:
- on-policy distillation
- knowledge distillation
- llm alignment
- preference optimization
- large language models
- tool use
- agents
- PEFT
- QLoRA
- consumer GPU
- single GPU
- post-training
- reproducibility
references:
- type: article
title: "Distilling the Knowledge in a Neural Network"
authors:
- family-names: Hinton
given-names: Geoffrey
- family-names: Vinyals
given-names: Oriol
- family-names: Dean
given-names: Jeff
year: 2015
url: "https://arxiv.org/abs/1503.02531"
notes: "Source of the high-temperature forward-KL correction; reverse-KL and JSD use it only as an explicit heuristic."
- type: article
title: >-
On-Policy Distillation of Language Models: Learning from Self-Generated
Mistakes
authors:
- family-names: Agarwal
given-names: Rishabh
year: 2024
url: "https://arxiv.org/abs/2306.13649"
notes: "Generalized JSD and training on student-sampled sequences."
- type: article
title: "On-Policy Context Distillation for Language Models"
authors:
- family-names: Ye
given-names: Tianzhu
- family-names: Dong
given-names: Li
- family-names: Wu
given-names: Xun
- family-names: Huang
given-names: Shaohan
- family-names: Wei
given-names: Furu
year: 2026
url: "https://arxiv.org/abs/2602.12275"
notes: "The privileged-context teacher mode follows this formulation."
- type: article
title: "Entropy-Aware On-Policy Distillation of Language Models"
authors:
- family-names: Jin
given-names: Woogyeol
- family-names: Min
given-names: Taywon
- family-names: Yang
given-names: Yongjin
- family-names: Wei
given-names: Dennis
- family-names: Zhou
given-names: Yi
- family-names: Kadhe
given-names: Swanand Ravindra
- family-names: Baracaldo
given-names: Nathalie
- family-names: Lee
given-names: Kimin
year: 2026
url: "https://arxiv.org/abs/2603.07079"
notes: >-
Motivates recording per-token teacher entropy. Entropy-aware divergence
mixing is a roadmap item and is not implemented in miniVERL.