You have trained a draft model. It sits next to a big target model — Qwen3-8B, say — and its whole job is to guess the next handful of tokens so the target can verify a block at a time instead of crawling one token per forward pass. You benchmarked it (I did too, in a previous post): a real speedup, tokens flying. So you do the obvious thing to make it better — you train it more. More SFT epochs, more target-generated data. And the curve does something quietly infuriating: it stops. Accuracy on the training objective keeps ticking up, but the number you actually care about — how many guessed tokens the target accepts per round — flattens into a plateau and refuses to move.

你训练好了一个草稿模型。 它挨着一个大目标模型 —— 比如 Qwen3-8B —— 它唯一的活儿就是猜出接下来的几个 token， 好让目标模型一次验证一整块， 而不是一个 forward pass 只吐一个 token 地爬。 你跑过 benchmark（我也跑过， 见之前那篇）： 实打实的加速， token 飞起来。 于是你做了最显然的事去让它更好 —— 多训一会儿。 更多 SFT epoch、 更多目标模型生成的数据。 然后曲线做了一件安静得令人抓狂的事： 它停了。 训练目标上的准确率还在往上走， 可你真正在乎的那个数字 —— 每一轮目标模型接受几个猜测的 token —— 摊成一条高原线， 死活不动。

This paper — Draft-OPD: On-Policy Distillation for Speculative Draft Models (Haodi Lei and ten co-authors from Shanghai Jiao Tong, Shanghai AI Lab, Tsinghua, CUHK, Peking, and Zhejiang) — is about that plateau: what causes it, why more data cannot fix it, and a training recipe that gets past it by teaching the drafter from the exact tokens it gets wrong. The result they report is a draft model that, at matched training compute, beats EAGLE-3 by 23% and DFlash by 13% in average decoding speedup on reasoning models. This is not a new draft architecture — it is a new way to train the draft architectures you already have.

这篇论文 —— Draft-OPD： 面向投机草稿模型的在线蒸馏（Haodi Lei 与十位合作者， 来自上海交大、 上海 AI Lab、 清华、 港中文、 北大、 浙大）—— 讲的就是这个高原： 它由什么造成、 为什么更多数据修不了、 以及一个能越过它的训练配方 —— 拿草稿模型猜错的那些 token 去教它。 他们报告的结果是： 在训练算力相当的前提下， 一个草稿模型在推理模型上的平均解码加速比， 比 EAGLE-3 高 23%、 比 DFlash 高 13%。 这不是一种新的草稿架构 —— 而是一种训练你已经有的那些草稿架构的新办法。

A note on where I sit. I work on AMD GPU kernels — MI300X, MI355X, ROCm, AITER, Triton-for-ROCm — and speculative decoding is one of the largest single levers on decode-time throughput for the models we serve. So I read this paper twice: once as a machine-learning idea, and once through a serving-and-roofline lens, asking "what would this change on an MI300X node?" I flag the second reading in § 8.

说一句我的位置。 我做的是 AMD GPU kernel —— MI300X、 MI355X、 ROCm、 AITER、 ROCm 上的 Triton —— 而投机解码是我们服务的模型里， 对 decode 阶段吞吐影响最大的单个杠杆之一。 所以这篇论文我读了两遍： 一遍当成机器学习的想法读， 一遍带着"服务与 roofline"的视角读， 问自己"这东西放到 MI300X 节点上会改变什么？" 第二遍的读法我放在 § 8。

§ 1 · The loop you are acceleratingSpeculative decoding, in one loop

§ 1 · 你在加速的那个循环投机解码， 一个循环就讲完

Everything in this paper is a consequence of one loop, so we earn it first. Autoregressive decoding is slow for a boring reason: to produce token t+1 the big model must read all of tokens up to t, and each token costs one full forward pass over billions of parameters. The pass is memory-bound — you spend most of your time streaming weights from HBM, not doing math — so a pass that produces one token wastes almost all of the compute it paid to load those weights.

这篇论文里的一切， 都是一个循环的推论， 所以我们先把它挣出来。 自回归解码慢有一个无聊的原因： 要产出第 t+1 个 token， 大模型必须读完到第 t 为止的所有 token， 而每个 token 都要花一整个 forward pass、 扫过几十上百亿参数。 这个 pass 是访存受限的 —— 你大部分时间花在从 HBM 里搬权重， 而不是做数学 —— 所以一个只产出一个 token 的 pass， 几乎浪费掉了它为搬那些权重所付的全部算力。

Speculative decoding exploits that waste. A verifier forward pass can score K tokens for almost the same cost as scoring one, because the weights are already loaded. So: let a small, cheap draft model q guess the next K tokens; then run one target p pass to check all K at once; keep the longest prefix the target agrees with; and — this is the guarantee — because the check uses the target's own probabilities, the output distribution is exactly the target's. Speculative decoding is lossless: same text as running the target alone, produced in fewer target passes.

投机解码利用的正是这份浪费。 一个 verifier forward pass 可以给 K 个 token 打分， 花的代价和给一个 token 打分几乎一样 —— 因为权重反正已经加载好了。 于是： 让一个小而便宜的草稿模型 q 猜出接下来 K 个 token； 再跑一次目标模型 p 的 pass， 一口气检查全部 K 个； 保留目标模型认可的最长前缀； 而且 —— 这是那条保证 —— 因为检查用的是目标模型自己的概率， 输出分布严格等于目标模型的分布。 投机解码是无损的： 和单独跑目标模型产出的文本一模一样， 只是用了更少的目标 pass。

The metric that governs the whole thing has a name: acceptance length \tau — the number of drafted tokens the target accepts per verification round. If the drafter proposes 16 tokens and the target accepts the first 6, then rejects the 7th, \tau = 6 for that round. Higher \tau means each expensive target pass "buys" more output tokens, which is the entire source of speedup. The paper reports two numbers side by side, and the difference between them is worth internalizing now:

支配这整件事的指标有个名字： 接受长度 \tau —— 每一轮验证里目标模型接受的草稿 token 数。 如果草稿模型提议 16 个 token， 目标模型接受了前 6 个、 在第 7 个上拒绝， 那这一轮的 \tau = 6。 \tau 越大， 每一次昂贵的目标 pass"买到"的输出 token 越多 —— 这就是加速的全部来源。 论文并排报了两个数字， 它俩之间的区别现在就值得刻进脑子里：

§ 2 · The train / inference gapWhy SFT plateaus

§ 2 · 训练 / 推理的裂缝为什么 SFT 会停滞

Here is the standard way to train a draft model, and it is exactly what plateaus. You take the target model, have it generate a big pile of trajectories, and train the drafter to imitate the target's next-token distribution on every prefix of those trajectories. This is supervised fine-tuning (SFT) — offline distillation on a fixed corpus. EAGLE-3 and DFlash are both trained this way, and both are excellent. But the paper's Figure 1 shows the trap: keep running SFT and acceptance length on test data stops improving — it fluctuates around a fixed ceiling. Worse, if you feed the on-policy data back through the SFT objective, \tau can actually drop. The limitation is not compute. It is which states the drafter trains on.

这是训练草稿模型的标准做法， 而它恰恰就是那个会停滞的做法。 你拿目标模型生成一大堆轨迹， 然后训练草稿模型， 让它在这些轨迹的每个前缀上模仿目标模型的下一 token 分布。 这就是监督微调（SFT） —— 在固定语料上的离线蒸馏。 EAGLE-3 和 DFlash 都是这么训的， 而且都很出色。 但论文的 Figure 1 揭出了陷阱： 继续跑 SFT， 测试数据上的接受长度就不再改善 —— 它围着一个固定天花板波动。 更糟的是， 如果你把在线采集的数据再喂回 SFT 目标， \tau 甚至会下降。 瓶颈不在算力。 在于草稿模型在哪些状态上训练。

The mismatch, said plainly

把这个错配说白

Recall Eq. 1: at inference, each drafted token stands on the drafter's own earlier guesses inside the block. So the states that decide acceptance — the prefixes the target is judging — are draft-induced. But SFT only ever showed the drafter target-induced states: every prefix in the training corpus came from the target. The drafter is trained on one distribution of states and graded on another. Its characteristic errors — the exact moments where it derails and a block gets rejected — never appear in the SFT data. You cannot fix an error the training set never contains, no matter how many epochs you run.

回想 Eq. 1： 推理时， 每个草稿 token 都踩在草稿模型自己在这一块里更早的猜测上。 所以决定接受与否的那些状态 —— 目标模型正在评判的那些前缀 —— 是草稿诱导的。 而 SFT 只给草稿模型看过目标诱导的状态： 训练语料里的每个前缀都来自目标模型。 草稿模型在一种状态分布上训练， 却在另一种上被打分。 它那些特征性的错误 —— 它跑偏、 整块被拒的那些确切时刻 —— 从来不出现在 SFT 数据里。 训练集里根本没有的错误， 你训再多 epoch 也修不了。

§ 3 · The obvious fix, and its two trapsOn-policy distillation — and why it breaks

§ 3 · 显然的修法， 和它的两个陷阱在线蒸馏 —— 以及它为何失灵

If the problem is that training and grading use different state distributions, the textbook remedy is on-policy distillation (OPD): sample states from the student's own policy, and have the teacher supervise there. Formally, let the student generate its own trajectory and take its prefixes as the training states:

如果问题在于训练和打分用了不同的状态分布， 教科书上的解药就是在线蒸馏（OPD）： 从学生自己的策略里采样状态， 让老师在那里做监督。 形式化地说， 让学生生成自己的轨迹， 取它的前缀作为训练状态：

Beautiful in principle. Now watch it break on a real draft model. Eq. 2 quietly assumes the student can generate a usable trajectory on its own. A model-based drafter — a small model from the same family — can. But the strong modern drafters, EAGLE-3 and DFlash, cannot: they are built to propose short blocks under target guidance, not to run as standalone autoregressive generators. Try to make one roll out a whole sequence alone and you get, in the paper's words, "repetitive or degenerate samples." Garbage states, useless supervision. That is Trap 1.

原理上很漂亮。 现在看它在一个真实草稿模型上怎么崩。 Eq. 2 悄悄假设了学生能自己生成一条可用的轨迹。 一个基于模型的草稿器 —— 同族的一个小模型 —— 能做到。 但强大的现代草稿器， EAGLE-3 和 DFlash， 做不到： 它们生来就是在目标模型引导下提出短块的， 不是拿来当独立自回归生成器用的。 硬让它一个人 rollout 出整条序列， 你得到的是论文原话说的"重复或退化的样本"。 垃圾状态， 无用监督。 这是陷阱一。

So you try the other thing: generate the trajectory with actual speculative decoding — draft proposes, target verifies. Now the sequence is fluent and stable. But look what verification did: it kept the accepted tokens and threw away every rejected proposal. The surviving trajectory follows the target's distribution, not the draft's. You are back to training on target-induced states — the exact mismatch you were trying to escape — and the tokens you deleted, the rejected ones, were the most valuable signal in the entire process. That is Trap 2, and it is the sharper one.

于是你试另一条路： 用真正的投机解码去生成轨迹 —— 草稿提议、 目标验证。 现在序列流畅又稳定了。 但看看验证干了什么： 它保留了被接受的 token， 却扔掉了每一个被拒的提议。 幸存下来的轨迹服从目标模型的分布， 不是草稿的。 你又回到了在目标诱导状态上训练 —— 正是你想逃开的那个错配 —— 而你删掉的那些 token、 那些被拒的 token， 恰恰是整个过程里最宝贵的信号。 这是陷阱二， 而且是更尖锐的那个。

§ 4 · Getting both at onceDraft-OPD: the three-stage trick

§ 4 · 同时拿到两者Draft-OPD： 三段式技巧

The key realization is a decoupling: generate the trajectory one way, choose the training states another. The trajectory comes from target-assisted speculative decoding (stable, target-quality — solves Trap 1). The training states are the drafter's actual proposed blocks, including the rejected tail, recovered afterward (on-policy — solves Trap 2). Three stages make that happen.

关键的领悟是一次解耦： 用一种方式生成轨迹， 用另一种方式挑训练状态。 轨迹来自目标辅助的投机解码（稳定、 目标级质量 —— 解决陷阱一）。 训练状态是草稿模型实际提出的那些块、 连被拒的尾巴一起， 事后再恢复出来（在线 —— 解决陷阱二）。 三个阶段把这件事做成。

Stage 1 · Rollout, and record the anchors

阶段一 · Rollout， 并记下 anchor

Run ordinary speculative decoding on a prompt. At each step m, note the anchor a_m — the index where this drafted block begins — then let the drafter propose its K-token block (Eq. 4). The target verifies, accepts r_m tokens, and the next anchor advances: a_{m+1} = a_m + r_m. What you record is cheap: the anchor list, the proposed blocks, and the accept counts. The paper's own line is the hinge: "Anchors preserve the draft model's local actions without requiring the draft model to generate an entire sequence alone." The global trajectory is target-stabilized; the local action at every anchor is a genuine draft-policy state.

在一个 prompt 上跑普通的投机解码。 每一步 m， 记下 anchor a_m —— 这一块草稿从哪个位置开始 —— 然后让草稿模型提出它的 K token 块（Eq. 4）。 目标验证、 接受 r_m 个 token， 下一个 anchor 前移： a_{m+1} = a_m + r_m。 你记录的东西很便宜： anchor 列表、 提出的块、 接受计数。 论文自己那句话是枢纽： "Anchor 保留了草稿模型的局部动作， 而不要求草稿模型独自生成整条序列。" 全局轨迹由目标模型稳住； 每个 anchor 处的局部动作， 是一个货真价实的草稿策略状态。

Stage 2 · Replay from the anchors

阶段二 · 从 anchor 处回放

After the rollout, for each anchor build the replay context c_m = (x,\ y_{\le a_m}) — the prompt plus the verified prefix up to that anchor — and teacher-force the exact drafted block d_m (rejected tokens included) through both models, reading off the full next-token distributions at every position (Eqs. 6–7). This is the move that recovers the discarded signal: you are asking the target to score the draft's actual mistakes, not just the tokens that happened to survive. And it is cheap — the hidden states over those verified prefixes were already computed during the rollout, so replay reuses them and only pays for the K-token block on top; every anchor's block can be scored in parallel.

Rollout 之后， 对每个 anchor 构造回放上下文 c_m = (x,\ y_{\le a_m}) —— prompt 加上到这个 anchor 为止的已验证前缀 —— 然后把那个确切的草稿块 d_m（连被拒的 token 一起）teacher-force 地过两个模型， 在每个位置读出完整的下一 token 分布（Eq. 6–7）。 这一步就是把被丢弃的信号恢复回来的动作： 你在让目标模型给草稿的实际错误打分， 而不只是给那些碰巧幸存的 token 打分。 而且它很便宜 —— 那些已验证前缀上的 hidden state 在 rollout 时就已经算过了， 所以回放复用它们， 只为上面那 K token 的块付代价； 每个 anchor 的块都可以并行打分。

Stage 3 · Distill, but treat accepted and rejected tokens differently

阶段三 · 蒸馏， 但对接受和被拒的 token 区别对待

Now you have, for every drafted position, both models' distributions — and a label: was this position accepted or rejected? Draft-OPD's objective uses that label to point each token's supervision in the right direction. That is the heart of the paper, and it earns its own section next. But the shape of the whole method is already here: a stable rollout you can trust, replay that resurrects the rejected tokens, and a loss that knows which is which.

现在对每个草稿位置， 你都有了两个模型的分布 —— 还有一个标签： 这个位置是被接受还是被拒？ Draft-OPD 的目标函数用这个标签， 把每个 token 的监督指向正确的方向。 这是全篇的心脏， 下一节它值得单独占一节。 但整个方法的形状已经在这儿了： 一条你能信任的稳定 rollout、 让被拒 token 复活的回放、 以及一个知道谁是谁的损失函数。

§ 5 · The deep endThe acceptance-aware objective, up close

§ 5 · 最深的一段近看接受度感知的目标函数

Partition every drafted position by the label from Stage 1. For block m with r_m accepted tokens, positions 1 \le k \le r_m are the accepted set I_{\mathrm{acc}}; positions r_m < k \le K are the rejected set I_{\mathrm{rej}}. The objective applies opposite KL directions to the two sets — and the reason it does is the single most elegant idea in the paper.

按阶段一的标签给每个草稿位置分组。 对于接受了 r_m 个 token 的块 m， 位置 1 \le k \le r_m 是接受集 I_{\mathrm{acc}}； 位置 r_m < k \le K 是拒绝集 I_{\mathrm{rej}}。 目标函数给这两个集合用了相反的 KL 方向 —— 而它这么做的理由， 是全篇最优雅的一个想法。

Why forward KL for accepted, reverse KL for rejected

为什么接受用 forward KL、 拒绝用 reverse KL

This is worth slowing down for, because it is where the paper turns a bookkeeping distinction into a principle. Appendix B derives it. A forward KL D(p\|q) is what falls out of target-weighted cross-entropy \mathbb{E}_{y\sim p}[-\log q(y)] = H(p) + D_{\mathrm{KL}}(p\|q) — it is mode-covering: it forces the student to put mass everywhere the teacher does. That is the right thing at accepted positions, because those are verified, target-distributed states where you want the drafter to cover the target and keep agreement high.

这里值得慢下来， 因为这正是论文把一个记账上的区分变成一条原则的地方。 Appendix B 推导了它。 forward KL D(p\|q) 是从目标加权的交叉熵 \mathbb{E}_{y\sim p}[-\log q(y)] = H(p) + D_{\mathrm{KL}}(p\|q) 里掉出来的 —— 它是覆盖模式（mode-covering）的： 它逼学生在老师有质量的地方处处放上质量。 在接受位置上这是对的， 因为那些是已验证、 目标分布的状态， 你想让草稿模型覆盖目标、 把一致性维持得很高。

A reverse KL D(q\|p) is the opposite: the expectation is under q, so the gradient concentrates on tokens the drafter is confident about but the target is not. It is mode-seeking: it punishes probability mass the drafter puts where the target has little. And that is exactly a rejected token — a place the drafter was confidently wrong, and got caught by verification. Reverse KL directly suppresses the drafter's confident wrong modes, which are the literal cause of rejections. Use forward KL here instead and you would waste the small drafter's capacity trying to cover the teacher's entire distribution at a state it already fails.

reverse KL D(q\|p) 是反过来的： 期望取在 q 下， 所以梯度集中在草稿模型很自信、 目标模型却不认可的 token 上。 它是寻找模式（mode-seeking）的： 它惩罚草稿模型放在目标模型几乎没质量处的那些概率质量。 而这恰恰就是一个被拒的 token —— 草稿模型自信地错了、 被验证逮住的地方。 reverse KL 直接压制草稿模型自信的错误模式， 那正是拒绝的字面成因。 这里若改用 forward KL， 你就会浪费小草稿模型的容量， 去在一个它本就失败的状态上覆盖老师的整个分布。

The \gamma = 0.8 position decay

\gamma = 0.8 位置衰减

One more piece. The rejected-token loss weights position k by w_k = \gamma^{k-1} with \gamma = 0.8 — later positions in the block get exponentially less weight. Why? Go back to Plate I: an early error kills the whole tail. If the drafter fails at position 3, positions 4–16 are dead regardless of what they contained — and worse, in the replay they are conditioned on an already-wrong prefix, so their supervision is noisy. The earliest failure is the one that actually cost you acceptance length, so it gets the most weight. The ablation shows removing this decay costs about 0.4× speedup on MATH-500 — not free.

还有一块。 拒绝 token 的损失把位置 k 的权重设为 w_k = \gamma^{k-1}， \gamma = 0.8 —— 块里越靠后的位置权重指数级越小。 为什么？ 回到 Plate I： 一个早期错误会杀掉整条尾巴。 如果草稿模型在位置 3 失败， 位置 4–16 无论内容如何都作废了 —— 更糟的是， 在回放里它们依赖的是一个已经错了的前缀， 所以它们的监督是有噪声的。 最早的那个失败才是真正让你损失接受长度的那个， 所以它拿到最多权重。 消融显示去掉这个衰减在 MATH-500 上损失约 0.4× 加速 —— 不是白给的。

§ 6 · Reading the numbers honestlyWhat it actually buys

§ 6 · 诚实地读数字它到底换来什么

Setup, so you can weigh the numbers: targets are Qwen3-4B, Qwen3-8B, and Qwen3-30B-A3B-Thinking-2507 (a MoE); drafts are DFlash-style, 5 transformer layers for the two dense targets and 8 for the MoE, block size 16. Seven benchmarks — GSM8K, MATH-500, AIME25, MBPP, HumanEval, SWE-bench Lite, MT-Bench. Everything runs on NVIDIA H200, batch size 1, thinking mode on unless stated. And the fairness knob that makes the comparison mean something: baselines get matched FLOPs — 10 SFT epochs versus Draft-OPD's 6 SFT + 8 OPD — so the gains are from how you train, not how much.

先交代设置， 你才好掂量数字： 目标模型是 Qwen3-4B、 Qwen3-8B、 Qwen3-30B-A3B-Thinking-2507（一个 MoE）； 草稿是 DFlash 风格， 两个稠密目标用 5 层 transformer、 MoE 用 8 层， 块大小 16。 七个 benchmark —— GSM8K、 MATH-500、 AIME25、 MBPP、 HumanEval、 SWE-bench Lite、 MT-Bench。 全部跑在 NVIDIA H200 上， batch size 1， 除非另说都开 thinking 模式。 还有那个让比较有意义的公平旋钮： 基线拿到相当的 FLOPs —— 10 个 SFT epoch 对 Draft-OPD 的 6 SFT + 8 OPD —— 所以增益来自怎么训， 不是训多少。

The headline — and the number you must read carefully

头条 —— 以及那个你必须仔细读的数字

The abstract says "over 5× lossless acceleration for thinking models." That phrasing is verbatim and true — but if you write it down as "Draft-OPD is 5× faster" you have misread it. Here is the actual Table 1, thinking mode, temperature 0, and the honest averages:

摘要说"对 thinking 模型超过 5× 无损加速"。 这句措辞是原文原话、 也没错 —— 但如果你把它记成"Draft-OPD 快 5 倍"， 你就读错了。 这是真实的 Table 1， thinking 模式、 temperature 0， 以及诚实的平均值：

Two-model average: \tau rises from 5.35 → 5.85, and average speedup is 4.88× across the seven benchmarks — which is below 5×. The "over 5×" is real but selective: it holds on individual thinking-mode benchmarks (MATH-500 hits 5.55× on Qwen3-4B) and as the non-thinking average (5.17×). The clean, defensible claim is the relative one: +23% over EAGLE-3, +13% over DFlash, on average speedup, at matched FLOPs. That is the number to quote.

两模型平均： \tau 从 5.35 升到 5.85， 七个 benchmark 上的平均加速是 4.88× —— 低于 5×。 "超过 5×"是真的、 但有选择性： 它在单个 thinking 模式 benchmark 上成立（MATH-500 在 Qwen3-4B 上到 5.55×）， 也作为非 thinking 平均成立（5.17×）。 干净、 站得住的说法是那个相对值： 比 EAGLE-3 高 23%、 比 DFlash 高 13%， 在相当 FLOPs 下的平均加速。 这才是该引用的数字。

Non-thinking, and the serving test

非 thinking， 以及服务端测试

With thinking disabled, everything is easier to draft (less reasoning entropy — more on that in § 7), and Draft-OPD averages \tau = 6.33 and 5.17× speedup. But the result I trust most is the SGLang one, because it answers the question a serving engineer actually asks: does a higher \tau survive a real inference engine, at real concurrency? They serve Draft-OPD vs DFlash on SGLang with the FA3 backend, sweeping concurrency 1 → 32.

关掉 thinking 后， 一切都更好草稿（推理熵更少 —— § 7 细说）， Draft-OPD 平均 \tau = 6.33、 加速 5.17×。 但我最信的结果是 SGLang 那个， 因为它回答了服务工程师真正会问的问题： 更高的 \tau 能不能在一个真实推理引擎里、 在真实并发下存活？ 他们在 SGLang 上用 FA3 后端服务 Draft-OPD 对 DFlash， 把并发从 1 扫到 32。

Two things make this the strongest section. First, the \tau gains translate to throughput gains almost 1:1 — so in SGLang+FA3 the draft overhead is small enough that acceptance length really is the binding variable. Second, and this is the surprising part: the relative gain does not shrink at higher concurrency. Speculative decoding classically loses value as batch grows (verification becomes compute-bound), yet here the average relative gain at concurrency 32 is even higher than at 1 — because the advantage over DFlash comes purely from \tau, which is batch-independent. The MoE target has the lowest absolute \tau (hardest to draft) but the largest relative gain (up to +17%): on-policy training helps most exactly where the offline draft was weakest.

有两点让这一节最有说服力。 第一， \tau 的增益几乎 1:1 地转成吞吐增益 —— 所以在 SGLang+FA3 里草稿开销小到足以让接受长度真正成为决定变量。 第二， 也是意外的地方： 相对增益在更高并发下并不缩水。 投机解码传统上随 batch 增大而失去价值（验证变成计算受限）， 可这里并发 32 的平均相对增益甚至高于并发 1 —— 因为对 DFlash 的优势纯粹来自 \tau， 而 \tau 与 batch 无关。 MoE 目标的绝对 \tau 最低（最难草稿）却相对增益最大（最高 +17%）： 在线训练恰恰在离线草稿最弱的地方帮助最大。

§ 7 · What actually mattersThe ablations, and a ceiling

§ 7 · 到底什么在起作用消融实验， 与一个天花板

The ablations are a clean factorization of the method into four independent knobs (Qwen3-4B, thinking on, τ=0; speedup / τ on MATH-500, HumanEval, MT-Bench). Read them as "how much does each design choice earn?"

消融实验把方法干净地拆成四个独立旋钮（Qwen3-4B、 thinking 开、 τ=0； MATH-500、 HumanEval、 MT-Bench 上的加速 / τ）。 把它们读成"每个设计选择赚了多少？"

The ranking is instructive: full (4.63×) > all-forward (4.48×) > no-decay (4.39×) ≈ all-reverse (4.38×) > random anchors (4.33×). Every knob contributes, and the single largest lever is where you replay — random anchors is the worst row. The paper's line: "not all on-policy states are equally useful." Being on-policy is necessary but not sufficient; you have to replay at the errors. Note also that all-forward is a respectable default and all-reverse is the worst KL choice — which is exactly what Plate V predicts, since reverse KL everywhere would treat verified accepted states as if they were errors.

这个排序很有教益： full（4.63×）> 全 forward（4.48×）> 去衰减（4.39×）≈ 全 reverse（4.38×）> 随机 anchor（4.33×）。 每个旋钮都有贡献， 而单个最大的杠杆是你在哪里回放 —— 随机 anchor 是最差的一行。 论文原话： "并非所有在线状态都同等有用。" 在线是必要但不充分的； 你必须在错误处回放。 还要注意全 forward 是个体面的默认、 全 reverse 是最差的 KL 选择 —— 这正是 Plate V 所预测的， 因为处处 reverse KL 会把已验证的接受状态当成错误来对待。

The sharpest single result

最锋利的单个结果

Section 5.3 isolates the core thesis in one number. Take Draft-OPD but train only on the verified continuation — i.e. keep the target-assisted rollout for stability but throw the rejected tokens away (this reduces to KL-loss SFT on target-distributed data). Average speedup drops from 4.63× to 4.29× — a 7.3% relative fall. Same data-collection cost, same target-quality text; the only difference is whether the rejected proposals get rescored and trained on. That 7.3% is the paper's entire argument compressed to a single controlled comparison.

Section 5.3 把核心论点浓缩进一个数字。 拿 Draft-OPD、 但只在已验证的续写上训练 —— 也就是为了稳定保留目标辅助 rollout， 却把被拒 token 扔掉（这退化成在目标分布数据上的 KL 损失 SFT）。 平均加速从 4.63× 掉到 4.29× —— 相对下降 7.3%。 同样的数据采集代价、 同样的目标级文本； 唯一的区别是被拒的提议有没有被重新打分、 被拿去训练。 那 7.3% 就是整篇论文的论点压缩成一次受控比较。

The draftability ceiling — a lovely piece of analysis

可草稿性天花板 —— 一段可爱的分析

Every method is slower in thinking mode than not (Qwen3-8B Draft-OPD: 4.89× thinking vs 5.60× non-thinking). Why? Appendix C's answer is information-theoretic and, to me, the most quietly interesting result in the paper. During long reasoning the target itself is more uncertain — at each step there are multiple plausible next tokens. The authors measure this directly: the target's own next-token NLL on its own generations is higher in thinking mode across ShareGPT, AoPS, and Nemotron. Less concentrated target mass means less predictable structure for a small drafter to capture. Draftability is upper-bounded by the target's entropy — a ceiling you can measure before training any drafter. Draft-OPD narrows the gap; it cannot repeal the ceiling.

每个方法在 thinking 模式下都比不开时慢（Qwen3-8B Draft-OPD： thinking 4.89× 对非 thinking 5.60×）。 为什么？ Appendix C 的答案是信息论式的， 在我看来是全篇最安静有趣的结果。 在长推理过程中， 目标模型本身更不确定 —— 每一步都有多个说得通的下一 token。 作者直接测了它： 目标模型在自己生成上的下一 token NLL， 在 thinking 模式下更高， 在 ShareGPT、 AoPS、 Nemotron 上都如此。 目标质量越不集中， 一个小草稿模型能捕捉的可预测结构就越少。 可草稿性被目标模型的熵从上方封顶 —— 这个天花板你在训练任何草稿模型之前就能测出来。 Draft-OPD 缩小了差距； 它废除不了这个天花板。

§ 8 · The serving-and-roofline readWhat this changes on an MI300X node

§ 8 · 服务与 roofline 视角这东西在 MI300X 节点上改变了什么

Now the second reading. This paper is NVIDIA-and-Qwen throughout — H200, SGLang+FA3, Qwen3. Nothing in the method is NVIDIA-specific, which is exactly why it is interesting to me: it is a training recipe that produces a better draft checkpoint, and a better draft checkpoint helps any serving stack. Here is how I read it against the two goals I actually work on.

现在读第二遍。 这篇论文从头到尾是 NVIDIA 加 Qwen —— H200、 SGLang+FA3、 Qwen3。 但方法里没有任何 NVIDIA 专属的东西， 这恰恰是它吸引我的地方： 它是一个产出更好草稿 checkpoint 的训练配方， 而更好的草稿 checkpoint 对任何服务栈都有帮助。 下面是我对照自己实际在做的两个目标怎么读它。

Why acceptance length is the right target on AMD too

为什么接受长度在 AMD 上也是对的目标

The roofline logic that makes speculative decoding win is architecture-agnostic. On MI300X, decode is memory-bound for the same reason it is on H200 — 192 GB of HBM3 to stream weights through, and a single-token forward pass leaves the matrix engines mostly idle. A target pass that verifies a 16-token block amortizes that weight-streaming cost over up to \tau + 1 output tokens. So raising \tau is a first-order win on our hardware, and the SGLang result — \tau gains surviving to concurrency 32 — matters more for us, not less, because our production serving runs at batch, not batch-1.

让投机解码取胜的 roofline 逻辑是与架构无关的。 在 MI300X 上， decode 是访存受限的， 原因和在 H200 上一样 —— 192 GB HBM3 要把权重流过去， 而单 token 的 forward pass 让矩阵引擎大半闲着。 一个验证 16 token 块的目标 pass， 把这份权重搬运代价摊到最多 \tau + 1 个输出 token 上。 所以抬高 \tau 在我们硬件上是一阶的胜利， 而那个 SGLang 结果 —— \tau 增益存活到并发 32 —— 对我们更重要而非更次要， 因为我们的生产服务跑在 batch 上， 不是 batch-1。

The part that maps onto multi-agent kernel work

和多智能体 kernel 工作对得上的那部分

Strip Draft-OPD to its shape and it is a small student learning from a big teacher's accept/reject signal on the student's own rollouts — generate, verify, keep the informative failures, iterate. That is the same loop I want an agent team running over kernels: generate a variant, benchmark it against roofline, keep the failures as signal (not just the wins), iterate. The paper even makes the case I keep making about iteration speed — its whole "target-assisted rollout" idea exists because you cannot get a usable training signal without a fast, stable generate-verify loop. Two concrete transfers I am filing:

把 Draft-OPD 剥到只剩形状， 它就是一个小学生在自己的 rollout 上、 从大老师的接受 / 拒绝信号里学习 —— 生成、 验证、 留住有信息量的失败、 迭代。 这就是我想让一个 agent team 在 kernel 上跑的同一个循环： 生成一个变体、 对着 roofline benchmark、 把失败当信号留住（不只是胜利）、 迭代。 这篇论文甚至印证了我一直在讲的关于迭代速度的话 —— 它整个"目标辅助 rollout"的想法之所以存在， 就是因为没有一个快速稳定的生成-验证循环、 你根本拿不到可用的训练信号。 我记下两个具体的迁移：

• Keep the rejects. The single sharpest result (7.3% from rescoring rejected tokens) is a reminder that in any generate-verify search, the failed attempts carry more signal than the successes. A kernel-optimization loop that logs only the fastest variant is throwing away its 7.3%.
• 留住被拒的。 那个最锋利的单个结果（重新打分被拒 token 带来的 7.3%）提醒我们： 在任何生成-验证搜索里， 失败的尝试比成功的携带更多信号。 一个只记录最快变体的 kernel 优化循环， 正在扔掉它自己的那 7.3%。
• Match the objective direction to who generated the state. The forward/reverse-KL split is really "supervise differently depending on whether this data came from the teacher or from the student's own error." That is a transferable design principle for any distillation or RL loop, kernels included.
• 让目标方向匹配是谁生成了这个状态。 forward / reverse KL 的拆分本质是"根据这份数据来自老师、 还是来自学生自己的错误， 用不同方式监督"。 这是一条可迁移的设计原则， 适用于任何蒸馏或 RL 循环， kernel 也不例外。

The honest caveat, which the paper itself flags: it is Qwen3-only, DFlash-architecture-only, and NVIDIA-only in evaluation. Porting the recipe to an AMD-served MoE would mean re-running the rollout-and-replay loop with our own target on our own hardware — the method transfers, but the checkpoints do not. That re-tuning is the broad-goal work: real NV→AMD porting was never just syntax translation, it is re-fitting the training loop to the new target and the new roofline.

诚实的告诫， 论文自己也标了： 评估上它只用了 Qwen3、 只用了 DFlash 架构、 只在 NVIDIA 上。 把这个配方移植到一个 AMD 服务的 MoE 上， 意味着要用我们自己的目标、 在我们自己的硬件上重跑 rollout-and-replay 循环 —— 方法可迁移， checkpoint 不行。 而那次重新调优就是广义目标的工作： 真正的 NV→AMD 移植从来不只是语法翻译， 它是把训练循环重新拟合到新目标和新 roofline 上。

§ 9 · What to take awayThree things to keep

§ 9 · 该带走什么三件要留下的事

If you keep three things from this paper, keep these.

如果你从这篇论文只留三件事， 留这三件。

One — the plateau is an exposure-bias problem, and the fix is on-policy training. A draft model trained on the target's clean trajectories never sees its own mistakes, so it cannot fix them. This is behavior cloning hitting its wall, and the answer is the same as RL-over-SFT: train on the states your policy actually visits.

一 —— 高原是一个 exposure-bias 问题， 而修法是在线训练。 一个在目标模型干净轨迹上训练的草稿模型， 从没见过自己的错误， 所以修不了它们。 这是 behavior cloning 撞墙， 而答案和"用 RL 取代 SFT"一样： 在你的策略真正走到的状态上训练。

Two — the trick is decoupling generation from state selection. Target-assisted rollout for a stable trajectory; error-position replay to resurrect the drafter's rejected tokens as on-policy signal; an acceptance-aware loss that points forward KL at what works and reverse KL at what fails. The 7.3% from rescoring rejected tokens is the whole thesis in one controlled number.

二 —— 诀窍是把生成和状态选择解耦。 用目标辅助 rollout 拿到稳定轨迹； 用 error-position replay 把草稿被拒的 token 复活成在线信号； 用一个接受度感知的损失， 让 forward KL 对准有效的、 reverse KL 对准失败的。 重新打分被拒 token 带来的 7.3%， 是整个论点浓缩成的一个受控数字。

Three — read the headline critically. "Over 5×" is the peak; 4.88× is the honest thinking-mode average; +23% / +13% is the robust relative claim. And the draftability ceiling is a genuinely useful idea beyond this paper: speculative-decoding speedup on a reasoning model is upper-bounded by the target's own next-token entropy, which you can measure before you train anything.

三 —— 带着批判读头条。 "超过 5×"是峰值； 4.88× 是诚实的 thinking 模式平均； +23% / +13% 是稳健的相对说法。 而可草稿性天花板是一个超出本文、 真正有用的想法： 推理模型上的投机解码加速， 被目标模型自己的下一 token 熵从上方封顶， 而这个熵你在训练任何东西之前就能测。

Source: Haodi Lei, Yafu Li, Haoran Zhang, Shunkai Zhang, Qianjia Cheng, Xiaoye Qu, Ganqu Cui, Bowen Zhou, Ning Ding, Yun Luo, Yu Cheng, "Draft-OPD: On-Policy Distillation for Speculative Draft Models" (arXiv:2605.29343, cs.CL, 28 May 2026). Numbers verified against the arXiv HTML v1: Table 1 (main results), Table 2 (SGLang serving), Tables 3–4 (ablations), and Appendices A–C. Companion reading in this Library: my EAGLE-3 on Qwen3-Coder benchmark (the drafter this paper improves) and the chunked-pipeline reading (the serving engine these numbers run on).

原文： Haodi Lei、 Yafu Li、 Haoran Zhang、 Shunkai Zhang、 Qianjia Cheng、 Xiaoye Qu、 Ganqu Cui、 Bowen Zhou、 Ning Ding、 Yun Luo、 Yu Cheng， 《Draft-OPD: On-Policy Distillation for Speculative Draft Models》（arXiv:2605.29343， cs.CL， 2026-05-28）。 数字均对照 arXiv HTML v1 核过： Table 1（主结果）、 Table 2（SGLang 服务）、 Table 3–4（消融）、 附录 A–C。 本文库里的同伴： 我的 EAGLE-3 on Qwen3-Coder benchmark（本文改进的那个草稿器）和 chunked-pipeline 精读（这些数字运行其上的服务引擎）。