The question问题
A colleague noticed the layout while reading the config and asked the obvious question: three of every four attention layers are KDA — so how much of the decode time is that?
同事在读 config 的时候注意到这个排布， 提了一个很自然的问题： 每四层注意力里有三层是 KDA， 那它占了多少解码时间？
The layout is real. linear_attn_config.kda_layers lists 69 indices and full_attn_layers lists 24, interleaved so that layers 1–3 are KDA, layer 4 is full attention, and so on to the tail, where 92 and 93 are both full attention. Every layer past the first also carries a mixture-of-experts feed-forward with 896 experts, 16 routed plus 2 shared.
这个排布是真的。 linear_attn_config.kda_layers 里有 69 个下标， full_attn_layers 里有 24 个， 交替排成第 1–3 层 KDA、第 4 层全注意力， 一直到末尾， 第 92 和 93 层都是全注意力。 除了第一层， 每层还挂着一个 896 专家的 MoE 前馈， 每 token 激活 16 个路由专家加 2 个共享专家。
But a fraction of layers only becomes a fraction of time if the layers cost the same, and here they cannot. The two mechanisms differ in the one thing that decides decode cost: what they have to read. A full-attention layer reads a cache that grows with every token generated. A KDA layer reads a fixed-size matrix that does not. Those are different functions of context length, so their time curves cannot have the same shape — the only real questions are where they cross and how steep the crossing is.
但层数占比要变成时间占比， 前提是每层成本相同， 而这里显然不同。 两种机制在决定解码成本的那件事上是相反的： 它们要读什么。 全注意力层读的缓存随生成的每个 token 增长， KDA 层读的是一个固定大小的矩阵。 这是两个不同的上下文长度函数， 时间曲线不可能同形—— 真正的问题只是它们在哪里相交、交叉有多陡。
The repeating unit is three KDA layers then one full-attention layer, 22 times, with a two-layer full-attention tail. The two boxes are the reason the time cannot follow the layer count: one mechanism carries a fixed state, the other carries a per-token cache.
重复单元是三层 KDA 加一层全注意力， 重复 22 次， 末尾是两层连续的全注意力。 下面两个框就是时间无法跟随层数的原因： 一种机制带的是固定状态， 另一种带的是随 token 增长的缓存。
Attributing a kernel to a layer type把一个核函数归到某类层上
The measurement sounds like it should be a profiler flag. It is not, and the reason is worth stating because it recurs in any attribution work on a fused, tensor-parallel serving stack.
这个测量听起来像是打开一个 profiler 开关就行。 并不是。 原因值得写下来， 因为在任何融合过的、张量并行的推理栈上做归因都会遇到同一件事。
Why kernel names cannot do it为什么靠核函数名字不行
A GPU profile is a list of kernel launches with names and durations. The obvious move is to bucket by name. That fails here because the names describe the shape of the computation, not its purpose. The dense GEMMs come from Tensile and hipBLASLt, whose kernel names encode tile geometry — Cijk_Alik_Bljk_..._MT256x16x64_... — so a KDA layer's q_proj and a shared expert's projection can be the same kernel, launched with the same tile, byte-identical in the trace. Any name-based rule that separates them is a guess.
一份 GPU profile 就是一串带名字和时长的 kernel launch， 最直接的做法是按名字分桶。 这里行不通， 因为名字描述的是计算的形状， 不是它的用途。 稠密 GEMM 来自 Tensile 和 hipBLASLt， 名字编码的是 tile 几何—— Cijk_Alik_Bljk_..._MT256x16x64_... —— 所以 KDA 层的 q_proj 和共享专家的投影可能是同一个核函数、同一个 tile， 在 trace 里逐字节相同。 任何基于名字去区分它们的规则都是在猜。
The fix is to ask the host side instead of the device side. A one-line-per-site patch wraps the three cost centres of KimiK3DecoderLayer in torch.profiler.record_function ranges named K3/kda, K3/full_attn and K3/moe. Each GPU kernel carries a correlation id shared with the host call that launched it, and that host call sits inside whichever range was open — so the innermost enclosing range names the block. The patch is gated behind SGLANG_K3_PROF_RANGES and compiles to a no-op class otherwise, because as section 07 shows, leaving it on is not free.
解法是去问主机侧， 而不是设备侧。 一个每处一行的补丁把 KimiK3DecoderLayer 的三个成本中心包进 torch.profiler.record_function 区间， 命名为 K3/kda、K3/full_attn、K3/moe。 每个 GPU 核函数都带一个 correlation id， 和发起它的主机调用共享； 那次主机调用落在当时打开的区间里—— 最内层的那个区间就是它所属的块。 补丁由 SGLANG_K3_PROF_RANGES 控制， 关闭时编译成一个空操作类， 因为第 07 节会说明， 一直开着并不是免费的。
Why that is not enough for decode为什么这对 decode 还不够
For prefill the ranges are the whole method. Prefill does not run through a CUDA graph in this configuration, so the profiler sees every kernel and the ranges label them directly.
对 prefill 来说， 区间就是全部方法。 这个配置下 prefill 不走 CUDA graph， profiler 能看到每个核函数， 区间直接给它们打上标签。
Decode is captured in a HIP graph, and the torch profiler cannot see kernels replayed from inside one. This is not subtle in the data: a decode trace of the running server showed 1.8 ms of GPU work per step against a real step time of about 26 ms, and the kernels it did show were the draft model and the speculative bookkeeping — everything outside the graph. Profiling a graph-mode decode with the torch profiler measures the parts you did not want.
Decode 被捕获进了 HIP graph， 而 torch profiler 看不到从 graph 内部重放的核函数。 这一点在数据里非常明显： 对运行中的服务器做一次 decode trace， 每步只看到 1.8 ms 的 GPU 工作， 而真实每步约 26 ms； 看到的那些核函数是 draft 模型和投机解码的簿记， 也就是 graph 之外的部分。 用 torch profiler 去测 graph 模式的 decode， 量到的恰好是你不想要的那部分。
So the ranges are used once, in eager mode, not to produce the answer but to learn the map from kernel name to block type. That map is then applied to traces taken with graphs on. The step is only legitimate if the map is nearly injective, and it is: 81% of non-collective dispatches carry a name unique to one block. The one compute kernel genuinely shared between KDA and full attention is the 1536→7168 o_proj, and it is shared in the most benign possible way — both blocks issue the identical shape, 69 times and 24 times per step, so splitting its time 69:24 is exact rather than an approximation.
于是区间只在 eager 模式下用一次， 目的不是给出答案， 而是学出核函数名字到块类型的映射。 然后把这张映射用到打开 graph 时采集的 trace 上。 这一步成立的前提是映射近乎单射， 而事实确实如此： 81% 的非集合通信派发的名字唯一属于一个块。 唯一真正被 KDA 和全注意力共享的计算核函数是 1536→7168 的 o_proj， 而且共享方式是最良性的一种—— 两个块发出的形状完全相同， 每步分别 69 次和 24 次， 所以按 69:24 切分是精确的， 不是近似。
A learned map is only worth as much as its validation. Every context point re-derives three structural counts before its numbers are used: exactly 69 KDA recurrence dispatches, 24 MLA stage-1 dispatches and 92 expert-GEMM dispatches per decode step — one per layer of each kind — and an integral per-step count for every mapped name. All five decode points pass, with 3372 dispatches per step and no unmapped kernel names, at 4K and at 1M alike. If the map did not describe the trace, a non-integral count would say so immediately.
一张学出来的映射， 价值全在于它的验证。 每个上下文点在使用数字之前都会重新导出三个结构性计数： 每个 decode step 恰好 69 次 KDA 递推派发、24 次 MLA stage-1 派发、92 次专家 GEMM 派发—— 各类层每层一次—— 并且每个已映射名字的每步派发次数必须是整数。 五个 decode 点全部通过， 每步 3372 次派发， 没有未映射的名字， 4K 和 1M 一样。 如果映射描述不了这份 trace， 一个非整数的计数会立刻暴露它。
Two approaches fail before the working one: names are ambiguous by construction, and the decode forward is invisible to the profiler because it is replayed from a graph. The ranges are therefore demoted from measurement instrument to map-learning instrument, and the map is validated by counting layers.
在可行方案之前有两条路走不通： 名字天然有歧义， 而 decode 的前向因为是从 graph 重放的， profiler 看不见。 于是区间从「测量工具」降级为「学映射的工具」， 再用数层数的方式验证这张映射。
Decode, 4K to 1MDecode： 4K 到 1M
Batch size 1, no speculative decoding. That is the realistic operating point at long context — at 1M the KV cache barely fits one sequence — and it removes the draft model and the accept length as confounds. Each point is the mean of 24 consecutive decode steps on rank TP0, with the profiler armed only after prefill has produced its first token.
批大小 1， 不开投机解码。 这是长上下文下的真实工作点—— 1M 时 KV cache 基本只装得下一条序列—— 同时也去掉了 draft 模型和接受长度这两个混淆因素。 每个点取 rank TP0 上连续 24 个 decode step 的平均， profiler 在 prefill 产出首个 token 之后才武装。
The blue band and the violet band are the same height in all five bars. Everything that makes the rightmost bar twice the height of the leftmost is the flame band — the 24 full-attention layers.
五根柱子里蓝色和紫色两段的高度完全一样。 让最右边这根柱子比最左边高一倍的， 全部来自橙红色那一段—— 那 24 层全注意力。
| Context上下文 | KDA · 69 L | Full attn · 24 L | MoE · 92 L | Attn-res残差库 | Other其他 | Device total设备合计 | Measured实测 |
|---|---|---|---|---|---|---|---|
| 4K | 3.54 · 17.1% | 3.10 · 14.9% | 10.55 · 50.8% | 2.29 · 11.0% | 1.29 · 6.2% | 20.77 | 19.39 |
| 32K | 3.55 · 16.3% | 4.05 · 18.6% | 10.57 · 48.5% | 2.29 · 10.5% | 1.33 · 6.1% | 21.79 | 20.39 |
| 64K | 3.54 · 15.7% | 4.81 · 21.3% | 10.55 · 46.8% | 2.28 · 10.1% | 1.36 · 6.0% | 22.53 | 21.19 |
| 512K | 3.59 · 11.5% | 12.67 · 40.7% | 10.64 · 34.2% | 2.33 · 7.5% | 1.92 · 6.2% | 31.16 | 29.67 |
| 1M | 3.52 · 8.8% | 21.23 · 52.9% | 10.52 · 26.2% | 2.32 · 5.8% | 2.51 · 6.3% | 40.10 | 38.75 |
Milliseconds of GPU time per decode step and share of device compute time. The last column is measured inter-token latency with CUDA graphs on, same server configuration. Full data: decode-composition.csv.
每个 decode step 的 GPU 毫秒数及其占设备计算时间的比例。 最后一列是同一服务器配置下打开 CUDA graph 后实测的 token 间延迟。 完整数据： decode-composition.csv。
KDA costs 3.52 to 3.59 ms per step across a 256-fold change in context. Its share of decode time falls from 17.1% to 8.8% purely because the denominator grows. MoE is equally flat at 10.5 ms and falls from 50.8% to 26.2% for the same reason. The 24 full-attention layers go from 3.10 ms to 21.23 ms, a factor of 6.8, and that single term accounts for essentially the whole 19.3 ms increase in step time.
在上下文变化 256 倍的范围内， KDA 每步耗时 3.52 到 3.59 ms。 它占解码时间的比例从 17.1% 降到 8.8%， 纯粹是因为分母变大了。 MoE 同样平坦， 稳定在 10.5 ms， 出于同样的原因从 50.8% 降到 26.2%。 24 层全注意力从 3.10 ms 涨到 21.23 ms， 6.8 倍， 而每步时间总共增加的 19.3 ms 基本全部来自这一项。
Composition was measured in eager mode, so it needs an anchor. Measured latency with CUDA graphs on, divided by the eager compute-kernel sum, is 0.933, 0.936, 0.940, 0.952 and 0.966 at the five points — a tight, slowly rising ratio. Real latency comes in slightly below the eager kernel sum, which means real collectives and launch gaps together net out to roughly nothing, and the composition transfers to wall-clock time without a correction term.
构成是在 eager 模式下测的， 所以需要一个锚。 打开 CUDA graph 后实测的每步延迟， 除以 eager 下计算核函数的时间之和， 在五个点上分别是 0.933、0.936、0.940、0.952、0.966—— 一个很紧、缓慢上升的比值。 真实延迟略低于 eager 的核函数时间和， 说明真实的集合通信与发射空隙加起来几乎相互抵消， 构成可以不加修正项地搬到墙钟时间上。
Why the two diverge — and why only 6.8×两者为何分化—— 以及为何只有 6.8 倍
Count the bytes each mechanism has to move for one decoded token, per rank. KDA reads and writes a [128 × 128] fp32 state for each of its 12 local heads in each of its 69 layers: 54.3 MB in, 54.3 MB out, unchanged whether the sequence is a thousand tokens or a million. MLA reads a compressed latent cache of 512 + 64 dimensions per token per layer, in bf16, over 24 layers: 27.0 KiB per token, which at 1M context is 28.96 GB for a single step.
按每生成一个 token、每张卡需要搬运的字节数来算。 KDA 在它的 69 层里， 每层为 12 个本地头各读写一个 [128 × 128] 的 fp32 状态： 读 54.3 MB， 写 54.3 MB， 序列是一千个 token 还是一百万个都一样。 MLA 读的是压缩后的 latent 缓存， 每 token 每层 512 + 64 维、bf16， 24 层合计每 token 27.0 KiB， 在 1M 上下文下单步就是 28.96 GB。
That is a ratio above 500×. The time ratio is 6.8×. The gap between those two numbers is the interesting part, and it is not a rounding error — it is the two kernels sitting in different bottleneck regimes.
这是 500 倍以上的字节差。 时间差是 6.8 倍。 这两个数字之间的落差才是有意思的地方， 而且它不是舍入误差—— 它是两个核函数处在完全不同的瓶颈区间。
Left: the two mechanisms' decode-time memory traffic, on a log axis. Right: what fraction of the machine's bandwidth the MLA KV scan converts that traffic into. The 533× byte gap becomes a 6.8× time gap because the KDA kernel cannot use the machine and the MLA kernel only uses a fifth of it.
左： 两种机制在解码时的访存量， 对数轴。 右： MLA 的 KV 扫描把这些访存量转化成了机器带宽的多少比例。 533 倍的字节差之所以只变成 6.8 倍的时间差， 是因为 KDA 的核函数用不满机器， 而 MLA 的核函数也只用了五分之一。
KDA's recurrence kernel is latency-bound, not bandwidth-bound. At batch 1 its grid is 48 workgroups of 64 threads — four V-blocks by twelve local heads — on a GPU with 256 compute units. It moves its 108.5 MB at an effective 0.24 TB/s, 3% of peak, because there is no way for 48 workgroups to saturate anything. Its 6.5 µs per layer is the cost of touching memory at all, and it would be roughly 6.5 µs if the state were half the size.
KDA 的递推核函数是延迟受限， 不是带宽受限。 批大小 1 时它的 grid 是 48 个 workgroup、每个 64 线程—— 四个 V 分块乘以十二个本地头—— 而 GPU 有 256 个计算单元。 它以 0.24 TB/s 的有效带宽搬完那 108.5 MB， 只有峰值的 3%， 因为 48 个 workgroup 无论如何也喂不满任何东西。 它每层 6.5 µs 的开销就是「碰一次内存」的代价， 状态哪怕小一半， 也还是大约 6.5 µs。
The MLA KV scan is bandwidth-bound and therefore does scale with the bytes, but it converts them at 1.64 TB/s against an 8 TB/s peak. That number is the single most actionable result in this experiment: at 1M the scan alone costs 17.61 ms of a 40.10 ms step, and four fifths of the machine's bandwidth is sitting idle while it runs.
MLA 的 KV 扫描是带宽受限的， 所以确实随字节数增长， 但它的转化效率是 1.64 TB/s， 而峰值是 8 TB/s。 这个数字是本次实验里最具可操作性的结果： 1M 时仅这一次扫描就要 17.61 ms， 占 40.10 ms 每步时间的 44%， 而它运行期间机器有五分之四的带宽是闲着的。
Inside the blocks拆开每个块
Block-level totals are the answer to the question that was asked. The breakdown inside each block is where the optimization targets are, and in two of the three cases it is not where you would guess.
块级别的合计回答了最初的问题。 而每个块内部的拆解才是优化靶子所在， 三个块里有两个的答案和直觉不一样。
A KDA layer is mostly not KDA一层 KDA 里大部分不是 KDA
Of the 3.54 ms that 69 KDA layers spend per decode step, the gated delta-rule recurrence — the actual linear-attention mathematics — is 445 µs, or 12.6%. Projection GEMMs are 2233 µs, 63%. The short depthwise convolution over the 4-token window is 297 µs and the gated output norm 269 µs. The proportions do not move between 4K and 1M.
69 层 KDA 每步花掉的 3.54 ms 里， 门控 delta-rule 递推—— 也就是线性注意力真正的数学部分—— 是 445 µs， 占 12.6%。 投影 GEMM 是 2233 µs， 占 63%。 4-token 窗口上的深度可分离短卷积是 297 µs， 门控输出 norm 是 269 µs。 从 4K 到 1M 这个比例不变。
If someone proposes making KDA cheaper by optimizing the recurrence kernel, the ceiling on that work is 12.6% of 8.8–17.1% of decode time — under 2% end to end, even if the kernel became free. The projections are ordinary bf16 matmuls that any attention variant would also need. The honest reading is that KDA's decode cost is already close to the floor for any mechanism with the same projection widths, and the way to reduce it is to make the layer narrower, not to make the recurrence faster.
如果有人提议通过优化递推核函数来降低 KDA 的成本， 这项工作的天花板是「解码时间的 8.8–17.1% 里的 12.6%」—— 端到端不到 2%， 哪怕核函数变成零开销也是如此。 那些投影是普通的 bf16 矩阵乘， 任何注意力变体同样需要。 更诚实的解读是： KDA 的解码成本已经接近「任何具有相同投影宽度的机制」的下限， 要降它得把层做窄， 而不是把递推做快。
A full-attention layer changes character with context全注意力层的性质随上下文改变
At 4K the KV scan is 308 µs, only 9.9% of the block; projections (1029 µs) and the split reduction (905 µs) dominate. At 1M the scan is 17609 µs and 83% of the block, while projections are unchanged at 1036 µs. So the same 24 layers are a fixed-overhead problem at short context and a pure bandwidth problem at long context — and an optimization that helps at one end may be irrelevant at the other.
4K 时 KV 扫描是 308 µs， 只占这个块的 9.9%； 投影（1029 µs）和 split 归约（905 µs）才是主项。 1M 时扫描变成 17609 µs、占 83%， 而投影仍是 1036 µs 不变。 所以同样这 24 层， 在短上下文下是固定开销问题， 在长上下文下是纯带宽问题—— 一端有效的优化在另一端可能毫无意义。
At batch 1, MoE does not spend its time on experts批大小 1 时， MoE 的时间没花在专家上
MoE is a flat 10.5 ms at every context, and the split inside it is counterintuitive. The routed-expert MXFP4 GEMMs — the thing the architecture is named after — are 1439 µs, 13.6%. The two dense shared experts cost 3836 µs, 36.4%. Routing, sorting and quantization cost 2959 µs, 28.1%.
MoE 在每个上下文点都稳定在 10.5 ms， 而它内部的拆分很反直觉。 被路由专家的 MXFP4 GEMM—— 这个架构因之得名的东西—— 是 1439 µs， 占 13.6%。 两个稠密共享专家花掉 3836 µs， 占 36.4%。 路由、排序和量化花掉 2959 µs， 占 28.1%。
The reason is structural rather than an implementation flaw. Sparsity removes work in proportion to how few experts fire, but it removes nothing from the parts that always run. One token through 16 of 896 experts is a set of very small GEMMs; the two shared experts run at full width for that same token; and the routing machinery — gate, top-k, sort, scatter — has a cost per token that is independent of how sparse the routing is. Sparsity scales down the numerator and leaves the fixed terms alone, so at batch 1 the fixed terms are what you see.
原因是结构性的， 不是实现缺陷。 稀疏化按「激活了多少专家」成比例地减少工作量， 但它对那些始终要跑的部分毫无作用。 一个 token 过 896 个专家里的 16 个， 是一组非常小的 GEMM； 而两个共享专家对同一个 token 是全宽度运行的； 路由机制—— gate、top-k、排序、scatter—— 的每 token 成本与稀疏程度无关。 稀疏化缩小的是分子， 固定项原封不动， 所以批大小 1 时你看到的就是固定项。
| Block块 | Component组成部分 | µs / step | Share of block占该块 | Scales with context?随上下文变化？ |
|---|---|---|---|---|
| KDA | Projection GEMMs投影 GEMM | 2233 | 63.2% | no否 |
| Gated delta-rule recurrence门控 delta-rule 递推 | 445 | 12.6% | no否 | |
| Short conv, kernel 4短卷积， kernel 4 | 297 | 8.4% | no否 | |
| Gated output RMSNorm门控输出 RMSNorm | 269 | 7.6% | no否 | |
| Full attn | KV scan, stage 1KV 扫描， stage 1 | 308 → 17609 | 9.9% → 83.0% | yes, linearly是， 线性 |
| KV-split reduction, stage 2KV-split 归约， stage 2 | 905 → 1731 | 29.2% → 8.2% | weakly弱相关 | |
| Projection GEMMs投影 GEMM | 1029 → 1036 | 33.2% → 4.9% | no否 | |
| MoE | Shared-expert GEMMs, dense共享专家 GEMM， 稠密 | 3836 | 36.4% | no否 |
| Routing, sort, quantize路由、排序、量化 | 2959 | 28.1% | no否 | |
| Routed-expert GEMMs, MXFP4被路由专家 GEMM， MXFP4 | 1439 | 13.6% | no否 |
Ranges show 4K → 1M where the component moves. Full 150-row breakdown: decode-block-internals.csv.
带箭头的区间表示该组成部分从 4K 到 1M 的变化。 完整的 150 行拆解： decode-block-internals.csv。
Prefill, 1K to 32K — the answer invertsPrefill： 1K 到 32K—— 答案反了过来
Prefill is the easier measurement and the more surprising result. Easier because there is no CUDA graph, so the ranges label the real trace directly, and because summed kernel time lands within 1% of measured TTFT at every size — 169.6 against 170 ms, 330.6 against 333, 588.5 against 593, 3052 against 3051 — which means the GPU is saturated throughout and the composition is the composition of wall-clock time, collectives included.
Prefill 是更容易的测量， 也是更意外的结果。 容易是因为没有 CUDA graph， 区间可以直接给真实 trace 打标签； 也因为核函数时间之和在每个尺寸上都落在实测 TTFT 的 1% 以内—— 169.6 对 170 ms， 330.6 对 333， 588.5 对 593， 3052 对 3051—— 这说明 GPU 全程饱和， 于是这个构成就是墙钟时间的构成， 集合通信也包含在内。
Right panel: the two lines cross at about 8K. Left of the crossing a full-attention layer is cheaper than a KDA layer, so the 3:1 layout costs prefill time rather than saving it — 69 KDA layers take 22.7% of an 8K prefill against 7.5% for 24 full-attention layers, purely on layer count.
右图： 两条线在 8K 附近相交。 交点左侧， 一层全注意力比一层 KDA 更便宜， 所以 3:1 的排布在 prefill 上是在花时间而不是省时间—— 8K 时 69 层 KDA 占 22.7%， 24 层全注意力占 7.5%， 差别纯粹来自层数。
| Input输入长度 | TTFT | Throughput吞吐 | KDA | Full attn全注意力 | MoE | One KDA layer单层 KDA | One full-attn layer单层全注意力 |
|---|---|---|---|---|---|---|---|
| 1K | 170 ms | 6,041 tok/s | 23.6% | 6.3% | 56.1% | 0.333 ms | 0.255 ms |
| 4K | 333 ms | 12,299 tok/s | 23.1% | 6.3% | 49.5% | 0.848 ms | 0.662 ms |
| 8K | 593 ms | 13,814 tok/s | 22.7% | 7.5% | 46.9% | 1.537 ms | 1.454 ms |
| 32K | 3051 ms | 10,741 tok/s | 15.8% | 37.6% | 30.2% | 6.005 ms | 41.046 ms |
Percentages are shares of prefill compute time with collectives excluded, matching the decode table. Full data including collectives and per-token cost: prefill-composition.csv.
百分比是剔除集合通信后占 prefill 计算时间的比例， 与 decode 那张表口径一致。 含集合通信与每 token 成本的完整数据： prefill-composition.csv。
Per input token the three mechanisms separate cleanly. KDA converges to 12.6 µs/token and MoE to 24.2 µs/token — both linear, both flattening once the fixed per-layer overheads are amortized. Full attention falls to 3.9 µs/token at 4K and then turns back up to 30.1 µs/token at 32K, which is what quadratic cost looks like once the quadratic term overtakes the constant one.
按每个输入 token 看， 三种机制分得很干净。 KDA 收敛到 12.6 µs/token， MoE 收敛到 24.2 µs/token—— 都是线性的， 固定的每层开销摊薄后就走平。 全注意力在 4K 降到 3.9 µs/token， 然后在 32K 反弹到 30.1 µs/token， 这正是二次项超过常数项之后二次成本的样子。
One kernel carries all of it. The MLA prefill attention kernel goes from 18.5 ms at 8K to 916.1 ms at 32K — about 50× for 4× the input — at which point it alone is 30% of all prefill GPU time. The KDA chunked kernels over the same span go from 41.2 ms to 176.9 ms, 4.3×, which is linear to within the noise. Section 06 takes that 50× apart; it is not what it looks like.
这一切由一个核函数承担。 MLA 的 prefill 注意力核函数从 8K 的 18.5 ms 涨到 32K 的 916.1 ms—— 输入 4 倍， 时间约 50 倍—— 到那时它一个就占了整个 prefill GPU 时间的 30%。 同一区间里 KDA 的分块核函数从 41.2 ms 涨到 176.9 ms， 4.3 倍， 在噪声范围内是线性的。 第 06 节会把这 50 倍拆开， 它并不是看上去那样。
Linear attention is not free at small L, it is merely flat. A chunked linear-attention prefill has to materialize and pass a state matrix between chunks and run secondary GEMMs whose cost is set by the head dimension, not the sequence length. Quadratic attention has a much smaller constant. So at small L the constant wins and the asymptotics have not started paying yet; the crossing point is wherever the quadratic term catches the linear constant, and for these dimensions that is around 8K in prefill and around 4K in decode. Below those lengths, replacing three quarters of the layers with KDA buys memory, not time.
线性注意力在 L 很小时并不是免费的， 它只是平。 分块线性注意力的 prefill 必须在 chunk 之间物化并传递状态矩阵， 还要跑一批二级 GEMM， 它们的成本由 head 维度决定， 与序列长度无关。 二次注意力的常数项则小得多。 所以 L 小的时候常数项占上风， 渐进优势还没开始兑现； 交叉点就在二次项追上线性常数的地方， 对这组维度来说 prefill 大约在 8K， decode 大约在 4K。 在这些长度以下， 把 3/4 的层换成 KDA 买到的是显存， 不是时间。
Taking the 50× apart: the two chunks run different MLA拆开那 50 倍： 两个 chunk 跑的不是同一种 MLA
A 32K prefill is two chunks of 16384. Splitting its trace at the forward-pass boundary and comparing kernel by kernel gives an unusually clean result: every kernel is identical between the two chunks except attention. KDA's chunk kernel is 68.84 against 68.88 ms, MoE's reduction 34.72 against 34.73. The kernels chunk 2 runs and chunk 1 does not total 8.9 ms — 1% of the 783 ms gap between them. The entire difference sits in one kernel, launched the same 24 times in both.
32K 的 prefill 是两个 16384 的 chunk。 把它的 trace 按前向边界切开、逐核函数对比， 结果异常干净： 除注意力之外每个核函数都一样。 KDA 的分块核 68.84 对 68.88 ms， MoE 的归约 34.72 对 34.73。 chunk 2 有而 chunk 1 没有的核函数加起来 8.9 ms—— 只占两者 783 ms 差距的 1%。 全部差异都在同一个核函数里， 而且两边都是 24 次派发。
The launch geometry recorded in the trace explains it. extend_attention_fwd launches (batch, heads, cdiv(max_extend_len, BLOCK_M)), and on gfx950 _get_block_sizes_for_extend_attention picks BLOCK_M = 128 only when 128 < Lq <= 256, otherwise 64. Both chunks extend the same 16384 tokens, so a grid of [1, 12, 128] versus [1, 12, 256] pins Lq at 192 in chunk 1 and 576 in chunk 2 — the decompressed MHA form and the absorbed latent form respectively.
trace 里记录的启动几何解释了这件事。 extend_attention_fwd 的 grid 是 (batch, heads, cdiv(max_extend_len, BLOCK_M))， 而在 gfx950 上 _get_block_sizes_for_extend_attention 只在 128 < Lq <= 256 时取 BLOCK_M = 128， 否则取 64。 两个 chunk 扩展的都是同样的 16384 个 token， 所以 [1, 12, 128] 与 [1, 12, 256] 的差别就把 Lq 钉死在 chunk 1 的 192 和 chunk 2 的 576—— 分别是解压后的 MHA 形式和吸收后的 latent 形式。
| chunk 1 · no prefixchunk 1 · 无前缀 | chunk 2 · 16K prefixchunk 2 · 16K 前缀 | ratio比值 | |
|---|---|---|---|
| MLA formMLA 形式 | decompressed MHA解压后的 MHA | absorbed latent吸收后的 latent | — |
| Lq / Lv | 192 / 128 | 576 / 512 | 3.40× |
| BLOCK_M · grid | 128 · [1,12,128] | 64 · [1,12,256] | — |
| query-key pairsquery-key 对数 | 1.34e8 | 4.03e8 | 3.00× |
| FLOP per layer每层 FLOP | 1.03e12 | 1.05e13 | 10.2× |
| ms per layer每层耗时 | 2.76 | 35.41 | 12.8× |
| achieved达到的算力 | 373 TFLOP/s | 297 TFLOP/s | 0.80× |
A first pass through this data blamed the super-quadratic 8K→32K growth on kernel efficiency, and that was wrong — it assumed both chunks ran the same arithmetic. They do not. Chunk 2 performs 10.2× the FLOPs at 1.26× lower efficiency, which is 12.8× the time. The kernel is behaving reasonably; what costs is the choice of form.
第一轮读这份数据时， 把 8K→32K 超出二次的增长记在了核函数效率上， 那是错的—— 它默认两个 chunk 做的是同样的算术。 并不是。 chunk 2 做了 10.2 倍的 FLOP、效率只低 1.26 倍， 合起来就是 12.8 倍的时间。 核函数的表现是合理的； 贵的是形式的选择。
Why does it switch? Once a prefix exists, MLA moves to the absorbed form and attends directly over the 576-dimensional latent, which avoids decompressing the prefix KV into 320 dimensions per head. For decode that is unambiguously right: one query against a very long key sequence, so a decompression could never be amortized. For a prefill chunk carrying 16384 queries the trade inverts — you pay 3.4× the FLOPs on every query-key pair to avoid a decompression that 16384 queries would have amortized many times over.
为什么会切换？ 一旦存在前缀， MLA 就换成吸收形式， 直接在 576 维的 latent 上做注意力， 省掉把前缀 KV 解压成每头 320 维的那一步。 对 decode 来说这毫无疑问是对的： 一个 query 面对很长的 key 序列， 解压的成本永远摊不掉。 但对一个带着 16384 个 query 的 prefill chunk， 取舍就反了—— 为了省一次能被 16384 个 query 反复摊薄的解压， 每个 query-key 对要多付 3.4 倍的 FLOP。
SGLang's chunked prefix cache is exactly the mechanism for this: it runs the prefix portion in the decompressed form instead. It is gated on the attention backend — CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS lists flashinfer, fa3, fa4, flashmla, cutedsl_mla and cutlass_mla — and this configuration runs triton, so maybe_disable_chunked_prefix_cache turns it off at load time without the recipe ever asking for it. The server log confirms it: the line "Chunked prefix cache is turned on" never appears. Every prefill past one chunk therefore pays the absorbed form's FLOP bill.
SGLang 的 chunked prefix cache 正是为此存在的机制： 它让前缀那部分改用解压形式跑。 它由注意力后端决定是否可用—— CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS 里是 flashinfer、fa3、fa4、flashmla、cutedsl_mla 和 cutlass_mla—— 而这套配置用的是 triton， 于是 maybe_disable_chunked_prefix_cache 在加载时就把它关掉了， 配方本身从没要求过。 服务器日志可以印证： 「Chunked prefix cache is turned on」这行从未出现。 因此每一次超过一个 chunk 的 prefill， 都在为吸收形式的算力账单买单。
Attention is 30.0% of GPU time in the 32K trace but only 0.55% of its slices — 48 out of 8727 — and 93% of that time sits in the last 24 of them, between t≈1.5 s and 3.05 s of a 3.06 s trace. Zoom anywhere in the first 2.8 seconds and attention is effectively invisible while MoE and KDA carpet the timeline with thousands of sub-millisecond slices. Sort by total duration; visual density is measuring slice count, not time.
在 32K 这份 trace 里， 注意力占 GPU 时间的 30.0%， 却只占切片数量的 0.55%——8727 个里只有 48 个—— 而且其中 93% 的时间集中在最后 24 个上， 位于 3.06 秒 trace 的 t≈1.5 到 3.05 秒之间。 在前 2.8 秒里随便放大， 注意力几乎是隐形的， 而 MoE 和 KDA 用成千上万个亚毫秒切片铺满了时间轴。 要按总时长排序； 视觉密度衡量的是切片数量， 不是时间。
One wave, one kernel一个波， 一个核函数
The claim that the KDA recurrence is latency-bound rather than bandwidth-bound deserves evidence below the kernel-timing level, so the kernel was thread-traced. rocprofv3 --att captures an instruction-level Advanced Thread Trace; the ROCprof Trace Decoder turns it into a directory that ROCprof Compute Viewer opens.
「KDA 递推是延迟受限而非带宽受限」这个论断值得有核函数计时层面以下的证据， 所以给这个核函数做了 thread trace。 rocprofv3 --att 抓的是指令级的 Advanced Thread Trace， 再由 ROCprof Trace Decoder 解成一个可以被 ROCprof Compute Viewer 直接打开的目录。
The two panels say opposite-sounding things that are both true. Inside a wave the kernel is ALU-heavy and only 30% stalled — each wave is working. But only 48 waves exist, so 81% of the compute units never see this dispatch at all.
两幅图说的话听起来相反， 但都成立。 在一个波内部， 这个核函数是 ALU 主导的、只有 30% 的时间在 stall—— 每个波都在干活。 但一共只有 48 个波， 所以 81% 的计算单元根本没参与这次派发。
This resolves the apparent contradiction in section 03. The kernel is not slow because it waits on memory; a traced wave spends 55.7% of its cycles in vector ALU doing the per-K decay and the delta-rule update on a [128 × 128] fp32 state. It is slow relative to the bytes it moves because the grid at batch 1 is batch × heads × V-blocks = 1 × 12 × 4, and no amount of per-wave tuning changes that. The practical consequence: KDA decode cost is a fixed per-layer overhead that only amortizes with batch size — the same 6.5 µs serves one sequence or, up to the occupancy limit, many.
这解开了第 03 节里表面上的矛盾。 这个核函数慢， 不是因为它在等内存； 被追踪的波有 55.7% 的周期花在向量 ALU 上， 对一个 [128 × 128] 的 fp32 状态做 per-K 衰减和 delta-rule 更新。 它相对于搬运的字节数显得慢， 是因为批大小 1 时 grid 是 batch × heads × V-blocks = 1 × 12 × 4， 再怎么调单波效率也改变不了这一点。 实际后果是： KDA 的解码成本是一项固定的每层开销， 只能靠批大小摊薄—— 同样的 6.5 µs， 服务一条序列， 或者在占用率上限之内服务很多条。
Two details cost real time. Whole-server tracing is not possible: rocprofv3 per-dispatch interception across 8 TP ranks drove the scheduler past its 300 s watchdog with no output, and runtime --attach injected successfully but produced nothing. The trace therefore comes from a standalone reproduction at the server's exact per-GPU shapes — sound here precisely because the kernel's launch geometry does not depend on context length. Second, the default --att-target-cu captures nothing, because 48 workgroups on 256 CUs will usually miss any particular one; --att-shader-engine-mask 0xFF with --att-consecutive-kernels 8 is what makes waves land in the trace.
有两个细节花了不少时间。 整机 trace 做不到： rocprofv3 在 8 个 TP rank 上逐派发拦截， 把 scheduler 拖过了 300 秒看门狗， 而且没有任何输出； 运行时 --attach 注入成功但什么也没产出。 所以这份 trace 来自一个与服务器完全同形状的独立复现—— 在这里成立， 恰恰是因为这个核函数的启动几何与上下文长度无关。 第二， 默认的 --att-target-cu 什么也抓不到， 因为 256 个 CU 上只有 48 个 workgroup， 通常落不到指定的那一个； 要靠 --att-shader-engine-mask 0xFF 配合 --att-consecutive-kernels 8 才能让波出现在 trace 里。
The decoded trace is archived at att/rcv-ui-output/ — per-wave JSON for all eight shader engines, occupancy and wave-state series, and the Triton source correlation — alongside the per-instruction table in instruction-stats.csv.
解码后的 trace 归档在 att/rcv-ui-output/—— 包含全部八个 shader engine 的逐波 JSON、occupancy 与 wave-state 序列、以及 Triton 源码对应关系—— 逐指令表在 instruction-stats.csv。
Three ways the measurement lied测量骗了我三次
Each of these produced a plausible, wrong number that survived until something else contradicted it. They are recorded because the corrections are more transferable than the results.
这三次都产生了一个看起来合理、实际错误的数字， 直到别的证据与它矛盾才被发现。 记录下来， 是因为这些修正比结果本身更有迁移价值。
1 · Collectives that are not communication1 · 不是通信的集合通信
The first eager decode profile put 135 of 156 ms per step in cross_device_reduce. Taken at face value, TP communication would be 87% of decode. It is not: at batch 1 the all-reduce payload is 7168 bf16 values, 14 KB, which no interconnect takes 600 µs to move. The kernel busy-waits on its peers, so its duration measures the skew between ranks, and in eager mode that skew is the host launch jitter accumulated over the ~18 kernels between consecutive collectives, multiplied by 187 collectives per step.
第一份 eager decode profile 把每步 156 ms 里的 135 ms 记在了 cross_device_reduce 上。 照字面理解， TP 通信占了解码的 87%。 并非如此： 批大小 1 时 all-reduce 的载荷是 7168 个 bf16， 14 KB， 任何互连都不需要 600 µs 去搬。 这个核函数在自旋等待对端， 所以它的时长量的是各 rank 之间的偏移； eager 模式下这个偏移就是相邻两次集合通信之间约 18 次 kernel launch 累积的主机抖动， 再乘以每步 187 次集合通信。
The correction is to exclude them from the decode composition and to calibrate the exclusion against graphs-on latency, which is what the 0.93–0.97 ratio in section 02 is for. Prefill is the opposite case and the contrast is instructive: there a chunk all-reduces 235 MB per layer and the surrounding kernels are millisecond-scale, so the same kernel now reports something real and is reported.
修正办法是把它们从 decode 的构成里剔除， 并用打开 graph 后的实测延迟去校准这次剔除—— 第 02 节那个 0.93–0.97 的比值就是干这个用的。 Prefill 是相反的情形， 对比很有启发： 那里一个 chunk 每层要 all-reduce 235 MB， 周围的核函数都是毫秒级的， 于是同一个核函数报告的就是真实开销， 也就照实计入。
2 · The probe changed the thing it measured2 · 探针改变了被测对象
The first prefill pass put the GPU at 29% busy during a 1K prefill and 48% at 4K, which reads as a host-launch-bound regime and would have been an interesting finding. It was an artifact of the record_function ranges: with them compiled in, 4K TTFT was 692 ms; without, 333 ms. At 32K both were about 3.05 s. Annotations are negligible against a saturated GPU and are not negligible against an idle one, and the shape of that error is exactly the shape of the conclusion it produced.
第一轮 prefill 测下来， 1K 时 GPU 只有 29% 忙、4K 时 48%， 读起来像是主机发射受限， 而且本来会是个有意思的发现。 它其实是 record_function 区间造成的假象： 带着区间时 4K 的 TTFT 是 692 ms， 去掉后是 333 ms； 而 32K 两种情况都是约 3.05 s。 注解对饱和的 GPU 可以忽略， 对空闲的 GPU 不能忽略， 而这个误差的形状恰好就是它导出的那个结论的形状。
Re-measured on a server built without the annotations, summed kernel time is within 1% of TTFT at every prefill size — the GPU is saturated even at 1K. Kernel durations were never affected, so the composition numbers stood; only the wall-clock column moved. The general form: when an instrument's overhead lands in the same place as the effect you are looking for, the measurement will confirm you.
在一台不带注解的服务器上重测， 每个 prefill 尺寸的核函数时间之和都落在 TTFT 的 1% 以内—— 连 1K 时 GPU 都是饱和的。 核函数时长从头到尾没有受影响， 所以构成数字站得住， 动的只是墙钟那一列。 一般化的说法是： 当仪器的开销恰好落在你要找的效应所在的位置时， 测量就会替你确认它。
3 · A profiler that cannot see the work3 · 看不见工作的 profiler
The very first decode profile looked clean — a well-formed trace, plausible kernels, no errors — and reported 1.8 ms of GPU time per step against a known step time near 26 ms. Nothing in the tool signals that a HIP graph replay is invisible to it; the trace simply contains what it can see. The tell was arithmetic, not a warning: the kernels present were 5 attention dispatches per step, which matches the 5-layer draft model, not a 93-layer target model.
最早那份 decode profile 看起来很干净—— trace 结构完整、核函数合理、没有报错—— 报告每步 1.8 ms 的 GPU 时间， 而已知每步约 26 ms。 工具本身不会提示「HIP graph 重放对我不可见」， trace 里就只有它能看到的东西。 暴露问题的是算术而不是警告： trace 里每步有 5 次注意力派发， 对得上 5 层的 draft 模型， 对不上 93 层的目标模型。
The habit worth keeping is to check a profile against an independently known total before reading anything else out of it. Every one of these three was caught by a number that had to add up and did not.
值得保留的习惯是： 在从一份 profile 里读出任何结论之前， 先拿它和一个独立已知的总量对一下。 上面这三次， 每一次都是被一个「本该对上却没对上」的数字抓出来的。
What to fix on MI355XMI355X 上该修什么
Ranked by measured headroom rather than by how interesting the mechanism is.
按实测的可优化空间排序， 而不是按机制本身有多有趣。
| Target目标 | Measured today当前实测 | Why there is room为什么有空间 | Prize收益 |
|---|---|---|---|
| MLA decode KV scanMLA 解码 KV 扫描 | 1.64 TB/s, 20.6% of peak; 17.61 ms of a 40.10 ms step at 1M1.64 TB/s， 峰值的 20.6%； 1M 时占 40.10 ms 中的 17.61 ms | Pure streaming read of a contiguous latent cache — the one access pattern HBM3E is built for对连续 latent 缓存的纯流式读取—— 正是 HBM3E 最擅长的访问模式 | −12 ms/step at 1M1M 时每步 −12 ms |
| MLA prefill attentionMLA prefill 注意力 | 916 ms at 32K, 30% of prefill; ~50× growth for 4× input32K 时 916 ms， 占 prefill 的 30%； 输入 4 倍、时间约 50 倍 | Growth outruns the quadratic expectation between 8K and 32K, so part of it is kernel efficiency, not algorithm8K 到 32K 之间的增长超过二次预期， 说明其中一部分是核函数效率而非算法 | TTFT above 32K32K 以上的 TTFT |
| MoE routing at low batch低批量下的 MoE 路由 | 2959 µs/step, 28.1% of MoE, versus 1439 µs for the actual experts每步 2959 µs， 占 MoE 的 28.1%， 而真正的专家只有 1439 µs | Gate, top-k, sort and scatter cost per token regardless of sparsity — pure fixed overhead at batch 1gate、top-k、排序、scatter 的每 token 成本与稀疏度无关—— 批大小 1 时是纯固定开销 | up to 7% of decode最多解码的 7% |
| KDA decode occupancyKDA 解码占用率 | 48 of 256 CUs, 0.24 TB/s effective, 6.5 µs per layer256 个 CU 中占 48 个， 有效 0.24 TB/s， 每层 6.5 µs | Grid is batch × heads × V-blocks; nothing to gain at batch 1, but it amortizes as batch growsgrid 是 batch × heads × V-blocks； 批大小 1 时无从优化， 但会随批量增大而摊薄 | batch, not kernel靠批量， 不靠核函数 |
The ordering carries the real message. The KDA kernels — the novel part of the architecture, and the part it is easiest to get excited about optimizing — are last, and the honest entry in their prize column is "not a kernel problem". The first two rows are both the same conventional attention path that every dense model also has, and together they are where the machine is being wasted.
这个排序本身就是结论。 KDA 的那些核函数—— 架构里最新颖、也最容易让人想去优化的部分—— 排在最后， 而它们收益一栏诚实的写法是「这不是核函数的问题」。 前两行都是同一条常规注意力路径， 每个稠密模型也都有它； 机器被浪费掉的地方就在这两行。
Reproducing this复现
Four server configurations, because no single one can produce all four numbers. The composition needs eager mode (the profiler is blind to graph replay); the latency anchor needs graphs on; prefill composition needs the ranges compiled in; and the prefill wall-clock reference needs them compiled out, for the reason in section 07.
需要四种服务器配置， 因为没有哪一种能同时给出全部四类数字。 构成需要 eager 模式（profiler 看不见 graph 重放）； 延迟锚点需要打开 graph； prefill 构成需要编入区间； 而 prefill 的墙钟参考值需要去掉区间， 原因见第 07 节。
| Produces产出 | Server服务器 | Driver驱动脚本 |
|---|---|---|
| decode compositiondecode 构成 | GRAPH=off RANGES=0 | sweep2.sh ctx |
| decode latency anchordecode 延迟锚点 | GRAPH=on RANGES=0 | rp3_drive.py --max-new 40 |
| prefill compositionprefill 构成 | GRAPH=on RANGES=1 | sweep-prefill.sh pf |
| prefill wall clockprefill 墙钟 | GRAPH=on RANGES=0 | prefill_ttft.py --reps 3 |
| the name map名字映射 | GRAPH=off RANGES=1 | name_map.py traces/<tag> |
| the thread tracethread trace | none — standalone无—— 独立进程 | rocprofv3 --att … kda_micro.py |
All of it, with the exact flags, in README.md; the harness itself is under scripts/, and the one-line-per-site instrumentation is sglang-k3-profiler-ranges.patch.
完整命令与参数见 README.md； 测量脚本在 scripts/， 每处一行的插桩补丁是 sglang-k3-profiler-ranges.patch。
All nine chrome traces — five decode points and four prefill points, rank TP0 — are in a public gist, 52 MB, openable directly in Perfetto without decompressing. It also carries kernel-inventory.csv: every distinct kernel, which block launches it, how many times per forward pass, and whether any two blocks share it.
全部九份 chrome trace—— 五个 decode 点和四个 prefill 点， rank TP0—— 都在一个公开 gist 里， 共 52 MB， 不用解压就能在 Perfetto 里直接打开。 里面还有 kernel-inventory.csv： 每个不同的核函数、由哪个块发起、每次前向发起多少次、以及有没有被两个块共用。
Two costs to plan for. Prefilling 1M tokens takes 1462 s at chunked-prefill 16384, so the two long decode points are most of the wall time in the whole study. And the first call at any new shape runs roughly twice the warm one, because aiter picks a GEMM configuration on the fly for shapes it has not seen — every sweep here warms up first and every TTFT is the best of three.
有两项成本要预留。 在 chunked-prefill 16384 下 prefill 一百万 token 要 1462 秒， 所以两个长上下文的 decode 点占了整个研究的大部分墙钟时间。 另外任何新形状的第一次调用大约是热身后的两倍， 因为 aiter 会为没见过的形状现场挑选 GEMM 配置—— 这里每次扫描都先热身， 每个 TTFT 都取三次里的最好值。
Epilogue尾声
Three of every four layers are linear, and they cost between 8.8% and 23.6% of the time depending on which phase you are in and how long the context is. The architecture does what it claims at long context: 74% of the layers hold their absolute cost constant from 4K to 1M while the remaining 26% grow by a factor of 6.8 and end up taking more than half the step.
每四层里有三层是线性的， 而它们占用的时间在 8.8% 到 23.6% 之间， 取决于你在哪个阶段、上下文有多长。 架构在长上下文上确实兑现了它的承诺： 74% 的层从 4K 到 1M 绝对成本保持不变， 剩下 26% 增长 6.8 倍， 最终吃掉了超过一半的每步时间。
What the measurement adds beyond that is a threshold and a warning. The threshold: a KDA layer is only cheaper than a full-attention layer above roughly 4K in decode and 8K in prefill; below that, the 3:1 layout is buying KV-cache memory and paying for it in time. The warning: the part of a KDA layer that is actually KDA is an eighth of it, and the part of decode that is actually attention-of-any-kind is a third at 4K. It is easy to spend a quarter's optimization budget on the mechanism with the interesting name and leave the streaming read that owns 44% of a 1M decode step running at a fifth of the bandwidth it could have.
这次测量在此之外多给了一个阈值和一个警告。 阈值： 一层 KDA 只有在 decode 约 4K 以上、prefill 约 8K 以上时才比一层全注意力便宜； 在那以下， 3:1 的排布买到的是 KV cache 显存， 代价是时间。 警告： 一层 KDA 里真正属于 KDA 的部分只有八分之一， 而 4K 解码里真正属于「任何一种注意力」的部分只有三分之一。 把一个季度的优化预算花在名字最有意思的那个机制上、同时让那个占据 1M 解码步 44% 的流式读取以五分之一的带宽跑着—— 这是很容易发生的事。