What a tuned RMSNorm kernel actually costs on Hopper在 Hopper 上， 一个调优过的 RMSNorm kernel 到底要花多少

01Why measure this为什么要测

We are about to implement RMSNorm forward and backward on MI355X. The question that decides whether that implementation is finished is not "does it run" but "how close to the memory system is it, and how close should it be." That number has to come from somewhere.我们即将在 MI355X 上实现 RMSNorm 的前向与反向。 判断这个实现是否算完成的问题， 不是"能不能跑"， 而是"离显存系统还有多远， 以及本来应该有多近"。 这个数字总得有个来源。

Quack's CuTe DSL implementation is a reasonable answer. It is a mature, well-tuned kernel on hardware whose memory system is well understood, written by people optimising for exactly this. Measuring it carefully gives the MI355X work a target rather than a vibe — and, more usefully, tells us which part of RMSNorm is hard. Those turn out to be very different for forward and backward.Quack 的 CuTe DSL 实现是个合理的参照。 它是一个成熟、 调优良好的 kernel， 跑在显存系统被充分理解的硬件上， 由专门做这件事的人写成。 认真测一遍， 能给 MI355X 的工作一个明确目标而不是一种感觉； 更有用的是， 它会告诉我们 RMSNorm 究竟哪一部分难。 而前向和反向的答案完全不同。

RMSNorm is memory bound. Every row is read, reduced, scaled and written; the arithmetic is about one operation per byte, so quoting TFLOPS would report a number that says nothing about the kernel. The throughput unit that means something here is GB/s, and the efficiency reading is that figure over the machine's measured roofline.RMSNorm 是带宽受限算子。 每一行被读入、 归约、 缩放、 写出， 算术强度约为每字节一次运算， 所以报 TFLOPS 只会给出一个与 kernel 好坏无关的数字。 这里有意义的吞吐单位是 GB/s， 而效率读数是它除以本机实测的 roofline。

Both are reported throughout, and they answer different questions. Latency and GB/s say what the kernel costs and are what a future MI355X run gets placed directly beside. Percent of roofline says how much of that machine the kernel extracted, and is the only figure that survives a change of memory system. Quoting either one alone is how benchmark records become unusable a month later.两者本文都会给出， 它们回答的是不同问题。 延迟和 GB/s 说明这个 kernel 要花多少代价， 也是未来 MI355X 的结果可以直接并排放的东西。 占 roofline 的百分比说明 kernel 榨出了那台机器的多少， 也是换一套显存系统后唯一还成立的数字。 只报其中之一， 正是 benchmark 记录在一个月后变得没法用的原因。

02What a clean run requires一次干净的测量需要什么

A percent-of-peak number is a ratio, and the denominator is measured, not quoted from a spec sheet. The harness probes the device three ways — a copy, a pure write, and a two-read-one-write pattern — and takes the best as the achievable roofline. That choice matters later, because the two GPUs did not peak on the same probe.占峰值的百分比是一个比值， 而分母是实测出来的， 不是从规格表上抄的。 测试框架用三种方式探测设备—— copy、 纯写、 以及 2 读 1 写—— 取最优值作为可达上限。 这个选择在后面会变得重要， 因为两张卡的峰值并不出现在同一种探测上。

Three disciplines make the ratio trustworthy. Every cell is validated against an fp32 reference before it is timed, so a fast wrong kernel cannot score. Inputs rotate through cloned buffers with a cache-evicting copy between calls, so small shapes cannot quietly serve from L2. And the bandwidth probe is repeated at the end of the sweep: if the closing number has drifted from the opening one, a co-tenant arrived and every ratio in the run is suspect.三条纪律让这个比值可信。 每个测试点在计时之前都与 fp32 参考实现校验， 所以一个又快又错的 kernel 拿不到分数。 输入在多组克隆缓冲之间轮转， 两次调用之间插入一次冲刷缓存的拷贝， 所以小 shape 无法悄悄从 L2 供数。 带宽探测在 sweep 结束时再做一次： 如果收尾值相对开场值发生漂移， 说明中途来了邻居， 这一轮所有比值都要打问号。

GPUs were chosen by inspection, not by hope: 0 MiB allocated and 0% utilisation, re-checked immediately before launch. On the H200 node five of the other seven cards were in use by someone else, which is exactly the situation the canary exists for.GPU 是看着挑的， 不是赌来的： 显存占用 0 MiB、 利用率 0%， 并在启动前立刻复查一次。 H200 那台机器上另外七张卡有五张正被别人使用—— 这正是哨兵存在的意义。

Run environment, read back from each run's environment.json. Note the two hosts do not share a toolchain, and did not peak on the same bandwidth probe.运行环境， 取自各自的 environment.json。 注意两台机器的工具链并不相同， 峰值也不出现在同一种带宽探测上。

03Two config paths, only one of them searched两条配置路径， 只有一条真的搜索过

Quack exposes RMSNorm twice. rmsnorm_fwd and rmsnorm_bwd pick a launch configuration from a hand-written analytical ladder — a few thresholds on row width that select thread count, threads per row and cluster size. rmsnorm_fwd_tuned and rmsnorm_bwd_tuned hand the same decision to @autotune, which compiles and benchmarks the whole config space and caches the winner on disk.Quack 把 RMSNorm 暴露了两次。 rmsnorm_fwd 和 rmsnorm_bwd 从一条手写的解析式阶梯里挑 launch config—— 几个基于行宽的阈值， 决定线程数、 每行线程数和 cluster 大小。 rmsnorm_fwd_tuned 和 rmsnorm_bwd_tuned 则把同一个决策交给 @autotune， 由它编译并实测整个配置空间， 再把赢家缓存到磁盘。

This distinction is easy to miss and it changes the headline. The un-tuned entry points are what a caller gets by default, so they are the honest "out of the box" number. The tuned ones are the ceiling. Both were measured here, in the same sweep, against the same roofline.这个区别很容易被忽略， 而它会改变结论。 未 tune 的入口是调用者默认拿到的东西， 所以那是诚实的"开箱"数字； tune 过的则是上限。 这里两者都测了， 在同一次 sweep 里， 对同一个 roofline。

04Forward: already at the wall前向： 已经贴着墙了

The forward kernel saturates, and it does so from the default path. At the widest row the analytical config reaches 93.4% of achievable bandwidth on the H200 and 93.8% on the H100 — the same fraction of two very different memory systems, which is what a kernel doing its job looks like. In absolute terms: 266.5 us at 4029.5 GB/s on the H200, and 358.9 us at 2991.8 GB/s on the H100.前向是打满的， 而且是用默认路径打满的。 在最宽的行上， 解析式配置在 H200 上达到可达带宽的 93.4%， 在 H100 上达到 93.8%—— 两套差异很大的显存系统上相同的比例， 这正是一个称职的 kernel 应有的样子。 换成绝对量： H200 是 266.5 us / 4029.5 GB/s， H100 是 358.9 us / 2991.8 GB/s。

Searching does not improve on that. On the H100 autotune lands at 94.0%, indistinguishable from the default; on the H200 it picks a worse config and drops to 84.4%. Section 06 covers why a search can lose. For forward the practical conclusion is that the hand-written ladder is already right, and paying 20 s per cell to confirm it is a bad trade.搜索并不能改进它。 H100 上 autotune 落在 94.0%， 与默认无从区分； H200 上它挑了一个更差的配置， 掉到 84.4%。 搜索为什么会输见第 06 节。 对前向来说， 实用的结论是手写的那条阶梯已经是对的， 而为了确认这一点每个测试点付 20 秒并不划算。

The ramp below the wall is not inefficiency, it is arithmetic. A short row cannot amortise launch and reduction cost, so the small shapes report a low fraction because they are latency bound, not because the kernel is bad. The useful reading is where the curve reaches its plateau — around 32768x2048, and only the last two shapes are genuinely at the wall.墙以下的那段爬升不是低效， 而是算术。 短行摊不掉 launch 和归约的开销， 所以小 shape 的比例低是因为它们受延迟约束， 而不是 kernel 写得差。 有用的读法是看曲线在哪里进入平台—— 大约在 32768x2048， 而真正贴到墙上的只有最后两个 shape。

Forward, per shape, bfloat16. moved is the logical bytes the operation must touch. Latency is the median of 12 samples for each config path; best GB/s and % peak use whichever path was faster on that host.前向逐 shape 数据， bfloat16。 moved 是该操作必须触及的逻辑字节数。 延迟是两条配置路径各自 12 次采样的中位数； best GB/s 与 % peak 取该机器上更快的那条路径。

05Backward: the weaker half反向： 较弱的一半

Backward moves more bytes for the same row — it reads the input, the incoming gradient, the weight and the fp32 reciprocal standard deviation, and writes both the input gradient and a per-CTA fp32 partial weight gradient. It also does not finish inside the kernel: the partials are summed by PyTorch afterwards, so a real backward call is a CuTe kernel plus a reduction the kernel does not own.反向在同样的行上搬运更多字节—— 它要读输入、 上游梯度、 权重和 fp32 的 rstd， 并写出输入梯度以及每个 CTA 一份的 fp32 部分权重梯度。 它也不在 kernel 内部收尾： 这些部分和随后由 PyTorch 求和， 所以一次真实的反向调用是一个 CuTe kernel 加上一段不归它管的归约。

This is where the two config paths separate. On the default path backward stalls at 75.7% of achievable bandwidth on the H200 and 85.1% on the H100 at the widest row — visibly short of the low nineties forward manages on the same card and the same memory system.两条配置路径在这里分道。 走默认路径时， 最宽行上反向停在 H200 的 75.7% 与 H100 的 85.1%—— 明显低于同一张卡、 同一套显存系统上前向能做到的九十出头。

In absolute terms that is 493.4 us on the H200 and 595.1 us on the H100 to move 1536.2 MiB.换成绝对量， 搬运 1536.2 MiB 在 H200 上要 493.4 us， 在 H100 上要 595.1 us。

Searched, it closes almost all of that: 407.1 us and 540.6 us, which is 3956.5 GB/s and 2979.6 GB/s, or 91.7% and 93.6% of roofline. So the gap is not the algorithm and it is not the extra bytes — the same kernel reaches forward-class efficiency once it is allowed a configuration the ladder cannot express. What remains after that, a point or two, is the reduction that happens outside the kernel.而搜索之后， 这个缺口几乎被补平： 407.1 us 与 540.6 us， 也就是 3956.5 GB/s 与 2979.6 GB/s， 相当于 roofline 的 91.7% 与 93.6%。 所以缺口既不是算法造成的， 也不是多出来的那些字节造成的—— 同一个 kernel 一旦被允许使用阶梯表达不出的配置， 就能达到前向级别的效率。 剩下那一两个点， 才是发生在 kernel 之外的归约。

Backward, per shape, bfloat16. Same column meanings as the forward table. Note how much wider the analytical-versus-tuned latency gap gets in the last two rows.反向逐 shape 数据， bfloat16。 列含义同前向表。 注意最后两行里 analytical 与 tuned 的延迟差距明显拉大。

06Where autotune pays, and whyautotune 在哪里有回报， 以及为什么

Comparing the two config paths cell by cell gives a clean answer: forward has nothing left to win, and backward has a lot — but only at the wide rows.把两条配置路径逐点对比， 答案很干净： 前向已经无利可图， 反向则有相当大的空间—— 但只在宽行上。

The backward gain at the widest row is +17.5% on the H200 and +9.2% on the H100. That is not a mistuned threshold. Reading _for_hopper_bwd in quack/rmsnorm_config.py shows it returns a config with use_tma=False written as a literal, and it never passes smem_stages, so the dataclass default of 2 stands. The autotune winners on both hosts chose use_tma=True with smem_stages=3.最宽行上反向的收益是 H200 +17.5%、 H100 +9.2%。 这不是阈值没调好。 读 quack/rmsnorm_config.py 里的 _for_hopper_bwd 会看到它返回的配置里 use_tma=False 是字面量写死的， 并且它从不传 smem_stages， 于是 dataclass 的默认值 2 生效。 而两台机器上 autotune 选出的赢家都是 use_tma=True 配 smem_stages=3。

Read the other direction, the same plate says the analytical ladder is good. Across forward it is within noise of an exhaustive search, and at narrow rows it matches backward too. A search that costs 20 s per forward cell and up to 160 s per backward cell, to win nothing on most of the matrix, is a poor default — which is presumably why the un-tuned path is the default.反过来读， 同一张图也说明这条解析式阶梯是好的。 在前向上它与穷举搜索的差距落在噪声之内， 在窄行上反向也追平。 一次搜索前向每个测试点约 20 秒、 反向最多 160 秒， 却在矩阵的大部分上一无所获—— 作为默认路径并不划算， 这大概正是未 tune 的那条被设为默认的原因。

Autotune is not a free winautotune 不是白拿的

Across all 45 backward cells the search improves 21 of them on the H200 and 34 on the H100, but it also makes 8 and 2 respectively measurably worse, with a worst case of −16.7%. Forward regresses in 4 cells on each card, including the −10.7% at the widest H200 row quoted in section 04.在全部 45 个反向测试点里， 搜索在 H200 上改善了 21 个、 在 H100 上改善了 34 个， 但同时也分别让 8 个和 2 个明显变差， 最差的一例是 −16.7%。 前向在两张卡上各有 4 个测试点回退， 包括第 04 节引用的 H200 最宽行那 −10.7%。

The reason is a methodology mismatch, not a broken search. The autotuner scores candidates with its own L2-cold benchmark; this harness times steady-state calls through rotating buffers. Those two regimes reward different configurations — a config that wins when every access misses cache is not necessarily the config that wins in a training loop where the previous layer just left the cache warm. A search only optimises the metric it is given.原因是评测口径不一致， 而不是搜索坏了。 autotuner 用它自己的 L2-cold 基准给候选打分， 而这套 harness 计时的是经由轮转缓冲的稳态调用。 这两种口径奖励的是不同的配置—— 一个在"每次访问都不命中缓存"时胜出的配置， 未必是在"上一层刚把缓存捂热"的训练循环里胜出的那个。 搜索只会优化你交给它的那个指标。

The practical form of this: do not ship a tuned config without re-measuring it under the access pattern you actually run. The backward win at wide rows survives that check here, which is why it is reported as a finding; the forward "wins" do not.落到实践上： 不要在没有用真实访存模式复测的情况下就把某个 tuned 配置发出去。 这里宽行上的反向收益经受住了这道复核， 所以才作为结论报告； 前向那些"收益"没有。

07Dtype sensitivity at the widest row最宽行上的 dtype 敏感性

At 32768x8192 both kernels are fully bandwidth bound, which makes it the one shape where dtype effects are readable without latency confounding them.在 32768x8192 上两个 kernel 都完全进入带宽受限区间， 这使它成为唯一一个能在不被延迟干扰的情况下读出 dtype 效应的 shape。

Percent of achievable bandwidth at M=32768, N=8192, autotuned configs. Compare against the default-path figures quoted in the prose below. The weight column is the parameter dtype: matching the activations, or an fp32 master weight.M=32768、 N=8192 下占可达带宽的百分比， 使用 autotune 配置。 权重列是参数 dtype： 与激活一致， 或使用 fp32 主权重。

Tuned, there is barely any dtype sensitivity left: every cell above sits between the low nineties and 96%. That is worth pausing on, because on the default path there very much is one — backward at this shape runs at 75.7% with 16-bit activations on the H200 but 95.7% with fp32.调优之后， 几乎看不到 dtype 敏感性了： 上表每一格都落在九十出头到 96% 之间。 这一点值得停一下， 因为在默认路径上敏感性是明显存在的—— 同一个 shape 上， H200 的反向在 16 位激活下只有 75.7%， 而 fp32 下是 95.7%。

So the dtype dependence was never a property of the algorithm. It came from one configuration being wrong for narrow activations specifically: with fp32 rows the analytical config happens to be near-optimal, and with 16-bit rows — where more of the row fits in shared memory and there is more latency for a deeper pipeline and TMA to hide — it is not. Tuning removes the dependence entirely rather than reducing it, which is the signature of a config artifact rather than a bandwidth effect.所以这个 dtype 依赖从来就不是算法的属性。 它来自一个配置恰好只对窄激活是错的： fp32 行下解析式配置碰巧接近最优， 而 16 位行下—— 那里更多行能装进共享内存， 也有更多延迟可供更深的流水和 TMA 去掩盖—— 它就不是了。 调优是把这个依赖完全消除而不是减轻， 这正是"配置产物"而非"带宽效应"的特征。

The two forward cells in the eighties on the H200 are the autotune regressions from section 06, not a dtype effect; the analytical path reaches the low nineties on both of them.H200 上那两个落在 80% 段的前向格子是第 06 节说的 autotune 回退， 不是 dtype 效应； 解析式路径在这两格上都能到九十出头。

08What limits these numbers这些数字的边界

Three things would change a conclusion drawn from this data, so they belong next to it rather than in a footnote.有三件事会改变从这批数据得出的结论， 所以它们应该紧挨着数据， 而不是塞进脚注。

The two hosts do not share a toolchain两台机器的工具链不同

The H200 ran PyTorch 2.9.1+cu128 with CUTLASS DSL 4.6.1; the H100 ran 2.11.0+cu130 with 4.6.0. The Quack source is identical, so kernel-level differences are attributable, but any absolute microsecond gap between the cards carries a compiler and runtime component too. This is why the plates report percent of peak and not time.H200 上是 PyTorch 2.9.1+cu128 配 CUTLASS DSL 4.6.1； H100 上是 2.11.0+cu130 配 4.6.0。 Quack 源码完全一致， 所以 kernel 层面的差异是可归因的， 但两卡之间任何绝对微秒差里都混有编译器与 runtime 的贡献。 这正是图表报占峰值百分比而不是时间的原因。

The roofline is a different probe on each host两台机器的 roofline 来自不同探测

The H200 peaked at 4312.7 GB/s and the H100 at 3182.3 GB/s, but not on the same access pattern. The method is identical — best of three probes — yet the denominators are not shaped alike, so percent of peak is a within-host efficiency measure. Comparing H200 91% against H100 93% is comparing two ratios with different denominators, and the difference between them is not evidence about the kernel.H200 的峰值是 4312.7 GB/s， H100 是 3182.3 GB/s， 但并非来自同一种访存形态。 方法完全一致—— 三种探测取最优—— 但分母的形状不同， 所以占峰值百分比是一个机内效率指标。 拿 H200 的 91% 去比 H100 的 93%， 是在比两个分母不同的比值， 它们之间的差不构成关于 kernel 的证据。

The H200 node was sharedH200 整机是共享的

Our card was idle throughout, but five of the other seven held allocations. The closing bandwidth probe came back at 0.999 of the opening one against 0.999 on the exclusive H100 — inside the harness's quiet band, but at its edge. H200 figures carry more noise than H100 ones, and a difference of a percent or two between the cards should not be read as real.我们那张卡全程空闲， 但另外七张里有五张持有显存分配。 收尾的带宽探测是开场值的 0.999 倍， 而独占的 H100 是 0.999—— 落在框架的安静区间内， 但在边缘上。 H200 的数字比 H100 噪声更大， 两卡之间一两个百分点的差异不应被当真。

09What this sets as the target for MI355X这为 MI355X 定下了什么目标

The point of the exercise. Four things transfer, none of them a microsecond count.这才是这件事的意义所在。 有四点可以迁移， 其中没有一个是微秒数。

Forward should reach the low nineties as a percentage of a locally measured roofline. Two different Hopper memory systems both got there, so it is a property of the algorithm rather than of the hardware. A forward RMSNorm sitting at 70% on MI355X is not finished, and the gap is not explained by the architecture. The concrete anchors at 32768x8192 bf16, moving 1536.2 MiB: forward 266.5 us / 4029.5 GB/s on H200 and 358.9 us / 2991.8 GB/s on H100; backward 407.1 us / 3956.5 GB/s and 540.6 us / 2979.6 GB/s.前向应当达到本机实测 roofline 的九十出头。 两套不同的 Hopper 显存系统都做到了， 说明这是算法的属性而不是硬件的属性。 一个在 MI355X 上停在 70% 的前向 RMSNorm 不算完成， 而且这个缺口无法用架构来解释。 32768x8192 bf16、 搬运 1536.2 MiB 时的具体锚点： 前向 H200 266.5 us / 4029.5 GB/s， H100 358.9 us / 2991.8 GB/s； 反向 407.1 us / 3956.5 GB/s 与 540.6 us / 2979.6 GB/s。

Compare like this. Put the MI355X GB/s next to these directly — the logical byte count is hardware-independent, so the same shape moves the same bytes and the ratio of GB/s is a real throughput ratio. Then compute percent against an MI355X roofline measured by the same three probes, and compare that to the percentages here. If GB/s is lower but percent is equal, the memory system explains it and the kernel is fine.这样来比。 把 MI355X 的 GB/s 直接并排放在这些数字旁边—— 逻辑字节数与硬件无关， 同一个 shape 搬运同样多的字节， 所以 GB/s 的比值是真实的吞吐比。 然后用同样三种探测测出 MI355X 的 roofline 算出百分比， 再拿那个百分比和这里的百分比比较。 如果 GB/s 更低但百分比持平， 那就是显存系统的差异， kernel 本身没问题。

Backward should get there too — the gap is not inherent. This was the surprise. Backward moves more bytes and finishes with a reduction outside the kernel, so a few points of shortfall looked structural; measured, it is worth about one point once the configuration is right. Do not budget a large backward penalty in advance. If an MI355X backward sits well below its forward, suspect the config before the algorithm.反向也应该达到同样水平—— 那个缺口不是固有的。 这是最意外的一点。 反向搬运更多字节， 还要在 kernel 之外收尾归约， 所以看起来天然该差几个点； 但实测下来， 配置正确之后只差约一个点。 不要预先给反向留很大的性能预算。 如果 MI355X 上反向明显低于前向， 先怀疑配置， 再怀疑算法。

Do not let a config heuristic hardcode a memory-path knob. The single largest finding here is worth +17.5% on one shape and comes entirely from two constants. The AMD equivalents — whether to use the async copy path, and how deep to make the LDS pipeline — deserve to be in the search space from the start, even if the shipped default is a heuristic.不要让配置启发式把访存路径的旋钮写死。 这里最大的一个发现在某个 shape 上值 +17.5%， 而它完全来自两个常量。 AMD 侧的对应项—— 是否走异步拷贝路径、 LDS 流水做多深—— 应该从一开始就留在搜索空间里， 哪怕最终发布的默认值是一条启发式。

Measure the roofline on the machine, every run. A spec-sheet denominator would have hidden both the 10% drift on the shared H200 node and the fact that the two cards peak on different access patterns. Neither is exotic; both would have quietly corrupted a comparison.每一次运行都在本机测 roofline。 用规格表当分母的话， 共享的 H200 节点上那 10% 的漂移、 以及两张卡峰值出现在不同访存形态这件事， 都会被藏起来。 这两件事都不罕见， 而且都会悄悄毁掉一次比较。

10Reproducing this复现

Raw results, run logs, the exact commands and the harness patch are committed under data/quack-rmsnorm-hopper/. The figures on this page are generated from those CSVs by scripts/gen-rmsnorm-plates.mjs, so the prose and the plates cannot drift apart.原始结果、 运行日志、 完整命令和 harness 补丁都提交在 data/quack-rmsnorm-hopper/ 下。 本页的图表由 scripts/gen-rmsnorm-plates.mjs 从那些 CSV 生成， 所以正文与图表不会各说各话。

bash data/quack-rmsnorm-hopper/sweep-rmsnorm.sh <repo-root> <idle-gpu> <out-dir>

# regenerate this page's plates from the archived CSVs
node scripts/gen-rmsnorm-plates.mjs
node scripts/gen-rmsnorm-plates.mjs --check   # non-zero if the page is stale

The sweep script refuses to start on a GPU that is not idle, and prints the resolved quack.rmsnorm module path before it begins, because those are the two ways this measurement silently goes wrong.这个 sweep 脚本在 GPU 不空闲时会拒绝启动， 并在开始前打印解析到的 quack.rmsnorm 模块路径—— 因为这正是这类测量会悄悄出错的两个途径。