PRESERVE: Prefetch Weights and KV-Cache in Distributed LLM Serving
Authors: Ahmet Caner Yüzügüler, Jiawei Zhuang, Lukas Cavigelli | Affiliation: Huawei Zurich Research Center | PDF: PRESERVE_Prefetch_Weights_KV_Cache_LLM_Serving_2025.pdf
一句话总结
PRESERVE 在分布式 decode 中于 collective communication 期间从 HBM prefetch 权重与 KV-cache 至 L2，overlap 通信与内存读，商业加速器上最高 1.6× 端到端加速；考虑 prefetch 的最优 L2 104 MB 带来 1.25× perf/$。
核心贡献
- Comm-overlap prefetch：突破仅 fuse 相邻 GEMM+allreduce 的限制，KV 路径亦可 hide latency
- Graph optimization pass：编译期插入 prefetch 流、跟踪 L2 占用上限防 cache pollution
- Weight + KV 联合 prefetch：长上下文下 KV 读可超过 weight — 二者同等关键
- Decode memory-bound 分析：OI ~16 Op/word vs roofline >100 — HBM BW 主导
- Accelerator DSE：prefetch-aware 最优 on-chip cache 显著大于传统 8 MB 基线
关键数字
| 设置 | 结果 |
|---|---|
| End-to-end speedup | Up to 1.6× |
| Optimal L2 (with prefetch) | 104 MB (vs 8 MB baseline) |
| Performance per cost | 1.25× vs baseline design |
| Decode OI (typical) | ~16 Op/word |
| Example L2 sizes | GB200 126 MB, MI300X 256 MB L3 |
与 wiki 交叉引用
- Disaggregated Inference — 多设备 collective 与 serving 拓扑
- vLLM — KV-cache 布局与分布式 shard
- GEMM vs GEMV — decode 低 OI、memory-bandwidth bound
- Inference Capacity Trap — 设备 idle 于 comm 阶段的利用率损失
- Prefill Decode Divergence — decode 阶段 HBM 读主导
Citations
[1] PRESERVE_Prefetch_Weights_KV_Cache_LLM_Serving_2025.pdf — Yüzügüler et al. (2025) [2] preserve-prefetch-weights-kv-cache.md — 结构化摘录