Every µs Matters: Achieving Near Speed-of-Light Latency in GPU Collectives每个 μS 都至关重要：在 GPU 集合通信中实现接近光速的 LatencySiyuan Shen*Zürich, Switzerland siyuan.shen@inf.ethz.ch瑞士苏黎世 siyuan.shen@inf.ethz.chAnton Korzh NVIDIA Corporation Santa Clara, California akorzh@nvidia.comNVIDIA Corporation 加利福尼亚州圣克拉拉akorzh@nvidia.comJohn Bachan NVIDIA Corporation Santa Clara, California jbachan@nvidia.comNVIDIA Corporation 加利福尼亚州圣克拉拉jbachan@nvidia.comTiancheng Chen ETH ZürichZürich, Switzerland tiancheng.chen@inf.ethz.ch瑞士苏黎世 tiancheng.chen@inf.ethz.chArnav GoelNVIDIA CorporationSanta Clara, California arnavg@nvidia.com加利福尼亚州圣克拉拉 arnavg@nvidia.comLudwig SchneiderNVIDIA CorporationSanta Clara, California加利福尼亚州圣克拉拉lschneider@nvidia.comPouya KoushaNVIDIA CorporationSanta Clara, California加利福尼亚州圣克拉拉pkousha@nvidia.comZhenhao He NVIDIA Corporation Zürich, Switzerland瑞士苏黎世zhenhaoh@nvidia.comSylvain Jeaugey NVIDIA Corporation Grenoble, France法国格勒诺布尔sjeaugey@nvidia.comKamil Iskra NVIDIA Corporation Chicago, Illinois伊利诺伊州芝加哥kiskra@nvidia.comJeff R. Hammond NVIDIA Helsinki Oy Helsinki, Finland赫尔辛基，芬兰jeffpapers@nvidia.comTorsten Hoefler ETH Zürich Zürich, Switzerland瑞士苏黎世torsten.hoefler@inf.ethz.chAbstract摘要GPU collective communication is typically optimized for bandwidth, yet many emerging workloads are increasingly limited by latency. Long-context decode-heavy large language model (LLM) inference is a prime example, where serving large models requires multiple GPUs, and many small collectives lie directly on the critical path of token generation. Therefore, even µs of overhead can impact performance and cost. In this work, we study how to approach the hardware Speed-of-Light (SoL) lower bound for GPU collectives within a scale-up network. We identify key principles for near-optimal designs, including barrier-free synchronization and efficient use of symmetric memory and multicast. Building on NCCL’s device-side API, we develop lowlatency interfaces for constructing custom collective kernels and use them to implement new symmetric collectives in NCCL. Microbenchmarks show substantial latency reductions for small and medium messages, reducing overhead to within 7% of the absolute SoL lower bound. When integrated into real applications, these kernels improve inter-token latency and throughput in LLM inference and accelerate cuSOLVERMp,GPU集体通信通常针对带宽进行优化，但许多新兴工作负载正日益受到Latency的限制。长上下文、以decode为主的大型语言模型(LLM)Inference就是一个典型例子：为大模型提供服务需要多个GPU，而许多小型集体通信直接位于Token生成的关键路径上。因此，即使只有μS的开销，也会影响性能和成本。在这项工作中，我们研究了如何在 scale-up 网络内使 GPU 集体通信逼近硬件的光速(SoL)下界。我们识别出实现近似最优设计的关键原则，包括无barrier同步，以及对 symmetric memory和multicast 的高效利用。基于NCCL的设备端 API，我们开发了用于构建自定义集体通信kernel 的低Latency接口，并利用这些接口在NCCL中实现了新的 symmetric 集体通信。微基准测试表明，对于小型和中型消息，Latency 显著降低，将开销降至距离绝对 SoL下界仅7%之内。当集成到实际应用中时，这些kernel 改善了LLM Inference 中的 inter-token Latency和 Throughput，并加速了cuSOLVERMp，展示了其对 AI Inference 和传统 HPC 工作负载的双重收益。I. Introduction1. 引言Deep learning has driven rapid growth in GPU cluster scale and interconnect capability. As large language models (LLMs) scale, inference increasingly spans multiple GPUs, bringing collective communication onto the critical path of token generation. For example, serving DeepSeek-V3-class models [1] requires at least 8×H2oo GPUs [2]. The impact of collective communication is especially evident in decode-heavy workloads, where modern systems may generate millions of tokens per request in applications such as code generation and agent-style workflows [3], [4]. In these settings, even small communication overheads accumulate and directly impact the深度学习推动了 GPU 集群规模和互连能力的快速增长。随着Large Language Model(LLMs)不断扩展，Inference越来越多地跨越多个 GPU 进行，使得集合通信进入了 token 生成的关键路径。例如，提供 DeepSeek-V3 级别模型 [1]的服务至少需要 8×H200 GPU [2]。集合通信的影响在以 decode 为主的工作负载中尤为明显，在代码生成和 agent-style 工作流等应用中，现代系统每个请求可能生成数百万个 Tokens [3]，[4]。在这些场景下，即使是很小的通信开销也会不断累积，并直接影响Figure 1: Fig. 1: In long-context, small-batch tensor-parallel (TP) LLM inference, many small AllReduce operations lie on the critical path, making collective latency a crucial bottleneck. The microbenchmark shows that our NCCL low-latency kernel reduces small-message AllReduce latency relative to other implementations, approaching the speed-of-light (SoL) bound. This translates into lower inter-token latency (ITL) and higher cost savings for Llama-3.1-7oB inference.图1：图1：在长上下文、小批量张量并行(TP)LLM Inference中，许多小型AlIReduce 操作位于关键路径上，使得collective Latency 成为关键瓶颈。微基准测试表明，相较于其他实现，我们的 NCCL 低 Latency kernel 降低了小消息AllReduce Latency，并接近光速(SoL)界限。这转化为更低的 inter-token Latency(ITL)以及更高的 Llama-3.1-70B Inference 成本节省。quality of service. In addition, for long-context inference, the KV-cache memory grows with sequence length, and the batch size is often reduced to fit within device memory. As a result, collectives such as AllReduce are invoked frequently with relatively small message sizes during decoding, making latency, rather than bandwidth, the dominant bottleneck. Consequently, recent inference frameworks such as vLLM [3], SGLang [5],服务质量。此外，对于长上下文 Inference，KV-cache 内存会随着 Sequence Length 增长，而 batch size 往往会缩减以适应设备内存。因此，在解码过程中，诸如 AllReduce 之类的 collective会以相对较小的消息大小被频繁调用，使得Latency 而非带宽成为主要瓶颈。因此，近期的Inference 框架，如 vLLM [3]、SGLang [5]，and TensorRT-LLM [6] treat communication latency as a firstclass optimization target and implement custom low-latency GPU kernels for operations such as AllReduce.以及 TensorRT-LLM [6]，都将通信 Latency 视为首要优化目标，并为 AIllReduce 等操作实现了定制的低 Latency GPUkernels.Low-latency collectives are essential not only for LLM inference, but also for many traditional scientific and HPC applications. Many simulations and solvers perform frequent small global reductions within tightly synchronized phases, such as time-stepping loops and particle simulations. In these settings, collective latency lies on the critical path and can limit strong scaling. Prior work such as LLAMP shows that widely used HPC workloads, including MILC and LULESH, are measurably sensitive to collective latency [7]. At the same time, many GPU-accelerated scientific applications do not yet fully exploit GPU-native communication libraries. These observations indicate that low-latency GPU collectives can improve not only LLM inference, but also scalability and timeto-solution in traditional scientific workloads.低 Latency 集合通信不仅对 LLM Inference 至关重要，对许多传统科学与 HPC 应用同样如此。许多仿真和求解器会在紧密同步的阶段内频繁执行小规模全局归约，例如时间步进循环和粒子仿真。在这些场景中，集合通信Latency位于关键路径上，并且会限制强扩展。先前工作如LLAMP表明，广泛使用的HPC工作负载，包括MILC和LULESH，对集合通信Latency 具有可测量的敏感性[7]。与此同时，许多 GPU 加速的科学应用尚未充分利用原生面向 GPU 的通信库。这些观察表明，低 Latency 的 GPU 集合通信不仅可以提升 LLM Inference，也可以改进传统科学工作负载的可扩展性和求解时间。Despite this growing recognition, existing approaches still leave performance on the table. We observed that even the best available implementations often remain above the “speed-of-light” (SoL) bound, by which we mean the absolute hardware lower bound imposed by the interconnect and memory system. Figure 1 illustrates this effect for the long-context, smallbatch decode setting that we target. On 4 GB2oo GPUs, the NCCL low-latency kernel we introduce in this work reduces average latency for small messages from 11.0µ s for NCCL ring to 2.37µ s, yielding an 8.7% ITL reduction for the inference workload of Llama-3.1-70B. Using CoreWeave’s on-demand price of $42/hour for 4 GB20o [8] and converting output throughput into cost per iM output tokens, the measured data implies that each µs removed from AllReduce latency reduces cost by about o.9%. While this saving may appear insignificant, it compounds into substantial cost reduction at the trillion-token scale of modern LLM services [4], [9].尽管这一认识日益增强，现有方法仍然未能榨于全部性能。我们观察到，即使是当前最好的可用实现，其性能也常常仍高于“光速”(SoL)界限；这里我们所说的 SoL，是指由互连和内存系统施加的硬件绝对下界。图1 展示了我们所针对的长上下文、小批量 decode 场景中的这一现象。在4 个 GB200 GPU 上，我们在本文中引I入的 NCCL 低 Latencykernel 将小消息的平均 Latency 从 NCCL ring 的11.0μ s降至 2.37μ s，从而使 Llama-3.1-70B 的 Inference 工作负载的 ITL 降低了 8.7%。采用 CoreWeave 对 4 个 GB200 提供的按需价格 $42/小时 [8]，并将输出 Throughput 换算为每100 万个输出 Tokens 的成本，实测数据表明，AllReduce Latency 每减少 μs，成本约可下降 0.9%。虽然这节省看起来可能并不显著，但在现代 LLM 服务达到万亿 Tokens 规模时，这种节省会累积为可观的成本降低[4]， [9]。In this work, we begin by identifying global memory barriers across participating GPUs as a key source of latency in existing collective implementations. To this end, we present several techniques, including LL, sentinel-based synchronization, double buffering, and a novel two-shot AllReduce algorithm. By combining these techniques, we eliminate expensive global memory barriers entirely while preserving correctness and efficiency. Building on these, we develop a set of experimental application programming interfaces (APIs) on top of NCCL’s latest device communication APIs that encapsulate these lowlatency mechanisms into reusable primitives for efficiently prototyping custom low-latency kernels. Leveraging this interface, we implement several new AllReduce kernels within NCCL that are directly usable in practice.在这项工作中，我们首先识别出参与的 GPU 之间的全局内存屏障是现有collective 实现中Latency 的一个关键来源。为此，我们提出了多种技术，包括LL、基于sentinel的同步、双缓冲，以及一种新的 two-shot AllReduce 算法。通过结合这些技术，我们在保持正确性和效率的同时，完全消除了代价高昂的全局内存屏障。在此基础上，我们基于NCCL最新的设备通信API开发了一组实验性的应用程序编程接口(APIs)，将这些低Latency机制封装为可复用的原语，以便高效原型化定制的低 Latency kernel。利用该接口，我们在NCCL 中实现了几种新的 AllReduce kernel，可直接在实践中使用。To evaluate these techniques and the resulting collectives, we conduct detailed microbenchmarks showing that our designs approach the hardware SoL latency bound across a wide range of node configurations. We also integrate our low-latency kernels into real workloads, including vLLM and cuSOLVERMp. These case studies demonstrate consistent and measurable performance improvements over standard NCCL collectives and other state-of-the-art frameworks, confirming为了评估这些技术及其产生的collective，我们进行了详细的微基准测试，结果表明我们的设计在广泛的节点配置下都接近硬件 SoL Latency 上界。我们还将我们的低 Latency kernel 集成到真实工作负载中，包括 vLLM 和 cuSOLVERMp。这些案例研究表明，相比标准 NCCL collective 和其他最先进框架，能够实现一致且可测量的性能提升，从而证实Figure 2: Fig. 2: Overview of device-initiated communication and symmetric memory in NCCL [19]. When GPUs are in the same node or NVLink domain, LSA operations are supported over PCIe and NVLink, while multimem operations are enabled via NVLink SHARP for hardware-accelerated multicast and reduction. For inter-node communication, GPUinitiated networking (GIN) supports GDAKI and proxy-assisted data transfers over InfiniBand and RoCE [2o].图 2：NCCL [19] 中设备发起通信和对称内存的概览。当 GPU 位于同一节点或同一 NVLink 域内时，LSA 操作可通过PCle 和 NVLink 支持，而 multimem 操作则通过 NVLink SHARP 启用，以实现硬件加速的多播和归约。对于跨节点通信，GPU 发起网络（GIN)支持通过 InfiniBand 和 RoCE [20] 进行 GDAKI 和代理辅助的数据传输。the practical benefits of latency-centric collective optimization for both LLM inference and traditional HPC workloads.以 Latency 为中心的 collective 优化对于 LLM Inference 和传统 HPC 工作负载都具有实际收益。Our contributions in this work are as follows:我们在这项工作中的贡献如下：- We systematically analyze existing techniques for reducing collective latency, characterize where they are effective. and compose them into barrier-free designs that approach the hardware lower bound.我们系统性地分析了现有的降低collective Latency的技术，刻画了它们各自生效的场景，并将其组合为无barrier的设计，使其接近硬件下界。- We design and implement a set of low-latency communication APIs on top of NCCL’s device-side APIs to facilitate development of custom collective kernels.我们在 NCCL的 device-side API 之上设计并实现了一组低 Latency 通信 API，以便于开发自定义 collectivekernel.- We develop new low-latency AllReduce algorithms, including a novel two-shot LL128 atomic design, using the proposed low-latency APIs.我们利用所提出的低 Latency API 开发了新的低 Latency AllReduce 算法，包括一种新颖的 two-shot LL128atomic 设计。- We conduct extensive evaluations through microbenchmarks and real workloads, including vLLM inference and cuSOLVERMp, demonstrating noticeable performance gains.我们通过 microbenchmark 和真实工作负载开展了广泛评估，其中包括 vLLM Inference 和 cuSOLVERMp，结果表明可带来显著的性能提升。II. BackgroundⅡ1. 背景A. GPU Communication LibrariesA. GPU 通信库Efficient communication is essential in modern GPUaccelerated systems, where both AI and scientific workloads rely on tightly coupled GPU execution [1o]. To support this, specialized GPU communication libraries have emerged as a critical software layer. NCCL is one of the most widely used libraries, providing collective and point-to-point operations optimized for various interconnects [11]. While NCCL primarily targets optimized collectives, NVSHMEM adopts a partitioned global address space (PGAS) model that enables one-sided communication and exposes finer device-side control [12]. Similar libraries exist across vendors, including AMD’s RCCL and rocSHMEM, and Intel’s oneCCL [13]-[15]. In contrast, traditional frameworks like MPI [16], originally designed for CPU-based systems, have been extended to support GPUs (e.g., CUDAaware MPI [17]), but often underperform vendoroptimized libraries for large-scale collectives while remaining competitive for point-to-point communication [18]. These observations highlight that GPU-native communication libraries have become an essential complement to traditional frameworks in modern HPC and AI systems.高效通信在现代GPU加速系统中至关重要，在这些系统中，AI和科学计算工作负载都依赖于紧密耦合的 GPU执行[10]。为支持这一点，专用的GPU通信库已成为关键的软件层。NCCL是使用最广泛的库之一，提供针对各种互连优化的 collective 和 point-to-point 操作 [11]。虽然 NCCL 主要面向优化的 collective，但 NVSHMEM 采用 partitionedglobal address space(PGAS) 模型，从而支持 one-sided 通信，并提供更细粒度的设备端控制[12]。类似的库在各家厂商中也都存在，包括 AMD 的 RCCL 和 rocSHMEM，以及 Intel 的 oneCCL [13]-[15]。相比之下，像 MPI [16] 这样的传统框架最初是为基于CPU的系统设计的，后来被扩展以支持 GPU(例如，CUDA-aware MPI[17])；但在大规模collective 方面，其性能往往不如厂商优化的库，而在 point-to-point 通信方面仍保持竞争力[18]。这些观察表明，在现代 HPC 和 AI 系统中，原生面向 GPU 的通信库已成为传统框架的重要补充。B. Device-Initiated Communication and Symmetric MemoryB. 设备发起通信与对称内存In addition to optimized collective primitives, modern CCLs increasingly support device-driven communication, enabling kernels to directly initiate and orchestrate data movement. Early support for this appeared in NVSHMEM, which allows kernels to perform remote memory operations (e.g., put/get) and synchronization without host involvement. More recently, NCCL 2.28 introduces device-side communication APIs that enable kernels to directly invoke communication primitives [2o], [21]. These capabilities are enabled by underlying hardware and runtime support, including GPU Virtual Memory Management (VMM), which provides a unified virtual address space across GPUs, and GPUDirect Async Kernel-Initiated (GDAKI), which allows GPUs to directly interact with network interfaces without CPU intervention.除了优化的集合通信原语之外，现代 CCL 也越来越多地支持设备驱动的通信，使kernel能够直接发起并编排数据移动。对此的早期支持出现在NVSHMEM 中，它允许 kernel 在无需 host 参与的情况下执行远程内存操作(例如put/get)和同步。最近，NCCL 2.28 引入了设备侧通信 API，使 kernel 能够直接调用通信原语[20]，[21]。这些能力由底层硬件和运行时支持实现，包括 GPU Virtual Memory Management (VMM)，它在多个 GPU 之间提供统一的虚拟地址空间，以及 GPUDirect Async Kernel-Initiated (GDAKI)，它允许 GPU 在无需 CPU 干预的情况下直接与网络接口交互。Symmetric memory originates from the SHMEM family of PGAS models. Remotely accessible data objects, called symmetric objects, have identical type, size, and layout on each processing element (PE), which corresponds to a GPU in this case. This allows remote access using the same logical address together with a PE identifier. These objects reside in the symmetric heap, which supports one-sided operations such as get, put, and atomics.对称内存起源于 SHMEM系列的 PGAS 模型。可远程访问的数据对象称为对称对象，它们在每个处理元素(PE)上具有相同的类型、大小和布局，这里对应于一个GPU。这使得可以结合PE标识符，使用相同的逻辑地址进行远程访问。这些对象驻留在对称堆中，对称堆支持诸如 get、put 和 atomics 等单边操作。On GPUs connected through PCIe or NVLink and supported by CUDA Virtual Memory Management (VMM), symmetric memory regions can be mapped into a unified virtual address space, making them load/store accessible (LSA). On systems with NVSwitch and NVLink SHARP (NVLS), multimem load/store instructions can further accelerate communication by enabling multicast and in-network reduction. A visualization is shown in Fig. 2. Overall, symmetric memory reduces address translation overhead and enables low-latency data exchange. Thus, NCCL is gradually replacing its collectives with symmetric-memory-based kernels, and previous implementations are now referred to as legacy kernels [19], [22].在通过 PCle 或 NVLink 互连并由 CUDA Virtual Memory Management (VMM) 支持的 GPU 上，对称内存区域可以映射到统一虚拟地址空间中，从而使其可通过 load/store 访问(LSA)。在具有 NVSwitch 和 NVLink SHARP (NVLS)的系统上，multimemload/store 指令还可以通过支持组播和网络内归约来进一步加速通信。图2展示了其可视化示意。总体而言，对称内存减少了地址转换开销，并实现了低Latency的数据交换。因此，NCCL正在逐步以基于对称内存的 kernel 替换其 collective，而先前的实现现在被称为 legacy kernel [19], [22]。We base our implementation on NVIDIA hardware and NCCL because this stack represents one of the most widely adopted platforms for GPU collectives in modern AI and HPC ecosystems [23], [24]. We chose NCCL rather than NVSHMEM since NCCL is extensively used as a communication backend across both deep learning frameworks and other scientific libraries, including PyTorch [25], TensorFlow [26], vLLM [3], cuSOLVERMp [27], and cuBLASMp [28]. This level of integration makes NCCL a more suitable choice.我们基于 NVIDIA 硬件和 NCCL 进行实现，因为这一技术栈代表了现代 AI 和 HPC 生态系统中 GPU collective 最广泛采用的平台之一[23]，[24]。我们选择 NCCL 而不是 NVSHMEM，是因为 NCCL 被广泛用作深度学习框架和其他科学计算库中的通信后端，包括 PyTorch [25]、TensorFlow [26]、vLLM [3]、cuSOLVERMp [27] 和 cuBLASMp [28]。这种集成程度使NCCL 成为更合适的选择。Most of the design principles are, nevertheless, not NVIDIA-specific. LL, sentinel synchronization, and double buffering require GPU-initiated access to peer memory, remote writes that become visible to GPU-side polling, and deviceside ordering or fence operations before buffer reuse. All platforms providing these properties can implement the proposed protocols and kernels.不过，大多数设计原则并不特定于 NVIDIA。LL、sentinel synchronization 和 double buffering 需要由 GPU 发起对对等内存的访问、对GPU侧轮询可见的远程写入，以及在缓冲区复用之前进行设备端排序或fence 操作。所有提供这些特性的平台都可以实现所提出的协议和 kernel。Additionally, this work focuses on collectives within a scaleup network, specifically GPUs residing in the same NVLink domain. We exclude scale-out communication for several reasons. First, in modern LLM inference, which is a primary此外，这项工作重点关注 scale-up 网络内的 collective，具体来说是位于同一 NVLink 域中的 GPU。我们出于几个原因排除 scale-out 通信。首先，在现代 LLM Inference 中，这是一个主要的Figure 3: Fig. 3: Barrier latency on GB2oo as a function of the number of GPUs for unicast and multicast implementations.图 3：GB200 上单播和多播实现中，Barrier Latency 随 GPU 数量变化的关系。target of this work, parallel groups are typically confined to a single scale-up domain. Second, multi-node systems commonly employ hierarchical collectives that separate local and scale-out phases, making improvements within the scaleup domain complementary to higher-level optimizations [29], [3o]. Finally, emerging GPU systems increasingly expand the size and capability of these domains, allowing a growing fraction of latency-sensitive workloads to executeentirely within a single scale-up network [31], [32]本工作的目标是，并行组通常被限制在单个纵向扩展域内。其次，多节点系统通常采用分层集合通信，将本地阶段与横向扩展阶段分离，因此，scale-up 域内的改进可作为更高层优化的补充[29]，[30]。最后，新兴的 GPU系统正不断扩展这些域的规模和能力，使得越来越大比例的Latency敏感型工作负载能够完全在单个纵向扩展网络内执行[31][32]。III. Toward Near Speed-of-Light AllReduceIII. 迈向接近光速的 AIlReduceIn this section, we describe how we approach near speedof-light (SoL) latency, i.e., the hardware lower bound, for AllReduce (R is capitalized following NCCL’s notation). We focus on AllReduce because it is one of the most widely used collectives in HPC and distributed machine learning and a frequent optimization target in practice [33]-[38]. Moreover, many implementations decompose AllReduce into ReduceScatter and AllGather or Reduce and Broadcast, so techniques that minimize AllReduce latency often apply directly to these building blocks. Thus, optimizing AllReduce benefits a broader class of collectives.在本节中，我们描述如何让 AlIReduce(R 按照NCCL 的记法大写）的 Latency 接近光速(SoL)Latency，即硬件下界。我们聚焦于 AllReduce，因为它是 HPC 和分布式机器学习中使用最广泛的 collective 之一，并且在实践中也是一个常见的优化目标 [33]-[38]。此外，许多实现会将 AllReduce 分解为 ReduceScatter 和 AllGather，或 Reduce 和Broadcast，因此，能够最小化 AllReduce Latency 的技术通常也可直接应用于这些基础构件。因此，优化 AllReduce会使更广泛的一类 collective 受益。A. Low-Latency AllReduce AlgorithmsA. 低 Latency AllReduce 算法When the message size of an AllReduce operation is small, its latency is primarily determined by the number of synchronizations needed, or communication phases. Consequently, algorithms such as tree-based or recursive-doubling AllReduce, which require O(log N) rounds of synchronization for N ranks, typically outperform ring-based algorithms that require O(N) rounds for small messages [11], [39]. In a scale-up network, the number of synchronizations can be reduced further to O(1) using one-shot or two-shot AllReduce algorithms, which are widely adopted in most communication libraries and frameworks [3], [6], [11], [12], [30]当 AllReduce 操作的消息大小较小时，其Latency 主要由所需的同步次数（或通信阶段数）决定。因此，对于N个rank，诸如基于树或递归倍增的 AllReduce 算法通常只需O(log N)轮同步，通常比对小消息需要O(N)轮的基于ring 的算法表现更好 [11], [39]。在 scale-up 网络中，使用 one-shot 或 two-shot AllReduce 算法可以将同步次数进一步减少到 O(1)，这类算法已被大多数通信库和框架广泛采用 [3], [6], [11], [12], [30]。One-shot AllReduce: In a one-shot AllReduce, the entire reduction is completed in a single communication phase, where each GPU fetches data from all peers, performs the reduction locally, and writes the result to the output. In pull mode, GPUs read remote data via loads, while in push mode they write data to remote buffers before reducing locally. Push is generally faster, requiring only half a GPU-to-GPU RTT versus a full RTT for remote loads, but needs additional buffering for incoming data. Consequently, push-based one-shot designs are often preferred for latency-sensitive collectives.一次性 AllReduce：在一次性 AlIReduce中，整个归约在单个通信阶段内完成，每个 GPU从所有对端获取数据，在本地执行归约，并将结果写入输出。在pull模式下，GPU通过load 读取远程数据；而在 push 模式下，它们先将数据写入远程缓冲区，再在本地进行归约。Push 通常更快，只需要一半的 GPU 到 GPU RTT，而远程load 需要完整的 RTT，但它需要为传入数据提供额外缓冲。因此，基于 push 的一次性设计通常更适合对 Latency 敏感的 collective。Two-shot AllReduce: A two-shot AllReduce is decomposed into two phases. In the ReduceScatter phase, each GPU exchanges partitions of its input with peers and performs the两次式 AllReduce：两次式 AllReduce 被分解为两个阶段。在 ReduceScatter 阶段，每个 GPU 与对端交换其输入的分区，并执行其Figure 4: Fig. 4: Example illustrating bidirectional communication with double buffering. Two ranks exchange chunked inputs, where chunk i from rank r is denoted Cγ. The scratch space is divided into Buffers o and 1, each can store two chunks. The black border marks the active buffer. Steps (1), (2), and (3) show events in chronological order, with time progressing to the right. Arrows represent cross-GPU data transfer. The mechanism is independent of whether LL or sentinel synchronization is used.图 4：图 4：使用双缓冲进行双向通信的示例。两个 rank 交换分块输入，其中来自 rank r 的 chunk i 记为CriC _ { r } ^ { i }Cri​。暂存空间被划分为 Buffer 0 和 1，每个都可存储两个 chunk。黑色边框标记活动 buffer。步骤(1)、(2)和(3) 按时间顺序展示事件，时间向右推进。箭头表示跨 GPU 数据传输。该机制独立于是否使用 LL 或 sentinel 同步。reduction on its assigned chunk, producing a partial result. Either push or pull semantics may be used in this phase. In the AllGather phase, the reduced chunks are exchanged so that every GPU has the complete result. Compared to one-shot, twoshot introduces an additional synchronization but significantly reduces the communication volume, improving performance for moderate message sizes. For N ranks reducing M bytes of data, one-shot incurs O(N · M) total communication volume, whereas two-shot reduces this to O(M).对其分配的 chunk 进行 reduction，生成部分结果。在此阶段可以使用 push 或 pull 语义。在 AllGather 阶段，交换已reduction 的 chunk，使每个 GPU 都拥有完整结果。与 one-shot 相比，twoshot 引入了额外的一次同步，但显著减少了通信量，从而提升了中等消息大小下的性能。对于 N 个 rank 对 M 字节数据进行 reduce，one-shot 产生的总通信量为O(N·M)，而 two-shot 将其降低到O(M)。B. Cost of Memory BarriersB. 内存屏障的成本After examining several one-shot and two-shot AllReduce implementations in state-of-the-art frameworks and libraries [3], [6], [12], [3o], we observe that, regardless of push or pull communication, these designs typically rely on explicit memory barriers to synchronize peers and signal data readiness at the thread-block level. Using NCCL’s ncclLsaBarrierSession, which implements a memory barrier for load-store accessible (LSA) devices, we measure the overhead of such synchronization under relaxed memory ordering and report the results in Fig. 3. Although NCCL’s implementation may not be fully optimized, alternative designs follow the same fundamental pattern: a flag is propagated to all peers, and each GPU waits until it observes the corresponding signals from every other participant. Therefore, it is representative of the inherent cost in such approaches.在考察了先进框架和库[3]、[6]、[12]、[30] 中的若于 one-shot 和 two-shot AllReduce 实现之后，我们观察到，无论采用push还是pull通信，这些设计通常都依赖显式内存屏障在线程块级别同步对等方并发出数据就绪信号。我们使用NCCL 的 nccILsaBarrierSession(其为 load-store accessible (LSA) 设备实现了—个内存屏障)来测量在 relaxedmemory ordering 下此类同步的开销，并在图 3 中报告结果。尽管 NCCL 的实现可能尚未完全优化，但其他设计遵循相同的基本模式：一个标志被传播给所有对等方，而每个GPU都会等待，直到它观察到来自其他每个参与者的相应信号。因此，它可以代表此类方法的固有成本。The measured barrier latency shows that each barrier will incur more than 1μs of overhead. In many AllReduce kernels, two such barriers are required. As illustrated in Fig. 1, when a small-message AllReduce completes in roughly 5μs on four GPUs, two barrier calls alone will account for about 40% of the total latency. As emphasized earlier, when targeting nearSoL performance, every µs matters. Thus, eliminating these barriers entirely can yield substantial performance gains.测得的 barrier Latency 表明，每个 barrier 都会带来超过 1μs的开销。在许多 AllReduce kernel 中，需要两个这样的barrier。如图 1 所示，当四个 GPU 上的一次小消息 AlIReduce 大约在5μs内完成时，仅两次 barrier 调用就约占总Latency 的 40%。正如前文所强调的，当目标是接近 SoL 性能时，每个μs都至关重要。因此，彻底消除这些 barrier可以带来显著的性能提升。IV. Designing Barrier-Free CollectivesIV. 设计无 barrier 的 collectiveHaving established that memory barriers introduce nonnegligible overhead, we next consider alternative synchronization mechanisms that achieve the same purpose with lower latency. As discussed in Section III-A, AllReduce can be implemented using either push or pull communication. Since our goal is to approach the SoL, we focus on push mode, which trades additional buffer space, referred to here as a scratch buffer, for roughly half of a GPU-to-GPU RTT.既然已经确定 memory barrier 会引入不可忽略的开销，我们接下来考虑能够以更低 Latency 实现相同目的的替代同步机制。如II-A 节所述，AllReduce 可以使用 push 或 pull 通信来实现。由于我们的目标是接近 SoL，我们聚焦于 push模式，它以额外的缓冲区空间(这里称为 scratch buffer)为代价，换取大约一半的 GPU-to-GPU RTT。LL : The first technique we introduce is LL, short for low latency, which originates from the LL protocol in NCCL [11] and is also used in libraries such as NVSHMEM [12] and MSCCL++ [3o]. Conventional synchronization signals data arrival using explicit flags and enforced ordering between data and signals. LL removes the signaling step by packing the 8byte flag with the 8-byte data and transmitting them atomically with 16-byte atomic stores, allowing the receiver to determine data readiness by checking the flag directly. This design halves the effective payload bandwidth and doubles scratch buffer usage, making LL mostly suitable for very small messages.LL：我们引I入的第—种技术是 LL，即 low latency，源自 NCCL [11] 中的 LL protocol，也被用于 NVSHMEM [12]和 MSCCL++[30] 等库中。传统同步通过显式 flag 以及在数据与信号之间强制执行顺序来表示数据到达。LL 通过将8byte flag 与 8-byte data 打包，并使用 16-byte atomic stores 原子地传输它们，从而移除了 signaling 步骤，使接收方能够通过直接检查 flag 来判断数据是否就绪。该设计将有效 payload bandwidth 减半，并使 scratch buffer 使用量翻倍，因此LL大多只适用于非常小的消息。Sentinel: Instead of embedding a signal in the transmitted data, the receiving scratch buffer can be initialized with a sentinel value that is unlikely to appear in valid computations, such as the floating-point value -NaN. Data is then written directly to the buffer, and the receiver polls until the value changes from the sentinel, indicating that valid data has arrived. Compared to LL, this approach preserves full effective bandwidth and uses less scratch space, making it more efficient for moderately larger messages. However, it has a few drawbacks. First, unlike LL, which updates its flag each iteration, the sentinel method requires the buffer to be reset before reuse, complicating buffer management. Second, transmitted values must never match the sentinel because a matching value would prevent the receiver from detecting data arrival. Users must therefore exclude such values to ensure correctness.哨兵：接收端的暂存缓冲区可以用一个在有效计算中不太可能出现的哨兵值进行初始化，而不是在传输的数据中嵌入信号，例如浮点值-NaN。随后，数据会直接写入该缓冲区，接收端则持续轮询，直到该值从哨兵值变为其他值，这表明有效数据已经到达。与LL相比，这种方法能够保留完整的有效带宽，并使用更少的暂存空间，因此对于中等偏大的消息更加高效。然而，它也有一些缺点。首先，不同于LL在每次迭代中都会更新其标志，哨兵方法在缓冲区重用之前需要将其重置，这使缓冲区管理更加复杂。其次，传输的值绝不能与哨兵值相同，因为一旦值匹配，接收端就无法检测到数据到达。因此，用户必须排除这类值以确保正确性。Bidirectional Communication Double Buffering: Although LL and sentinel synchronization eliminate explicit memory barriers for single exchanges, they are insufficient when messages require multiple iterations due to limited buffer space. In such cases, inputs are partitioned and processed in chunks. Conventional approaches insert barriers between iterations to prevent buffer overwrites. We can eliminate them by using bidirectional communication and double buffering双向通信与双缓冲：尽管LL 和 sentinel 同步消除了单次交换所需的显式内存屏障，但当由于缓冲区空间有限而使消息需要多次迭代时，它们仍然不够用。在这种情况下，输入会被划分并按块处理。传统方法会在迭代之间插入屏障以防止缓冲区被覆写。我们可以通过使用双向通信和双缓冲来消除这些屏障。Fig. 4 demonstrates the mechanism. Consider two ranks whose scratch space cannot hold all input data simultaneously. In step (1), both ranks begin with the scratch buffer set to Buffer o. In step (2), they exchange their first chunksCr.0C _ { r . } ^ { 0 }Cr