NVIDIA / nccl

NVIDIA's C++/CUDA library of optimized collective communication primitives (all-reduce, all-gather, broadcast, etc.) for multi-GPU and multi-node systems.