fix(runtime): support NCCL2 communicator binding

[apbose]

Fixes #4658

[pkisfaludi-nv]

Independent repro that matches this fix, in case it's useful as corroboration.

Building torch-tensorrt from source with ENABLE_TRT_NCCL_COLLECTIVES (8× A100, TRT 11.2.1.2, torch 2.15.0.dev20260902+cu130, ENABLED_FEATURES.native_trt_collectives == True), every test that goes through distributed_context(group) dies at exactly the two lines this PR deletes:

RuntimeError: [Error thrown at core/runtime/TRTEngine.cpp:916]
Expected nccl_pg != nullptr to be true but got false
Backend is not ProcessGroupNCCL

getBackend(BackendType::NCCL) returns non-null, so it is specifically the dynamic_cast<c10d::ProcessGroupNCCL*> that fails — consistent with the backend now being c10d::nccl2::ProcessGroupNCCL on this torch.

Split by whether the binding path is exercised:

test	result
TestMultirankNccl::test_all_reduce_correctness (no distributed_context)	passed
TestMultirankNccl::test_distributed_mode_subgroup	failed, TRTEngine.cpp:916
TestMultirankNccl4GPU::test_two_dimensional_mesh_routing (from #4380)	failed, TRTEngine.cpp:916

One unrelated gotcha that cost me a while, in case anyone else hits it on a PCIe box: on this node NCCL P2P is broken, and every 2-rank group hung in dist.barrier() for 600 s before any of the above surfaced. NCCL_P2P_DISABLE=1 makes 2-rank collectives work and turns the hangs into the clean error above.

Happy to re-run #4380's 4-GPU mesh-routing test against this branch once it's ready — it's currently the only thing blocking a runtime verdict there.

[cehongwang]

This is generated by AI. I am not sure since I am not very familiar with NCCL but I think it is worth referring to:

This grep matches on PyTorch builds that have no nccl2 backend whatsoever, so the check validates nothing — and because it can only ever subtract from the header result, its sole possible effect is to silently disable nccl2 support in a build that needed it.

I ran the exact command against torch 2.13.0+cu132, which ships no torch/include/torch/csrc/distributed/c10d/nccl2/ directory at all:

$ grep -q "nccl2" $TORCH/lib/libtorch_cuda.so; echo $?
0
$ grep -ao "nccl2" $TORCH/lib/libtorch_cuda.so | wc -l
4
All four hits are mangling artifacts, not the namespace you are looking for:

_ZN5torch4cuda4nccl26all2all_single_equal_splitERN2at6TensorES4_iPvRN3c104cuda10CUDAStreamE
  => torch::cuda::nccl::all2all_single_equal_split(at::Tensor&, at::Tensor&, int, void*, c10::cuda::CUDAStream&)

4nccl (namespace nccl, 4 chars) is immediately followed by 26, the length prefix of the 26-character all2all_single_equal_split. The substring nccl2 straddles that boundary. So the "make sure the matching implementation is present" comment is not true of the code beneath it.

The check is also redundant. _has_process_group_nccl2 is only reached inside if has_nccl: → if found: (defs.bzl:65), and has_nccl is itself established by grepping ProcessGroupNCCL in the very same libtorch_cuda.so. A PyTorch that ships the nccl2 header ships the nccl2 implementation in the same wheel, so the header test alone already carries the signal. The only outcome the grep can add is a false negative — say libtorch_cuda.so is absent under some SBSA or CPU-only layout — which omits TORCHTRT_HAS_PROCESS_GROUP_NCCL2, compiles the nccl2 branch out, and produces a wheel that throws your new "Unsupported NCCL backend" on the default backend.

Simplest fix is to delete it:

def _has_process_group_nccl2(repository_ctx, torch_path, lib_path):
    """Check that the installed PyTorch exposes the NCCL2 backend."""
    header = torch_path + "/include/torch/csrc/distributed/c10d/nccl2/ProcessGroupNCCL.hpp"
    header_result = repository_ctx.execute(["test", "-f", header])
    if header_result.return_code != 0:
        return False
    # The header alone is insufficient: make sure the matching implementation
    # is present in the PyTorch CUDA library that the detector found.
    symbol_result = repository_ctx.execute(["grep", "-q", "nccl2", lib_path])
    return symbol_result.return_code == 0
    return repository_ctx.execute(["test", "-f", header]).return_code == 0
That also drops the now-unused lib_path parameter at the call site. If you would rather keep a symbol check, grep the mangled namespace token instead of the bare string — 5nccl216ProcessGroupNCCL (nccl2 as a 5-char namespace followed by the 16-char ProcessGroupNCCL) cannot collide with torch::cuda::nccl.

[apbose]

removed the binary grep and the now unused lib path parameter. NCCL2 support should be detected from the ProcessNCCL2.hpp header