NCCL 2.27 or newer
Reads NCCL's own metrics, with no change to your code.
GPU Straggler Detector
Find the straggler GPU that slows down training
One late GPU makes every other GPU wait. Zymtrace finds it, shows why, and correlates it to the corresponding GPU profiles.
- NCCL 2.27+
- No code changes
- Slurm-aware
Possible causes
The likely cause behind each straggler
Zymtrace compares the late GPU's clock with its peers', checks its throttle reasons, such as a power cap, and looks for NVLink errors.
- Clock checks across the communicator
- How much of its peers' time goes to waiting for it
- One click to its CUDA and CPU flamegraphs
Slow operations
NCCL collectives that run slower than usual
Zymtrace tracks each collective's bandwidth per communicator and message size, and flags drops of 30% or more from its recent normal.
- Peak slowdown in bandwidth and in time
- Intra-node and inter-node communicators
- Zoom the page to any episode
Profile-aware
Each straggler links to its CUDA flamegraph
Detection runs on the profiler's own data, so each straggler opens the CUDA flamegraph of its GPU and window, down to the NCCL kernel that waits.
- On one 8× H100 node, the fix it pointed to made an FSDP job 1.42× faster
- Scope stragglers to a Slurm job ID
- MCP tools let coding agents run the same investigation
How it works
Built into the CUDA profiler
In ByteDance's OSDI '25 study, stragglers slowed 42.5% of traced jobs and wasted 10.4% of all allocated GPU hours.
Off until you turn it on
Set ZYMTRACE_CUDAPROFILER__NCCL=true, or turn it on in a CUDA profiler rule.
NCCL collectives
AllReduce, AllGather, ReduceScatter and more, per communicator and message size.
Hardware checks
Clock against its peers', throttle reasons such as power caps, and NVLink errors.
Slurm-aware
Scope stragglers and slow operations to a Slurm job, a host or a communicator.
Ready for agents
MCP tools for stragglers, slow collectives and their causes.
FAQ
GPU Straggler Detector FAQ
A GPU that reaches collectives late and makes the rest of its communicator wait. Its own time per collective is the shortest, because its peers' clocks keep running while they wait for it.
NCCL 2.27 or newer and
ZYMTRACE_CUDAPROFILER__NCCL=true on every process of the job, or NCCL collection in a CUDA profiler rule. The launch post walks through a full investigation.It covers NCCL collectives. Point-to-point Send/Recv and NIC or InfiniBand diagnosis aren't covered yet.
Find the slow rank
Zymtrace is fully self-hosted. 5 minutes is all you need to get it up and running.