NVLink since 13:21
GPU 2 ↔ GPU 5 degraded
4.2 Flit CRC errors/s 1.3 replays/s
Drain the node, reseat the link, then recheck the interconnect
GPU topology & health
GPU topology and health, with recommended fixes
Degraded NVLink and poor NUMA placement can slow workloads without an obvious software bottleneck. Zymtrace identifies affected GPUs, surfaces hardware errors and recommends NUMA fixes, so engineers can investigate the actual cause.
- NVLink and PCIe
- XID and ECC errors
- NUMA affinity
XID errors
XID errors with what to do next
XID events are decoded with NVIDIA's guidance and pinned to the GPUs they hit.
- ECC, row remapping and PCIe errors per GPU
- Flit CRC and replay errors per NVLink
- Slack alerts linked to the affected GPU
NUMA affinity
NUMA affinity for GPU processes
Zymtrace flags processes that run on CPUs far from their GPU, with the command that pins them back.
- GPUs grouped by NUMA node, with their CPUs
- Non-affine CPUs flagged per process and GPU
- A
numactlcommand to pin each process
Fleet-wide metrics
GPU metrics per device and MIG instance
NVIDIA, AMD and AWS Neuron metrics come from the same agent that profiles.
- Utilization, memory, power, clocks and throttling
- Per device, per MIG instance and per process
- Tensor Core utilization per instance
Metrics and profiles
Open the GPU profile behind any metric
A dashboard can't explain a dip. Click any point on any chart to open the CUDA or ROCm profile of that moment, down to the code that caused it.
- CUDA or ROCm, matched to the GPU
- The full flamegraph or the top functions
Topology health
The hardware layer under your profiles
Interconnect matrix
Each GPU pair, NV1 to NV18 or the PCIe path, by NUMA node.
NVLink health
Flit CRC, replay and recovery errors per link.
Device health
ECC errors, retired pages, row remapping and XID events.
MIG layout
MIG profiles and instances on the GPUs that have them.
Warp stall distribution
Where the host's GPU time stalls, at a glance.
No extra agent
The profiler collects it all. No sidecar, no exporter.
FAQ
GPU topology & health FAQ
No. The profiler collects topology, health and metrics with the profiles.
GPU topology covers NVIDIA GPUs. GPU metrics cover NVIDIA GPUs, including MIG instances, AMD GPUs and AWS Neuron devices.
Every 30 seconds, exported once a minute by default.
Yes. XID errors can post to Slack, linked to the affected GPU.
See your GPU fleet's health
Zymtrace is fully self-hosted. 5 minutes is all you need to get it up and running.