Squeeze more FLOPs from the hardware you already own.
Profile-guided, agentic optimization for training and inference workloads.
Silicon-aware and always-on, whether your fleet is all NVIDIA or a mix. Zymtrace sits between your AI workloads and the accelerators beneath them, turning execution data into hardware-aware optimization.
zymtrace
Silicon-aware · Always-on · Agentic optimization
CUDA · NVLink When a performance problem isn't in your code, it's in your hardware. Zymtrace surfaces GPU health, topology, and interconnect issues that waste the compute you paid for.
Ran on CPUs 16-19, which are on NUMA 0 and not affine with this GPU.
fix: numactl --cpunodebind=1 --membind=1
Zymtrace picks up execution on the CPU with eBPF and traces it down to the accelerator. An MRI for your stack: it exposes the bottlenecks hiding between the layers, with minimal overhead.
↑ observe from silicon up
Use Zymtrace to debug and optimize inference and training on the engines you already use, from vLLM and SGLang to distributed training, so you maximize throughput per GPU, per watt.
Metrics and profiles collected from the inference engines themselves, across prefill and decode, on heterogeneous accelerators. Drill from a dip in token throughput straight into the stall behind it.
SGLang 
llm-d 
TensorRT-LLM · Dynamo One execution flow across the CPU/GPU boundary, from Python down to the exact GPU instruction.
41% of GPU time waiting on memory, attributed to one instruction
Continuous profiles of every process on the box, compiled and interpreted alike, with automatic symbolization of native and CUDA frames.
Everything above feeds one ranked list: what to change, why it matters, and where to apply it.
Switch the KV cache to FP8
Decode is bandwidth-bound: 41% of GPU time stalls on memory dependency, not compute. An FP8 cache halves KV bytes per token and keeps more sequences resident, so batches grow instead of waiting.
We enrich coding agents with perf data context from Zymtrace over MCP, so they find the bottleneck and open a pull request.
Flamegraphs, top functions, and GPU metrics exposed as MCP tools. Any MCP-capable agent can query production directly.
Ready-made skills teach your coding agent the whole optimization workflow: rank consumers, read flamegraphs, interpret stall reasons, apply the fix.
Read about agent skills →Fan out specialized subagents to investigate many workloads in parallel. Each one returns an evidence-backed diagnosis and fix, not a guess.
Most GPU clusters run under 40% utilization. These teams found what was actually stalling their workloads and fixed it without provisioning a single extra GPU.
Zymtrace is one of the most remarkable pieces of technology I have ever used.
Deep visibility into application behavior, resource utilization, and performance bottlenecks is no longer a luxury. It is a foundational capability for operating at scale.
“Excited to welcome Zymtrace to the Walmart Sparkcubate program to help build the AI infrastructure observability layer.”Vini Jolly, Walmart Global Tech →
We spent so much time hunting down why our GPUs were being used inefficiently, and often we couldn't even reproduce the issue when we tried to profile it. … Zymtrace solved both problems: it runs continuously, so the data is always there when issues happen, and the entire team can understand it without needing specialized GPU profiling knowledge.
Oscar Holroyd and Michael Negus have done a massive refactor on the Vanellus solver over the last couple of weeks, removing some gnarly GPU memory transfer bottlenecks the Zymtrace GPU profiler helped us identify.
No re-architecting. No code changes. The agent drops into whatever you already run.
Slurm HPC scheduler native We pioneered, open-sourced, and donated the eBPF profiler to OpenTelemetry. With Zymtrace, we're extending that same low-level engineering to GPU-bound workloads, building a scalable profiling platform purpose-built for today's distributed, heterogeneous fleets.
compliant # zero code changes, no restarts. up in 5 minutes.
$ helm repo add zymtrace https://charts.zymtrace.com
$ helm install zymtrace zymtrace/profiler \
--set gateway.url=$ZYMTRACE_GATEWAY
✓ agent running fleet-wide
✓ CPU + GPU profiles streaming
✓ overhead <1% CPU · <250MB RAMEngineering notes on profiling, GPUs, and efficient AI infrastructure.
Zymtrace is fully self-hosted. 5 minutes is all you need to get it up and running.
Try it now
