Zero-fill memsets dominate GPU decode step
20 hot tracestriton
GPU · Memory LLM · Inference · vLLM
2.4 GPU hours
Recommendations
Performance recommendations, written from your profiles
Performance issues compete for engineering time, but their costs are not equal. Zymtrace quantifies each issue over time and recommends a fix with estimated savings, so your team can prioritize the changes worth shipping.
- CUDA and CPU
- Estimated savings
- Your own AI model
Usage over time
See what each issue costs
Each recommendation charts GPU time or CPU cores per hot trace, so you know how big the problem is before you fix it.
- One series per hot trace
- The hosts, containers and executables it spans
- Affected entities one click away
Recommended actions
Ranked fixes with estimated savings
Each fix is specific, down to the hook to install, the flag to set or the call to replace.
- Savings in GPU time or CPU cores
- The evidence and source locations behind each fix
- Regenerate the report after you ship a fix
Slack
New recommendations, straight to Slack
New recommendations post to a Slack channel with their diagnosis, impact and a link to the report.
- Impact in GPU time or CPU cores
- Tags to route on, such as
gpu:collective:nccl - GPU XID errors can post too
Fleet ranking
Know which application to fix first
Recommendations are ranked by peak usage across your fleet, so the applications burning the most GPU and CPU time come first.
- One recommendation per application, even across clusters
- CUDA and CPU tabs, filtered by tags
- Dismiss what doesn't apply
Autonomous optimization
Profile-guided agentic optimization
We enrich coding agents with perf data context from Zymtrace over MCP, so they find the bottleneck and open a pull request.
01
Pull Filters the fleet, then pulls GPU flamegraphs and inference metrics over MCP.
02
Correlate Cross-checks the CPU profiles for the same run to separate dispatch from compute.
03
Diagnose Names the hot frame, why it stalls, and what it costs in tokens/sec.
04
Fix and verify
↺ LOOP
Applies the change, then re-profiles to prove the win.
coding agent · Zymtrace skills
MCP server
Flamegraphs, top functions, and GPU metrics exposed as MCP tools. Any MCP-capable agent can query production directly.
Zymtrace agent skills
Ready-made skills teach your coding agent the whole optimization workflow: rank consumers, read flamegraphs, interpret stall reasons, apply the fix.
Read about agent skills →Subagents at fleet scale
Fan out specialized subagents to investigate many workloads in parallel. Each one returns an evidence-backed diagnosis and fix, not a guess.
Under the hood
Built on the profiles you already collect
Rolling analysis window
The last six hours by default, regenerated as the workload changes.
CUDA and CPU
CUDA reports cover the launching code and the kernels.
One report per application
Per deployment, container or executable.
Estimated savings
In GPU time or CPU cores, plus an annual cost for CPU.
Your choice of AI model
Claude, GPT, Gemini or any OpenAI-compatible endpoint.
Token limits
A separate model per analysis step and an hourly token cap.
Customer results
Fixes that shipped
TypeSafe, makers of Jev
TypeSafe profiled Jev's training and inference with Zymtrace. Three optimizations compounded on the same hardware, and removing a hidden device-to-host sync sped up inference.
- training throughput
- +46%
- faster inference
- ~10%
Anam traced slow avatar inference to Python runtime overhead and GIL contention, then removed it with TorchScript and CUDA graphs.
- lower latency
- 2.5×
- inference throughput
- +90%
FAQ
Recommendations FAQ
No. Recommendations run in the background and update within minutes. You can also generate one for any flamegraph.
CUDA and on-CPU profiles. Recommendations for AMD ROCm are coming soon.
Yes. Connect a channel and new recommendations post there with a link to the report.
The one you configure: Claude, GPT, Gemini or any OpenAI-compatible endpoint, including a model you host yourself.
Zymtrace is self-hosted. Only the traces being analyzed go to the model you configure, so a self-hosted model keeps everything in your network.
Yes. The MCP server gives agents the recommendations, flamegraphs and metrics, and Zymtrace Agent Skills package the workflow for Claude Code, Codex and Cursor.
Let Zymtrace do the analysis
Zymtrace is fully self-hosted. 5 minutes is all you need to get it up and running.