Nvidia NeMo Switchyard routes AI tasks to lower-cost models
Nvidia has released NeMo Switchyard, a model-routing library that aims to reduce agent costs, though its savings figures remain company-reported.
By Dominic Okoye · Staff Writer
· 3 min read
Nvidia released NeMo Switchyard on Aug. 11 alongside its 30-billion-parameter Nemotron 3.5 Lightning model, offering enterprises a way to send different stages of an AI-agent workflow to different language models. The Nvidia NeMo Switchyard pitch is straightforward: reserve costly frontier models for harder work, and use smaller or specialized models for routine steps where they can meet the required quality bar.
Switchyard is software, not networking equipment. Nvidia describes it as an open-source, provider-agnostic LLM routing decision engine and library. It selects a backend from a developer-configured pool of open, proprietary and Nvidia models for an individual request or a step within a larger agent workflow.
How does Nvidia NeMo Switchyard reduce AI agent costs?
Model routing targets total task-completion cost rather than a provider’s listed price per token. A lower-priced model can cease to be economical if it uses far more tokens to finish the same job. Conversely, an expensive frontier model can be wasteful for bounded tasks such as classification, extraction or formatting.
Nvidia says Switchyard can assess request characteristics, available context, model capability, latency, cost profiles and infrastructure signals. It also supports strategies based on a classifier or changing agent state, allowing routing to change as tools return results or a workflow becomes more or less complex. Developers can set policies that weight quality, latency and cost differently.
Lightning is the companion model in that system-of-models argument. Nvidia positions the open mixture-of-experts model for high-volume, specialized agent tasks, while a stronger model could handle planning or ambiguous reasoning. The company says Lightning can also run locally or on premises on specified Nvidia hardware, though its product claims are not independent measurements of an enterprise’s economics.
What evidence supports Nvidia’s savings claim?
Nvidia says an internal benchmark maintained frontier-level accuracy while bringing task-completion cost to nearly one-third of running Anthropic’s Opus 4.8 alone. That is a vendor benchmark, not an independently verified production result.
A more specific test reported by VentureBeat came from LangChain: across 145 multi-turn Deep Agents tasks, LangChain said routing only 7% of calls to a frontier model reduced cost by 74%, with a 6% accuracy tradeoff. The result illustrates the central operational tradeoff, rather than establishing a general savings rate for other workloads.
For teams considering the approach, the relevant comparison is their own production mix of tasks, failure tolerance and model pool. They need to measure quality as well as spend, including whether routing decisions remain reliable as workflows and underlying models change. A disciplined AI model evaluation process is needed before translating benchmark economics into a deployment plan.
Nvidia enters an established category. OpenRouter, LiteLLM, Not Diamond and RouteLLM already offer routing-related tools or frameworks. Nvidia says Switchyard has integrations or partnerships involving LangChain, Cognition, Kong, LiteLLM and OpenRouter, so its differentiator is the pairing of routing with its own open model line rather than a new routing concept.
There are deployment limits. Nvidia’s documentation labels the NeMo Relay Switchyard plugin experimental and says it is not enabled in default Relay builds. That integration currently requires a separately running Switchyard Decision API, and Nvidia says the plugin’s configuration and contracts may change.
This story draws on original reporting from The Register.
