Nemotron 3.5 Lightning and NeMo Switchyard: NVIDIA's Bet on Model Routing

The short version

NVIDIA just shipped two things that matter if you run agents in production: Nemotron 3.5 Lightning, a 30-billion-parameter mixture-of-experts open model built for high-volume agentic tasks, and NeMo Switchyard, an open source routing library that sends each step of an agent workflow to the model best suited for it.

One is a cheaper worker. The other is a dispatcher. Together they point at the architecture most serious agent systems are already converging on: not one big model doing everything, but a system of models where a frontier planner delegates to fast specialists.

Why "one model for everything" is the expensive mistake

Most businesses wire their agent to a single frontier model and call it done. It works, and then the invoice arrives.

In a real workflow, the steps are wildly uneven. Classifying an inbound message, extracting a phone number, checking whether a ticket needs escalation, or formatting a summary are not hard problems. Planning a multi-step refund investigation is. Paying frontier prices for all of it means you subsidize the easy 80% to protect the hard 20%.

The alternative used to be manual: write your own routing logic, maintain it, and re-tune it every time a model updates. That is integration work that quietly eats a deployment timeline.

Nemotron 3.5 Lightning: the specialist

Lightning is a 30B mixture-of-experts model, meaning only a slice of its parameters activate per token. NVIDIA reports up to 4x faster output speed and roughly 30% faster agentic task completion versus comparable models in its class.

The parts that matter operationally:

Companies already customizing it include CrowdStrike (cybersecurity), Harvey (legal), CodeRabbit (code review), Lila Sciences, and Fastino Labs. That spread tells you the intended use: narrow, repeated, high-volume tasks inside a bigger system.

NeMo Switchyard: the dispatcher

Switchyard is an open source model router for agents. It inspects each request in a workflow and sends it to the most capable and efficient model available — across your own mix of open, proprietary, and NVIDIA models — without rewriting the application.

You tune the router to your priorities: quality, latency, or cost. NVIDIA's internal benchmarks show it holding frontier-level accuracy while cutting task completion cost to roughly one-third of running a top-tier model alone.

Partner results are the more interesting evidence, because they were measured on real workloads:

Notice the shape of every number: large cost reduction, small or no accuracy cost. That is what a good router buys you.

The tradeoff nobody puts on the slide

Routing adds a decision point, and decision points fail. Three things to plan for:

1. Misroutes are silent. A frontier-worthy request sent to a small model doesn't error — it returns a confident, mediocre answer. You need evals on the routed path, not just the model.

2. The 6% accuracy tradeoff is not free everywhere. For a support summary it's noise. For a compliance decision or a medical intake it isn't. Route by task risk, not just task cost.

3. Two systems to version. Model updates and router policy updates can each regress you. Pin both, and change one at a time.

What we'd actually build with this

If you're running agents for a service business, the practical pattern looks like this:

That's not an exotic setup. It's the difference between an agent that's a line item and one that pays for itself.

Where to get it

Nemotron 3.5 Lightning is available on Hugging Face, ModelScope, OpenRouter, and build.nvidia.com. NeMo Switchyard is open source and already wired into Kong AI Gateway, LiteLLM, LangChain, and Hermes.

If you'd rather not assemble the routing layer, the evals, and the fallback logic yourself, that's the part we do. We scope the workflow, pick the model mix, and have it deployed in 14 days.