{"id":79491,"date":"2026-09-02T15:29:21","date_gmt":"2026-09-02T19:29:21","guid":{"rendered":"https:\/\/overcentral.com\/en\/?p=79491"},"modified":"2026-09-02T15:29:21","modified_gmt":"2026-09-02T19:29:21","slug":"switchyard-llm-proxy-79491","status":"publish","type":"post","link":"https:\/\/overcentral.com\/en\/switchyard-llm-proxy-79491\/","title":{"rendered":"Switchyard Routes and Translates LLM Calls Across OpenAI & Anthropic"},"content":{"rendered":"

Teams running coding agents<\/a> inevitably hit the same wall. Claude Code speaks the Anthropic Messages API, Codex CLI speaks OpenAI, and the model a team actually wants to serve sits behind vLLM, NVIDIA NIM, or Ollama. Rewriting the agent is not an option, so the translation layer has to live somewhere else. Switchyard<\/a>, released by NVIDIA under Apache 2.0, is designed to be that layer: a Rust proxy and library for LLM traffic that routes requests across providers, translates between OpenAI and Anthropic wire formats, records operational metrics, and exposes typed, composable routing algorithms. For teams wrestling with heterogeneous model backends, it offers a single point of integration that decouples the agent\u2019s API from the backend\u2019s API entirely.<\/p>\n

What Switchyard Does: Format Translation and Provider Routing<\/h2>\n

At its core, Switchyard functions as an intelligent intermediary. Clients keep their native API \u2014 they send requests in the format they always use. Switchyard decodes the inbound request into provider-neutral Rust types, runs a routing algorithm to pick a backend, re-encodes the request in that backend\u2019s own wire format, calls it, and translates the response, including streaming events, back into the shape the client expects. This decoupling is the fundamental point: the agent\u2019s API and the backend\u2019s API no longer<\/a> have to match.<\/p>\n

The server accepts three inbound formats: OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Any of the<\/a> three inbound formats can address any route, and each configured LLM client selects one upstream format of its own. This means a team running Claude Code, which natively uses the Anthropic Messages API, can route its traffic through Switchyard and have it translated on the fly to the OpenAI Chat Completions format for a backend like Codex CLI or vLLM. Conversely, a client using the OpenAI format can be translated to Anthropic Messages for a backend serving Claude models.<\/p>\n

How a Request Travels Through Switchyard<\/h3>\n

The flow is straightforward but powerful. A client posts a request to the Switchyard server. The server decodes that request into its internal, provider-neutral representation. A routing algorithm then selects the appropriate target backend. The request is re-encoded into the target\u2019s native format and sent upstream. The backend\u2019s response, including any streaming events, is translated back into the original client\u2019s expected format and returned. The entire process is transparent to both the client and the backend.<\/p>\n

Deployability and Maturity: An Evaluation Tool, Not Production Ready<\/h2>\n

Is Switchyard deployable? Yes, but with a critical caveat. The binary installs from crates.io and the launcher from PyPI, and it self-hosts anywhere, making it easy to spin up for evaluation. However, NVIDIA explicitly labels Switchyard as pre-alpha and experimental, warns it is not for production use, and expects the API and algorithms to change significantly before v1.0. This is a tool for teams to experiment with routing strategies and format translation, not a system to depend on in a customer-facing application.<\/p>\n

Three Ways to Run Switchyard<\/h2>\n

NVIDIA provides three distinct paths for running Switchyard, catering to different use cases and levels of integration.<\/p>\n

The Launcher Path for Coding Agents<\/h3>\n

The launcher path targets coding agents directly. Install the published tool with uv tool install --python 3.12 \"nemo-switchyard[cli]\"<\/code>codecodecodecode, then run switchyard launch claude<\/code>codecodecodecode, switchyard launch codex<\/code>codecodecodecode, or switchyard launch openclaw<\/code>codecodecodecode against a packaged deployment or your own TOML configuration file. This is the quickest way to get a Switchyard proxy running for a specific agent.<\/p>\n

The Server Path for Custom Deployments<\/h3>\n

The server path installs the standalone proxy with cargo install --locked switchyard-server<\/code>codecodecodecode. It validates a configuration with the --dry-run<\/code>codecodecodecode flag and serves on a host and port you choose. This path offers more control over the server process and is suitable for integration into existing infrastructure for evaluation purposes.<\/p>\n

The Library Path for Embedded Routing<\/h3>\n

The library path uses switchyard-libsy<\/code>codecodecodecode, which embeds the routing algorithms in a Rust application without owning an HTTP stack. This is a significant design choice. The library never calls a model itself; the algorithm decides which target to use and hands every model call back to the caller. This allows developers to integrate Switchyard\u2019s routing intelligence directly into their own Rust-based tools and services, without the overhead of a separate proxy server.<\/p>\n

Routing Algorithms: From Simple Passthrough to LLM Classifiers<\/h2>\n

The routing algorithms in Switchyard are its most compelling feature. They transform the proxy from a simple format translator into an intelligent traffic manager.<\/p>\n

Passthrough and Random Routes<\/h3>\n

The passthrough<\/code>codecodecodecode algorithm sends every request to a single target. This is the simplest route type, useful for basic format translation or testing a single backend. The random<\/code>codecodecodecode algorithm splits traffic across targets using optional relative weights. An optional seed reproduces the selection sequence, making it suitable for A\/B testing and cost experiments where reproducibility is required.<\/p>\n

The LLM Classifier Route<\/h3>\n

The llm_classifier<\/code>codecodecodecode route is where Switchyard starts to show its intelligence. It calls a classifier target for a capability verdict, then routes the request to either a weak or strong target. Configuration parameters like base_threshold<\/code>codecodecodecode, min_confidence<\/code>codecodecodecode, capability_elevated_floor<\/code>codecodecodecode, and session_affinity<\/code>codecodecodecode tune the classifier\u2019s behavior. If the judge cannot decide, the request falls through to the strong target. An alternative mode = \"escalation\"<\/code>codecodecodecode runs every turn on the weak tier first and lets the judge decide whether to rerun it on the strong tier. This is a powerful paradigm for optimizing cost and performance by routing simple queries to cheaper, faster models and complex queries to more capable ones.<\/p>\n

The Stage Router<\/h3>\n

The stage_router<\/code>codecodecodecode algorithm scores tool-result and agent-progress signals from recent turns to pick a capable or efficient target. This avoids the extra classifier call on most turns, making it more efficient than the LLM classifier for multi-turn agent interactions. It is designed for scenarios where the model\u2019s performance in the current context can be inferred from recent tool calls and progress signals, rather than requiring an explicit classification.<\/p>\n

Observability: Prometheus Metrics and Routing Logs<\/h2>\n

Switchyard exposes a rich set of metrics for operational visibility. GET \/metrics<\/code>codecodecodecode returns Prometheus text from the server\u2019s process-wide OpenTelemetry provider. The metric families cover requests, errors, model-call latency, full-turn latency, prompt tokens, completion tokens, cached tokens, cache-creation tokens, and reasoning tokens. Upstream HTTP attempts are tracked by outcome and code.<\/p>\n

The most distinctive metric is switchyard_routing_overhead_ms<\/code>codecodecodecode, which reports the algorithm\u2019s run time minus the call that served the request. This allows teams to directly measure the cost of routing decisions. Classifier calls are not subtracted, so an LLM-classifier route reports its classification time here, while passthrough<\/code>codecodecodecode and random<\/code>codecodecodecode report the sub-millisecond cost of picking a target. The metric\u2019s buckets start at 0.1 ms, providing fine granularity. Additionally, --routing-log-file<\/code>codecodecodecode appends a JSON record per completed response, and GET \/v1\/routing\/session-stats<\/code>codecodecodecode returns per-session call and token totals from that log.<\/p>\n

Configuration Structure: Clients, Targets, and Routes<\/h2>\n

A TOML deployment has three distinct layers. llm_clients<\/code>codecodecodecode define base URL, wire format, credential environment variable, and retry policy. targets<\/code>codecodecodecode bind one upstream model ID to a client. routes<\/code>codecodecodecode expose one client-visible model ID and its algorithm. Secrets never sit in the file, since api_key_env<\/code>codecodecodecode only names an environment variable. max_retries<\/code>codecodecodecode defaults to 2 and applies to transport failures, timeouts, HTTP 408\/429, and 5xx responses. This layered configuration makes it easy to manage complex multi-provider deployments where the same model might serve different roles in different routes.<\/p>\n

Switchyard is a thoughtful and technically ambitious project from NVIDIA. Its combination of format translation, intelligent routing algorithms, and detailed observability makes it a compelling evaluation tool for any team building coding agents or other LLM-powered applications that need to interact with multiple model providers. The pre-alpha maturity warning is critical, but the architectural decisions \u2014 particularly the use of a Rust core and the library path for embedded routing \u2014 suggest a solid foundation. For teams looking to break the lock-in between their agent\u2019s API and their model\u2019s API, Switchyard is well worth the evaluation time.<\/p>\n","protected":false},"excerpt":{"rendered":"

Teams running coding agents inevitably hit the same wall. Claude Code speaks the Anthropic Messages API, Codex CLI speaks OpenAI, and the model a team actually wants to serve sits behind vLLM, NVIDIA NIM, or Ollama. Rewriting the agent is not an option, so the translation layer has to live somewhere else. Switchyard, released by […]<\/p>\n","protected":false},"author":7,"featured_media":82888,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/cards.overcentral.com\/cards\/en\/79491.png","fifu_image_alt":"Switchyard Routes and Translates LLM Calls Across OpenAI & Anthropic","footnotes":""},"categories":[31],"tags":[],"class_list":["post-79491","post","type-post","status-publish","format-standard","has-post-thumbnail","category-technology"],"fifu_image_url":"https:\/\/cards.overcentral.com\/cards\/en\/79491.png","fifu_image_alt":"Switchyard Routes and Translates LLM Calls Across OpenAI & Anthropic","_links":{"self":[{"href":"https:\/\/overcentral.com\/en\/wp-json\/wp\/v2\/posts\/79491","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/overcentral.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/overcentral.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/overcentral.com\/en\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/overcentral.com\/en\/wp-json\/wp\/v2\/comments?post=79491"}],"version-history":[{"count":0,"href":"https:\/\/overcentral.com\/en\/wp-json\/wp\/v2\/posts\/79491\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/overcentral.com\/en\/wp-json\/wp\/v2\/media\/82888"}],"wp:attachment":[{"href":"https:\/\/overcentral.com\/en\/wp-json\/wp\/v2\/media?parent=79491"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/overcentral.com\/en\/wp-json\/wp\/v2\/categories?post=79491"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/overcentral.com\/en\/wp-json\/wp\/v2\/tags?post=79491"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}