The University of Idaho runs MindRouter, an open-source LLM inference gateway, in production on its own GPU infrastructure. According to a PEARC'26 short paper by Lucas Sheneman, director of Research Computing and Data Services at the university, the system has been in production since early 2026 and spans 13 GPU nodes with 40 GPUs, including NVIDIA H200, H100, RTX A8000 and A4000 cards, serving 64 open-weight models across Ollama and vLLM backends. It is designed for FERPA-regulated, controlled unclassified and export-controlled data, and all data remains on campus.
MindRouter is a containerized Python application built on the FastAPI ASGI framework. Its gateway exposes OpenAI-compatible endpoints, Ollama-compatible endpoints and an Anthropic Messages API endpoint. Instead of point-to-point translators, a canonical intermediate schema normalizes chat requests, messages, tool definitions, embedding requests and streaming responses, with one inbound translator per client protocol and one outbound translator per backend engine. An application using the Anthropic SDK can therefore target a model hosted on Ollama, and structured outputs and tool calling are preserved across protocols.
Requests are ordered by a Weighted Deficit Round Robin scheduler adapted for LLM inference. Each user belongs to a group with a scheduling weight, for example administrator 10, faculty 5, staff 2 and student 1. The scheduler keeps per-user deficit counters, lets burst credits accumulate during idle periods so one user can consume idle capacity, and decays them quickly under contention. Backend selection first filters on hard constraints such as model availability, modality support, capacity and memory fit, then ranks candidates on soft factors including whether the model is already loaded in GPU memory, GPU utilization, observed latency, queue length and recent throughput.
The cluster is modeled as physical nodes and logical backends. Each node runs a lightweight sidecar agent, a minimal FastAPI service using pynvml, that reports per-GPU utilization, memory, temperature and power draw; the gateway polls each sidecar once and derives backend-level utilization from the cached node data. Automatic context-length capping for vLLM and Ollama prevents GPU memory oversubscription.
User provisioning is integrated with Azure AD single sign-on through Microsoft Entra ID with just-in-time provisioning, and institutional groups determine scheduler weights and quotas. API keys are stored as Argon2 hashes and issued through a self-service dashboard, giving programmatic access from the OpenAI Python SDK, LangChain, LlamaIndex, IDE extensions and agentic harnesses. The security model, which includes role-based access control, non-root Docker containers behind nginx TLS termination and full request and response audit logging, was validated through institutional CISO review.
Since early 2026 the deployment has processed 3.7 billion tokens.