The Partnership for an Advanced Computing Environment (PACE) at the Georgia Institute of Technology built SPIN (Service for Persistent INference), a fully on-premises AI gateway that gives users one OpenAI-compatible API endpoint in front of several inference servers. It targets HPC sites that manage GPUs through a batch scheduler rather than Kubernetes or OpenShift.
The gateway is LiteLLM with a PostgreSQL database for user management and usage tracking. Both containers run in a single Podman pod on a CPU-only virtual machine, defined as systemd units through Quadlet so that the pod gets systemd dependency ordering, startup and recovery behaviour. Administrators issue API keys per user and per team and set rate limits, token pricing and budgets at global, user or team level.
Inference runs on GPU nodes of the existing Slurm cluster. Each server is a Slurm job with a 12-hour walltime limit, and scrontab tries to resubmit it every 5 minutes with --dependency=singleton, so a new job starts only when none is running. vLLM runs as an Apptainer instance from the vllm/vllm-openai image. Before the server starts, the job script finds an open port and generates a random API key; afterwards it calls the LiteLLM API to add the model or to update its hostname, port and key. User requests name only the model and the fixed LiteLLM endpoint, so they keep working unchanged. A per-GPU Slurm reservation and models cached on network storage keep each restart to an observed 5 to 10 minutes of downtime.
Image generation uses NVIDIA Triton Inference Server serving Stable Diffusion XL, with PyTorch models converted to ONNX and compiled into TensorRT engines. Because LiteLLM has no native Triton support for image generation, the team wrote an OpenAI-compatible FastAPI adapter exposing /v1/images/generations and /v1/models; it builds the input tensors, calls Triton through tritonclient.http and returns base64-encoded PNG images.
The initial models are Meta-Llama-3.1-70B-Instruct (AWQ INT4) for text, InternVL3_5-14B-Instruct for image and video to text, granite-embedding-278m-multilingual for text embeddings and stable-diffusion-xl-base-1.0 for text to image.
In Locust load tests sending 1000 concurrent requests per second against a mock inference engine, the gateway on a VM with 8 vCPUs and 8 LiteLLM workers returned 757.88 responses per second, 76% of the ideal rate; a bare-metal host with 8 workers reached 981.85.
SPIN is in limited availability for pilot groups of students and researchers. Students in the Human-Augmented Analytics Group built a Model Context Protocol tool that uses the image and video model to detect and describe birds in wildlife camera footage. The Georgia Tech Open Source Program Office adapted its training tutorials to SPIN. Pilot accounts are provisioned manually by PACE staff. Configuration and code are published on GitHub under GPL v3.0.