Superhuman, the AI-native email client now part of Grammarly, jointly engineered with Databricks a GPU inference platform that handles more than 200,000 queries per second across Superhuman's email AI features — Write with AI, Ask AI, Auto Summarize, and Instant Reply — using a combination of elastic data-streaming (EDS) power-of-two load balancing, image accelerator integration, and FP8 quantization that lifted per-pod throughput by about 60% while keeping p99 latency low.
The platform was built because Superhuman's email AI features were originally served from a more traditional autoscaling deployment that did not exploit the high concurrency GPU inference patterns that large language models can support when batching is done correctly. As features like Write with AI and Instant Reply became core to the product experience, the engineering team needed an inference layer that could handle bursty, request-rate-driven load from millions of mailboxes while preserving the millisecond-level latency that users expect from a keyboard-driven email client. The team worked with Databricks to apply techniques from large-scale ML serving to the consumer product workload.
The technical architecture combines several innovations. Elastic data-streaming (EDS) uses power-of-two-choices load balancing across GPU pods so that each incoming request is routed to the pod with the smaller of two randomly selected queue depths, dramatically reducing tail latency under bursty load compared with single-random or round-robin load balancing. Image acceleration offloads request decoding and tokenization to dedicated accelerator hardware, freeing the GPU compute budget for actual model inference. FP8 quantization reduces the model's memory footprint and arithmetic intensity, allowing each GPU pod to serve roughly 60% more requests per second at the same quality bar. The combined effect is a serving layer that scales elastically with request volume while keeping per-pod economics efficient enough to make consumer email AI viable at Superhuman's user count.
The operational impact is that Superhuman can now serve 200,000-plus QPS across its email AI features with a small fleet of GPU pods, each running at roughly 60% higher throughput than the previous generation, with p99 latency that meets the product's interactive response budget. The same serving patterns are now being reused for additional Superhuman AI features beyond the original four, and the joint engineering work has become a reference design inside Databricks for consumer-scale GPU inference. The architecture is documented publicly by both teams as a case study in elastic, high-concurrency LLM serving.