Slack's engineering team describes how it evolved the infrastructure that serves large language models for Slack AI features such as channel summaries, Recaps and AI Search, from a first deployment in early 2023 to a multi-cloud architecture in 2026. The 28 May 2026 post by Shaurya Kethireddy, a lead member of technical staff, sets out four phases, each driven by enterprise security requirements, GPU scarcity, regional outage risk and the speed at which better models appear.
Slack AI first ran on AWS SageMaker, chosen for its security, FedRAMP compliance, model availability and control. Anthropic models were hosted through an escrow virtual private cloud, a zero-knowledge arrangement in which Slack's data stayed private to Slack and the provider's model weights stayed inaccessible to Slack. Containers were deployed across several AWS regions, and Slack's teams managed cross-region IAM roles, balanced routing across model endpoints, capacity planning and auto-scaling themselves. Initialisation times prevented instant scaling, A100 and H100 GPU instances were often unavailable, and idle capacity had to be held to meet peak service levels; by early 2024 the team was relying on On-Demand Capacity Reservations and cron-based scaling. New model versions also reached Amazon Bedrock weeks or months before SageMaker.
In mid-2024, after Bedrock reached FedRAMP Moderate, Slack migrated. Bedrock measures capacity in Model Units, each a fixed throughput in tokens per minute, so the team stopped scaling individual GPU instances. Provisioned Throughput served latency-sensitive features such as channel summaries, and On Demand capacity served bursty scheduled work such as Recap, with provisioned capacity moved first. The migration went through Legal, Security and FedRAMP sign-off, load tests and shadow requests to find the Model Unit count that matched the SageMaker baseline, A/B tests and evaluation frameworks comparing output quality and latency side by side, and a gradual traffic shift behind feature flags that could be reversed in seconds. Slack reports zero customer-facing incidents. Capacity work shifted to forecasts several weeks ahead shared with AWS account teams, and AI Search could be upgraded quickly to newer high-reasoning models. Two gaps remained: provisioning for the US East and West Coast morning peak left capacity idle during APAC and EU hours and at weekends, and provisioned commitments of one to six months delayed moves to better models.
The third phase shifted more traffic to On Demand capacity, which mattered most for features with a tenfold gap between peak and off-peak usage and let the team move a feature to a new model within a day once it passed internal quality bars. A hybrid routing strategy kept high-volume, latency-sensitive features on provisioned capacity, sent asynchronous work such as nightly Recaps to on-demand capacity, and spilled requests over to on-demand endpoints whenever a surge exceeded reserved limits. Bedrock's routing across US regions found capacity while keeping to Slack's regional data boundaries. To offset the shared-capacity risk, Slack gave every AI feature a model hierarchy that falls back automatically when the primary model degrades, measured by elevated time to first token, throttling errors or a downward trend in customer feedback.
In early 2026 Slack added Google Cloud Vertex AI, both for provider-level redundancy and to reach models available only on other clouds. The integration involved Security, Risk and Compliance, Trust and Integrity, AI Quality and Legal teams, and used secretless authentication and an API normalisation layer so that, for example, a rate-limit response from one provider and a throttling exception from another are handled identically. The resulting routing layer selects a model for each feature from internal quality metrics and designates backup models; routes a percentage of traffic to a new model for A/B tests, which cut the feedback loop for Recaps model trials from weeks to days; and runs a circuit breaker that trips on elevated time to first token, 5xx error spikes or a p90 latency threshold, diverts traffic to a healthy alternative, then lets a small trickle back through and ramps it up as the endpoint recovers. Matching models to features gave about a 10% improvement in quality metrics for complex reasoning tasks and about a 67% reduction in latency for high-velocity, low-token workloads.
Slack lists the costs of running across clouds: provider-specific APIs, error codes and rate limits that needed normalising, a unified monitoring stack built to combine telemetry from each cloud into one view for on-call engineers, billing instrumentation to attribute cost per feature as traffic moves between providers, and on-call engineers who must now work across several cloud environments. The team treats an LLM service that is up but slow as failing, counting p90 spikes and feedback trends as soft failures in its routing decisions.