China Merchants Bank (CMB) runs DeepSeek-V4 Flash large-EP inference in production on domestic AI chips, orchestrated with SGLang RBG on Kubernetes. The live topology is 1 Router plus 2 Prefill and 2 Decode instances, each Prefill and Decode instance spanning 16 NPUs split into 1 Leader and 15 Worker Pods. Upgrading one instance from SGLang 0.5.8 to 0.5.9 takes about 5 to 8 minutes under the conventional RecreatePod path and a steady 3 to 4 minutes with in-place upgrade, against tens of minutes for a full-cluster rolling update.
Written by CMB's IT Department Architecture Management Team and published on InfoQ China on 2026-06-10, the article argues that large-model inference is moving from single machines to distributed clusters and disaggregated architectures, while the native Kubernetes workload primitives, Deployment and StatefulSet, were never designed for inference scenarios needing multi-role collaboration, topology sensitivity, fast and reliable upgrades and fault linkage.
DeepSeek-V4 Flash is a Mixture of Experts model with parameters in the hundreds of billions, experts spread across cards under Expert Parallelism while Prefill and Decode are split into instance groups coordinated by a Router. The structure nests three levels, roles, instances and worker Pods, which Kubernetes cannot express, so operators maintained three YAML files with cross-role references hardcoded; in the article's 2 Prefill plus 2 Decode example the Router command alone needs 32 --prefiller-hosts and 32 --decoder-hosts entries. Because cross-role KV Cache transfer over RDMA is latency sensitive, Pods run with hostNetwork: true and each inference Pod needs three to four ports, meaning two Pods on one node collide unless nodeSelector forces one Pod per node and blocks scaling.
The first setup resolved IPs with getent hosts and hardcoded port lists in the SGLang command line, which inherits a DNS race: Kubernetes refreshes DNS records only once the target Pod is Ready, so a lookup that fires too early returns a stale address and cascades into startup failure during drift restart. Faults cascade at three levels: within an instance, the collective communication library cannot hot-rejoin a single rank, so one failed worker blocks the whole communicator; across roles, the Bootstrap session registry and the Transfer engine's RDMA QP cache keep stale entries after a Decode restart, so Prefill transfers fail on session mismatch while the Router keeps routing into the dead link; and at cluster level, a naive health check plus restart policy turns one failure into a restart storm.
The team moved to SGLang RBG, a Kubernetes API extension that defines Router, Prefill and Decode as role groups under a single CR, then added an in-place upgrade path. RBG's controller delivers the four core mechanisms. Dynamic port allocation uses a global allocator picking randomly inside range isolation, with RoleScoped ports such as bootstrap and PodScoped ports such as HTTP and gRPC; allocation and reference annotations are resolved at Pod creation, so a reference like prefill.leader.grpc is injected as an environment variable and the one-Pod-per-node limit falls away. Service discovery runs in three layers, standardized environment variables, a topology ConfigMap mounted at /etc/rbg/config.yaml, and component-level annotations resolving a leader's address and port. A cluster-level ClusterEngineRuntimeProfile packages the chip driver init container, the metrics exporter and the runtime configuration, referenced by profileName so the same inference YAML can move between hardware.
Fault self-healing works at the RoleInstance level, where the default strategy RecreateRoleInstanceOnPodRestart rebuilds every Pod of an instance when any one of them fails; cascade protection pairs an in-memory LRU cache with a persistent Condition on the API Server and clears only once the instance returns to Ready. A CoordinatedPolicy keeps Prefill and Decode in step, allowing 1 percent skew during rolling updates and 10 percent during scaling. In-place upgrade comes in three forms, RecreatePod, InPlaceIfPossible and InPlaceOnly; the controller permits an in-place update only when the change is limited to container images, drains the Pod, waits out a 30 second grace period so long Prefill requests finish, patches the image and confirms completion by comparing the container ImageID, with tini as PID 1 so SIGTERM reaches the inference process.
The article is candid about what remains open: random port allocation carries a theoretical conflict risk when the port range is small, the ConfigMap reconcile cycle leaves a brief discovery inconsistency window, in-place upgrade is limited to image changes so volume or network edits still fall back to RecreatePod, and cross-role traffic removal after a Decode failure still depends on the inference framework itself. The stated next steps are to add elastic scaling on RBG that adjusts the Prefill to Decode ratio against live load, and to feed production problems and solutions back to the upstream open-source community.