China Merchants Bank deploys DeepSeek-V4 Flash large-EP inference on domestic AI chips via SGLang RBG — industry first

Title

China Merchants Bank deploys DeepSeek-V4 Flash large-EP inference on domestic AI chips via SGLang RBG — industry first

Content

China Merchants Bank (CMB) deployed DeepSeek-V4 Flash large-EP inference on Huawei Ascend NPU production clusters — the industry's first DeepSeek-V4 large-EP inference rollout on a domestic AI chip stack with Kubernetes + SGLang RBG cloud-native orchestration. The production topology is 1 Router + 2 Prefill + 2 Decode instances, each inference instance spanning 16 NPUs (1 Leader + 15 Worker Pods) — 64 inference NPUs + 1 Router total. "Large-model inference is moving to distributed clusters and disaggregated architectures, but Kubernetes native primitives (Deployment, StatefulSet) were not designed for inference scenarios requiring multi-role collaboration, topology sensitivity, fast reliable upgrades, and fault linkage." — CMB IT Department Architecture Management Team, InfoQ (2026-06-10). DeepSeek-V4 Flash uses MoE, so the standard pattern is PD-disaggregation + large-EP — Prefill and Decode as separate instance groups, unified via a Router. On Kubernetes-managed domestic heterogeneous compute, the complexity far exceeds ordinary microservices. Three-tier nested multi-role topology needs three separate YAMLs plus hardcoded cross-role refs (Router alone needs 32 --prefiller-hosts + 32 --decoder-hosts). hostNetwork port conflicts arise (each Pod ≥4 ports). DNS races cascade on Pod drift-restart. And faults cascade three levels — collective comms (HCCL/NCCL) → Bootstrap session table → Transfer engine (Mooncake, NIXL) RDMA QP cache — beyond native restartPolicy. Evolution followed three steps. The initial approach used getent hosts + hardcoded ports in SGLang commands — hours-per-deploy and error-prone. Then CMB adopted SGLang RBG (RoleBasedGroup), a Kubernetes API extension that unifies Router/Prefill/Decode under one CR; the Controller auto-handles port allocation, generates a service-discovery ConfigMap and rebuilds on fault. The third step added an in-place upgrade flow: InPlaceIfPossible rolling update with 30s gracePeriodSeconds, and tini as PID 1 to fix SIGTERM-not-propagated-to-child under sh -c. The stack is Kubernetes + SGLang RBG Controller + SGLang 0.5.9 + Huawei Ascend NPU + Ascend disaggregation backend + RDMA + Mooncake/NIXL. Four core mechanisms make it work. Dynamic port allocation uses two scopes — RoleScoped (role-consistent, e.g. bootstrap) and PodScoped (per-Pod, e.g. HTTP) — via allocation and reference annotations, exposing them as container env vars. Three-layer service discovery flows from env vars (RBG_GROUP_NAME etc.) to a topology ConfigMap (/etc/rbg/config.yaml) to component-level component-discovery refs. A ClusterEngineRuntimeProfile cluster-level CRD encapsulates NPU driver init, Service Registration Sidecar and Metrics Normalization Sidecar; workload YAMLs only reference profileName (e.g. "ascend-npu-runtime"), enabling chip-agnostic migration. The Metrics Normalization Sidecar rewrites SGLang's sglang_num_queue_reqs, vLLM's vllm:num_requests_waiting and Dynamo's dynamo_pending_requests into one unified Prometheus metric namespace. In production the topology runs 1 Router + 2×16 = 32 Prefill NPUs + 2×16 = 32 Decode NPUs, with cross-host Prefill↔Decode KV Cache over RDMA and hostNetwork: true. Dynamic port allocation enables multi-replica-per-node, breaking the nodeSelector "one instance per node" constraint. readinessProbe waits for multi-minute model load; livenessProbe detects stuck inference. A standalone Decode Pod restart cannot recover service — Bootstrap retains stale worker registrations, and the Transfer engine retains stale RDMA QP cache — so instance-level rebuild is required. CMB's stated roadmap is a "best-fit deployment" strategy: dynamically selecting targets based on each model's performance and accuracy across different chips. Three concrete next steps follow. First, EngineRuntimeProfile will abstract hardware differences across domestic AI chips (Ascend, Cambricon, Hygon, Iluvatar) so DeepSeek-V4 Flash runs on any chip class. Second, in-place upgrades will continue to be favoured as SGLang and vLLM evolve, to avoid tens-of-minutes service-window jitter. Third, PD-disaggregation + large-EP will be abstracted into a reusable banking template that other joint-stock banks can adopt.