SMBC Americas Migrates 170 Risk Attributes to Azure Databricks, Cuts Shuffle I/O 67% and Runtime 40-60%

SMBC Americas, the U.S. subsidiary of Sumitomo Mitsui Banking Corporation, migrated a core risk-data workload of 170 attributes to Azure Databricks, where re-architected Delta Lake pipelines reduced shuffle I/O by 67% and improved end-to-end job runtime by 40-60% over the legacy on-premises Hadoop-based risk data platform. The migration was driven by SMBC Americas's need to consolidate regional risk data onto a single governed lakehouse that could serve cross-region analytics and machine-learning workloads without duplicating infrastructure in each location. The legacy Hadoop environment required nightly batch windows long enough to constrain the risk team's ability to iterate on new analytics use cases, and shuffle-heavy Spark jobs were the dominant cost driver. Azure Databricks was chosen because it offered a managed Spark and Delta Lake environment that could integrate with SMBC's existing Azure Active Directory controls, while Photon-accelerated compute and Adaptive Query Execution addressed the shuffle bottleneck directly. The technical architecture uses Delta Lake as the storage layer for risk data, with partitions and Z-ordering tuned to the access patterns of the 170 core risk attributes. Spark jobs were rewritten to use higher-order functions, fewer wide transformations, and Adaptive Query Execution so that the runtime hot loop runs as a single stage where possible, cutting the shuffle stages that had previously dominated wall time. Unity Catalog provides the cross-region data sharing and access controls that SMBC Americas needs to expose the same risk datasets to both risk analysts and ML engineers without copying data between environments. The operational impact is that the 170-attribute risk workload runs 40-60% faster end-to-end on Azure Databricks than on the legacy Hadoop platform, with shuffle I/O reduced by 67%, freeing nightly batch windows and allowing risk analysts to iterate on new analytics during business hours. The same lakehouse foundation now supports additional AI-driven risk, treasury, and credit workflows that SMBC Group has been recognised for in the 2026 Databricks Customer Awards, with ML models for early-warning indicators and credit memo automation running on the same governed data products.