MiniMax reduced compute resource consumption by 50% on a unified Alibaba Cloud data warehouse, and further refinements brought overall compute usage down by 75%, according to Alibaba Cloud's case study. Data lifecycle management policies lowered storage costs by 40%, and key data now lands in the warehouse approximately one hour earlier. The platform handles tens of PBs of total data, with daily processing volumes reaching hundreds of TBs.
Starting in 2024, MiniMax products including Hailuo AI, MiniMax Audio and Talkie grew rapidly at home and abroad, and user data scaled to tens of petabytes. MiniMax had adopted different cloud providers in China and overseas and deployed separate data platforms, so teams maintained multiple development standards across disparate engines. Its data governance tools were built on a customized version of the open-source Dolphin Scheduler, which was costly to iterate and maintain, and running dual systems meant permission management, resource monitoring and troubleshooting had to be done on both platforms.
Alibaba Cloud helped MiniMax build a globally unified, cloud-native data warehouse centered on DataWorks. DataWorks real-time data integration synchronizes heterogeneous sources into the warehouse, with metadata management, quality monitoring and access control. Realtime Compute for Apache Flink processes Kafka streaming data, Hologres handles real-time ingestion, updates and analytics with sub-second response times, MaxCompute runs batch processing and offline analytics, and Elasticsearch serves full-text search and ad-hoc queries. Object Storage Service acts as the cold data tier, integrated with MaxCompute for hot and cold data tiering. Using DataWorks visual ETL, MiniMax synchronizes full and incremental data from source systems directly into Hologres, and cross-engine federation between MaxCompute and Hologres decoupled real-time storage from offline computation.
For LLM training, MiniMax then built a fully managed Data + AI data processing platform on MaxCompute using the MaxFrame distributed computing framework, replacing legacy open-source solutions. Problems it addressed included out-of-memory errors and failed full-dataset MinHash deduplication tasks during preprocessing. MaxFrame provides distributed operators compatible with Pandas and MinHash, supports distributed processing and offline inference, and offers a Python environment with custom Docker images. With MaxCompute's mix of monthly reserved and on-demand elastic resources, MiniMax can scale to hundreds of thousands of CPU cores within minutes, compute resource utilization improved by 30%, and operational resource investment fell by 50%. Batch inference for FastText text classification runs on MaxCompute's elastic CPU resources.
MiniMax and Alibaba Cloud say they will continue joint innovation in large-model data preprocessing and multimodal data processing.