ING Belgium Raises IT Incident Prediction F1 Score From 76% to 97% With Synthetic Training Data

A team at ING Belgium used synthetic data to lift the F1 score of an IT incident prediction model from 76% to 97%, which the case study describes as a 21% improvement. The case study was written by Jan Lennartz, Data Scientist in Site Reliability Engineering at ING Belgium, and Kalyan Veeramachaneni, Co-founder and CEO of DataCebo. The team set out to build a machine learning model that predicts whether a planned change will result in an incident, defined as Major, P1 or P2 events, and what type of incident it might be. With that prediction, teams can plan mitigations, communicate more broadly, enforce stricter change windows or delay the change. The challenge was that the number of changes linked to incidents was significantly smaller than what is optimal for finding patterns, and a model trained on such imbalanced data tends to overfit to incident-related patterns, reducing overall accuracy. The team collected historical data and completed data labeling, feature engineering, model training, testing and validation before deciding to explore whether synthetic data could help. Model training starts from historical, featurized, model-ready data in which each record is a change with a label showing whether it led to an incident. The team followed seven steps. They split the data into train and test sets. They trained a generative model on the training data using SDV Enterprise, the Synthetic Data Vault. They used SDV Enterprise's conditional sampling to generate synthetic records for the less common category, changes that led to Major, P1 or P2 incidents. They combined the real training data and the synthetic data into a balanced dataset. They performed feature selection to choose the most relevant attributes. They trained the incident prediction model with LightGBM. Finally they tested it on the test data set aside at the beginning. The resulting incident prediction model achieved a 97% F1 score, up from 76%. The authors note that reaching this result meant answering how much synthetic data to add, at what stage of model development to generate it, and how to test the model's robustness. They say a technical deep dive with best practices for augmenting training data with synthetic data will follow.