Checkr AI Background Screening - 1.5M+ Screenings

Title

Checkr AI Background Screening - 1.5M+ Screenings

Content

Checkr provides personnel background checks per month for more than 100,000 businesses, a process that requires generative AI (genAI) and machine learning tools to sift through massive amounts of unstructured data. The automation engine produces a report about each potential job prospect based on background information from various sources, categorizing criminal or other issues described in the report. Of Checkr's unstructured data, about 2% is considered "messy" — records that can't be easily processed with traditional machine learning automation software. Checkr initially tried OpenAI's GPT-4 large language model, which achieved only 88% accuracy on background checks (82% on messy data). Adding retrieval augmented generation (RAG) improved bulk accuracy to 96% but dropped messy data accuracy to 79%, with response times of 15 and 7 seconds respectively. Checkr's machine learning team then switched to an open-source small language model (SLM). Vlad Bukhin, Checkr's machine learning engineer, fine-tuned the SLM using data collected over years to teach what the company sought in employee background checks. The accuracy rate for bulk data rose to 97% (from 88%) and for messy data to 85% (from 82%). Query response times dropped to just half a second. The cost to fine-tune an SLM based on Llama-3 with 8 billion parameters was one-fifth of a 1.8 billion-parameter GPT-4 model. To fine-tune, Checkr used Predibase, a cloud platform that connects thousands of historical examples to fine-tune the Llama-3 SLM. After a few hours of work, Bukhin had a custom model built. Predibase also developed LoRAX, an open-source framework for serving hundreds of fine-tuned LLMs at low cost, using LoRA (low-rank adaptation) to customize a small percentage of model parameters while sharing the same base model across multiple use cases. The production deployment runs on a single A100 GPU, with multiple use cases served via LoRAX adapters. Cost dropped from $7,000-$12,000 per month with GPT-4 to $800 per month with the SLM. Checkr processes 1.5 million background checks per month, resulting in one complex charge annotation request every three seconds. "I don't know that I would have been able to run a production instance for this problem using GPT. These big models are very costly, and there's always a tradeoff between cost and scale," said Bukhin. The SLM approach enables Checkr to handle its scale while maintaining high accuracy on both bulk and messy data.