Meta deployed Catching Just-in-Time Tests (JiTTests) in its development infrastructure to surface potential regressions before code changes land. In its engineering account, Meta reports that the system played a key role in catching 74 serious privacy bugs at the code-diff stage during the first half of 2026, when the overall occurrence of severe privacy incidents decreased by 20%. These are Meta-reported results; the source does not isolate how much of the overall decrease was independently caused by JiTTests.
The system runs overnight on spare capacity and selects risky changes submitted on the previous day using a mechanism similar to Meta’s Diff Risk Score. Two test-generation workflows examine each change. An intent-aware workflow uses a language model to infer the change’s purpose and possible risks, constructs mutants representing potential defects, and generates tests to distinguish their behavior from the original. A dodgy-diff workflow treats the change itself as a potential mutant and produces tests that reveal behavioral differences from the parent revision. Tests that fail on the actual change become candidate catches.
To rank candidates and reduce false positives, Meta combines an ensemble of language models that reviews test code, execution traces, diffs and inferred intent with a deterministic rule-based assessor called RubFake. Engineers define criteria and review aggregate outcomes, and the system reaches out to engineers when ranked findings merit attention. Meta reports that 589 engineers engaged with JiTTests and reported a 60% true-positive rate.
Details
- City
- Menlo Park
- Organization
- Meta Platforms
- Continent
- North America
- Country
- United States
- Category
- Interactive Media & Services