Oossa

Meta says JiTTests caught 74 privacy bugs before release

Meta’s AI-generated tests flagged 74 serious privacy bugs in the first half of 2026. The company says engineers judged 60% of reported findings to be real issues.

By Published by Oossa: 2 min read

Katie Moum · Unsplash

Meta has put its Catching JiTTests system into production, using large language models (LLMs)—the kind of AI behind chatbots—to generate tests aimed at finding bugs in code changes before they ship. Meta says the system helped catch 74 serious privacy bugs during the first half of 2026, before those changes reached production.

The company also says the system lowered the overall number of privacy incidents it classifies as severe by 20% during that period. Engineers who engaged with its findings reported a 60% true-positive rate, meaning they considered about six in ten flagged issues to be genuine problems. These are Meta’s results; the company has not provided an independent evaluation in the supplied material.

How does it work?

JiTTests differ from AI tools that read code and comment on possible faults. Meta’s system generates executable tests designed to fail when a code change causes a behavior it is meant to catch. The tests target changes that Meta’s risk-scoring system considers more likely to cause serious regressions.

The system runs overnight on spare computing capacity and examines risky code changes submitted the previous day. It uses two approaches: one asks an LLM to infer what a change is meant to do and where it could go wrong; the other treats the change as potentially faulty and generates tests to expose differences in behavior. Tests that fail become candidates for review.

Meta then uses a rules-based checker and an ensemble of LLMs—several models assessing a result—to filter likely false alarms. Engineers set criteria and review outcomes at an aggregate level rather than checking every generated test. Meta calls this a “human-on-the-loop” approach.

What do the results cover?

Meta says 589 engineers engaged with the system, representing 64% of its outreach. It has not said in the supplied material how many code changes were tested or how the 60% true-positive rate was measured beyond engineer feedback. The figures therefore describe Meta’s internal deployment, not a result established across other companies or codebases.

For Meta engineers, the system adds another check before code lands, with the goal of finding some problems that ordinary tests may miss. It is part of Meta’s own infrastructure; the supplied information does not describe a public tool or access for outside developers.

Why it matters

For Meta engineers, JiTTests offers an additional chance to catch privacy bugs before code ships, without requiring them to inspect every generated test. Meta says the system reduced severe privacy incidents, but the figures are internal company results and do not show whether the approach would work as well elsewhere.

Was this article useful?
Share

Read next

Oossa · Newsletter

The week in AI, explained

Every Monday: the stories worth knowing, in plain language. Free, no spam.