Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale

• Over the last year we’ve been sharing our work leveraging large language models to catch bugs on the fly — a novel approach we call Catching Just-in-Time Tests (JiTTests).
• Now, we’re sharing the results and positive impact we’ve seen with Catching JiTTests deployed into production.
• Our paper, “Just-in-Time Catching Test Generation at Meta,” fully details our initial deployment and testing with Catching JiTTests.

Every code change improves our platform, but it also introduces new complexity. With hundreds of millions of lines of code constantly evolving at Meta, traditional testing can’t keep pace. Catching risks proactively, before code lands, enables us to identify risks earlier, respond more effectively, and deliver reliable experiences at scale.

Over the last year, we’ve shared our research into applying large language models (LLMs) to generate software tests. More specifically, we’ve shared our work with Just-in-Time Tests (JiTTests), a novel testing approach where we leverage LLMs to catch bugs on the fly, even ones that traditional testing might not catch.

In the five months since we first unveiled this work, we’ve deployed our Catching JiTTests approach to determine its scalability and applicability to the wider industry. Today, Catching JiTTests are no longer a research prototype; they’re part of Meta’s infrastructure, and they’re helping our engineers catch more bugs with less load and a smaller chance of false positives.