---

Resource Evaluation: "Just-in-Time Catching Test Generation at Meta"

Source: https://arxiv.org/abs/2601.22832 Article title: "The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive It" Author: Mark Harman (Meta) — founding figure of mutation testing and search-based software engineering Date: January 30, 2026 Type: Industrial research paper (arXiv cs.SE) — production deployment at Meta Evaluated: 2026-03-22

---

📄 Content Summary

• Core concept: JiTTests (Just-in-Time Tests) are LLM-generated tests created on-the-fly at PR submission time, targeting regressions introduced by a specific code change rather than general code quality
• Mechanism: Infer intent from diff → create mutants (deliberate faults) → generate tests to catch those mutants → ensemble rule-based + LLM assessors to filter false positives → report only true positive failures
• Production results at Meta: 41 candidate catches reported to engineers, 8 confirmed true positives, 4 preventing serious failures from reaching production; assessors reduce human review load by 70%
• Scale context: Deployed on a codebase of hundreds of millions of lines of code
• Key numbers verified by Perplexity: code-change-aware methods improve catch generation 4x over hardening tests and 20x over coincidentally failing tests
• Paradigm shift: Tests are ephemeral — generated per-PR, never committed, requiring zero maintenance

---

🎯 Relevance Score

Score: 4/5

Justification: This is not a think-piece — it is an industrial paper with production numbers from Meta at scale. The guide's testing section (~3,500 lines across TDD, BDD, ATDD, and methodologies) does not address what happens when AI generates code faster than humans can write and maintain test suites. Section 9.18.6 explicitly states "tests are the specification for agents" and "tests written manually, NOT delegated to agents" — accurate for today, but without a response to the 10x velocity problem that agentic development introduces. JiTTests are the most rigorous published answer to exactly that problem. The gap is real and will become more visible as the guide's audience shifts toward reviewing AI-generated PRs rather than writing code themselves.

---

⚖️ Comparative Analysis

---

📍 Integration Recommendations

Where: guide/core/methodologies.md, after the ATDD block (~line 199). A 150-200 word subsection titled "Testing in AI-Accelerated Workflows" or "When Code Outpaces Tests."

Secondary: Link from guide/workflows/tdd-with-claude.md (advanced patterns section, currently ~line 244) as a forward pointer.

What to include:

1. The core tension: TDD assumes human-paced code authoring; agentic development breaks that assumption
2. JiTTests as the most rigorous industrial response: intent inference at PR boundary, ephemeral mutation-based test generation
3. Production context: deployed at Meta at scale, 70% false-positive reduction, 4 serious failures prevented
4. Honest framing: no open-source implementation yet; engineers can approximate today by prompting Claude to generate regression-targeting tests before merging a PR, then discarding them
5. Link to arxiv paper for readers who want depth

Priority: Medium. The testing section is not broken today — this is a future-proofing addition that will age better than the current "humans write tests, agents implement" framing.

---

🔥 Challenge (technical-writer agent)

The agent challenger pushed the score from an initial 3/5 to 4/5 based on:

Arguments for higher score:

• Mark Harman's authorship is load-bearing: he built the theoretical foundations of mutation testing and now leads industrial application at Meta. This is not a community post.
• The "catching" vs "hardening" distinction is conceptually important and entirely absent from the guide. Ephemeral tests that fail by design are a different category from everything the guide currently documents.
• The "death of traditional testing" title is rhetorical — the paper's actual argument is more measured: traditional testing practices are mismatched to AI-generated code pace, not obsolete.

Risks of NOT integrating: The guide's testing section will age poorly as the audience shifts from code writers to code reviewers. No section currently answers "how do you test 200 lines of agent-generated code per hour?" That's an increasingly common situation with no guidance today.

Score adjusted: 4/5 (up from initial 3/5)

---

✅ Fact-Check

Note: The article text provided is a blog summary — it omits all production numbers. The actual paper has substantially more substance than the article implies. The article underestimates the resource's value.

Corrections: No corrections needed to the paper claims. The article framing ("death of traditional testing") is rhetorical overreach; the paper's actual claims are measured and verified.

---

🎯 Final Decision

• Final score: 4/5
• Action: Integrate
• Confidence: High — production numbers verified, paper is real, gap in guide is real
• Timeline: Medium priority — include in next methodologies update