hollowtest: find tests that pass but prove nothing

Coverage can be green while your test suite is empty of proof. That is not a hypothetical. Stack Overflow’s 2025 developer survey found that a majority of developers spend more time fixing almost-right AI output — and tests are one of the places that “almost right” hides best. An AI assistant will happily write a function, then write a test that calls it, prints something, and ends with assert result is not None. Coverage goes up. Confidence should not. I built hollowtest to catch those tests statically, before they land in main. The problem Mutation testing is the gold standard for “do my tests actually catch bugs?” It is also slow, noisy, and a poor default for every PR. Style linters do not care whether you asserted anything meaningful. Coverage only asks whether a line ran. What we needed in the AI-coding loop was a fast, deterministic gate: did this test body contain real proof, or is it hollow? Hollow patterns I kept seeing in AI-generated suites: Empty bodies — pass, a docstring, or an ellipsis left as a “TODO test” No assertions — call the code, maybe print, never check a result Tautologies — assert True, assert x == x Existence theater — only assert result is not None / assert result / assert len(x) > 0 Mock theater — mock.assert_called_once_with(...) and nothing about real outcomes All of these pass. All of them inflate coverage. None of them prove behavior. What hollowtest does pip install git+https://github.com/SybilGambleyyu/hollowtest.git hollowtest tests/ Enter fullscreen mode Exit fullscreen mode It walks Python test files (pytest and unittest conventions), finds test functions, and classifies their assertions via the AST. No network, no model, no API key — stdlib only. Rule Severity Meaning HT001 error No assertions HT002 error Only tautologies HT003 warning Only presence/truthiness checks HT004 warning Only mock call verifications HT005 error Empty / docstring-only / pass Meaningful patterns are left alone: assert result == expected, self.assertEqual, with pytest.raises(...), or a mock check plus a real value assertion. Example against a tiny sample: test_app.py:12:1: warning HT003 in test_add_exists Existence assertions only: Only checks presence/truthiness ... test_app.py:18:1: error HT005 in test_placeholder Empty test body: Test body is empty, docstring-only, or just `pass` test_app.py:23:1: error HT001 in test_smoke_print No assertions: No assert / self.assert* / pytest.raises found ... Enter fullscreen mode Exit fullscreen mode Exit codes work for CI (--fail-on error|warning), and there is JSON plus GitHub Actions annotation output. What it is not Not coverage. Coverage says a line ran; hollowtest asks whether any assertion proved something. Not mutation testing. It will not tell you if your equality check is too weak for every bug class. It catches the empty and trivial cases in milliseconds. Not a style linter. Use ruff for that. Python only, for now. Why ship this The gap I cared about was not “another AI that reviews your AI.” It was a boring, reliable check that treats test hollowness as a first-class defect class. If your agents write tests (or your teammates paste them), run this on the suite the same way you run the suite itself. # GitHub Actions sketch - run: pip install git+https://github.com/SybilGambleyyu/hollowtest.git - run: hollowtest tests/ --format github --fail-on warning Enter fullscreen mode Exit fullscreen mode Get it Source: github.com/SybilGambleyyu/hollowtest Release: v0.1.0 If you find false positives or hollow patterns it misses, open an issue — that feedback is the roadmap. MIT licensed. Zero runtime dependencies. Python 3.10+.