No. 15of 22WaypointHackathons
Flaky-test detector
FlakyTestX
A developer tool that reruns a pytest suite many times, scores how flaky each test is, and charts the result. Built in a day for the Kyiv QA Automation Hackathon 2025; on its own sample suite it flagged the 1 flaky test among 13.
- When
- Apr 2025
- Status
- Built
Progress
Progress65%
Detector, reports and dashboard work end to end. The language-model insight step ran only in fallback mode.
What comes next
- Run it on a real open-source project's test suite and report what it finds.
- Do one live run of the language-model step.
1 of 13
tests flagged as flaky in the saved run
The project's own sample suite, with the flakiness injected on purpose. Not a real-world codebase.
Saved run results (results/flaky_results_20250402_*.json)Verified. Backed by a file or public page that has been checked.Saved run results (results/flaky_results_20250402_*.json) — Read in the project audit of 13 November 2026.92.31%
suite stability
Simply 12 stable tests out of 13, on the same sample suite.
Saved run results (results/flaky_results_20250402_*.json)Verified. Backed by a file or public page that has been checked.Saved run results (results/flaky_results_20250402_*.json) — Read in the project audit of 13 November 2026.5 / 5
passes and failures of the flaky test over 10 iterations (flakiness score 1.0)
A separate 5-iteration run, the one in the pictures, scored it 0.80 with 2 passes.
Saved run results (results/flaky_results_20250402_104649.json)Verified. Backed by a file or public page that has been checked.Saved run results (results/flaky_results_20250402_104649.json) — Read in the project audit of 13 November 2026.Exhibit 01 · Gallery
A saved run, in the tool's own output
The summary of a 5-iteration run on the sample suite: 13 tests, 1 flaky. 
The tool's flakiness chart from the same 5-iteration run. In the 10-iteration run the same test passed 5 and failed 5, a score of 1.0. 
The tool's stability chart: 12 of 13 tests always pass. The suite is a sample, with the one flaky test planted.
01
What it is
A flaky test passes on one run and fails on the next with no change to the code, which makes a whole test suite hard to trust. FlakyTestX reruns a pytest suite many times, tracks every test's passes and failures, and gives each test a flakiness score and the suite a stability figure. It was built on 2 April 2025 for the Kyiv QA Automation Hackathon. No result from the event is on record.
02
What was built
A detector that repeats the suite with pytest-repeat and xdist, JSON reports and matplotlib charts, a Streamlit dashboard, a script that runs the whole chain, and a sample test suite with flakiness injected on purpose. One more step sends each flaky test's context to a language model for likely causes, with a fallback for when the model cannot be reached.
03
What was found
The saved runs are on the project's own sample suite, not a real codebase. Over 10 iterations the tool analyzed 13 tests and flagged 1: an async API test that passed 5 times and failed 5. The other 12 were stable, which is where the suite stability of 92.31% comes from. In the saved run the language-model step returned only its fallback message, so the tool's explanations have not been shown working.
One question
The answer is on this page. Get it right and it goes in your passport on the map.
Challenge· stamped
FlakyTestX reran a sample suite of 13 tests ten times. How many did it flag as flaky?
One: an async API test that passed 5 times and failed 5. The other 12 were stable, which gives the suite stability of 92.31%. The suite is the project's own sample, with the flakiness injected on purpose. Source: the saved run results, read in the project audit of 13 November 2026.
Show the answer
1. One: an async API test that passed 5 times and failed 5. The other 12 were stable, which gives the suite stability of 92.31%. The suite is the project's own sample, with the flakiness injected on purpose. Source: the saved run results, read in the project audit of 13 November 2026.
- Stack
- Python
- pytest (pytest-repeat, xdist)
- Streamlit
- pandas
- matplotlib
- OpenAI SDK (optional insight step)