AI ML in software testing: six techniques, how each one fails, and what still needs a tester

Where does AI ML in software testing actually help today?#

AI and ML help software testing in six places today, from writing tests to choosing which ones run, and each one fails in a way a tester still has to catch. The role of artificial intelligence and machine learning in software testing is narrower than tool pages suggest. That is because each technique ranks options and then acts on the top one.

Here is the map, with the source each section below is built on:

• Test generation. A large language model (LLM) writes test code from your source or a plan, as in GitHub Copilot, an AI coding assistant, or the Playwright Test Agents generator.
• Self-healing locators. A broken element lookup is swapped for the closest match, as in Healenium, an open-source self-healing library for Selenium tests.
• Visual diffing. Screenshots are compared with a model that skips rendering noise, as in Applitools Eyes, a visual testing service.
• Test selection. A model trained on past results picks the tests a change needs, as in Gradle's Develocity Predictive Test Selection and the Facebook paper.
• Flaky-test detection. Tests that pass and fail on the same code get flagged, mostly by reruns.
• Defect prediction. A classifier scores which files are likely to hold bugs, which a 2026 study on arXiv says needs checking per project.

In short, AI ML testing is not one product. It is six bets, and each bet has a known way to lose. When people search for artificial intelligence and machine learning in testing, these six are what they find under the label.

What is the future of software testing?#

The future of software testing, and of test automation in particular, already shows in shipped tools: models write, heal and select tests, and a tester decides what each may change.

Each of those jobs has a documented tool behind it. The Playwright Test Agents page describes a planner, a generator and a healer for browser tests. Meta's 2024 TestGen-LLM paper describes a generator that keeps a test only after it clears build, pass and coverage filters. And Develocity Predictive Test Selection uses a model trained on past results to choose which tests a change needs.

What the tools do not change is who answers for the result. Each one moves work from writing tests to reviewing them: a generated assertion, a heal patch, a skipped test. So the software tester's future is less typing and more judging, and what follows takes that review one technique at a time.

What is the payoff worth, and what does it cost in missed failures?#

Facebook's 2018 arXiv paper on predictive test selection, accessed October 2026, reports testing infrastructure cost halved while over 95% of individual test failures still reached developers. That is the clearest measured payoff of ML in software testing. Still, it is a compute saving, not a quality gain.

The paper says its method is "learned from a large dataset of historical test outcomes." It uses "basic machine learning techniques," in the paper's own words. In practice, the model looks at a change and at past results, then skips the tests least likely to fail.

However, read the bound the other way round. The 2018 Facebook paper guarantees over 95%, so up to one test failure in twenty can be held back on a given change. Because over 99.9% of faulty changes were still caught, the trade was judged worth it. But that judgement leans on a full suite running later, and that is the part people forget when they copy the idea.

How good are tests that AI writes, and what do they get wrong?#

In Meta's 2024 TestGen-LLM paper, accessed October 2026, 75% of generated test cases built, 57% passed reliably and 25% increased coverage. Also, those figures come from Meta's trial on the Instagram Reels and Stories products. Meta posted the paper to arXiv on 14 February 2024.

In brief, TestGen-LLM is Meta's tool for extending tests that people already wrote. And the key word in the paper is filters. According to the Meta paper, the tool checks that its tests "successfully clear a set of filters" before anyone sees them. That is how it deals with made-up code. In short, most tests a model writes do not survive a build. So a generator is only safe behind filters that build, rerun and measure each test before a person reviews it.

By contrast, off-the-shelf tools hand that filter to you. GitHub's Copilot docs, read in October 2026, say "you should always review the generated code." The same docs add that you should "add any additional tests that may be necessary." Also, the Playwright Test Agents page warns that generated tests "may include initial errors." Playwright is an open-source browser testing framework, and its generator turns a Markdown test plan into test files.

So what do these tests get wrong? First, they test what the code does, not what it should do. A bug can get a passing test written around it. Second, a test that passes reliably may still add nothing. In the 2024 Meta numbers, 57% passed but only 25% added coverage, so more than half of the passing tests added no coverage.

Does the model behind a test generator change what you get?#

Across 25 npm packages, the TestPilot study, accessed October 2026, measured median statement coverage of 70.2% with gpt3.5-turbo and 54.0% with StarCoder, against 51.3% for a non-LLM generator. TestPilot is a research tool that writes JavaScript unit tests. Schaefer and colleagues first posted the study to arXiv on 13 February 2023.

And given a different model, the same tool lost 16 points of coverage. For comparison, Nessie, the baseline, is a JavaScript test generator with no LLM in it. The 2023 study's authors tie the gap to "the size and training set of the LLM."

Therefore, a vendor's coverage figure tells you about the vendor's model on the vendor's code. Coverage from generated tests depends on the model and the prompt as much as on the tool. Since your code is not their code, run the generator on one of your own modules first. Then measure coverage before you trust it.

How do self-healing locators work, and how do they fail?#

Healenium heals a broken locator to the stored candidate whose match probability clears its score-cap, 0.5 by default in its README accessed October 2026, so a similar sibling element can win. With a score-cap of 0.5, the README says "healing will be performed for new healed locators where probability of match with target one is >=50%."

In other words, a self-healing locator replaces a failed lookup with the most similar element above a threshold. As a result, a renamed button can heal onto a look-alike, and the test passes while clicking the wrong thing.

How does a heal land on the wrong button?#

Take a made-up checkout page as an example. Your test finds the Place order button by its id, and a release renames the id. Next to it sits a Save draft button with the same tag, classes and parent. Say Save draft scores 0.62 as a match. Then it clears the 0.5 default from the README, read in October 2026, so the heal picks it and the test clicks Save draft. The run goes green, yet no order was placed and no step failed.

Why do absence checks and skipped tests hide failures?#

The second failure is quieter. For example, an absence check asks whether an element is gone. If healing is on, a removed Cancel button can heal onto a nearby button. Then "is Cancel shown?" comes back true. The README names this case and gives an opt-out "if you want to verify that element is not present on the page."

Similarly, Playwright's healer agent carries its own version of the risk. Per the Playwright Test Agents page, read in October 2026, the healer replays the failing steps. Then it "Inspects the current UI to locate equivalent elements or flows." It re-runs the test "until it passes or until guardrails stop the loop." Its output is "a passing test, or a skipped test if the healer believes that functionality is broken." Meanwhile, a skipped test does not turn a build red. So someone has to read every skip and every patch.

What does a self-healing setup look like in code, and what replaces it?#

The smallest example wraps the driver in Healenium's SelfHealingDriver, sets score-cap in healenium.properties, and marks every absence check with @DisableHealing so it cannot heal into a pass. Each step below comes from the Healenium README, read in October 2026. Before that, the README asks you to start the Healenium backend.

Steps one and three are Java, for a Selenium test:

// 1. Wrap your Selenium driver so failed lookups can heal
WebDriver delegate = new ChromeDriver();
SelfHealingDriver driver = SelfHealingDriver.create(delegate);

// 3. Never let an absence check heal into a pass
@DisableHealing
public boolean isCancelPresent() {
    try {
        return driver.findElement(By.cssSelector("#cancel-order")).isDisplayed();
    } catch (NoSuchElementException e) {
        return false;
    }
}

Step two is the config file, under src/test/resources/healenium.properties:

# 2. Heal only when the match is at least 50%
recovery-tries = 1
score-cap = 0.5
heal-enabled = true

If you raise score-cap, fewer heals fire, so more tests fail loudly instead of passing on a look-alike. Also, the README lets you switch healing off for a run with -Dheal-enabled=false. That is worth doing on release builds.

But the better fix is a locator that rarely needs healing. The Playwright locators guide, read in October 2026, says "we recommend prioritizing user-facing attributes and explicit contracts." Its own example of such a contract is page.getByRole(). In TypeScript, that is one line:

await page.getByRole('button', { name: 'Place order' }).click();

If the button's text changes, this line fails, and it should. A failure that names the button is cheaper than a pass on the wrong one.

Which pages should a team build from?#

Build from the official pages in this order: the Playwright locators guide, the Healenium README, the Playwright Test Agents page, then Develocity's predictive test selection docs. Each one covers one layer, and each was read in October 2026.

1. Playwright locators: role and test-id locators first, so most lookups never break.
2. Healenium README: SelfHealingDriver, score-cap and @DisableHealing, for suites that cannot move off fragile selectors yet.
3. Playwright Test Agents: the planner, generator and healer, and what a skipped-test output means.
4. Develocity Predictive Test Selection: the selection profiles and how to run the remaining tests later.

What does visual AI change compared with a pixel diff?#

Applitools Eyes replaces pixel-by-pixel comparison with match levels, and its Layout level ignores content changes, which is exactly where a wrong price would hide. The Applitools core concepts page, read in October 2026, sums up Layout in one line. It "Ignores content changes but detects shifts in layout or structure."

First, pixel diffs fail for a dull reason. As the Applitools page puts it, they raise false alarms from "anti-aliasing, font smoothing" and similar rendering noise. So the vendor offers levels instead. Strict is the default, and the others are Layout, Ignore Colors, Content and Dynamic. And Dynamic suppresses diffs "due to dynamic data such as emails, dates, credit card numbers."

For example, a checkout page tested at Layout level will not flag a total that shows the wrong amount. That is because the amount is content. Visual AI cuts the rendering noise that makes pixel diffs unusable. Yet every match level is a decision about what to ignore.

Meanwhile, Percy, BrowserStack's visual testing service, shows the other half of the job. Its visual testing basics page, read in October 2026, compares new screenshots "against previously approved ones (baseline)." The same page also says: "Your build is auto-approved by default on all changes on the main branch." As a result, a bad screen merged to main can become the new baseline. So a person still approves each new baseline, and that person is the real test.

How does predictive test selection decide what to skip?#

In Gradle's Develocity documentation example, accessed October 2026, predictive test selection ran 5 of 22 test classes under the Standard profile. The same example log in Gradle's Develocity 2026.3 docs reports "saving 23m 45s serial time."

Put simply, a selection model ranks tests by predicted relevance to a change and skips the rest. Gradle's 2026.3 docs say the model is built "by observing code changes and test outcomes." They add that it is "calibrated to catch over 99% of non-flaky test task/goal failures." Then the profile you pick moves the line. Conservative selects more tests, Standard is the default, and Fast "uses a higher selection threshold."

In particular, the docs are frank about the trade. Gradle says the feature "trades testing comprehensiveness for faster feedback." It also suggests running the remaining tests in later stages. So a skipped test is a deferred risk, not an irrelevant one.

Still, the research behind it is wider than one vendor. In 2021, Pan and colleagues published a systematic review on arXiv of 29 primary studies of ML-based test selection and prioritisation. One of their questions was whether those studies could be reproduced at all. So treat any published accuracy figure as a starting point to measure, not a promise.

Do flaky-test detection and defect prediction really use machine learning?#

Develocity marks a test FLAKY when it fails and then passes within a single task, which is retry arithmetic, while learned flaky predictors and defect scores need revalidation per project. The Develocity flaky test guide, read in October 2026, says an outcome is "marked as FLAKY if it fails and succeeds" within one task. Playwright's retries page uses the same rule. There, a test that "failed on the first run, but passed when retried" is flaky.

So most shipped flaky detection is reruns, not a model. Yet a model can work without reruns. In 2023, Gruber et al. trained classifiers on 100 flaky test histories from one industrial system, plus the same number of non-flaky ones. Their best model reached an F1 score of 95.5% from three signals. Those were the test's flip rate, recent changes to source files and the files changed in the latest pull request.

But in that 2023 study, Gruber et al. measured the 95.5% on the same system the model learned from. And defect prediction runs into the same limit. A study posted to arXiv on 2 July 2026 tested 16 classifiers within projects and across projects. When they crossed projects, the 2026 authors found that uncertainty signals "do not reliably transfer." They conclude: "Transferred probabilities should be revalidated before guiding quality-assurance decisions."

In short, a defect score from someone else's codebase is a guess about yours. Use it to order a review, and check it against your own bug history before it decides anything. Besides, why defects happen is often a human question, and a score built on code churn cannot see it.

When is an AI testing tool the wrong tool, and what still needs a tester?#

In the TestPilot study, accessed October 2026, generated tests reached 70.2% median statement coverage but only 52.8% branch coverage, so business rules hiding in branches still need a tester. The 2023 TestPilot study by Schaefer and colleagues reports both figures for the same 25 packages. And branches are where discounts, limits and odd cases live.

AI testing tools are the wrong tool in three cases, and each has a better one:

• The risk lives in a business rule. A generator copies what the code does today. Instead, write the rule as a test from the spec, by hand, with the tester and the product owner in the room.
• The check is about absence. A healer turns "gone" into "found something close." Use a strict locator with healing off, as in @DisableHealing above.
• Correct is a judgement. A visual model at Layout level cannot tell a wrong amount from a right one. A person reviews the baseline, the same way a quality analyst reviews what the tools leave behind.

Everything else can use the tools, as long as a tester owns three reviews. Those are every heal patch, every new baseline and every generated assertion. That is the honest answer to the role of AI and machine learning in software testing: it speeds up the routine work, and a tester keeps the judgement.

Where should a team go from here?#

Start with one technique on one suite, measure its misses against a full run, and read the related posts on framework choice and the agentic lifecycle next. With AI ML in software testing, pick the one whose failure you can see first, such as test selection with a nightly full run beside it.

If you are still choosing the framework the AI features plug into, read Selenium, Cypress or Playwright. For where testing sits once agents write the code, read the agentic SDLC. And for why a person still catches what the automation misses, read how manual testing improves test automation quality.

Teams that want a hand can look at hiring QA testers or AI-augmented development. Still, the four documentation pages above are enough to run the pilot on your own.