Why test automation stalls — and how to make it work

Test automation often begins with energy. A tool is selected, a specialist is hired and the first regression suite is built. The demonstration is compelling: tests that once took hours now run in minutes. A few months later, confidence declines. Scripts are unstable, the pipeline is regularly red and no one dares to change the suite. The technology works, but the organisation has not built a sustainable testing product.

Continuous Connect studied the adoption of test automation across 17 projects at organisations in the Netherlands. Through semi-structured interviews, test automation consultants shared their positive experiences and long-term challenges. The research is a qualitative snapshot, not statistical evidence for every organisation. Its patterns, however, remain recognisable and relevant.

What the research revealed

Across the projects, test automation was often treated as a temporary implementation project. An external team delivered scripts, while lasting ownership, a maintenance budget and a suitable architecture lagged behind. Only one application in the study was considered sustainably successful by respondents.

That result does not mean automation rarely works. It shows that a functioning suite requires more than scripts. Knowledge transfer, integration with development, test data, environments and daily follow-up together determine whether automation continues to deliver value.

Six reasons automation stalls

1. There is no explicit objective

'We need more automation' is not a strategy. Are you aiming to give developers faster feedback, protect a critical customer journey, shorten manual regression or release more frequently? The objective determines the tests, speed and reliability required.

2. All attention goes to the wrong layer

GUI tests are visible, but often slow and sensitive to change. Many checks are cheaper and more stable at unit, component or API level. Automate as low as possible and as high as necessary. Keep a limited set of end-to-end tests for the most important business flows.

3. Automation remains owned by a specialist

When only one consultant understands the code, a new risk is created. Developers, testers and operations must agree together on reviews, standards, failures and priorities. External expertise can accelerate progress, but internal ownership must be built from the start.

4. Test data and environments are unreliable

A test may be technically correct and still fail because data is missing, integrations are unavailable or a shared environment changes during the run. Treat data, simulations, configuration and clean-up as part of the automation architecture.

5. Maintenance is not planned

Test code is production code: it requires design, version control, reviews, refactoring and technical stewardship. Reserve capacity for maintenance and remove tests that no longer provide useful information. A larger suite is not automatically a better suite.

6. Red tests become normal

Flaky tests undermine trust. If teams rerun failures without analysis, the suite loses its function as an early-warning system. Make instability visible, quarantine unreliable tests and prioritise fixing the root cause.

Design a strategy before selecting a tool

A test automation strategy need not be long, but it should make clear choices about:

• which product risks and user journeys take priority;
• which test levels provide the most efficient controls;
• when tests run and how quickly feedback is needed;
• who owns the framework, scripts, data and environments;
• which quality criteria apply to test code;
• how results are reviewed and failures followed up;
• how knowledge is shared and continuity protected.

Select tooling only after those decisions. A tool must fit the technology, team skills, pipeline and operating context. A popular solution that no one can maintain is a poor investment.

Measure information, not script counts

The number of automated tests or a general automation percentage says little about value. Use a small set of indicators that encourages improvement:

• time from code change to reliable feedback;
• percentage of runs failing because of instability rather than a product defect;
• time required to diagnose and repair a failed test;
• coverage of critical risks and business flows;
• defects first discovered in a later test phase or production;
• maintenance effort compared with manual regression time saved.

Always combine numbers with analysis. Fast feedback has value only when teams act on it.

A practical route in four steps

1. Select one meaningful journey

Start with a stable, frequently used and business-critical flow. Map its risks, current test effort and expected benefit.

2. Build a thin slice

Automate a few checks at different levels, include test data and pipeline integration, and demonstrate how a failure is investigated. This tests the whole operating model, not only the tool.

3. Make the team accountable

Use pair or mob sessions, introduce code reviews and record decisions close to the code. Internal team members should be able to change the suite independently before external support is reduced.

4. Evaluate and scale deliberately

Measure reliability, feedback time and maintenance. Scale only when the first set is demonstrably used and managed. Stop or rewrite checks that consistently create more noise than insight.

Automation does not replace exploratory testing

Automated checks excel at quickly and repeatedly recognising known deviations. People are better at investigating unexpected behaviour, usability and new risks. A mature approach combines both. Automation creates time for deeper testing; it does not replace quality thinking.

The central lesson from the research is organisational: treat test automation as a lasting product capability. With a clear objective, shared ownership and maintenance built in, it can provide faster feedback and demonstrably support releases.

Further reading

• ISTQB — Test Automation Engineering

← Back to all blogs