Challenges in implementing test automation

We conducted research about implementing test automation within organizations. Read this blog to learn about the key findings.

Software has changed dramatically from small programs to complex, comprehensive IT systems over the last decade. Our lives are increasingly determined by its presence everywhere — from our toothbrush to our cars. The quality of software therefore stands to be of crucial importance. Organizations that rely on manual testing methods have to deal with increasing pressure on faster time-to-market cycles. This is where test automation plays an important role.

Based on our research, we see that organizations start enthusiastically with the implementation of test automation, but as time progresses, the enthusiasm decreases and the organization ceases.

Research approach

We examined 17 different projects regarding the implementation of test automation within test and QA organizations of different companies in the Netherlands. As a research method, we used semi-structured interviews. The Test Automation Consultants (TACs) who implement or work with test automation were interviewed. This research focused on the question: "What are the positive experiences and challenges with test automation within the different organizations in the long run?"

Key results of the research

There was only one successful application of test automation, which is not a lot and means that test automation needs work — hard work.

Test automation is started using a project-based approach in various organizations, where the organization and the test professionals are enthusiastic about automating current tests on different levels, from a unit to a Graphical User Interface (GUI). After the implementation, some challenges arise, such as maintenance on test scripts or code — because not all test professionals have the skills to maintain them.

Most test professionals hired by an organization are outsourced. Thus, test automation remains dependent on individuals or external parties. Transferring the knowledge to the rest of the organization therefore becomes a challenge. It was also noticeable that not all organizations have a specific test strategy or architecture when implementing test automation.

Practical recommendations

• Start with the WHAT. What do you want to test, and by whom?
• Identify the regressions and routine tasks that can be automated using test automation tools.
• Maintenance of your test automation is part of your test architecture.
• Make sure you are doing enough unit testing to avoid finding defects on higher levels.
• Is there enough knowledge regarding test automation within the organization?
• Use it often in your tests to overcome the maintenance bottlenecks.

The learning points from the past are still valid for many organizations. We know it is not easy to do test automation; however, it is important to take a closer look at your organization and decide what makes sense.

← Back to all blogs