7 mistakes you want to avoid before setting up automation tests
Published: 2023-01-09 12:01:43

Automation testing is a brilliant tool for ensuring the quality and reliability of software. It allows teams to run large numbers of tests quickly and efficiently, helping to identify bugs and issues early in the development process. However, like any tool, automation testing can be misused or not used to its full potential. In this article, we will explore seven common mistakes that people make when using automation testing, and provide examples of how these mistakes can impact the effectiveness of the testing process. By understanding and avoiding these mistakes, teams can make the most of their automation testing efforts and improve the quality of their software.


!["Automation gone wrong" by Midjourney, 2023](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f4f386eszrqea648d085.png)

## 1. Not having clear goals
Automation testing is most effective when it is used to accomplish specific, well-defined goals. Without clear goals, it can be difficult to determine the appropriate test cases to automate and to measure the success of the automation efforts.

For example, let's say the team is working on a web-based application and they decide to automate their testing without first setting clear goals. They may begin by automating a large number of test cases that cover various aspects of the application, such as the user interface, the database, and the back-end logic. However, because they have not defined specific goals for the automation, they may not know which of these test cases are the most important or which ones are most likely to uncover bugs. As a result, they may end up automating _a large number of test cases that do not effectively address any specific problem_ or need.

This lack of clear goals can also make it difficult for the team to measure the success of their automation efforts. Without defined goals, it is difficult to determine _whether the automation is providing value_ or whether it could be more effective if it were focused on different test cases or areas of the application.

In order to avoid this mistake, it is important for teams to clearly define their goals for automation testing before they begin. This may involve identifying specific areas of the application that are most important to test, setting targets for how many test cases should be automated, or defining other metrics that can be used to measure the success of the automation effort.

## 2. Relying too heavily on automation
While automation can be a powerful tool, it is not a replacement for manual testing. Automation is best used to supplement manual testing, not replace it entirely.

Your team may believe that automation will be faster and more efficient than manual testing, and they may decide to discontinue manual testing altogether in favor of automation.

However, this approach can be problematic for several reasons. First, automation is not always able to detect every type of bug or issue. Some issues, such as usability problems or issues with the user interface, may be more easily detected through manual testing. If the team is not performing any manual testing, these issues may go undetected.

Second, automation testing can be expensive to set up and maintain. If the team is automating all of their testing, they may need to invest a significant amount of time and resources into creating and maintaining their automation test suite. This can be especially challenging if the software is changing frequently, as the automation test suite will also need to be regularly updated.

Finally, it is important to remember that _automation testing is not a replacement for human judgment_. Even with a comprehensive test suite, it is still possible for bugs and issues to slip through the cracks. By relying solely on automation, the team may miss the opportunity to use their own expertise and judgment to identify and fix problems.

In order to avoid this mistake, it is important for teams to use a combination of both manual and automation testing. Automation can be an effective tool for running large numbers of test cases quickly and efficiently, but it should be used in conjunction with manual testing to provide a more comprehensive and effective quality assurance process.

## 3. Not maintaining the test automation suite
Automation testing requires maintenance, just like any other software. If the test suite is not kept up-to-date, it can become unreliable.

Naive may believe that, once the test suite is in place, it will run smoothly indefinitely without any additional effort.
However, as the software changes and evolves, the automation test suite may need to be updated to reflect these changes.

For example, let's say the team is working on a web-based application and they have automated a number of test cases to validate the functionality of the application. Over time, the team makes a number of changes to the application, such as adding new features or modifying the user interface. If the team does not update their automation test suite to reflect these changes, the test cases may no longer be relevant or accurate. As a result, the test suite may produce _false negatives_ (tests that should have failed but did not) or _false positives_ (tests that should have passed but did not).

Another thing that can really poison your automation is _test flakiness_, or simply unstable tests. This problem is often encountered at the integration or end-to-end layer of tests. Almost every team I've worked for has encountered this problem and tried to ignore it. You may think that one or two tests can just be re-run or ignored, but if you do that, you will see that over time, that number grows to a point where you need to invest considerable resources to fix all the instabilities or even rewrite those tests from scratch. What's even worse is that the automation loses the trust of the team, because it's hard to differentiate whether this red test is just another instability or a real bug that crept its way into your project.

In order to avoid this mistake, it is important for teams to regularly maintain their automation test suite as the software changes. This may involve updating existing test cases or adding new test cases to cover new or modified functionality.
A good practice is not only updating the tests with the production code as a part of a task, but to also have regular hackathons for that or even scheduled backlog items to keep your tests green and clean.

## 4. Not testing the right things
It is important to carefully consider what to automate and what to test manually. Automating tests for features that change frequently may not be the most efficient use of resources, as the tests will need to be frequently updated.

Imagine an application that includes a number of features that change frequently, such as the layout of a webpage or the content of a news feed. If the team decides to automate all of their testing, they may need to constantly update their automation test cases to reflect these changes. This can be time-consuming and frankly a waste of resources.

In order to avoid this mistake, it is important for teams to carefully consider which test cases are most suitable for automation before implementing them. _Test cases that are stable and do not change frequently_ may be good candidates for automation, while test cases that are subject to frequent change may be better suited for manual testing. By carefully considering what to automate and what to test manually, teams can make the most efficient use of their resources and ensure that they are testing the right things.

## 5. Not having good test coverage
Even though this sound pretty damn obvious, it is nevertheless had to be said. _Nothing's more important to have a comprehensive set of test cases_ to ensure that the software is thoroughly tested. If the test coverage is not sufficient, there may be gaps in the testing that could result in undiscovered bugs.

My developer friend not so long ago shared his frustration about experience on a project where the team did not have sufficient test coverage for their software. They were just writing autotests for their features as they developed them, they haven had dedicated processes and performed testing by themselves without QA professionals. This was a mobile application that was designed to track a user's fitness activities and provide personalized recommendations based on their data.

The team had automated a plethora of test cases to validate the recommendations of the application, some really next level machine learning stuff, but they had not thoroughly tested all aspects of the application. As a result, when the application was released, it contained a number of bugs that were not detected during testing.

One of the most significant issues was a bug that caused the application to crash whenever the user tried to manually add a new activity to their profile, which as it turned out was a major user scenario. This bug was not discovered during testing because the team had not thoroughly tested the feature that allowed users to add new activities. They had a lot of isolated tests for different sub-features of it, they even had a test for importing an activity, but simply haven't checked a basic user path before getting into the details.

The consequences of this mistake were significant. Users who downloaded the application were frustrated by the frequent crashes and left negative reviews, leading to a drop in the application's rating in the app store. The team also received negative feedback from their client, who was disappointed by the low quality of the software.

In order to avoid this mistake in the future, the team implemented a more comprehensive testing process that included both manual and automation testing. They also invested in a dedicated QA Engineer and in tools to help them track their test coverage and identify areas of the application that were not being thoroughly tested. As a result, the team was able to deliver a higher quality application on their next project and now has a stellar reputation with both users and their clients.

## 6. Not involving the right people
Previous point is naturally connected with this. Automation testing should be a team effort, involving both developers and QA/QC professionals.

I have had experience with this mistake on a project where the automation testing was solely the responsibility of the development team. The testers on the team were not involved in the automation process at all, and as a result, they did not have a clear understanding of what was being tested or how the automation was being implemented.

This led to a number of problems. First, the testers did not have a clear understanding of what to test manually and what had already been automated. As a result, they ended up _duplicating some of the work that had already been done_ by the automation, leading to wasted effort and confusion.

Second, because the testers were not involved in the automation process, they were not able to provide valuable input or feedback on the test cases being automated. This meant that the automation may not have been as effective as it could have been, as the testers were not able to contribute their expertise or perspective to the process.

Finally, because the automation testing was solely the responsibility of the development team, there was a lack of communication and collaboration between the testers and the developers. This led to a breakdown in the testing process and _a lack of trust between the two groups_.

To avoid this mistake in the future, it is important to involve both developers and testers in the automation testing process. This can be achieved through regular communication and collaboration between the two groups, as well as through the use of tools and processes that support teamwork and collaboration. By involving the right people in the automation testing process, teams can ensure that the automation is effective and efficient, and can build trust and collaboration within the team.

## 7. Not investing in the right tools
Choosing the right automation tools is important for the success of the automation testing effort. It is worth investing in high-quality tools that are well-suited to the needs of the project and the team.

I have had practical experience with this mistake on a project where the team did not invest in finding the right automation tools. We were working on a mobile application and the team decided to use an internal framework just because it was used by other teams in our company for other apps for that operating system.

As a result, we struggled to effectively automate our test cases. The framework was convoluted, unreliable and frequently produced false negatives, leading to a lack of confidence in the results of the automation testing. Additionally, the framework was not well-documented and was difficult to use, making it time-consuming and frustrating for the team to work with.

In order to address this problem, we decided to invest in a different automation tool that was better suited to our needs. We conducted research and compared a number of different tools, ultimately choosing one that was more reliable, well-documented, and user-friendly. This investment paid off, as the new tool allowed us to automate our test cases more efficiently and effectively, and it also improved the trust and confidence of the team in the automation process.

In hindsight, we learned that it is important to _invest_ in high-quality automation tools that are well-suited to the needs of the project and the team. By taking the time to carefully evaluate and choose the right tools, we were able to improve the efficiency and effectiveness of our testing process and deliver a higher quality product.

---

**Automation testing is a thing without which I can't imagine today's software development**. However, it is important to approach automation with care and consideration, in order to avoid common mistakes that can impact the effectiveness of the testing process. By understanding and avoiding these mistakes, teams can make the most of their automation testing efforts and improve the quality of their software. By setting clear goals, relying on a combination of manual and automation testing, maintaining the test automation suite, testing the right things, having good test coverage, involving the right people, and investing in the right tools, teams can make the most of their automation testing efforts and deliver high quality software.