Click Run, Skip Thinking

In my previous articles I covered some knowledge and tricks that I’m using. This time I’ll be sharing an experience I learned the hard way. It sounds a bit dramatic, but it is not - it’s just something to keep in mind so you don’t fall into a trap you set yourself.

Whenever I began working with some project that is unknown to me I always started with exploratory testing. Exploratory testing in combination with some product manuals or product specification is the best way to learn dark corners and hidden behaviors of the product. No matter how much the product has been tested previously (by someone else), this way of familiarizing yourself with the product hugely increases chances to reveal some inconsistencies and bugs.

By the time you’ve done an exploratory testing, wrote most of the test specification and started developing test automation, you feel like the king of the product - you know every piece of it, every feature, every component, every highlight and every flaw. This is a special kind of feeling, I must admit, because developers respect your opinion, support relies on your feedback, and your superiors are much more confident about product reliability.

This is the point where, by the SDLC standards, product maintenance ‘kicks in’ and in most scenarios the QA’s responsibility onward is supervising the automation tests and maintaining them, especially if they are running as a part of CI/CD. Basically you just take a look at the outcome of the tests and if everything is green you just continue with your other tasks.

I really like to use the following saying, so here it is - And here comes the highlight of this article: you start forgetting the product. Without the need to constantly or periodically browse and explore through the product, you just start forgetting it. Even knowledge of basic features and actions of the product, which you were very familiar with, start to fade. This is also the point where your product starts losing reliability, because you no longer recognize every change made to the product. I am aware that some of you might argue about that part where product loses reliability after introducing the testing automation, because test automation is supposed to increase the reliability of the product, right? It is, but not by completely eliminating manual testing, exploratory testing, and overall human interaction with the product during the regression testing. Testing automation is just a leverage to help you with maintaining the product, including all other types of testing. Getting a car with all the modern safety features won’t make you any safer if you decide to start driving with your eyes closed.

Some signs that you are sliping into “Click run, skip thinking” sindrom are:

• You trust green pipelines without questioning what’s actually covered
• You struggle to explain a feature without opening the app
• Bugs reported by users feel “surprising” instead of “missed”
• You rarely open the product outside debugging automation failures

Now that we’ve identified the problem, the solution is straightforward - but it requires discipline. Do the manual regression testing once in a while, no matter how detailed coverage of the test automation is. Manually cover both tests that are a part of test automation and the ones that are not, especially some edge cases. Edge cases not only cover the ’edge’ scenarios, but might also lead to discovering an issues that also affect other parts of the system. A practical way to approach this is by introducing small rituals:

• Weekly 30 minutes exploratory sessions
• Manual smoke run on a release day
• Monthly “break the product” session focusing only on edge cases

Test specification is also a nice way of keeping the knowledge of product up to date. By reviewing and updating the test specification you are keeping the knowledge of product in good shape. Just keep in mind not to duplicate the test cases, as quantity doesn’t necessaryly guarantee quality.

Reviewing test specification for duplicate test cases is fantastic example of use case for AI - just feed the AI with test specifications and ask it to check for duplicates and it might also check for gaps in coverage or highlight tests that validate the same behavior in slightly different ways.

I learned this the hard way on one project where everything looked stable - pipelines were green for weeks. A small UI change broke a core user flow, but automation didn’t catch it. I didn’t either, because I hadn’t actually used that part of the product in a long time.

Automation should support your thinking, not replace it. Because the moment you only click “Run”, you’ve already skipped the most important part.