Tests in Order, Please

Should test case IDs reflect execution order at all? If yes, here is the pain and my solution. If no, here is the alternative philosophy.

At some point I probably mentioned that most of the things I write on this blog came from my personal experience. It might be some interesting discovery I made, an advice I want to share, or question I want to clarify. This time I want to share one of (workaround) solutions I made, but I’m also raising a question in order to cope with this challenge more efficient in the future. This might also help other QA colleagues who will enroll on new projects and want to be as prepared as possible.

Attributes used to define the test cases in test specifications are very similar no matter what kind of project or testing is being performed. Some basic attributes would be:

• Test case ID
• Test case title
• Steps to reproduce
• Expected result
• Actual result

Attributes can be added or removed, depending on what we need. Even some more custom attributes, which might contribute to our tests, might be added.

Regarding some additional attributes, read my first post about object/action attribute.

Most of these are self-explanatory, so I won’t be describing them and their purpose, but I will take this opportunity to discuss the use, benefit and potential challenges of test case ID, and I will be raising a question, as already mentioned. Test case ID is used to distinguish each test case and it goes hand in hand with test case title. It has a unique value, and while test case title is used to describe the meaning of test case so humans can understand it, test case ID is a string which is used strongly to define the identity of test case. Just as we have our first and last name we also have some personal identifier number issued to us by the government. Patterns for defining a test case ID can vary depending on how the author of test specification wants to structure it, but some (pretty much) universal way I like to do it is something like this: HG-03-008. Let’s define it a bit better:

• HG - Name of the project under test, in this case name of the project is Hugo.
• 03 - Label of scenario, ranging from 01 up to 99, with assumption we won’t need more than 99 test scenarios.
• 008 - Label of test case, ranging from 001 up to 999, with assumption we won’t have more than 999 test cases per scenario.

First test case in the whole test specification would have an ID HG-01-001, second one would be HG-01-002, and so on. You may have noticed I am trying to keep the ID follow some order of test cases, or to be more specific - order of executing the test cases. I intentionally use IDs as execution-order markers, as it it easier to navigate by them. Creating test cases from scratch is very straightforward and I do not expect for problems to arise at that moment. But here is the challenge I wanted to address, and believe me, it gave me more headache and ate more precious time than I could expect.

Imagine the situation where you already have the whole test specification defined, including all the UI E2E tests which are executing in Cypress.

A minor change in one of features is made and you decided to cover that with an additional test case.

Test scenario already has 60 test cases, and new test case would be 5th in order.

Since I am writing this article as my own experience, I will take in consideration that test specification is written in some tool/platform which easily handles changes of IDs, and in my current situation this is Excel. Test case management tools such as Testmo or Testrail support usage of generic sequence IDs or even some custom fields which can be used as your own custom IDs, but I will explore how these tools handle ID changes in future post, in case you add some new test case in the middle of scenario. I will give you my insight when I confirm the behavior of this feature.

But incrementing all these test cases in code of your automation tests might become a bit of pain in the ***. Adding or removing one test requires some time to manually adjust all test case IDs, depending on the amount of tests cases you need to adjust. But adding few test cases in one spot, changing few in other, removing couple in third place - now you would need to concentrate hard and lose some time just to do one thing - update the IDs. When you also consider the possibility to make a mistake while updating the IDs, the frustration just gets heavier.

So, as announced earlier, here comes my question(s) and my (workaround) solution;

How do you cope with this issue?

How do you update the IDs in your automation code?

Are you using some GUID instead of a sequence IDs?

Are you laughing at this because it is a minor issue to you?

If you are laughing at this because it is a minor issue to you, what is the allmighty solution you are using?

One could argue that test case IDs should be immutable and should not reflect execution order at all. In that case, adding a new test would not require renumbering anything. However, in my approach, I intentionally use IDs to reflect structure and execution order, which introduces this challenge.

After some significant time of manually updating the test case IDs in my automation code, I wrote a script to check all of the test cases in my automation code, compare it with test cases in my test specification, and update the IDs according to the test specification. It basically does the following algorithm:

Fetch test case Title in automation code Search for the test case with the matching test case Title in test specification and copy the test case ID Update the copied test case ID in automation code

As a reference, I need to attach my test specification to the script. In current case I am using Excel as my test case management tool. Also, here is another question I am curious about:

How would you use this script if test cases are located in some test case management tool such as Xray, Testrail or Testmo.

Up until my next post, keep testing and be tenacious! Also, feel free to ask me anything, or comment if you have some ideas to share. I’ll have to install some comment component here :)