Software Testing Series: Organizing a Test Suite with Tags Part Three

Organizing a test suite with tags (part 3)

This is the third in a three-part post about using tags as a means to organize an automated unit test suite.

Part 3 of this post can be read as a standalone article. If it were, it would be titled Design elements of an automated unit test framework using tags. If you’re only reading this post and not parts 1 and 2, pretend that this is the title.

• In part one of this post we developed an understanding of tagging as a technology, including its pros and cons.
• In part two of this post we defined the top five opportunities and problems inherent in the organization of automated test suites.
• In part three of this post we will consider the key design associated with using tags as a mechanism for organization.

Setting expectations

We designed a custom unit test automation system based on the use of tags to organize automated tests for a client. That system isn’t the subject of this post, but it did provide us with context and insight into the problem we’ve addressed. In this post we won’t be presenting a completed design – there are many good tools out there already for automating unit tests. We will be talking about a subset of the design decisions that are associated with the use of tags as a mechanism to organize the unit testing within the suite. In Marc Clifton’s advanced unit testing articles, he walks readers through the creation of a test automation tool for C#. The concepts presented here can be incorporated into a design relatively easily, but the details of doing that are a little too off-topic for most of our readers.

We are writing about designing software that tests other software. To keep our language consistent and easy to follow in this post, we will use two terms. Tool represents the test automation software that we are designing. Application represents software being tested with the tool, or more specifically, with tests maintained within the tool.

Design approach

After identifying the use cases and functional requirements for the tool, we began iterating on screen designs, business (requirement) object models and architectural (implementation) object models. We created an object oriented analysis (OOA) diagram to represent the concept concisely.
OOA diagram

An object oriented analysis diagram of the key relationships between scripts, inspections and tags.
A script is the embodiment of a user session in the application – it represents a set of actions that a user of the application would take. The user of the tool will create a script (as a separate file), and will create a reference to that script in the tool.

An inspection is an unit test of the application. The inspection evaluates a particular condition or makes a specific assertion about the state of the application (a properly filled out order is submitted when the user clicks “submit”) . The code that executes that inspection is represented outside of the tool. The user of the tool will create a reference to the inspection within the tool.

Inspections can be associated explicitly with any number of scripts (including none). An association between inspection 1 and script A is an instruction to the tool to run script A within the application, and evaluate inspection 1 against the script.

The processing of scripts and inspections is outside the scope of this document, but is covered in many other references, including Marc Clifton’s.

Any number of tags can be associated with each script. Tags could be used to represent different user actions (like deleting items from a shopping cart), different specific selections (user adds 1000 of an item to the shopping cart), different situations (shipping address does not match billing address), or any other relevant descriptor of the user session. A single script could have multiple tags.

Each inspection can be transitively associated with a set of scripts by explicitly associating it with one or more tags. By associating an inspection with a tag, we are instructing the tool to dynamically associate the inspection with all scripts that are associated with all of the identified tags. There are two benefits to this approach. First, this approach reduces the amount of time that a user of the tool must spend to associate an inspection with a set of relevant existing scripts. Second, this indirect mapping is utilized to assure that an inspection that is mapped to a tag will also automatically become associated with any future scripts that are added – as long as they have associations with the same tag or tags as the script. This reduces the cost of creating future mappings when scripts are added to the suite in the future.

We expect this design approach to provide significant labor savings in maintaining a test suite. We built our business case for this project upon that assumption. We also expect that this design approach will result in better testing coverage of the application by users of the tool. We did not incorporate that expectation into our cost benefit analysis when calculating the ROI of this project.

Followup

We will follow-up some months from now when we can evaluate data and draw conclusions from use of a tool built along similar lines for a client. Until then, we have confidence that it will work very well, but no tangible data.

Summary

• In part one of this post we developed an understanding of tagging as a technology, including its pros and cons.
• In part two of this post we defined the top five opportunities and problems inherent in the organization of automated test suites.
• In part three of this post we considered the key design elements associated with using tags as a mechanism for organization.

– – -Check out the index of software testing series posts for more articles.