Software Testing Series: Organizing a Test Suite with Tags Part One

Organizing a test suite with tags

Tagging is a method of organizing information that is pushing into the mainstream now through the success of sites like Flickr and Del.icio.us, and blogging software like WordPress. We can apply this idea to managing our automated test suites. An automated test suite is a critical component of any continuous integration process.
First steps first…

This post is a follow-up to our previous case study on incorporating unit testing into an existing team’s development environment. The case study is based on a real solution that has already started reaping rewards for our client, and is gaining momentum. We’re now looking at making it easier for the development team to maintain this test suite, and proposing some extensions – including a form of tagging.

• In part one of this post we develop an understanding of tagging as a technology, including it’s pros and cons.
• In part two of this post we will define the top five opportunities and problems inherent in the organization of automated test suites.
• In part three of this post we will explore an approach to combining tagging with test suite organization.

Understanding tagging

Tagging allows users to define their own categories for describing the items that they care about. The technical term for a free-form approach to labeling items is folksonomy as opposed to taxonomy, which is a classical categorization approach.

The Wikipedia presents the following definitions for folksonomy and taxonomy:

Folksonomy, a portmanteau word combining “folk” and “taxonomy,” refers to the collaborative but unsophisticated way in which information is being categorized on the web. Instead of using a centralized form of classification, users are encouraged to assign freely chosen keywords (called tags) to pieces of information or data, a process known as tagging. Examples of web services that use tagging include those designed to allow users to publish and share photographs, personal libraries, bookmarks, social software generally, and most blog software, which permits authors to assign tags to each entry.

Taxonomy (from Greek verb tassein = “to classify” and nomos = law, science, cf “economy”) may refer to:

• the science of classifying living things (see alpha taxonomy)
• a classification

Initially taxonomy was only the science of classifying living organisms, but later the word was applied in a wider sense, and may also refer to either a classification of things, or the principles underlying the classification. Almost anything, animate objects, inanimate objects, places, and events, may be classified according to some taxonomic scheme.

There is debate about the value of tags

Several people have voiced concerns that tagging is simply a bad idea, with some compelling arguments. The Net Takeaway has a post and links to previous posts.

Look, if I am looking for something specific, then I type those terms in. Say I use a search engine. If I am looking for a phrase, I use quotes and type in all the words (up to 10 for most engines) and I get hits with that phrase.

But usually, I want stuff “like” or “similar” to my words. […]

But that’s now how tagging systems work. Instead, you have to know the terms up front to find anything.[…] Note that every popular “tagging” system, to date, has been for consumer fun stuff (flickr, etc.) and not for real knowledge management.

There are some good rebuttals in the comment thread as well, providing insight into what is intended by tagging, suggestions on how to use it, and alternative comparisons with search. If you only read one, read comment #3. Then go back and read the rest of them.

Benjamin Booth writes, in The Present Failure of Tagging, about the challenges. Benjamin’s approach is “I like tagging, how do we make it work the way we want?”

The general problem can be seen as the task of 1) externalizing knowledge-retrieval ‘landmarks’ when encountering information you want to store (in some context) and then, 2) being able to quickly find these landmarks when trying to recall the information later on, potentially in a completely different context from the one in which you created the landmark in the first place.

Near the end of his post he begins:

We need refactoring for tagging.[…]

Benjamin makes reference to the concept mapping tool from IHMC that we talked about previously.

Rashmi Sinha has an outsanding article where she applies the science of cognitive psychology to the issues with tagging usability.

Cognitively, we are equipped to handle making category decisions. So, why do we find this so difficult, especially in the digital realm – to put our email into folders, categorize our bookmarks, sort our documents. Here are some factors that lead to what I call the “post-activation analysis paralysis”.

Our conclusions about tagging

After reading the linked posts and their discussion threads, we are pretty well versed about the pros and cons of tagging.

Pros

• Tagging eliminates the “How do I organize this?” analysis paralysis that happens when trying to start organizing
• Tagging allows for a dynamic classification system that grows over time. If we make a bad decision early, we can grow out of it.
• The “free association” approach to tagging that exists in the digital world today is consistent with the way our brains function when storing information.
• There are a bunch of smart people working on tagging right now, so there is plenty of opportunity to leverage their good ideas.

Cons

• Tagging makes retrieval of information difficult. If we don’t know how we previously tagged something, it can be hard to find it later.
• The existing (digital) approaches to tagging don’t provide an analog for the way our brains function when continuously updating and refining our “free associations” as we learn.
• People rooted in traditional taxonomy-based classification systems struggle with the concept of tagging. This probably characterizes 90% of people, so gaining mindshare outside of the technorati will be difficult, and user adoption could be a challenge.

Reminder:

• In part one of this post we develop an understanding of tagging as a technology, including it’s pros and cons.
• In part two of this post we will define the top five opportunities and problems inherent in the organization of automated test suites.
• In part three of this post we will explore an approach to combining tagging with test suite organization.

– – –

Check out the index of software testing series posts for more articles.