Are there any decent testing metrics?

Last week I was asked by my company to define some decent metrics to start tracking for my team.  I have been thinking on this for a while, and the report I ended up with is one that seemed very appropriate for a blog post.

I read a tweet the other day from Janet Gregory:

“I like visible metrics that are aimed at measuring a problem that we are trying to solve. Once the problem is solved, stop.”  (original tweet)

That resonates with me. So I question what problems is it that we are actually trying to solve through collecting metrics? The way these metrics were framed by $person, we are looking for numbers to provide corporate to keep them from forcing us to gather metrics that may not be applicable in our scenario. $person also mentioned justification of new hires and additional resources.

I appreciate the heads up and the suggestion to get some metrics in place before we are forced to do so.I don’t believe  that 2 or 3 numbers will provide any real insight into our team, especially at the corporate level. I also lack an understanding of what would satisfy their need for data. This doesn’t seem like an actual problem that I know how to solve.

As far as justification for additional resources, I understand ‘resources’ to include hiring new people into the team, as well funding for training and improvement of the current team. Through the page coverage metrics below, we will be able to show the page coverage possible with our current team. These metrics will not show any inefficiencies in time usage, but it seems with the coming of $newtool we will get a complete suite of metrics there than can track that. $testerperson will also be tracking time spent on helpdesk tasks to help facilitate hiring a full time helpdesk tech.

The problem I chose to address is one that is authentic to myself and my team.  The coverage metrics listed below are ones that I think will assist in managing the testing effort of my team.  As each release date draws nearer, these coverage metrics will provide insight into areas of the application that may need further testing. As mentioned below, coverage metrics are fallible, and this is only one tool to aid in testing, and showing the story of our testing to others in the company.

Metrics for QA

Hot fixes required to fix critical bugs in production

This is the most important metric for tracking the efficacy of QA as far as I am concerned. Hot fixes are pushed to resolve critical issues clients face. This metric should be as close to 0 as possible for any given period of time. While we have not been tracking this explicitly, we have noticed a decrease in this recently. I think a brief description of each of these incidents should also be recorded.

Time Spent on Support Tasks

One of the problems we are facing right now is the lack of a full time support employee. $testerperson and Wade (the two main QA members that work with customers) will track their time spent on support related tasks to assist in hiring a full time employee.

Page Coverage in Automation

This week we will be adding a tool to $app that will allow us to track all page views by user. We will then be able to compare this against current count of pages in the app and come up with a decent metric for % coverage. This will only be a coverage metric for how many pages are viewed, it is not branch coverage, code coverage, logic coverage, or any other sort of coverage that someone may think of, but this will give us some insight into what portion of the app is being tested by automation.

Page Coverage in Manual Testing

With the same tracking tool that will allow us to track automation’s progress, we will also be able to track manual coverage. This will be useful for tracking coverage at each release cycle to take into consideration for .

These are the metrics I plan to track for my team in the coming months.  Like I mentioned above, the page coverage metric is fallible, but I think in my context we will be able to use the metric for some useful information.  I don’t necessarily want this metric to reach 100% coverage, as there are pages that currently exist in our app that are not actively used by clients, or even available for use by them.

Once I get a feel for how these metrics starts to illuminate our process, the metrics or our processes might change.  Until then, this is what we will use.

Respond to this post