Significance of Test Metrics Management

If you are a Test manager or any part of Testing group in an organization, then you surely would have encountered the below mentioned queries:

How much time will it take to finish the test cycle?
How stable is the functionality you are testing?
How much of testing is remaining to be done in test areas assigned to you?
What’s the status of reviews you are doing?
How much percentage of Build is testable?

The list of these queries can go on. The important thing here is that these queries forms a part of daily routine of tester’s job.

More often, answering the Metrics related queries results in an uncomfortable experience for the tester. The reasons for this are:

1. Tester is not prepared to present the asked data.
2. There is an inadequate data available with the tester.
3. Tester was not at all aware that he/she had to be prepared with the asked metrics.

In absence of data, which may be due to one or all of above mentioned reasons, a tester is often prompted to leave the task at hand and go from scratch to collect the data and put the data in presentable form. This activity, which is primarily due to lack of planning and ineffective management, may take long time to finish and causes a precious loss of testing time.

Considering the fact that testing time is often squeezed during the life cycle of a product in order to meet important Product adlines, there is an utmost need to save time and focus on important tasks at hands. Managing the metrics effectively will help achieve this objective in a definite way.

As Testers are the people who are most exposed to Software before its actual Release or Beta Release, there is always a possibility tester will be asked to present various kinds of data during different stages of Life cycle.

Thus, metrics forms an important part of a tester’s job. So, Metrics needs to be managed in an efficient manner and it will in turn help to enhance the productivity of a tester. Both Managers and Testers have a role to play in managing metrics.

Manager’s Role in Test Metrics Management

Indeed, Managers have a vital role to play in managing Test metrics effectively. What Managers can do is - PLAN BETTER

There is always a scope of Improvement in this area. Yes, this is definitely an area of Improvement as often Metrics are missed or not handled with priority at the Planning stage.

Most of the QA Plan or Master Test Plan templates don’t say much about the metrics that would be used during the whole life cycle of Product testing.

This is where Manager’s can act and plan the usage of metrics before hand i.e. details of data that would be expected from tester’s during various stages of Product life cycle.

Of course, all the requirements related to metrics are difficult to think of so early but even some meaningful Inputs at planning stage will definitely help testers to be prepared. This will help to make tester’s work more focused.

Tester’s Role in Test Metrics Management

To start with, the following listed attributes/ skills can help testers in this area-

1. Sense of anticipation
2. Discipline
3. Usage of Tools

Sense of anticipation is a quality that would definitely help tester manage him/her better. Its just thinking in advance, what kind of metrics would be expected in a particular time or a phase of the product. To add to this, sense of anticipation can be gained from one’s own experience or somebody else’s experiences.

To cite an example- during testing phase, one can always expect to be asked for time required for execution of test cases or about status of component under test.
Judging this, a tester can always be prepared with required stuff much before it is asked for. Anticipation can help testers in great deal to escape from phrases such as “You were supposed to be present with this data”.

Often metrics collection and management is a tedious activity and it sometimes prompts a tester to do repetitive tasks. Discipline is an attribute that can help tester in this aspect. Tester’s motive should be to find a better way of doing things and this should be taken as a challenge.

Usage of appropriate tools can help a great deal in managing things better. Data storage, Data retrieval and Data presentation are the important aspects that need to be considered while selecting a tool. It should be noted that data presentation is an important aspect as effective presentation helps anyone who is reviewing the data save a lot of time and gather and analyze more in a very less time. Already existing small-scale databases such as Microsoft Access, Microsoft Excel can assist a lot.

One more tip in this area is that if there is already existing tool in place then queries for retrieval of data should be planned and made available for use anytime and by anyone.

Reengineer The Test Management

Most organizations don’t have a standard process for defining, organizing, managing, and documenting their testing efforts. Often testing is conducted as an ad hoc activity, and it changes with every new project. Without a standard foundation for test planning, development, execution, and defect tracking, testing efforts are nonrepeatable, nonreusable, and difficult to measure.
Generating a test status report is very time consuming and many times not reliable. It is difficult to procure testing information such as
How much testing needs to be done?
How much testing has been completed to date?
What are the results of these tests?
Who tested what, and when was it last tested?
What is the defect number for this failed test case?
Do we have test results for this build?
Do we have a history of test case results across different builds?
How can we share the test cases with a remote team?
Does our product meet the requirements that we originally set?
What is the requirements test coverage?
Is our product ready for release?
Getting this information fast is critical for software product and process quality. But many times, it is difficult to get this information, depending on the way test cases and execution results are defined, organized, and managed.
Most organizations still use word processing tools or spreadsheets to define and manage test cases. There are many problems associated with defining and storing test cases in decentralized documents:
1. Tracking. Testing is a repetitive task. Once a test case has been defined, it should be reusable until the application is changed. Unstructured testing without following any standard process can result in creation of tests, designs, and plans that are not repeatable and cannot be reused for future iterations of the test. It is difficult to locate and track decentralized test documents.
2. Reuse. Because it is difficult to locate test cases for execution, they are seldom used in day to day testing execution.
3. Duplication of test cases and efforts. It is difficult to locate a test case, so there are chances of duplicating the same and wasting the testing effort.
4. Version control. Since there is no central repository, version control becomes difficult, and individual team members may use different versions of test cases.
5. Changes and maintenance. Changes to product features can happen many times during a product development lifecycle. In such scenarios, test cases can become obsolete, rendering the whole effort in test planning a fruitless exercise. It is important to keep the test case list updated to reflect changes to product features; otherwise, for the next phase of testing there will be a tendency to discard the current test case list and start over again.
6. Execution results—logging and tracking. The test execution result history is difficult to maintain. It is difficult to know what testing has been done, which test cases have been executed, results of each test case that is executed, and if a problem report has been written against this failed test case.
7. Incomplete and inconsistent information for decision-making. A defect database provides only one side of the information necessary for knowing the quality of a product. It tells what is broken in a product, and what has been fixed. It does not tell what has been tested and what works. This is almost as important as the defect information.
8. Test metrics. If we cannot have a history of test results, it is difficult to generate test metrics like functional test coverage, defect detection effectiveness, test execution progress, etc.
9. Difficult to associate related information. We also need to deal with huge quantities of information (documents, test data, images, test cases, test plans, results, staffing, timelines, priorities, etc.).
10. Nonpreservation of testware. It is essential that testware (test plans, test cases, test data, test results, and execution details) be stored and preserved for reuse on subsequent versions of a single application or sharing between applications. Not only does this testware save time, but over a period of time it gives the organization a pattern, knowledge base, and maturity to pinpoint the error-prone areas in the code development cycle, fix them, and prevent the errors from recurring.
11. Inconsistent processes. Organizations are not static. People move from project to project. If the testing job is performed differently for each project or assignment, the knowledge gained on one assignment is not transferable to the next. Time is then lost on each assignment as the process is redefined.
12. Requirements traceability and coverage. In the ideal world, each requirement must be tested at least once, and some requirements will be tested several times. With decentralized documents, it is difficult to cross-link test cases with requirements. Test efforts are ineffective if we have thousands of test cases but don’t know which one tests which requirement.

The problems due to unstructured, decentralized test management can be solved by reengineering the test management process. A testing project starts by building a test plan and proceeds to creating test cases, implementing test scripts, executing tests, and evaluating and reporting on results.
The objectives of reengineering test management are to
1. assist in defining, managing, maintaining, and archiving testware
2. assist in test execution and maintaining the results log over different test runs and builds
3. centralize all testing documentation, information, and access
4. enable test case reuse
5. provide detailed and summarized information about the testing status for decision support
6. improve tester productivity
7. track test cases and their relationship with requirements and product defects
A Reengineered test management process can help in improving key processes in test definition, tracking, execution, and reporting.

Four Keys to Better Test Management

When I started working as an onsite coordinator for various vendors(NIIT, Keane etc.) of SEI in USA, there seemed to be a disjoint between offshore development and onsite test groups.

There were four things that became very obvious to me, that were necessary to get better organized:

1. To have a common set of ground rules on the test progress, defect reporting, and verification.
2. Be able to convey how is testing going on a frequent basis.
3. Be able to determine what do I need to test and stand behind the reasons why.
4. Maintain good communication with the technical leaders to help move the product through the development phases by being proactive rather than reactive.

Provide Test Expertise to Test Group

The test manager should provide some test expertise to the test group. This expertise comes in a variety of forms:

Ability to discuss, mentor, and/or train current employees on general testing techniques.
Ability to look forward to the products in development and planned for development, to foresee what new expertise and training is needed by the group.
Ability to hire people based on current and future testing needs.

Note that this does not mean the test manager has to provide specific low-level tests for the product. A test lead's job is developing tests, not the test manager. As a test manager, you provide leverage to all the testers and to corporate management. You need to define for yourself the difference between mentoring and doing other people's work.

Develop Test Strategies

The ship criterias should have a significant impact on the test strategy for the product. For example, if the product is an operating system that has to run on multiple hardware platforms, ship criteria that emphasize the number of platforms the product has to run on, and de-emphasize the specific features of the operating system will push your strategy to one of hardware combination testing, not depth-first testing across many features on one particular platform. The test manager does not necessarily develop the entire strategy for testing, but should have input to the strategy, and participate in reviewing the strategy with the test group.

Define and Verify Product Ship Criteria

As the test manager, we have an opportunity to negotiate the criteria with marketing and the development groups. You want to verify criteria with customers or representatives of the customer. The development group's job is to figure out how to achieve what the company wants to achieve. Using the customer requirements to figure out what to do provides the developers a complete view of the product and how it works. Once the product is defined in terms of what and how, the testing effort can verify how well the product meets the real requirements.
It's important for testers to prioritize their testing so that the product ship criteria can be met. Since very few projects have enough time to do everything, getting the testers enough information for what and when to test is a very important role for the test manager.
Corporate managers need to understand the product ship criteria enough to be able to judge whether the product is ready to ship at a given date. I don't believe the test group should be either the holder of product approval or rejection-that role belongs to corporate management. Having pre-negotiated, agreed-upon ship criteria helps corporate managers judge for themselves whether or not a product is ready to ship. Pre-negotiated criteria helps the project team decide what product quality is when one no one is stressed from the project work.

Gather Product Information

The successful test manager also gathers product information, in the form of defect counts, test pass rates, and other meaningful data. The test manager defines the data, and then gathers it for presentation to corporate management. For example, I gather what I consider to be the "basic" metrics:
Defect find and close rates by week, normalized against level of effort (are we finding defects, and can developers keep up with the number found and the ones necessary to fix?)
Number of tests planned, run, passed by week (do we know what we have to test, and are we able to do so?)
Defects found per activity vs. total defects found (which activities find the most defects?)
Schedule estimates vs. actuals (will we make the dates, and how well do we estimate?)
People on the project, planned vs. actual by week or month (do we have the people we need when we need them?)
Major and minor requirements changes (do we know what we have to do, and does it change?)

Measure what you’ve done

Few days ago, at a testing conference, I met a few of incredibly busy test managers. They were taking conference calls, sending and receiving email, and still trying to attend sessions at the conference. I found that one of the reasons managers are so busy is that some of them don’t know whether they’ve gotten anything done. They’re all crazy-busy, but most of them feel as if they’ve accomplished nothing.

When your efforts have more to do with team performance and avoiding disasters than a tangible product, how can you measure what you’ve done in a concrete way? If you are a test manager, here are some ways to measure what you’ve done:

How many people have you unwedged or unblocked this week?

Sometimes our staff may not realize they are stuck trying to solve problems. If you are able to recognize that someone is stuck and point them in a more useful direction, then you’ve accomplished valuable work.

How many of your staff were able to get more work done this week, based on what you said?

Managers leverage the work other people do. When you facilitate other people’s work, then you’re doing management work. If you discussed test plans or your staff’s choices about which tests to implement, then you’ve helped people leverage their work.

How much strategic work did you do this week?

Managers set the strategy for their groups, and then put the tactics in place to make it happen. Maybe you’re working on improving your test capability with a new test lab, or arranging ongoing education for your testers. Maybe you’re working on decreasing time to market by changing how you plan your projects, or how the developers and testers work together. Strategic goals are long-term goals, so you’d expect to make only a little progress on strategic work each week. But the effort you invest and the progress you make certainly count.

How many crises did you prevent?

Every time you solve a problem and prevent a disaster or crisis, you’ve allowed your technical staff to go forward. For example, if you realized early enough in the project that the developers and testers were working toward different goals, and you started developing release criteria, you’ve averted plenty of problems and a potential ship crisis.

How many meetings were you able to cancel?

Unfortunately, too many of us work in meeting-happy organizations. A meeting that doesn’t include the following characteristics is not likely to help your project or your team move forward:
It starts on time
It ends on time or early
You can leave the meeting if you’re not needed
Action items come out of the meeting
The meeting has an agenda and minutes
If you are able to cancel meetings that don’t actually help move a project forward, then you’ve done wonderful management work. Look for these opportunities and take advantage of them when you see them.

These are just five ways to measure your managerial output. You may want to create a short checklist that you review each week to keep track of your efforts. Any work you do that helps your staff and the long-term health of your organization is good, tangible work. Count it.