Making Software Quality Metrics Actually Work

There is no way to control things which are not being measured properly.
Being able to measure quality of software and having understanding what contributes to it, it’s possible to improve the quality and eliminate (or minimize) negative factors.
But, it seems, just measuring the quality is not enough. Even to have right methodology is not enough. Even good tools may not help. Why?

I would like to list 7 things, which can address question “why”, help to adopt, increase efficiency and accuracy of software quality measurement process.

1. Measuring quality is always a tough task because of employees perception.

People tend to take personally any indicators which potentially could harm them. Hence, developers or testers will do everything to make them look better in terms of quality metrics. Managers too.

2. It should be very clear stated that quality metrics will not be used to punish people or affect them negatively.

If people are afraid to be punished, they will do all kind of tricks: window-dressing, opposition to software quality adoption, complaining about accuracy of metrics data, challenging methodology, even sabotaging. Rewards are also not always work.
One company came up with “great idea”, they offered some amount of money to testers for each bug found and to developers for each bug fixed. After a few weeks they terminated this incentive because of number of found (and fixed) bugs skyrocketed. Turned out, testers and developers made a deal to raise easy money.

3. Quality Metrics should help regular employees.

When you ask a developer to do something extra on daily basis (fill extra form, or track some data, for example) it always should be a good reason for that. Things like “it helps company to track X amount of additional metrics” don’t work. These metrics should help development process overall and this developer in particular. What would work is: “having clear picture our software quality will help to streamline the process and YOU will be having more organized environment and less overtimes and around-the-clock red-eye nights”. It should not be just words. If there is no way to improve employee’s life via new metrics, try to collect them without extra efforts.

4. Interpretation of metrics data can be improved if it includes the following questions.

“What this data tells me?”, “Based on the data, can we tell if we are improving or going worse?”, “What is the question this metric is trying to answer?”, “Can it tell me what are the positive / negative contributors to the metric?”, “Does this metric stimulate people to do right things?”, “What kind of goal could we achieve, if we use this metrics as a primary one?”.

5. No one quality metric can provide an accurate and complete picture.

Software development process is not one-dimensional thing. Sometimes increase of incoming defects means positive things. Sometimes absence of defects is a bad indicator. Sometimes new release has much more defects and it’s OK. Depending on the context, one metric can be interpreted differently, that’s why metrics should be used as a package.

6. Too much metrics can create a mess.

Optimal number of measurements should be not more than 10-12 and not less than 4-6. Too much metrics will confuse people too less – provide incomplete picture.

7. Software quality metrics should follow the development process.

There is no “silver bullet” and universal metrics set for everybody and everything. Metrics should measure existent process and follow it, not otherwise. Breaking the process for the sake of “innovative metrics methodology” could cost more than the adoption could potentially save.