Wednesday, September 27, 2023

8 min

Essential metrics to measure the effectiveness of QA process



As an indispensable part of the IT world, quality assurance (QA) has become an important area in the lives of developers and testers. As websites and apps have become more sophisticated in recent years, the quality assurance process has also become equally active. More extensive websites and applications usually require more extensive and thorough testing before release.  


The quality assurance process must be carefully planned and observed to be sufficiently successful. The most effective way to monitor the effectiveness of the quality assurance process is to use appropriate metrics.  


When discussing the applicability of quality assurance performance standards, it is also important to note that quality assurance standards do not refer to the company as a single unit. Instead, they refer to the different levels that exist, each level having its own set of standards. For example, you may have metrics in the design phase, and some metrics are related to the number of defects in a given period. Such metrics are best suited for their specific products, although you can also evaluate them at the overall department level.  


Before determining quality assurance standards, it is necessary to understand what questions are intended to be answered by the application of these standards.


  So the questions to ask beforehand are:


1.How long will the test take?

2.How much money is allocated for the testing process?

3.What kind of bugs do we have? (bug status)

4.What proportion of total software testing do existing tests provide (what percentage?)

5.Are the estimated dates sufficient to carry out the testing?  


Basic standards of quality assurance


Now let's move on to the metrics used to measure the effectiveness of the testing process.


There are absolute values that are always represented by numbers:  


1.Number of test cases

2.The number of passed test cases

3.Number of failed test cases

4.Number of blocked test cases

5.Number of bugs found

6.Number of accepted bugs

7.The number of rejected bugs

8.Number of delayed bugs

9.Number of critical bugs

10.Number of bugs found since release


But ultimately, all the above indicators are not enough to measure the success and efficiency of the testing process. To see and evaluate the overall picture, we need more specific and calculated data.


 Measurement of testing effort:


Metrics include number of tests performed, bugs per hour of testing, average time to fix bugs, etc.  


1.Number of tests in a given period = Number of tests/Total time

2.Test design efficiency = Number of tests designed / Total time

3.Test Review Efficiency = Number of Tests Reviewed/Total Time

4.Number of bugs per test = Total number of bugs/Total number of tests


Test coverage measurement:


Helps to understand which parts of the application should be tested. Assuming the tests are of good quality, this metric can reveal which parts of the software have the most defects.      


1.Test coverage percentage = (number of tests performed/number of tests to be performed) X 100

2.Claims Coverage = (Number of Claims Coverage/Total Number of Claims) X 100


Measuring Team Performance:


This evaluates the distribution of work among team members and the results obtained by the respective team member. Experts recommend never using these criteria to pit individual testers against each other, but rather to promote progress within the team and achieve success through collaborative efforts.  


1.Number of bugs returned by each team member

2.Number of bugs opened by each team member

3.Number of test cases assigned to each team member

4.Number of test cases performed by each team member    


Bug distribution measurement:


It helps to understand which part of the software is most susceptible to defects and therefore where to focus testing efforts.


1.Distribution of bugs by cause

2.Breakdown of bugs by feature/functional area

3.Distribution of bugs by severity

4.Allocating bugs by priority

5.Distribution of bugs by type

6.Breakdown of bugs by tester – Dev, QA, UAT or End-user


Measuring Test Performance:


In this way, we can evaluate how successful the testing is.

(bugs found in 1 test / total number of bugs found in tests + after release) X 100


Our experience


As a quality assurance company, we cannot avoid measuring the effectiveness of our work using the above criteria.

Typically, the metrics used by our team depend on the current project the team/couple is working on.

If it is a small project involving two of our specialists (as we mentioned in one of our previous blogs, we mostly work on projects in small teams or pairs), there is no need to measure, for example, "Team Effectiveness".

But in general, there are metrics that should always be considered and measured well, no matter what type of project we're dealing with, such as "Defect Distribution" metrics, as well as "Test Coverage" metrics, which are of primary importance.

Content