High Efficiency Defect Removal for Software Projects

Capers Jones, VP and CTO, Namcook Analytics LLC
350
581
114

Software quality depends upon two important variables. The first variable is that of “defect potentials” or the sum total of bugs likely to occur in requirements, architecture, design, code, documents, and “bad fixes” or new bugs in bug repairs. Defect potentials are measured using function points, since “lines of code” cannot deal with requirements and design defects. The second variable is “defect removal efficiency” (DRE) or the percentage of bugs found and eliminated before release of software to clients.

Defect potentials and defect removal efficiency (DRE) are useful quality metrics developed by IBM circa 1973 and widely used by technology companies.

Defect potentials are the sum total of bugs found in requirements, architecture, design, code, and other sources of error. The approximate U.S. average for defect potentials is shown in table 1.

Table 1: Average defect potentials circa 2015 for the United States average

Requirements                1.00 defects per function point

  • Architecture                   0.30 defects per function point
  • Design                          1.15 defects per function point
  • Code                            1.35 defects per function point
  • Security code flaws      0.25 security flaws per function point
  • Documents                  0.50 defects per function point
  • Bad fixes                     0.40 defects per function point
  • Totals                         4.95 defects per function point

Defect potentials are of necessity measured using function point metrics. The older “lines of code” metric cannot show requirements, architecture, and design defects not any other defect outside the code itself.

Defect potentials range from < 2.00 per function point for top teams and quality-strong methods to > 6.00 defects per function point for inexperienced teams and quality-weak methods.

Defect removal efficiency (DRE) is also a powerful and useful metric.  Every important project should top 99 percent in DRE, but few do.

DRE is measured by keeping track of all bugs found internally during development, and comparing these to customer-reported bugs during the first 90 days of usage. If internal bugs found during development total 95 and customers report 5 bugs, DRE is 95 percent.

Table 2 shows approximate DRE values for common pre-test and test methods although there are variations for each method and also for the patterns of methods used.

To illustrate the principles of optimal defect prevention, pre-test removal, and test defect removal table 2 shows a sequence of pre-test and test stages that will top 99 percent  in defect removal efficiency (DRE). Table 2 illustrates 1,000 function points and about 53,000 Java statements.

“Software quality depends upon two important variables— Defect Potentials and Bad Fixes”

Table 2: DRE > 99% Defects
Requirements defect potential 134
Design defect potential 561
Code defect potential 887
Document defect potential 135
Total Defect Potential 1717
Per function point 1.72
Per KLOC 32.20

 

Defect Prevention Efficiency Remainder Bad Fixes Costs
Joint Application Design (JAD) 27% 1262 5 $28052
Quality Function Deployment 30% 888 4 $39633
Prototype 20% 713 2 $17045
Models 68% 229 5 $42684
Subtotal 86% 234 15 $127415

 

Pre-Test Removal Efficiency Remainder Bad Fixes Costs
Desk check 27% 171 2 $13225
Static analysis 55% 78 1 $7823
Inspections 93% 5 0 $73791
Subtotal 98% 6 3 $94839

 

Test Removal Efficiency Remainder Bad Fixes Costs
Unit 32 4 0 $22390
Function 35 2 0 $39835
Regression 14 2 0 $51578
Component 32 1 0 $57704
Performance 14 1 0 $33366
System 36 1 0 $63747
Acceptance 17 1 0 $15225
Subtotal 87 1 0 $283845

 

PRE-RELEASE COSTS --- 1734 3 $506099
POST-RELEASE REPAIRS TECHNICAL DEBT 1 0 $658
MAINTENANCE OVERHEAD       $46545
COST OF QUALITY COQ     $553302

 

Defects delivered 1
High severity 0
Security flaws 0

High severity %

11.58%

 

Delivered Per FP 0.001
High severity per FP 0
Security flaws per FP 0
Delivered Per KLOC 0.014
High severity per KLOC 0.002
Security flaws per KLOC 0.001
Cumulative Removal Efficiency 99.96%

DRE measures can be applied to any combination of pre-test and testing stages. The U.S. norm is to use static analysis before testing and six kinds of testing:  unit test, function test, regression test, performance test, system test, and acceptance test.  This combination usually results in about 95 percent  DRE. 

In order to top 99 percent  in DRE table 2 shows several forms of defect prevention and includes inspections as an important pre-test removal method. Formal inspections have the highest DRE of any known method, and over 50 years of empirical data.

Due to inspections, static analysis, and formal testing by certified test personnel, DRE for code defects can top 99.75 percent .  It is harder to top 99 percent  for requirements and design bugs since both resist testing and can only be found via inspections, or by text static analysis.

Conclusions

The combination of defect potential and defect removal efficiency (DRE) measures provide software engineering and quality personnel with powerful tools for predicting and measuring all forms of defect prevention and all forms of defect removal.

Read Also

Lessons from 2014 for a Better Security Strategy in 2015

John McClurg, Chief Security Officer and VP, Dell

The Crucial Link in Contract Lifecycle Management

Ulf Zetterberg, Founder & CEO, Seal Software

Moving Forward With Virtualized Desktop Is Something New

Richard Ferrara, CTO & EVP, Woodforest National Bank

Implementing Technology That Understands Your Business

Michael Everly, CIO, D&H Distributing