The Cost of Static Analysis Backlogs

As we visit leading software development organizations around the world, one thing that we see again and again is a huge backlog of defects reported by their static analysis tool. Static analysis (or source code analysis) tools are great at quickly finding lots of critical problems in software code. No test cases and minimal setup are required and so there are few barriers to getting immediate value out of the tool. In addition, while analysis requires a significant amount of processing, the wall clock time required to get results is insignificant. To analyze millions of lines of code takes just hours resulting in thousands of potential problems.

But, many organizations run the tool, cherry pick a few problems and then leave the backlog there, sometimes sitting there for weeks and months. One would ask why would anyone do this:

• there are real critical bugs in the backlog, some of which are likely to be showstoppers
• bugs reported from static analysis tools are easy to fix, because they point to problems in right in the source code
• license costs for static analysis tools are not insignificant
• the value of a static tool is not in how bugs are found, but in how many good bugs are fixed

But there are real reasons why these backlogs sit unaddressed:

• handling source code analysis results are not a part of the standard development process. When it becomes an extra step, it won't be done without the right incentives and consequences.
• getting developers trained on code analysis takes an investment in time and resource. Most organizations don't have the political will and timing is often important.
  
• source code analysis results vary in quality. Keep in mind that a source code analysis solution can only derive information from the source code and has no knowledge of environmental issues or usage/intent issues. Depending upon the quality of the tool, you may find that 10-20% of problems reported are high priority issues. The challenge, of course is knowing which ones are the 10-20%. Engineers must wade through lower priority issues, false positives and "don't cares" in order to find the good ones. Some tools have higher than a 50% false positive rate. Developers quickly tune out after a few false positives or "don't care" reports.
  
• people don't have enough time. Everyone is being pulled in 10 different directions and the results from a static analysis solution get deprioritized and ignored. It becomes a thing they'll get to when they have time.
  

There is a high probability that there are true, highly critical bugs in the backlog, some of which may cost the organization hundreds of thousands of dollars once they surface. And the bugs are not hard to fix - it's just that there are so many of them - how can one get started?

We've seen many organizations deal with this problem. Some organizations have allocated resources on their team and created "bug fest" days to bring the backlog down to a manageable level. Other organizations have looked to outside firms such as ours to bring in expertise to kill a backlog quickly and efficiently. Others have "bit the bullet" and rolled static analysis to the general development organization with quota's and status reports to address the backlog.

Getting a backlog down to zero is an important part of starting a larger rollout of static analysis within the organization. In many organizations, the incremental changes to a codebase are minimal compared to the size of the overall legacy code. By addressing the backlog first, you get rid of any major potential problems and start with a clean slate to address any new defects that may arrive from code changes. It sets the stage for a successful rollout, declutters the reports coming from the tool and addresses potentially serious problems that may be lurking, unaddressed in your defect reports.

Depending upon the size of the backlog and the cost of a typical bug reported by a static analysis solution (typically crashes, memory leaks, buffer overruns, etc.) not addressing a backlog may be an extremely costly mistake.