The Hidden Cost of Source Code Analysis

Static analysis (or source code analysis) seems like a no-brainer. The analysis runs on your source code, bugs get produced and you fix them. The sooner you fix problems, the cheaper it is, often by orders of magnitude. However, there is a hidden cost to static analysis -- one that most organizations need to take into consideration but rarely do, until the rollout is well under way.

Let's Take a Deeper Look
Are all static analysis defects created equal? Certainly not. There are some issues that are more important to address than others. Every organization categorizes defects differently. For some a memory leak may be important, particularly for long running applications where even a tiny memory leak can add up to a big problem. For others applications, like a desktop application that loads and exits frequently, memory may be hardly an issue but crashes may be a serious priority. Other factors that affect priority can include effect (such as data corruption), frequency, compliance, etc. Regardless of any company's individual priority, one can roughly bucket static analysis defects in these categories:

1. Critical - we should fix it now (e.g. don't check it in)
   
2. High - we should fix it before the next milestone
3. Medium - we should fix it at the next appropriate milestone
4. Low - nice to have - fix it if opportunistic situation
5. Ignore - analysis is doing the right thing but the problem doesn't need to be fixed (might be in test code, 3rd party code or perhaps the defect is incorrect because it couldn't take into account an environmental assumption)
6. False positive - the analysis goofed and the bug report is wrong

The Problem With Prioritization
Keep in mind that most bugs are prioritized based on the effect of the bug, which is easier to tie to a business problem. Static analysis defects are reports of flaws in the code -- for instance, it might show a potential memory leak along a specific path in the code. It may be difficult to discern how much memory is being leaked, what the specific steps are that cause the leak and what happens if enough memory leaks. Static analysis defects are thus, harder to prioritize.

A "Typical" Distribution
The distribution of defect types varies greatly among organizations. Some organizations (like a company that writes mil-aero software) consider any coding defect a serious issue. Based on experience with customers at Code Integrity Solutions, we often see defect distributions like this:

1. Critical - 10%
   
2. High - 10%
3. Medium - 10%
4. Low - 10%
5. Ignore - 45%
   
6. False positive - 15%

With different codebases, different static analysis solutions and even different triagers you may get varying results.

The Biggest Cost?
In addressing the defects reported from a static analysis tool, you need to:

1. Make sure the analysis is set up properly
2. Someone needs to triage the results meaning interpreting the results for action
3. Someone needs to address the issue, typically by fixing the problem
4. Someone needs to verify that the issue is addressed

#1 is typically handled by an administrator skilled in the static analysis tool. #2 may take anywhere from 5-10 minutes on average (we'll assume 10 minutes on average because some go quick, and some take longer). #3 often takes 20 minutes to throughly understand the issue, find the right solution, make the necessary code changes and then review/QA the change. #4 we don't have reliable data for.

Back of the Envelope
Most organizations purchase a static analysis solution for a product after the product has already been released. Therefore there is a considerable amount of "legacy" code already there. When you run a static analysis solution it finds problems consistently across old and new code. Thus, many organizations who recently purchased a static analysis start by running the analysis and creating a backlog of sometimes thousands, if not tens of thousands of defects.

Let's use a fictitious example. Let's say an organization has 3 million lines of code and finds 3000 defects in their backlog. In order to triage all of the defects, it would take 3000 defects x 10 minutes per defect = 500 hours (62.5 developer days). For a team of 12 developers, this would take a little over 5 business days or essentially a full week. Assuming the distribution above, the distribution of defects would be around:

1. Critical - 10% = 300 defects
2. High - 10% = 300 defects
   
3. Medium - 10% = 300 defects
   
4. Low - 10% = 300 defects
   
5. Ignore - 45% = 1350 defects
   
6. False positive - 15% = 450 defects
   

To fix 300 defects would take approximately 300 defects x 20 minutes per defect = 100 hours. For the same team of 12 developers, this would take 8.3 hours or a little over 1 day.

The Hidden Cost
So to triage all the defects takes a team of 12 developers 1 full week. To fix the defects that you care most about, it takes 1 day. That's a whopping ratio of 5:1. Clearly, one of the largest costs of static analysis is triaging the results - to sift through to determine which are the ones worth fixing and which are the ones worth ignoring. In addition, some static analysis tools have much higher false positive rates which means much of the time is spent in reviewing defects rather than the actual time performing the fixes.

The Case for Outside Help
For most organizations, developer time is precious - the most constrained of resources. A week of a team of 12's time can mean a significant number of customer-focused features developed. Many organizations are turning to outside teams to help review defects so that their developers can focus on their core competence - developing capabilities within their product rather than developing expertise reading static analysis results.

An additional strategy is to address the defects even earlier in the software development process, e.g. to the developers as they are coding. This may slightly reduce triage time overall.

However, whether you use an outside team or not, the time required to triage defects must be taken into consideration. It's surprising for many that the most time consuming aspect of using static analysis is in the triaging of results rather than the actual fixing of defects.