The Challenges (and Benefits) of Source Code Analysis

Source code analysis, (or static analysis) has been used for many years now to perform detection of defects. Lint was one of the more popular early technologies for catching potential problems in code. Since then, the technology for finding defects has greatly improved enabling software development organizations to automatically detect problems with greater depth and accuracy than ever before.

The qualitative arguments have always been apparent. Finding and fixing a defect after a product has shipped can be orders or magnitude more expensive than if you found the problem during development. The cost of recreating a bug, debugging it and then fixing it is significantly larger than a tool pointing to where in the source code your problem is while you are developing it.

While many leading organizations agree that source code analysis is important, it is often too hard for these same organizations to fully bake it into their process and use it in a disciplined way. There are numerous reasons why:

• Results from static analysis tools always have false positives. The tolerance level of a software developer for false positives is very low. If they see more false positives than true positives, then they begin to mistrust the results. The downward cycle begins as they investigate each new report as likely to be incorrect. We saw in several customer situations where developers will incorrectly ignore a report because they assumed it was incorrect and did not investigate it as fully as they should have.
• Source code analysis is not yet part of the standard "reference architecture" for software development. Everyone needs a source control system, bug tracking system, editors, etc. but source code analysis is still not standard issue for most organizations. Adoption of such tools thus requires more effort in training and customization for rollout. In addition, organizations can utilize static analysis at many different points -- some in the IDE, some as part of a nightly build and some on a per sprint or release basis. Very few organizations have experienced static analysis practitioners to establish the right structure and processes for the organization.
  
• Source code analysis can be more "vitamin" than "aspirin". When an executive comes to engineering complaining about the latest bug that is causing you to lose customers, there is an easy way to value the pain. Get it done and don't go home until you do. With a problem reported in static analysis, the pain levels are perceived to be much lower. It is often too difficult to understand how a source code flaw would manifest in production. A developer can be too far removed from how it might affect customers (and which customers). Business metrics are also hard because it isn't easy to cleanly measure field impact to defects fixed from static analysis.
  
• In these economic times, organizations have to do more with less. Resources are constantly being pulled on to different projects. Organizational structures can change several times in a year. Priorities seem like they change with the wind to meet market conditions. Time and again, we see many organizations purchase tools but then get 10% of the value because they don't have the resources. Due to the way budgeting is performed, hiring headcount means the "wrong" kind of costs. The result is that no one has true ownership of making it happen and the tool effectively sits on the shelf.
  

It's a shame. Coverity has provided some metrics that they have been able to increase developer productivity by 12.5%, accelerate development cycles by 10-15% and reduce field defects by 30%. Caper Jones says that defects can be reduced by up to a factor of six. Although the mileage varies with each of these statistics, these are nevertheless significant numbers, particularly if your business is the technology that you're building. Many leading firms like Microsoft and Cisco and government organizations like the Air Force have made significant investments into static analysis and have successfully made it an integral part of their software development process. The road isn't hard but not easy either. The right combination of executive sponsorship, appropriate processes and incentives, accurate and effcient tools, ownership and in-house and outsourced expertise can help most any organization improve their software development organization significantly.