Code Integrity Blog

The Speed of Development

2011-06-10T17:01:00.000-07:00

We were recently working with a company that had a very large codebase. It took 2 hours to build and took another 4 to 24 hours to run a variety of tests such as unit tests and static analysis. Developers were doing full builds locally on their sad, underpowered machines. In this time of multi-core computing, developers were doing parallel computing the old fashion way: using the lengthy build and test time to catch up on email.

In the worst case, you find a problem that you need to fix. Guess what? Verifying whether the fix was sufficient creates yet another cycle. Not only that, if the build systems have a problem, the turnaround on fixing those problems can literally fritter away critical days from the development cycle. Enlightened companies realize that this is a lot of time wasted but most importantly, creates a latency to the information you need to move code toward the finish line. If you are going to fail, you should fail fast so you can fix it quickly.

Software is complex and brittle. Not surprisingly, the infrastructure to build and test and manage the software can be similarly unreliable. There is good reason: software systems must support many multiple operating systems, integrations and devices and development cycle times have been compressed with Agile methodology.

After this company spent effort (and some hardware) to move their build to 15 minutes and all of their tests to run in 5 hours, their developers could at least get same day response times. One more iteration meant that higher quality features could be checked in faster making their developers more productive.

They created a rough matrix:

Build/Test
========
15 minute = "immediate" feedback
30 minute = basically "on demand"
1 hour -> 3 hours = run multiple times per day
4 hours -> 6 hours = potentially same day response time
7 hours -> 14 hours = overnight process (ready by next morning)

Their next goal, through further optimizations was to get to multiple times per day.

Building a Better Build System

2011-05-27T16:26:00.000-07:00

Here's an interesting and lengthy interview with Peter Smith, author of a new book, Software Build Systems: Principles and Experiences. For the full interview, please see this link:

http://www.informit.com/articles/article.aspx?p=1702971

Another Tuning Story...

2011-04-07T14:07:00.000-07:00

One customer that we worked with in the past has a medium sized codebase that handles financial-based transactions. Their business deals with a lot of money and is highly regulated. Management has very high expectations that their software development team build their products with superior quality. Not surprisingly they have turned to many modern techniques and technologies to keep up with the pace and scale of development. They began using source code analysis a few years ago in order to detect and fix problems before release.

In their first run alone, the source code analysis tool reported over 20,000 potential defects. They didn't know where to begin. Even if it took 1 minute to go over each defect it would take over 40 business days to complete. Realistically to query up the result, understand the defect report in the code, prioritize and document it briefly would take more on the order of 5-10 minutes per issue. That's of course averaging the occasional really easy issue and really difficult issue. At 7.5 minutes per issue that becomes 312 business days -- a significant investment by any standard. Of course static analysis is one of the best ways to improve the systemic quality of your code as it is much less expensive to find and fix problems before release.

But there is a better way than using brute force to fix static analysis defects. Tuning the analysis almost always results in significant improvements in the analysis quality, which means fewer false defects and more real defects. After just a couple of days with this customer we knocked their queue down to about 9000 defects. Actually, we knocked it down to about 8000 defects but then found an additional 1000 new defects that weren't being detected before. The time savings, as you can imagine are large. The psychological effect was even better. Although 9000 is still a mountain to climb, it's psychologically a lot better to hike up Kilimanjaro than scale Mount Everest.

Power to the Developer

2011-03-11T21:23:00.007-08:00

Increasingly, we've seen software development organizations wanting to give power to their developers to find and fix problems prior to check-in. There are several very good reasons for this:

If management has instituted an acceptance criteria - such as no new static analysis defects, then naturally developers will be evaluated based on whether they fix static analysis defects. A developer analysis gives power to the developers to find and fix problems locally before it becomes publicly visible in a system analysis.
Finding and fixing problems before check-in makes the codeline higher quality. Rather than checking in potential bugs, the code can be cleaned earlier. Everyone pulls from higher quality code, particularly in agile and continuous integration environments.
Giving developers analysis results right when they want it increases efficiency. Rather than waiting for the nightly or even weekly results, minimizes context switching. In addition, fixing a problem can generate another problem. Being able to iterate multiple times quickly at one time is much more efficient than fixing and then waiting a day or even a week to see if no new problem has been created.

Of course working from the desktop may require more coordination. You have to ensure that the analysis being used locally is the same as being used for the system build -- consistency ensures that everyone is working towards the same goal. Generally, you don't want developers to be analyzing either more or less than the established criteria. Also system analyses can generate some different bugs than the desktop analysis - most of the times very slightly. This is natural because the system analysis usually has more complete information and thus has better analyses at its disposal. Still, this can cause confusion when a developer analysis doesn't pick up something that the system analysis picked up.

Desktop analysis is also an extra flow that requires additional time and training. However, many would say that it is a small price to pay given the benefits.

Managing Branches in Static Analysis

2011-01-11T12:37:00.000-08:00

In some respects, making software isn't much different than making sausage - it's messy and most would rather skip the details. For those that choose the software development career, you have to live it everyday. One area where I've seen significant mess is in the management of branches. Delivery of software is seldom a one-time effort - with future versions expected to be delivered frequently. Add to that the complexity of:

software being available on different platforms and operating systems
emergency software patches being made available
different versions of the software customized to specific customers
new major updates/rearchitectures/research simultaneously in the works as a production version

Pretty soon you may have branches all over the place. Most organizations have a main branch which is the "gold master" with development branches being forked from it. Many organizations will make it a requirement that the main branch should be release ready at any time. Some organizations won't even allow edits on the main branch, instead requiring all changes to be made in forked development branch from the main branch. Other organizations have developers happily chipping away at the main branch as well as creating multiple branches that may eventually be released. Chaos ensues in these environments.

For those organizations that at least have a main and development branches, changes can be tested in the development branch prior to merge into the main branch. Of course there may have been other merges that have taken place during the time that development branch was worked on and thus integration can be painful. Software has lots of dependencies and when the assumptions are changed, software breaks in unpredictable ways.

The increasingly popular continuous integration process is designed to limit the pain of branch integration by doing it early and often. Hand in hand with Agile, continuous integration has reduced the risk of complexity and therefore delay in releasing software.

Where does static analysis fit in?

There are lots of places where you can run the analysis. You can run it regularly on the main branch (which probably already has a nightly scheduled build or continuous integration process). This is a great way to quickly find out if something has broken, particularly during a merge.

Some organizations want to run it also on development branches - in a sense, creating a no new defects policy from branch pull to branch integration. Of course when the development branch merges back into main, an analysis should be performed as part of the testing to see what breaks when things get merged. This can be done on a manual basis.

Performing analysis on the development branch may not be entirely required -- it depends on when you want to be made aware of problems and when you want to enforce your acceptance criteria. Analyzing every development branch will add to the IT costs as you will need more hardware to run the analyses on development branches. For some organizations this is overkill - for others this is essential as their branches may be long lived and they need to meet their criteria as early as possible.

One additional way to still have the acceptance criteria enforced at the main branch but still enable developers to find and fix problems as they develop in the development branches is to have developers analyze the code they are working on in their own desktops. Some static analysis tools provide a "desktop" version of their tool so that developers have the power to analyze the code they are working on and iteratively fix problems well in advance of a merge. This enables them to find and fix problems early. However, one must keep in mind that the development branch and particularly the code being worked on by a specific developer may vary quite a bit from the main branch and thus not all problems can be found at the desktop. Frequent merging should help solve that problem. Regardless, doing some analysis on the development branches enables developers to fix problems sooner which pays off in higher quality software earlier and better productivity.

Static Analysis Reporting For Success

2010-12-15T14:27:00.000-08:00

When looking at effective reporting with static analysis, you have to consider the following:

Who is the audience of the information?
How do you turn data into actionable information?
How does the information support business goals?
How is this information delivered?

Static analysis improves software quality and security and so there are a number of potential "actors" that are affected by the information. These may include:

CEO/CTO
VP of Engineering, VP of Products
Director of Engineering, QA
Manager of Engineering, QA
Director of Tools, Infrastructure, Build/Release
Manager of Tools, Infrastructure, Build/Release
Architect
Engineer / Developer / Programmer
Tools, Build/Release Engineer
QA Engineer
Governance / Compliance Analyst
Product Manager

Each individual has a stake in the successful usage of static analysis. Each constituent has their piece of the puzzle to manage or be aware of. We unfortunately do not have room in the blog post to cover each of these roles.

Business Goals
Creating metrics without business goals as a framework will create wasted cycles, missing information and superfluous data. Some common business goals that we see for static analysis are:

No static analysis defects in code (sometimes called "Clean")

The business goal is to improve software quality and security by addressing all potential issues reported by static analysis.
Variations of this include: "no critical or high priority issues" or no defects in a particular safety-critical component of the code
This often requires it to be "managed" down, meaning that there should be planned downward slope to the number of defects.
Targets may be established at certain milestones, for instance, the number of defects should be reduced by 10% for every release or month. Possibly only the high and critical defects would be managed in this way.
Some organizations even manage down "false positives" with the goal that the code should be written so cleanly that a static analyzer couldn't be confused.

No new defects (sometimes called "No new harm")
- The business goal is to at least keep software quality and security at status quo. The business argument is that new code tends to be the buggiest and that legacy code has already had a chance to be "exercised."
- Variations of this include: "no new critical or high priority issues" or no new defects in a particular safety-critical component of the code. Complexity metrics can also be used as indicators of trouble areas.
- A baseline should be established (such as the beginning of a branch pull) and all new defects (or a high and critical subset) introduced through changes in the code should be fixed
Improve Quality and Security Through Voluntary Usage

The business goal is to provide a convenience to the developers to optionally fix software problems. In practice though, usage typically decreases over time.
Total number of defects fixed should be reported in order to quantify tool's worth

Benchmark Against Competition

The business goal is to be at parity or better than the industry benchmarks
Defect density's are compared to publicly available data, such as open source.

Sample Reports

The number and types of reports that organizations use to more effectively use static analysis is too numerous to put in blog or article form. We include just a few report concepts to give you an idea of the type of reporting that has proven useful to organizations:

Executive Dashboard

Number of defects outstanding in current development branch. Defects of course are likely to be the types of defects that are required to be fixed as part of an acceptance criteria.
Trend of high priority defects that have been reported and that have been fixed since the branch pull. Graph with pretty pictures always impress executives.
Benchmark of quality level compared to industry standards
Defect density (current and trend)
Latest analysis run statistics - was it broken? How much coverage was there?

Managerial dashboard

Number of new defects reported since yesterday
Number of open defects for each component owner
Ranked list of number of fixes by component and by developer
Trend in complexity and defect type
False positive rate overall and by component and by developer
List of open defects by priority and by age

Administrator dashboard

Latest build statistics including lines of code analyzed, number of new defects, analysis times, coverage statistics
New false positives marked to review for configuration changes
Alerts for broken builds or builds that exceed certain performance and coverage thresholds

Architects

Complexity trend over time
Ranking of complexity by function
New false positives marked for review to audit
Defect density by component

Developers

New defects introduced in my code
Outstanding defects in my queue
Code complexity for my code
Benchmarks against other developers in my company

And much, much more.

Information Delivery
How information is received plays an important role in how usable the system is. In general, utilize whatever existing process flows exist - for instance:

Generate email as alerts with a link to get to the details
Create a new bug tracking entry for new defects (if they meet a specific criteria). Some organizations group static analysis bugs into a single bug tracking entry
Display information in an existing continuous integration dashboard
Publish information in a wiki or equivalent intranet
Display information in a code review tool
Generate a PDF of information as part of an executive dashboard

The fewer additional steps required the more likely the tool will be used. Creating separate, independent paths to the information will often cause the tool to not be used.

What's In Your Static Analysis Build Log and Why You Should Care

2010-11-23T15:26:00.001-08:00

Static analysis tools are often used to help detect defects in source code. Simple in concept, they analyze the code and report potential defects. Of course, what everyone wants is a tool with zero false positive reports and zero unreported defects (or false negatives). Developers in particular will be resistant to using a tool that they do not see as accurate. Improving the tools accuracy often requires looking under the hood.

Like most build engines, your static analysis tool(s) create a log somewhere of the steps it takes to analyze the code. Most tools I'm familiar with call it a build log. The build log, I believe, is the first important place to look. Why should you look there? There are errors that are reported in the build log that are not reported any where else. Specifically errors are reported by the compiler ( for each file it looks at) and then by the analysis engine itself. In their default configuration not all compilation errors are visible in the defect list.

What this means is if you run an analysis and then look at just the results these tools provide you may not be getting the whole picture. For example if an entire file fails to compile, static analysis tools will often record this as a error and be duly reported to the end user or administrator. However if just a single function or data structure is unable to be resolved or in some cases include files cannot be found, this may actually go unreported except in the log files. Of course when details are lost so too will analysis fidelity.

To take a recent example I was working with a customer where the analysis build summary reported zero errors. However as is my practice, I searched the build log for error messages from the analysis compiler. I found that the analysis compiler reported 75 errors in which it could not find some include files. In this particular case when we added the include files to the compiler configuration we found that out of 1362 defect that were reported by the original analysis, 315 became "fixed", resulting in a reduction of approximately 25% in the defect density.

So what's an analysis administrator to do? I recommend you count the errors in your build log at the end of every build and if any occur, evaluate them and attempt to fix them. Usually it's pretty straight forward. Sometime it takes more effort, sometimes it's not worth the effort. Usually the payoff is worthwhile. In this case, it took a couple of hours to resolve 315 defect reports. As a result I reduced the number of defects needed to be triaged for this team by 25% (without any loss of real bugs), saving significant time.

-- Kenneth Kron

Static Analysis Metrics - A Window into Development

2010-11-09T06:51:00.000-08:00

You can't manage what you can't measure right? In addition to detecting problems in source code, static analysis can be used to produce metrics that provide insight into productivity, code quality, security and more. We all know the dangers of managing solely by metrics, yet, metrics serve as a highly useful window into your software development process. In this post, we most certainly won't focus on the general challenges of using metrics because they've been written about in many other forums. We will discuss some static analysis metrics here that when pieced with other information can provide useful information.

Static analysis is typically integrated at the system build level. These days, most organizations build their code on a frequent basis - which are convenient points of time to generate static analysis results. If you are running a nightly build, you can get static analysis output on a daily basis. If you run it on a continuous integration basis you get output at the frequency you choose (sometimes static analysis takes longer than a continuous integration cycle if you haven't tuned it but then simply perform static analysis on every 2nd or 3rd cycle). Metrics can be used to get a sense for where you are (status) and point to potential problem areas too (alerts). Thresholds can be set to spur action - such as block a release, or trigger a code review or design review.

Every organization has different priorities and thus every organization should have a different set of metrics that they should track to maintain alignment with goals. We list just a few common metrics that can come from static analysis tools.

Management Level Metrics

Most static analysis tools can easily generate output such as:

Lines of code, number of new lines of code, churn
Number of defects outstanding, number of new defects introduced
Defects per thousand line of code (defect density)
Number of defects fixed so far, number of defects fixed in last period
Comments per line of code
Cyclomatic complexity, function points

Supporting Metrics to Ensure Compliance

Code coverage for tests (100% or near 100% should be attained)
Average time to resolve issues
Number of defects marked as false positives (or low priority)

A couple of considerations about this data:

Breakdowns
The entire codebase should be measured and tracked however convenient breakdowns can be created to make the information more actionable. For instance, data could be collected on a team by team basis (or component by component). Individuals metrics can also be calculcated in order to hone in on specific problem areas or better yet, to highlight improvement areas worth rewarding. In addition, team leads may be interested in tracking information on a function by function basis.

All defect related metrics should be broken down by priorities and/or severities. Metrics that lump in low priorities become diluted. This is particularly important if you have an acceptance criteria that requires all critical and high defects to be fixed before release.

What Is Measured
Because static analysis usually operates at the build level, you can get meaningful statistics on a per codebase perspective. Thus if you have multiple targets (e.g. different operating systems, devices, etc.) you can produce statistics for each. You can find build target specific bugs as well as bugs in shared code.

Absolute Versus Relative
A number without context provides almost no value. Let's say the tool spits out a defect density rate of 1 bug per 1000 lines of code. What can you do with the information? The data becomes rich information when you are able to compare it to other numbers. The most common way to use these metrics is to benchmark against industry standards. Coverity has an interesting database of open source defects from their Department of Homeland Security grant. While comparing your codebase with open source statistics may not be ideal (wouldn't it be great if you could benchmark against your competitors), it provides at least some real world data that you can use to compare and contrast. Are you above or below the average and by how much?

Another common way to interpret the numbers is to evaluate them over time. Take a baseline - a snapshot in time and then measure against that baseline to see how you are trending. A convenient time to take a baseline is at the beginning of a branch pull. Some organizations have the goal of toeing the line - meaning "no new harm." Quite simply the goal is to maintain the same level of quality as prior to any new development efforts. Others at least want to see improvement over time - showing that they are making progress. The high level defect density as well as the overall defect number are very typical KPI's (key performance indicators) chosen by development organizations.

Frequency
Note again that the frequency is as often as you run static analysis. As part of a system build, this is typically run regularly, either every evening, every few days or every continuous integration cycle. Advanced organizations also provide static analysis at the developer level so that they can run on-demand static analysis. Klocwork has some strong capabilities in this area. In this way developers can proactively identify and fix potential problems prior to check-in. Giving the power to the developers to view and improve the metrics as they work, results in better system-level metrics faster and much less chance of having to deal with an emergency later in the development cycle.

Trending
All data should be stored and accessible to get a historical context. Graphical charts enable a quick interpretation of results. Historical context helps you see how you are doing relatively. Start with a baseline and see how you are doing from that point onward.

Data Consistency
Static analysis tools are complex, with many settings that are tweakable in order for it to understand the myriad of different codebases that it can analyze. Think what kind of bugs would be useful to find in a 10,000 line of code embedded device (like a stack overflow) versus a 10 million line of code server application (like a concurrency bug). If not set up properly, the static analysis tool can create challenges in keeping the data stable so that you get consistent numbers to compare over time. For instance, if you upgrade the static analysis tool, new and improved checkers may suddenly change the defect count. New checkers will find new bugs, which may falsely signal that there was a sudden increase in the number of defects in your codebase. Those bugs were always there :-). Similarly, an improved checker may suddenly report many fewer false positives but maybe at the cost of a few real bugs that are no longer found.

If analyses can be run in multiple places (for instance, developers running the analysis tool locally on their machine) then discrepancies can occur if the configurations and the code being analyzed are not identical. Even changing some settings that you wouldn't imagine could possibly cause a data consistency problem can mysteriously change the numbers. While static analysis tools are designed to find bugs, that doesn't mean they don't have bugs themselves. These tools are doing a lot of complex analyses behind the scenes and don't always provide consistent results in every possible scenario.

In a future blog post, we'll look at some sample dashboards so that you get a sense for what can be used for reporting and how these data points can be used.

Technical Debt And Static Analysis

2010-10-24T13:51:00.000-07:00

CAST Software, a maker of software analysis and measurement solutions published some interesting findings on the quality and security of software applications across a number of different industries and applications. (75 organizations, 288 custom applications).

While clearly, vendor funded need to be taken with a grain of salt (for instance the calculation was based on their tool's defect density and a fix rate presumably from their experience), the approach and findings do cast an interesting light on the quantification of quality initiatives. To start with, they define "technical debt" as "cost of the effort required to fix problems that remain in the code when an application is released to operation. Like financial debt, Technical Debt incurs interest in the form of the extra effort it takes to maintain and enhance an application due to the structural quality flaws left in that code."

In summary, some of the findings are:

The average technical debt was found to be about $1M with the average code size being 374,000 lines of code.
The average maintenance cost per line of code was $2.82 assuming a $75/hour average, fully loaded wage for a developer and that a bug takes about an hour to fix.
C/C++ applications had wider variability but on average had lower technical debt that some of the Web technologies such as Java and .NET.

My assumption though is that this number was calculated from the defects that came from the Cast tool that would presume to be fixed and therefore is a subset of the actual technical debt for an application (presumably there are many other bugs that would need to be fixed, not found by a static tool). If true, I would conclude that they are saying that if you use the Cast tool to the level they suggest, that it would save $1M in technical debt, an amount that over time would grow larger due to interest.

One could argue either way whether the number is bigger or smaller. On the one hand, there is a cost to fixing, starting from the license cost all the way to the extra hardware, training and usage cost. On the other hand, the calculation of technical debt was only done based on developer time and not on the effect of a defect, which can be many multiple times more costly than simply the time it takes to fix a bug. We sometimes call these the "million dollar bug" and it varies significantly among industries and application types. While this cost may not technically be a part of the technical debt calculation, you may wish to figure the cost of a defect in deciding whether to use static analysis or not. Sure, these defects are few and far between but if you can find just one...

Either way, static analysis does appear to save money and help developers produce better quality software - the hard part has been in measuring it. Technical debt is one interesting way to measure it.

Why Have Static Analysis Tools Not Yet Hit the Mainstream

2010-10-08T23:16:00.000-07:00

Over the past few weeks, there has been an interesting and lively discussion going on in a Linkedin group focused on static analysis. The original question was along the lines of "Why hasn't static analysis penetrated the market further?" The conversation has reached 97 comments and counting from developers, managers and vendors alike.

One could argue that static analysis is actually fairly pervasive, particularly among companies who produce shipped software as their business and where quality and security is critical. If you count the organizations who use static analysis (and ignore some overlap) it would number well into the thousands. In addition, opensource tools like Findbugs and others have a decent following as do free static analysis tools bundled with IDE's and compilers. And yet, static analysis still has a way to go before it becomes a part of the standard toolchain that most every organization uses. Getting static analysis used effectively within the organization is a second order challenge once an organization decides to use static analysis.

I will attempt to summarize some of the interesting points that were surfaced in the sometimes heated discussion. Some I agree with and some I don't.

Static analysis tools can be noisy - producing false positives. Does static analysis consistently produce real issues? Are the issues reported comprehensible and actionable?
Static analysis tools were overhyped many years ago causing expectations to be set too high.
Developers are protective of their code and don't like a tool poking at their "treasured creations".
Static analysis vendors are still small or medium-sized and therefore don't yet have the marketing budgets to communicate to the broad market. HP and IBM's recent acquisitions of Fortify Software and Ounce Labs may change this at least in the security space.
Static analysis is often viewed as as an additional step after a "successful build" when it should be release criteria for passing or failure.
The term "static analysis" is boring and doesn't reflect the sophistication of the analysis
A multitude of defects are reported and developers don't know where to start or how to create a real process that addresses the issues
Ignorance of the value of tools in the software development process. The all too frequent way that managers and senior staff treat software as an art that cannot be improved through better process and tools.
Need a clearer demonstration of the value static analysis provides (the benefit minus the costs).

Static analysis is growing particularly among organizations where quality issues and security vulnerabilities are most damaging but it will take a combination of improvements to the technology, changes to the attitude of development organizations and industry accepted best practices to take static analysis past the chasm towards the mainstream. On the services front, we help many organizations overcome these issues through best practices in people/process/technology, mentoring, training, support and a multitude of other services. When done right, organizations who take on static analysis do gain a competitive advantage. These companies not only improve the quality and security of their products, they do so often with an overall productivity gain. Invest in the right way and get some payback. Approach it with the wrong steps and a complacent attitude and then you are left to cross that chasm yourself.

False Positives versus Intentional's

2010-09-24T10:49:00.000-07:00

Bugs need to be categorized so that the appropriate action can be taken. The traditional bug reported by customers, support, field engineers, etc. are examined by developers to determine if it is a bug. If it is a bug, its priority and severity are usually assessed. If it is not a valid bug report it may have been rejected as not a bug, unfixable, already fixed, duplicate or any of a number of different categories that say that no further work will be done on it.

Static analysis defects have a slightly different flavor. Static analysis reports potential defects, meaning an inconsistency or flaw has been detected that needs to be verified by a developer or review team and handled appropriately. Bugs determined to be real ("true positives") need to be assessed for priority - usually a combination of type of bug reported, whether it is on a specific conditional path/exception path, what part of the code it is found in (new code/old code/component/3rd party library), etc. Static analysis solutions may also include a "score" to give further information to help prioritize it.

Some static analysis results however may be "false positives" which are essentially "not a bug" characterizations. Herein lies a distinction. Vendors often like to make a distinction between a "false postive" and an "intentional" status. A "false positive" signals a situation where the analysis incorrectly identified a problem as a bug. If you will, it's a bug or limitation of the static analysis tool. An "intentional" status signals a situation where a valid bug is reported but that the developer had intentionally coded it this way and so no further action is required. The presumption here is that for "intentional's", the analysis did what it was supposed to do correctly.

Some organizations lump everything that they don't want to fix into the "false positive" category. If they weren't going to change any code, then it shouldn't have been reported. Other organizations take a more sophisticated approach by making this distinction - the presumption being that intentional's should be reviewed, like a sanity check. A variety of reasons why they categorize it as such include:

wanting to get an accurate "false positive rate" for the tool. Lumping "intentional's" artificially increases the false positive rate
the workflow for "intentional" and "false positive" may be different. While both should be reviewed/audited, a static analysis administrator may review "false positives" for tuning and configuration opportunities. False positives should be reported back to the vendor for fixing as well. "Intentional's" may be reviewed by architects to understand what strange coding practices are taking place. Intentional's typically need a good reason to explain why it was coded in the way it was.

Of course the advantage to finer grain categorizations have to be weighed with the cost of explaining the distinctions. There is a cost to data acquisition, but if you value the ability to measure the tool's effectiveness and have a process for improving your analysis and improving your code through examination of intentional's then it may be well worth it.

Power of 10 for Safety Critical Code

2010-08-17T11:01:00.000-07:00

Just reminding programmers of a good set of safety critical programming rules from the rocket scientists at JPL/NASA. Gerard Holzmann wrote the Power of 10 rules which are a set of good programming practices that should be considered for programs that demand high levels of quality.

The rule that most applies to static analysis is "Rule 10" which is noted below:

"Compile with all warnings enabled, in pedantic mode, and use one or more modern static source code analyzers. All code must be compiled, from the first day of development, with all compiler warnings enabled at the compiler's most pedantic setting. All code must compile with these setting without warnings. All code must be checked on each build with at least one, but preferably more than one, state-of-the-art static source code analyzer and should pass the analyses with zero warnings." (Rule 10 of Power of 10 Rules)

Gerard notes that "there is no excuse for any serious software development effort not to make use of this technology." Some may view the "rule of zero warnings" as draconian but Gerard notes that if the analysis gets confused enough to report a problem, then the code should be rewritten to be "trivially valid." Good code doesn't need to be complex. If the code is clean, it's not only much less error-prone but also much easier to manage and maintain ongoing. Think also about the next developer who may be inheriting the code. Static analysis will find bugs but will also help code be more readable and maintainable.

Dr. Dobbs Article on Porting to 64-bit Platforms

2010-08-06T14:32:00.000-07:00

Irving Rabin, Senior Consultant at Code Integrity Solutions recently published an article on porting to 64-bit platforms.

You can read it on Dr. Dobbs at: Porting to 64-bit Platforms.

Enjoy and have a great weekend!

Getting Buy-In For Static Analysis From Management.

2010-08-02T12:03:00.000-07:00

I recently read an interesting article (8-Ways To Get Buy In From Management) on how to get buy-in from IT Management. While the document is mostly focused on IT related activities, there are some valuable lessons that can be learned for any technical undertaking within an organization. In this entry, we discuss some of the issues focused on static source code analysis.

1. Learn what management values.

Software development management has a different set of concerns than most individual contributors have. Clearly there should be alignment between management and worker-bees but that is too often not the case. Static source code analysis will help you find bugs so that developers can fix them early in the development process. From management's perspective, static analysis is almost a no-brainer. Caper Jones' estimates that static analysis can have a defect removal efficiency that can top 95% (presumably in a well configured, effective system and process). Other ways to describe the benefits include:

Fewer defects and vulnerabilities in the field, meaning lowered support costs, fewer patches and better customer satisfaction
Increased developer productivity
Lower risk of late releases

Particularly for technology companies, arming developers with good tools so that they increase their effectiveness is a must. How management couches the benefits is industry and company dependent.

2. Propose projects in management terms, not technical terms.

The benefits in #1 are real, but how tangible are they? The more the benefits can be quantified, the easier it is to make a decision whether to institute static analysis or not. A convenient way to break down benefits is to look at increases in opportunities, lowering of costs and reduction in risk. Static analysis typically does not increase opportunities directly but an argument can be made that any reduction in development time enables more value-added features to be built. Lowered costs are definitely significant. Lowered support costs (in technical support, developer support (patches, etc.)) as well as increased productivity can be estimated. Many companies use automated source code analysis to find problems quickly in an fast-paced Agile environment. This enables increased throughput overall. Reduction in risk can take many forms - the risk of late release, the risk of poor quality, the risk of lowered customer satisfaction, the risk of failing compliance etc.

3. Show management case studies and examples that support your point for cost savings or revenue enhancement.

Certainly leverage the vendor. While they are not impartial, they often have good case studies. References are also very important. A lot of companies are using static analysis so leverage your contacts. Better always to do backdoor references so you can get a balanced view. People seem happy to discuss their experiences.

4. Find ways to minimize the project's expense.

While reducing license cost is a subject better left for closed door negotiations, there are ways to maximize the benefit you get from static analysis tools. Make sure you take the time to tune it for your needs, integrate it into your process (both technically and organizationally) and try to tie metrics of value to release criteria. Another way to get more value is to use the tool's extensibility capabilities to build your own custom checkers so that you leverage the analysis engine and the place in the workflow. Wouldn't it be great to find and fix additional problems, violations in coding standards and more borrowing from what you are already doing. When more people use the tool, more value is derived. To get more people to use the tool, make it super easy to use, make the results as relevant as possible and make it clear as possible what is expected of all participants.

5. If your department's IT budget isn’t big enough, get help from another department.

A lot of folks can benefit from static analysis. We've seen static analysis budgets come from not only the development organization but also from quality assurance, central tools, security groups and even governance organizations. While they are not users, they can clearly benefit from the results and may help fund its usage. Static analysis can also come from other overarching budgets such as from a quality and security initiative or projects related to moving to Agile / continuous integration and software development infrastructure automation.

6. Break up the project into easy-to-swallow pieces.

Breaking up projects into pieces not only makes it easier to swallow, it reduces risk. As you see success in the early projects, it gives you knowledge (and confidence) to continue with the other phases of the project. Start with a pilot group and see how that goes. Set clear metrics for what is considered success for that pilot and work to meet it. You'll learn significantly from the project to help you make better decisions down the line. Use of static analysis as an integral part of a software development process is relatively new and thus haven't been ingrained into the DNA of most people and organizations. That's changing but we're not quite there yet.

7. Look for ways to outsource part of the project.

This advice may be a bit self-serving, but I will mention that it never ceases to amaze me how many companies opt to do everything in-house. The short term benefit from doing this is that organizations build some expertise in-house while not incrementally add budget. But, they rarely get a best practices implementation. In addition, they have to give up some opportunity cost of having that developer take on a nonstrategic project versus something more relevant to the goals of the company. In addition, by doing it all in-house it often takes a lot longer for them to get it done. All too often, we see engineers struggle to do it in-house only to never really do a good job, never complete it and waste significant amounts of time doing so. Outsource for expertise, outsource for opportunity cost, outsource to save a considerable amount of time.

8. Pick your battles carefully.

Static tools are not cheap -- not only to buy but also to effectively put into one processes. The evaluation process for many static tools give you a good sense for what it can do for you so take full advantage of it. If you need to, start small. De-risk the project as best you can through prototypes and smaller rollouts. Bringing in static analysis does require a commitment but pays off big-time when done correctly. By its very nature, static analysis is proactive. If you don't get funding the first time around, it shouldn't take long for the next emergency to make the situation more "favorable"for bringing in tools to more efficiently improve quality and security.

Detecting Endian Issues With Static Analysis

2010-07-20T16:43:00.000-07:00

Our own Carl Ek, architect at Code Integrity Solutions had his article on detecting Endian issues published in the esteemed Dr. Dobbs Journal. Here is the first paragraph and link:

"There are 0010 0000 kinds of people in the world: Those that understand the difference between Big Endian and Little Endian, and those that do not."

Since all binary processors (hardware or software) have an endian design, correct processing of the data based on that endian design is extremely important. The statement above is a version of another joke, but with a twist: the binary is represented in little endian, giving some mild humor for those that understand. For those that don't understand endianness, the humor is lost, much like a processor which has an endian processing defect. In this article, I describe the kinds of defects which occur, and methods where static analysis tools can help detect programming errors and enforce correct programming.

Detecting Endian Issues With Static Analysis

What Should You Be Good At?

2010-07-13T17:01:00.000-07:00

Any reasonably powerful software tool has enough complexity and flexibility to handle a multitude of scenarios. Having a lot of knobs and buttons are essential in the software development arena where environments are typically comprised of stitched together home-grown and commercial tools. Rarely is the whole toolchain packaged and nicely integrated. The ability for development tools to be customized means that it is highly possible to solve your business problem using best of breed technologies. It also doesn't hurt that engineering teams are highly capable of building most anything. However, the path to achieving that goal through in-house resources is often difficult and oftentimes impractical.

Any department, particularly the engineering department, must recognize that they must have a set of core competencies that are strategically essential for their company to succeed. Providers of software are in a highly competitive environment where differentiation is critical. Not surprisingly, these core competencies are the areas that must be focused on, cultivated and maintained from an engineering standpoint. Some examples of core competencies are the ability to:

produce particularly high performance, highly scalable transaction systems
add relevant features quickly to be first to market
consistently deliver the most reliable systems
deliver the most elegantly designed and easy to use systems

Successful companies deliver on the levers that matter the most in a competitive industry. It could be feature leadership, process leadership, user experience leadership or likely a combination of things. The ability to leverage your core competencies comes from having the right talent and the right environment to help them succeed.

Where are your levers? Where should you focus your team's efforts? What competencies should you build and what should you outsource? Where should you invest and by how much? Should your team be experts at building features that matter or having expertise in software development infrastructure?

In the most extreme case, one could argue that you should outsource everything that is not within your core competence. The benefit is that you can focus on being best of breed in the areas that matter to your business while getting best of breed from outside providers in the areas that matter less. In the other extreme, you can do everything in-house, the "not invented here" syndrome. The benefit is that you have control but the downside is that your focus is diluted and your costs greatly increase as your needs change. Culturally, engineers have tended towards the "build it in-house" approach because they can.

With regards to static analysis, how much of software development infrastructure is truly strategic to your company? How much can and should be outsourced to some outside help? Where can you get the most leverage from your team's resources to focus on the areas that will make your software development organization a leader in the industry?

Static Analysis Typically Results in 85% Defect Removal Efficiency

2010-07-09T10:12:00.000-07:00

Caper Jones put together a nice article titled "Getting Software Quality Right." He makes an important case for static analysis helping to increase the defect removal efficiency. He also notes:

Combining inspections, static analysis, and testing is cheaper than testing by itself and leads to much better defect removal efficiency levels. In concert, these approaches also shorten development schedules by more than 45% because, when testing starts after inspections, almost 85% of the defects already will have been addressed.

The range for defect removal efficiency for static analysis is listed from 55% - 90% depending upon how correctly it is used. It's a wide range and most likely highly dependent upon whether best practices are implemented or not. Clearly a tool can be easily misapplied both from a process and product implementation perspective.

One Easy Way to Improve the Quality of Third Party Components

2010-06-30T15:52:00.000-07:00

Software products these day are increasingly made up of components from a variety of different sources. These components provided by the open source community, supplier companies and software vendors provide functionality for user interface management, operating system access, security interaction, connections to device and more. In addition, software developers also use code generation tools that enable developers to manage the application at a higher abstraction.

By using third party components, an application with much more functionality can be pieced together much more quickly than ever before.

Yet the customer who receives the product, be it a smartphone, router, medical device, software application certainly doesn't care how it's developed - only that it works. Developers increasingly must be able to manage not just the code that they've written but also the code that they integrate into the overall solution. Although it is in one sense, out of their sphere of control, they must manage the integration of the code so that the overall solution has the needed quality, security, performance, reliability and more that is expected of the finished product. Managing the interfaces well in some cases is not enough -- many developers must opt to own the source code so that they can make the needed modifications to make it work for their needs.

We work with many companies who run into this situation. They analyze source code using static analysis on their codebase to find all sorts of problems to fix. In order for the source code analysis to work better, they also analyze the third party code that is bundled with their end product. This is an important step because many bugs stem from the improper interaction of components - and without analyzing the whole program you either miss out on a lot of bugs or get an inordinate number of false positives depending upon the static analyzer's conservative or aggressive assumptions. Clearly, defects that result from the direct interaction with these third party components should be fixed. This is something within the control of developers. Defects found in the source code of the third party code should also be fixed although many developers find themselves powerless to change the vendor's code or realize that they must increase their overhead by making changes to the source code and managing that change for future versions. It's extra work.

But the developer has some leverage. We have worked with many forward thinking companies who lean on their suppliers to do their own code analysis, and to fix up the problems. For those companies with extra leverage, they have even demanded their suppliers to provide clean code as an acceptance criteria. These companies take a whole product view of the code, realizing that a bug in a third party component doesn't make a bit of difference to an end customer who's system or application is broken.

So if you do run an analysis and find problems in third party code, put the burden on the supplier to fix up their code. It's very reasonable for a company to run a quick analysis and to put a concerted effort to fix the problems (although they will have to obtain their own license). You may be surprised at how much leverage you have. And demanding good, clean code is something in everyone's interest. It will save you a significant amount of time downstream and more importantly provide the quality experience that your users expect.

Defect Database Bugs Versus Static Source Code Analysis Bugs

2010-05-24T12:29:00.000-07:00

Static analysis has been around for quite some time. Recently they've been pointed to the very practical application of finding bugs in source code. Many modern tools do a very good job of uncovering very good bugs -- the big value of which that the tools find them early in the development cycle. Because the bugs are found in code, the reports point specifically to where in the source code the problem manifests. Easy to find, easy to fix.

But there are some subtle differences that can greatly affect the expectations that users have of the tool and the process workflow that is needed to support these tools well.

A Dose of Reality
Contrary to what every developer desires, static source code analysis tools do not ONLY find critical, crash-causing defects. Like any other discovery or analysis tool, it's going to find a mix of great stuff, good stuff, so-so stuff and stuff that's plain wrong. In industry parlance, these would be problems categorized as critical, high, medium, low priorities and "false positives." In static analysis, there's even an additional state commonly used called "intentional" which is that the tool correctly found something that looks like a problem but that the programmer intentionally coded it that way (usually due to an assumption that the static analysis tool is not privy to - such as assumptions in the environment that the code operates in). Modern tools have improved their analysis significantly to reduce the lower value stuff and raise the likelihood of uncovering higher value stuff.

That being said, every organization has a different definition of priority but for the vast majority of software development organizations, priorities are mapped to a stage in the process. Often, priorities such as "critical" and "high" are mapped to specific requirements - such as part of an acceptance criteria for release. Mediums and lows are relegated to "if you have time, fix it" or "we'll never get to it." That's fine - rarely does an organization have the resources to address all issues (safety critical code being the notable exception). Defects reported by static analysis tools should be treated no differently. Some are important to fix and others are less important to fix. When we go over static analysis defects with groups of developers, you'll invariably have the whole room resoundingly agree that a particular segment of code is poorly written and "should be rewritten." However, resource constraints don't always make that the best business decision. Most developers know that the regular low priority defects in their bug tracking database should be fixed, but will likely never be touched again. Everyone wants to fix them but nobody has the time. Lower priority results from static analysis tools fit that too, with the only caveat that the chance for opportunistic fixes is greater because they are discovered during coding.

When introducing static analysis tools to developers, they should be of the mindset that it's not going to find every critical problem and that the problems it does find are not all going to be critical. These are unrealistic expectations for any tool.

Checks and Balances
Another key difference with static analysis defect handling is that they are being reported by a tool and not by a customer, support or QA. Thus the decision process for prioritization is not handled by an "outside" party to engineering. Some organizations let the developers decide on defects which is okay as long as developers are properly trained and all have the same minimum standard for what is acceptable. Many developers without the right expectations and training end up marking bugs incorrectly as false positives or perform unnatural acts to make the "tool shut up". To mitigate these problems, organizations can set up a different review/prioritization team or have an audit process on the tail end to ensure accuracy and build learning into the organization.

Making Lemonade Out of Lemons
One might think that a "false positive" or "intentional" is a terrible waste of time and needs to go straight to the wastebucket without further ado. However, these reports do provide information that can be useful to the organization:

False positives show when the tool did not correctly report a problem. This is often a case where tuning the analysis can improve the results. By configuring the analysis to understand your code better, you can significantly lower false positives and increase the valid bug count for future analyses. In this case, some false positives should be assigned to a static analysis expert to retune the analysis to improve the results. Fixing false positives in this way is significantly more efficient than having developers wade through incorrect results. False positive "fatigue" is a frequent cause for miscategorization as well.
False positives often are a signal of poor coding practices. If the tool is confused by the code, then it is more likely that the code may be confusing to a new developer inheriting the code. False positives tend to show around code that could be improved through refactoring.
Intentional's are also an interesting area to look. The most common reason for dismissing a bug report as intentional is that the tool did not have access to an assumption. Of course, the root of many errors is a changed assumption which causes problems to occur in the future. Intentional's often signal an opportunity for defensive coding.

Process and Alignment
Static analysis tools often come with their own defect tracking information, which is usually quite useful because they also include a browser which helps developers more quickly debug a report. And yet, having another database and UI is extra work.

Borrowing as much of the already existing bug workflow is key. Developers don't want to learn an additional process on top of their already busy day. Borrow what you can, recognize the differences and institute unobtrusive additional processes to handle them. Every solution we've instituted follow the same general principles but have varying levels of implementation because everyone's process is different.

Offshoring Quality in Static Source Code Analysis

2010-05-06T10:03:00.000-07:00

Companies who buy static source code analysis solutions will invariably run into the problem of the backlog. After all, they already have a mountain of working code, accumulated from decades to hundreds of person-years of development. There are bugs in the code and a static source code analysis tool will dutifully march through the code and find all sorts of potential problems.

At Code Integrity Solutions, we've seen backlogs as small as a few hundred on a small codebase to over 50,000 on the largest codebases in the world. Average backlogs are in the thousands. Getting through this backlog takes time, effort and skill. The rewards are usually worthwhile because within this backlog are usually some very good showstoppers. And yet software development teams are always short on resources. Rarely will a product owner stop all development to through the backlog to find those gems hidden among the other high, medium, low and false positive results.

Oftentimes, we see the engineering management suggest, "why don't we have our offshore team do it." The offshore team may already be charged with maintenance and so it seems a logical group to address a big backlog of static analysis defects. It certainly is a cost effective solution and it will make the backlog go away. Problem neatly solved and management is happy.

Of course, successful development through offshore teams is a discipline all unto its own. We won't repeat any of that here. We discuss here the static analysis flavors of these challenges below. These problems exist in any team but are amplified in an offshore environment.

The backlog must be prioritized. The question is who should own the prioritization? Who has a vested interest in quality that you trust to make that decision? The backlog will go away but will it be done in a way that results in significantly improved quality?
Teams are heterogeneous and thus the quality of the prioritization and the quality of the fix will be heterogeneous
Configuration and optimization almost always beats brute force. If a static source code analysis tool misunderstands a construct in your code, this can result in a cascade of false positives. Oftentimes a quick configuration change can wipe away hundreds to thousands of defects saving time (now and in the future) and making the analysis better understand the codebase.

What can result from mismanagement is poor results done inefficiently. Falsely identifying issues, fixing problems that aren't real, and plowing through report after report inefficiently can result in wasted dollars, missed opportunities to improve the quality and tensions between offshore and on-shore teams. Keep in mind that when the future showstopper invariably hits, management will ask, "was this problem found by our static analysis tool and could we have avoided it if we had done it right the first time."

Any good manager knows that they need to provide the right support, infrastructure and structure to make their team successful. For example:

Set good standards for diagnosis of issues. Everyone needs to understand what a critical, high, medium, low, etc. categorization is. Some prioritize based on checker types (e.g. memory leaks, concurrency, etc.). Some prioritize based on whether the bug is found in a critical path, conditional path or in an exception path. Some prioritize based on what part of the code it is found in. Some use the built-in prioritization mechanisms of the static tool. Most use a combination.
Train the team. We found mentoring the team through real bugs is the best way to get everyone on the same page. Expectation setting is critical here. These sessions help gain consistency in results. In addition, finding the "ah-ha" moments gets the team jazzed and energized to do good.
Set up a trust but verify model where results are audited. Use these audits as learning opportunities to steer the team in the right direction and finetune further. We've audited results and identified many opportunities to improve standards, to train the team appropriately on certain topics and to open up discussion on how best to handle certain situations.
Have an expert on hand who is aggressively seeking ways to tune the analysis. It is much more efficient than brute-force going through the backlog and it set ups the analysis to be much more accurate in the future. A simple configuration change can save weeks of effort. Paul Anderson, the VP of Engineering at Grammatech recently did a presentation at the ESC Show in San Jose that suggested that the more false positives there are the more exponentially higher misdiagnoses there are.

Of course the challenge of instituting a successful static source code analysis process takes a lot more than following a few bullet points. If you decide to go off-shore instead of outsourced or doing it with your on-shore team, it can be successful. With the right effort and expertise, you can effectively kill the backlog, improving your quality without having to stop everything else in its tracks.

Cramming More Value Into the Static Analysis Workflow

2010-04-29T20:01:00.000-07:00

Static analysis tools find bugs in source code during the earliest part of development. Where in the process static analysis is used varies widely, depending upon the software development organization's specific goals and the environment in which it will operate in. Some organizations run static analysis on entire codebases just before release. Others run static analysis on a nightly basis or several times a day during a continuous integration process. Still others run static analysis locally on a developer's desktop so that they can find and fix problems on the snippet of code they are working on.

One can make a distinction of when static analysis MUST be run versus when it is CONVENIENT to be run. Organizations are increasingly using the results of static analysis as part of the acceptance criteria for a codebase. Meeting acceptance criteria may be just before release, before collapsing a development branch into the main line or even before a developer checks in code. It makes a lot of sense to ensure that all reported issues have been addressed as part of graduating the code from one level to the next.

Of course, we all know it's a good practice to prepare well in advance of a future requirement. By providing static analysis throughout the development process, you eliminate the situation where you're two days before release and you run the static analysis tool and find that you have hundreds of problems yet to be addressed. Clearly, fixing bugs as you develop will improve the quality and security of the software early, letting you more comfortably coast into and through your acceptance process.

What else should be in your acceptance process? The static analysis workflow is a convenient place to find and fix other problems other than just bugs. Here are some additional things that static analysis can help you enforce, conveniently borrowing the same workflow.

Industry coding standards such as MISRA, Scott Meyers Effective C++
Government compliance standards such as DO-178B, PCI and FDA
Internal coding standards
Architectural coding standards

Using the same workflow makes it easier and efficient for developers and administrators. Tools are starting to combine their capabilities to help software development organizations solve bigger problems. One of the major vendors in the static analysis space recently announced the support for MISRA. Many vendors offer the ability to author custom checks to find and fix a wide range of problems. They provide a huge benefit because these checks can find problems across all codebases in the company and can be modified to meet the changing needs of the organization. Custom checks can be written to enforce coding standards, architectural requirements and find defect categories that are particularly important to your company. Custom checking can also serve as a complement to code review.

So leverage existing workflows by adding other types of checking during the right points in the development process. And then arm the developers with the ability to meet their goals easily. The more you can push checking up front, the more likely you'll meet your time pressure goals.

Lowering Developer Resistance to Static Analysis Tools

2010-04-22T14:08:00.000-07:00

Imagine a team of software engineers who embraces the software development process. They dutifully create their unit tests before writing a line of code, run and fix all necessary static analysis and dynamic tests prior to check-in, fix all compiler warnings, participate actively in code review. They write code that is brilliant, yet easy to understand and maintain. They write unbrittle code that can withstand the inevitable changes to the assumptions that surround the code. Everything of course is well documented.

They realize that their code is bigger than themselves -- that it has to work as part of a larger ecosystem of code created by engineers from around the world. They realize also that their code may last for decades and may change ownership several times. They realize that taking proactive efforts to make good code mean less reaction down the road. They welcome new process that will help them write better code overall even if it means a little more work upfront.

Most developers won't disagree with these ideals, however, when instituting process that is "good for us to do", they bristle at, argue with and/or ignore the change. It's not their fault. Developers are often set up to fail at following the process right from the beginning.

With every new change comes an investment in time and energy to make it happen. If you were asked to add a new step in your development process, you have to make an investment (sometimes sizable) at the beginning to understanding and instituting the change and subsequent investments every time you perform that new added step.

We all know that developers are under immense pressure to deliver more for less. They simply aren't given enough time to make the upfront and ongoing investments. Routinely asking engineers to do feature X in 2 days instead of 3 days will result in cries of "foul" every time something might get in the way of their already oversubscribed status quo. Clearly the managers and project managers must understand and manage properly the time pressures.

In addition, the upfront and ongoing investments in time and energy can be significantly reduced. When instituting static analysis tools at various companies, we've found the following to be helpful:

Training can be a huge time saver. But not all people learn the same way. Some like it fed to them and some like to go and get it. We've found that taking a multi-pronged approach, by offering training, group and individual mentoring, a centralized resource with good, simple step-by-step instructions, a helpdesk, a forum/wiki and more get just about everyone onboard.
Package it well. Developers can figure out just about anything technical. But, it doesn't mean they have to. Having to fiddle with getting a tool installed may provide some individual technical satisfaction but doesn't mean they've done anything useful towards meeting a business goal. Multiply that by how many developers you have and you've created a large time sink. You'll also lose support by developers in this way by making it hard to use.
Keep the standard workflow simple. We worked with one organization which gave every developer the standard training and then washed their hands of it. After having thrown it over to the developers, it was no wonder that a year later hardly anyone was using it. One of the biggest problems was the tool was given to the developer with no workflow and no purpose. No queries were set up. No screens were customized. The developers had to suddenly become experts at using the tool -- a foolish use of time given tool usage was certainly not a core competence of the organization. Spoon feeding a simple, easy-to-follow process doesn't mean you think your developers are dumb. It means that they get to spend less time wrestling with the tool and more time doing what they are paid to do - write great code that solves the company's mission. Again, without the right process, you'll lose support.
Automation is key. Having multiple steps in the process across multiple tools in the toolchain mean added time, human error and frustration. Integrating static analysis with bug tracking, source control management, LDAP, policy databases, build scripts, etc. lowers the ongoing investment.
Another area ripe for improvement is performance. We've run into codebases where the analysis takes 30 hours or more. Static analysis has to analyze a lot of stuff but there are ways to improve the performance dramatically. We've tuned some analyses down to minutes depending upon the deployment model. This significantly lowers the "cost" and opens up new possibilities for when and where analysis can be performed
One of the biggest complaints you'll hear from developers is that the results have false positives - meaning results that are incorrectly noted as issues. Every static analysis tool will generate false positives but the more the tool understands your code (usually through configuration), the more it will produce better results. Every time a developer has to examine an issue that is eventually is false is wasted time (although some will argue that not all of the time is wasted because false positives can signal poorly written code). Lower this cost and raise the value of the tool through tuning.
If you aren't a developer get them involved. There's nothing like having a champion or two help you get buy-in. Involve them in defining the process - after all, nobody knows the process better than the developers themselves. Developers have to see and believe in the true value, not the inflated value. Do this by running mentoring sessions where everyone can plainly see the value and the limitations of the tool. Set expectations all around accordingly.

In concept, static analysis tools are simple to use. In addition, your developers are smart, so it's tempting to just give them the tools and let them figure out the value. It's a given your developers have too many features to deliver in too short a time. You need to lower the cost of adding static analysis to their workflow. You'll lower resistance and improve their efficiency while improving the overall quality and security of the code.

Two Views of Static Analysis

2010-04-11T08:33:00.000-07:00

Engineers have differing expectations of static source code analysis tools. Opinions of its usefulness vary widely but generally fall into one of two extreme camps.

On the one hand, some engineers think that program analysis couldn't possibly produce anything worthwhile. They believe all the results are useless and not worth the effort to examine. They may have had experience in the past with products such as Lint, which produced copious amounts of “useless” warnings. These “naysayer” engineers will rarely give static analysis a real chance and instead avoid using it, or worse yet, dismiss every result as a “false positive” or “never going to happen”.

Conversely, some engineers have the unrealistic expectation that static analysis will only produce valid, critical defects that must be fixed. These “naive” engineers may not have a good grounding of how static analysis technologies work and what its limitations are. They may equate static analysis tools with dynamic tools, which almost always report real bugs but lack the coverage and ease of static tools. These engineers will very quickly be disappointed. Static analysis finds potential problems based on sophisticated algorithms. These algorithms operate on source code and build-level information and may not have access to all of the run-time assumptions. Highly complex or very custom code can also cause the tool to misfire unless the tool has been properly tuned. Naive engineers may also "fix" issues that don't really need fixing on blind faith or to quiet the tool. Attempted bug fixes can cause other areas to break.

Nearly every organization has these polar extremes and everything in between. Naysayers cause others to lose confidence in the tool. The naive engineers may lose credibility by overstating the tool’s capability. Somewhere in between is the truth -- that static analysis produces good and useful but not perfect results. There are going to be good bugs you really want to fix. There are also going to be low priority issues that you don’t really need to fix (similarly, very few organizations fix even a quarter of the bugs logged in their bug tracking system). Other issues will be just plainly wrong (a false positive) and signals that the tool couldn’t understand that part of the code well or that the analysis needs to be configured.

Engineers need to understand not just what the tool can do, but what it can’t do as well. When an engineer is trying to interpret a result, it helps to understand what the tool did to produce that result. There are many strategies for getting a group of engineers to understand the range of value a static tool provides. Training, documentation and process changes are necessary but not every engineer learns by reading a manual or attending training class. One of the most effective ways for engineers to learn is to do some joint triaging of results. For Code Integrity Solutions, we’ve ran very successful sessions by simply going through bugs with a group of engineers in the room. We discuss a handpicked set of bugs, debate them and resolve them together. These sessions (even as short as an hour) enable everyone to come to a general conclusion around a common set of results and start the basis for a standard by which to triage defects. Senior engineers shine in these types of environments because they not only quickly understand the tool's capabilities but also are able to impart their standards to the less experienced team members.

We often hear these points made:
* This is not really a bug but this is poorly written code and needs to be redone
* Ah, I see why the tool is reporting this incorrectly. All I have to do is make a mod to the analysis to fix that
* I wouldn't have thought that it was a bug. I now see why it is after taking a deeper look.
* If we change this assumption slightly, then this would become a bug. For defensive coding reasons, we should change it, especially since we occasionally change ownership of code.
* I understand why this tool is not catching this type of problem. I need to use a different method for finding these types of problems.

Invariably the naysayer comes to understand the value the tool can provide for him or her and the rest of the organization. By also showing false positives, the naive engineers can see any limitations of the tool. For everybody involved, they see the nuances of the tool and the level of effort required for effectively evaluating results. A few hours of time makes for weeks of productivity gains.

The False False Positive

2010-03-20T15:00:00.000-07:00

Static analysis tools report potential bugs in source code by analyzing the structure of the code for inconsistencies and flaws. Sometimes they get it right and sometimes they get it wrong depending upon how strong the analysis is and how complex the code may be. These days, static analysis tools are getting more and more sophisticated, doing statistical analysis and interprocedural analysis (where information is tracked across functions) across all the logical paths (sometime numbering many millions) in the code. Dataflow analysis can track values across functions to produce bug reports that span multiple levels of calls.

While increased sophistication means that static analysis tools can catch more problems with a higher degree of accuracy, the burden increases on the reviewer of the results to interpret them correctly. If you were grep'ing through some code for something you can quickly review (and dismiss) many of the results because you understand what your "analysis" is doing. With static source code analysis, this is much less apparent.

We see many engineers look at a complex bug report and not take the necessary time to understand the problem and fix it. This is mostly because they don't understand what the static analysis tool is doing and how deep it is analyzing the code. The result is a real bug being marked as a false positive - or a "false false positive" if you will. These bugs then disappear off the queue never to be seen again - a lost opportunity.

How do you minimize the number of false false positives? Lots of different ways:

Training and/or mentoring of the reviewer team helps the team see how to properly disposition a defect
Auditing of false positives either periodically or all-the-time. Incorrect markings are great learning opportunities to correct incorrect usage.
Metrics that monitor the general false positive rate so that individuals can be benchmarked against them to detect anomalies. If one engineer is marking 80% of the defects as false positives versus a general 30% average, then this is likely a correctable problem.
Have double reviews of defects. Similar to pair programming make sure each disposition has "two pairs of eyes" on it

By making triaging more accurate you get much more value from your tool and minimize the incorrect throwaway.

Does One Size Fit All With Static Source Code Analysis?

2010-03-08T12:32:00.000-08:00

If you take a look at all the software written in this world you'd find a vastly heterogeneous set of applications. The list of these applications include medical devices, military/aerospace, banking, games, network telecommunications, automotive, gaming, accounting, wireless and much much more.

As to the importance of quality, security and reliability, even a gaming application can demand good quality and high performance. While a game crashing may be a trivial situation for an individual games customer, the game publisher may be faced with an expensive patch they must rollout or a poor review to hurt their sales. Good software quality and quality is good business.

Static source code analysis is up to the task of helping software developers improve the quality and security of their code in the earliest part of the development process. Finding critical problems in source code is simply easier and faster to fix. However, codebases are very different. They use different libraries, have autogenerated code. They have different memory allocation mechanisms and concurrency and locking mechanisms. They have different internal coding standards by which they adhere to. They support many different platforms. They may come in very different flavors like C, C++, C# and Objective-C some of which bear very little resemblance to the other. Even within C, there are different implementations of the C standard. What's a static analyzer to do?

Static source code analysis does its best job to interpret the code at the cost of potential false positives. Clearly if the static analyzer cannot understand the codebase it has less information to make good decisions. Static source analysis vendors ship out their best configuration but it's a one-size fit all. A medical devices company analyzing 50,000 lines of code for a pacemaker is getting the same settings as a networking company with a 18,000,000 lines of code operating system. What's important for one company may be extremely different from another. For instance, a company who provides accounting software that runs on the desktop could care less about memory leaks as the application is frequently exited and reentered. Counter that with network routing software that must be up 99.999% of the time and may run continuously for years. A crash on an embedded system on an airplane or automobile may be different in priority than to a crash on a web application where automatic failover to a different server is built into the infrastructure.

In addition, the way an application is coded may be different. The application may use standard memory allocators provided by the operating system or the application may use its own proprietary memory management techniques.

The analysis may be more conservative or aggressive than desired. A medical devices company who wants to find every potential bug has a different tolerance for the "noise" than a large gaming application which has to deal with tens of thousands of bugs and only wants to hit the highest priority ones.

One size cannot fit all. Only one version of the software comes from the provider with the same factory settings as everybody else. That same version is used for all industries and even in vendor presales situations as well. It's understandable that this is the case. Vendors understandably must provide the tool with the best possible setting for as many different situations as possible.

But clearly it can't address all situations optimally. With just some smart tuning, the analysis can be much smarter about the specific codebase it is looking at and make the results more relevant and accurate. In some cases, we've been able to tune away thousands of defects with just a single configuration change which saves weeks of collective developer time. Other cases of tuning have resulted in hundreds of more critical bugs being reported that wouldn't have otherwise with the "factory settings." The payoff to good expert tuning is both immediate and long term.

Who's Guarding the Henhouse When Fixing Source Code Analysis Bugs?

2010-02-27T07:23:00.000-08:00

What happens when you have a large development team where each developer is responsible for their own quality? On the face of it, this is what every development organization wants. Developers should feel ownership of their code. However, without the right minimum standards instituted you end up with inconsistency and a false sense of security.

Development organizations are heterogeneous. You have great developers and you have developers that may be well, not so great. You have experienced developers and you have junior developers. One thing is for sure, is that everyone has their own sense for what is sufficient. Training, code review, institution of clear standards are all important ways to raise the bar at least to what is acceptable. This is why a transparent acceptance criteria is so important.

Yet, source code analysis often misses the bar. Many organizations have started to integrate source code analysis into their development process and have put in acceptance criteria gates. They are most commonly, "no new static analysis defects should be introduced" or even the more stringent "no static analysis defects in code." All very good criteria that greatly improve the quality and security of the developed product while also improving developer efficiency.

But many organizations fall down in execution. Developers analyze code, they see the results and fix the bugs. All seems good. But, I've seen in many high end software development organizations, where developers routinely mark reported defects as "false positive" or "intentional" (meaning this is how I intended it to be - not a bug). We find by auditing these defects that they are indeed real bugs, some of them quite nasty.

The developer who is under time pressure just marks these problems away. Other developers, who may have their own biases against source code analysis simply don't take time to understand what the tool is uncovering. Good bugs that could be caught at its earliest point in the development cycle get missed, only to be found in later, more expensive downstream process.

The problem occurs when you have developers deciding on their own criteria. Unlike bugs from a bug database, many organizations let the developers self-prioritize. In doing so, the "bad" developers do whatever they can to shut the tool up and move on. The good developers take the time to understand. Source code analysis is great for good developers but even more needed for bad developers.

What Can Be Done?
Training is useful. Prioritization by a different developer or a review team is important and can be tied to release criteria. Code review can create good learning experiences. Spot-checking and other type of auditing can help catch bugs and present new learning opportunities.

A good way to start is to do a good audit of your "false positives" and "intentional"'s. At Code Integrity Solutions, we provide audits to companies' results so that companies can

get a sense for how pervasive the problem is
catch a few good bugs that they would have missed
identify problem areas for training
identify learning opportunities to help developers know what is acceptable
help provide needed data for coding standards and acceptance criteria

Having the developers take ownership of quality through static analysis is critical to creating a good scalable process. But without the right checks and balances in place you defeat the benefit of improving the areas that need it the most.

Must Read Article on Static Analysis

2010-02-07T08:35:00.000-08:00

Coverity recently published an insider's view to building a static analysis tool. It does a great job of explaining the practical issues in building a world-class tool that is depended upon by so many software development organizations in so many different industries.

If you've ever used Coverity, or any other static analysis tool, you will have a more healthy appreciation of how difficult static analysis can be. It just might explain some headscratchers you may be experiencing. Static analysis can never be perfect, but it can be darn good enough to provide significant value in the software development process.

Handling an Embarrassment of Riches

2010-01-29T09:15:00.000-08:00

A good commercial static analysis tool will probably find more bugs in your company's code than you have time to fix. This sounds like bad news, but take a moment for some attitude adjustment: The bugs were there all along, so this is actually a good thing, albeit embarrassing. But it means that your top priority is prioritizing the bugs—which, unfortunately, is a weak point of these tools.

So tweak the heck out of your settings. There's a lot you can configure, and the vendor’s support staff probably left you with a fairly generic configuration—something that won’t cause problems for them, but which is suboptimal for your needs.

Triage a representative sample of the defects, and if a checker doesn't seem to be producing a lot more signal than noise, move it to the bottom of your public priority list. (More on this later.) If a checker seems totally bogus (which is fairly rare) or is irrelevant to your company's concerns (far more common), turn it off completely. Do this sparingly, as it may be politically impossible to reenable a checker once you've turned it off. (Conversely, it's also worth looking at the checkers your vendor turns off by default—some of them produce surprisingly important defects.) If a checker seems flakey, check the configuration and error logs. It's regrettably easy to waste a lot of time on false positives caused by build errors. And if build configuration problems are causing obvious false positives, they're probably causing invisible false negatives as well.

Be cynically realistic about which part of your codebase you're actually going to fix soon. Define components so you can skip analysis on the code you're not going to fix. For instance, if your product uses third-party code, in theory you're responsible for its quality, and should patch bugs in it. In practice, such bugs rarely get fixed. And if it's mature third-party code, the false positive rate will be high.

Conversely, in-house code that everyone knows is going to be replaced, can be a rich source of accurate bugs that nobody cares about. Try to get buy-in on excluding such code from analysis. Skipping don't-care code will also speed up the analysis. Components also help with assigning individual responsibility for defects, which I've found crucial to getting them fixed.

Give a lot of thought to how you report defects, and in what order; the commercial tools aren't going to share your priorities. Their primary means of reporting defects is on a web page, but their page won't fit your concerns. Spend a little time reverse-engineering the URL format for browser display of defects. Do a little URL scripting to generate a simple web page with the links your organization cares about, in priority order. Give this page a short, simple, URL and publicize it internally. (I once used a lovely, scary Coverity poster of a bug under a microscope for this; I put the URL and poster over the corporate pinball machine.)

If different parts of your organization own different components, segment the report so that an engineer can just look at what he or she cares about. Be as fine-grained as possible in the report; ideally, have a separate row for each engineer, with his or her name prominently displayed. (Programmers' pride is one of your most important tools.)

Within each section, put the most important links first. This may just be a live link for defects found by the checker(s) that do the best on your code—where what’s best will take judgement and experience to decide. It will be a tradeoff between how rarely the checker is wrong, versus how serious a bug is when it's right.

Another candidate for top billing on your report is new bugs. If you've started with an unmanageable number of bugs in your old code, you may at least be able to forbid adding new ones. One last thing you might want on the report is a manually-maintained Top Ten List of particularly amusing bugs. Used carefully, humor can be an important tool for getting developers' attention, and it can take the sting out of facing harsh truths—which static analysis is embarrassingly good at turning up.

Acknowledgements

This posting is partly based on a talk I gave at the Stanford Computer Science Department and a posting at StackOverflow. My main debt is to "Weird Things that Surprise Academics Trying to Commercialize a Static Checking Tool" by Dawson Engler, Andy Chou, Ben Chelf, Chris Zak, et al., and to "The Benefits of Source Code Analysis" by Andy Yang. After I wrote this post, a follow-on to "Weird Things" was published, which covers some of the same issues, and which I highly recommend: "A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World."

The Irrelevant Tools Team

2010-01-22T11:11:00.000-08:00

Software development tools teams have always been a critical part of a software development organization. The tools engineers, build and release engineers and architects help ensure that the software development infrastructure is contributing to the best and most efficient software development. But what is the extent with which a tools group can positively affect the development organization?

A Tale of Two Cities
Recently I was speaking with a tool manager who described his organization as "IT" (information technology) for the software development organization. His job was to support the development organizations essentially doing "whatever the development managers wanted him to do." It was clear that his team was handed tasks to do and his job was to manage the resources to address those needs. He clearly wasn't setting the agenda - instead he was really acting as a direct report to the development organization.

Contrast that with a similarly sized organization with a completely different tools team. The tools team there is responsible for much larger range issues - not just the installation and operational aspects of the tools infrastructure. There they play a pivotal role in defining process and architecting solutions that meet their largest quality, productivity and security goals. In this type of organization, the tools team is well recognized and funded by the engineering executive management. By centralizing these functions, they take full advantage of improvements across all the different development groups. In addition, they provide a healthy counterpoint to the software development organization - pushing for improvements that need to be made that in many other organizations, software developer's resist.

Case in Point
At Code Integrity Solutions we work with a lot of organizations to successfully institute static analysis into their organization. In the first organization, the software development managers had purchased the tool, thinking it would be a good productivity tool. They then passed ownership over the static analysis solution over to the tools team. The tools team's idea of success was to install the static analysis solution and make the URL available to the developers and to provide support for them if any problems arose. As you can guess, nobody ended up using the tool and their investment in static analysis had a negative return.

In the second organization, the purchase was made by the development managers and the tools team in order to solve their larger quality and productivity needs. There, the tools team was charged with a very different set of criteria: no new static analysis defects would be introduced for any software development. The tools team then had to figure out how to set up a deployment and process that would meet that goal. This necessitated changes to each developer's process, reports to measure progress to multiple parties and the creation of a robust infrastructure that changed the way they developed software.

The Search for Relevance
In recounting these situations, I'm reminded of all the experiences and literature I've read about IT organizations in large corporations. IT, when considered as just the folks who operate the helpdesk, distribute your username/passwords and come over to fix your machine when it breaks down, is a support infrastructure that is nonstrategic and a target for outsourcing to whoever can provide it the cheapest. IT organizations who search for ways to create significant efficiencies or enable new business models are the treasured goal of any CIO or VP of IT.

Tools teams are not that much different in this respect. When tools teams make the business of developing software better, faster and cheaper, they have a seat at the table, playing a strategic role in making product development organizations more competitive. Which organization type do you think brings more value?

Now is the Time.

2010-01-06T10:34:00.000-08:00

It's a new decade. Time to turn over a new leaf. Time to make those significant changes that you had always planned. We speak to so many software development organizations who have had grand plans to improve the competitiveness of their software development organizations but lack the time, energy and resource to do so. The fact is, most software development organizations over the years have only made minimal, incremental changes to their software development processes and practices. The emergencies of the day always take precedence.

However, even simple improvements to improve the productivity of developers can mean that many more market-winning features. Being able to deliver fractionally higher productivity than the competition can mean the difference for market leadership. Engineering will always be the bottleneck, with a long backlog of desired features. Improving the productivity of that bottleneck provides huge gains to product companies.

So when is the right change to change the wheels on a moving car? This economy has provided a convenient time to retrench. After the drastic cutting of resources, the sluggish nature of the economy has provided an ample opportunity for software development organizations to take a step back and improve the software development process. When the economy turns around, more new people are hired, the market becomes more competitive and the chance to make internal improvements dwindle. With the new decade ahead of us, and the economy still not yet turning the corner, Q1 2010 represents a golden opportunity to make improvements for improved scalability of team and business.

We've met many such companies looking at this time as a way to institute new practices. Software development organizations mired in waterfall are now using this time to move to Agile development. Teams used to doing massive, painful integrations are moving to continuous integration. Organizations used to performing significant manual labor are moving to automation like static source code analysis so that they can squeeze not only more productivity from their employees but also improve the quality of the output. They know changing the wheels now is a lot easier than changing it later.

What do you have to show for the last year and a half of economic woe? Will it have been a time of "sitting on your hands" or a time to wipe the slate clean and make those bold changes that will make your organization a strategic part of the business?

Upper management won't look kindly to a "hold the fort" strategy. If you take the opportunity during this down period to make those impactful changes, you'll be using this time wisely to set yourself up for the inevitable upswing. It's time to take action. Change is hard but the alternative is worse.

Who Owns Quality?

2009-12-13T20:18:00.000-08:00

The architect at Code Integrity Solutions and I were talking the other day about ownership of static analysis. Who truly "owns" static analysis and therefore drives its success or failure?

If you look at most large organizations, a central tools group ends up supporting the static analysis tool. I say "ends up" purposely because oftentimes they were not the drivers for bringing in the static analysis tool -- in some cases, they find out after the fact and simply told to support it once brought in-house. How well that tool is supported is a big factor in the tool's eventual success within the organization - however what is more important is knowing who is truly responsible for its adoption. Without a business driver, and therefore the metrics to understand whether the goal is being met, the source code analysis tool is left as a voluntary, nice-to-have tool that gets underutilized for the price paid.

Tools organizations without these strong business drivers are left to try to deploy a "carrot" solution, enticing the developers with its value. Left to their own devices, many developers have too much on their plate to do and may not see the big picture of how it can not only improve quality but also productivity. Inexperienced developers see the short term.

Some tools teams are very well tied into the business of building software and have the right knowledge and influence to make successful tools rollouts. Most tools team feel it is sufficient to set it up and let the developers decide if they want to use it. Someone in the organization has to care about its adoption and successful use. In some cases it's the development managers. In other cases it's the executive team who's tired of talking about quality and would rather talk about features. Quality assurance and central security also should have a vested interest in the use of these types of tools in the development process.

With the right business driver(s), the right process can be put in place to ensure usage, individual developers know that there are incentives and consequences to use of static analysis and progress can be measured to ensure successful adoption.

Inserting Static Analysis Into Your Agile Processes

2009-10-19T13:55:00.000-07:00

Most software development organizations are moving to Agile development processes. All flavors of companies have started instituting Agile whether the organization is an IT organization supporting their business units or product companies who are developing software for customers. An important benefit of Agile is the ability to improve cycle times. Agile enables organizations to iterate much faster on market requirements and to iterate much faster on delivering working software to the market.

To meet these demands, software development organizations are using everything they can get their hands on to help them accomplish their goals. Whether it's continuous integration servers or static analysis solutions, development teams are arming themselves with new tools and new processes to capitalize on their ability to meet market needs faster and with less risk.

From Nightly Analysis to Continuous Analysis.
Static analysis can help in a number of ways. Many organizations already have a nightly build in which they require their developers to check-in only code that has been verified as clean. By making sure their software builds successfully every day, they minimize the risk of having broken code and complex integration steps towards the latter stage of the development process. It's not a stretch to increase the check-in requirement to also include addressing of all new static analysis defects. In fact, being alerted to these issues during development make them much easier to fix than when reported later in the development process.

Continuous integration is a further step in ensuring that the code is being constructed earlier and often. The challenge for most organizations is having static analysis operate at a reasonable performance to keep up with the build frequency. A typical large project may take only 15 minutes to build in an optimized environment but take 4-5 hours to analyze. With the right tweaks, static analysis can be optimized to be much faster. Often, it takes some outside expertise to assist.

Inserting static analysis whenever you build is an ideal time. Builds are a convenient time to run statistics, run automated tests and perform static analysis on your code.

Meeting the Demands
No developer wants to be the person who breaks the build. If there are new criteria required to check-in code, then those requirements should be testable by the developer in their development environment so they get a chance to address any problems prior to the more visible tests. They should therefore be able to make changes to their code and test, fix and iterate until they feel sure that their changes will pass the criteria. Arming the developers with this capability is not easy. Static analysis is doing complicated stuff. The computations that occur take time. Moving static analysis to the desktop requires both proper performance levels as well as the right usability model. At Code Integrity Solutions we've worked with extremely large codebases (tens of millions of lines of code) that whose analysis times can be chopped down to just minutes). Better quality code gets checked in when developers can iterate multiple times. Investing in a developer desktop model provides orders of magnitude benefit including better quality software earlier in the process and improved developer productivity.

Invest in a good, fast solution for both developers and your software development infrastructure and you'll see your Agile pay off faster!

Static Analysis: Isn't This Supposed to Make my Life Easier?

2009-09-21T13:53:00.001-07:00

Software developers have a tough job. They have to produce working, good quality code from scratch, under immense time pressure and often without a good specification to provide guidance. Project estimates are always optimistic which leaves the developer in a situation where they have to work twice as hard to deliver what had been promised. Rarely are developers able to write code to the level of quality that they want to write. There just isn't enough time in the day. Is it any wonder that software developers resist adding more things to do in their process?

Along comes static source code analysis. Most developers don't want to add an additional step in their already harried process to check bugs that were found by another program. Here are the typical rants we hear from software developers?

False positives!! Nothing hurts the credibility of the tool more than false positives. If a tool is more wrong than right, developers won't trust the tool and consider it a waste of time to use. For most static analysis tools, you spend more time reviewing defects than actually fixing them. Fix the analysis so that it reports less wrong reports. Have a team (inside or outside the company) weed out the bad stuff so the development team can focus on developing.

Blend it into the process!! Forcing developers to take extra steps "out-of-band" not only slows them down but also makes them forget to run it in the first place. Make it run automatically with each build. Make it automatically assign defects to developers. Make it easily create a bug tracking report. Notify the developers rather than request them to look for it. Make it easy for them to do their jobs. Make it so that you don't need any documentation.

Tuning!! Developers are already busy enough -- they don't want to add static analysis expert to their growing list of responsibilities. If the analysis falsely identifies an issue as a bug, there are usually steps that can be taken to improve the analysis so that it understands the code better. Requiring the engineer to learn the system to be able to modify a configuration, add rules, etc. is a waste of time. Asking them to write their own static analysis checks is an effort that is often wasted. Have a static analysis expert make it understand the codebase and have them write the checks that are needed to be productive.

Make it faster!! Waiting for builds is bad enough. Waiting for static analysis to finish can be a time sink especially if it's a requirement to check in "clean code." Have an expert optimize the performance of the system. Cut down the analysis so that it analyzes only the stuff that has been changed.

Truly getting adoption from developers is a tough challenge. If they don't see the value, they won't use it. Maximizing the benefit returned from using the tool is just as important as minimizing the cost of learning and operating the tool. The key to bringing value is to manage this equation so that maximum return is achieved at an individual developer level. You'll never get perfect adoption without creating some artificial requirements (like not allowing a release without addressing all bugs). However, you'll have to fight a lot fewer battles if you make your user base as happy as possible.

Are Static Source Code Analysis Solutions One Size Fit All?

2009-09-09T10:53:00.001-07:00

Automated source code analysis solutions worth their salt are complex products that do heavy duty analysis on complex codebases. They churn through thousands to millions of lines of code written by teams of developers each with their own skill levels and style. Millions of paths are typically analyzed in a single run producing what it thinks are valid defects to be examined.

On one level, you'll hear people say that code is the same whether it is written in Java, C++, C or C#. After all, some universities give computer science homework assignments that can be written in your language of choice. On the other hand, there is a high degree of variability in the customer base that a static analysis vendor must provide. There are just some of the significant differences:

Industry - some industries care about different things. A router company is going to care more about memory leaks because their code may be in operation for months to years at a time. A desktop application may not care at all about memory leaks because they know their application will be open and closed frequently. You can imagine that a military aerospace or medical devices company would have vastly different priorities than a SaaS product where the system can be controlled much more easily. Some organizations may want to squeeze every last possible defect the tool can find at the cost of getting some additional noise. Others may be looking for a good "bang for the buck" and fix bugs opportunistically without having to wade through false positives.
Development environment - the software development tools space is highly fragmented. There are hundreds of vendors that provide source control systems, compilers, bug tracking systems, build and continuous integration tools, automated test tools, etc. On top of that, there may be third party libraries in use and/or code generation tools that supplement the hand-written codebase. Software applications are almost always built in a custom way and it's hard for a third party tool to get an accurate picture of what the shipped code actually is without a significant amount of tweaking and integration. A static tool may simply not understand a custom memory allocator or a source/sink for a data flow analysis. They may also have some compiler incompatibility issues. What can happen is that you don't get 100% coverage. Tools will typically "fail" silently or make it nonobvious how to find out what your true coverage is.
Development process - similar to the diversity of the software development infrastructure, software development organizations are often very chaotic. Some organizations follow the traditional waterfall methodology and others follow Agile processes. These requirements place demands on how a source code analysis is deployed and used. Some organizations are global with offshore development teams making significant contributions.
Platforms - software development organizations typically commit to a single development platform (usually a specific Unix flavor, Linux, Windows, Mac OS X, etc.). However, they may be building software that will run on many different target platforms. Each of these applications use different libraries and may have significant functionality differences.

What's a static analysis tool to do to satisfy everybody? Should the default settings be set conservatively or aggressively? Are the settings designed to give a good demo/trial or are they set for a specific industry? The clear answer is that no tool comes out of the box optimized for your specific codebase. There is too much variability.

What's important to do is to build or obtain the expertise to optimize the tool for your codebase. After all, you are paying the license fee for the full benefit of the tool. If you don't take the extra effort to make sure you are optimized, then you are overpaying for the tool.

We've worked with many organizations who have taken different approaches. Some of built their expertise in-house with just the standard training as their starting point. This can take years to obtain expertise, however, the competency remains in-house. Others take a hybrid approach and allocate the right resources and get an expert to mentor them through the process. Others use consultants to get them all set up and running and then use their in-house staff to manage and administrate it. Others outsource everything related to the tool to an outside expert so that they can focus on their core competencies.

What You Didn't Buy When You Bought That Static Analysis Tool

2009-08-30T16:57:00.000-07:00

When you are buying things, be it at the store or through a vendor for your company, you are really buying results. The tool or object you purchase is just a means to an end. Like the old adage goes, you don't go to a hardware store to buy drills, you go there to buy holes. Static source code analysis solutions are great tools for helping software development organizations find defects earlier in the development process. However, because they are a technology entering the mainstream, most organizations don't have the experience to know entire effort required to effectively use static analysis.

When you purchase a static analysis solution you are really buying the defects that are valuable enough for you to take action to fix. A good measure of the value of the tool is the number of defects that you indeed fix. What you aren't purchasing are:

the set up and customization required to get the tool set up, tuned and integrated into the toolchain
the management and administration needed to keep the tool running
the additional hardware required to run it with good performance
the training and rollout required to bake the tool into the process
the political process required to get people to change their behavior
the additional steps required for each user to change their process
the defects that were reviewed and put in the 'not fix' bucket. These include false positives, don't care alerts, low priority reports and ones that haven't been reviewed.
the time and expertise required to further improve upon the tool or take advantage of features and capabilities not yet used

Even with all of this effort taken together, it's hard to beat the value that a good static analysis tool can bring. Just remember that buying the tool is just the first in a series of steps that need to be taken to get to success. Sometimes you have to pay a little bit to get the big payoff.

Who's Really In Charge Now?

2009-08-23T22:38:00.001-07:00

Static analysis is fast becoming a standard part of every software development effort. Companies ranging from medical devices companies to Internet ecommerce vendors to car manufacturers to network equipment providers are purchasing static analysis solutions on the promise that finding defects sooner in their development process will save them time, money and embarrassment. The concept sells well at the developer and management levels because it makes sense.

And, static analysis tools are good these days. They aren't perfect, but with the right usage they provide a good bang for the buck. I stress the word "usage". Many companies buy the tool. The vast majority install the tool but a surprisingly high number don't get past running the tool even a couple of times. The product sits on the shelf.

There are many reasons why this occurs but the one we'll explore today is ownership. Someone has to own the tool and what happens to the results reported by the tool. Technical ownership is fairly straightforward. Static analysis tools are usually owned the tools group, who is responsible for setting up the software development infrastructure. They set up the factory floor, making sure that the engineers have all the tools they need to be efficient and effective. Automation plays a key role, helping to scale benefits across an entire team. However, unless the proper organizational and process infrastructure is in place, they are likely doomed to failure.

The purview of the tools group varies but does not include setting up new processes in isolation from the software development team. Oftentimes static tools are thrust on the tools group. The tools group then sets it up with the only goal of offering it as a voluntary service to the software development team. What ends up happening is a static analysis tool that is used by only a few enterprising software developers who believe in the value of tools and of software quality. The tools group is powerless to force more developers to use the tool and only a fraction of the potential value of the tool is realized.

Tools groups can lower the cost of usage by making it as simple as possible to use, by integrating the tools within their development process more effectively and by providing training, support and mentorship to developers. Tools groups also can take responsibility for tuning the analysis to make sure the results are as relevant as possible. But while carrot approaches are important and necessary, it won't get everyone over. You need to foster experts within the organization who can establish and defend the tool's worth. When you can set check-in, commit or release criteria that include static analysis, then you start getting the usage that will provide the payoffs that were paid for.

We've seen organizations that mandate that all new issues get addressed. Even false positives are "fixed" because if the code is confusing enough to fool a static tool, it will likely be confusing to the next person who must maintain the code. Others simply require that at least everything needs to be reviewed and it is up to the developers to determine whether they should fix them (the developers must document why they don't fix a problem). Others require specific classes of bugs to be fixed. The rules simply must match the business goals.

So even though tools groups are essential to the success of static analysis, they cannot be the only owner. Like any change within an organization, it requires executive sponsors, champions and a technical and organizational infrastructure to make it happen.

Hiring, Repurposing, Consulting, Contractors, Outsourcing, Offshoring or Part-timers for Static Source Code Analysis?

2009-08-03T16:42:00.000-07:00

When it comes to getting resources to get static analysis underway within your organization, you have a lot of options. You can hire full-time (or use an already existing teammember), you can get a contractor, you can get consultants and you can outsource/offshore.

When it comes to designing and operating your static analysis solution, there are many options you can go with. I like to think that the institution of static analysis in an organization as being divided into several phases.

Requirements - what goals do you hope to get out of it, usage models
Design / Architecting of the solution within your existing infrastructure
Implementation - the fine art of setting it all up and automating it
Maintenance - the ongoing management of the solution, including dealing with the inevitable changes to the code, build environment and the static analysis solution

Clearly, each step requires a different skill set. These steps can be handled by one senior person (who has all the skills) or by a combination of people each specializing in their own areas.

We walk through the various choices with their own advantages and disadvantages:

Full-time
Part-time
Contractor
Consulting Firm
Outsourcing / Offshoring

As someone who has hired many people full and part-time, and who has worked with contractors, consultants and outsourcing firms and have founded and worked in two consultancies, I have a few experiences and scars to share.

Full-Time Employee

Advantages:

Institutionalize all the expertise with your employees
Flexibility to control your resources and move them to different responsibilities rapidly
Employees benefit from already existing relationships in the company

Disadvantages:

The fully loaded costs of a full-time employee are significantly higher than the salary. Put in the cost of insurance, training, office space, machine costs, etc. and a $100,000 employee starts looking like a $150,000 cost. The budgeting process reflects these extra costs and the cost of extra headcount.
Employees are often fairly flexible resources that are assigned many different responsibilities but are therefore constantly being pulled onto other urgent projects of the day.
Hiring takes extra effort. Employee hiring should be looked upon as a permanent action. Will this person with these specialized skills be the optimum fit years from now?
You may need to train this person. It's hard to hire someone who is both flexible for the long term and highly specialized for your need today. It takes months to even years of investment to develop a good level of expertise in-house.
Gaps in time such as for vacation and sick days require coverage, particularly for ongoing management and administration.

Part-Time Employee

Advantages:

Some roles don't need to be full-time. Particularly for a younger company, you may not be able to justify a full-time build engineer or a static analysis expert.
Part-time enables you to pay for only what you need but still get what you need for the amount that you need it for.

Disadvantages:

If you predict growth, then part-timers are not very scalable.
Part-timers also carry a fully-loaded cost, oftentimes higher in proportion to a full-time employee.

Contractor

Advantages:

Contractors need to prove their existence every day. They know they have to provide a high level of service or be replaced.
Contractors can be started up quickly and can be removed at the end of the project without any HR headache. Some projects make sense for a period of time and not forever.
You can hire contractors that are highly specialized. Perhaps it's a particular product expertise, or a particular process expertise - you can hire highly experienced people from a temporary time period to get best practices.
Contractors enable you to focus your best resources on the most strategic activities while leaving the rest to nonstrategic resources.

Disadvantages:

Contractors need start up time. They are used to getting up to speed fast but still require ramp up time.
Contractors may or may not be available after the project completes.
If you don't plan properly for this, contractors will leave at the end of the project (or even midway in extenuating circumstances) with all the institutional knowledge gained during the process. You should build in time early to transition knowledge through mentorship and documentation throughout the project.
Contractors need to develop relationships during the project.

Consulting Firm

Advantages:

You have a firm to back you up. If you need to have varying levels of expertise throughout a project, it's up to the consulting firm to provide the right person. You can draw upon a larger set of expertise from a consulting firm. You can get the best experts in every area you need that don't need any training.
Consulting firms tend to be solutions and results focused.
You can ramp up and down a team easily - from a whole team down to a part-time consultant.
You get coverage when you need it. No need to worry about sick and vacation days.
You have one place to go to for as many of your needs as possible
You allocate budget and responsibilities and are less able to be sidetracked than an employee
Consulting firms enable customers to focus their best resources on their company's strategic needs while still leaving nonstrategic items to experts in their field

Disadvantages:

Consultants require some ramp up time
If you don't plan properly for this, consultants will leave at the end of the project with institutional knowledge. You need to work in transition time through mentorship and documentation.
Depending upon market rates, consultants may be more expensive than the fully loaded costs of an employee.
Consultants must develop relationships during the project.

Outsourcing / Offshoring

Advantages:

The cost advantage is usually significant
Outsourcing is a good way to pass off non-core competence work leaving your developers to focus on more strategic work

Disadvantages:

Communication is usually difficult. Many outsourced teams are located in different countries with different timezones and languages spoken.
Management overhead is increased. Interfaces between teams must be micromanaged.

Hybrid approach.
Every organization is different. No one solution fits all. The key though is to have the expertise you need when you need it in order to institute static analysis as quickly, efficiently and effectively as possible. Consider these questions as you consider what are your best options:

Many organizations recognize the investment required to develop expertise in-house. Is the investment of becoming static analysis experts worthwhile or not?
Do you value best practices? Will having an expert who's done it many times before guide you through the process more efficiently than trying to do it yourself?
What are my architecting versus operating costs? How can I create the best solution that requires the least amount of management and administration?
How much time is required and what expertise is needed for architecting and operating static analysis? It may not be full-time.
What are my contingency plans if priorities change or people leave?
What are your true core competences? What do you want to be best at doing?
What kind of budget do you have? Initially and ongoing? What is the return you get to justify the expenditure?
Are pieces of the requirements-architect-implement-administrate better served by the same person or different people? Is continuity of resource more valuable than best practices in each area?

What Motivates People to Buy Static Analysis

2009-07-24T16:18:00.000-07:00

We’ve visited a lot of companies who are using static analysis to improve their software development. What drives companies to use static analysis varies considerably from company to company and from industry to industry. We attempt to categorize the most common drivers for using static analysis into various buckets. We’d love to hear about other drivers as well so please let us know if you have others to contribute.

There are a lot of factors to consider when trying to understand what drives a company to use static analysis. You have to consider not only what are the business drivers and benefits but also who are the people who are driving and influencing the decision.

We have bucketed them accordingly
• The opportunists
• The code reviewers
• The engineering management
• The quality portfolio manager
• The compliers
• The automators

No categorization is discrete. There’s quite a bit of overlap and different roles can exist within the same company. Benefits can be felt across a wide range of people.

The Opportunists
The Opportunists use static analysis tactically, to capture the low hanging fruit in the codebase. They recognize that one of the biggest benefits of static analysis is finding and fixing bugs are much cheaper than most any other means. But, Opportunists only want to invest as little time in the tool as possible. They try to find and fix as many of the easy problems as they can find and focus only on the areas that they feel are most important to them for example like focusing on crashes or memory leaks.

The downside of this approach is that incrementally, Opportunists get a really high ROI because their "Investment" is low and their "Return" is good. However, they must take into consideration the license and support costs plus any other costs associated with the tool such as training, set up and configuration. With all parts of the equation adequately modeled, the ROI can suddenly become much less attractive. In order to keep the costs low, many Opportunists just run the tool out of the box with the default settings, not even taking the time to tune the analysis to find more of the low hanging fruit that they want to fix. Clearly there is more benefit opportunity for Opportunists, to better increase the marginal utility from the marginal cost they put into it.

The opportunist is usually a single developer or a few developers who are highly proactive in improving the quality and security of their code. They are also usually extremely busy with little time to devote to static analysis.

The Code Reviewers
Code Reviewers can live within the software development organization (such as a code review team) or outside, such as in the case of a security review team. Manual code review is a tedious, error-prone and non-scalable process. While static analysis doesn’t replace manual code review, it helps make the manual code review process much more focused on higher value activities. At the same time, static analysis helps reviewers gain significantly more coverage, reduce human error and provide more consistency in the review process.

Code reviewers like to have a well-tuned analysis, specific to the codebase with all the right settings to find the class of problems that are most relevant to them, for example, security reviewers like sources and sinks defined as part of their data flow analyses (without it, the data flow analysis doesn't have much to work with). The analysis can also be tuned to find a wider range of problems at the cost of a little less accuracy. Code Reviewers can also write analysis rules to find specific classes of problems or enforce coding rules. When put into a regression test, these rules can make sure the problem or coding violation doesn’t arise again either in the original place it was found in or in any other place in the codebase or even in other codebases that are being analyzed.

The main challenge in these environments is the natural friction caused by reviewers versus developers. When handled improperly, developers can become defensive and push back on static analysis reports as “not bugs.” Code Reviewers need to be sensitive in how they assign and discuss potential problems. Developers take a lot of pride in their work but should also be open minded of efficient ways to improve the quality of their output. When working well, this model has great checks and balances in place to ensure proper usage of the tool.

The Engineering Management
We’ve all heard the analogy of how software development should borrow more from the manufacturing industry. Many software development managers recognize that static analysis actually saves overall development time and can improve cycles times. Finding defects earlier in the process can most definitely save on overall time and costs.

Management’s big challenge is to ensure broad, consistent usage so that the benefits can be realized. Not every developer recognizes the benefits amidst the negatives. Static analysis is not perfect and requires some learning, to understand defect reports, process changes and most importantly dealing with false positives and “don’t care” reports which, if not handled properly, can decimate the morale of the developers.

The biggest challenges to these environments is lowering the cost of usage through deep integration with the current software development process and automation to minimize any manual work. In addition, tuning of the analysis to lower the rate of false positives and “don’t care” reports will lower the cost significantly and provide a big psychological boost. Using outsourced teams to review the defects first help maximize your most precious resource – your developer’s time.

Administrating the tool requires effort and most tools engineers are already too busy to not only learn the tool but manage it ongoing. Contractors and consultants can help either to jump start the process or to take ownership of the management and administration ongoing.

The Quality Portfolio Manager
There’s no silver bullet to quality and security. In order to get acceptable quality and security, you have to take a multi-pronged approach. Unit tests, code review and now, static analysis are parts of the overall strategy to lower the risk of field defects.

Holistic approaches to quality are often driven by a forward thinking engineering or QA manager, or as a response to a serious bug or serious class of bugs that has now driven a top-down quality initiative. Other companies may realize that bugs are causing a high recall rate or high support costs which is driving down profitability. This realization create the necessary organizational capital to move static analysis forward.

The challenges that arise from these types of initiatives stem higher than those faced by the Engineering Management example. Specific ownership and clear goals are too often not considered when instituting a static analysis solution from the top down. When static analysis becomes just a checkbox among a range of other strategies, it often doesn't get the necessary attention to make it succeed. Ensure that the ownership of the tool has teeth, whether it be owned by someone internally or externally.

The Complier
In some industries, you just don’t get much choice. If you are developing software that will be implanted into someone, or go up into space, or guide a missile, then you have to be concerned about regulations and industry guidelines such as FDA, DO-178B and MISRA. In all of these industries the government either requires or recommends the use of static code analysis to improve the quality and security of the delivered product.

The challenges faced by the Complier are similar to the Quality Portfolio Manager. The further away the driver for a solution is from the implementer and maintainer of the solution, the more that will be lost in translation and commitment. Developing the expertise in-house is not insignificant. Creating ownership with real incentives and consequences is critical.

The Automator
Automation is the fashion, particularly with the movement towards Agile methodology where cycle times are much more rapid. Continuous integration builds are becoming more common and creates better software earlier in the process. Automators recognize that static analysis is perfect for this. Static analysis doesn’t require test cases and can be automated as part of the build process (as most modern static analysis is integrated with the build process). Results can be churned out in minutes or hours. Developers can even analyze code as they are developing it allowing organizations to institute rules that code must be free from any static analysis errors before check-in.

The largest challenges here is that automation takes some work. It’s not hard but it’s also not necessarily turnkey. Being able to have the right deployment model so that the analysis is fast for each developer can be tricky but is most certainly possible. Automation usually is two pronged – where developers can run a fast analysis locally on their changed code while a central analysis handles a full integration analysis. Automation is well worth the effort even in the short term.

Many Reasons, Many Options
Clearly there are many drivers for static analysis. In conert, the optimum deployment model varies considerably. For example, tuning an analysis to find everything under the sun may not be a good model for the Opportunists. Likewise, a full developer deployment model may not be a good choice for the Reviewer's. Matching the deployment model and the analysis tuning to the business need will increase by orders of magnitude the ROI you get from a static tool. Companies have woken up to the fact that developing this expertise in-house is an investment that is appealing to some and unappealing to others. Some prefer a jump start mentoring program to help them get pointed in the right direction and others prefer to outsource to get best practices static analysis expertise. Regardless, static analysis can provide a multitude of benefits for most software development organizations. By recognizing the drivers, you can create a great custom solution that makes almost everybody who involved get what they need out of it.

Do Your Software Quality Practices Have Teeth?

2009-07-22T17:26:00.000-07:00

These days, no one has time to do the extra things that should be done. We only have time to do the things that absolutely need to be done. We don’t strive for perfection – instead we strive for barely acceptable. Working in business today is all about compromise -- prioritizing our efforts on the things that are important AND urgent with little to no room for anything else. Over time, urgent takes precedence over important. In general, quality clearly falls into the important but nonurgent category. Proactive measures, such as using static analysis to improve quality are abandoned or delayed to address the need of the day.

Organizations see static analysis (or source code analysis) as a great way to:

find costly bugs and security vulnerabilties earlier in the development process
save developer time and money and to shorten delivery cycles
detect severe defects before letting them get out into production
increase code coverage along many more paths than dynamic testing and code review
improve software as the code is being written to get better working software faster

While organizations may spend significant money to rightfully purchase the best static analysis tools, they frequently don’t set up the right process and role structure for success. The ownership of the tool becomes a side responsibility for an already harried tools engineer and the tool is simply made available for developer use on a voluntary basis. Time and again, the engineers who least need static analysis, end up using it the most – these are the engineers who use everything to their advantage to improve the quality of their code. While a generalization, the engineers who complain the most often are the ones who should be using it more. Usage then degrades rapidly over time and it becomes an expensive tool that is run only occasionally delivering only a fraction of the benefit.

In most organizations, addressing the results from a static tool is frankly not considered urgent. Resolving static analysis issues provides great value but it is not putting out fires. When urgent AND important issues come up (which is status quo), the use of static analysis falls to the wayside, or never even gets started. Why?

static analysis reports flaws in source code. The effect on a customer isn't always apparent. For instance, it might detect a memory leak on a specific line of code but it doesn't have the same effect as a bug report from your biggest customer.
static analysis defects are reported from a tool, not from customer support or a sales person trying to win a deal. Problems found by a tool in the development process don’t have nearly as much urgency.
static analysis tools aren’t always right. In fact, some tools have very high false positive rates where the tool is more wrong than it is right. This makes it very hard for many developers to trust the tool and feel it is a good use of their time.

What's a development organization to do? How does a software development group save time overall while improving quality with static analysis? As with any new process you want to introduce to a group of people, it has to have some teeth to it. If using static analysis is purely voluntary then there's not much hope you’ll achieve the type of benefits you had hoped for.

If you want to have some teeth in using static analysis you have to have both tangible incentives and consequences for using static analysis. Or, if you prefer, this is the carrot and stick analogy. Because every organization is varied in people's skill levels and motivations, having real incentives and consequences is critical to moving the whole organization in a consistent fashion.

Some examples of incentives are:

educating the team on the benefits of static analysis. This is typically the job of the hampion or managers to make sure that the benefits to the organization and to the individual are spelled out.
peer pressure is also a powerful tool to motivate people to use the tool. If the top developers are bought into the process and are using the tool then you're more likely to get more people in the organization to follow. Cultivating the influential leaders is necessary.
public recognition drives many developers, oftentimes more than salary and benefits. Recognizing proper usage of the tool helps the rest of the organization see the benefits of using static analysis and drive developers to perform the right behaviors.
lower the barriers for usage. Make it simple to use. Automate everything as much as possible so that you lower the cost for a developer to use static analysis. Integrate with the workflow and minimize change to the process flow.
don't penalize people for its usage. Some developers are afraid of showing flaws in their code. After all, it's a tool that points out mistakes, sometimes really dumb ones. Set up usage models that enable developers to find all of their mistakes up front to give them a chance to fix problems before everyone knows.

Some examples of consequences are:

make it a check-in criteria to have all static analysis issues inspected and addressed. Developers have a lot of pride in their work. Nobody wants to be the developer that holds up the release.
managers typically set MBO's for their employees. Managers can set specific static analysis goals such as meeting thresholds for fixing critical problems as minimizing regressions
published metrics can create peer pressure. Breaking out reports by individual developers can promote healthy, productive competition. Using negative reinforcement techniques have to be coupled with the right culture. Metrics can also create the wrong behavior. For instance, publishing defect fixes could cause gaming – the creation of intentional defects in order to produce a high fix rate. Metrics focused on the overall desired results such as the number of times the developers checked code with unaddressed defects.

You either want the tool to be used or you don't. Without the proper incentives and consequences the tool will likely languish. There's a reason why people deride the phrase, "If you build it, they will come." Simply putting a tool out there for use doesn't mean it's going to be used. If the tool isn't used, then what are the consequences? Are there any ramifications for choosing not to use the tool?

There are too many things to do with too few hours to do them. Change is only worthwhile if you are going to follow through with it. Setting yourself up for success with static analysis is not hard, but because it falls into the important but nonurgent bucket, you need to set up the right structure so that it is addressed to the level where it will provide good value.

Usage Models for Static Analysis

2009-07-13T13:56:00.000-07:00

Static analysis (or source code analysis) hasn't yet hit the mainstream yet. Leading organizations are using it to gain a quality and security advantage over their competitors, but it's not yet part of the standard tool chain. With regards to tooling, nearly every software organization starts with a source code repository, bug tracking database, compiler, automated test frameworks, etc. They standardize on these functions because a software development team needs to collaborate effectively to produce a final product.

However, there are other parts of the tool chain which are often (but not always) left to individual preferences, such as editors like emacs, vi, Eclipse. Other tools that often fit in this category can be architectural tools, debugging tools and static analysis tools, such as Lint. Rarely is it a requirement for every developer to look at all Lint issues. Lint is more of an option, left to the individual developer's decision on if and when to use it. Recent static analysis vendors like Cove rity and Klocwork and Fortify Software are providing to the market place a more holistic solution, meant for every developer in an organization. The market has evolved quickly in recent years with more team and organization oriented tools.

The deployment model needed for a given organization depends heavily on what the goal is they want to accomplish. Success and failure can take many different forms when using static analysis. Some organizations are perfectly happy finding the occasional killer bug and leaving the rest of the reported defects unaddressed. Others, want to use the tool as essentially a cornerstone to their code review efforts and to find every possible problem that could exist. All of these call for different analysis options and for different deployment models.

Process Structure of Code Analysis
This is a simplification, but for most static analysis deployments the following roles need to be addressed:

Administrator - who installs and manages the tool
Triager - who reviews the reports that come from the tool and prioritizes the results
Fixer - who fixes the problems
Verifier - who verifies the fix worked and didn't introduce new problems

These roles can be filled by internal and external staff in various combinations. The most common deployment models we see are the following:

"The static analysis guy" -- this is where one developer has been dubbed as "the static analysis guy" and performs all of the above functions. This person runs the tool on their machine periodically usually for a couple of minutes every day or at a milestone during the latter part of a release cycle. This developer often reviews the defects and either fix the bugs or distribute them to developers. This option is great for companies who care only about the "low hanging fruit" bugs. This options has several advantages:

+ one person who is responsible for everything and therefore "one neck to wring"
+ few handoffs

But...

- Process is inherently unscalable
- No checks and balances in place to make sure the person responsible hasn't made any mistakes or is using the tool suboptimally
- Putting someone in charge of static analysis takes someone "off the bench". Usually a senior developer is put in charge of static analysis causing s/he to invest considerable time in becoming a tools expert than focusing on building features for their product.

Regrettably, some organizations who have every intention to roll static analysis out to their organization never get past this point.

"Special teams" - this deployment option takes advantage of different specialties. The administration side is taken care of by the tools group. The triaging is typically handled by a separate team such as, a security review team or a quality review team who examine the problems up front and then distribute them to the developers who handle the fixing. The verification process is usually handled by the security or quality team to ensure the fixes are done right and that no new issues are introduced. The advantage of this model include:

+ Great checks and balances in play that help the overall quality improve. Healthy tensions between triagers, fixers and verifiers drive quality.
+ Saves significant time from your most precious resources - your developers. The largest cost for many organizations using static analysis is in reviewing the defects (see earlier post on the hidden cost of static analysis).
+ Highly scalable structure

But of course it is not without its costs:
- Handoffs of defects requires more coordination and can create errors
- Development skills are required in every role. A tools engineer must be able to understand code and be able to tune the analysis based on developer feedback. Security reviewers, quality engineers, etc. must also have software development skills.

"Managed Service" -- this option focuses the software development team on their core competences . When people buy static analysis, they are buying essentially the defects that they end up fixing. They aren't buying the administration burden, they aren't buying false positive reports that come from the tool -- they are buying the defects that are reported which they take action upon. In this model, the administration is managed by an outside consultant. The triaging is handled by a specialist security or quality review team. The fixing is handled by an off-shore software development team - the same type of team that regularly handle maintenance for software products. The verification is handled by either a skilled QA team or the review team. There are several key advantages and disadvantages:

+ Consultants are specialists in their field - so you get the best practices implementation without false starts
+ Consultants are responsible for hand delivering only the defects you want to fix so that you don't have to "lift a finger" in managing the tool or triaging the results. This enables the team to focus on what they do best - building features. This is one of the main reasons why managed services in general have taken off.
+ Resources can be scaled up and down as needed, for instance you can have a larger team handle the triage of a big backlog and then once the backlog is down to zero, have a smaller team deal with the bugs caused by incremental changes. For most organizations, administration is only a part-time job.
+ Consulting organizations take all of the brunt of scheduling. You don't have to worry about sick/vacation days or deal with HR issues.

However,

- Working with consultants requires some handoff both in startup and ongoing work.
- Consultants also require some overhead to manage.
- In particular for fixing, consultants can't know the code as well as your developers. Some collaboration is required in order to make the consultants efficient and effective, just as in the case when an offshore team takes responsibility for a codebase.

"Every developer is responsible for quality" In this model, each and every developer owns quality and security. In this case, the software developers are responsible for reviewing the results and fixing them. Verification can also be done by the developers depending upon how much you tolerate the "fox guarding the henhouse" approach. The advantages:

+ Puts more of the responsibility for quality into the hands of the developers
+ Brings defects to the developer's at the absolute earliest time - right as they are coding versus waiting for results from a nightly build.
+ Helps developers check in better code, a benefit for Agile development and continuous integration models.

But,

- Requires a consistent approach. There is a wide variance between the best and the worst developers in a given software development group. A common saying in the static analysis community is that the developers who ignore or rebel against static analysis tend to be the ones who need it the most.
- Lack of checks and balances. Over time, some developers will just write code that makes the tool "shut up" even at the cost of making less stable code. If the improper incentives are put in place then this kind of suboptimal behavior can result.
- While the total time to review bugs every day may be small, added up across every developer every day can result in a significant amount of time. For most organizations, the time for triage significantly surpasses the time it actually takes to fix bugs.
- Training is required for every developer as well as every future developer who comes into the organization.

"Hybrid approach" Nearly every other combination exists in some way shape or form. Many organizations opt for a hybrid of the above, most typically a tools group responsible for the administration of the tool (in-house or outsourced), a review team (in-house or outsourced engineers) and an in-house development team to fix the reported defects. In this way, developers are focused on what they do best, building features, not on becoming experts at running a static analysis tool. In addition, having a separate triage team saves time, freeing your developers from having to wade through "don't care"'s, false positives and low priority items.

Have it Your Way
Using a good static analysis tool can significantly improve the quality and security of code. Whether the goal is to address compliance, find every last possible memory leak, or to occasionally find some bugs, there is a deployment model that should work. Each method has its pluses and minuses. Over time needs will change, the deployment model should change as well. We don't doubt that there are quite a few other potential configurations out there to get the most out of the static analysis tools.

The Hidden Cost of Source Code Analysis

2009-07-06T12:57:00.000-07:00

Static analysis (or source code analysis) seems like a no-brainer. The analysis runs on your source code, bugs get produced and you fix them. The sooner you fix problems, the cheaper it is, often by orders of magnitude. However, there is a hidden cost to static analysis -- one that most organizations need to take into consideration but rarely do, until the rollout is well under way.

Let's Take a Deeper Look
Are all static analysis defects created equal? Certainly not. There are some issues that are more important to address than others. Every organization categorizes defects differently. For some a memory leak may be important, particularly for long running applications where even a tiny memory leak can add up to a big problem. For others applications, like a desktop application that loads and exits frequently, memory may be hardly an issue but crashes may be a serious priority. Other factors that affect priority can include effect (such as data corruption), frequency, compliance, etc. Regardless of any company's individual priority, one can roughly bucket static analysis defects in these categories:

Critical - we should fix it now (e.g. don't check it in)
High - we should fix it before the next milestone
Medium - we should fix it at the next appropriate milestone
Low - nice to have - fix it if opportunistic situation
Ignore - analysis is doing the right thing but the problem doesn't need to be fixed (might be in test code, 3rd party code or perhaps the defect is incorrect because it couldn't take into account an environmental assumption)
False positive - the analysis goofed and the bug report is wrong

The Problem With Prioritization
Keep in mind that most bugs are prioritized based on the effect of the bug, which is easier to tie to a business problem. Static analysis defects are reports of flaws in the code -- for instance, it might show a potential memory leak along a specific path in the code. It may be difficult to discern how much memory is being leaked, what the specific steps are that cause the leak and what happens if enough memory leaks. Static analysis defects are thus, harder to prioritize.

A "Typical" Distribution
The distribution of defect types varies greatly among organizations. Some organizations (like a company that writes mil-aero software) consider any coding defect a serious issue. Based on experience with customers at Code Integrity Solutions, we often see defect distributions like this:

Critical - 10%
High - 10%
Medium - 10%
Low - 10%
Ignore - 45%
False positive - 15%

With different codebases, different static analysis solutions and even different triagers you may get varying results.

The Biggest Cost?
In addressing the defects reported from a static analysis tool, you need to:

Make sure the analysis is set up properly
Someone needs to triage the results meaning interpreting the results for action
Someone needs to address the issue, typically by fixing the problem
Someone needs to verify that the issue is addressed

#1 is typically handled by an administrator skilled in the static analysis tool. #2 may take anywhere from 5-10 minutes on average (we'll assume 10 minutes on average because some go quick, and some take longer). #3 often takes 20 minutes to throughly understand the issue, find the right solution, make the necessary code changes and then review/QA the change. #4 we don't have reliable data for.

Back of the Envelope
Most organizations purchase a static analysis solution for a product after the product has already been released. Therefore there is a considerable amount of "legacy" code already there. When you run a static analysis solution it finds problems consistently across old and new code. Thus, many organizations who recently purchased a static analysis start by running the analysis and creating a backlog of sometimes thousands, if not tens of thousands of defects.

Let's use a fictitious example. Let's say an organization has 3 million lines of code and finds 3000 defects in their backlog. In order to triage all of the defects, it would take 3000 defects x 10 minutes per defect = 500 hours (62.5 developer days). For a team of 12 developers, this would take a little over 5 business days or essentially a full week. Assuming the distribution above, the distribution of defects would be around:

Critical - 10% = 300 defects
High - 10% = 300 defects
Medium - 10% = 300 defects
Low - 10% = 300 defects
Ignore - 45% = 1350 defects
False positive - 15% = 450 defects

To fix 300 defects would take approximately 300 defects x 20 minutes per defect = 100 hours. For the same team of 12 developers, this would take 8.3 hours or a little over 1 day.

The Hidden Cost
So to triage all the defects takes a team of 12 developers 1 full week. To fix the defects that you care most about, it takes 1 day. That's a whopping ratio of 5:1. Clearly, one of the largest costs of static analysis is triaging the results - to sift through to determine which are the ones worth fixing and which are the ones worth ignoring. In addition, some static analysis tools have much higher false positive rates which means much of the time is spent in reviewing defects rather than the actual time performing the fixes.

The Case for Outside Help
For most organizations, developer time is precious - the most constrained of resources. A week of a team of 12's time can mean a significant number of customer-focused features developed. Many organizations are turning to outside teams to help review defects so that their developers can focus on their core competence - developing capabilities within their product rather than developing expertise reading static analysis results.

An additional strategy is to address the defects even earlier in the software development process, e.g. to the developers as they are coding. This may slightly reduce triage time overall.

However, whether you use an outside team or not, the time required to triage defects must be taken into consideration. It's surprising for many that the most time consuming aspect of using static analysis is in the triaging of results rather than the actual fixing of defects.

Not All Bugs Have to be Fixed

2009-06-29T15:36:00.000-07:00

We live in a world where there just aren't enough software developers. No matter what stage of development you may be at, your team could always use just a few more developers to build that great feature marketing wants, fix that extra bug that's been nagging technical support, help build some tools so that software development can work more efficiently, etc. But sadly, we live in a world of constraints and that means that the marginal cost of any investment has to be paired with the marginal benefit it will bring.

The approach to fixing defects is no different. When fixing problems you need to look at the resource required and the benefit it brings. You'll see a wide range of perspectives when you ask developers about source code analysis. Naysayers view the results as noise, "nice-to-have" defects that never reveal themselves in the field. Supporters believe source code analysis helps them find not only critical defects but also potential coding problems that can affect future maintainability and scalability. Most developers fall in between these two extremes. We often see a good mix of problems reported:

critical problems that need to be fixed soon
medium to low priority items that are certainly bugs or poor programming but aren't of high priority to fix immediately
false positives - where the tool reports something that is clearly wrong
"ignores" - where the analysis is doing what it should be doing but the prioritizer doesn't think it's important to do anything about

Depending upon the organization's needs, the numerical distribution of these categories vary greatly. For a company sending a satellite into space, they may view most issues as in the "need to fix" category, even going so far as to saying that if a static analysis tool is confused enough to get it wrong, then the code should be changed so that the code is easier to maintain for the owner, code reviewers and all future owners. For these organizations, there is zero tolerance for failure.

For other organizations that are strapped for resources and have more than enough critical bugs they need to fix: then addressing just the "low hanging fruit" may be sufficient. Such organizations can tactically use static analysis to help them but they probably have larger issues they need to deal with. Of course, most companies fit somewhere in between the extremes.

However, it's important to revisit the marginal cost versus marginal benefit equation. Defects reported by static analysis tools point to specific lines of code where the problem may exist. This level of granularity makes the marginal cost of fixing a defect miniscule in comparison to fixing defects reported by QA or by customer support. No test cases, no debugging tools, no setting up the environment to recreate the problem is required. With a marginal cost so low, the pool of defects worth fixing suddenly expands. For most organizations, not every defect needs to be fixed, but the number of defects worth fixing is considerable.

Tools versus Solutions

2009-06-19T16:24:00.000-07:00

"People don't go to the hardware store to buy drills -- they go to buy holes"

When buying for a business, it always comes down to the bottom line - a problem must be solved, a pain alleviated, a cost saved or additional revenue made. Simply buying a tool won't solve any problem. When making a tool into a solution one must consider:

Does this tool fit into my process? How will it be integrated?
Who will install and configure this tool custom to my needs?
Who will use this tool? Does the users' process require them to use the tool? What are the incentives to use it and consequences for not using the tool?
Will the users be able to understand the tool? Will training be required and effective?
Who will administrate the tool? How will they gain the expertise to administrate it well?
Who will own the success and failure of the tool?
Who will be able to evolve the tool as the group's priorities and environment shift?

Organizations must consider how much they wish to invest in the tool in terms of time and resource. Building expertise within the organization can be expensive - almost always more than you expect. Some organizations go so far as to outsource everything that isn't within their core competence knowing that specialists can do a far better job. When taking on a tool, think about what it truly takes to bring a tool to success within your organization.

At Code Integrity Solutions, we've see this often. Organizations have the best of intentions when purchasing a tool but fail to make the necessary investment to make it a success. The word "investment" covers a lot of ground - it not only means license cost but all the other associated costs in making it happen. When purchasing a tool, no one has the intention to see it fail, and yet if it's not given the right priority and investment, it's chances of success are slim.

With static analysis, it's common to underestimate the needed cost. Static analysis is not yet an established part of every software development organization. Many companies don't know what it takes to administrate the tool, configure it properly and to how to integrate it well into their process. Many firms take a "hope" strategy by making the tool available as a service, not realizing that the people who end up using the tool are usually the people who least need it.

Yet, leading companies see the advantage it brings and are using it to gain a competitive advantage but many get just a fraction of its benefits. Usually the solution settles into an implementation that is "good enough for now". It's not a bad place to be temporarily but a tool is always a tool no matter what the problems are. A solution can solve much bigger problems if the right approach, right expertise and right investment are made.

Source Code Analysis and Continuous Integration - A Perfect Marriage

2009-06-07T10:00:00.000-07:00

Saw a good post last week on continuous integration (CI). Software development organizations are moving towards continuous integration in order to improve cycle times. By successfully integrating early and often, they minimize the risk of integration while reducing the overall integration and therefore delivery time.

In this post, the blogger mentioned his desire for his continuous integration server to do more, including:

Monitor source repository and build on changes
Run unit tests
Run various static analysis metrics
Publish assemblies for usage (could be to a test server, a file server or ftp server)
Build a clean copy of the database from setup scripts
Be able to commit versioned assemblies back into the repository for other teams to use (if needed)
Create visual graphs/trends for tracking
Be able to run a build which creates and packages an installer (for deployment)
Send notifications (emails/post to discussion boards, twits, etc) when there are issues

The thing I would add here is to include source code analysis by a static analysis tool for detecting defects and problems. Just as code should build cleanly with no compile errors, code should be free from poorly written code that results in obvious problems. Static tools are great at quickly finding logic type problems that are obviously wrong and can be easily fixed. In addition, for more sophisticated static analysis tools who can do a full-analysis across components, many complex integration oriented problems can be discovered. In addition, they don't require test cases, and can easily be automated into the process to find scores of problems with the newly written code. Minimizing the time between finding problems and fixing them increases the efficiency of building quality into the code by many times.

Some static tools though, can be noisy and therefore can be more trouble than they are worth. However, by skimming through the problems regularly, developers can catch the low hanging fruit and fix the obvious problems before a backlog builds and gets out of hand.

More advanced software development organizations make it a requirement to review and address every problem as they pop up but it often takes either a separate group or a few consultants to handpick the high priority stuff so that the developers can be most productive.

Static analysis for defect detection is increasingly becoming an integral part of the continuous build process for many leading companies. By cleaning up the code early and often, software development organizations improve their quality while also improving their overall efficiency.

Agile Software Development and Automated Static Analysis

2009-06-04T12:18:00.000-07:00

I was recently at an Agile conference where I heard from leading software development organizations. They described their experiences bringing Agile software development practices into their organizations. Some of these organizations were the typical, nimble Internet companies while others were large scale, healthcare and medical devices companies.

I was particularly keen on the aspects of software quality and Agile. Agile enables organizations to be more responsive to ever changing market demands. Agile enables organizations to have much faster cycle times, providing more of an iterative approach to delivery than in the traditional waterfall. With this comes features that better match the current market needs, faster delivery to market and in some cases improved quality.

Most of these organizations were fully in process with Agile software development but were playing "catchup" with testing. QA plays an interesting role in Agile environments. Many of these Agile practitioners went through the realization that QA no longer plays the same role that they did previously. Now that product owners are much more involved in the process, QA plays less of a role in determining whether the release meets requirements. QA instead plays more of a role on the delivery team - working closely with engineers to ensure that the proper tests are built and that the project passes all of the tests. Setting up the right acceptance tests are a critical role for the QA engineer. The goal of the developer is to write code that will pass the acceptance tests that are generally written at the beginning of the project.

The role of the developer also changes significantly. In order to meet quality needs, all agreed that test driven development (TDD) was the way to go. They also lamented about the challenges, such as developer resistance to writing and performing tests.

The biggest recommendation for testing was to automate tests as much as possible. Given the rapid cycle times, there is little time for traditional testing. Automating the test enables organizations to include testing within the sprint and deliver a production ready project at the end. Organizations that were in process of building good automation had to resort to follow on sprints after the sprint for testing. They would schedule a mini-sprint where developers and QA would test what was created in the prior sprint to make sure the quality was at a releasable state. Of course with a good test framework in place, developers and testers could automate their tests as they were "sprinting" so that during the next iteration, they could perform the same test without any manual intervention.

Static analysis also plays a big role here. Static analysis (or source code analysis) enables organizations to bring automated testing right to the developer. By analyzing their code before checking in their code, they can find potentially critical problems during the earliest part of the development cycle. Minimizing the time between coding and detection of problems makes developers makes them more efficient. Static analysis is an ideal automated testing tool because it requires no testcases and can generate a lot of good bugs in a short amount of time. Several leading organizations make it a requirement that all static analysis reports are at least looked at and fixed. Fixing a static analysis defect also usually takes a trivial amount of time.

Continuous integration was also another area of discussion. With multiple sprints on the same codebase going on in parallel, performing a system test on the entire system was a challenge for many organizations. Ensuring continuous integration enabled organizations to gain more efficiency from testing, especially with the fast cycle times many of these organizations have established and results in better code earlier.

Almost half of all software development organizations have moved or are moving to Agile. Building quality into the development process becomes even more paramount in an Agile process -- but it's not nearly as easy as it sounds. It's a long road to get there, fraught with not just tools issues but political and process issues -- but it's an integral part of making Agile work.

The Cost of Static Analysis Backlogs

2009-06-04T09:46:00.000-07:00

As we visit leading software development organizations around the world, one thing that we see again and again is a huge backlog of defects reported by their static analysis tool. Static analysis (or source code analysis) tools are great at quickly finding lots of critical problems in software code. No test cases and minimal setup are required and so there are few barriers to getting immediate value out of the tool. In addition, while analysis requires a significant amount of processing, the wall clock time required to get results is insignificant. To analyze millions of lines of code takes just hours resulting in thousands of potential problems.

But, many organizations run the tool, cherry pick a few problems and then leave the backlog there, sometimes sitting there for weeks and months. One would ask why would anyone do this:

there are real critical bugs in the backlog, some of which are likely to be showstoppers
bugs reported from static analysis tools are easy to fix, because they point to problems in right in the source code
license costs for static analysis tools are not insignificant
the value of a static tool is not in how bugs are found, but in how many good bugs are fixed

But there are real reasons why these backlogs sit unaddressed:

handling source code analysis results are not a part of the standard development process. When it becomes an extra step, it won't be done without the right incentives and consequences.
getting developers trained on code analysis takes an investment in time and resource. Most organizations don't have the political will and timing is often important.
source code analysis results vary in quality. Keep in mind that a source code analysis solution can only derive information from the source code and has no knowledge of environmental issues or usage/intent issues. Depending upon the quality of the tool, you may find that 10-20% of problems reported are high priority issues. The challenge, of course is knowing which ones are the 10-20%. Engineers must wade through lower priority issues, false positives and "don't cares" in order to find the good ones. Some tools have higher than a 50% false positive rate. Developers quickly tune out after a few false positives or "don't care" reports.
people don't have enough time. Everyone is being pulled in 10 different directions and the results from a static analysis solution get deprioritized and ignored. It becomes a thing they'll get to when they have time.

There is a high probability that there are true, highly critical bugs in the backlog, some of which may cost the organization hundreds of thousands of dollars once they surface. And the bugs are not hard to fix - it's just that there are so many of them - how can one get started?

We've seen many organizations deal with this problem. Some organizations have allocated resources on their team and created "bug fest" days to bring the backlog down to a manageable level. Other organizations have looked to outside firms such as ours to bring in expertise to kill a backlog quickly and efficiently. Others have "bit the bullet" and rolled static analysis to the general development organization with quota's and status reports to address the backlog.

Getting a backlog down to zero is an important part of starting a larger rollout of static analysis within the organization. In many organizations, the incremental changes to a codebase are minimal compared to the size of the overall legacy code. By addressing the backlog first, you get rid of any major potential problems and start with a clean slate to address any new defects that may arrive from code changes. It sets the stage for a successful rollout, declutters the reports coming from the tool and addresses potentially serious problems that may be lurking, unaddressed in your defect reports.

Depending upon the size of the backlog and the cost of a typical bug reported by a static analysis solution (typically crashes, memory leaks, buffer overruns, etc.) not addressing a backlog may be an extremely costly mistake.

The Challenges of Source Code Analysis

2009-05-26T08:52:00.000-07:00

What are the challenges of Source Code/Static Analysis (SCA)
SCA in practice is far from perfect. Simply buying a tool doesn’t solve any problems. Almost certainly making a tool available to your team is not going to result in adoption. In fact, it rarely does.

Think of the old adage: “you don’t go to a hardware store to buy a drill, you go to buy holes.” When you purchase an SCA tool you are buying fixed defects and addressed vulnerabilities. The tool, by itself, will get you only small part of the way there. The biggest challenges we’ve seen from hundreds of deployments are:

• Lack of integration into the process
• Misconfiguration and setup of technology to match business goals
• Lack of trust in what the tool is reporting
• Lack of management support
• Lack of commitment
• Insufficient staffing in number and in expertise
• Lack of ownership

When you purchase an SCA tool, your work has only just begun. Just as with most other tools, after the license and support costs, there are tangible and intangible costs to operate the tool effectively from both a technical and process standpoint. These investments have to be made otherwise, the tool will sit on the shelf, nobody will use it and the purchase will have been a waste. Luckily, these costs are not generally large and most of the issues can be addressed with some good planning and allocation of the right resources and budget.

Belief in the tools value
It is very difficult to calculate the ROI on an SCA tool. By definition, you are fixing problems before they go out into the field, and so it is difficult to predict the full extent of what the benefit would have been if you hadn’t used the tool. Several companies have tried to measure a tool’s value by correlating found results to past defects with some success. Results have varied greatly depending upon the codebase and the methodology used to analyze the results. Most see a big benefit but are not able to quantify it well.

In addition, SCA tools provide at best, educated guesses and therefore can be incorrect. This should be fully expected. For example, SCA tools have limited ability to account for environmental assumptions like configuration settings or usage constraints. False positives (incorrectly flagged defects) or irrelevant (“that will never happen” issues) can cause users to lose confidence in it. Defect reports pile up or worse yet, valid bugs are incorrectly marked as false because the developer didn’t trust the tool enough to investigate the report in enough detail. SCA reported defects then become low priority and remain unaddressed. Bugs and vulnerabilities you could have caught in development instead slip through into production.

The initial usage of the tool is an important time when lasting impressions are formulated. Many SCA tools report problems equally with little guidance as to what is higher priority than others. To be fair, priority is different for every organization and is usually comprised of severity of effect, how likely it will occur in production and how it affects functionality of the product. Therefore it is typically not calculated. To the developer using the tool with a large pile of bug reports, they don’t know where to begin. SCA tools do not prioritize well.

In software development organizations, there is a constant battle between what is urgent and what is important. SCA tools fit more under the important category and are often deprioritized from daily activities in an environment where everything is urgent. Some organizations unfortunately are more reactive than proactive. If everything is urgent, then in general you are understaffed. If people are pulled onto different projects and can never accomplish their goals, then you have not made a serious commitment to SCA. Get ready to put the SCA tool on the shelf.

Changing behavior is hard
Most organizations who use SCA require the software developer who owns the code to fix the problems reported. Getting participants to change their process is never easy. Extra steps are never welcome. Expect resistance from a vocal minority.

If the tool is not easy then it won’t get used. It has to be baked into the process. The tool has to be easy to use and the users trained. In addition, individuals have to feel that it adds value to their job. Incentives and consequences are extremely important. People who use the tool to accomplish business goals should be rewarded loudly. Likewise, there need to be penalties or consequences for not fixing problems reported from the SCA tool. Techniques like the “wall of shame” and MBO’s help align everybody to the common goal but must be used carefully. Displaying high bug counts can be a powerful political force but can cause undesired effects such as developers overzealously marking their defects as false or ignore’s. You have to balance the culture with the proper checks and balances to have the right effect.

Management support has to be strong and steady. It’s hard to prioritize quality and security over features and functionality. If management doesn’t believe it is a priority, then it will never succeed. Setting up the right process and the correct incentive structures are critically important to changing behavior. That being said, management should always be evaluating whether an investment in a tool is required or not. Having the right reporting in place can help management monitor the contribution SCA is making to the overall development process. Market conditions and therefore priorities change rapidly from day to day.

Giving the right support
Lack of ownership is a huge issue. In addition to management support, there needs to be ownership from both a tools operation perspective and from the user perspective. Often, a small percentage of a single resource is charged with owning the SCA tool. Most SCA tools come preconfigured to accomplish goals that may be misaligned with your own. Without the proper expertise and time commitment, most SCA administrators resort to using the SCA tools out of the box, resulting in higher false positives rates, incomplete code coverage, increased false negatives (where the tool should have reported a bug but didn’t) and suboptimal performance. SCA tools are sophisticated and can be tuned to optimum capabilities for a specific codebase. Expecting your SCA operator (who may have many other responsibilities) to be an expert administrator of the tool is unrealistic and a set up for failure.

Companies who gain the most value from SCA allocate expert staff with “static analysis” in their titles. They, in turn, took years to develop the level of proficiency that transform their organizations into big quality and security success stories.

At Code Integrity Solutions, we've worked with many leading organizations (both small and large) to help them overcome these challenges. But every organization is different. Let us know what challenges you've seen with source code analysis/static analysis in your organization.

Whitepaper Excerpt: The Benefits of Source Code Analysis

2009-05-22T10:30:00.000-07:00

I recently authored a whitepaper focused on how companies can succeed with their source code analysis efforts -- most importantly to match their source code analysis efforts with whatever their business-focused goals are. We at Code Integrity Solutions have seen dozens to hundreds of deployments. We have taken what we consider the best practices and put the information in a whitepaper. If you are interested in getting a copy of the whitepaper, please contact us at info@codeintegritysolutions.com. I included an introductory excerpt below:

What is Static Source Code Analysis
Source code analysis, sometimes known as static analysis, is a technology that uses sophisticated algorithms to analyze source code (such as C, C++, Java and C# code) and report on findings. Typically, source code analysis (hereby known as SCA) is performed to find defects or security vulnerabilities in source code or provide structural information on the architecture of the code. Many modern SCA tools can analyze all the paths in the code providing significantly more coverage than traditional dynamic testing tools.

What are the Benefits of SCA
Finding defects and vulnerabilities in source code enables software development organizations to develop better quality, more secure software cheaper. Finding and fixing a bug in the development process can be 100 times cheaper than having the bug slip through to production. SCA doesn’t require test cases (since it doesn’t operate on running code), can report hundreds and thousands of defects in a short amount of time and can report problems right in the source code, making it easier for software developers to fix.

Many organizations have used SCA to greatly improve the quality and security of released code. The business benefits include:

Better Engineering Efficiency
• increased developer productivity
• faster time to market
• better trained developers who write better quality code as part of the development process
• more maintainable code

Compliance
• adherence to industry standards
• compliance to government standards such as FDA requirements

Minimized Risk
• decreased quality and security risk
• increased confidence in offshore or outsourced code
• increased confidence in third party licensed code

Lower Business Cost
• lower cost of recalls, returns
• lower support costs, field issues
• increased customer satisfaction
• increased renewals

Stronger marketing claims
• higher availability and uptime statistics
• fewer security vulnerabilities

Just to name a few. In our next post, we'll talk about some of the challenges!

People, Process and Technology

2009-05-16T18:15:00.000-07:00

A former colleague of mine who was in enterprise sales used to always say, it's all about people, process and technology. I've found that to be a great way to frame how software tools not only get sold but also adopted. Having a great technology is only one small part of the battle for success. In fact, many companies become highly successful without having the best technology compared to the competition. Having a best in class product makes success easier, but it does not nearly guarantee success. Many piece are required to fall into place.

Process considerations are highly underrated. Many software companies clearly see the value their tool brings to their customers but they often don't see the changes that must be made in the way the target market adopts the solution. Making their customer's lives easier in one aspect of their job may introduce headaches in others just as easily. Making a change to accommodate another tool may be just too much for an already overstretched company to handle. Even if the result is a much more streamlined process, the initial cost of changing the organization to a newer process may be daunting in both effort and risk. Organizations may be experts at what they build, but may not be experts at instituting a new process that helps them get to their goals faster.

People issues always are the most dynamic and unpredictable. Organizations are made up of people who have their own MBO's and most importantly, their own personal agendas. Finding who owns the pain and who can alleviate that pain is critical. Often many initiatives fail to have a true owner, someone who has the skill, the fortitude and the necessary bandwidth to gain success. No wonder project success rates are abysmal. People are pulled on different projects as priorities change, and a string of unfinished initiatives are left hanging. Taking on an initiative also has its challenges. The risk of failure can be high and nobody wants to be associated with a failed initiative. Taking on risk is important for any company to survive -- but minimizing the risk through proper resource allocation, outside expertise and good planning is essential.

At Code Integrity Solutions, we have worked with many different customers in many different industries. It's amazing to see how such skilled individuals are being asked to do the impossible - to drive important initiatives in the face of constantly changing requirements. We've seen people responsible for numerous other important initiatives while also trying to drive a source code analysis intiative throughout a large organization. The challenges are great for companies trying to find solutions that satisfy not just their technology needs but also their people and process issues.

The Challenges (and Benefits) of Source Code Analysis

2009-05-02T14:09:00.001-07:00

Source code analysis, (or static analysis) has been used for many years now to perform detection of defects. Lint was one of the more popular early technologies for catching potential problems in code. Since then, the technology for finding defects has greatly improved enabling software development organizations to automatically detect problems with greater depth and accuracy than ever before.

The qualitative arguments have always been apparent. Finding and fixing a defect after a product has shipped can be orders or magnitude more expensive than if you found the problem during development. The cost of recreating a bug, debugging it and then fixing it is significantly larger than a tool pointing to where in the source code your problem is while you are developing it.

While many leading organizations agree that source code analysis is important, it is often too hard for these same organizations to fully bake it into their process and use it in a disciplined way. There are numerous reasons why:

Results from static analysis tools always have false positives. The tolerance level of a software developer for false positives is very low. If they see more false positives than true positives, then they begin to mistrust the results. The downward cycle begins as they investigate each new report as likely to be incorrect. We saw in several customer situations where developers will incorrectly ignore a report because they assumed it was incorrect and did not investigate it as fully as they should have.
Source code analysis is not yet part of the standard "reference architecture" for software development. Everyone needs a source control system, bug tracking system, editors, etc. but source code analysis is still not standard issue for most organizations. Adoption of such tools thus requires more effort in training and customization for rollout. In addition, organizations can utilize static analysis at many different points -- some in the IDE, some as part of a nightly build and some on a per sprint or release basis. Very few organizations have experienced static analysis practitioners to establish the right structure and processes for the organization.
Source code analysis can be more "vitamin" than "aspirin". When an executive comes to engineering complaining about the latest bug that is causing you to lose customers, there is an easy way to value the pain. Get it done and don't go home until you do. With a problem reported in static analysis, the pain levels are perceived to be much lower. It is often too difficult to understand how a source code flaw would manifest in production. A developer can be too far removed from how it might affect customers (and which customers). Business metrics are also hard because it isn't easy to cleanly measure field impact to defects fixed from static analysis.
In these economic times, organizations have to do more with less. Resources are constantly being pulled on to different projects. Organizational structures can change several times in a year. Priorities seem like they change with the wind to meet market conditions. Time and again, we see many organizations purchase tools but then get 10% of the value because they don't have the resources. Due to the way budgeting is performed, hiring headcount means the "wrong" kind of costs. The result is that no one has true ownership of making it happen and the tool effectively sits on the shelf.

It's a shame. Coverity has provided some metrics that they have been able to increase developer productivity by 12.5%, accelerate development cycles by 10-15% and reduce field defects by 30%. Caper Jones says that defects can be reduced by up to a factor of six. Although the mileage varies with each of these statistics, these are nevertheless significant numbers, particularly if your business is the technology that you're building. Many leading firms like Microsoft and Cisco and government organizations like the Air Force have made significant investments into static analysis and have successfully made it an integral part of their software development process. The road isn't hard but not easy either. The right combination of executive sponsorship, appropriate processes and incentives, accurate and effcient tools, ownership and in-house and outsourced expertise can help most any organization improve their software development organization significantly.

Greetings from Code Integrity Solutions (CIS)

2009-05-01T20:17:00.000-07:00

Code Integrity Solutions (CIS) is a provider of software integrity professional services. Started by former executives of Coverity, Inc., CIS's mission is to help leading software organizations gain the most from their static analysis investment.

Through this blog, we hope to provide you with news, insight and commentary on software quality, source code analysis, software security and a host of other software development issues. We invite your comments and a healthy debate on issues that software development organizations face every day.

Thanks for listening!