The Speed of Development

We were recently working with a company that had a very large codebase.  It took 2 hours to build and took another 4 to 24 hours to run a variety of tests such as unit tests and static analysis.  Developers were doing full builds locally on their sad, underpowered machines.  In this time of multi-core computing, developers were doing parallel computing the old fashion way:  using the lengthy build and test time to catch up on email.

In the worst case, you find a problem that you need to fix.  Guess what? Verifying whether the fix was sufficient creates yet another cycle.  Not only that, if the build systems have a problem, the turnaround on fixing those problems can literally fritter away critical days from the development cycle. Enlightened companies realize that this is a lot of time wasted but most importantly, creates a latency to the information you need to move code toward the finish line.  If you are going to fail, you should fail fast so you can fix it quickly.

Software is complex and brittle.  Not surprisingly, the infrastructure to build and test and manage the software can be similarly unreliable.  There is good reason:  software systems must support many multiple operating systems, integrations and devices and development cycle times have been compressed with Agile methodology.

After this company spent effort (and some hardware) to move their build to 15 minutes and all of their tests to run in 5 hours, their developers could at least get same day response times.  One more iteration meant that higher quality features could be checked in faster making their developers more productive.

They created a rough matrix:

Build/Test
========
15 minute = "immediate" feedback
30 minute = basically "on demand"
1 hour -> 3 hours = run multiple times per day
4 hours -> 6 hours = potentially same day response time
7 hours -> 14 hours = overnight process (ready by next morning)

Their next goal, through further optimizations was to get to multiple times per day.

Building a Better Build System

Here's an interesting and lengthy interview with Peter Smith, author of a new book, Software Build Systems: Principles and Experiences.  For the full interview, please see this link:

http://www.informit.com/articles/article.aspx?p=1702971

Another Tuning Story...

One customer that we worked with in the past has a medium sized codebase that handles financial-based transactions.  Their business deals with a lot of money and is highly regulated. Management has very high expectations that their software development team build their products with superior quality.  Not surprisingly they have turned to many modern techniques and technologies to keep up with the pace and scale of development.  They began using source code analysis a few years ago in order to detect and fix problems before release.

In their first run alone, the source code analysis tool reported over 20,000 potential defects.  They didn't know where to begin.  Even if it took 1 minute to go over each defect it would take over 40 business days to complete.  Realistically to query up the result, understand the defect report in the code, prioritize and document it briefly would take more on the order of 5-10 minutes per issue.  That's of course averaging the occasional really easy issue and really difficult issue.  At 7.5 minutes per issue that becomes 312 business days -- a significant investment by any standard.  Of course static analysis is one of the best ways to improve the systemic quality of your code as it is much less expensive to find and fix problems before release.

But there is a better way than using brute force to fix static analysis defects.  Tuning the analysis almost always results in significant improvements in the analysis quality, which means fewer false defects and more real defects.  After just a couple of days with this customer we knocked their queue down to about 9000 defects.  Actually, we knocked it down to about 8000 defects but then found an additional 1000 new defects that weren't being detected before.  The time savings, as you can imagine are large. The psychological effect was even better.  Although 9000 is still a mountain to climb, it's psychologically a lot better to hike up Kilimanjaro than scale Mount Everest.

Power to the Developer

Increasingly, we've seen software development organizations wanting to give power to their developers to find and fix problems prior to check-in.  There are several very good reasons for this:

• If management has instituted an acceptance criteria - such as no new static analysis defects, then naturally developers will be evaluated based on whether they fix static analysis defects.  A developer analysis gives power to the developers to find and fix problems locally before it becomes publicly visible in a system analysis.
• Finding and fixing problems before check-in makes the codeline higher quality.  Rather than checking in potential bugs, the code can be cleaned earlier.  Everyone pulls from higher quality code, particularly in agile and continuous integration environments.
• Giving developers analysis results right when they want it increases efficiency.  Rather than waiting for the nightly or even weekly results, minimizes context switching.  In addition, fixing a problem can generate another problem.  Being able to iterate multiple times quickly at one time is much more efficient than fixing and then waiting a day or even a week to see if no new problem has been created.

 Of course working from the desktop may require more coordination.  You have to ensure that the analysis being used locally is the same as being used for the system build -- consistency ensures that everyone is working towards the same goal.  Generally, you don't want developers to be analyzing either more or less than the established criteria.  Also system analyses can generate some different bugs than the desktop analysis - most of the times very slightly.  This is natural because the system analysis usually has more complete information and thus has better analyses at its disposal.  Still, this can cause confusion when a developer analysis doesn't pick up something that the system analysis picked up.

Desktop analysis is also an extra flow that requires additional time and training.  However, many would say that it is a small price to pay given the benefits.

Managing Branches in Static Analysis

In some respects, making software isn't much different than making sausage - it's messy and most would rather skip the details.  For those that choose the software development career, you have to live it everyday.  One area where I've seen significant mess is in the management of branches.  Delivery of software is seldom a one-time effort - with future versions expected to be delivered frequently.  Add to that the complexity of:

• software being available on different platforms and operating systems
• emergency software patches being made available
• different versions of the software customized to specific customers
• new major updates/rearchitectures/research simultaneously in the works as a production version

Pretty soon you may have branches all over the place.  Most organizations have a main branch which is the "gold master" with development branches being forked from it.  Many organizations will make it a requirement that the main branch should be release ready at any time.  Some organizations won't even allow edits on the main branch, instead requiring all changes to be made in forked development branch from the main branch.  Other organizations have developers happily chipping away at the main branch as well as creating multiple branches that may eventually be released.  Chaos ensues in these environments.

For those organizations that at least have a main and development branches, changes can be tested in the development branch prior to merge into the main branch.  Of course there may have been other merges that have taken place during the time that development branch was worked on and thus integration can be painful.  Software has lots of dependencies and when the assumptions are changed, software breaks in unpredictable ways.

The increasingly popular continuous integration process is designed to limit the pain of branch integration by doing it early and often.  Hand in hand with Agile, continuous integration has reduced the risk of complexity and therefore delay in releasing software.

Where does static analysis fit in?

There are lots of places where you can run the analysis.  You can run it regularly on the main branch (which probably already has a nightly scheduled build or continuous integration process).  This is a great way to quickly find out if something has broken, particularly during a merge.

Some organizations want to run it also on development branches - in a sense, creating a no new defects policy from branch pull to branch integration.  Of course when the development branch merges back into main, an analysis should be performed as part of the testing to see what breaks when things get merged.  This can be done on a manual basis.

Performing analysis on the development branch may not be entirely required -- it depends on when you want to be made aware of problems and when you want to enforce your acceptance criteria.  Analyzing every development branch will add to the IT costs as you will need more hardware to run the analyses on development branches.  For some organizations this is overkill - for others this is essential as their branches may be long lived and they need to meet their criteria as early as possible.

One additional way to still have the acceptance criteria enforced at the main branch but still enable developers to find and fix problems as they develop in the development branches is to have developers analyze the code they are working on in their own desktops.  Some static analysis tools provide a "desktop" version of their tool so that developers have the power to analyze the code they are working on and iteratively fix problems well in advance of a merge.  This enables them to find and fix problems early.  However, one must keep in mind that the development branch and particularly the code being worked on by a specific developer may vary quite a bit from the main branch and thus not all problems can be found at the desktop.  Frequent merging should help solve that problem.  Regardless, doing some analysis on the development branches enables developers to fix problems sooner which pays off in higher quality software earlier and better productivity.

Static Analysis Reporting For Success

When looking at effective reporting with static analysis, you have to consider the following:

• Who is the audience of the information?
• How do you turn data into actionable information?
• How does the information support business goals?
• How is this information delivered?

Static analysis improves software quality and security and so there are a number of potential "actors" that are affected by the information.  These may include:

• CEO/CTO
• VP of Engineering, VP of Products
• Director of Engineering, QA
• Manager of Engineering, QA
• Director of Tools, Infrastructure, Build/Release
• Manager of Tools, Infrastructure, Build/Release
• Architect
• Engineer / Developer / Programmer
• Tools, Build/Release Engineer
• QA Engineer
• Governance / Compliance Analyst
• Product Manager

Each individual has a stake in the successful usage of static analysis. Each constituent has their piece of the puzzle to manage or be aware of.  We unfortunately do not have room in the blog post to cover each of these roles.

Business Goals
Creating metrics without business goals as a framework will create wasted cycles, missing information and superfluous data.  Some common business goals that we see for static analysis are:

• No static analysis defects in code (sometimes called "Clean")

• The business goal is to improve software quality and security by addressing all potential issues reported by static analysis.
• Variations of this include:  "no critical or high priority issues" or no defects in a particular safety-critical component of the code
• This often requires it to be "managed" down, meaning that there should be planned downward slope to the number of defects.
• Targets may be established at certain milestones, for instance, the number of defects should be reduced by 10% for every release or month.  Possibly only the high and critical defects would be managed in this way.
• Some organizations even manage down "false positives" with the goal that the code should be written so cleanly that a static analyzer couldn't be confused.

• No new defects (sometimes called "No new harm")
  
  • The business goal is to at least keep software quality and security at status quo.  The business argument is that new code tends to be the buggiest and that legacy code has already had a chance to be "exercised."
  • Variations of this include:  "no new critical or high priority issues" or no new defects in a particular safety-critical component of the code.  Complexity metrics can also be used as indicators of trouble areas.
  • A baseline should be established (such as the beginning of a branch pull) and all new defects (or a high and critical subset) introduced through changes in the code should be fixed
• Improve Quality and Security Through Voluntary Usage
• The business goal is to provide a convenience to the developers to optionally fix software problems.  In practice though, usage typically decreases over time.
• Total number of defects fixed should be reported in order to quantify tool's worth

• Benchmark Against Competition
• The business goal is to be at parity or better than the industry benchmarks
• Defect density's are compared to publicly available data, such as open source.

Sample Reports

The number and types of reports that organizations use to more effectively use static analysis is too numerous to put in blog or article form.  We include just a few report concepts to give you an idea of the type of reporting that has proven useful to organizations:

Executive Dashboard

• Number of defects outstanding in current development branch.  Defects of course are likely to be the types of defects that are required to be fixed as part of an acceptance criteria.
• Trend of high priority defects that have been reported and that have been fixed since the branch pull.  Graph with pretty pictures always impress executives.
• Benchmark of quality level compared to industry standards
• Defect density (current and trend)
• Latest analysis run statistics - was it broken?  How much coverage was there?

Managerial dashboard

• Number of new defects reported since yesterday
• Number of open defects for each component owner
• Ranked list of number of fixes by component and by developer
• Trend in complexity and defect type
• False positive rate overall and by component and by developer
• List of open defects by priority and by age

Administrator dashboard

• Latest build statistics including lines of code analyzed, number of new defects, analysis times, coverage statistics
• New false positives marked to review for configuration changes
• Alerts for broken builds or builds that exceed certain performance and coverage thresholds

Architects

• Complexity trend over time
• Ranking of complexity by function
• New false positives marked for review to audit
• Defect density by component 

Developers

• New defects introduced in my code
• Outstanding defects in my queue
• Code complexity for my code
• Benchmarks against other developers in my company

And much, much more.

Information Delivery
How information is received plays an important role in how usable the system is.  In general, utilize whatever existing process flows exist - for instance:

• Generate email as alerts with a link to get to the details
• Create a new bug tracking entry for new defects (if they meet a specific criteria).  Some organizations group static analysis bugs into a single bug tracking entry
• Display information in an existing continuous integration dashboard
• Publish information in a wiki or equivalent intranet
• Display information in a code review tool
• Generate a PDF of information as part of an executive dashboard

The fewer additional steps required the more likely the tool will be used.  Creating separate, independent paths to the information will often cause the tool to not be used.

What's In Your Static Analysis Build Log and Why You Should Care

Static analysis tools are often used to help detect defects in source code. Simple in concept, they analyze the code and report potential defects. Of course, what everyone wants is a tool with zero false positive reports and zero unreported defects (or false negatives).  Developers in particular will be resistant to using a tool that they do not see as accurate.  Improving the tools accuracy often requires looking under the hood.

Like most build engines, your static analysis tool(s) create a log somewhere of the steps it takes to analyze the code. Most tools I'm familiar with call it a build log.  The build log, I believe, is the first important place to look.  Why should you look there?  There are errors that are reported in the build log that are not reported any where else. Specifically errors are reported by the compiler ( for each file it looks at) and then by the analysis engine itself.  In their default configuration not all compilation errors are visible in the defect list.

What this means is if you run an analysis and then look at just the results these tools provide you may not be getting the whole picture. For example if an entire file fails to compile, static analysis tools will often record this as a error and be duly reported to the end user or administrator.  However if just a single function or data structure is unable to be resolved or in some cases include files cannot be found, this may actually go unreported except in the log files.   Of course when details are lost so too will analysis fidelity.

To take a recent example I was working with a customer where the analysis build summary reported zero errors.  However as is my practice, I searched the build log for error messages from the analysis compiler. I found that the analysis compiler reported 75 errors in which it could not find some include files.  In this particular case when we added the include files to the compiler configuration we found that out of 1362 defect that were reported by the original analysis, 315 became "fixed", resulting in a reduction of approximately 25% in the defect density.

So what's an analysis administrator to do?  I recommend you count the errors in your build log at the end of every build and if any occur, evaluate them and attempt to fix them.  Usually it's pretty straight forward.  Sometime it takes more effort, sometimes it's not worth the effort. Usually the payoff is worthwhile.  In this case, it took a couple of hours to resolve 315 defect reports.  As a result I reduced the number of defects needed to be triaged for this team by 25% (without any loss of real bugs), saving significant time.

-- Kenneth Kron