tag:blogger.com,1999:blog-57049014584701355572024-03-13T09:11:09.324-07:00Code Integrity BlogCode Integrity Solutions (CIS) is a professional services firm that helps customer get the most out of their source code analysis and build tools investment.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.comBlogger51125tag:blogger.com,1999:blog-5704901458470135557.post-75287341461694137912011-06-10T17:01:00.000-07:002011-06-10T17:02:24.140-07:00The Speed of DevelopmentWe were recently working with a company that had a very large codebase. It took 2 hours to build and took another 4 to 24 hours to run a variety of tests such as unit tests and static analysis. Developers were doing full builds locally on their sad, underpowered machines. In this time of multi-core computing, developers were doing parallel computing the old fashion way: using the lengthy build and test time to catch up on email.<br />
<br />
In the worst case, you find a problem that you need to fix. Guess what? Verifying whether the fix was sufficient creates yet another cycle. Not only that, if the build systems have a problem, the turnaround on fixing those problems can literally fritter away critical days from the development cycle. Enlightened companies realize that this is a lot of time wasted but most importantly, creates a latency to the information you need to move code toward the finish line. If you are going to fail, you should fail fast so you can fix it quickly.<br />
<br />
Software is complex and brittle. Not surprisingly, the infrastructure to build and test and manage the software can be similarly unreliable. There is good reason: software systems must support many multiple operating systems, integrations and devices and development cycle times have been compressed with Agile methodology.<br />
<br />
After this company spent effort (and some hardware) to move their build to 15 minutes and all of their tests to run in 5 hours, their developers could at least get same day response times. One more iteration meant that higher quality features could be checked in faster making their developers more productive.<br />
<br />
They created a rough matrix: <br />
<br />
Build/Test<br />
========<br />
15 minute = "immediate" feedback<br />
30 minute = basically "on demand"<br />
1 hour -> 3 hours = run multiple times per day<br />
4 hours -> 6 hours = potentially same day response time<br />
7 hours -> 14 hours = overnight process (ready by next morning)<br />
<br />
Their next goal, through further optimizations was to get to multiple times per day.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-46157930069320408682011-05-27T16:26:00.000-07:002011-05-27T16:26:53.389-07:00Building a Better Build SystemHere's an interesting and lengthy interview with Peter Smith, author of a new book, Software Build Systems: Principles and Experiences. For the full interview, please see this <a href="http://www.informit.com/articles/article.aspx?p=1702971">link</a>: <br />
<br />
<a href="http://www.informit.com/articles/article.aspx?p=1702971">http://www.informit.com/articles/article.aspx?p=1702971</a>Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-56240494052394596382011-04-07T14:07:00.000-07:002011-04-07T14:07:15.989-07:00Another Tuning Story...One customer that we worked with in the past has a medium sized codebase that handles financial-based transactions. Their business deals with a lot of money and is highly regulated. Management has very high expectations that their software development team build their products with superior quality. Not surprisingly they have turned to many modern techniques and technologies to keep up with the pace and scale of development. They began using source code analysis a few years ago in order to detect and fix problems before release.<br />
<br />
In their first run alone, the source code analysis tool reported over 20,000 potential defects. They didn't know where to begin. Even if it took 1 minute to go over each defect it would take over 40 business days to complete. Realistically to query up the result, understand the defect report in the code, prioritize and document it briefly would take more on the order of 5-10 minutes per issue. That's of course averaging the occasional really easy issue and really difficult issue. At 7.5 minutes per issue that becomes 312 business days -- a significant investment by any standard. Of course static analysis is one of the best ways to improve the systemic quality of your code as it is much less expensive to find and fix problems before release.<br />
<br />
But there is a better way than using brute force to fix static analysis defects. Tuning the analysis almost always results in significant improvements in the analysis quality, which means fewer false defects and more real defects. After just a couple of days with this customer we knocked their queue down to about 9000 defects. Actually, we knocked it down to about 8000 defects but then found an additional 1000 new defects that weren't being detected before. The time savings, as you can imagine are large. The psychological effect was even better. Although 9000 is still a mountain to climb, it's psychologically a lot better to hike up Kilimanjaro than scale Mount Everest.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-32320277037912689632011-03-11T21:23:00.007-08:002011-03-11T21:23:30.970-08:00Power to the DeveloperIncreasingly, we've seen software development organizations wanting to give power to their developers to find and fix problems prior to check-in. There are several very good reasons for this:<br />
<ul><li>If management has instituted an acceptance criteria - such as no new static analysis defects, then naturally developers will be evaluated based on whether they fix static analysis defects. A developer analysis gives power to the developers to find and fix problems locally before it becomes publicly visible in a system analysis.</li>
<li>Finding and fixing problems before check-in makes the codeline higher quality. Rather than checking in potential bugs, the code can be cleaned earlier. Everyone pulls from higher quality code, particularly in agile and continuous integration environments.</li>
<li>Giving developers analysis results right when they want it increases efficiency. Rather than waiting for the nightly or even weekly results, minimizes context switching. In addition, fixing a problem can generate another problem. Being able to iterate multiple times quickly at one time is much more efficient than fixing and then waiting a day or even a week to see if no new problem has been created.</li>
</ul> Of course working from the desktop may require more coordination. You have to ensure that the analysis being used locally is the same as being used for the system build -- consistency ensures that everyone is working towards the same goal. Generally, you don't want developers to be analyzing either more or less than the established criteria. Also system analyses can generate some different bugs than the desktop analysis - most of the times very slightly. This is natural because the system analysis usually has more complete information and thus has better analyses at its disposal. Still, this can cause confusion when a developer analysis doesn't pick up something that the system analysis picked up.<br />
<br />
Desktop analysis is also an extra flow that requires additional time and training. However, many would say that it is a small price to pay given the benefits.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-31169824549715884752011-01-11T12:37:00.000-08:002011-01-11T12:37:25.832-08:00Managing Branches in Static AnalysisIn some respects, making software isn't much different than making sausage - it's messy and most would rather skip the details. For those that choose the software development career, you have to live it everyday. One area where I've seen significant mess is in the management of branches. Delivery of software is seldom a one-time effort - with future versions expected to be delivered frequently. Add to that the complexity of:<br />
<ul><li>software being available on different platforms and operating systems</li>
<li>emergency software patches being made available</li>
<li>different versions of the software customized to specific customers</li>
<li>new major updates/rearchitectures/research simultaneously in the works as a production version</li>
</ul>Pretty soon you may have branches all over the place. Most organizations have a main branch which is the "gold master" with development branches being forked from it. Many organizations will make it a requirement that the main branch should be release ready at any time. Some organizations won't even allow edits on the main branch, instead requiring all changes to be made in forked development branch from the main branch. Other organizations have developers happily chipping away at the main branch as well as creating multiple branches that may eventually be released. Chaos ensues in these environments.<br />
<br />
For those organizations that at least have a main and development branches, changes can be tested in the development branch prior to merge into the main branch. Of course there may have been other merges that have taken place during the time that development branch was worked on and thus integration can be painful. Software has lots of dependencies and when the assumptions are changed, software breaks in unpredictable ways.<br />
<br />
The increasingly popular continuous integration process is designed to limit the pain of branch integration by doing it early and often. Hand in hand with Agile, continuous integration has reduced the risk of complexity and therefore delay in releasing software.<br />
<br />
Where does static analysis fit in?<br />
<br />
There are lots of places where you can run the analysis. You can run it regularly on the main branch (which probably already has a nightly scheduled build or continuous integration process). This is a great way to quickly find out if something has broken, particularly during a merge.<br />
<br />
Some organizations want to run it also on development branches - in a sense, creating a no new defects policy from branch pull to branch integration. Of course when the development branch merges back into main, an analysis should be performed as part of the testing to see what breaks when things get merged. This can be done on a manual basis.<br />
<br />
Performing analysis on the development branch may not be entirely required -- it depends on when you want to be made aware of problems and when you want to enforce your acceptance criteria. Analyzing every development branch will add to the IT costs as you will need more hardware to run the analyses on development branches. For some organizations this is overkill - for others this is essential as their branches may be long lived and they need to meet their criteria as early as possible.<br />
<br />
One additional way to still have the acceptance criteria enforced at the main branch but still enable developers to find and fix problems as they develop in the development branches is to have developers analyze the code they are working on in their own desktops. Some static analysis tools provide a "desktop" version of their tool so that developers have the power to analyze the code they are working on and iteratively fix problems well in advance of a merge. This enables them to find and fix problems early. However, one must keep in mind that the development branch and particularly the code being worked on by a specific developer may vary quite a bit from the main branch and thus not all problems can be found at the desktop. Frequent merging should help solve that problem. Regardless, doing some analysis on the development branches enables developers to fix problems sooner which pays off in higher quality software earlier and better productivity.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-20816005691819874062010-12-15T14:27:00.000-08:002010-12-15T14:27:30.395-08:00Static Analysis Reporting For SuccessWhen looking at effective reporting with static analysis, you have to consider the following:<br />
<ul><li>Who is the audience of the information?</li>
<li>How do you turn data into actionable information?</li>
<li>How does the information support business goals?</li>
<li>How is this information delivered?</li>
</ul>Static analysis improves software quality and security and so there are a number of potential "actors" that are affected by the information. These may include:<br />
<ul><li>CEO/CTO</li>
<li>VP of Engineering, VP of Products</li>
<li>Director of Engineering, QA</li>
<li>Manager of Engineering, QA</li>
<li>Director of Tools, Infrastructure, Build/Release</li>
<li>Manager of Tools, Infrastructure, Build/Release</li>
<li>Architect</li>
<li>Engineer / Developer / Programmer</li>
<li>Tools, Build/Release Engineer</li>
<li>QA Engineer</li>
<li>Governance / Compliance Analyst</li>
<li>Product Manager</li>
</ul>Each individual has a stake in the successful usage of static analysis. Each constituent has their piece of the puzzle to manage or be aware of. We unfortunately do not have room in the blog post to cover each of these roles.<br />
<br />
<b>Business Goals</b><br />
Creating metrics without business goals as a framework will create wasted cycles, missing information and superfluous data. Some common business goals that we see for static analysis are:<br />
<ul><li>No static analysis defects in code (sometimes called "Clean") </li>
</ul><ul><ul><li>The business goal is to improve software quality and security by addressing all potential issues reported by static analysis.</li>
<li>Variations of this include: "no critical or high priority issues" or no defects in a particular safety-critical component of the code</li>
<li>This often requires it to be "managed" down, meaning that there should be planned downward slope to the number of defects.</li>
<li>Targets may be established at certain milestones, for instance, the number of defects should be reduced by 10% for every release or month. Possibly only the high and critical defects would be managed in this way.</li>
<li>Some organizations even manage down "false positives" with the goal that the code should be written so cleanly that a static analyzer couldn't be confused. </li>
</ul></ul><ul><li>No new defects (sometimes called "No new harm") <br />
<ul><li>The business goal is to at least keep software quality and security at status quo. The business argument is that new code tends to be the buggiest and that legacy code has already had a chance to be "exercised."</li>
<li>Variations of this include: "no new critical or high priority issues" or no new defects in a particular safety-critical component of the code. Complexity metrics can also be used as indicators of trouble areas. </li>
<li>A baseline should be established (such as the beginning of a branch pull) and all new defects (or a high and critical subset) introduced through changes in the code should be fixed</li>
</ul></li>
<li>Improve Quality and Security Through Voluntary Usage</li>
<ul><li>The business goal is to provide a convenience to the developers to optionally fix software problems. In practice though, usage typically decreases over time.</li>
<li>Total number of defects fixed should be reported in order to quantify tool's worth </li>
</ul></ul><ul><li>Benchmark Against Competition</li>
<ul><li>The business goal is to be at parity or better than the industry benchmarks</li>
<li>Defect density's are compared to publicly available data, such as open source.</li>
</ul></ul><b>Sample Reports</b><br />
<b> </b><br />
The number and types of reports that organizations use to more effectively use static analysis is too numerous to put in blog or article form. We include just a few report concepts to give you an idea of the type of reporting that has proven useful to organizations:<br />
<br />
<i>Executive Dashboard</i><br />
<ul><li>Number of defects outstanding in current development branch. Defects of course are likely to be the types of defects that are required to be fixed as part of an acceptance criteria.</li>
<li>Trend of high priority defects that have been reported and that have been fixed since the branch pull. Graph with pretty pictures always impress executives.</li>
<li>Benchmark of quality level compared to industry standards</li>
<li>Defect density (current and trend)</li>
<li>Latest analysis run statistics - was it broken? How much coverage was there?</li>
</ul><i>Managerial dashboard</i><br />
<ul><li>Number of new defects reported since yesterday</li>
<li>Number of open defects for each component owner</li>
<li>Ranked list of number of fixes by component and by developer</li>
<li>Trend in complexity and defect type</li>
<li>False positive rate overall and by component and by developer</li>
<li>List of open defects by priority and by age</li>
</ul><i>Administrator dashboard</i><br />
<ul><li>Latest build statistics including lines of code analyzed, number of new defects, analysis times, coverage statistics</li>
<li>New false positives marked to review for configuration changes</li>
<li>Alerts for broken builds or builds that exceed certain performance and coverage thresholds</li>
</ul><i>Architects</i><br />
<ul><li>Complexity trend over time</li>
<li>Ranking of complexity by function</li>
<li>New false positives marked for review to audit</li>
<li>Defect density by component </li>
</ul><i>Developers</i><br />
<ul><li> New defects introduced in my code</li>
<li>Outstanding defects in my queue</li>
<li> Code complexity for my code</li>
<li>Benchmarks against other developers in my company</li>
</ul>And much, much more.<br />
<br />
<b>Information Delivery</b><br />
How information is received plays an important role in how usable the system is. In general, utilize whatever existing process flows exist - for instance:<br />
<ul><li>Generate email as alerts with a link to get to the details</li>
<li>Create a new bug tracking entry for new defects (if they meet a specific criteria). Some organizations group static analysis bugs into a single bug tracking entry</li>
<li>Display information in an existing continuous integration dashboard</li>
<li>Publish information in a wiki or equivalent intranet</li>
<li>Display information in a code review tool</li>
<li>Generate a PDF of information as part of an executive dashboard</li>
</ul>The fewer additional steps required the more likely the tool will be used. Creating separate, independent paths to the information will often cause the tool to not be used.<br />
<ul></ul>Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-34652808573995120892010-11-23T15:26:00.001-08:002010-11-24T08:15:50.813-08:00What's In Your Static Analysis Build Log and Why You Should Care<style>
@font-face {
font-family: "Arial";
}@font-face {
font-family: "Arial Unicode MS";
}p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman"; }p.MsoBodyText, li.MsoBodyText, div.MsoBodyText { margin: 0in 0in 6pt; font-size: 12pt; font-family: "Times New Roman"; }span.BodyTextChar { font-family: "Arial Unicode MS"; }div.Section1 { page: Section1; }
</style> <br />
<div style="margin-bottom: 0.0001pt;"><span style="color: black; font-family: Arial; font-size: 10pt;">Static analysis tools are often used to help detect defects in source code. Simple in concept, they analyze the code and report potential defects. Of course, what everyone wants is a tool with zero false positive reports and zero unreported defects (or false negatives). Developers in particular will be resistant to using a tool that they do not see as accurate. Improving the tools accuracy often requires looking under the hood.</span></div><div style="margin-bottom: 0.0001pt;"><br />
</div><span style="color: black; font-family: Arial; font-size: 10pt;">Like most build engines, your static analysis tool(s) create a log somewhere of the steps it takes to analyze the code. Most tools I'm familiar with call it a build log. The build log, I believe, is the first important place to look. Why should you look there? There are <i>errors</i> that are reported in the build log that are not reported any where else. Specifically errors are reported by the compiler ( for each file it looks at) and then by the analysis engine itself. In their default configuration not all compilation errors are visible in the defect list.</span><br />
<div style="margin-bottom: 0.0001pt;"><span style="color: black; font-family: Arial; font-size: 10pt;">What this means is if you run an analysis and then look at just the results these tools provide you may not be getting the whole picture. For example if an entire file fails to compile, static analysis tools will often record this as a error and be duly reported to the end user or administrator. However if just a single function or data structure is unable to be resolved or in some cases include files cannot be found, this may actually go unreported except in the log files. Of course when details are lost so too will analysis fidelity.</span></div><div style="margin-bottom: 0.0001pt;"><br />
</div><div style="color: black; margin-bottom: 0.0001pt;"><span style="font-family: Arial; font-size: 10pt;">To take a recent example I was working with a customer where the analysis build summary reported zero errors. However as is my practice, I searched the build log for error messages from the analysis compiler. I found that the analysis compiler reported 75 errors in which it could not find some include files. In this particular case when we added the include files to the compiler configuration we found that out of 1362 defect that were reported by the original analysis, 315 became "fixed", resulting in a reduction of approximately 25% in the defect density.</span></div><div style="margin-bottom: 0.0001pt;"><br />
</div><div style="margin-bottom: 0.0001pt;"><span style="color: black; font-family: Arial; font-size: 10pt;">So what's an analysis administrator to do? I recommend you count the errors in your build log at the end of every build and if any occur, evaluate them and attempt to fix them. Usually it's pretty straight forward. Sometime it takes more effort, sometimes it's not worth the effort. Usually the payoff is worthwhile. In this case, it took a couple of hours to resolve 315 defect reports. As a result I reduced the number of defects needed to be triaged for this team by 25% (without any loss of real bugs), saving significant time.</span></div><div style="margin-bottom: 0.0001pt;"><br />
</div><div style="margin-bottom: 0.0001pt;"><span style="color: black; font-family: Arial; font-size: 10pt;">-- Kenneth Kron </span> </div>Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-72309693168047034802010-11-09T06:51:00.000-08:002010-11-09T06:51:17.438-08:00Static Analysis Metrics - A Window into DevelopmentYou can't manage what you can't measure right? In addition to detecting problems in source code, static analysis can be used to produce metrics that provide insight into productivity, code quality, security and more. We all know the <i>dangers of managing solely by metrics</i>, yet, metrics serve as a highly useful window into your software development process. In this post, we most certainly won't focus on the general challenges of using metrics because they've been written about in many other forums. We will discuss some static analysis metrics here that when pieced with other information can provide useful information.<br />
<br />
Static analysis is typically integrated at the system build level. These days, most organizations build their code on a frequent basis - which are convenient points of time to generate static analysis results. If you are running a nightly build, you can get static analysis output on a daily basis. If you run it on a continuous integration basis you get output at the frequency you choose (sometimes static analysis takes longer than a continuous integration cycle if you haven't tuned it but then simply perform static analysis on every 2nd or 3rd cycle). Metrics can be used to get a sense for where you are (status) and point to potential problem areas too (alerts). Thresholds can be set to spur action - such as block a release, or trigger a code review or design review.<br />
<br />
Every organization has different priorities and thus every organization should have a different set of metrics that they should track to maintain alignment with goals. We list just a few common metrics that can come from static analysis tools.<br />
<br />
<b>Management Level Metrics</b><br />
<br />
Most static analysis tools can easily generate output such as:<b> </b><br />
<ul><li>Lines of code, number of new lines of code, churn </li>
<li>Number of defects outstanding, number of new defects introduced<br />
</li>
<li>Defects per thousand line of code (defect density)</li>
<li>Number of defects fixed so far, number of defects fixed in last period</li>
<li>Comments per line of code </li>
<li>Cyclomatic complexity, function points</li>
</ul>Supporting Metrics to Ensure Compliance<br />
<ul><li>Code coverage for tests (100% or near 100% should be attained)</li>
<li>Average time to resolve issues</li>
<li>Number of defects marked as false positives (or low priority)</li>
</ul>A couple of considerations about this data:<br />
<br />
<b>Breakdowns</b><br />
The entire codebase should be measured and tracked however convenient breakdowns can be created to make the information more actionable. For instance, data could be collected on a team by team basis (or component by component). Individuals metrics can also be calculcated in order to hone in on specific problem areas or better yet, to highlight improvement areas worth rewarding. In addition, team leads may be interested in tracking information on a function by function basis.<br />
<br />
All defect related metrics should be broken down by priorities and/or severities. Metrics that lump in low priorities become diluted. This is particularly important if you have an acceptance criteria that requires all critical and high defects to be fixed before release.<br />
<br />
<b>What Is Measured</b><br />
Because static analysis usually operates at the build level, you can get meaningful statistics on a per codebase perspective. Thus if you have multiple targets (e.g. different operating systems, devices, etc.) you can produce statistics for each. You can find build target specific bugs as well as bugs in shared code.<br />
<br />
<b>Absolute Versus Relative</b><br />
A number without context provides almost no value. Let's say the tool spits out a defect density rate of 1 bug per 1000 lines of code. What can you do with the information? The data becomes rich information when you are able to compare it to other numbers. The most common way to use these metrics is to benchmark against industry standards. <a href="http://www.coverity.com/">Coverity</a> has an interesting database of open source defects from their Department of Homeland Security grant. While comparing your codebase with open source statistics may not be ideal (wouldn't it be great if you could benchmark against your competitors), it provides at least some real world data that you can use to compare and contrast. Are you above or below the average and by how much?<br />
<br />
Another common way to interpret the numbers is to evaluate them over time. Take a baseline - a snapshot in time and then measure against that baseline to see how you are trending. A convenient time to take a baseline is at the beginning of a branch pull. Some organizations have the goal of toeing the line - meaning "no new harm." Quite simply the goal is to maintain the same level of quality as prior to any new development efforts. Others at least want to see improvement over time - showing that they are making progress. The high level defect density as well as the overall defect number are very typical KPI's (key performance indicators) chosen by development organizations.<br />
<br />
<b>Frequency</b><br />
Note again that the frequency is as often as you run static analysis. As part of a system build, this is typically run regularly, either every evening, every few days or every continuous integration cycle. Advanced organizations also provide static analysis at the developer level so that they can run on-demand static analysis. <a href="http://www.klocwork.com/">Klocwork</a> has some strong capabilities in this area. In this way developers can proactively identify and fix potential problems prior to check-in. Giving the power to the developers to view and improve the metrics as they work, results in better system-level metrics faster and much less chance of having to deal with an emergency later in the development cycle.<br />
<br />
<b>Trending</b><br />
All data should be stored and accessible to get a historical context. Graphical charts enable a quick interpretation of results. Historical context helps you see how you are doing relatively. Start with a baseline and see how you are doing from that point onward.<br />
<br />
<b>Data Consistency</b><br />
Static analysis tools are complex, with many settings that are tweakable in order for it to understand the myriad of different codebases that it can analyze. Think what kind of bugs would be useful to find in a 10,000 line of code embedded device (like a stack overflow) versus a 10 million line of code server application (like a concurrency bug). If not set up properly, the static analysis tool can create challenges in keeping the data stable so that you get consistent numbers to compare over time. For instance, if you upgrade the static analysis tool, new and improved checkers may suddenly change the defect count. New checkers will find new bugs, which may falsely signal that there was a sudden increase in the number of defects in your codebase. Those bugs were always there :-). Similarly, an improved checker may suddenly report many fewer false positives but maybe at the cost of a few real bugs that are no longer found.<br />
<br />
If analyses can be run in multiple places (for instance, developers running the analysis tool locally on their machine) then discrepancies can occur if the configurations and the code being analyzed are not identical. Even changing some settings that you wouldn't imagine could possibly cause a data consistency problem can mysteriously change the numbers. While static analysis tools are designed to find bugs, that doesn't mean they don't have bugs themselves. These tools are doing a lot of complex analyses behind the scenes and don't always provide consistent results in every possible scenario.<br />
<br />
In a future blog post, we'll look at some sample dashboards so that you get a sense for what can be used for reporting and how these data points can be used.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-84661457271490998992010-10-24T13:51:00.000-07:002010-10-24T18:44:15.387-07:00Technical Debt And Static Analysis<a href="http://www.castsoftware.com/">CAST Software</a>, a maker of software analysis and measurement solutions <a href="http://www.castsoftware.com/Resources/Research-Labs.aspx">published</a> some interesting findings on the quality and security of software applications across a number of different industries and applications. (75 organizations, 288 custom applications).<br />
<br />
While clearly, vendor funded need to be taken with a grain of salt (for instance the calculation was based on their tool's defect density and a fix rate presumably from their experience), the approach and findings do cast an interesting light on the quantification of quality initiatives. To start with, they define "technical debt" as "cost of the effort required to fix problems that remain in the code when an application is released to operation. Like financial debt, Technical Debt incurs interest in the form of the extra effort it takes to maintain and enhance an application due to the structural quality flaws left in that code."<br />
<br />
In summary, some of the findings are:<br />
<ul><li>The average technical debt was found to be about $1M with the average code size being 374,000 lines of code. </li>
<li>The average maintenance cost per line of code was $2.82 assuming a $75/hour average, fully loaded wage for a developer and that a bug takes about an hour to fix.</li>
<li>C/C++ applications had wider variability but on average had lower technical debt that some of the Web technologies such as Java and .NET.</li>
</ul>My assumption though is that this number was calculated from the defects that came from the Cast tool that would presume to be fixed and therefore is a subset of the actual technical debt for an application (presumably there are many other bugs that would need to be fixed, not found by a static tool). If true, I would conclude that they are saying that if you use the Cast tool to the level they suggest, that it would save $1M in technical debt, an amount that over time would grow larger due to interest.<br />
<br />
One could argue either way whether the number is bigger or smaller. On the one hand, there is a cost to fixing, starting from the license cost all the way to the extra hardware, training and usage cost. On the other hand, the calculation of technical debt was only done based on developer time and not on the effect of a defect, which can be many multiple times more costly than simply the time it takes to fix a bug. We sometimes call these the "million dollar bug" and it varies significantly among industries and application types. While this cost may not technically be a part of the technical debt calculation, you may wish to figure the cost of a defect in deciding whether to use static analysis or not. Sure, these defects are few and far between but if you can find just one...<br />
<br />
Either way, static analysis does appear to save money and help developers produce better quality software - the hard part has been in measuring it. Technical debt is one interesting way to measure it.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-5486046356606742402010-10-08T23:16:00.000-07:002010-10-09T06:50:23.504-07:00Why Have Static Analysis Tools Not Yet Hit the MainstreamOver the past few weeks, there has been an interesting and lively discussion going on in a <a href="http://www.linkedin.com/groups?mostPopular=&gid=1973349">Linkedin group</a> focused on static analysis. The original question was along the lines of "Why hasn't static analysis penetrated the market further?" The conversation has reached 97 comments and counting from developers, managers and vendors alike.<br />
<br />
One <i>could</i> argue that static analysis is actually fairly pervasive, particularly among companies who produce shipped software as their business and where quality and security is critical. If you count the organizations who use static analysis (and ignore some overlap) it would number well into the thousands. In addition, opensource tools like Findbugs and others have a decent following as do free static analysis tools bundled with IDE's and compilers. And yet, static analysis still has a way to go before it becomes a part of the standard toolchain that most every organization uses. Getting static analysis used effectively within the organization is a second order challenge once an organization decides to use static analysis.<br />
<br />
I will attempt to summarize some of the interesting points that were surfaced in the sometimes heated discussion. Some I agree with and some I don't.<br />
<ul><li>Static analysis tools can be noisy - producing false positives. Does static analysis consistently produce real issues? Are the issues reported comprehensible and actionable?</li>
<li>Static analysis tools were overhyped many years ago causing expectations to be set too high. </li>
<li>Developers are protective of their code and don't like a tool poking at their "treasured creations".</li>
<li>Static analysis vendors are still small or medium-sized and therefore don't yet have the marketing budgets to communicate to the broad market. HP and IBM's recent acquisitions of Fortify Software and Ounce Labs may change this at least in the security space.</li>
<li>Static analysis is often viewed as as an additional step after a "successful build" when it should be release criteria for passing or failure. </li>
<li>The term "static analysis" is boring and doesn't reflect the sophistication of the analysis</li>
<li>A multitude of defects are reported and developers don't know where to start or how to create a real process that addresses the issues</li>
<li>Ignorance of the value of tools in the software development process. The all too frequent way that managers and senior staff treat software as an art that cannot be improved through better process and tools.</li>
<li>Need a clearer demonstration of the value static analysis provides (the benefit minus the costs).</li>
</ul>Static analysis is growing particularly among organizations where quality issues and security vulnerabilities are most damaging but it will take a combination of improvements to the technology, changes to the attitude of development organizations and industry accepted best practices to take static analysis past the chasm towards the mainstream. On the services front, we help many organizations overcome these issues through best practices in people/process/technology, mentoring, training, support and a multitude of other services. When done right, organizations who take on static analysis do gain a competitive advantage. These companies not only improve the quality and security of their products, they do so often with an overall productivity gain. Invest in the right way and get some payback. Approach it with the wrong steps and a complacent attitude and then you are left to cross that chasm yourself.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com2tag:blogger.com,1999:blog-5704901458470135557.post-42346453361179567342010-09-24T10:49:00.000-07:002010-09-24T10:50:09.035-07:00False Positives versus Intentional'sBugs need to be categorized so that the appropriate action can be taken. The traditional bug reported by customers, support, field engineers, etc. are examined by developers to determine if it is a bug. If it is a bug, its priority and severity are usually assessed. If it is not a valid bug report it may have been rejected as not a bug, unfixable, already fixed, duplicate or any of a number of different categories that say that no further work will be done on it.<br />
<br />
Static analysis defects have a slightly different flavor. Static analysis reports potential defects, meaning an inconsistency or flaw has been detected that needs to be verified by a developer or review team and handled appropriately. Bugs determined to be real ("true positives") need to be assessed for priority - usually a combination of type of bug reported, whether it is on a specific conditional path/exception path, what part of the code it is found in (new code/old code/component/3rd party library), etc. Static analysis solutions may also include a "score" to give further information to help prioritize it.<br />
<br />
Some static analysis results however may be "false positives" which are essentially "not a bug" characterizations. Herein lies a distinction. Vendors often like to make a distinction between a "false postive" and an "intentional" status. A "false positive" signals a situation where the analysis incorrectly identified a problem as a bug. If you will, it's a bug or limitation of the static analysis tool. An "intentional" status signals a situation where a valid bug is reported but that the developer had intentionally coded it this way and so no further action is required. The presumption here is that for "intentional's", the analysis did what it was supposed to do correctly.<br />
<br />
Some organizations lump everything that they don't want to fix into the "false positive" category. If they weren't going to change any code, then it shouldn't have been reported. Other organizations take a more sophisticated approach by making this distinction - the presumption being that intentional's should be reviewed, like a sanity check. A variety of reasons why they categorize it as such include:<br />
<ul><li>wanting to get an accurate "false positive rate" for the tool. Lumping "intentional's" artificially increases the false positive rate</li>
<li>the workflow for "intentional" and "false positive" may be different. While both should be reviewed/audited, a static analysis administrator may review "false positives" for tuning and configuration opportunities. False positives should be reported back to the vendor for fixing as well. "Intentional's" may be reviewed by architects to understand what strange coding practices are taking place. Intentional's typically need a good reason to explain why it was coded in the way it was.</li>
</ul>Of course the advantage to finer grain categorizations have to be weighed with the cost of explaining the distinctions. There is a cost to data acquisition, but if you value the ability to measure the tool's effectiveness and have a process for improving your analysis and improving your code through examination of intentional's then it may be well worth it.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-39356800967684672522010-08-17T11:01:00.000-07:002010-08-17T11:01:31.201-07:00Power of 10 for Safety Critical CodeJust reminding programmers of a good set of safety critical programming rules from the rocket scientists at JPL/NASA. Gerard Holzmann wrote the <a href="http://www.spinroot.com/p10/">Power of 10</a> rules which are a set of good programming practices that should be considered for programs that demand high levels of quality.<br />
<br />
The rule that most applies to static analysis is "Rule 10" which is noted below:<br />
<br />
"Compile with all warnings enabled, in pedantic mode, and use one or more modern static source code analyzers. All code must be compiled, from the first day of development, with all compiler warnings enabled at the compiler's most pedantic setting. All code must compile with these setting without warnings. All code must be checked on each build with at least one, but preferably more than one, state-of-the-art static source code analyzer and should pass the analyses with zero warnings." <span style="font-size: x-small;"><i> (Rule 10 of Power of 10 Rules)</i></span><br />
<br />
Gerard notes that "there is no excuse for any serious software development effort not to make use of this technology." Some may view the "rule of zero warnings" as draconian but Gerard notes that if the analysis gets confused enough to report a problem, then the code should be rewritten to be "trivially valid." Good code doesn't need to be complex. If the code is clean, it's not only much less error-prone but also much easier to manage and maintain ongoing. Think also about the next developer who may be inheriting the code. Static analysis will find bugs but will also help code be more readable and maintainable.<br />
<table><tbody>
<tr><td valign="top" width="10%"><br />
</td><td width="5%"><br />
</td><td><br />
</td><td><br />
</td><td><br />
</td></tr>
</tbody></table>Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com2tag:blogger.com,1999:blog-5704901458470135557.post-80425074588070281252010-08-06T14:32:00.000-07:002010-08-10T08:50:38.349-07:00Dr. Dobbs Article on Porting to 64-bit PlatformsIrving Rabin, Senior Consultant at <a href="http://www.codeintegritysolutions.com/">Code Integrity Solutions</a> recently published an article on porting to 64-bit platforms.<br />
<br />
You can read it on Dr. Dobbs at: <a href="http://www.drdobbs.com/cpp/226600156;jsessionid=ZVI3ENP4W0F0VQE1GHOSKH4ATMY32JVN?queryText=porting+64-bit+platforms">Porting to 64-bit Platforms</a>.<a href="http://www.drdobbs.com/cpp/226600156;jsessionid=ZVI3ENP4W0F0VQE1GHOSKH4ATMY32JVN?queryText=porting+64-bit+platforms"></a><br />
<br />
Enjoy and have a great weekend! <br />
<br />
<span style="font-size: 13px;"><br />
</span> <br />
<span style="font-size: 13px;"><br />
</span>Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com1tag:blogger.com,1999:blog-5704901458470135557.post-61162400845278369232010-08-02T12:03:00.000-07:002010-08-02T12:03:41.270-07:00Getting Buy-In For Static Analysis From Management.I recently read an interesting article (<a href="http://www.itmanagement.com/features/8-ways-to-get-buyin-061708/">8-Ways To Get Buy In From Management</a>) on how to get buy-in from IT Management. While the document is mostly focused on IT related activities, there are some valuable lessons that can be learned for any technical undertaking within an organization. In this entry, we discuss some of the issues focused on static source code analysis.<br />
<br />
<b>1. Learn what management values. </b><br />
<br />
Software development management has a different set of concerns than most individual contributors have. Clearly there <i>should</i> be alignment between management and worker-bees but that is too often not the case. Static source code analysis will help you find bugs so that developers can fix them early in the development process. From management's perspective, static analysis is almost a no-brainer. Caper Jones' estimates that static analysis can have a defect removal efficiency that can top 95% (presumably in a well configured, effective system and process). Other ways to describe the benefits include:<br />
<ul><li>Fewer defects and vulnerabilities in the field, meaning lowered support costs, fewer patches and better customer satisfaction</li>
<li>Increased developer productivity</li>
<li>Lower risk of late releases</li>
</ul>Particularly for technology companies, arming developers with good tools so that they increase their effectiveness is a must. How management couches the benefits is industry and company dependent.<br />
<br />
<b>2. Propose projects in management terms, not technical terms.</b><br />
<br />
The benefits in #1 are real, but how tangible are they? The more the benefits can be quantified, the easier it is to make a decision whether to institute static analysis or not. A convenient way to break down benefits is to look at increases in opportunities, lowering of costs and reduction in risk. Static analysis typically does not increase opportunities directly but an argument can be made that any reduction in development time enables more value-added features to be built. Lowered costs are definitely significant. Lowered support costs (in technical support, developer support (patches, etc.)) as well as increased productivity can be estimated. Many companies use automated source code analysis to find problems quickly in an fast-paced Agile environment. This enables increased throughput overall. Reduction in risk can take many forms - the risk of late release, the risk of poor quality, the risk of lowered customer satisfaction, the risk of failing compliance etc.<br />
<br />
<b>3. Show management case studies and examples that support your point for cost savings or revenue enhancement.</b><br />
<br />
Certainly leverage the vendor. While they are not impartial, they often have good case studies. References are also very important. A lot of companies are using static analysis so leverage your contacts. Better always to do backdoor references so you can get a balanced view. People seem happy to discuss their experiences.<br />
<br />
<b>4. Find ways to minimize the project's expense.</b> <br />
<br />
While reducing license cost is a subject better left for closed door negotiations, there are ways to maximize the benefit you get from static analysis tools. Make sure you take the time to tune it for your needs, integrate it into your process (both technically and organizationally) and try to tie metrics of value to release criteria. Another way to get more value is to use the tool's extensibility capabilities to build your own custom checkers so that you leverage the analysis engine and the place in the workflow. Wouldn't it be great to find and fix additional problems, violations in coding standards and more borrowing from what you are already doing. When more people use the tool, more value is derived. To get more people to use the tool, make it super easy to use, make the results as relevant as possible and make it clear as possible what is expected of all participants.<br />
<br />
<b>5. If your department's IT budget isn’t big enough, get help from another department.</b><br />
<br />
A lot of folks can benefit from static analysis. We've seen static analysis budgets come from not only the development organization but also from quality assurance, central tools, security groups and even governance organizations. While they are not users, they can clearly benefit from the results and may help fund its usage. Static analysis can also come from other overarching budgets such as from a quality and security initiative or projects related to moving to Agile / continuous integration and software development infrastructure automation.<br />
<br />
<b>6. Break up the project into easy-to-swallow pieces.</b><br />
<br />
Breaking up projects into pieces not only makes it easier to swallow, it reduces risk. As you see success in the early projects, it gives you knowledge (and confidence) to continue with the other phases of the project. Start with a pilot group and see how that goes. Set clear metrics for what is considered success for that pilot and work to meet it. You'll learn significantly from the project to help you make better decisions down the line. Use of static analysis as an integral part of a software development process is relatively new and thus haven't been ingrained into the DNA of most people and organizations. That's changing but we're not quite there yet.<br />
<b><br />
7. Look for ways to outsource part of the project. </b><br />
<br />
This advice may be a bit self-serving, but I will mention that it never ceases to amaze me how many companies opt to do everything in-house. The short term benefit from doing this is that organizations build some expertise in-house while not incrementally add budget. But, they rarely get a best practices implementation. In addition, they have to give up some opportunity cost of having that developer take on a nonstrategic project versus something more relevant to the goals of the company. In addition, by doing it all in-house it often takes a lot longer for them to get it done. All too often, we see engineers struggle to do it in-house only to never really do a good job, never complete it and waste significant amounts of time doing so. Outsource for expertise, outsource for opportunity cost, outsource to save a considerable amount of time.<br />
<b><br />
8. Pick your battles carefully.</b><br />
<br />
Static tools are not cheap -- not only to buy but also to effectively put into one processes. The evaluation process for many static tools give you a good sense for what it can do for you so take full advantage of it. If you need to, start small. De-risk the project as best you can through prototypes and smaller rollouts. Bringing in static analysis does require a commitment but pays off big-time when done correctly. By its very nature, static analysis is proactive. If you don't get funding the first time around, it shouldn't take long for the next emergency to make the situation more "favorable"for bringing in tools to more efficiently improve quality and security.<b><br />
</b>Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-2657982242529886802010-07-20T16:43:00.000-07:002010-07-20T16:43:04.295-07:00Detecting Endian Issues With Static AnalysisOur own Carl Ek, architect at <a href="http://www.codeintegritysolutions.com/">Code Integrity Solutions</a> had his article on detecting Endian issues published in the esteemed Dr. Dobbs Journal. Here is the first paragraph and link:<br />
<br />
<i>"There are 0010 0000 kinds of people in the world: Those that understand the difference between Big Endian and Little Endian, and those that do not."</i><br />
<br />
Since all binary processors (hardware or software) have an endian design, correct processing of the data based on that endian design is extremely important. The statement above is a version of another joke, but with a twist: the binary is represented in little endian, giving some mild humor for those that understand. For those that don't understand endianness, the humor is lost, much like a processor which has an endian processing defect. In this article, I describe the kinds of defects which occur, and methods where static analysis tools can help detect programming errors and enforce correct programming.<br />
<a href="http://www.blogger.com/goog_2107333007"><br />
</a><br />
<a href="http://www.drdobbs.com/windows/226000073;jsessionid=M2RNTVOZTQ42LQE1GHOSKH4ATMY32JVN">Detecting Endian Issues With Static Analysis</a>Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com2tag:blogger.com,1999:blog-5704901458470135557.post-10328914199494658302010-07-13T17:01:00.000-07:002010-07-13T17:01:28.552-07:00What Should You Be Good At?Any reasonably powerful software tool has enough complexity and flexibility to handle a multitude of scenarios. Having a lot of knobs and buttons are essential in the software development arena where environments are typically comprised of stitched together home-grown and commercial tools. Rarely is the whole toolchain packaged and nicely integrated. The ability for development tools to be customized means that it is highly possible to solve your business problem using best of breed technologies. It also doesn't hurt that engineering teams are highly capable of building most anything. However, the path to achieving that goal through in-house resources is often difficult and oftentimes impractical.<br />
<br />
Any department, particularly the engineering department, must recognize that they <i>must</i> have a set of core competencies that are strategically essential for their company to succeed. Providers of software are in a highly competitive environment where differentiation is critical. Not surprisingly, these core competencies are the areas that must be focused on, cultivated and maintained from an engineering standpoint. Some examples of core competencies are the ability to: <br />
<ul><li>produce particularly high performance, highly scalable transaction systems</li>
<li>add relevant features quickly to be first to market</li>
<li>consistently deliver the most reliable systems</li>
<li>deliver the most elegantly designed and easy to use systems </li>
</ul>Successful companies deliver on the levers that matter the most in a competitive industry. It could be feature leadership, process leadership, user experience leadership or likely a combination of things. The ability to leverage your core competencies comes from having the right talent and the right environment to help them succeed.<br />
<br />
Where are your levers? Where should you focus your team's efforts? What competencies should you build and what should you outsource? Where should you invest and by how much? Should your team be experts at building features that matter or having expertise in software development infrastructure?<br />
<br />
In the most extreme case, one could argue that you should outsource everything that is not within your core competence. The benefit is that you can focus on being best of breed in the areas that matter to your business while getting best of breed from outside providers in the areas that matter less. In the other extreme, you can do everything in-house, the "not invented here" syndrome. The benefit is that you have control but the downside is that your focus is diluted and your costs greatly increase as your needs change. Culturally, engineers have tended towards the "build it in-house" approach because they can.<br />
<br />
With regards to static analysis, how much of software development infrastructure is truly strategic to your company? How much can and should be outsourced to some outside help? Where can you get the most leverage from your team's resources to focus on the areas that will make your software development organization a leader in the industry?Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-83234542384093361562010-07-09T10:12:00.000-07:002010-07-09T10:12:04.332-07:00Static Analysis Typically Results in 85% Defect Removal EfficiencyCaper Jones put together a nice article titled "<a href="http://www.drdobbs.com/architecture-and-design/225701139;jsessionid=5Z044V44D45LKQSNDLSCKHA">Getting Software Quality Right</a>." He makes an important case for static analysis helping to increase the defect removal efficiency. He also notes:<br />
<blockquote>Combining inspections, static analysis, and testing is cheaper than testing by itself and leads to much better defect removal efficiency levels. In concert, these approaches also shorten development schedules by more than 45% because, when testing starts after inspections, almost 85% of the defects already will have been addressed.</blockquote>The range for defect removal efficiency for static analysis is listed from 55% - 90% depending upon how correctly it is used. It's a <b>wide</b> range and most likely highly dependent upon whether best practices are implemented or not. Clearly a tool can be easily misapplied both from a process and product implementation perspective.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-4688860691278049592010-06-30T15:52:00.000-07:002010-06-30T15:52:46.350-07:00One Easy Way to Improve the Quality of Third Party ComponentsSoftware products these day are increasingly made up of components from a variety of different sources. These components provided by the open source community, supplier companies and software vendors provide functionality for user interface management, operating system access, security interaction, connections to device and more. In addition, software developers also use code generation tools that enable developers to manage the application at a higher abstraction.<br />
<br />
By using third party components, an application with much more functionality can be pieced together much more quickly than ever before. <br />
<br />
Yet the customer who receives the product, be it a smartphone, router, medical device, software application certainly doesn't care how it's developed - only that it works. Developers increasingly must be able to manage not just the code that they've written but also the code that they integrate into the overall solution. Although it is in one sense, out of their sphere of control, they must manage the integration of the code so that the overall solution has the needed quality, security, performance, reliability and more that is expected of the finished product. Managing the interfaces well in some cases is not enough -- many developers must opt to own the source code so that they can make the needed modifications to make it work for their needs.<br />
<br />
We work with many companies who run into this situation. They analyze source code using static analysis on their codebase to find all sorts of problems to fix. In order for the source code analysis to work better, they also analyze the third party code that is bundled with their end product. This is an important step because many bugs stem from the improper interaction of components - and without analyzing the whole program you either miss out on a lot of bugs or get an inordinate number of false positives depending upon the static analyzer's conservative or aggressive assumptions. Clearly, defects that result from the direct interaction with these third party components should be fixed. This is something within the control of developers. Defects found <i>in the source code of the third party code </i>should also be fixed although many developers find themselves powerless to change the vendor's code or realize that they must increase their overhead by making changes to the source code and managing that change for future versions. It's extra work.<br />
<br />
But the developer has some leverage. We have worked with many forward thinking companies who lean on their suppliers to do their own code analysis, and to fix up the problems. For those companies with extra leverage, they have even demanded their suppliers to provide clean code as an acceptance criteria. These companies take a whole product view of the code, realizing that a bug in a third party component doesn't make a bit of difference to an end customer who's system or application is broken.<br />
<br />
So if you do run an analysis and find problems in third party code, put the burden on the supplier to fix up their code. It's very reasonable for a company to run a quick analysis and to put a concerted effort to fix the problems (although they will have to obtain their own license). You may be surprised at how much leverage you have. And demanding good, clean code is something in everyone's interest. It will save you a significant amount of time downstream and more importantly provide the quality experience that your users expect.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-49156371313126236702010-05-24T12:29:00.000-07:002010-05-24T12:30:20.666-07:00Defect Database Bugs Versus Static Source Code Analysis BugsStatic analysis has been around for quite some time. Recently they've been pointed to the very practical application of finding bugs in source code. Many modern tools do a very good job of uncovering very good bugs -- the big value of which that the tools find them early in the development cycle. Because the bugs are found in code, the reports point specifically to where in the source code the problem manifests. Easy to find, easy to fix.<br />
<br />
But there are some subtle differences that can greatly affect the expectations that users have of the tool and the process workflow that is needed to support these tools well.<br />
<br />
<b>A Dose of Reality</b><br />
Contrary to what every developer desires, static source code analysis tools do not <i>ONLY</i> find critical, crash-causing defects. Like any other discovery or analysis tool, it's going to find a mix of great stuff, good stuff, so-so stuff and stuff that's plain wrong. In industry parlance, these would be problems categorized as critical, high, medium, low priorities and "false positives." In static analysis, there's even an additional state commonly used called "intentional" which is that the tool correctly found something that looks like a problem but that the programmer intentionally coded it that way (usually due to an assumption that the static analysis tool is not privy to - such as assumptions in the environment that the code operates in). Modern tools have improved their analysis significantly to reduce the lower value stuff and raise the likelihood of uncovering higher value stuff.<br />
<br />
That being said, every organization has a different definition of priority but for the vast majority of software development organizations, priorities are mapped to a stage in the process. Often, priorities such as "critical" and "high" are mapped to specific requirements - such as part of an acceptance criteria for release. Mediums and lows are relegated to "if you have time, fix it" or "we'll never get to it." That's fine - rarely does an organization have the resources to address all issues (safety critical code being the notable exception). Defects reported by static analysis tools should be treated no differently. Some are important to fix and others are less important to fix. When <a href="http://www.codeintegritysolutions.com/">we</a> go over static analysis defects with groups of developers, you'll invariably have the whole room resoundingly agree that a particular segment of code is poorly written and "should be rewritten." However, resource constraints don't always make that the best business decision. Most developers know that the regular low priority defects in their bug tracking database <i>should</i> be fixed, but will likely never be touched again. Everyone wants to fix them but nobody has the time. Lower priority results from static analysis tools fit that too, with the only caveat that the chance for opportunistic fixes is greater because they are discovered during coding. <br />
<br />
When introducing static analysis tools to developers, they should be of the mindset that it's not going to find <i>every</i> critical problem and that the problems it does find are not <i>all</i> going to be critical. These are unrealistic expectations for any tool.<br />
<br />
<b>Checks and Balances</b><br />
Another key difference with static analysis defect handling is that they are being reported by a tool and not by a customer, support or QA. Thus the decision process for prioritization is not handled by an "outside" party to engineering. Some organizations let the developers decide on defects which is okay as long as developers are properly trained and all have the same minimum standard for what is acceptable. Many developers without the right expectations and training end up marking bugs incorrectly as false positives or perform unnatural acts to make the "tool shut up". To mitigate these problems, organizations can set up a different review/prioritization team or have an audit process on the tail end to ensure accuracy and build learning into the organization.<br />
<br />
<b>Making Lemonade Out of Lemons</b><br />
One might think that a "false positive" or "intentional" is a terrible waste of time and needs to go straight to the wastebucket without further ado. However, these reports do provide information that can be useful to the organization:<br />
<ul><li>False positives show when the tool did not correctly report a problem. This is often a case where tuning the analysis can improve the results. By configuring the analysis to understand your code better, you can significantly lower false positives and increase the valid bug count for future analyses. In this case, some false positives should be assigned to a static analysis expert to retune the analysis to improve the results. Fixing false positives in this way is significantly more efficient than having developers wade through incorrect results. False positive "fatigue" is a frequent cause for miscategorization as well.</li>
<li>False positives often are a signal of poor coding practices. If the tool is confused by the code, then it is more likely that the code may be confusing to a new developer inheriting the code. False positives tend to show around code that could be improved through refactoring.</li>
<li>Intentional's are also an interesting area to look. The most common reason for dismissing a bug report as intentional is that the tool did not have access to an assumption. Of course, the root of many errors is a changed assumption which causes problems to occur in the future. Intentional's often signal an opportunity for defensive coding.</li>
</ul><b>Process and Alignment</b> <br />
Static analysis tools often come with their own defect tracking information, which is usually quite useful because they also include a browser which helps developers more quickly debug a report. And yet, having another database and UI is extra work.<br />
<br />
Borrowing as much of the already existing bug workflow is key. Developers don't want to learn an additional process on top of their already busy day. Borrow what you can, recognize the differences and institute unobtrusive additional processes to handle them. Every solution we've instituted follow the same general principles but have varying levels of implementation because everyone's process is different.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-52039844926116262412010-05-06T10:03:00.000-07:002010-05-06T10:04:40.282-07:00Offshoring Quality in Static Source Code AnalysisCompanies who buy static source code analysis solutions will invariably run into the problem of the backlog. After all, they already have a mountain of working code, accumulated from decades to hundreds of person-years of development. There are bugs in the code and a static source code analysis tool will dutifully march through the code and find all sorts of potential problems.<br />
<br />
At <a href="http://www.codeintegritysolutions.com/">Code Integrity Solutions,</a> we've seen backlogs as small as a few hundred on a small codebase to over 50,000 on the largest codebases in the world. Average backlogs are in the thousands. Getting through this backlog takes time, effort and skill. The rewards are usually worthwhile because within this backlog are usually some very good showstoppers. And yet software development teams are always short on resources. Rarely will a product owner stop all development to through the backlog to find those gems hidden among the other high, medium, low and false positive results.<br />
<br />
Oftentimes, we see the engineering management suggest, "why don't we have our offshore team do it." The offshore team may already be charged with maintenance and so it seems a logical group to address a big backlog of static analysis defects. It certainly is a cost effective solution and it <i>will</i> make the backlog go away. Problem neatly solved and management is happy.<br />
<br />
Of course, successful development through offshore teams is a discipline all unto its own. We won't repeat any of that here. We discuss here the static analysis flavors of these challenges below. These problems exist in <b>any</b> team but are amplified in an offshore environment. <br />
<ul><li>The backlog must be prioritized. The question is who should own the prioritization? Who has a vested interest in quality that you trust to make that decision? The backlog will go away but will it be done in a way that results in significantly improved quality?</li>
<li>Teams are heterogeneous and thus the quality of the prioritization and the quality of the fix will be heterogeneous</li>
<li>Configuration and optimization almost always beats brute force. If a static source code analysis tool misunderstands a construct in your code, this can result in a cascade of false positives. Oftentimes a quick configuration change can wipe away hundreds to thousands of defects saving time (now and in the future) and making the analysis better understand the codebase.</li>
</ul>What can result from mismanagement is poor results done inefficiently. Falsely identifying issues, fixing problems that aren't real, and plowing through report after report inefficiently can result in wasted dollars, missed opportunities to improve the quality and tensions between offshore and on-shore teams. Keep in mind that when the future showstopper invariably hits, management will ask, "was this problem found by our static analysis tool and could we have avoided it if we had done it right the first time." <br />
<br />
Any good manager knows that they need to provide the right support, infrastructure and structure to make their team successful. For example:<br />
<ul><li>Set good standards for diagnosis of issues. Everyone needs to understand what a critical, high, medium, low, etc. categorization is. Some prioritize based on checker types (e.g. memory leaks, concurrency, etc.). Some prioritize based on whether the bug is found in a critical path, conditional path or in an exception path. Some prioritize based on what part of the code it is found in. Some use the built-in prioritization mechanisms of the static tool. Most use a combination.</li>
<li>Train the team. We found mentoring the team through real bugs is the best way to get everyone on the same page. Expectation setting is critical here. These sessions help gain consistency in results. In addition, finding the <a href="http://www.informationweek.com/news/development/tools/showArticle.jhtml?articleID=224600143">"ah-ha" </a>moments gets the team jazzed and energized to do good.</li>
<li>Set up a trust but verify model where results are audited. Use these audits as learning opportunities to steer the team in the right direction and finetune further. We've audited results and identified many opportunities to improve standards, to train the team appropriately on certain topics and to open up discussion on how best to handle certain situations.</li>
<li>Have an expert on hand who is aggressively seeking ways to tune the analysis. It is much more efficient than brute-force going through the backlog and it set ups the analysis to be much more accurate in the future. A simple configuration change can save weeks of effort. <a href="http://www.grammatech.com/products/codesonar/media/ESC_SJ_2010/">Paul Anderson, the VP of Engineering at Grammatech recently did a presentation at the ESC Show in San Jose</a> that suggested that the more false positives there are the more exponentially higher misdiagnoses there are.</li>
</ul>Of course the challenge of instituting a successful static source code analysis process takes a lot more than following a few bullet points. If you decide to go off-shore instead of outsourced or doing it with your on-shore team, it can be successful. With the right effort and expertise, you can effectively kill the backlog, improving your quality without having to stop everything else in its tracks.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-31696165719456156642010-04-29T20:01:00.000-07:002010-04-29T20:01:21.490-07:00Cramming More Value Into the Static Analysis WorkflowStatic analysis tools find bugs in source code during the earliest part of development. Where in the process static analysis is used varies widely, depending upon the software development organization's specific goals and the environment in which it will operate in. Some organizations run static analysis on entire codebases just before release. Others run static analysis on a nightly basis or several times a day during a continuous integration process. Still others run static analysis locally on a developer's desktop so that they can find and fix problems on the snippet of code they are working on.<br />
<br />
One can make a distinction of when static analysis MUST be run versus when it is CONVENIENT to be run. Organizations are increasingly using the results of static analysis as part of the acceptance criteria for a codebase. Meeting acceptance criteria may be just before release, before collapsing a development branch into the main line or even before a developer checks in code. It makes a lot of sense to ensure that all reported issues have been addressed as part of graduating the code from one level to the next.<br />
<br />
Of course, we all know it's a good practice to prepare well in advance of a future requirement. By providing static analysis throughout the development process, you eliminate the situation where you're two days before release and you run the static analysis tool and find that you have hundreds of problems yet to be addressed. Clearly, fixing bugs as you develop will improve the quality and security of the software early, letting you more comfortably coast into and through your acceptance process.<br />
<br />
What else should be in your acceptance process? The static analysis workflow is a convenient place to find and fix other problems other than just bugs. Here are some additional things that static analysis can help you enforce, conveniently borrowing the same workflow.<br />
<ul><li>Industry coding standards such as MISRA, Scott Meyers Effective C++</li>
<li>Government compliance standards such as DO-178B, PCI and FDA</li>
<li>Internal coding standards</li>
<li>Architectural coding standards</li>
</ul>Using the same workflow makes it easier and efficient for developers and administrators. Tools are starting to combine their capabilities to help software development organizations solve bigger problems. One of the major vendors in the static analysis space recently announced the support for <a href="http://www.klocwork.com/news/press-releases/releases/2010/PR-2010_04_27-Klocwork-Announces-Support-For-MISRA.php">MISRA</a>. Many vendors offer the ability to author custom checks to find and fix a wide range of problems. They provide a huge benefit because these checks can find problems across all codebases in the company and can be modified to meet the changing needs of the organization. Custom checks can be written to enforce coding standards, architectural requirements and find defect categories that are particularly important to your company. Custom checking can also serve as a complement to code review.<br />
<br />
So leverage existing workflows by adding other types of checking during the right points in the development process. And then arm the developers with the ability to meet their goals easily. The more you can push checking up front, the more likely you'll meet your time pressure goals.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-59747051558516839212010-04-22T14:08:00.000-07:002010-04-22T14:08:20.432-07:00Lowering Developer Resistance to Static Analysis ToolsImagine a team of software engineers who embraces the software development process. They dutifully create their unit tests before writing a line of code, run and fix all necessary static analysis and dynamic tests prior to check-in, fix all compiler warnings, participate actively in code review. They write code that is brilliant, yet easy to understand and maintain. They write unbrittle code that can withstand the inevitable changes to the assumptions that surround the code. Everything of course is well documented.<br />
<br />
They realize that their code is bigger than themselves -- that it has to work as part of a larger ecosystem of code created by engineers from around the world. They realize also that their code may last for decades and may change ownership several times. They realize that taking proactive efforts to make good code mean less reaction down the road. They welcome new process that will help them write better code overall even if it means a little more work upfront.<br />
<br />
Most developers won't disagree with these ideals, however, when instituting process that is "good for us to do", they bristle at, argue with and/or ignore the change. It's not their fault. Developers are often set up to fail at following the process right from the beginning.<br />
<br />
With every new change comes an investment in time and energy to make it happen. If you were asked to add a new step in your development process, you have to make an investment (sometimes sizable) at the beginning to understanding and instituting the change and subsequent investments every time you perform that new added step. <br />
<br />
We all know that developers are under immense pressure to deliver more for less. They simply aren't given enough time to make the upfront and ongoing investments. Routinely asking engineers to do feature X in 2 days instead of 3 days will result in cries of "foul" every time something might get in the way of their already oversubscribed status quo. Clearly the managers and project managers must understand and manage properly the time pressures.<br />
<br />
In addition, the upfront and ongoing investments in time and energy can be significantly reduced. When instituting static analysis tools at various companies, we've found the following to be helpful:<br />
<ul><li>Training can be a huge time saver. But not all people learn the same way. Some like it fed to them and some like to go and get it. <a href="http://www.codeintegritysolutions.com/">We</a>'ve found that taking a multi-pronged approach, by offering training, group and individual mentoring, a centralized resource with good, simple step-by-step instructions, a helpdesk, a forum/wiki and more get just about everyone onboard.</li>
<li> Package it well. Developers can figure out just about anything technical. But, it doesn't mean they have to. Having to fiddle with getting a tool installed may provide some individual technical satisfaction but doesn't mean they've done anything useful towards meeting a business goal. Multiply that by how many developers you have and you've created a large time sink. You'll also lose support by developers in this way by making it hard to use.</li>
<li>Keep the standard workflow simple. We worked with one organization which gave every developer the standard training and then washed their hands of it. After having thrown it over to the developers, it was no wonder that a year later hardly anyone was using it. One of the biggest problems was the tool was given to the developer with no workflow and no purpose. No queries were set up. No screens were customized. The developers had to suddenly become experts at using the tool -- a foolish use of time given tool usage was certainly not a core competence of the organization. Spoon feeding a simple, easy-to-follow process doesn't mean you think your developers are dumb. It means that they get to spend less time wrestling with the tool and more time doing what they are paid to do - write great code that solves the company's mission. Again, without the right process, you'll lose support.</li>
<li>Automation is key. Having multiple steps in the process across multiple tools in the toolchain mean added time, human error and frustration. Integrating static analysis with bug tracking, source control management, LDAP, policy databases, build scripts, etc. lowers the ongoing investment.</li>
<li>Another area ripe for improvement is performance. We've run into codebases where the analysis takes 30 hours or more. Static analysis has to analyze a lot of stuff but there are ways to improve the performance dramatically. We've tuned some analyses down to minutes depending upon the deployment model. This significantly lowers the "cost" and opens up new possibilities for when and where analysis can be performed</li>
<li>One of the biggest complaints you'll hear from developers is that the results have false positives - meaning results that are incorrectly noted as issues. Every static analysis tool will generate false positives but the more the tool understands your code (usually through configuration), the more it will produce better results. Every time a developer has to examine an issue that is eventually is false is wasted time (although some will argue that not all of the time is wasted because false positives can signal poorly written code). Lower this cost and raise the value of the tool through tuning. </li>
<li>If you aren't a developer get them involved. There's nothing like having a champion or two help you get buy-in. Involve them in defining the process - after all, nobody knows the process better than the developers themselves. Developers have to see and believe in the true value, not the inflated value. Do this by running mentoring sessions where everyone can plainly see the value and the limitations of the tool. Set expectations all around accordingly.</li>
</ul>In concept, static analysis tools are simple to use. In addition, your developers are smart, so it's tempting to just give them the tools and let them figure out the value. It's a given your developers have too many features to deliver in too short a time. You need to lower the cost of adding static analysis to their workflow. You'll lower resistance and improve their efficiency while improving the overall quality and security of the code.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-26753891975705619152010-04-11T08:33:00.000-07:002010-04-12T06:53:11.707-07:00Two Views of Static AnalysisEngineers have differing expectations of static source code analysis tools. Opinions of its usefulness vary widely but generally fall into one of two extreme camps. <br /><br />On the one hand, some engineers think that program analysis couldn't possibly produce anything worthwhile. They believe all the results are useless and not worth the effort to examine. They may have had experience in the past with products such as Lint, which produced copious amounts of “useless” warnings. These “naysayer” engineers will rarely give static analysis a real chance and instead avoid using it, or worse yet, dismiss every result as a “false positive” or “never going to happen”. <br /><br />Conversely, some engineers have the unrealistic expectation that static analysis will only produce valid, critical defects that must be fixed. These “naive” engineers may not have a good grounding of how static analysis technologies work and what its limitations are. They may equate static analysis tools with dynamic tools, which almost always report real bugs but lack the coverage and ease of static tools. These engineers will very quickly be disappointed. Static analysis finds potential problems based on sophisticated algorithms. These algorithms operate on source code and build-level information and may not have access to all of the run-time assumptions. Highly complex or very custom code can also cause the tool to misfire unless the tool has been properly tuned. Naive engineers may also "fix" issues that don't really need fixing on blind faith or to quiet the tool. Attempted bug fixes can cause other areas to break.<br /><br />Nearly every organization has these polar extremes and everything in between. Naysayers cause others to lose confidence in the tool. The naive engineers may lose credibility by overstating the tool’s capability. Somewhere in between is the truth -- that static analysis produces good and useful but not perfect results. There are going to be good bugs you really want to fix. There are also going to be low priority issues that you don’t really need to fix (similarly, very few organizations fix even a quarter of the bugs logged in their bug tracking system). Other issues will be just plainly wrong (a false positive) and signals that the tool couldn’t understand that part of the code well or that the analysis needs to be configured.<br /><br />Engineers need to understand not just what the tool can do, but what it can’t do as well. When an engineer is trying to interpret a result, it helps to understand what the tool did to produce that result. There are many strategies for getting a group of engineers to understand the range of value a static tool provides. Training, documentation and process changes are necessary but not every engineer learns by reading a manual or attending training class. One of the most effective ways for engineers to learn is to do some joint triaging of results. For <a href="http://www.codeintegritysolutions.com">Code Integrity Solutions</a>, we’ve ran very successful sessions by simply going through bugs with a group of engineers in the room. We discuss a handpicked set of bugs, debate them and resolve them together. These sessions (even as short as an hour) enable everyone to come to a general conclusion around a common set of results and start the basis for a standard by which to triage defects. Senior engineers shine in these types of environments because they not only quickly understand the tool's capabilities but also are able to impart their standards to the less experienced team members.<br /><br />We often hear these points made:<br />* This is not really a bug but this is poorly written code and needs to be redone<br />* Ah, I see why the tool is reporting this incorrectly. All I have to do is make a mod to the analysis to fix that<br />* I wouldn't have thought that it was a bug. I now see why it is after taking a deeper look.<br />* If we change this assumption slightly, then this would become a bug. For defensive coding reasons, we should change it, especially since we occasionally change ownership of code.<br />* I understand why this tool is not catching this type of problem. I need to use a different method for finding these types of problems.<br /><br />Invariably the naysayer comes to understand the value the tool can provide for him or her and the rest of the organization. By also showing false positives, the naive engineers can see any limitations of the tool. For everybody involved, they see the nuances of the tool and the level of effort required for effectively evaluating results. A few hours of time makes for weeks of productivity gains.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0tag:blogger.com,1999:blog-5704901458470135557.post-5236194027592557492010-03-20T15:00:00.000-07:002010-03-20T15:15:07.931-07:00The False False PositiveStatic analysis tools report potential bugs in source code by analyzing the structure of the code for inconsistencies and flaws. Sometimes they get it right and sometimes they get it wrong depending upon how strong the analysis is and how complex the code may be. These days, static analysis tools are getting more and more sophisticated, doing statistical analysis and interprocedural analysis (where information is tracked across functions) across all the logical paths (sometime numbering many millions) in the code. Dataflow analysis can track values across functions to produce bug reports that span multiple levels of calls.<br /><br />While increased sophistication means that static analysis tools can catch more problems with a higher degree of accuracy, the burden increases on the reviewer of the results to interpret them correctly. If you were grep'ing through some code for something you can quickly review (and dismiss) many of the results because you understand what your "analysis" is doing. With static source code analysis, this is much less apparent. <br /><br />We see many engineers look at a complex bug report and not take the necessary time to understand the problem and fix it. This is mostly because they don't understand what the static analysis tool is doing and how deep it is analyzing the code. The result is a real bug being marked as a false positive - or a "false false positive" if you will. These bugs then disappear off the queue never to be seen again - a lost opportunity.<br /><br />How do you minimize the number of false false positives? Lots of different ways:<br /><ul><li>Training and/or mentoring of the reviewer team helps the team see how to properly disposition a defect</li><li>Auditing of false positives either periodically or all-the-time. Incorrect markings are great learning opportunities to correct incorrect usage.<br /></li><li>Metrics that monitor the general false positive rate so that individuals can be benchmarked against them to detect anomalies. If one engineer is marking 80% of the defects as false positives versus a general 30% average, then this is likely a correctable problem.</li><li>Have double reviews of defects. Similar to pair programming make sure each disposition has "two pairs of eyes" on it<br /></li></ul>By making triaging more accurate you get much more value from your tool and minimize the incorrect throwaway.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com1tag:blogger.com,1999:blog-5704901458470135557.post-49817229118277603302010-03-08T12:32:00.000-08:002010-03-08T17:05:18.013-08:00Does One Size Fit All With Static Source Code Analysis?If you take a look at all the software written in this world you'd find a vastly heterogeneous set of applications. The list of these applications include medical devices, military/aerospace, banking, games, network telecommunications, automotive, gaming, accounting, wireless and much much more.<br /><br />As to the importance of quality, security and reliability, even a gaming application can demand good quality and high performance. While a game crashing may be a trivial situation for an individual games customer, the game publisher may be faced with an expensive patch they must rollout or a poor review to hurt their sales. Good software quality and quality is good business.<br /><br />Static source code analysis is up to the task of helping software developers improve the quality and security of their code in the earliest part of the development process. Finding critical problems in source code is simply easier and faster to fix. However, codebases are very different. They use different libraries, have autogenerated code. They have different memory allocation mechanisms and concurrency and locking mechanisms. They have different internal coding standards by which they adhere to. They support many different platforms. They may come in very different flavors like C, C++, C# and Objective-C some of which bear very little resemblance to the other. Even within C, there are different implementations of the C standard. What's a static analyzer to do? <br /><br />Static source code analysis does its best job to interpret the code at the cost of potential false positives. Clearly if the static analyzer cannot understand the codebase it has less information to make good decisions. Static source analysis vendors ship out their best configuration but it's a one-size fit all. A medical devices company analyzing 50,000 lines of code for a pacemaker is getting the same settings as a networking company with a 18,000,000 lines of code operating system. What's important for one company may be extremely different from another. For instance, a company who provides accounting software that runs on the desktop could care less about memory leaks as the application is frequently exited and reentered. Counter that with network routing software that must be up 99.999% of the time and may run continuously for years. A crash on an embedded system on an airplane or automobile may be different in priority than to a crash on a web application where automatic failover to a different server is built into the infrastructure.<br /><br />In addition, the way an application is coded may be different. The application may use standard memory allocators provided by the operating system or the application may use its own proprietary memory management techniques. <br /><br />The analysis may be more conservative or aggressive than desired. A medical devices company who wants to find every potential bug has a different tolerance for the "noise" than a large gaming application which has to deal with tens of thousands of bugs and only wants to hit the highest priority ones.<br /><br />One size cannot fit all. Only one version of the software comes from the provider with the same factory settings as everybody else. That same version is used for all industries and even in vendor presales situations as well. It's understandable that this is the case. Vendors understandably must provide the tool with the best possible setting for as many different situations as possible.<br /><br />But clearly it can't address all situations optimally. With just some smart tuning, the analysis can be much smarter about the specific codebase it is looking at and make the results more relevant and accurate. In some cases, we've been able to tune away thousands of defects with just a single configuration change which saves weeks of collective developer time. Other cases of tuning have resulted in hundreds of more critical bugs being reported that wouldn't have otherwise with the "factory settings." The payoff to good expert tuning is both immediate and long term.Andyhttp://www.blogger.com/profile/06705630945588101293noreply@blogger.com0