Inserting Static Analysis Into Your Agile Processes

Most software development organizations are moving to Agile development processes. All flavors of companies have started instituting Agile whether the organization is an IT organization supporting their business units or product companies who are developing software for customers. An important benefit of Agile is the ability to improve cycle times. Agile enables organizations to iterate much faster on market requirements and to iterate much faster on delivering working software to the market.

To meet these demands, software development organizations are using everything they can get their hands on to help them accomplish their goals. Whether it's continuous integration servers or static analysis solutions, development teams are arming themselves with new tools and new processes to capitalize on their ability to meet market needs faster and with less risk.

From Nightly Analysis to Continuous Analysis.
Static analysis can help in a number of ways. Many organizations already have a nightly build in which they require their developers to check-in only code that has been verified as clean. By making sure their software builds successfully every day, they minimize the risk of having broken code and complex integration steps towards the latter stage of the development process. It's not a stretch to increase the check-in requirement to also include addressing of all new static analysis defects. In fact, being alerted to these issues during development make them much easier to fix than when reported later in the development process.

Continuous integration is a further step in ensuring that the code is being constructed earlier and often. The challenge for most organizations is having static analysis operate at a reasonable performance to keep up with the build frequency. A typical large project may take only 15 minutes to build in an optimized environment but take 4-5 hours to analyze. With the right tweaks, static analysis can be optimized to be much faster. Often, it takes some outside expertise to assist.

Inserting static analysis whenever you build is an ideal time. Builds are a convenient time to run statistics, run automated tests and perform static analysis on your code.

Meeting the Demands
No developer wants to be the person who breaks the build. If there are new criteria required to check-in code, then those requirements should be testable by the developer in their development environment so they get a chance to address any problems prior to the more visible tests. They should therefore be able to make changes to their code and test, fix and iterate until they feel sure that their changes will pass the criteria. Arming the developers with this capability is not easy. Static analysis is doing complicated stuff. The computations that occur take time. Moving static analysis to the desktop requires both proper performance levels as well as the right usability model. At Code Integrity Solutions we've worked with extremely large codebases (tens of millions of lines of code) that whose analysis times can be chopped down to just minutes). Better quality code gets checked in when developers can iterate multiple times. Investing in a developer desktop model provides orders of magnitude benefit including better quality software earlier in the process and improved developer productivity.

Invest in a good, fast solution for both developers and your software development infrastructure and you'll see your Agile pay off faster!

Static Analysis: Isn't This Supposed to Make my Life Easier?

Software developers have a tough job. They have to produce working, good quality code from scratch, under immense time pressure and often without a good specification to provide guidance. Project estimates are always optimistic which leaves the developer in a situation where they have to work twice as hard to deliver what had been promised. Rarely are developers able to write code to the level of quality that they want to write. There just isn't enough time in the day. Is it any wonder that software developers resist adding more things to do in their process?

Along comes static source code analysis. Most developers don't want to add an additional step in their already harried process to check bugs that were found by another program. Here are the typical rants we hear from software developers?

False positives!! Nothing hurts the credibility of the tool more than false positives. If a tool is more wrong than right, developers won't trust the tool and consider it a waste of time to use. For most static analysis tools, you spend more time reviewing defects than actually fixing them. Fix the analysis so that it reports less wrong reports. Have a team (inside or outside the company) weed out the bad stuff so the development team can focus on developing.

Blend it into the process!! Forcing developers to take extra steps "out-of-band" not only slows them down but also makes them forget to run it in the first place. Make it run automatically with each build. Make it automatically assign defects to developers. Make it easily create a bug tracking report. Notify the developers rather than request them to look for it. Make it easy for them to do their jobs. Make it so that you don't need any documentation.

Tuning!! Developers are already busy enough -- they don't want to add static analysis expert to their growing list of responsibilities. If the analysis falsely identifies an issue as a bug, there are usually steps that can be taken to improve the analysis so that it understands the code better. Requiring the engineer to learn the system to be able to modify a configuration, add rules, etc. is a waste of time. Asking them to write their own static analysis checks is an effort that is often wasted. Have a static analysis expert make it understand the codebase and have them write the checks that are needed to be productive.

Make it faster!! Waiting for builds is bad enough. Waiting for static analysis to finish can be a time sink especially if it's a requirement to check in "clean code." Have an expert optimize the performance of the system. Cut down the analysis so that it analyzes only the stuff that has been changed.

Truly getting adoption from developers is a tough challenge. If they don't see the value, they won't use it. Maximizing the benefit returned from using the tool is just as important as minimizing the cost of learning and operating the tool. The key to bringing value is to manage this equation so that maximum return is achieved at an individual developer level. You'll never get perfect adoption without creating some artificial requirements (like not allowing a release without addressing all bugs). However, you'll have to fight a lot fewer battles if you make your user base as happy as possible.

Are Static Source Code Analysis Solutions One Size Fit All?

Automated source code analysis solutions worth their salt are complex products that do heavy duty analysis on complex codebases. They churn through thousands to millions of lines of code written by teams of developers each with their own skill levels and style. Millions of paths are typically analyzed in a single run producing what it thinks are valid defects to be examined.

On one level, you'll hear people say that code is the same whether it is written in Java, C++, C or C#. After all, some universities give computer science homework assignments that can be written in your language of choice. On the other hand, there is a high degree of variability in the customer base that a static analysis vendor must provide. There are just some of the significant differences:

• Industry - some industries care about different things. A router company is going to care more about memory leaks because their code may be in operation for months to years at a time. A desktop application may not care at all about memory leaks because they know their application will be open and closed frequently. You can imagine that a military aerospace or medical devices company would have vastly different priorities than a SaaS product where the system can be controlled much more easily. Some organizations may want to squeeze every last possible defect the tool can find at the cost of getting some additional noise. Others may be looking for a good "bang for the buck" and fix bugs opportunistically without having to wade through false positives.
  
• Development environment - the software development tools space is highly fragmented. There are hundreds of vendors that provide source control systems, compilers, bug tracking systems, build and continuous integration tools, automated test tools, etc. On top of that, there may be third party libraries in use and/or code generation tools that supplement the hand-written codebase. Software applications are almost always built in a custom way and it's hard for a third party tool to get an accurate picture of what the shipped code actually is without a significant amount of tweaking and integration. A static tool may simply not understand a custom memory allocator or a source/sink for a data flow analysis. They may also have some compiler incompatibility issues. What can happen is that you don't get 100% coverage. Tools will typically "fail" silently or make it nonobvious how to find out what your true coverage is.
  
• Development process - similar to the diversity of the software development infrastructure, software development organizations are often very chaotic. Some organizations follow the traditional waterfall methodology and others follow Agile processes. These requirements place demands on how a source code analysis is deployed and used. Some organizations are global with offshore development teams making significant contributions.
• Platforms - software development organizations typically commit to a single development platform (usually a specific Unix flavor, Linux, Windows, Mac OS X, etc.). However, they may be building software that will run on many different target platforms. Each of these applications use different libraries and may have significant functionality differences.

What's a static analysis tool to do to satisfy everybody? Should the default settings be set conservatively or aggressively? Are the settings designed to give a good demo/trial or are they set for a specific industry? The clear answer is that no tool comes out of the box optimized for your specific codebase. There is too much variability.

What's important to do is to build or obtain the expertise to optimize the tool for your codebase. After all, you are paying the license fee for the full benefit of the tool. If you don't take the extra effort to make sure you are optimized, then you are overpaying for the tool.

We've worked with many organizations who have taken different approaches. Some of built their expertise in-house with just the standard training as their starting point. This can take years to obtain expertise, however, the competency remains in-house. Others take a hybrid approach and allocate the right resources and get an expert to mentor them through the process. Others use consultants to get them all set up and running and then use their in-house staff to manage and administrate it. Others outsource everything related to the tool to an outside expert so that they can focus on their core competencies.

What You Didn't Buy When You Bought That Static Analysis Tool

When you are buying things, be it at the store or through a vendor for your company, you are really buying results. The tool or object you purchase is just a means to an end. Like the old adage goes, you don't go to a hardware store to buy drills, you go there to buy holes. Static source code analysis solutions are great tools for helping software development organizations find defects earlier in the development process. However, because they are a technology entering the mainstream, most organizations don't have the experience to know entire effort required to effectively use static analysis.

When you purchase a static analysis solution you are really buying the defects that are valuable enough for you to take action to fix. A good measure of the value of the tool is the number of defects that you indeed fix. What you aren't purchasing are:

• the set up and customization required to get the tool set up, tuned and integrated into the toolchain
  
• the management and administration needed to keep the tool running
• the additional hardware required to run it with good performance
  
• the training and rollout required to bake the tool into the process
• the political process required to get people to change their behavior
  
• the additional steps required for each user to change their process
  
• the defects that were reviewed and put in the 'not fix' bucket. These include false positives, don't care alerts, low priority reports and ones that haven't been reviewed.
• the time and expertise required to further improve upon the tool or take advantage of features and capabilities not yet used

Even with all of this effort taken together, it's hard to beat the value that a good static analysis tool can bring. Just remember that buying the tool is just the first in a series of steps that need to be taken to get to success. Sometimes you have to pay a little bit to get the big payoff.

Who's Really In Charge Now?

Static analysis is fast becoming a standard part of every software development effort. Companies ranging from medical devices companies to Internet ecommerce vendors to car manufacturers to network equipment providers are purchasing static analysis solutions on the promise that finding defects sooner in their development process will save them time, money and embarrassment. The concept sells well at the developer and management levels because it makes sense.

And, static analysis tools are good these days. They aren't perfect, but with the right usage they provide a good bang for the buck. I stress the word "usage". Many companies buy the tool. The vast majority install the tool but a surprisingly high number don't get past running the tool even a couple of times. The product sits on the shelf.

There are many reasons why this occurs but the one we'll explore today is ownership. Someone has to own the tool and what happens to the results reported by the tool. Technical ownership is fairly straightforward. Static analysis tools are usually owned the tools group, who is responsible for setting up the software development infrastructure. They set up the factory floor, making sure that the engineers have all the tools they need to be efficient and effective. Automation plays a key role, helping to scale benefits across an entire team. However, unless the proper organizational and process infrastructure is in place, they are likely doomed to failure.

The purview of the tools group varies but does not include setting up new processes in isolation from the software development team. Oftentimes static tools are thrust on the tools group. The tools group then sets it up with the only goal of offering it as a voluntary service to the software development team. What ends up happening is a static analysis tool that is used by only a few enterprising software developers who believe in the value of tools and of software quality. The tools group is powerless to force more developers to use the tool and only a fraction of the potential value of the tool is realized.

Tools groups can lower the cost of usage by making it as simple as possible to use, by integrating the tools within their development process more effectively and by providing training, support and mentorship to developers. Tools groups also can take responsibility for tuning the analysis to make sure the results are as relevant as possible. But while carrot approaches are important and necessary, it won't get everyone over. You need to foster experts within the organization who can establish and defend the tool's worth. When you can set check-in, commit or release criteria that include static analysis, then you start getting the usage that will provide the payoffs that were paid for.

We've seen organizations that mandate that all new issues get addressed. Even false positives are "fixed" because if the code is confusing enough to fool a static tool, it will likely be confusing to the next person who must maintain the code. Others simply require that at least everything needs to be reviewed and it is up to the developers to determine whether they should fix them (the developers must document why they don't fix a problem). Others require specific classes of bugs to be fixed. The rules simply must match the business goals.

So even though tools groups are essential to the success of static analysis, they cannot be the only owner. Like any change within an organization, it requires executive sponsors, champions and a technical and organizational infrastructure to make it happen.

Hiring, Repurposing, Consulting, Contractors, Outsourcing, Offshoring or Part-timers for Static Source Code Analysis?

When it comes to getting resources to get static analysis underway within your organization, you have a lot of options. You can hire full-time (or use an already existing teammember), you can get a contractor, you can get consultants and you can outsource/offshore.

When it comes to designing and operating your static analysis solution, there are many options you can go with. I like to think that the institution of static analysis in an organization as being divided into several phases.

1. Requirements - what goals do you hope to get out of it, usage models
   
2. Design / Architecting of the solution within your existing infrastructure
   
3. Implementation - the fine art of setting it all up and automating it
   
4. Maintenance - the ongoing management of the solution, including dealing with the inevitable changes to the code, build environment and the static analysis solution
   

Clearly, each step requires a different skill set. These steps can be handled by one senior person (who has all the skills) or by a combination of people each specializing in their own areas.

We walk through the various choices with their own advantages and disadvantages:

• Full-time
• Part-time
• Contractor
• Consulting Firm
• Outsourcing / Offshoring
  

As someone who has hired many people full and part-time, and who has worked with contractors, consultants and outsourcing firms and have founded and worked in two consultancies, I have a few experiences and scars to share.

Full-Time Employee

Advantages:

• Institutionalize all the expertise with your employees
  
• Flexibility to control your resources and move them to different responsibilities rapidly
• Employees benefit from already existing relationships in the company
  

Disadvantages:

• The fully loaded costs of a full-time employee are significantly higher than the salary. Put in the cost of insurance, training, office space, machine costs, etc. and a $100,000 employee starts looking like a $150,000 cost. The budgeting process reflects these extra costs and the cost of extra headcount.
  
• Employees are often fairly flexible resources that are assigned many different responsibilities but are therefore constantly being pulled onto other urgent projects of the day.
• Hiring takes extra effort. Employee hiring should be looked upon as a permanent action. Will this person with these specialized skills be the optimum fit years from now?
  
• You may need to train this person. It's hard to hire someone who is both flexible for the long term and highly specialized for your need today. It takes months to even years of investment to develop a good level of expertise in-house.
• Gaps in time such as for vacation and sick days require coverage, particularly for ongoing management and administration.
  

Part-Time Employee

Advantages:

• Some roles don't need to be full-time. Particularly for a younger company, you may not be able to justify a full-time build engineer or a static analysis expert.
• Part-time enables you to pay for only what you need but still get what you need for the amount that you need it for.
  

Disadvantages:

• If you predict growth, then part-timers are not very scalable.
• Part-timers also carry a fully-loaded cost, oftentimes higher in proportion to a full-time employee.
  

Contractor

Advantages:

• Contractors need to prove their existence every day. They know they have to provide a high level of service or be replaced.
• Contractors can be started up quickly and can be removed at the end of the project without any HR headache. Some projects make sense for a period of time and not forever.
• You can hire contractors that are highly specialized. Perhaps it's a particular product expertise, or a particular process expertise - you can hire highly experienced people from a temporary time period to get best practices.
• Contractors enable you to focus your best resources on the most strategic activities while leaving the rest to nonstrategic resources.
  

Disadvantages:

• Contractors need start up time. They are used to getting up to speed fast but still require ramp up time.
  
• Contractors may or may not be available after the project completes.
• If you don't plan properly for this, contractors will leave at the end of the project (or even midway in extenuating circumstances) with all the institutional knowledge gained during the process. You should build in time early to transition knowledge through mentorship and documentation throughout the project.
  
• Contractors need to develop relationships during the project.
  

Consulting Firm

Advantages:

• You have a firm to back you up. If you need to have varying levels of expertise throughout a project, it's up to the consulting firm to provide the right person. You can draw upon a larger set of expertise from a consulting firm. You can get the best experts in every area you need that don't need any training.
  
• Consulting firms tend to be solutions and results focused.
  
• You can ramp up and down a team easily - from a whole team down to a part-time consultant.
• You get coverage when you need it. No need to worry about sick and vacation days.
  
• You have one place to go to for as many of your needs as possible
• You allocate budget and responsibilities and are less able to be sidetracked than an employee
• Consulting firms enable customers to focus their best resources on their company's strategic needs while still leaving nonstrategic items to experts in their field
  

Disadvantages:

• Consultants require some ramp up time
• If you don't plan properly for this, consultants will leave at the end of the project with institutional knowledge. You need to work in transition time through mentorship and documentation.
  
• Depending upon market rates, consultants may be more expensive than the fully loaded costs of an employee.
• Consultants must develop relationships during the project.
  

Outsourcing / Offshoring

Advantages:

• The cost advantage is usually significant
• Outsourcing is a good way to pass off non-core competence work leaving your developers to focus on more strategic work
  

Disadvantages:

• Communication is usually difficult. Many outsourced teams are located in different countries with different timezones and languages spoken.
• Management overhead is increased. Interfaces between teams must be micromanaged.

Hybrid approach.
Every organization is different. No one solution fits all. The key though is to have the expertise you need when you need it in order to institute static analysis as quickly, efficiently and effectively as possible. Consider these questions as you consider what are your best options:

• Many organizations recognize the investment required to develop expertise in-house. Is the investment of becoming static analysis experts worthwhile or not?
• Do you value best practices? Will having an expert who's done it many times before guide you through the process more efficiently than trying to do it yourself?
• What are my architecting versus operating costs? How can I create the best solution that requires the least amount of management and administration?
  
• How much time is required and what expertise is needed for architecting and operating static analysis? It may not be full-time.
• What are my contingency plans if priorities change or people leave?
• What are your true core competences? What do you want to be best at doing?
• What kind of budget do you have? Initially and ongoing? What is the return you get to justify the expenditure?
• Are pieces of the requirements-architect-implement-administrate better served by the same person or different people? Is continuity of resource more valuable than best practices in each area?
  

What Motivates People to Buy Static Analysis

We’ve visited a lot of companies who are using static analysis to improve their software development. What drives companies to use static analysis varies considerably from company to company and from industry to industry. We attempt to categorize the most common drivers for using static analysis into various buckets. We’d love to hear about other drivers as well so please let us know if you have others to contribute.

There are a lot of factors to consider when trying to understand what drives a company to use static analysis. You have to consider not only what are the business drivers and benefits but also who are the people who are driving and influencing the decision.

We have bucketed them accordingly
• The opportunists
• The code reviewers
• The engineering management
• The quality portfolio manager
• The compliers
• The automators

No categorization is discrete. There’s quite a bit of overlap and different roles can exist within the same company. Benefits can be felt across a wide range of people.

The Opportunists
The Opportunists use static analysis tactically, to capture the low hanging fruit in the codebase. They recognize that one of the biggest benefits of static analysis is finding and fixing bugs are much cheaper than most any other means. But, Opportunists only want to invest as little time in the tool as possible. They try to find and fix as many of the easy problems as they can find and focus only on the areas that they feel are most important to them for example like focusing on crashes or memory leaks.

The downside of this approach is that incrementally, Opportunists get a really high ROI because their "Investment" is low and their "Return" is good. However, they must take into consideration the license and support costs plus any other costs associated with the tool such as training, set up and configuration. With all parts of the equation adequately modeled, the ROI can suddenly become much less attractive. In order to keep the costs low, many Opportunists just run the tool out of the box with the default settings, not even taking the time to tune the analysis to find more of the low hanging fruit that they want to fix. Clearly there is more benefit opportunity for Opportunists, to better increase the marginal utility from the marginal cost they put into it.

The opportunist is usually a single developer or a few developers who are highly proactive in improving the quality and security of their code. They are also usually extremely busy with little time to devote to static analysis.

The Code Reviewers
Code Reviewers can live within the software development organization (such as a code review team) or outside, such as in the case of a security review team. Manual code review is a tedious, error-prone and non-scalable process. While static analysis doesn’t replace manual code review, it helps make the manual code review process much more focused on higher value activities. At the same time, static analysis helps reviewers gain significantly more coverage, reduce human error and provide more consistency in the review process.

Code reviewers like to have a well-tuned analysis, specific to the codebase with all the right settings to find the class of problems that are most relevant to them, for example, security reviewers like sources and sinks defined as part of their data flow analyses (without it, the data flow analysis doesn't have much to work with). The analysis can also be tuned to find a wider range of problems at the cost of a little less accuracy. Code Reviewers can also write analysis rules to find specific classes of problems or enforce coding rules. When put into a regression test, these rules can make sure the problem or coding violation doesn’t arise again either in the original place it was found in or in any other place in the codebase or even in other codebases that are being analyzed.

The main challenge in these environments is the natural friction caused by reviewers versus developers. When handled improperly, developers can become defensive and push back on static analysis reports as “not bugs.” Code Reviewers need to be sensitive in how they assign and discuss potential problems. Developers take a lot of pride in their work but should also be open minded of efficient ways to improve the quality of their output. When working well, this model has great checks and balances in place to ensure proper usage of the tool.

The Engineering Management
We’ve all heard the analogy of how software development should borrow more from the manufacturing industry. Many software development managers recognize that static analysis actually saves overall development time and can improve cycles times. Finding defects earlier in the process can most definitely save on overall time and costs.

Management’s big challenge is to ensure broad, consistent usage so that the benefits can be realized. Not every developer recognizes the benefits amidst the negatives. Static analysis is not perfect and requires some learning, to understand defect reports, process changes and most importantly dealing with false positives and “don’t care” reports which, if not handled properly, can decimate the morale of the developers.

The biggest challenges to these environments is lowering the cost of usage through deep integration with the current software development process and automation to minimize any manual work. In addition, tuning of the analysis to lower the rate of false positives and “don’t care” reports will lower the cost significantly and provide a big psychological boost. Using outsourced teams to review the defects first help maximize your most precious resource – your developer’s time.

Administrating the tool requires effort and most tools engineers are already too busy to not only learn the tool but manage it ongoing. Contractors and consultants can help either to jump start the process or to take ownership of the management and administration ongoing.

The Quality Portfolio Manager
There’s no silver bullet to quality and security. In order to get acceptable quality and security, you have to take a multi-pronged approach. Unit tests, code review and now, static analysis are parts of the overall strategy to lower the risk of field defects.

Holistic approaches to quality are often driven by a forward thinking engineering or QA manager, or as a response to a serious bug or serious class of bugs that has now driven a top-down quality initiative. Other companies may realize that bugs are causing a high recall rate or high support costs which is driving down profitability. This realization create the necessary organizational capital to move static analysis forward.

The challenges that arise from these types of initiatives stem higher than those faced by the Engineering Management example. Specific ownership and clear goals are too often not considered when instituting a static analysis solution from the top down. When static analysis becomes just a checkbox among a range of other strategies, it often doesn't get the necessary attention to make it succeed. Ensure that the ownership of the tool has teeth, whether it be owned by someone internally or externally.

The Complier
In some industries, you just don’t get much choice. If you are developing software that will be implanted into someone, or go up into space, or guide a missile, then you have to be concerned about regulations and industry guidelines such as FDA, DO-178B and MISRA. In all of these industries the government either requires or recommends the use of static code analysis to improve the quality and security of the delivered product.

The challenges faced by the Complier are similar to the Quality Portfolio Manager. The further away the driver for a solution is from the implementer and maintainer of the solution, the more that will be lost in translation and commitment. Developing the expertise in-house is not insignificant. Creating ownership with real incentives and consequences is critical.

The Automator
Automation is the fashion, particularly with the movement towards Agile methodology where cycle times are much more rapid. Continuous integration builds are becoming more common and creates better software earlier in the process. Automators recognize that static analysis is perfect for this. Static analysis doesn’t require test cases and can be automated as part of the build process (as most modern static analysis is integrated with the build process). Results can be churned out in minutes or hours. Developers can even analyze code as they are developing it allowing organizations to institute rules that code must be free from any static analysis errors before check-in.

The largest challenges here is that automation takes some work. It’s not hard but it’s also not necessarily turnkey. Being able to have the right deployment model so that the analysis is fast for each developer can be tricky but is most certainly possible. Automation usually is two pronged – where developers can run a fast analysis locally on their changed code while a central analysis handles a full integration analysis. Automation is well worth the effort even in the short term.

Many Reasons, Many Options
Clearly there are many drivers for static analysis. In conert, the optimum deployment model varies considerably. For example, tuning an analysis to find everything under the sun may not be a good model for the Opportunists. Likewise, a full developer deployment model may not be a good choice for the Reviewer's. Matching the deployment model and the analysis tuning to the business need will increase by orders of magnitude the ROI you get from a static tool. Companies have woken up to the fact that developing this expertise in-house is an investment that is appealing to some and unappealing to others. Some prefer a jump start mentoring program to help them get pointed in the right direction and others prefer to outsource to get best practices static analysis expertise. Regardless, static analysis can provide a multitude of benefits for most software development organizations. By recognizing the drivers, you can create a great custom solution that makes almost everybody who involved get what they need out of it.