Monday, June 29, 2009

Not All Bugs Have to be Fixed

We live in a world where there just aren't enough software developers. No matter what stage of development you may be at, your team could always use just a few more developers to build that great feature marketing wants, fix that extra bug that's been nagging technical support, help build some tools so that software development can work more efficiently, etc. But sadly, we live in a world of constraints and that means that the marginal cost of any investment has to be paired with the marginal benefit it will bring.

The approach to fixing defects is no different. When fixing problems you need to look at the resource required and the benefit it brings. You'll see a wide range of perspectives when you ask developers about source code analysis. Naysayers view the results as noise, "nice-to-have" defects that never reveal themselves in the field. Supporters believe source code analysis helps them find not only critical defects but also potential coding problems that can affect future maintainability and scalability. Most developers fall in between these two extremes. We often see a good mix of problems reported:
  • critical problems that need to be fixed soon
  • medium to low priority items that are certainly bugs or poor programming but aren't of high priority to fix immediately
  • false positives - where the tool reports something that is clearly wrong
  • "ignores" - where the analysis is doing what it should be doing but the prioritizer doesn't think it's important to do anything about
Depending upon the organization's needs, the numerical distribution of these categories vary greatly. For a company sending a satellite into space, they may view most issues as in the "need to fix" category, even going so far as to saying that if a static analysis tool is confused enough to get it wrong, then the code should be changed so that the code is easier to maintain for the owner, code reviewers and all future owners. For these organizations, there is zero tolerance for failure.

For other organizations that are strapped for resources and have more than enough critical bugs they need to fix: then addressing just the "low hanging fruit" may be sufficient. Such organizations can tactically use static analysis to help them but they probably have larger issues they need to deal with. Of course, most companies fit somewhere in between the extremes.

However, it's important to revisit the marginal cost versus marginal benefit equation. Defects reported by static analysis tools point to specific lines of code where the problem may exist. This level of granularity makes the marginal cost of fixing a defect miniscule in comparison to fixing defects reported by QA or by customer support. No test cases, no debugging tools, no setting up the environment to recreate the problem is required. With a marginal cost so low, the pool of defects worth fixing suddenly expands. For most organizations, not every defect needs to be fixed, but the number of defects worth fixing is considerable.

No comments:

Post a Comment