Defects: Find Them Early, Fix Them Fast

“Most defects end up costing more than it would have cost to prevent them. Defects are expensive when they occur, both the direct costs of fixing the defects and the indirect costs because of damaged relationships, lost business, and lost development time.”

— Kent Beck, Extreme Programming Explained

Software defects, also referred to as issues, bugs, observations, or oopsies (my personal favorite), are one of your largest sources of wasted time and money.

In traditional software development with separate development and QA staff, quality is viewed mostly as the testers’ responsibility. The developer’s job is to develop. With a separate QA safety net, programmers are free to code fast and furiously with no need to worry about defects because 1. that slows them down and 2. that’s not their job!

Putting politics and human nature aside, what really happens when we defer testing?

The later you detect and fix a defect, the longer it will take to fix. In software development, time is money. Team members get paid regardless. The ultimate cost of a given user story hinges on how long it takes to develop and deploy successfully.

Why? It’s the complexity factor:

1. As you take a new feature from idea to design, to testing, then to release, the complexity of the solution rises exponentially. The farther you progress through a feature’s lifecycle, the more inter-dependencies that feature inherits, making defects more difficult to fix.
2. If you take time to validate the requirements and design, the fix can be a simple text change without repercussions on any other existing infrastructure or integrated interdependencies.
3. The further you progress into implementation, the greater the complexity. A defect found late in the game requires a complex unwinding of interrelated threads of requirements, design, interdependent classes and services, and test cases.

According to the National Institute of Standards and Technology (NIST), as a defect passes undetected through each stage of your system, its cost to fix increases exponentially.

Consider the following scenario:

A customer submits a request to add emoji capability to your financial software communication module. You decide emojis are a good idea and your team starts working on it. In this scenario, from the beginning, there is a nasty hidden defect we’ll call Bugsy.

Depending at what stage you find and fix Bugsy will determine how long it will take to fix Bugsy.

1. If Bugsy is found during requirements and design development, it will take about 1x time to fix. This fix would be a simple text or design change.

2. If Bugsy is found during coding, it will take about 5x longer to fix. The developer would need to write code, test code, find Bugsy, fix Bugsy, re-test code and if finds errors, fixes those and retests. This cycle continues until the code is defect free. Therefore, fixing a defect in coding takes longer than in design.

3. If Bugsy is found during integration testing, it will take about 10x longer to fix. Integration is when we are testing to see how our multiple components interact with each other. This testing takes longer to do because it is more complex. Likewise, if a defect is found, it will also take longer to fix due to the greater complexity.

4. If Bugsy is found during acceptance testing, it will take about 15x longer to fix. Acceptance testing is often performed by a tester, not the person who developed the feature. This adds delays. Here is a common flow for acceptance testing interchanges:

• • Tester tests feature
  • Tester also performs some level of regression testing to make sure the new feature didn’t break another feature
  • Tester writes up defects and pushes to the development team
  • The development team fixes defects and pushes to test (again)
  • Tester evaluates fix and either closes defect or pushes back to the development team for rework. The cycle continues until no defects found.

Whew, that is a lot of potential delays! No wonder it takes so much longer to fix Bugsy when it isn’t found until acceptance testing

5. If Bugsy is found by a customer after the emoji feature is released it will take about 30x longer to fix. This is due to a variety of factors. First, the inherent time required for the back-and-forth communication between customer support and the customer, then the back-and-forth with support and the development team. Second, months may have passed since the team developed the feature. Even if the same developer that created the feature fixes the defect (not always likely), they will have long forgotten the details of the implementation. The developer will need a significantly longer amount of time to weed through the code, isolate the problem, then fix and retest. Finally, in the timeframe from when the customer found Bugsy and the emoji feature was delivered, the code base has grown. That increased complexity and change can increase the effort required to find, fix, and validate Bugsy’s demise.

The additional time needed to fix and validate a defect found late in the delivery process takes away time and money that could be better spent developing new features.

To increase your ability to find and fix defects in the earliest stages, you must test in every stage.

Before requirements and design are passed to the next stage, conduct a peer review, try to find holes and misassumptions, and fix immediately. Be sure the acceptance criteria are clearly defined, and the team has a shared understanding of them.

During coding, developers write automated unit tests to validate small chunks of code. They run static code analysis tools to defect coding flaws and fix immediately. They do a code-review with other developers. They run integration and acceptance tests before passing the feature to QA. Any defects found here, fix immediately!

Testers work with people in each stage as tests are being created to give input early and often. When the feature reaches stage, there should not be any easily detected defects. The testers will perform exploratory testing to find those sneaky defects that may be missed by automation. In collaboration with the developers, testers will also develop and maintain a set of reliable automated regression tests. When defects are found in QA, fix immediately! Don’t put them off.

Start one step at a time and incrementally expand and improve your testing pipeline and processes. Commit to baking quality in at the very beginning of your process and sharing responsibility for it across the team. If team members aren’t familiar with the new awesome testing tools, provide training and support. Investing in quality assurance at each stage in your lifecycle will decrease costs, improve employee morale, and improve customer satisfaction.

Fixing defects immediately is a Kanban practice that shortens lead time. Take our Kanban System Design and Kanban Systems Improvement courses to learn how to implement Kanban to decrease your lead times and increase your throughput and quality. See upcoming classes.