Nate Meyvis

Notes on integration testing

Integration testing1 is often necessary, but:

  1. They tend to be slower, and so keep you from iterating as quickly as you'd like to.1
  2. They produce more false negatives.
  3. They often share, or fight over, the same resources, so that some tests cause others to break.
  4. They can't be as exhaustive, for combinatorial reasons.2
  5. They're usually orders of magnitude more expensive.3

So, although integration tests are implicitly underrated by most people (who don't do integration testing at all), they're often overrated in environments where they're being used at all.

Besides being important for its own sake, this all suggests a couple of speculative psychology-of-software conjectures:

  1. We experience the benefits of well-written unit tests--they scale well, they're cheap, they're fast, and so on--so often and vividly that we default to assuming that those benefits apply to integration tests also.
  2. Writing and debugging integration tests feels like significant, responsible work. Often it is!4 But this can lead us to undertake that work without reflecting on whether its benefits merit it.

  1. Note that this means that, if the development environment is otherwise healthy, they reduce code quality by keeping you from making first-order improvements to the code. If the tests are good, this cost is worth it, but we should remember that the cost exists.↩

  2. I agree with those who say that in many contexts the distinction between "narrow-scope" and "wide-scope" tests is more useful than that between "unit tests" and "integration tests." But if you refuse to speak of "integration testing," you will usually just annoy people.↩

  3. Let's say you have four systems, each with 8 parameters. If you're unit-testing an 8-parameter system, you can test many relevant combinations of those parameters. (If those parameters can't take on many values, you can even test all such combinations!) But it's almost always impossible to test all relevant parameters of a 32-parameter system, especially if each test takes orders of magnitude longer.↩

  4. And it's often easier to justify to management than deleting the tests would be.↩

#psychology of software #software #testing