Blind Faith

Your automated tests are great. They run unattended every night, and each morning, you come in and see a little report on your desktop with a bunch of green success messages. You feel good. You know those tests are making sure the basic functionality of your app is running smooth, and there hasn't been a failure in days.

Then you give the first build of the app to the test team. Within an hour, 30 high priority bugs have been logged. You review the bugs and see that close to half of them are scenarios that are covered by the automated tests. Stupid testers, you think. They must be doing something wrong. You kick off your automated suite, and it passes. Then you try to run the same test by hand, and it fails. You try it again by hand, and it fails again.

Then you take a look at the automated test, and realize there's a flaw in its logic. You see that the test is configured to always pass, regardless of what's actually happening. You quickly dive into the code and fix the application, and then you go back and fix the tests.

The moral of the story here is to make sure you know exactly what your automated tests are doing. Have them code reviewed just like any other part of your project. Just because a report gets spit out saying everything passed, doesn't mean everything did.