Only Bitmap Comparisons? Bad Tester! Bad!

Many automation tools have the ability to take a screenshot of something in your AUT, and compare that to a baselined value. The intention is that you can use this to verify that images have rendered properly in your app; so things like your company logo, a product's photo, or the like can be validated easily. But I still come across people who want to use bitmap comparisons as their only means of validation in their tests.

Their reasoning seems sound enough. "If we just click through each screen of our app, and do a bitmap comparison of each full screen, then we can easily confirm the app is working. The bitmap comparison will automatically validate every value in every field, so we'll be able to create our tests even faster!"

It sounds good, doesn't it? But it doesn't work that way in practice. In fact, this has to be one of the most fragile, breakable, and ineffective approaches to automation. See, bitmap comparisons are going to do a pixel by pixel comparison of each screen. That means that if a single pixel is off, the test fails. In some cases, just running the same test on a different OS is enough to cause every bitmap comparison to fail, because a dialog in WinXP may have a screen where one pixel is slightly lighter than that same pixel in Vista. To your eyes, they don't look any different, so you won't know there's a problem until it's too late. Imagine that suddenly all your tests fail, and for no good reason. If this happens, you'll wind up in a maintenance nightmare as you struggle to track down that rogue pixel and somehow convince your automation tool that this is really ok. Automation is supposed to make testing easier and faster, not the other way around.

So please, take my advice. Use bitmap/region/image (or whatever your automated tool calls them) comparisons only to compare actual images. If you want to verify the contents of a field, read the actual value from that field. If you want to verify a button is enabled, check its actual Enabled property. This may require a bit more effort up front, but the payoff in the long run is well worth it. You'll have more efficient, more stable tests that will run clean, and it won't matter if that one rogue pixel off to the left is grey, a slightly darker shade of gray or a slightly lighter shade of gray than the ones around it.