Automation Mashers

Test automation is handy for regression testing, but automated regression tests rarely find new bugs. So instead of focusing exclusively on regression tests, try coming up with scenarios to detect bugs that would be difficult (or impossible) to find manually. For example, let's say my AUT can be used to generate reports. I enter some criteria, click "Generate" and the AUT fetches data from a database and dynamically creates a report based on my selections. Let's also say that this fetch and display operation causes a momentary spike in my CPU usage and memory usage before settling back to normal.

Now, we could create a automated regression test that generated a report and verified it generated the correct data. That may be useful, but we can make it much more useful by turning it into a "masher" test. Mashers are tests that perform a single operation over and over again, while tracking how the system performs over time. In doing this, we can find bugs (like memory leaks) that we wouldn't be able to find by manual testing alone.

So in my scenario above, I noticed that there's a momentary spike in CPU and memory use when the report is generated. Being suspicious, I wonder if there's the possibility for a memory leak somewhere in that code. So I'd create an automated test that ran my report and confirmed it was correct, and then loop it so that reports were being generated for 24 hours straight. As the test ran, it would log the system resources being used after each generation. At the end of the test, I'd graph the data that had been logged. If the system resource usage remains constant, then there's no problem. But if there's a steady increase in resource usage, then we have a memory leak, and we've found a bug that a regular regression test (manual or automated) would have missed.

So in addition to making manual testing easier, and taking care of regression tests, don't forget to look for places where automation can help you find bugs that would be impossible to find otherwise.