Monitor Your Environment

It's important to keep tabs on what's installed on your test systems. Changes in your environment can alter test results and really throw a wrench in your day.

I was running performance tests once. My developer and I had tweaked the systems to their best performance, and the automated tests had been refined. We ran tests before we went home that night, and confirmed our results were good. The next morning, our VP wanted us to run the tests again so he could watch. Imagine our surprise when all our operations were taking 1.5 - 2x longer.

My VP immediately suspected that the test tool was faulty, and that's never a good way to start a discussion. Luckily, I had a written a python app that ran in the background on all my systems. It was a simple little affair, it polled the Uninstall section of the Windows Registry and logged anytime a value was added or removed. During the night, a windows update had been automatically applied, which adversely impacted my system's performance. Without my utility, there's no way I would've caught that.

We spent a couple hours on the phone with Microsoft and got a workaround to the problem. That done, we ran the tests again, and viola, everything was happy. I quickly disabled automatic updates on all my systems from that point on.

I lost a couple of hours to this problem, and my test automation's rep was literally on the line because of an automatic update. So keep tabs on your environment, and make sure you know what's happening to it.