Performance Testing 101 - 3

Now that we've talked about scenarios, the next thing I want to discuss is your performance testing environment.

Your environment should be an isolated, standalone network. All "normal" network activities like virus scans, automatic system updates and backups should be disabled. You don't want anything running that could skew your test results. Now, some people may question this, saying it's not a "real world" scenario. At this point, you're trying to see how fast your system can perform under the absolute best conditions. It's a way to see, "ok, we are absolutely positive we can handle this number of transactions/second." My experience with this has been if you have other things running, and the numbers come back bad, people will immediately jump on that as the cause. "Oh, you had a virus scanner running - that's probably messing up the results. Uninstall the scanner and run the tests again" If your tests take days to run, that's not what you want to hear, especially if the test results come back the same as they were before.

Having your tests on an isolated network also ensures that normal day to day traffic isn't skewing your results. I once had four hours' worth of testing negated because someone three cubes down started downloading MSDN disks. I wound up having to do my tests at night after everyone had gone home in order to ensure there wasn't any rogue traffic. Again, having an isolated network will save you a lot of testing time and frustration in the long run.

The other thing to keep in mind is that your tests systems should match up to what you'll be using in production. So whatever hardware your system will run in the real world is what you should be testing with. Some IT folks balk at this, citing cost as a problem. Thing is, there are issues you won't be able to find if you're just testing with a bunch of hand-me-down systems. If you're not running on a box that has the same type of processor, how will you know if you're taking full advantage of that processor's architecture? This extends beyond just the systems themselves. Switches, NICs, even the network cable involved can impact your tests' performance. Don't fall into the trap of "well, we'll be using better hardware in the field, so performance will be better", because it might not.