Performance Testing 101 - 1

I've seen a lot of people jump into performance testing recently, and they've all encountered similar problems. So each Friday for the next few weeks, I'm going to post a few tips that will hopefully help folks out.

Let's start out with some basic terminology here. Performance testing is making sure that your app responds in a timely fashion. Note that this does not mean that you need to have hundreds or thousands of virtual users running in order to do a simple performance test. Consider this scenario: Let's say I'm testing a calculator application. I entered 2+2, and then pressed the = button, and got 4. From a functional testing standpoint, the test passed. However, if it took the calculator 30 seconds to return the answer, then we have a performance problem. That's all performance testing is - making sure your app responds within a user's expectations.

Load testing is what most people are thinking of when they refer to "performance testing." This is where you're expecting that 500 people will be performing a given activity on a web site, so you use a tool to emulate 500 people perfoming that activity. So load testing is ensuring that an application responds in a timely fashion while an expected number of people are working on it.

Stress testing is where you try to find the breaking point of your application. So if you've established that your system can handle the expected 500 users, can it handle 600? 700? More? This is where you push the system to its limits and find out what number of users make it fall over and die.

Ok, so now that the terms are out of the way, the first thing you should do is figure out the scenarios you want to performance test. If you're testing a site that's already live, you can cull through your web server logs to find out which pages are being hit most frequently. Then build your scenarios based on your actual customer behavior.

If you're testing a site that hasn't been released yet, then think of those activities that your users will perform most frequently. For example, if you're testing a website like Amazon, your most frequent scenarios will be things like searching for books, logging into the site, adding books to the cart and processing orders. Start with those, and then consider other activities. Remember, the point of load testing is not to do a functional test with hundreds of users, so you don't need to performance test every page or feature on your site. For example, how often do you search for a book on Amazon, compared to how often you change your login password? Those less often performed activities can be tested later, but they probably aren't high priority.

Next, run a single user performance test for each scenario. This will establish your current performance baseline. No extra tools other than a stopwatch or a wristwatch with a second hand are needed. Launch your application, and perform your test. For example, let's say one of your performance tests is to log in to a website and get to a Welcome screen. Enter your username and password, start your timer, and click the login button. Once you've reached your Welcome screen, stop the timer. Note how long it took to perform the login. Share this number with your developers and PMs. Let's say it was 2 seconds. Is this number acceptable? What's the acceptable range for this number? Is it ok if it goes up to 4 seconds?

Alternatively, if it takes 10 seconds to login, then you've already found a performance problem, and running the same test with more users probably won't yield any additional info. Just focus on getting the login process down to the desired 2 seconds, and then you can begin loading additional users.

We'll talk more about scenarios and their configuration next week.