Performance Testing 101 - 2

Ok, so now that we've taken care of terminology and gotten our performance baselines, it's time to start thinking about configuring our scenarios. This is where you figure out how many people will be performing a given action simultaneously. Again, if you are testing a pre-existing website, you can go through your web server's logs to see how many people typically access a given page at a time.

If you're testing a never-before released site, though, this part gets tricky. You'll need to make some educated guesses around your user's behavior. Let's say we're testing an e-commerce site, and we're going to roll it out to 1,000 users. Let's further assume that these users are equally divided between the east and west coasts of the USA (500 on each coast). For test cases, we want to perform logins, searches, and purchases.

Now, the knee-jerk reaction is to say that since you have 1,000 users, your system should be able to handle 1,000 simultaneous logins. That may be true, but is it a realistic scenario? In the real world, your users are separated by time zones, some of them may not be interested in purchasing a product today, some may be on vacation, some of them will only shop while they're on lunch break, and others will only shop at night after their kids have gone to bed. So in considering this, you may want to start out verifying that your system can handle 250 or 300 users logging in simultaneously. Then build up from there.

The next thing to consider is the frequency of the actions. Each user logs into the site once, but once they're in, how many times do they perform a search? If you go to Amazon, how many products do you search for in a single session? Between physical books, eBooks, movies and toys, I'd say I probably search for 2 or 3 items each time I log in. So that means searches are performed 2 or 3 times as often as logins. So if you're planning on having 250 - 300 users login to start with, then you're looking at 500 - 900 searches. Also consider the purchase scenario - not everyone that logs in and searches is going to buy something; Maybe half of the people, or a third of the people who login will actually make a purchase. So you're looking at 75 - 150 purchases for a scenario there.

So take some time and think about your end user's behavior when defining scenarios for load testing - don't make it strictly a numbers game.

Next week, we'll talk about your performance testing environment.