Performance Testing 101 - 1

I've seen a lot of people jump into performance testing recently, and they've all encountered similar problems. So each Friday for the next few weeks, I'm going to post a few tips that will hopefully help folks out.

Let's start out with some basic terminology here. Performance testing is making sure that your app responds in a timely fashion. Note that this does not mean that you need to have hundreds or thousands of virtual users running in order to do a simple performance test. Consider this scenario: Let's say I'm testing a calculator application. I entered 2+2, and then pressed the = button, and got 4. From a functional testing standpoint, the test passed. However, if it took the calculator 30 seconds to return the answer, then we have a performance problem. That's all performance testing is - making sure your app responds within a user's expectations.

Load testing is what most people are thinking of when they refer to "performance testing." This is where you're expecting that 500 people will be performing a given activity on a web site, so you use a tool to emulate 500 people perfoming that activity. So load testing is ensuring that an application responds in a timely fashion while an expected number of people are working on it.

Stress testing is where you try to find the breaking point of your application. So if you've established that your system can handle the expected 500 users, can it handle 600? 700? More? This is where you push the system to its limits and find out what number of users make it fall over and die.

Ok, so now that the terms are out of the way, the first thing you should do is figure out the scenarios you want to performance test. If you're testing a site that's already live, you can cull through your web server logs to find out which pages are being hit most frequently. Then build your scenarios based on your actual customer behavior.

If you're testing a site that hasn't been released yet, then think of those activities that your users will perform most frequently. For example, if you're testing a website like Amazon, your most frequent scenarios will be things like searching for books, logging into the site, adding books to the cart and processing orders. Start with those, and then consider other activities. Remember, the point of load testing is not to do a functional test with hundreds of users, so you don't need to performance test every page or feature on your site. For example, how often do you search for a book on Amazon, compared to how often you change your login password? Those less often performed activities can be tested later, but they probably aren't high priority.

Next, run a single user performance test for each scenario. This will establish your current performance baseline. No extra tools other than a stopwatch or a wristwatch with a second hand are needed. Launch your application, and perform your test. For example, let's say one of your performance tests is to log in to a website and get to a Welcome screen. Enter your username and password, start your timer, and click the login button. Once you've reached your Welcome screen, stop the timer. Note how long it took to perform the login. Share this number with your developers and PMs. Let's say it was 2 seconds. Is this number acceptable? What's the acceptable range for this number? Is it ok if it goes up to 4 seconds?

Alternatively, if it takes 10 seconds to login, then you've already found a performance problem, and running the same test with more users probably won't yield any additional info. Just focus on getting the login process down to the desired 2 seconds, and then you can begin loading additional users.

We'll talk more about scenarios and their configuration next week.

Dealing with Ego

I've been very lucky to work with some very talented developers over the last 11 years. (Wow, I suddenly feel old :)

However, we've all been on those teams where there's been a primadonna, that one person who thinks he or she is God's gift to software, and that everything they touch turns to gold. These people typically see testers as evil little gremlins who are hellbent on destroying their perfect code. As such, they display a certain level of standoff-ness to testers; maybe they aren't willing to answer questions, maybe they dismiss bugs as extreme edge cases, you get the idea.

I've always believed that my ego isn't part of my work. If someone finds a hole in a test I wrote, it's not personal. They aren't criticizing me. Most people I've worked with share this belief. It's not fun when someone finds bugs in the code you wrote, but it's not a personal attack either.

Sometimes just having an open conversation stating the above to the primadonna corrects the problem. Other times, they get even more huffy. So here's a trick I used on one primadonna. He was starting to brush off some of the concerns I was raising. I lifted a finger and said to him, "Mike*, my job as a tester is to make you look good. And if these items get addressed, the product's going to work better, people are going to love it, and as such, you're going to look great. So come on, help me make you look great."

He gave me all the help I needed after that. Sometimes it's all in how you play things.


*Mike wasn't actually his name. I always change names to protect the innocent. Then again, I'm not sure how innocent he was. His name was Lou.*


*Kidding! :)

The Most Important Qualty in a Tester

There are a lot of things that make a person good at a testing job. Technical inclination, an insight in to the end user's mind, and keeping up with new methodologies are a few. But I'd wager the most important thing of all is curiosity.

A really good tester wants to know how & why things work the way they do. If they encounter something they've never seen before, they start learning about it. They scour the web looking for articles on new technologies, they speak with their developers to get a better understanding of it, and then turn that new information into test cases that will be more insightful and useful.

You can spot a good tester by the questions he or she asks. If they see something they don't understand, they ask about it and keep asking questions until they get it. They'll propose user stories and scenarios to make sure they've got a handle on things, and then make those the basis for their test cases.

This really translates to all walks of life - whether you're a tester, a developer, a PM or a salesperson. Ask questions. Learn everything you can, and your career will go some really cool places.

Online Presentations and Slide Animations

A lot of people like to spruce up their presentations by adding animations. And if you look through the powerpoint menus, you've got tons of choices. You can have things fly in, fly out, materialize in, blink in, wipe in... you get the idea. They're fun, they're exciting... and they absolutely suck in an online presentation.

The issue is with screen refreshes. Some participants in your online presentation won't see a fluid animation. Instead of a smooth fly-in, They'll see a jerky motion where your text lurches across the screen. Instead of a neat Star Trek type beam-in animation, they'll see squares and lines.

If you're giving an online demo you have two choices for animations: Appear and disappear. They aren't flashy, they aren't sexy, but they work best in an online environment. I suggest you stick with these even in an "in person" presentation, too. They aren't as distracting as the other animations, and the less distracting your slides are, the more focused your audience stays on you.

Helpful Tool: Jing

We've all had times when someone, whether it's a co-worker, a customer, or a relative has sent an email saying "Hey, I've got this weird problem, can you take a look." They describe the problem, and you immediatley realize what it is. Then you let out a long sigh, because even though you know exactly what the problem is, typing out all the instructions to perform the fix is going to take you a while. Or worse, it's one of those problems whose solution really needs to be seen in order to be understood.

Enter Jing. Jing is a free screen capture utility that lets you create 5 minute videos (with audio) and then upload them to the web. I've used this successfully to show customers how to perform various operations, and the feedback has been tremendous. People really appreciate that I took the time to make this little video, and now they can share it with their co-workers. Jing has a very clean UI and is extremely easy to use. You can download it from here: http://www.jingproject.com/

The Rule of 10

When I was in college, we had a rule that if a professor was 10 minutes late for class, then the class was automatically canceled. I have a similar rule for online demos with my sales team.

It goes something like this. Let's say we've set up the demo for 2 pm. My sales rep and I will typically log on between 1:50 and 1:55. If the customer has not logged in by 2:05, the sales rep calls the customer. If there's no response, we wait another 5 mins and try calling again. If there's still no response, we cancel the call. We then leave a message saying something to the effect of : "We're sorry we weren't able to connect with you today, please give us a call and let us know when you'd like to reschedule the demo."

Do you have any similar approaches for when people don't show up to demos or meetings? Please let me know in the comments.