Autonomicon

Helpful Tool: PasteCopy

2011-01-07T21:20:00.000-05:00

Often when I'm giving demos, I'll need to type in a sentence or two into a text field. This is no big deal, I can type pretty fast, but we all have those days when we can't type to save our lives. And invariably, this happens when lots of people are watching. To help avoid this, I've been using a freeware clipboard manager called PasteCopy. This lets me store off sentences that I will use in demos, URLs to send to participants who are in webinars, or paths to files that I commonly need. I keep PasteCopy open on another monitor, and when I need to get at one of the stored strings, I just pop over to that monitor, click on the desired string in PasteCopy, and then it's ready to be pasted into the desired field. It works beautifully and has saved me lord knows how many incorrect keystrokes.

PasteCopy installs in about 3 seconds and has a very light footprint. You can check it out at http://www.pastecopy.net/

Alchemy, Not Magic

2010-08-30T12:42:00.002-04:00

I once had a conversation with a tester that went something like this -

"We need the automation to be able to recover from errors. I don't know what those errors will be, or what steps will need to be performed to address those errors, but the upshot is that the automation needs to get the AUT back into a good state before continuing."

I explained that in order to build this kind of logic, at least some knowledge was needed of what constituted an error, and how the error was to be addressed. They rather heatedly told me that if the automation couldn't "just know" that it was not going to be helpful.

Here's the thing. We need to set reasonable expectations for test automation. Automation can make testing easier, and can speed up tedious processes. However, automation is not some magical, omniscient force that can interpolate how your application works and how to best recover from problems. If it could, we wouldn't need testers or developers - all our software would write and test itself. Anyone who tells you that automated tests will just magically resolve themselves with no direction on your part (they might be cute and use the word "automagically" - watch out for those people) is someone who's selling something, or is clueless.

Automation is a tool. It's not magic. It's not smarter than you are. It's only as smart as you make it, and it can only do what you tell it. Your automation can never, and will never "just know" how to test your application. The computer is a stupid box. It's just fast. If you don't tell it what to do, it will do nothing, although it will do nothing very quickly.

In short, automation is not magic, it's more like alchemy. It can be complicated at times, but the end result of making the testing process smoother/easier/faster is akin to turning lead into gold.

The Julia Child Approach to Demos

2010-08-20T07:46:00.004-04:00

When you're giving live demos, it's always handy to have everything you need for that demo ready to go. Have powerpoint already loaded, have the application you're demo-ing already loaded, and have anything else ready to go, too. If you'll need to show a calculation, have Calculator already running. If you'll need to write some code in Visual Studio, have that already running & open to a blank project. The idea here is that your audience should never have to watch a splash screen or "Loading" dialog. Before I run a demo, I launch all the apps I need, and organize them on the Windows task bar so that they're in the order that I'll use them. Then I just click right through the task bar as I go from one activity to the next.

Additionally, if you're going to demo any feature that takes longer than 10 seconds to complete, consider having a "before & after" of that feature, where you walk through the steps needed to perform the action (the before), and then have a finished version of the feature ready to show off (the after).

(A note here - always tell your audience about how much time the operation should take to complete. You don't want to misrepresent how fast your application works, that's a sure-fire way to lose credibility in your customers' eyes. I usually say something like "This operation usually takes about 5 minutes to complete, so instead of making you stare at my hourglass for that much time, I've already got a finished sample right here")

I call this the Julia Child approach, because I was first exposed to it while watching Julia Child's cooking show back when I was a kid. Julia would have all her ingredients and utensils ready, so she'd smoothly go from one prep task to the next. For example, if she needed to chop some tomatoes, she already had the tomatoes washed, and there was a cutting board and a knife right there. You didn't need to watch her hunt around for anything, she had it ready to go. Then she'd walk you through all the steps to make a cake, then put that cake in the oven. Then she'd move over to a second oven, and remove a cake that had finished cooking. It was a great way to show both the preparation and the finished product, without needing a lot of filler in the presentation.

So having all the pieces of your demo ready to go will make the presentation go smoother, and having a "before & after" prevents awkward lapses in the presentation. All in all, this approach helps ensure a clean, smooth presentation that your customers will eat up. Bon Appetit!

Automate in Straight Lines

2010-08-13T10:08:00.002-04:00

Everyone's heard the adage "the shortest distance between two points is a straight line." This holds true in test automation as well. I recently spoke with someone who was struggling to get a UI automation tool to run against an application. This lady had no experience with test automation before, and the application was not "automation friendly." As such, she had spent almost an entire week struggling to get tests running.

I spoke with her on the phone for about 30 minutes, and during the course of the conversation, the intent of what she was doing came out. She was not testing the software, she was trying to populate a database with information, and this database would then be used for training purposes. She'd need to repopulate the db each time there was a class to teach. This GUI automation scenario she was trying to create would take hours to run, but she hoped she could start it on the night before a class and then have it be ready in the morning.

I asked her why she didn't just write a plain old SQL script that would populate the db, then create a backup of the database and restore it each time she needed to teach a class. There was silence on the other end of the line.

Look for the simple solutions. Databases can be written to directly, and then restored, over and over again. Webservices can be manipulated directly, methods can be called from dlls directly, the Windows Registry can be manipulated directly. If your intent is just to get data from point A to point B, do it in a straight line via script. Adding other layers of complexity (UIs, GUI tools, etc) is kind of like trying to go from New York City to Boston by way of Los Angeles.

MS Office Automation

2010-06-14T12:48:00.003-04:00

I've worked with a lot of people who have wanted to automate MS Office apps. Sometimes they're trying to just write results data out to a Word doc, other times, their company builds a plug-in to an Office app and they're trying to create automated tests for that plug-in. And they all start out by pointing a commercial record & playback tool at the app and clicking that red "record" button. At best, this results in tests that click on obscurely named objects, or at worst, tests that are driven exclusively by X/Y coordinates.

You see, MS Office apps (Word, Powerpoint Excel, Outlook) are not record & playback friendly. I learned this the hard way a long time ago, and I'm hoping to spare others the same pain. You will not be able to create a reliable, robust set of MS Office tests via a record and playback tool. The only way I've found to effectively work with Office apps is via scripting. The Office object model lets you programmatically access any bit of text, cell or slide. For example, this code launches Word, opens a new document, and writes Hello:

Set objWord = CreateObject("Word.Application")
objWord.Visible = True
Set objDoc = objWord.Documents.Add()
objDoc.Selection.TypeText("Hello")

If you try to do those same actions via record & replay, your results will be spotty at best. So when you need to work with Office apps, automate them via their object model. A little known fact is that MS ships help files with office that describe each app's object model, along with coding examples. The location of these files is [OfficeInstallDirectory]\[OfficeVersion]\1033
The app's help files are:

Word - VBAWD10.chm
Excel -VBAXL10.chm
Powerpoint - VBAPP10.chm

There's also an online version here: http://msdn.microsoft.com/en-us/library/y1xatbkd%28VS.80%29.aspx

Keep these handy next time you're doing MS Office automation, and your automation efforts will be much more successful.

Silver Bullets and Snake Oil

2010-06-11T09:10:00.004-04:00

After doing test automation for most of my professional life, I'm really tired of vendors claiming that automation is a silver bullet. I'm also tired of test managers claiming that automation is snake oil. Here's the thing. The silver bullet pitch ("Automate all tests with the click of a button; no coding knowledge or original thought needed") is snake oil. I've said before on this blog that if you want to automate, you need to learn how to write script. I still hold by that. So it's the vendors' claims that are the problem, not automation itself.

Let's look at this using a different example - the microwave oven. The little cookbook that came with my microwave claims the microwave can cook any food just as good as the traditional methods. But that's really not true. While the microwave can cook just about everything, the food comes out different. Chicken comes out rubbery, for example. The button marked Popcorn cooks popcorn for too long, and often burns it. There's a recipe for cooking a small turkey, but I'm not courageous enough to try that.

Now, the cookbook is making a silver bullet pitch. "The microwave cooks everything just as good as traditional means" And, based on my experience, that's not the case. However, does that mean that I should throw the microwave away, and denounce all microwave ovens as worthless? No. It means I still use the regular oven to cook chicken, and I manually key in how long to cook the popcorn. In short, I adapt the microwave to my needs and use it for what it does well. I still have a grill, a deep fryer, an oven and a stovetop. I microwave what's appropriate for me to microwave and that's it.

Same thing for automation. I would never recommend (or even try) to automate 100% of your tests. I would recommend trying to automate tasks that are difficult or impossible to perform by hand. I would recommend automating "prep" tasks, like loading a database, building strings for use in boundary testing. I would recommend using tools to automatically parse log files for errors, rather than trying to read them by hand. The application of automation is the important thing here; you need to be smart about what makes sense to automate, just like you need to be smart about what you try to cook in the microwave.

Silver bullets are for werewolves. Snake oil is for 19th century hucksters. Automation is neither. Automation is for testers and developers who want to put in some effort to speed up their existing processes. It's just a tool. That's all.

Ethics in Automation

2010-06-09T15:36:00.002-04:00

Automation really shines when it's used to speed up a process or to address some tedious task. Unfortunately, some less than scrupulous individuals have asked me to do some things with automation in the past that are just flat out wrong. Case in point: someone once asked me to create an automated test to search on google, and then perform click throughs on a competitor's adwords. The theory was that, since the competitor had to pay each time that link was clicked, it would cost the competitor a ton of money. I'm pretty sure that Google has some sort of mechanism set up to prevent click fraud, but the fact that someone would stoop to measures like this really pissed me off. I told that person no. Firmly.

Another time, someone else asked me to write a script that would artificially inflate the number of times a particular page had been viewed. He felt that if people saw the page had a high number of views, it would seem "more interesting" to other viewers and they'd focus on it. Again, I said no. Firmly.

I don't mean to go high-and-mighty, or holier-than-thou, but the fact of the matter is that unscrupulous business practices shouldn't be performed in the first place, and they sure as hell shouldn't be automated.

Always make sure to use your automation powers for good, not evil.

And now I'll get off my soapbox.

Cloud vs Traditional: Which Load Test Approach is Better?

2010-06-03T08:07:00.001-04:00

Hi All

I'd like to get your opinions on cloud based load testing tools like LoadStorm and BrowserMob vs more traditional tools like LoadRunner and WebPerformance. What advantage do the cloud tools provide that the traditional ones don't, and vice versa?

Here's my background - I was taught how to do load testing with LoadRunner. My mentor said that you should always set up an isolated network and run tests in that environment to get your app's code running well. Once that's done, move the code to a staging/production environment and then generate load from outside the network against your production environment to see how the hardware responds.

Is this still the recommended way of doing things? If so, it seems that the cloud based tools are at a disadvantage because they can't test against an isolated network. Or, can the results generated by the cloud based tools easily identify where the performance bottlenecks are?

Please sound off in the comments.

Thanks
Nick

Just Get On the Bike

2010-05-27T09:06:00.002-04:00

There's an old story that goes something like this:

Little Timmy is very late to school one morning. It's 10:30, and the teacher looks out the classroom window and sees Timmy just getting into the schoolyard. He's pushing his bicycle and looks very hot and out of breath. When he gets into the classroom, the teacher says:
"Timmy, why are you so late to school?"
Timmy replies, "Sorry teacher, I had to push my bicycle all the way here."
The teacher, confused, says, "Why didn't you ride your bicycle?"
Timmy replies, "Teacher, I was running so late this morning that I didn't have time to get on the bike."

I hear this kind argument all the time when people put off automating. "I have too much to do," they say. "When this release settles down, I'll have time then."

Bullshit.

Folks, I hate to say it like this, but you're never going to have that perfect moment where you do nothing but automate. If you're a tester, then there's always something that will need testing. You have to find ways to work automation development in to your regular day. If automation is something that you throw to the back burner every time your workload heats up, you'll never get any benefits from automation.

Sometimes this means explaining to your manager that automation can't be a side project and should be focused on sooner rather than later. Sometimes it means recruiting a developer to help get you jumpstarted, so you can get a test framework built up quicker. Sometimes it means hiring another person whose primary responsibility will be test automation.

Point is, if you're always waiting for that perfect moment, you'll never get anything automated. So do *something*. Even if it's just a small test to start with, that's something you can build on. You won't get a comprehensive set of automated tests overnight; so get one test automated/one helpful tool written and then add to that. Even if you only automate one new test a week, over the course of a month, that's 4 tests you don't have to run by hand. Or, if you're trying to build a tool that will augment manual testing, get that to your testers as soon as it can do something useful, then continue adding features. The time savings will get more and more significant as you go along.

So, long story short, just get on the bike!

TUU: The Missing Metric

2010-05-10T10:45:00.006-04:00

Before taking the plunge into test automation, many managers want to know how long until they see an ROI (return on investment). In the case of a commercial tool, some vendors will actually send out an Excel spreadsheet with formulas that let you plug in the amount you pay a tester per hour, and then the spreadsheet calculates how much it costs to find that bug during the test phase vs how much it costs to find it once the app has gone to the customer.

While I certainly understand the need to justify an expense, I think ROI by itself is the wrong way to look at test automation, because it puts the focus on dollars rather than on utility. If you buy a $9,000 automation tool, and pay your tester $40/hour, then using the ROI approach, your test automation tool needs to run 225 hours' worth of tests before your see an ROI. If you've bought multiple copies of that tool, then you need to multiply both the total cost and the total number of hours by the number of copies you purchased. So if you bought 5 copies of the test tool, that's $45,000 you spent, and you need to run 1,125 hours' worth of tests to break even. "That's too much time or money", someone says. "Look at a cheaper tool."

What I don't see people consider is how long it will take before the automation is doing something productive. So let's say that you're considering spending money on a commercial tool. Instead of saying "How long until I see an ROI", ask "How long until this lets me do something else?" Automation is implemented so that you don't have to do the same tasks over and over; you automate to get the computer doing regression tests so you can focus on other test activities. How long will it take you to create that first smoke test? How long until you are freed up to do the new tests? Let's say it takes you 2 work days to get your smoke test up and running. That means that two days after implementing automation, you start getting some of your time back. That's how long until the automation is doing something useful. I call this Time Until Useful (TUU).

I think that 16 hours until your automation is doing something useful is a lot more pallatable than 1,125 hours until your automation justifies its existence. Plus, now you're working and finding bugs that you didn't have time to find before. I've never seen that time savings or "new-bugs-found-during-exploratory-testing-while-the-automation-runs-a-smoketest" line item included on an ROI sheet.

If you're considering open source tools, TUU is much more a natural metric. Freed from cost concerns, a test team is given the chance to look at a bunch of tools and pick the one that's going to let them get the most done the fastest. I've seen teams where managers say 'oh, you're going open source?' and then promptly go deaf because they think that since there's no cost involved, they don't need to justify any expenses, therefore they don't care. Thing is, open source tools will have a bit of a learning curve as well, and you want to make sure that your team selects one that meets their needs and still has a good TUU. If the tool is free, but it takes 2 months before it's doing anything of value, is that worthwhile? No, you'd want them to look at a different open source tool that's going to do something sooner than that.

TUU is a metric that should be considered with the same weight (or maybe with greater weight) as ROI. Whether you're going commercial or open source, you should always consider how long it will be until the automation is doing something useful.

String Generator

2010-05-07T12:04:00.007-04:00

If you've ever had to do boundary testing, you probably did what my co-workers and I used to do. You need a 400 character string, so you open MS Word, hold down the 'x' key for a few seconds, and then use the word count feature to see how many characters you've got. If you've got too many, you delete some. If you don't have enough, you mash 'x' for a few more seconds. Later, rinse and repeat until you've got it right.

Some folks would even remember to save those word files for future use, but more often than not we closed them without thinking about it. (This was usually followed by the comment "Ah, crap, I shoulda saved that...")

To help take some of the sting out of this, I wrote a string generator application. It lets users generate strings of predefined lengths, and then automatically puts those strings on the clipboard so that they can be pasted into fields.

I wanted an easy way to know that a field could accept that number of characters, so I always had the string end with a capital Z. For example, if you needed a 10 character string, it would generate aaaaaaaaaZ. Then you'd paste that into the desired field, and if you didn't see the 'Z', you knew the field was truncating the data that was entered.

The app I wrote looked like this:

Each of the buttons you see on the left would automatically generate a string of the specified length. If you needed a string of a different length, you could enter that length in the box next to the custom button.

The string generation itself was handled by this code:


private string prgGenerateString(int intLen)
 {
  StringBuilder MyStringBuilder = new StringBuilder();
  MyStringBuilder.Capacity = intLen;
  for (int i = 1;i<intlen;i++)
  {
      MyStringBuilder.Append("a");
  }
  MyStringBuilder.Append("Z");
  return MyStringBuilder.ToString();
 }

private void prgGenerateStatusMessage(int intLen)
 {
  lblStatus.Text = intLen.ToString() + " character string placed on clipboard.";
 }

private void prgCreateStringAndPost(int intLen)
 {
  try
  {
   string txtValue = prgGenerateString(intLen);
   txtPreview.Text = txtValue;
   prgGenerateStatusMessage(intLen);
   Clipboard.SetDataObject(txtValue);
  }
  catch(Exception except)
  {
   lblStatus.Text = except.Message;
  }   
 }

Each button on the form had code like this:


  private void cmd129_Click(object sender, System.EventArgs e)
  {
      prgCreateStringAndPost(129);
  }

And here's the code for the custom button:


  private void cmdCustom_Click(object sender, System.EventArgs e)
  {
      if (txtCustomLength.Text != "")
      {
          try
          {
              int val = Convert.ToInt32(txtCustomLength.Text);
              prgCreateStringAndPost(val);
          }

          catch (Exception except)
          {
              lblStatus.Text = except.Message;
          }
      }

      else
          MessageBox.Show("Please enter a value in the box next to the Custom button");
  }

Feel free to take this code and build your own string generator. (If the source code got truncated in the post, just go to View Source in your browser and you can copy and paste from there)

Enjoy!

Time Diffs

2010-04-30T09:47:00.006-04:00

During one cycle of performance testing, the tool I was using only logged a test's start and end time. For some reason, it didn't log duration. So you'd see start time 10:34:23 and end time 10:39:00. Now, in the case of a short test, say, one that ran in 15 or 20 mins, that wasn't a big deal, it's easy to do that kind of math in your head. But when the durations were like this: start time 10:32:23 and the end time was 13:18:54, it would take a minute or so for me to calculate how long that was.

This was a frustrating experience, and one that was prone to human error. So I wrote a little C# app that allowed me to enter two times and see the difference between them. This let me note the test durations much faster and more precisely. To build your own time diff tool, fire up Visual Studio and create a form that looks like this:

For reference, I named the Start Time field txtStart, the End time field txtStop, the Calculate button is cmdCalculate, and the Clear button is cmdClear. I was lazy and left the label that you see as 00:00:00 as label1.

Now, use the following code for the calculate button:

private void cmdCalculate_Click(object sender, EventArgs e)
      {
          if ((txtStart.Text != "") && (txtStop.Text!=""))
          {
          DateTime start = DateTime.Parse(txtStart.Text);
          DateTime stop = DateTime.Parse(txtStop.Text);
          TimeSpan duration = new TimeSpan();
          duration = stop - start;
          label1.Text = duration.ToString();
          }
      }

Here's the code for the Clear button:

private void cmdClear_Click(object sender, EventArgs e)
      {
          txtStart.Text = "";
          txtStop.Text = "";
          label1.Text = "00:00:00";
      }

And that's it. You now have a little app that will calculate the differences in time for you. One thing to note here is that the app isn't intended to diff times that are greater than 24 hours, and you have to enter the times in a 24 hour clock format. So if your test ended at 2 pm, you'd enter 14:00:00.

This is a nice example of creating a tool that augments manual testing. It doesn't do any testing on its own, but it does let a tester perform a given activity faster and more efficiently. Always be looking for little opportunities like this, and your testers will thank you for them.

Focus on Test Approach, Not Test Tools

2010-04-28T12:48:00.004-04:00

There's a lot of news flying around right now about how powerpoint is making us stupid. Thing is, I don't think this is a powerpoint problem, I think it's a people problem. Slides probably aren't the best method to communicate this information, and that certainly needs to be remedied. So it's the approach that these presenters are taking that's the problem, not powerpoint itself.

This got me thinking about some of the negativity that comes up when the honeymoon period with a test automation tool ends. I've seen teams whose intent/dream/goal was to create UI automation tests via a record and playback tool. They envisioned doing a single recording and all their tests would then be able to run at the push of a button.

It's a nice dream, but in order to really get the most out of your automation efforts, you really need to know what you're doing. It's impossible to automate all your tests, and quite frankly, you probably don't need to. I once worked for a manager who insisted that all bugs have an automated regression test for them. Think about that for a minute. All bugs. Every one of them. Even the ones that were citing spelling mistakes.

Needless to say, that wasn't a good use of my time, and I'm pretty sure my blood pressure was in the near fatal range for a while there.

When you're thinking about automation, think about what makes the most sense to automate. Smoke test? Yep. Get that going and integrate it with your build process. Regression test? Yep, but don't go crazy. Focus on the primary workflows of your application, don't try to automate a test for every bug you've ever logged. That whizbang new functionality that Sally just rolled into the build? Probably not.

That whizbang new functionality probably should not be automated yet because it's UI will change, and most likely will change a lot over the course of the release cycle. Instead, use automation to augment manual and exploratory testing for this feature. Let's say that Sally's piece is a report generator that pulls data from a database. Build some automation that sets up different data in the db, allowing you to test more types of her reports more quickly. Maybe you can set up a batch file that calls out to a diff tool and compares a report from today's build to a baseline.

The approach you take will largely determine your automation effort's success. Don't assume that just because you bought a record-and-playback tool that you've got test automation covered. The test tool is a dumb computer program; you're the one with the brain. Know when to employ the tools you have, when to use something else, and your testing will go much smoother.

What's in it for me?

2010-04-09T08:01:00.000-04:00

As a presenter, that's the question you always have to be asking yourself. Imagine yourself in your audience's position - you are taking time out of your busy workday to view a demonstration. As such, you want to know how this new product or service will benefit you. When you first sit down to hear the presenter speak:

You do not care about the presenter's company history.
You do not care about the awards the presenter's company has won.
You do not care about the demonstrated product's history.

Yet it's amazing how many people start their presentations off with a bunch of slides that agonizingly detail those points. When I watch a demo, I want to know if a product is going to help me do my job effectively. Once I know that it will, then it might be nice to know more about the awards and company history. But that's not what's essential.

The essence of the demo should show the audience how your product will solve a problem they have. Before you present, make sure you understand the audience's pain. Are they performing a tedious task that is prone to errors? Are they trying to speed up their app's performance? Do they need a better way to communicate between mulitple departments?

Once you understand the problem, you can apply your product's features and show how those features turn into benefits for the audience. And that's the key - too many people will show a cool feature in their product, but not state how that feature benefits the audience. Always make sure you state the benefit - "Cool Feature X is important because it allows you to process your data 4x faster than before, which saves you a ton of time over the course of a release."

By tailoring your demos so you show how to solve your audience's problems, you'll be a much more effective presenter.

The Power of the Mute Button

2010-03-26T07:51:00.001-04:00

A big challenge any remote presenter faces is that it's too darned easy for an audience participant to tune out. You're already sitting in front of your computer, watching a demo, and then an important email shows up. Or an IM pops in. Or someone drops by your cube and wants to chat. So you start typing, or chatting, and if you haven't muted your phone line, that background noise is going to come through.

This is frustrating for the person presenting. Imagine if this scenario played out in a conference room. There, the presenter would politely ask to table the side discussion to later, or see if it was something he could help with. If it wasn't related to the item being presented, the people could step out of the room and not distract anyone. But when you're demoing remotely, this isn't an option. In that situation, you need to be able to mute the person remotely. If you can't, you're never going to get your message across.

Case in point:
A few months back, I attended a webinar that Microsoft was putting on about Visual Studio 2010. The presenter had to stop several times to ask people to mute their lines. Now here's the thing - if you're the person who's stopped paying attention to the call and is now typing, chatting, or playing video games, you're not going to notice the presenter asking people to mute. So he had to contend with a metric-crap ton of background noise. As an audience member, it was extremely frustrating to try and filter out the noise from the presenter's message, and to be honest, I can't remember much of what he talked about.

When I give remote demos, I have the ability to mute people remotely when stuff like this happens. The program I use (GoToMeeting) also has a chat window. So if there's a lot of background noise coming from a particular participant, I'll say "Ok, folks, it sounds like -insertPersonNameHere- has a really sensitive microphone that's picking up a lot of background noise. I'm going to mute them, but if they have questions, they'll be able to let me know via the chat window, and I'll unmute them."

So make sure you have the ability to mute people remotely. It will only make your presentations run smoother both for yourself, and for your audience.

Learning Regular Expressions

2010-03-19T07:26:00.000-04:00

During the course of your automated test efforts, you'll come across regular expressions. Regular expressions are an extremely powerful tool, but they're daunting when you're first learning them. I mean, really, can you just look at this line of code and tell me what it does?

re = /^((4\d{3})|(5[1-5]\d{2}))[ -]?(\d{4}[ -]?){3}$|^(3[4,7]\d{2})[ -]?\d{6}[ -]?\d{5}$/;

The above regular expression is used to verify that a credit card number is valid. (This is also a great example of why you should comment your code.) Such an expression is very useful, but how the heck do you write these things?

Fortunately the good folks over at Regular-Expressions.info have put together a great tutorial that explains regular expressions in detail. You'll learn what regular expressions are good for and how to write them. This is an extremely useful skill for an automator. You can perform more robust verifications with your tests, and you can use them to do other tasks like pull particular strings out of log files, or ensure email addresses are properly formatted.

It's also handy to have a tool that you can use to verify your expressions are working properly. For that, I recommend Roy Osherove's Regulator. It's free, and allows you to test your regular expression against a variety of inputs, allowing you to make sure you'll correctly match the patterns you're expecting.

Happy Automating!

Test Workflows, Not Features

2010-03-12T07:00:00.000-05:00

So you've got a shiny new feature in your application. Maybe it's a new button, or a new context menu item. You get your new build, load it up, and click the button. It works - Huzzah! Test passed, move on to the next feature!

Whoa, slow down there, chief. You've verified that the feature works in isolation, but how will your end users be using this button? Will they be clicking this button all by itself every time? Chances are there's a whole workflow that this button will be involved with, and you need to make sure you understand what that workflow is. So figure out what your users are going to do with this feature, and how they're going to do it. What other steps will they be performing before using this feature? What options could they set that would impact how this feature behaves?

Only when you have a good grasp on how the new feature will actually be used can you say whether or not it's working properly. Design your test cases to reflect real world behavior, and your application will be much more solid when it ships.

Forcing a BSOD

2010-03-05T07:10:00.000-05:00

The Blue Screen of Death is one of the most unpleasant events a person can face. It ranks right up there with doing your income taxes, the unexpected visit from your least favorite relative, and hemmorhodial flare ups. However, there are times when you actually want to have a BSOD event happen. Maybe you're doing some extreme failover testing, or maybe you want to show how your software can recover from a disastrous situation.

If that's the case, then you need a consistent way to force a BSOD. The steps below show you how to do it:

Go to Start>Run, and launch the Registry Editor (Regedit.exe).
Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\i8042prt\Parameters.
Go to Edit, select New | DWORD Value and name the new value CrashOnCtrlScroll.
Double-click the CrashOnCtrlScroll DWORD Value, type 1 in the Value Data textbox, and click OK.
Close the Registry Editor and restart your system.

When you want to force a BSOD, press and hold the [Ctrl] key on the right side of your keyboard, and then press the [ScrollLock] key twice. The BSOD will appear.

Note: If your system reboots instead of displaying the BSOD, you'll have to disable the Automatically Restart setting in the System Properties dialog box. To do so, follow these steps:

Press [Windows]-Break.
Select the Advanced tab.
Click the Settings button in the Startup And Recovery panel.
Clear the Automatically Restart check box in the System Failure panel.
Click OK twice.

Intepreter

2010-02-26T08:37:00.005-05:00

In the movie "The Fifth Element" Bruce Willis turns to another character and says, "Lady, I only speak two languages - English and Bad English."

And those are exactly the two languages we all need to speak in order to succeed, whether we're testers, developers, sales people or executives. The first is to ensure that we can communicate our ideas clearly and effectively to our audiences. The second is so we can understand and interpret what our customers, co-workers, and others are saying.

Now understand, when I say Bad English, I'm not referring to people who use dangling modifiers, end sentences with prepositions, or any of the other Grammatical Cardinal Sins that your 10th grade teacher warned you about. I'm refering to the rambling stream of consciousness email messages that some people send; the people who speak only in acronyms; the people who use obscure regional slang. In short, I'm saying you need to speak the language of the people who *can't* communicate effectively, so that you yourself *can*.

If you can read a confusing email message, interpret it correctly, clarify it, and then re-communicate it, you'll be able to help others get their messages across, and you'll be able to get your own messages across more effectively because you'll understand how these people think. How many times during a sales cycle or a development cycle have you seen other people bounce emails back and forth, never quite getting their points across, because they weren't speaking each other's languages? I've seen that more times than I can count, and it's a frustrating experience.

So how do you start becoming an interpreter of Bad English? For starters, you need to understand the context of the user's message. What feature or service are they talking about? Are there keywords in their message that might shed some light on what they're looking to do? Put the message in your own words, using the simplest possible language. Once you've done that, run it by the original sender to make sure you've understood correctly. If they say yes, great. If they say no, they'll elaborate on their situation. Don't worry about getting it right the first time; people will appreciate that you're trying to help them.

There are tons of reference materials out there on how to communicate effectively - just poke around on Amazon. If you want examples of Bad English, just ask your support team to show you the messages they receive every day. It's amazing how little detail or thought people will put into a message that's asking for help. It will show you how your users think, and that will help you communicate more effectively with them.

This same principal applies to any actual language - French and Bad French, Spanish and Bad Spanish, etc. The important thing is that you learn to understand what people are saying, and are able to respond to them. The other piece to this is that you're able to effectively communicate and convey your own ideas. Once you can do that, you're one step farther on the path to success in your field.

Perfomance Testing 101 - 4

2010-02-19T12:13:00.002-05:00

The last thing I want to talk for performance testing about is tools. There are a lot of great tools out there, both commercial and open source. Do some Googling to see what's available, and what other people think of those tools.

Spend some time evaluating tools before you make a decision. Make sure that the tools can provide the information you need in the format you want before you pull the trigger.

While you're running your load tests, you're going to need to keep tabs on your server's behavior. Make sure that the tool you're working with can monitor the server's resources as unobtrusively as possible. If your tool of choice doesn't have server monitoring capabilities, then you can use a tool built into Windows called Perfmon. Perfmon lets you keep tabs on many things, but at the miniumum, you'll want to track CPUUsage, Available MBytes (the total amount of RAM available on the system), Bytes Sent and Bytes received. This will give you a flavor for how your system's doing overall. You can also track items related to SQL databases or the .NET framework, so look through the list of what's available and see what meets your needs.

As you can see, there's a lot involved with performance testing, and these posts have just touched the surface. The thing to remember here is that performance testing is *very* different from functional testing. You'll need to learn the difference between a 302 and a 304 http return code. You'll need to understand what parts of your application uses system resources and why. If you've never done performance testing, be ready for a pretty steep learning curve. Make sure you budget time, whether it's on or off the clock, to learn as much as you can.

I'm not a performance testing expert by any means, so now I'll point you to the people who are :)

Corey Goldberg has a great blog and builds performance testing tools. Definitely check him out:
http://coreygoldberg.blogspot.com/
Twitter: @cgoldberg

Scott Barber is a perfomance testing guru:
http://www.perftestplus.com/scott_blog.php

Bob Dugan is an extremely smart performance tester and a great guy to work with:
http://www.stonehill.edu/x8127.xml

Here are a couple of helpful books on performance testing:

Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software
http://www.amazon.com/Performance-Solutions-Practical-Creating-Responsive/dp/0201722291

Performance Testing Microsoft .NET Web Applications
http://www.amazon.com/Performance-Testing-Microsoft-Applications-Pro-Developer/dp/0735615381/ref=sr_1_2?ie=UTF8&s=books&qid=1266416753&sr=8-2-spell

Performance Testing 101 - 3

2010-02-12T08:30:00.003-05:00

Now that we've talked about scenarios, the next thing I want to discuss is your performance testing environment.

Your environment should be an isolated, standalone network. All "normal" network activities like virus scans, automatic system updates and backups should be disabled. You don't want anything running that could skew your test results. Now, some people may question this, saying it's not a "real world" scenario. At this point, you're trying to see how fast your system can perform under the absolute best conditions. It's a way to see, "ok, we are absolutely positive we can handle this number of transactions/second." My experience with this has been if you have other things running, and the numbers come back bad, people will immediately jump on that as the cause. "Oh, you had a virus scanner running - that's probably messing up the results. Uninstall the scanner and run the tests again" If your tests take days to run, that's not what you want to hear, especially if the test results come back the same as they were before.

Having your tests on an isolated network also ensures that normal day to day traffic isn't skewing your results. I once had four hours' worth of testing negated because someone three cubes down started downloading MSDN disks. I wound up having to do my tests at night after everyone had gone home in order to ensure there wasn't any rogue traffic. Again, having an isolated network will save you a lot of testing time and frustration in the long run.

The other thing to keep in mind is that your tests systems should match up to what you'll be using in production. So whatever hardware your system will run in the real world is what you should be testing with. Some IT folks balk at this, citing cost as a problem. Thing is, there are issues you won't be able to find if you're just testing with a bunch of hand-me-down systems. If you're not running on a box that has the same type of processor, how will you know if you're taking full advantage of that processor's architecture? This extends beyond just the systems themselves. Switches, NICs, even the network cable involved can impact your tests' performance. Don't fall into the trap of "well, we'll be using better hardware in the field, so performance will be better", because it might not.

Performance Testing 101 - 2

2010-02-05T07:56:00.000-05:00

Ok, so now that we've taken care of terminology and gotten our performance baselines, it's time to start thinking about configuring our scenarios. This is where you figure out how many people will be performing a given action simultaneously. Again, if you are testing a pre-existing website, you can go through your web server's logs to see how many people typically access a given page at a time.

If you're testing a never-before released site, though, this part gets tricky. You'll need to make some educated guesses around your user's behavior. Let's say we're testing an e-commerce site, and we're going to roll it out to 1,000 users. Let's further assume that these users are equally divided between the east and west coasts of the USA (500 on each coast). For test cases, we want to perform logins, searches, and purchases.

Now, the knee-jerk reaction is to say that since you have 1,000 users, your system should be able to handle 1,000 simultaneous logins. That may be true, but is it a realistic scenario? In the real world, your users are separated by time zones, some of them may not be interested in purchasing a product today, some may be on vacation, some of them will only shop while they're on lunch break, and others will only shop at night after their kids have gone to bed. So in considering this, you may want to start out verifying that your system can handle 250 or 300 users logging in simultaneously. Then build up from there.

The next thing to consider is the frequency of the actions. Each user logs into the site once, but once they're in, how many times do they perform a search? If you go to Amazon, how many products do you search for in a single session? Between physical books, eBooks, movies and toys, I'd say I probably search for 2 or 3 items each time I log in. So that means searches are performed 2 or 3 times as often as logins. So if you're planning on having 250 - 300 users login to start with, then you're looking at 500 - 900 searches. Also consider the purchase scenario - not everyone that logs in and searches is going to buy something; Maybe half of the people, or a third of the people who login will actually make a purchase. So you're looking at 75 - 150 purchases for a scenario there.

So take some time and think about your end user's behavior when defining scenarios for load testing - don't make it strictly a numbers game.

Next week, we'll talk about your performance testing environment.

Performance Testing 101 - 1

2010-01-29T06:08:00.000-05:00

I've seen a lot of people jump into performance testing recently, and they've all encountered similar problems. So each Friday for the next few weeks, I'm going to post a few tips that will hopefully help folks out.

Let's start out with some basic terminology here. Performance testing is making sure that your app responds in a timely fashion. Note that this does not mean that you need to have hundreds or thousands of virtual users running in order to do a simple performance test. Consider this scenario: Let's say I'm testing a calculator application. I entered 2+2, and then pressed the = button, and got 4. From a functional testing standpoint, the test passed. However, if it took the calculator 30 seconds to return the answer, then we have a performance problem. That's all performance testing is - making sure your app responds within a user's expectations.

Load testing is what most people are thinking of when they refer to "performance testing." This is where you're expecting that 500 people will be performing a given activity on a web site, so you use a tool to emulate 500 people perfoming that activity. So load testing is ensuring that an application responds in a timely fashion while an expected number of people are working on it.

Stress testing is where you try to find the breaking point of your application. So if you've established that your system can handle the expected 500 users, can it handle 600? 700? More? This is where you push the system to its limits and find out what number of users make it fall over and die.

Ok, so now that the terms are out of the way, the first thing you should do is figure out the scenarios you want to performance test. If you're testing a site that's already live, you can cull through your web server logs to find out which pages are being hit most frequently. Then build your scenarios based on your actual customer behavior.

If you're testing a site that hasn't been released yet, then think of those activities that your users will perform most frequently. For example, if you're testing a website like Amazon, your most frequent scenarios will be things like searching for books, logging into the site, adding books to the cart and processing orders. Start with those, and then consider other activities. Remember, the point of load testing is not to do a functional test with hundreds of users, so you don't need to performance test every page or feature on your site. For example, how often do you search for a book on Amazon, compared to how often you change your login password? Those less often performed activities can be tested later, but they probably aren't high priority.

Next, run a single user performance test for each scenario. This will establish your current performance baseline. No extra tools other than a stopwatch or a wristwatch with a second hand are needed. Launch your application, and perform your test. For example, let's say one of your performance tests is to log in to a website and get to a Welcome screen. Enter your username and password, start your timer, and click the login button. Once you've reached your Welcome screen, stop the timer. Note how long it took to perform the login. Share this number with your developers and PMs. Let's say it was 2 seconds. Is this number acceptable? What's the acceptable range for this number? Is it ok if it goes up to 4 seconds?

Alternatively, if it takes 10 seconds to login, then you've already found a performance problem, and running the same test with more users probably won't yield any additional info. Just focus on getting the login process down to the desired 2 seconds, and then you can begin loading additional users.

We'll talk more about scenarios and their configuration next week.

Dealing with Ego

2010-01-27T09:28:00.002-05:00

I've been very lucky to work with some very talented developers over the last 11 years. (Wow, I suddenly feel old :)

However, we've all been on those teams where there's been a primadonna, that one person who thinks he or she is God's gift to software, and that everything they touch turns to gold. These people typically see testers as evil little gremlins who are hellbent on destroying their perfect code. As such, they display a certain level of standoff-ness to testers; maybe they aren't willing to answer questions, maybe they dismiss bugs as extreme edge cases, you get the idea.

I've always believed that my ego isn't part of my work. If someone finds a hole in a test I wrote, it's not personal. They aren't criticizing me. Most people I've worked with share this belief. It's not fun when someone finds bugs in the code you wrote, but it's not a personal attack either.

Sometimes just having an open conversation stating the above to the primadonna corrects the problem. Other times, they get even more huffy. So here's a trick I used on one primadonna. He was starting to brush off some of the concerns I was raising. I lifted a finger and said to him, "Mike*, my job as a tester is to make you look good. And if these items get addressed, the product's going to work better, people are going to love it, and as such, you're going to look great. So come on, help me make you look great."

He gave me all the help I needed after that. Sometimes it's all in how you play things.

*Mike wasn't actually his name. I always change names to protect the innocent. Then again, I'm not sure how innocent he was. His name was Lou.*

*Kidding! :)

The Most Important Qualty in a Tester

2010-01-25T06:34:00.000-05:00

There are a lot of things that make a person good at a testing job. Technical inclination, an insight in to the end user's mind, and keeping up with new methodologies are a few. But I'd wager the most important thing of all is curiosity.

A really good tester wants to know how & why things work the way they do. If they encounter something they've never seen before, they start learning about it. They scour the web looking for articles on new technologies, they speak with their developers to get a better understanding of it, and then turn that new information into test cases that will be more insightful and useful.

You can spot a good tester by the questions he or she asks. If they see something they don't understand, they ask about it and keep asking questions until they get it. They'll propose user stories and scenarios to make sure they've got a handle on things, and then make those the basis for their test cases.

This really translates to all walks of life - whether you're a tester, a developer, a PM or a salesperson. Ask questions. Learn everything you can, and your career will go some really cool places.

Online Presentations and Slide Animations

2010-01-22T06:30:00.000-05:00

A lot of people like to spruce up their presentations by adding animations. And if you look through the powerpoint menus, you've got tons of choices. You can have things fly in, fly out, materialize in, blink in, wipe in... you get the idea. They're fun, they're exciting... and they absolutely suck in an online presentation.

The issue is with screen refreshes. Some participants in your online presentation won't see a fluid animation. Instead of a smooth fly-in, They'll see a jerky motion where your text lurches across the screen. Instead of a neat Star Trek type beam-in animation, they'll see squares and lines.

If you're giving an online demo you have two choices for animations: Appear and disappear. They aren't flashy, they aren't sexy, but they work best in an online environment. I suggest you stick with these even in an "in person" presentation, too. They aren't as distracting as the other animations, and the less distracting your slides are, the more focused your audience stays on you.

Helpful Tool: Jing

2010-01-20T06:57:00.000-05:00

We've all had times when someone, whether it's a co-worker, a customer, or a relative has sent an email saying "Hey, I've got this weird problem, can you take a look." They describe the problem, and you immediatley realize what it is. Then you let out a long sigh, because even though you know exactly what the problem is, typing out all the instructions to perform the fix is going to take you a while. Or worse, it's one of those problems whose solution really needs to be seen in order to be understood.

Enter Jing. Jing is a free screen capture utility that lets you create 5 minute videos (with audio) and then upload them to the web. I've used this successfully to show customers how to perform various operations, and the feedback has been tremendous. People really appreciate that I took the time to make this little video, and now they can share it with their co-workers. Jing has a very clean UI and is extremely easy to use. You can download it from here: http://www.jingproject.com/

The Rule of 10

2010-01-18T11:58:00.000-05:00

When I was in college, we had a rule that if a professor was 10 minutes late for class, then the class was automatically canceled. I have a similar rule for online demos with my sales team.

It goes something like this. Let's say we've set up the demo for 2 pm. My sales rep and I will typically log on between 1:50 and 1:55. If the customer has not logged in by 2:05, the sales rep calls the customer. If there's no response, we wait another 5 mins and try calling again. If there's still no response, we cancel the call. We then leave a message saying something to the effect of : "We're sorry we weren't able to connect with you today, please give us a call and let us know when you'd like to reschedule the demo."

Do you have any similar approaches for when people don't show up to demos or meetings? Please let me know in the comments.

Never Stop Learning

2009-09-09T09:00:00.000-04:00

What's the one thing that consistently sets a successful person apart from the rest? Knowledge. The most successful people I know are always reading, watching industry trends, and keeping their ears to the ground for what's new & exciting.

I subscribe to about 30 different RSS feeds on coding, qa, and sales engineering. A lot of the people I follow on Twitter work in those fields. I read up on what's hot in their fields, and then I can speak more intelligently when I'm asked a question about Visual Studio 2010, agile test automation, or how to give a better presentation. That information helps me do a better job, which keeps me employed, which is a happy thing :)

No matter what your role, whether it's sales, testing, development, or something else, there's always something new for you to learn. Keep learning, and you'll never be bored, you'll keep your skillset current, and you'll be able to do your job more effectively.

Helpful Tool: Reflector

2009-09-04T09:00:00.000-04:00

Everyone who works with .NET applications should know about Reflector. This is a handy little tool that lets you inspect the code that makes up your application. Need to know what the heck one particular assembly does, and you don't have access to the source code or the developer who wrote the code? Reflector will show you all the methods and classes that live inside the app, what parameters they take, and so on. There are a bunch of plugins available too, that let you view more info about Silverlight apps, get quality metrics, and so on.

Check it out at http://www.red-gate.com/products/reflector/

Even Robots Need Maintenance

2009-09-02T09:00:00.001-04:00

When people are first exposed to test automation, one of the questions I often hear is: "If the application changes, like if they totally reorder & redesign the screens in a given workflow, will my automated test break?"

Well, yes. Your manual test cases will need to be changed as well. Your manual testers may also need to be retrained on your app if it changes that dramatically. The point is that your automated tests are going to need to change and evolve as your application changes and evolves. Anyone who tells you that an automated test is bullet-proof and can't be broken is selling something.

So yes, you will need to refactor, repair and update automated tests as you progress. You can make your tests change resilient, but they'll never be change proof. You'll need to plan for those changes in your release cycles, just as you would any other test activity. Be aware of that, keep it in mind, and your automated test efforts will go much smoother.

What do you do to plan for change in your automated tests? Sound off in the comments.

Helpful Tool: BGInfo

2009-08-31T09:06:00.000-04:00

If you have a lot of systems in your test lab, it can be hard to remember what each one has for specs. Enter BGInfo. This handy little app displays all manner of system information as your desktop background, including CPU, RAM, IP Address, System Name, and more. You can also customize it with script to show any other relevant info, like what version of MS Office or Firefox a particular system has.

BGInfo is free to download from this location.

Enjoy!

Helpful Tool: Recuva

2009-08-28T09:06:00.000-04:00

"Oh Sh*t!"

We've all heard a co-worker say something like that when they've accidentally deleted a file. Sometimes, it's in the Recycle Bin, but for those unfortunate souls who press Shift+Delete, deleted a file from a flash drive, or who emptied the Recycle Bin without thinking, there's Recuva.

Recuva's a great little tool for recovering deleted files. It's free, takes limited system resources, and can be run from a flash drive. It's a great tool for anyone who troubleshoots computers.

Check it out at http://www.recuva.com/

Monitor Your Environment

2009-08-26T09:07:00.000-04:00

It's important to keep tabs on what's installed on your test systems. Changes in your environment can alter test results and really throw a wrench in your day.

I was running performance tests once. My developer and I had tweaked the systems to their best performance, and the automated tests had been refined. We ran tests before we went home that night, and confirmed our results were good. The next morning, our VP wanted us to run the tests again so he could watch. Imagine our surprise when all our operations were taking 1.5 - 2x longer.

My VP immediately suspected that the test tool was faulty, and that's never a good way to start a discussion. Luckily, I had a written a python app that ran in the background on all my systems. It was a simple little affair, it polled the Uninstall section of the Windows Registry and logged anytime a value was added or removed. During the night, a windows update had been automatically applied, which adversely impacted my system's performance. Without my utility, there's no way I would've caught that.

We spent a couple hours on the phone with Microsoft and got a workaround to the problem. That done, we ran the tests again, and viola, everything was happy. I quickly disabled automatic updates on all my systems from that point on.

I lost a couple of hours to this problem, and my test automation's rep was literally on the line because of an automatic update. So keep tabs on your environment, and make sure you know what's happening to it.

Automation Honors

2009-08-24T10:45:00.002-04:00

I've just found out that my blog is a finalist for an Automation Honors award from ATI. ATI is a great resource for automated testing, and I'm honored to have been nominated.

Please head over here and vote for me!
http://www.automatedtestinginstitute.com/home/index.php?option=com_mad4joomla&jid=2&Itemid=137

You can learn more about ATI here: http://www.automatedtestinginstitute.com

Test Automation Does Not Replace Testers

2009-08-24T09:07:00.000-04:00

Back when automobile companies started putting robots on the assembly line, lots of blue collar workers lost their jobs. The robots were more efficient, faster, and didn't need bathroom breaks. By switching over to an automated process, the auto companies produced more cars faster. And according to them, they were produced cheaper and more efficiently.

Some folks have the same thought about automated testing. "If we automate all our tests, we can lay off half our test team! Think about the cost savings!"

These people are idiots.

See, testing isn't like auto manufacturing. An assembly line worker who's putting wheels on a car does just that. He's not inspecting the wheel for defects, he's not verifying the welds on the axle where the wheel goes, he's just attaching the wheel. There's an inspector later on down the line that checks the work. That assembler's work can be automated with no problem, because he's doing a simple, repetitive, monotonous task.

Testing is different. Yes, there are monotonous regression tests that need to be run, but even when your testers are following a test case, they're still observing beyond what's in the test. If a test step says "Click the OK button to see the login screen" and doing that shows the login screen, that's great. But if clicking that button shows the login screen and also turns the screen bright pink, a tester will log a bug, even though the scripted behavior is correct. Robots don't think beyond what they're told. They can't deduce, reason, or infer. Remember that.

Also, when a robot replaces an assembly line worker, it completely replaces all tasks that worker did. In my example above, the only thing that worker did was put tires on a car. It's highly unlikely that your manual test team only has a handful of test cases. More likely, they're scrambling to make sure the basic functionality test cases are covered. Automating the basic tests will free them up to work on more advanced tasks, which, let me assure you, there is no lack of.

Automation augments your testers, and it lets them work more efficiently. But it should never be viewed as a way to replace the people on your test team.

Helpful Tool: Process Monitor

2009-08-21T12:56:00.000-04:00

Have you ever wanted to watch and see exactly what files your app was calling, what registry keys it was working with, or what dlls it was loading? Then Process Monitor is for you. This handy little app can tell you everything that's running on your system, or can be filtered down to a specific process. You can download it for free from here.

It's Windows only, so if there are similar tools that you use for Linux, please sound off in the comments.

Helpful Tool: WinMerge

2009-08-19T12:55:00.000-04:00

Need a tool that can do a diff of those really long reports you just generated? Have two script files that look like they contain the exact same info, but they're behaving differently? You need a diff tool. From their site:

"WinMerge is an Open Source differencing and merging tool for Windows. WinMerge can compare both folders and files, presenting differences in a visual text format that is easy to understand and handle"

WinMerge is Windows only, so sound off in the comments if you have a favorite diff tool for OS X or Linux.

You can download WinMerge from here.

Helpful Tool: Wireshark

2009-08-17T20:55:00.000-04:00

When you're doing web testing, whether it's functional or performance, it's always a good idea to be able to "see" exactly what's going across the wire. That's where Wireshark comes in. Wireshark is an open source packet monitoring tool that lets you see each individual request that's made, and the response that gets returned. It's available for Windows, OS X & Linux, and can be downloaded for free from here:

www.wireshark.org

Simple Tests

2009-08-13T09:03:00.000-04:00

There's a great episode of Star Trek: The Next Generation where the omnipotent Q is made human. One of the tasks facing the crew during this episode is a moon that's losing orbit and threatening to crash into its planet. Q tries to help Geordi and Data prevent this disaster, and tells them that there's a simple solution. Geordi, excited, and asks what it is. Q's response: "Change the gravitational constant of the universe." Geordi's less than thrilled with this answer to say the least, but that's how Q would have handled the situation.

I see something similar when people are first evaluating automated test tools. They come up with a "simple test" and if the tool fails to perform that test, they immediately write the tool off as useless.

It's true there are times when a given automated tool isn't a good fit for a particular application, but keep perspective here. You, the tester, are on the same level as an empowered Q. You see everything in your application, and know how to make it work. The test tool doesn't have your ability to reason, to identify the cause of problems or to adapt. The test tool doesn't "see" the application the same way you do - it sees your application the way a *computer* sees it. You see a button labeled OK. Your test tool sees an extended Winforms control with a dynamically generated ID like cmdOK87823, or worse, the code may be obfuscated so that a test tool can't read any information from it.

The tool is operating on a much lower level playing field that you are, and it finding that dynamically generated button on an obfuscated application is just as impossible as it would be for Geordi to change the gravitational constant of the universe. So what to do in situations like this? You find ways to make the tool work with your application. In the TNG episode I mentioned, Data & Geordi found a way to send a warp field around the moon that would do something very similar to Q's solution. To find your dynamically generated controls, you could insert wildcards into your test scripts, so the tool would find cmdOK*, and thus match whatever identifier had been dynamically generated. You may need to run your tests against unobfuscated code. Just be aware that tasks like this may be a necessity, and you'll be much better off when it comes time to automate.

Snap Decisions

2009-08-12T09:01:00.000-04:00

Occasionally someone will contact me on a Wed afternoon and tell me that they need a demo of my software right away, because they have to make a decision whether to buy it tomorrow. Now, some people are saying this because they think they'll get attention faster, but some people legitimately plan on making a purchase within 24 hours.

These calls scare the daylights out of me. My company provides a 30 day evaluation of our software so that people can try it out and make sure that it works well for them. I can show things off against sample applications, but at the end of the day what matters is that it works with their application. By relying solely on my demo, that second part is completely overlooked.

I liken this to buying a car. Would you walk into a dealership and purchase a new car just based on the commercial you'd seen? No, you want to take it for a test drive. You want to know how well it corners, how loud the engine is, how comfortable the seats are.

So if you have someone on your team who's pushing for a snap decision on a product, do everything you can to keep that from happening. Make sure some time has been spent ensuring that the program meets your needs. Otherwise you could end up in a bad spot where a tool has been purchased, it doesn't work, and now you have no money to get something else in place.

Automating Installs

2009-08-10T09:30:00.000-04:00

A common scenario in a manual test case is something like this:

1 - Uninstall old version of application
2 - Install new build
3 - [Perform actual test here]

When people are starting out with test automation tools, they often want to automate steps 1 & 2. This makes complete sense, but the approach taken is almost always the wrong one. I've seen many people try to use record & playback tools to open the control panel, click Add/Remove programs, and uninstall.

Now, conceptually, this shouldn't be a big deal, but when you figure that there are differences in the control panel in almost every version of Windows, your recorded script will break quite easily. So instead, a better option is to use command line flags to remove your application.

Almost all the major install building programs allow for the creation of command line parameters. This lets you install your app with a command like "myapp.exe /AcceptLicenseAgreement /InstallToDefaultLocation" They also usually have uninstall commands as well. Installing/uninstalling your app in this fashion makes it a lot easier to get new versions of your programs loaded for use with automated tests.

Talk with your build engineer to get a list of the commands available for your product, and if there aren't any, work with him or her to get some implemented. It will only make your life easier in the long run.

LMGTFY

2009-08-07T11:20:00.000-04:00

We've all had those moments where someone emails us a question that could've been answered with a simple Google search. Things like "What does IIS stand for?" "How many GB in a TB?" "What's the capital of Iceland?" (ok, I've never actually been asked that one, but you know what I mean).

For those moments when you're feeling just a little snarky at someone for throwing one of these questions your way, there's Let Me Google That For You. LMGTFY takes a question and creates a URL that you can email in response to the question. When your associate clicks the link, they'll see a vid of the question being typed into the Google home page and the Google Search button being clicked, followed by a message saying "Was that so hard?" Then the user is taken to the actual results of the search.

Here's a sample with the aforementioned "What's the capital of Iceland?" question. Enjoy!

Blind Faith

2009-08-05T11:26:00.001-04:00

Your automated tests are great. They run unattended every night, and each morning, you come in and see a little report on your desktop with a bunch of green success messages. You feel good. You know those tests are making sure the basic functionality of your app is running smooth, and there hasn't been a failure in days.

Then you give the first build of the app to the test team. Within an hour, 30 high priority bugs have been logged. You review the bugs and see that close to half of them are scenarios that are covered by the automated tests. Stupid testers, you think. They must be doing something wrong. You kick off your automated suite, and it passes. Then you try to run the same test by hand, and it fails. You try it again by hand, and it fails again.

Then you take a look at the automated test, and realize there's a flaw in its logic. You see that the test is configured to always pass, regardless of what's actually happening. You quickly dive into the code and fix the application, and then you go back and fix the tests.

The moral of the story here is to make sure you know exactly what your automated tests are doing. Have them code reviewed just like any other part of your project. Just because a report gets spit out saying everything passed, doesn't mean everything did.

Ninjas or Pirates? Both.

2009-08-03T11:09:00.000-04:00

There's an age old debate on who would win in a fight, ninjas or pirates. But as I once saw in a Nodwick comic, if you create Ninja Pirates, then you've got something absolutely unstoppable.

For some reason, this got me thinking about when I hear people argue about whether it's worth it to exclusively do automated testing, or exclusive manual testing. On their own, both techniques are powerful, but you're going to find the most bugs when you combine the two techniques. So augment your manual tests with automated tools that will let your testers do things faster and more efficiently. The end result will be well worth it.

Plus, you can brag to your friends that you've got a bunch of Ninja Pirates working for you.

More than One Right Way

2009-07-13T10:40:00.000-04:00

In testing, we often focus on there being one right way to do something. This may be known as the "happy path" where you know a particular test scenario works when all the input parameters are exactly right. Then we charge off the happy path into the dark unknown, where we expect to find bugs. But as this Lifehacker post reminded me, sometimes there's more than one right way to do something.

This video (http://lifehacker.com/5311002/open-a-banana-like-a-monkey) shows how to open a banana from the bottom. This was a bit of a shock to me, as I've been using the stem of a banana like a pull tab my whole life. It's actually easier & cleaner to do the method shown in the vid. Either approach works, but they're both right.

When you're creating your test plans, take a few minutes and think about how many happy paths may exist. They'll help you create more thorough test cases and help you uncover more interesting bugs.

What goes down, must come up

2009-07-01T15:09:00.001-04:00

I have to admit, I'm a soda junkie. Pepsi's my preferred beverage, but I love Jolt, Coke, Mountain Dew (especially the World of Warcraft varieties), root beer and a bunch more. But I've learned the hard way that it's a *very bad* idea to have a Pepsi at arm's reach while doing a demo. The reason? Carbonation makes you burp , and depending on how much you drank, you may be burping a lot. A second issue is that odd 'quasi burp', where your throat constricts because you can feel a burp coming, but it doesn't. Either of these makes you look less than professional, and you don't want to have various bodily noises distracting people from the message you're trying to deliver.

So, avoid the carbonated beverages for about 30 mins before you give a presentation. Have a bottle of room temperature water handy in case you need a drink. (If the water's too cold, it can constrict your throat, and that's a whole different problem. )

Once you've given your killer, burp-free presentation, crack open a Pepsi for me!

Pull Up

2009-06-22T11:26:00.001-04:00

Back when I was first working in automation, my AUT was a java based course player. Users could learn about how to be a good manager, how to shake hands in Australia, and a myriad of other soft skill topics. At the end of each learning session, the user would be given a 10 question multiple choice quiz on what they learned.

My job was to have my automation tool launch courses, answer questions, and confirm that the scoring mechanism in the app was behaving properly.

What made this particularly challenging to automate was that the questions were displayed in random order, and there were anywhere from 30 - 50 potential questions per quiz. So you never knew which questions were coming, or what order they were coming in.

I voiced my concern to my team lead, and she said that I should maintain a list of every question that could be listed in the course, and the answers to those questions. Now, that might have been ok if I were only dealing with one course, but there were dozens of courses I had to work with to ensure the app was behaving properly.

One of the things I learned early on was to "pull up," which is my way of saying take a breath, step back, and look at the whole picture. Is there a better way to handle this? What I realized was that somehow, the AUT knew what the right answer to any question was. If Choice C was right, then the AUT knew that. It was so blindingly obvious that I was ticked at myself for not seeing it sooner.

I went over to one of my developers and spent the next 10 minutes discussing my situation, my intended solution, and what was required to implement it. By the end of that afternoon, I had a method in place that would call out to the AUT and tell me which answer was correct for any given question.

It was a heck of a lot cleaner than maintaining a question/answer list, and it made for good "press" internally with automation. It also raised my developer street-cred, because folks saw that test automation was a lot more than just point&click record and playback.

So when you face a challenge in your automation, remember to pull up and make sure that you're not missing something obvious.

The Kindle

2009-06-11T14:50:00.000-04:00

My family bought me a Kindle 2 for my birthday a couple months ago, and I am totally enamored with the device. I really do forget that I'm not reading a physical book. The electronic ink the Kindle uses is crisp, and very easy on the eyes. The only downside is how easy it's become to drop a ton of money at Amazon.

Here's a brief sampling of some of the titles I've picked up so far:

The Prince. This book should be in everyone's library. Machiavelli has a bad rep, but this book is extremely insightful. I have a paperback copy, but it's getting pretty tattered.

UR. I'm a big Stephen King fan, and this novella is a story about the Kindle. He ties it in nicely with the Dark Tower and From a Buick 8.

Agile Testing - A Practical Guide For Testers and Agile Teams. This is a great overview of agile testing, and what a tester's role is in an agile environment.

The Little Red Book of Selling - It's marketed as a sales book, but this little gem can be applied to achieving pretty much any goal you may have.

Goblin Hero - If you've ever read epic or heroic fantasy, you owe it to yourself to pick up the Goblin books by Jim C Hines. They give you a look at what it's like from the Goblin's point of view, and they're really funny.

I'll post more in-depth reviews as time goes on. What are you reading these days?

State

2009-06-03T09:31:00.000-04:00

One thing to remember with UI automation is that your application needs to be in the same state each time the test starts. You need to be on the right screen, with the right options enabled, and your test goes from there. This makes perfect sense to me, but I'm amazed at the people who get frustrated with that. They say "The tool should just know."

Now I agree that it's good to build logic into your tests that can get you onto the right screen if you aren't there already. It's also good to build recovery scenarios into your tests so you can get back to a good state if something goes wrong. However, to assume that the tool is just going to know what to do is ludicrous. The computer is a stupid box. You are the one with the brains. You tell the box what to do and it does it.

Come on, do you really want Skynet running your test efforts?

Stupid Password Masking

2009-05-13T14:33:00.001-04:00

Log In/Out tests are commonplace throughout software testing. Pretty much every portal, client/server app, or web site I've worked with has required that I log in with a valid user/pass combo. Those credentials were always listed in the test plans I used. "Step1 - Log in as User1/Pass123 and click Enter."

Now, there's no problem with putting the user/pass combo in the test plan, and then tacking it up on the wall. But why do the managers, whose teams are proudly displaying test user credentials alongside photos of their kids, suddenly balk if an automated tool stores passwords in plain text format?

"Someone else could learn the password" they cry. "This is horrible and insecure!"

OK, let's think about this, for a moment. The tests will be placed in source control. Source control can restrict who can access the tests. Boom - problem solved. Let's look at it from another angle. Let's say that your automated tool masked the password that it entered, so that it just appeared as asterisks. How do you know what password is being entered? If you login is failing over and over, is it because the tool is entering a bum password? You'll never know.

Maybe I'm missing something here. But as a tester, I want to know what password my tests are using. Toggling a "hide password" feature on and off seems foolish too, as I can just toggle it to "show password" and bang - there it is again.

What do you think? Is there something really obvious that I'm missing here?

Clean Up Your Screen

2009-05-06T17:42:00.000-04:00

When I watch demos, I always find myself distracted from the presenter's message by something on their desktop. Maybe there's an icon in the system tray I haven't seen before (or maybe something down there is blinking). Maybe they have a shortcut to World of Warcraft on their desktop, or maybe their desktop is nothing but shortcuts, giving the feeling that their Start Menu just threw up all over the screen.

I've whittled my desktop down to the Recycle bin, and 3 shortcuts. One is the application I demo with, and the other 2 are apps I use during my demonstration. Everything that the audience sees has a point to being there.

As for the System Tray (or notification area, if you prefer), I recently discovered there's a way to force Windows to hide icons that are displayed there. Right click on the tray and select Properties, then click Customize on the Taskbar tab. You'll see a list of the items in your tray, and you can select if you want them to always be hidden, always shown, or only hidden if inactive. I set everything except my system's volume control and GoToMeeting to Always Hide.

It's a small thing, but it helps keep people focused on the message I'm sending, rather than wondering if I'm playing Horde or Alliance.

Machiavellian Automation

2009-05-04T08:57:00.000-04:00

I spoke with a colleague recently about a company whose automated test efforts had suddenly stopped. They had been using a third party resource for all their automated testing, and the money for that resource had dried up. As such, automated test development stopped, and no one in house had the knowledge or expertise to pick up the effort.

This reminded me of something I'd read in Machiavelli's The Prince. Machiavelli argued that princes should not rely on mercenaries or auxiliaries, and instead should rely on their own people. Some of that applies here as well. I think it's great if you want to jump start or augment your efforts by bringing in third party resources, but you need to be building up your own people as well. Sooner or later, that outside resource will be unavailable, either due to funding on your part, or lack of time on theirs. If you've built up your own team so that they can take the effort over, you'll have a smooth transition and no lost time or effort. If you haven't, your people will be scrambling to figure out how the automation works, and you'll waste time and money.

So make the investment in your people. Send them to training, buy them books, encourage them to learn. The end result will be in house expertise, which will make all your efforts smoother in the long run.

Only Bitmap Comparisons? Bad Tester! Bad!

2009-05-01T10:10:00.000-04:00

Many automation tools have the ability to take a screenshot of something in your AUT, and compare that to a baselined value. The intention is that you can use this to verify that images have rendered properly in your app; so things like your company logo, a product's photo, or the like can be validated easily. But I still come across people who want to use bitmap comparisons as their only means of validation in their tests.

Their reasoning seems sound enough. "If we just click through each screen of our app, and do a bitmap comparison of each full screen, then we can easily confirm the app is working. The bitmap comparison will automatically validate every value in every field, so we'll be able to create our tests even faster!"

It sounds good, doesn't it? But it doesn't work that way in practice. In fact, this has to be one of the most fragile, breakable, and ineffective approaches to automation. See, bitmap comparisons are going to do a pixel by pixel comparison of each screen. That means that if a single pixel is off, the test fails. In some cases, just running the same test on a different OS is enough to cause every bitmap comparison to fail, because a dialog in WinXP may have a screen where one pixel is slightly lighter than that same pixel in Vista. To your eyes, they don't look any different, so you won't know there's a problem until it's too late. Imagine that suddenly all your tests fail, and for no good reason. If this happens, you'll wind up in a maintenance nightmare as you struggle to track down that rogue pixel and somehow convince your automation tool that this is really ok. Automation is supposed to make testing easier and faster, not the other way around.

So please, take my advice. Use bitmap/region/image (or whatever your automated tool calls them) comparisons only to compare actual images. If you want to verify the contents of a field, read the actual value from that field. If you want to verify a button is enabled, check its actual Enabled property. This may require a bit more effort up front, but the payoff in the long run is well worth it. You'll have more efficient, more stable tests that will run clean, and it won't matter if that one rogue pixel off to the left is grey, a slightly darker shade of gray or a slightly lighter shade of gray than the ones around it.

Befuddled

2009-04-29T12:25:00.000-04:00

I've posted previously about being honest when you deal with prospective customers. When a customer describes a scenario to me that I know my company's product isn't a good fit for, I state that. The last thing I want is for people to think I've lead them on so we can make a sale. People won't want to do repeat business with us, and hurts both my reputation and the company's.

But I've encountered one particular scenario that I just can't wrap my head around. I was speaking with a test manager and his team. They outlined their scenario, and what they'd done with my product so far, and the issues they had encountered. I listened carefully, and based on what they wanted to achieve, I knew my app wasn't a good fit. I told them this, and stated my reasons why. Several of the team members voiced their agreement. And then the manager said something that flat out stunned me.

"I think we can make it work. We'll buy it."

I couldn't think of anything to say other than restate my previous concerns. The manager told me that his team would deal with it, and that they would purchase.

How would you handle this?

I Can Haz Demo?

2009-04-23T13:32:00.001-04:00

It's funny how social media has changed our perception of language. We communicate much more informally via the emoticons and acronyms that we use in IM programs, Twitter and in-game chat programs. However, when communicating with first time customers, clients and prospects, this abbreivated means of communication strikes me as unacceptable. You're trying to portray yourself as professional, not show how hip you are. Additionally, if you are a customer requesting help, a demonstration, or requesting license information, your sales rep probably won't take you very seriously if you send in a message like the one I received last summer:

"Can u plz help in it?bcoz it is not clear from Demo how to test Web application. Waiting 4 ur reply."

While this person may have been genuinely interested in my company's product, this communication came across as unprofessional. My sales team sent off a canned email response along with links to some recorded demos. If this person had listed their questions in plain English, we definitely would've put a lot more effort into helping them.

I don't intend to come across as snobbish in this post. I use IM-speak a lot myself, and I abbreviate words when I'm on Twitter. But when I'm trying to solve a customer's problem, I'm not out to show what a l33t hax0r I am. And in the same vein, when someone approaches me with a problem to solve, I definitely prefer it to come in clear and concise language.

Am I being too old fashioned here? What do you folks think?

Honesty Is the Best Policy

2009-04-23T09:38:00.000-04:00

My dad used to have a plaque that read "If you can't dazzle 'em with brilliance, baffle 'em with bullshit." I saw this employed quite frequently when I was evaluating test tools.

When I would setup demos from tool vendors, I was always up front about my needs and what I was looking for. Then I'd watch whatever they prepared and ask any questions that sprang to mind. Sometimes those questions would stump the presenter, and he or she would come up with some answer that was choc full of buzzwords that didn't mean anything. I never felt the need to call people on this - because in doing this, they answered a different question, which is how honest they are. I tend to avoid doing business with people who BS me, because I never know if I can trust what they're saying. It's hard to get work done when a trust issue is always nagging in the back of your head.

I have no problem with someone telling me "I don't know", so long as that is immediately followed by "But I'll find out and get back to you." I appreciate the honesty and I respect the person more as a result of it, and I think many people share that sentiment. I would much rather work with someone who didn't know an answer and found out for me, rather than someone who'd invented something that sounded good at the time, in order to turn a quick sale.

Now that I'm the one giving the demos, I always try to prepare well in advance so I can address any questions my client may have. And when they stump me, I don't follow the advice on dad's plaque.

Screenshots Need Accompanying Info

2009-04-20T12:12:00.000-04:00

It's said that a picture is worth a thousand words, and quite frequently, testers attach screenshots to their bug reports to illustrate a particular issue. While this can provide additional information that's helpful to a developer, I've noticed a lot of people lately who just log screenshots as bugs, with no steps to reproduce. For example, I recently received an email saying "The tool doesn't work - see attached" The attached bitmap showed the application with a blank screen; there were no error messages, no data, nothing that told me what didn't work, nothing that told me how it had gotten into this state. The back-and-forth email chain that ensued could've been avoided completely if the person had given me the steps they were performing and what they were trying to achieve.

To encourage people to provide more information in their bug reports, try this at your next team meeting. Say "My car wouldn't start this morning. Why?" Before anyone can answer, quickly add "Oh, and here's a screenshot of it" and have a photo of your car. Hopefully this drives the point home.

You Have Control

2009-04-10T13:33:00.000-04:00

How many times have you walked out of a meeting, grinding your teeth because someone said something that upset you? I've seen this scenario happen a lot over the course of my career, and it usually ends up with the upset person venting to a friend or co-worker who had nothing to do with the situation. Nothing ever changes between the upset-ee and the upset-er, because they don't communicate with each other.

I experienced this earlier this week. I left a meeting and was steaming mad at one particular participant. While this person had valuable things to say, they were rude and condescending, and their behavior really insulted me. I ranted to my family for a while, but it didn't help. So I sat down and wrote a very polite email to this person. I explained that I valued their input, and that I wanted to work with them, but that they had been condescending.

I didn't demand an apology, I didn't use any harsh language. I sincerely thanked them for their comments (which I really did appreciate), and stated my issue with their behavior. Once I clicked the Send button for that email, it felt like a huge weight lifted off me, and my mood improved dramatically.

So remember, you can't control other people, but you can control yourself. I tried to handle the situation with diplomacy, and while the jury's still out on what the outcome will be, I felt better because I actively did something. So next time someone gets you riled up during a meeting, step back and think about what you can do to take a measure of control in the situation.

Free Presentation Helper Tools

2009-04-09T08:17:00.001-04:00

When I'm giving presentations & demos, there are a handful of tools that make things go much smoother. The best part is that all these tools are free.

ZoomIt - this is a magnification and annotation tool. I use it to zoom in on areas of interest within my application, and to draw callout arrows on various bits of the app I'm demonstrating.

Keyboard Jedi - this app displays any keyboard shortcuts you've pressed. For example, if you press Ctrl+Space in Visual Studio to open a code completion window, KeyboardJedi displayes Ctrl+Space. This keeps you from needing to repeat a bunch of keyboard commands to your audience as you're going through your presentation.

CoolTimer - This is a simple timer application. I set it up as a count down timer to show when a presentation will begin. This is nice because my audience knows exactly when things will be starting.

QRes - This is a command line application that changes your screen resolution. I have a batch file set up to change my resolution from high - 1680 x 1050 to something easier to read (1280x1024). I just run this batch file before my presentation and I don't need to worry about monkeying around with display settings.

Going Virtual

2009-04-08T08:54:00.000-04:00

Virtual machines are a great way to try out new apps without fear of completely hosing your system. I frequently use them both when I do testing, and when I give demonstrations. I use both VMWare and VirtualPC, depending on the scenario I'm working on.

One thing I've found is that Microsoft has several prebuilt virtual machines that show off some of their applications. The Visual Studio 2010 CTP is available as a VM, and there's a complete working TFS system as well.

For demonstration purposes, these are perfect because they're already set up and configured properly. That means you can spend your time on developing a worthwhile presentation for your client, rather than monkeying around with setting a system up.

So before you build a VM, run a quick search on Google to see if there's a pre-built one out there. It can save you lots of time and effort.

Dead Air

2009-04-03T13:17:00.000-04:00

When you give lots of presentations, sooner or later something bad will happen while you're in the middle of a demo. Either your machine will crash, or the app you're demo-ing will freeze, or one of any number of things. In times like this, you don't want to be standing in front of your audience just looking goofy. That's why it's always good to have a handful of canned conversation pieces at the ready.

I typically will use the opportunity to gather more information from my audience about what they've done in the past for test automation, or what their biggest challenges are. (Normally you should learn these before you give a presentation, but sometimes that doesn't work out. If you have learned what their challenges are, this gives the audience a great chance to further elaborate on their situation.)

Alternatively, you can use the time to mention certain "side notes" that may be relevant to the audience. These include things like user conferences, training resources, or books & blogs that may help them.

Some folks like to fill this time with jokes, or anecdotes about their kids. Do this with caution. Gauge your audience and decide if that sort of filler is appropriate. I've found that the more I can keep the discussion focused on the audience and their needs the better.

What sort of things have you done to help combat dead air? Drop me a note in the comments

Keep Your Automation Visible

2009-03-20T08:10:00.000-04:00

You and your automation are only useful if people know what you're doing. It's always a good idea to make sure the tools and scripts you've created have as much visibility as any other testing activity going on. Otherwise, someday someone's going to say "Hey, what does [insertYourNameHere] do all day, anyway?"

Here are a few thoughts on how you can keep people from asking that question:

Demonstrate your tools. Show the test & dev teams what you're working on. Do it during a lunch & learn and let them see what you've done. People may suggest some ideas on how to improve what you've created. Others may realize they have a use for what you've created. Either way, you're keeping your team informed on what you've built and how it can benefit them.

Send out updates. Between demonstrations, jot off a quick note to your team when you've updated a tool or script and how it can benefit them. When they realize you've added the ability to test foreign languages, or you've made it easier to verify values have been written to the registry, they'll be excited and will put those enhancements to work right away.

Blow your own horn. I hate doing this, but sometimes it's necessary. Your management needs to know that what you're doing is worthwhile, so sometimes you have to send off a mail saying "Hey, that tool I wrote helped Carla find 10 bugs in an hour" or "The smoke test I wrote caught a bunch of problems before it went to the test team, which kept them from wasting their time on a broken build." Especially in these economic times, being invisible is not the way to go.

Solicit feedback. Communication is a two way street. Always ask your test team "How could this be better?" when they see you implemented requests they asked for, they'll be more vested in your tools, and more excited about using them.

What other methods have you used to increase automation's visibility?

Your Automated Tests Are Stupid

2009-03-16T09:49:00.000-04:00

Yes, they are. And not because you didn't put a lot of effort into planning them, nor are they stupid because you implemented them poorly. Your tests are stupid because the computer is stupid. It' s going to check exactly what you told it to, and only what you told it to. Keep that in mind; if you write a test that verifies a field in your app correctly displays the sum of two numbers, then your test will make sure that the sum of 2+2 = 4. That's great, but the test will miss a lot of things that a manual tester would catch. For example, if the results are suddenly displayed in bright pink; if they're in a font size too small to see; if they're displayed offscreen. Your automated tests will also miss any spelling mistakes in fields adjacent to the field being tested, which a manual tester would (hopefully) notice.

The point I'm trying to make here is that automated tests will not and should not replace your manual testers. Automation should be used to augment your manual testers; in the case above, you could have an automated test verify hundreds of thousands of various input combinations, and then have a manual tester run through once to do a spot check. That way you get the best of both worlds - you're assured that the app is functioning properly from both a technical and a human perspective.

Reflecting on a Demo

2009-03-11T09:50:00.001-04:00

Last Friday I gave a TestComplete webinar to 314 people. The topic was Creating Script Extensions, which lets people extend the standard capabilities of TestComplete. I think it went really well overall, and I wanted to share a few of the things I learned/found helpful.

Let the audience know where you are in your presentation. I had a really short slide deck; most of the slides were used to track where I was in my agenda. This served as a transition between topics, and helped people keep track of where we were in the overall presentation.

Presenter Mode is a wonderful thing. PowerPoint 2007 has this great feature called Presenter Mode. This lets you have your presentation shown on one monitor, and your slides along with speaker notes and a timer on another. I was able to refer to my notes for each slide without having to print them off. This helped keep the presentation smooth, since I didn't have to go back and forth between a printout and what was displayed on screen.

Always have a cheat sheet. A good chunk of the demonstration involved teaching people how to write the code needed for a script extension. I had a blank Notepad++ window on the monitor I was presenting on, and on my other monitor I had the complete code I was writing, along with comments to help explain the code. This was insurance against me having a complete brain cramp while I was coding.

Load your applications ahead of time. No one wants to watch splash screens load or watch someone stumble through explorer windows and shortcut clicks. Have everything loaded in advance, and the demo goes smoother.

Order your applications in the taskbar. I used several programs during the course of the demonstration; PowerPoint, TestComplete, Notepad++, Paint.NET, Windows Explorer, and a couple of Notepad windows for my cheatsheets. It helps keep things smooth when the items in the Taskbar are ordered according to the order in which they'll be used. That way, you aren't searching for anything. There's a great little utility called TaskBar Shuffle that lets you reorder items in the Taskbar, in case you don't open apps in the order in which they'll be needed.

Poll your audience often. I was giving this demo remotely, so I couldn't see my audience. That meant I had no idea if they were engaged or not. I made it a point to ask them questions as I was presenting. Most of these were simple yes or no questions, but it was enough to keep them involved.

Have a wingman (or wingwoman). I had a co-worker riding shotgun with me during the demo. It was his job to answer questions about the telephone number needed to call in, and to alert me to any audio or visual problems that people were experiencing. This freed me up from having to respond to certain questions during the demonstration, and provided a mechanism to head trouble off before it became serious.

Breathe. I was really excited about giving this session. My energy level was pretty high, and I felt a little out of control in the first few minutes. I made myself slow down and take a breath, and was able to stay on a good pace for the session without rushing through things.

Record yourself. My wingman also recorded the demonstration. We wanted to be able to send this demo to clients who couldn't make it to the live session, but it's also a great tool for self-improvement. For example, I thought I had a really good handle on not saying "um" while presenting, but in listening to the recording I hear myself saying it, especially when I'm answering questions toward the end of the session. I'm going to keep working on that.

If you'd like to take a look at the recording, you can see it here (it's about 90 mins long). I'd welcome any feedback you can give.

You Have to Start Somewhere

2009-03-06T08:43:00.000-05:00

I was speaking with a client recently who asked a lot of great questions. He was the manager of a small QA team, and really had a good feel for automation. I addressed everything he asked about, and then he told me that he had one other member of his team who had just a few "quick questions."

After speaking with this person for a few minutes, it was blatantly obvious that not only did he have no clue about test automation, he didn't have a clue about testing in general. To top it off, his initial attitude toward me was rather rude. My first reaction was one of mixed shock and frustration, but then I did what William Ury suggests in his negotiation book "Getting Past No" - I went to the balcony.

Basically this concept means you distance yourself from your emotions and your natural reactions - in essence, you step back from the situation and look at it with different eyes. I tried to put myself in this person's shoes. He was being asked to do something that he wasn't qualified for, and he was probably frustrated about that. Given today's economy, he was probably also scared that if he couldn't grasp these concepts, he might be out of a job.

So I stepped back and listened to his concerns and what he was trying to achieve. Then I addressed each concern in turn, and gave him lots of opportunities to ask questions or get clarification. The end result was a successful session. He admitted he had a lot of learning to do, but he felt really good about his chances to succeed.

My takeaway here is that everyone has to start somewhere. After doing test automation for so long, it's easy to forget that there was once a time when I didn't know a thing about it. I've posted a note above my desk that says "You were an English Lit Major" to remind myself that I asked all those really basic questions once too, and that perspective is part of the key to making other people successful as they learn about testing and test automation.

The Automation Mantra

2009-03-04T07:44:00.000-05:00

I love tools. I love writing them, I love learning how to make the computer do more work for me. But it's easy to get caught up in the excitement of development and lose sight of what automation's really about. That's why I've adapted a mission statement that an old boss of mine had. Whenever I find myself writing a new app, or looking for a place to apply something I've created, I take a step back and say to myself -

"My job is to provide the QA team with helpful, easy to use tools. The output of these tools should be simple to understand and share."

If what I'm doing doesn't meet the above criteria, then I know I'm doing something wrong. It's a simple sanity check, but it keeps me from building things that aren't worthwhile. There are a lot of people out there who have been soured on automation because their experiences were with tools that did not follow these guidelines.

So when you're building tools for others, make sure they are:
Helpful - don't build a solution in search of a problem. Make sure you're really filling a need.
Easy to Use - don't make it harder to use the tool than to run the test without it.
Easy to Understand Output - put human readable English in your output messages, and store those messages in a file format that's easy to read. I've found XML/HTML are usually the best bets; PDF isn't bad either.

What other guidelines do you follow with your automation mantra?

Be an Educator, not an Enabler

2009-03-03T08:43:00.000-05:00

This weekend I set up a computer at a relative's house. I unboxed it, plugged it in, went through the Windows setup screen. I set the timezone, removed some of the preinstalled crap-ware and finally installed MS Office. The entire process only took about 35 mins, but when I went home that night, I was disappointed in myself; I had missed out on an opportunity to teach my family how to do something with a computer.

Let's be real here folks, how hard is it to set up a new PC? This one had 3 wires. The CPU power cord, the monitor power cord, and the cable that connected the monitor to the PC. The keyboard and mouse were wireless, and just needed batteries installed. Loading Office is just a matter of inserting a DVD and clicking "Next" a few times. I could have walked them through the process and then they'd have felt smarter and more confident with the technology. Instead, I've inadvertantly fostered the belief that working on computers is hard, and borders on black magic.

It's like that with test automation, too. People who don't have an understanding of what it does and how it works see it as some sort of dark art. It's up to us to educate them so that they learn the right way to do things, rather than us always doing it for them. Good teachers help people figure out the answers for themselves, rather than give the answers to them. I'm going to try and be more like that in the future.

What have you done to help educate your team/group on automation? Have you been an educator or an enabler?

Obfuscation Question

2009-02-26T12:50:00.000-05:00

Many companies employ obfuscation to keep their code from being reverse engineered. But this leads to an interesting problem, namely, a GUI that's been obfuscated can't be automated. The only workaround I've been able to come up with for GUI automators is to write & run tests against non-obfuscated code. The problem arises when a manager says "But then we aren't testing the same thing that we ship to our customers"

This got me thinking about how obfuscation impacts other kinds of testing. I would think that obfuscation would impact any tests that were run after the obfuscation process. So is it safe to say that companies who employ unit tests must be running those tests prior to the obfuscator running? If so, should the manager be concerned that the unit tests were run against non-obfuscated code?

Publish Post

Obfuscation would also prevent any other type of post-build white box testing; so any tools I've written that call methods directly from dlls or invoke web service calls would be stopped dead.

So what's the best solution here? Is it to do the bulk of your automated testing against non-obfuscated code and then do manual sanity checks against an obfuscated build? Or is there a better way that I'm not seeing? Please let me know in the comments.

Pointers on Pointing

2009-02-25T09:50:00.000-05:00

Gestures are a powerful part of any presentation. However, you have to be careful when you point at objects or people to ensure your body language doesn't send the wrong message. Here are a few things I've learned when pointing during presentations:

Don't use a laser. I love giving presentations; I get really excited when I'm in front of people. But that excited energy tends to make my hands shake a little. It's enough that I can't quite focus a laser dot on a single point without it bouncing a bit. If you're really nervous about being in front of people, the dot may dance all over the screen. That's distracting to your audience, and it takes away from the message you're trying to deliver. (Plus we've all seen those comics where the guy with the laser pointer accidentally beheads someone in his audience...)

Point at slides, point at objects, but don't point at people. We're all taught in school that it's not polite to point. Whether you intend it or not, pointing at people implies some measure of accusation. You want to invite your audience in to your message, not make them feel like they're on trial. So instead of pointing at someone, use an open hand with palm up and fingers out. This is a welcoming and encouraging gesture that brings people in rather than putting their backs up.

Use the mouse sparingly. If your mouse is bouncing around your screen throughout the presentation, people are going to get dizzy. Peter Cohen calls this "Pointer Palsy", and it's extremely distracting. Perform your actions slowly, so the mouse isn't racing around the screen. If you are giving a remote demo, you can gesture with the mouse, but use these gestures sparingly. If you don't need to gesture at something, take your hand off the mouse. You'd be surprised how many of us move the mouse around without realizing it or intending to.

Are there other "pointers" you've learned? If so, please share them in the comments.

Notes on Demos

2009-02-23T13:00:00.000-05:00

I give lots of demos every day, and I'm always on the lookout for things to make them better/smoother/cleaner. So the next time you have to give a technical presentation, here are some (hopefully) helpful tips:

Turn off all unnecessary applications. You don't want that IM from your significant other coming in during the middle of a discussion. Turn off your email, chat, twitter and any other programs that aren't relevant to your demonstration.

Remove any potentially offensive/confidential material from your recent items. The pics your brother-in-law sent of his latest Vegas exploit may be cool/funny, but they're not professional. On a related note, be professional when naming your computer. During a presentation I sat in on last week, the presenter's laptop was named WideAss. And since the name of his machine was prominently displayed in several places of the app being demonstrated, it's not like you could miss it. That's just not professional, and I couldn't ignore it.

Make it readable. If you're using slides, keep the font big. If you have to shrink the font to get everything to fit, edit, delete, or split it into multiple slides. If you're demo-ing code, keep that big, too. Scott Hanselman suggests using Lucinda Console 14 -18 pt font for all code demonstrations, and I've found that works very well. You lose your rythym as a presenter very quickly when you have to keep asking if everyone can read what's on your slides.

Finally, and most importantly practice! Spend some time saying exactly what you're going to say. Don't go halfway on this one; sitting in front of your slides saying "ok, now I'll talk about X" will not make an effective presentation. Say exactly what you're going to say, exactly how you want to say it. If that means you need to retreat to a conference room or hide in your car at lunch break, then do it. The more familiar you are with what you're going to say, the more confident you'll come across as.

I'll post more demo tips as I can, but what else have you done to make your presentations more effective?

Portable Tools

2009-02-20T10:38:00.000-05:00

There are times when your test machines may not have all the tools on them that you need to perform a test effectively. Sometimes you're not allowed to install other apps on a system due to a security concern. Luckily, many applications can be run from a flash drive. This lets you get your job done while not contaminating the system. Here's a quick list of some of my most frequently used portable apps. You can download a full suite of programs from PortableApps.com, and many more from PortableFreeware.com

Notepad++ - lightweight editor for pretty much any type of script.
Recuva - lets you quickly recover deleted files.
Wireshark - packet monitoring tool useful when load/performance testing.
PortablePython - a full python development environment on your flash drive.
FoxitPDF Reader - because sometimes, Acrobat just isn't there.
QueryExpress.exe - simple SQL query analyzer look alike.
WinMerge Portable - lets you quickly perform a diff between two files.

What other portable tools do you use?

Tips on Hiring an Automation Engineer

2009-02-18T08:29:00.000-05:00

Many QA managers don't know what to look for when hiring an automation engineer. They'll specify some generic criteria (knowledge of C#, knowledge of [insert major commercial tool name here]) and stop at that. Then, when an applicant comes in, the manager asks if they meet those criteria, and the technical side of things stops there. So here are a few tips I'd like to pass along when interviewing a candidate for an automation job.

Be wary of "automation engineers" who only know how to use commercial UI automation tools. Those tools are great, and they have their place, but they are not the end-all be-all of test automation. You also want to make sure they can write their own code outside of the automation tool. For example, many commercial vendors use VBScript as a programming language. If your candidate claims they can write VBScript, ask them to describe how they would perform a scenario outside of the UI tool in that scripting language.

If you already own a UI automation tool, don't hire someone who is an "expert" on a different tool. You're probably thinking that the candidate should be able to pick the other tool up quickly enough, and that may be true. But on a number of occasions I've seen people gripe and complain about how their favorite tool does A, B and C better, and this product is garbage. Some will even try to force a change in the department, to displace the existing tool with their preferred tool. This is disruptive and nothing but a time sink. The time spent arguing about tools is time that should be spent testing. If you've made an investment in a UI automation tool, hire people who are comfortable with that tool.

Describe your development process, and the app being developed. Then ask them where they see opportunities for automation. If they focus exclusively on the UI, then you have someone who won't be as effective as someone who talks about unit tests, who talks about testing an API, or testing the backend database. The key here is that a good automation engineer will look for places throughout the software development lifecycle where automation can be applied.

Ask if they've ever built their own tools, or if they've done any programming for fun. Someone who really enjoys coding will be much more effective than someone who just does it as a day job. Plus, someone who can build their own tools will be able to continue to automate in areas where commerical tool vendors don't. For example if you need to test against a beta operating system, db server or UI element, commerical tool vendors won't support you. Someone who can build their own tools will still be able to create automated tests.

Finally, and this is the most important, do they know how to test? Automation folks are testers too, and you don't want to hire a script monkey. You want to make sure that, as Karen N Johnson put it "...you feel confident that [they] can be handed an application, learn at least its primary aspects at a fairly rapid pace, jump in, join this project and find bugs." You want to make sure that these people can come up with useful scenarios for automation that will add value to your team; you don't want someone who's only able to take a manual test plan and turn it into an automated UI test.

What tips do you have for hiring an automation engineer?

Good Coding Practices

2009-02-16T08:00:00.000-05:00

Ben Simo (http://www.questioningsoftware.com/) mentioned via Twitter that he wanted to do a presentation on good automation coding practices. That got me thinking about the things I do to keep my code maintainable, readable and ultimately, useful as projects progress. Here's a quick list of some of my suggested coding practices:

Give functions/methods and variables descriptive names. When you're writing your code, don't name functions arbitrarily, like Test1. Instead, give them descriptive names so it's immediately obvious what the code is going to do. For example, "verifyAdminUserCanDeleteReport" may be a bit wordy, but you know exactly what it's going to do. Similarly, give your variables descriptive names as well. I like to prefix my variables with a shorthand representation of their type; for example, str for string, int for integer. So if I created a variable for a user name, I'd name it strUserName. This helps eliminate confusion, especially when you're dealing with a big test.

Comment everything. The code may seem completely clear to you as you write it, and so commenting may seem like a waste of time. But chances are when you leave it for 6 months to do something else, when you get back to it, you won’t remember what you were doing. Or, when someone else inherits your script, they won’t know what’s going on. Comments provide a nice, easy way to prevent that type of confusion. I like to include information like this with each function I create
/*
Name:

Description:

Syntax:

Parameters:

Return Values:

Example:

*/

You can also use utilities like TwinText to generate help files based on your comments. This provides a nice reference that people can browse for functions that may be useful in their own testing.

Don't chain tests. I see this a lot in UI automation, where "Test B" is dependent on "Test A"’s results. For example, let’s say you are testing a telephone directory application. Test Case A scrolls to the “Howland, Jessica” user record in the phone book and verifies its data. Test Case B clicks the Next button 3 times to get to the “Jameson, Rita” user record, and then verifies its data. If Test Case A fails to reach “Howland, Jessica”, there’s a chance that Test Case B will fail because the number of clicks needed to get to “Jameson, Rita” are dependant upon being at the “Howland, Jessica” record. This is a chain, and could lead to problems with your script. A better approach would be to start at a known point in the database (such as the first record), and use a parameterized loop to click Next until “Jameson, Rita” is displayed in the Name field.

Use source control. This lets you keep a versioned history of your tests, and lets you see who's edited what parts of the tests.

Keep your own backups. The source control server may be backed up, the network drive where you keep your working copies may be backed up, but do yourself a favor and keep your own backups of your tests. External hard drives are cheap, and writing a short script to copy everything from your working directory on to the external drive will only take a minute or two. This can save you a lot of headaches and frustration when servers crash; when your IT guys realize the backup drives are full and haven't succesfuly backed up in a month; or when you get told that your restore task is at the bottom of the todo list.

What good practices would you recommend?

Tools Do Not Convey Knowledge

2009-02-12T08:29:00.001-05:00

Let's say you've decided that you want to change the oil in your car. You don't want to take it to a shop, you want to do it yourself. So you go into Sears, find the Craftsman section, and purchase a mechanic's wrench set, an oil pan, and some jacks. You're all set, right?

One minor issue - you don't know *how* to change the oil; you've never done it before. But these tools were expensive, and they're supposed to make the job easy, so there shouldn't be a problem, right? So now, you get back to your house, crawl under the car and start messing around. After a few hours of frustrated cursing, you emerge from under the car covered in oil, your knuckles are torn up and you've damaged your car so badly that it'll need to be towed. You are all set to call Sears and light into them about what crappy tools they make because you couldn't even do a simple task like change your oil.

While extreme, and perhaps a little ridiculous, this scenario is exactly what happens when a company blindly introduces an automation tool without seeing if its people have the skills needed to use it properly. The statement "Automating tests is easy, you just click a record button" could be equated to "Changing the oil is easy, you just drain the oil and put in fresh stuff."

Make sure that you have a good understanding of what's going to be involved in your automation efforts before you put a tool into play. Do a small pilot project or proof of concept against a piece of your application and see if there are any surprises. That whizzy custom control your developer whipped up may only be accessible to someone who knows C#. That database test you want to perform may only be possible by someone who can write SQL stored procedures. Make sure that your people have those skills before you ask them to dive right in to automation.

Remember, tools enable people to work more effectively, but people need to understand how to use those tools before that can happen.

Testers don't break code

2009-02-11T09:31:00.000-05:00

It was about 9 am when Carl, our lead developer introduced me to Jesse, a recent college grad and our newest developer. Carl introduced me as "one of the QA team", to which Jesse said "Oh, you're one of the guys who breaks my code." I smiled at him and replied "Actually, the code's already broken by the time I get it." He paused for a moment, then laughed.

This perception of testers has always bothered me. I don't break anyone's code - I try to think of scenarios that the developer may have missed and help him make a better product. If I wanted to be a malicious little gremlin that "broke" code, I'd go into the source files and remove all the semicolons, or replace every instance of the letter e with a q. *That* would be breaking the code.

Testers are there to help developers build a better product. I think the issue arises when people take the issues testers log as personal negative feedback. One of the ways I was able to alleviate that was to do some pairs testing with a developer. We sat down, walked through the AUT, and I talked him through my thought process as I tested. At first he just watched, then started asking to try things as we went along. In short order, he had found issues that I wouldn't have thought to check for. Once he saw that I wasn't being malicious, his whole attitude toward the test team changed.

Have you encountered the "Testers break code" mentality? How did you resolve it?

UI Automation Safety Tip

2009-02-09T09:19:00.000-05:00

Here's a safety tip for everyone who works with UI automation record & playback tools. Turn off your email & IM programs before you start recording. Close down your Twitter, GMail and Yahoo-based items as well. This has nothing to do with system resources or anything like that. Rather, it's because there are a surprising number of people out there who will be recording a test, then respond to an IM or decide to jot off an email.

And when they do that, the recorder captures it and puts it into the script, where anyone else who has access to it can see. I had a co-worker once who used to IM his girlfriend during work hours. We sat down one afternoon to review a script he'd been working on, and right in the middle of it was his side of one of their conversations. There was nothing racy or controversial in there, but he was very embarrassed to say the least. It didn't occur to him that the tool was recording *every* action he performed.

So please, for your own sake, turn off those social media utils while you're doing record & playback. (Or at least remember to click the Stop Recording button before you use them.)

Incremental Automation

2009-02-04T08:19:00.000-05:00

When you're automating, it may seem logical to automate an entire test plan before you begin running it. For example, if you're automating a smoke test, you may want to have all pieces of the smoke test scripted so that you can run the test at the touch of a button. That's a good goal to strive for, but that doesn't mean you can't automate a piece of that smoke test and then start running it right away.

Back when I was first learning to automate, my smoke test was a 26 page test plan that touched all the major features of our product. When I sat down to automate it, I started on page 1 and went step by step. I was doing great until I hit page 11, when a control in my AUT wasn't recognized by my test tool. I spent a couple of days tracking down the issue and coming up with a workaround. A similar problem happened later in the test. All in all, it took about 3 days to automate the test, and another 3 days to troubleshoot issues, so it was over a week before the script ever ran. This was my first experience with automation, so I thought this was probably normal.

Looking back, I can see that by not running the parts of the test that were finished, I lost a great opportunity. I could have had the first 11 pages of that test running while I troubleshot the issues. Then people would have seen that the automation was doing something, and I could've shown my manager that while I was encountering some issues, our first crack at automation was off to a good start. I've heard stories where people tried to automate an entire test, and due to one problem or another, didn't actually run the test until two or three days before their product was set to release. That sort of approach doesn't benefit anyone, and winds up costing time and effort instead of saving it. It also reduces automation's credibility in management's eyes, which is part of the reason (I think, anyway) of why so many automation efforts get abandoned.

So the moral of the story is don't do "big bang automation", where you wait until everything is finished and perfect before you start running your automated tests. Get a piece of your test automated and then start running it. Then continue adding pieces until your entire test or suite of tests are finished. Your team will see results faster from automation, and your management will be more confident that the automation is helping the developing efforts.

You Can't Automate User Acceptance Testing

2009-02-03T12:25:00.000-05:00

One of the things I hear from clients is "we want our business analysts to be able to automate our user acceptance tests." I cringe every time that phrase gets mentioned. It doesn't work for two reasons.

The first is usually that the analyst does not have the skill set needed to effectively automate a test, and management is usually reluctant to make that kind of investment in them. The second reason is that you can't automate a user acceptance test. You can automate the functionality that makes up that test, make sure the correct dialogs and error messages are displayed at the appropriate time, but the "user acceptance" part only applies if a person with domain knowledge has tried the test and agreed "Yes, this is how the program should work."

I'm afraid that people then get lulled into a false sense of security. "Our automated acceptance tests are passing, so our product meets our users' needs."

What do you think? What have your experiences been with automating user acceptance testing?

Automation Mashers

2009-02-02T14:08:00.000-05:00

Test automation is handy for regression testing, but automated regression tests rarely find new bugs. So instead of focusing exclusively on regression tests, try coming up with scenarios to detect bugs that would be difficult (or impossible) to find manually. For example, let's say my AUT can be used to generate reports. I enter some criteria, click "Generate" and the AUT fetches data from a database and dynamically creates a report based on my selections. Let's also say that this fetch and display operation causes a momentary spike in my CPU usage and memory usage before settling back to normal.

Now, we could create a automated regression test that generated a report and verified it generated the correct data. That may be useful, but we can make it much more useful by turning it into a "masher" test. Mashers are tests that perform a single operation over and over again, while tracking how the system performs over time. In doing this, we can find bugs (like memory leaks) that we wouldn't be able to find by manual testing alone.

So in my scenario above, I noticed that there's a momentary spike in CPU and memory use when the report is generated. Being suspicious, I wonder if there's the possibility for a memory leak somewhere in that code. So I'd create an automated test that ran my report and confirmed it was correct, and then loop it so that reports were being generated for 24 hours straight. As the test ran, it would log the system resources being used after each generation. At the end of the test, I'd graph the data that had been logged. If the system resource usage remains constant, then there's no problem. But if there's a steady increase in resource usage, then we have a memory leak, and we've found a bug that a regular regression test (manual or automated) would have missed.

So in addition to making manual testing easier, and taking care of regression tests, don't forget to look for places where automation can help you find bugs that would be impossible to find otherwise.

Augment Manual Tests with Automation

2009-01-30T14:12:00.000-05:00

When people think of test automation, the knee-jerk reaction is to think of some tool that performs a test or test-related activity for you. That tool runs unattended and gives you results of some kind when it's finished. That's definitely part of automation, but by no means the only thing it can do. You can also use it to create utilities that testers can use to speed up the testing process.

Here's a scenario for you - a tester needs to confirm that a text box can accept strings up to a given length. If there are too many characters in the string, the AUT throws an error message. We've probably all seen and done this type of boundary test many times. And you probably created your input strings the same way I did - you opened up Word, held down the X key for a bit, then used the word count feature to see how long of a string you'd made. If it was too long, you deleted some of it. Too short, you added characters until you hit the right length. Then you copy/paste that string into your box, see if the right error was displayed and moved on to the next test.

That's a tedious way to perform a simple boundary test. I got frustrated with doing that and built a small helper utility whose sole purpose was to generate strings. It was just a small dialog that asked how many characters I wanted my string to be, and after I entered that, it generated a string of that length and automatically placed it on the clipboard. I pasted the string into the desired field, confirmed that my error was displayed, and moved on. I didn't have to mess around with Word anymore, and I could create those needed strings much faster and more accurately than before.

I shared that helper utility with other members of my team and it was very well received. Several of the developers on the team asked for it too, so they could do some preliminary tests before the code got passed over to the testers. Over time, the String Generator evolved so that people could just click a button to "quick-create" strings of a predefined length.

The string generator only took me a few minutes to write, and it turned out to be a real time saver. Always keep your eyes open for places where a little bit of code can save a lot of manual effort.

If You Want to Automate Tests, Learn to Script

2009-01-28T18:05:00.000-05:00

Many people who get tasked with automating test cases have never written code before. As such, they rely on the promises from UI tool vendors that they can automate all their tests without ever touching a line of code. While UI automation tools may be able to create simple tests, those tests tend to be brittle and prone to breakage. As such, a non-scripting user can quickly become frustrated and abandon test automation all together. Compare that to a person with some scripting experience - they can very easily modify a recorded script and turn it from something brittle into something nearly bulletproof.

When I started out in automation, my background was in technical writing. I had no programming experience or training, but I saw how powerful automation could be, and was determined to see it work. I spent a lot of time learning TSL, which was WinRunner's proprietary language, and found that I could make tests really sing by coding them myself. (My colleagues who were relying on record & playback were astounded that my scripts could recover from errors. )

Eventually, I realized that there were other automatable tasks that fell outside the scope of WinRunner. I learned Python and with that I was able to compare the contents of excel spreadsheets, verify that email messages had been posted to a server and then delete them, automatically reboot systems and put them into the proper state for testing. Those wound up being even bigger timesavers.

So the moral of the story is - if you're going to get into automation, learn how to script. Don't believe anyone who tells you that all your test automation goals can be achieved without ever touching a single line of code. By learning how to script, you become more marketable, and your company's automation efforts have a better chance of success.

There are a lot of great scripting languages out there - Python, Ruby, JavaScript. If there are developers in your company who already know one of them, ask them to recommend a book or website that will help you learn. If you're working with .NET, I highly recommend A Tester's Guide to .NET Programing
and Effective Software Test Automation

Find the Pain

2009-01-26T09:11:00.000-05:00

One of the biggest mistakes I see made with test automation is that people don't step back and determine where their manual testers are spending lots of their time. Instead, they just purchase a UI automation tool and then begin automating tests. In some cases, they may get lucky and automate a test that is a time saver. Other times, they may just take a test can be run manually in 5 minutes, and turn it into a 3 minute automated test. While there is a time savings there, it's going to take a lot of runs in order for that 2 minute savings to really add up.

UI automation tools have their place, but before you consider doing any type of automation, take a good look at where your manual testers are spending their time. Do they spend time populating databases with test data? Do they spend time reading through log files for error messages? Are they spending time manually comparing file names and version versions numbers when doing an install test? These are good candidates for automation. SQL scripts can be created to populate databases, perl or python (or whatever your language of choice is) can be used to parse log files for certain strings, and can be used to query file info as well.

In short, significant time savings can be achieved if you incorporate automation all parts of your test activities, not just the running of the tests themselves.

On Evaluating a UI Automation Tool

2009-01-23T09:04:00.000-05:00

At some point, manual software testers look at their ever-growing workload and say "Ok, this is getting out of control. There has to be a better way to do this." And the answer comes back from management, "Find an automated tool."

There are a bunch of automated tools out on the market. So many in fact, that it can be overwhelming to find the right one for you. Each tool has its own set of unique features that differentiate it from the others, and your first instinct may be to put together some sort of matrix that compares each tool. That's a fine idea, but hold off on doing it for the moment. Also resist the temptation to just start downloading all the tools blindly. Instead, take a step back and look at each tool with these criteria:

Does this tool support the language my app is written in, and the environments I require?
This may seem obvious, but there are people who never actually checked to see if their programming language was supported. Instead they compared all the "bells & whistles"-type features, and wound up spending thousands of dollars on products that didn't support their needs. One other thing to consider here is if your app makes use of 3rd party controls. To a manual tester, those controls don't seem any different than standard controls, but to an automation tool, they're very different. Ask your development team if there are 3rd party controls in your app, and then find out from the vendor if those are supported. You'll save yourself a lot of headaches and avoid surprises by doing this.

Is this tool in my budget?
I hate to make cost one of the first things to consider, but the unfortunate reality is that the QA department's budget is usually more restricted than other groups. The last thing you need is to spend a lot of time evaluating a tool only to find out you can't afford it.

Those two questions can usually be answered in a few minutes with a quick phone call to the vendor, or with a visit to the vendor's website. Once you've established that a tool or tools will work for you, download an evaluation copy of the product. Again, resist the urge to start a feature comparison matrix at this point. Instead, take two or three of your test cases and try to automate them with the evaluation software. It's a good idea to pick test cases from different areas of your application, to ensure that the tool will work properly for your needs. You don't want to find out after you purchased that while ToolA worked flawlessly against TestCase1, it failed to work at all against TestCases2 - 37.

Once you've done your initial automation, see if there are places where you have questions about how the tool worked. Jot those down and email them to your vendor. To keep the process moving quickly, be as specific as you can when describing your questions. Explain what you're trying to achieve to the vendor, what your application is written in, and then list your questions. Treat these questions the same way you would bug reports; the more info you can provide, the faster the vendor can get answers to you.

When the initial automation is done, and you're waiting for answers to your questions, now you can create that comprehensive feature matrix comparison.

The important thing is to make sure that the tool works well with your application first, and then worry about the niceties. I like to compare this to buying a car; sure, it's great if the car has an all leather interior, heated seats and an aux jack for your mp3 player, but if it doesn't get you from point A to point B reliably, those other features are moot.

Welcome

2009-01-23T08:58:00.000-05:00

Welcome to the Autonomicon. In this blog, I'll post notes about automated software testing, links to helpful utilities that can speed up your day to day activities, and offer my experiences with the QA process in general.