Just Get On the Bike

There's an old story that goes something like this:

Little Timmy is very late to school one morning. It's 10:30, and the teacher looks out the classroom window and sees Timmy just getting into the schoolyard. He's pushing his bicycle and looks very hot and out of breath. When he gets into the classroom, the teacher says:
"Timmy, why are you so late to school?"
Timmy replies, "Sorry teacher, I had to push my bicycle all the way here."
The teacher, confused, says, "Why didn't you ride your bicycle?"
Timmy replies, "Teacher, I was running so late this morning that I didn't have time to get on the bike."

I hear this kind argument all the time when people put off automating. "I have too much to do," they say. "When this release settles down, I'll have time then."

Bullshit.

Folks, I hate to say it like this, but you're never going to have that perfect moment where you do nothing but automate. If you're a tester, then there's always something that will need testing. You have to find ways to work automation development in to your regular day. If automation is something that you throw to the back burner every time your workload heats up, you'll never get any benefits from automation.

Sometimes this means explaining to your manager that automation can't be a side project and should be focused on sooner rather than later. Sometimes it means recruiting a developer to help get you jumpstarted, so you can get a test framework built up quicker. Sometimes it means hiring another person whose primary responsibility will be test automation.

Point is, if you're always waiting for that perfect moment, you'll never get anything automated. So do *something*. Even if it's just a small test to start with, that's something you can build on. You won't get a comprehensive set of automated tests overnight; so get one test automated/one helpful tool written and then add to that. Even if you only automate one new test a week, over the course of a month, that's 4 tests you don't have to run by hand. Or, if you're trying to build a tool that will augment manual testing, get that to your testers as soon as it can do something useful, then continue adding features. The time savings will get more and more significant as you go along.

So, long story short, just get on the bike!

TUU: The Missing Metric

Before taking the plunge into test automation, many managers want to know how long until they see an ROI (return on investment). In the case of a commercial tool, some vendors will actually send out an Excel spreadsheet with formulas that let you plug in the amount you pay a tester per hour, and then the spreadsheet calculates how much it costs to find that bug during the test phase vs how much it costs to find it once the app has gone to the customer.

While I certainly understand the need to justify an expense, I think ROI by itself is the wrong way to look at test automation, because it puts the focus on dollars rather than on utility. If you buy a $9,000 automation tool, and pay your tester $40/hour, then using the ROI approach, your test automation tool needs to run 225 hours' worth of tests before your see an ROI. If you've bought multiple copies of that tool, then you need to multiply both the total cost and the total number of hours by the number of copies you purchased. So if you bought 5 copies of the test tool, that's $45,000 you spent, and you need to run 1,125 hours' worth of tests to break even. "That's too much time or money", someone says. "Look at a cheaper tool."

What I don't see people consider is how long it will take before the automation is doing something productive. So let's say that you're considering spending money on a commercial tool. Instead of saying "How long until I see an ROI", ask "How long until this lets me do something else?" Automation is implemented so that you don't have to do the same tasks over and over; you automate to get the computer doing regression tests so you can focus on other test activities. How long will it take you to create that first smoke test? How long until you are freed up to do the new tests? Let's say it takes you 2 work days to get your smoke test up and running. That means that two days after implementing automation, you start getting some of your time back. That's how long until the automation is doing something useful. I call this Time Until Useful (TUU).

I think that 16 hours until your automation is doing something useful is a lot more pallatable than 1,125 hours until your automation justifies its existence. Plus, now you're working and finding bugs that you didn't have time to find before. I've never seen that time savings or "new-bugs-found-during-exploratory-testing-while-the-automation-runs-a-smoketest" line item included on an ROI sheet.

If you're considering open source tools, TUU is much more a natural metric. Freed from cost concerns, a test team is given the chance to look at a bunch of tools and pick the one that's going to let them get the most done the fastest. I've seen teams where managers say 'oh, you're going open source?' and then promptly go deaf because they think that since there's no cost involved, they don't need to justify any expenses, therefore they don't care. Thing is, open source tools will have a bit of a learning curve as well, and you want to make sure that your team selects one that meets their needs and still has a good TUU. If the tool is free, but it takes 2 months before it's doing anything of value, is that worthwhile? No, you'd want them to look at a different open source tool that's going to do something sooner than that.

TUU is a metric that should be considered with the same weight (or maybe with greater weight) as ROI. Whether you're going commercial or open source, you should always consider how long it will be until the automation is doing something useful.

String Generator

If you've ever had to do boundary testing, you probably did what my co-workers and I used to do. You need a 400 character string, so you open MS Word, hold down the 'x' key for a few seconds, and then use the word count feature to see how many characters you've got. If you've got too many, you delete some. If you don't have enough, you mash 'x' for a few more seconds. Later, rinse and repeat until you've got it right.

Some folks would even remember to save those word files for future use, but more often than not we closed them without thinking about it. (This was usually followed by the comment "Ah, crap, I shoulda saved that...")

To help take some of the sting out of this, I wrote a string generator application. It lets users generate strings of predefined lengths, and then automatically puts those strings on the clipboard so that they can be pasted into fields.

I wanted an easy way to know that a field could accept that number of characters, so I always had the string end with a capital Z. For example, if you needed a 10 character string, it would generate aaaaaaaaaZ. Then you'd paste that into the desired field, and if you didn't see the 'Z', you knew the field was truncating the data that was entered.

The app I wrote looked like this:












Each of the buttons you see on the left would automatically generate a string of the specified length. If you needed a string of a different length, you could enter that length in the box next to the custom button.

The string generation itself was handled by this code:

private string prgGenerateString(int intLen)
 {
  StringBuilder MyStringBuilder = new StringBuilder();
  MyStringBuilder.Capacity = intLen;
  for (int i = 1;i<intlen;i++)
  {
      MyStringBuilder.Append("a");
  }
  MyStringBuilder.Append("Z");
  return MyStringBuilder.ToString();
 }

private void prgGenerateStatusMessage(int intLen)
 {
  lblStatus.Text = intLen.ToString() + " character string placed on clipboard.";
 }

private void prgCreateStringAndPost(int intLen)
 {
  try
  {
   string txtValue = prgGenerateString(intLen);
   txtPreview.Text = txtValue;
   prgGenerateStatusMessage(intLen);
   Clipboard.SetDataObject(txtValue);
  }
  catch(Exception except)
  {
   lblStatus.Text = except.Message;
  }   
 }

Each button on the form had code like this:

  private void cmd129_Click(object sender, System.EventArgs e)
  {
      prgCreateStringAndPost(129);
  }     

And here's the code for the custom button:

  private void cmdCustom_Click(object sender, System.EventArgs e)
  {
      if (txtCustomLength.Text != "")
      {
          try
          {
              int val = Convert.ToInt32(txtCustomLength.Text);
              prgCreateStringAndPost(val);
          }

          catch (Exception except)
          {
              lblStatus.Text = except.Message;
          }
      }

      else
          MessageBox.Show("Please enter a value in the box next to the Custom button");
  }

Feel free to take this code and build your own string generator. (If the source code got truncated in the post, just go to View Source in your browser and you can copy and paste from there)

Enjoy!