Hi All
I'd like to get your opinions on cloud based load testing tools like LoadStorm and BrowserMob vs more traditional tools like LoadRunner and WebPerformance. What advantage do the cloud tools provide that the traditional ones don't, and vice versa?
Here's my background - I was taught how to do load testing with LoadRunner. My mentor said that you should always set up an isolated network and run tests in that environment to get your app's code running well. Once that's done, move the code to a staging/production environment and then generate load from outside the network against your production environment to see how the hardware responds.
Is this still the recommended way of doing things? If so, it seems that the cloud based tools are at a disadvantage because they can't test against an isolated network. Or, can the results generated by the cloud based tools easily identify where the performance bottlenecks are?
Please sound off in the comments.
Thanks
Nick
Subscribe to:
Post Comments (Atom)
Posts
both ways are recomended.
ReplyDeleteTo locate bottlenecks in algorythm and configurations it is enough to run load testing in isolated network.
For example I have to emulate the netwrok activity of 10 000 users from different hosts with different type of traffic. Simulating thi\s situation with classic tools(Load Runner, RPT, JMeter) is not acceptable - we have appropriate hardware resources to deploy large network.
@Minos - so ideally you'd want to be able to do both inside the network and from the cloud. Would it be safe to say that by load testing inside the network you can tweak your app's performance, and then you can get a "real world" feel for how your app behaves under large load from the cloud without needing to have a ton of your own hardware?
ReplyDeleteBoth methods have merit and are useful in different ways.
ReplyDeleteFor early-cycle type performance testing, I still think you need a local load generator and an isolated network. performance/load/scale testing for an *application* needs to be done on a low latency network. This is where you iteratively run tests to stress various software modules or to run load through an application itself. It is a constant cycle of tests and tuning. Exposing a test/development environment like this to the external Internet is often not feasible or desired.
Testing from the cloud gives you a different view. You have a "cloud" of uncontrolled systems communicating with your application over the internet. It is a *very* different model.
I work for SOASTA, so I obviously have a bias: That said, Corey is exactly right. We're best known for cloud testing but we actually started with, and continue to support, a behind the firewall model, using an appliance, with portability of tests between the two environments.
ReplyDeleteThe combination is useful for best meeting specific test goals, accommodating access restrictions, getting the best bang for your buck and appropriately leveraging resources.
reat question Nick. I cut my teeth on home made tools and then started to use tools like LoadRunner or SilkPerformer. Wonderful tools I did a lot of testing with those tools. And I think they still have a place today although that place has shrunk a whole lot. The advent of cloud based tools and lower cost even open source options have made load and performance testing a reachable option.
ReplyDeleteI agree with the previous posters in that each method has a valuable place. Working in an isolated environment is wondering for looking at your individual systems very closely. As Cory mentioned, this is very helpful for early systems development. However testing from the cloud allows you to get a better view of how your customers/clients experience your systems. Your customers/clients do not live inside your firewall, so for an accurate, real world view, I believe testing from outside your network - the cloud - gives you the best option.
I believe as cloud computing gets more stable and robust in its abilities you will see more and more load and performance testing being done from the cloud. Even HP recognizes this and and is thus going to be allowing people to run Amazon EC2 cloud servers that have LoadRunner on them. In reality HP is validating the cloud model by their move. And they are not the only ones. Many 'traditional' tools now have the ability to run their test from the cloud or internal to the network which in a wonderful option