Comparison of LoadRunner with OpenSTA

Comparison Between LoadRunner And OpenSTA

1.Protocols

The communication protocols that can be captured, manipulated and replayed by the tool.

Load Runner

Many supported. Protocols are charged per item. Has a multi-protocol recording feature.


Opensta

HTTP 1.0 / 1.1 / HTTPS (SSL) only.

2.Playback functions

Replaying of the script and script debugging facilities.


Load Runner

Extended logging supports view of parameter values and Server messages. Also view and comparison with ‘recorded’ version of web page view and client response messages. Debugging facilities in script generator, step and breakpoints.


Opensta

Similar playback facilities, but no integrated comparison function. The debugging functions are in the controller, including set break points and single stepping.


3.Scripting language

The medium used to represent the captured protocol data and manipulate the data for play-back.

Load Runner

Uses standard syntax for “C” & allows C function libraries to be added. Has extensive customised functions for the different protocols supported by the tool.

Opensta

SCL, it uses a proprietary, “BASIC” like language that has special automation scriptingfacilities. Is limited in available functions, such as string manipulation and supports direct DOM addressing.

Extensibility

The ability to increase the functionality of the tool.

1.Load Runner

Additional TSL or “C” function libraries, limited to functional capabilities of the tool.

2.OpenSTA

SCL script modules can be defined in â??Includeâ?? files. Open Source therefore new tool functionality can be added using C++.

Scripting Interface

The interfaces supplied by the tool application for the purpose of script editing.

1.Load Runner

Captures in several modes, high level context based and low level HTTP view. Has both a graphical tree structure & a script view. Script view has function sensitive entry.

2.OpenSTA

Has low-level HTTP protocol view and provides graphical tree representation of the â??Document Object Modelâ?? (DOM) structure. Viewable captured HTML rendering and addressable server-header table. Some language sensitive, syntax colour coding functionality.

Correlation

The task of substituting values in dynamic data to enable successful playback.

1.Load Runner

Automated correlation faculties. Including during recording, after recording and comparing recordings with playback results. Not available for all modes of capture.

2.OpenSTA

Manual correlation using graphical interactive â??DOMâ?? structure. Facility to automatically generate script code to aid variable substitution.

Cookie Management

Detection, recording and playback of HTTP cookies. Both tools need additional code to manage JavaScript generated cookies.

1.Load Runner

HTTP header cookies are managed automatically and can be manipulated manually if required.

2.OpenSTA

HTTP header cookies are managed automatically and can be manipulated manually if required.

Parameterisation

Automatically changing dynamic data values for more accurate emulation of real users. Often essential for session management.

1.Load Runner

Extensive facilities for data entry including wizard interface to DB interrogation. No standard function to lock data sources and maintain uniqueness of concurrently accessed data across distributed tests.

2.OpenSTA

Extensive facilities for data entry including wizard interface to automatically generate test data. Standard functions for sequential, random and pseudo-random data-file access. Has standard common locking facilities for maintaining uniqueness of parameters for an individual load injector or across all injectors on a distributed test.

Controller

Application that manages and conducts a test.

1.Load Runner

Facilities for real-time monitoring. Automatic scenario generation. Individual control of vusers, scripts and groups of scripts. Scheduling, percentage runs & iterations.

2.OpenSTA

Facilities for real-time monitoring. Simple drag & drop multi-scenario test configuration supporting modular scripting and addition of new scenarios/Vus at run-time. No automatic scenario generation. Allows test-wide and user-specific HTTP monitoring and debugging while multi-user load in progress.

Monitoring

Resource usage information is captured during execution.

1.Load Runner

It can be shown during execution and used to build performance reports. Many supported. On-line graphics during execution. Supplied with Apache, Netscape & IIS, other monitors are charged per item. Results used for reporting. New facility to allow remote users to monitor real-time results via a browser based interface. Note: To monitor through a firewall requires TCPIP access through a specific port. Future versions of LoadRunner should use HTTP messaging & avoid this issue.

2.OpenSTA

Supports Integrated real-time, graph view of Windows NT/2000 Performance (Perfmon) & SNMP collectors. Various measurements of test progress including VU-specific, custom status and activity information. Web-relay allows monitors to run on remote machines beyond firewall. On-line graphics during execution, & monitored results used for reporting.

Distributed tests

The ability to distribute the generation of load across multiple load-injector machines.

1.Load Runner

Supports multiple load-injectors managed by a single controller.

2.OpenSTA

Supports multiple load-injectors managed by a single controller. Uses TCP/IP if on the same network or the Web-relay feature uses HTTP to control load-injectors located within remote DMZs.


IP Spoofing


The ability to emulate the behaviour of different IP addresses accessing a system. Particularly useful with Load balancing systems.

1.Load Runner

Supports IP spoofing including automated router updates for IP forwarding.

2.OpenSTA

No inbuilt features to enable IP spoofing.


WAN/LAN emulation

The ability to emulate the behaviour of different network infrastructures during a test.

1.Load Runner

New feature added to version 7.6. Allows the emulation of latency, packet loss, link faults & dynamic routing effects over the LAN used in a test. Requires a special license.

2.OpenSTA

No inbuilt features to enable WAN/LAN emulation.

Caching

The ability to emulate the caching of pages as performed by a web browser.

1.Load Runner

Can control browser cache emulation during playback and control setting for each individual vuser.

2.OpenSTA

No specific faculties, although can be emulated with simple script code.

User connection speed emulation

The ability to emulate the different network speeds that can be used by real users.

1.Load Runner

Can emulate different network speeds during playback

2.OpenSTA

No inbuilt features to emulate user-connection speed emulation.

Distributed/Remote load generation

To generate large loads additional load generator machines are required. These machines must be centrally controlled.

1.Load Runner

Can control multiple load generators and collect results.
Can control load generators located at remote networks (through a firewall) if required using HTTP ports.

2.OpenSTA

Can control multiple load generators and collect results.
Can control load generators located at remote networks (through a firewall) if required using HTTP ports.


Reporting & analysis

The facilities to examine and investigate the results of a test including timers and monitored resources and display the results in graphical format.

1.Load Runner

Sophisticated & large range of charts and graphs with overlay facilities. -Automatic report generation into MS-Word. Analyser is a separate application that can be distributed to users.

2.OpenSTA

Simple charts and graphs sufficient for analysing key load related statistics and resource usage monitors. Resource usage monitors supports graph overlays. Can be exported to Microsoft Excel. No license restrictions on OpenSTA distribution thus stats can be viewed by any user with access. -Free tools and excel macros available through public user-forums.


Scalability

The ability of the tool to generate numbers of virtual users and the corresponding resource usage. Actual resource use depends on the number, size and complexity of the scripts.

1.Load Runner

Resource limitations are number of threads and RAM. Approx 1 vuser per ½ Mb RAM for NT/W2K. Windows 95, 98 & Unix are less efficient. Max. approx 1,500 VUs per PC.

2.OpenSTA

Main resource used is RAM. Tested for simple ASP pages, reaching up to 3,000 users on load generator of 1Gb RAM on a single P4 processor & W2K. Unconfirmed report of a limit of 1664 Vus per Win2K machine for complex scripts. Suspected thread limitation. No licensing limitations.


Initial Costs

Purchase costs of software and licenses, excluding upgrades or support.

1.Load Runner

From £16,000 for basic package with no virtual users. Additional charge for each protocol and monitored resource and for virtual users.

2.OpenSTA

FREE¦ download from SourceForge via www.OpenSTA.org. Downloads available: Previous versions; Automatic installer or current source code (with simple build instructions for MS C++ Visual Studio 6).
Costs of Virtual users
Most commercial tools charge on the basis of the number of virtual user available. Extra hardware is an additional cost.

Prices vary but for the purpose of expectation: additional vusers cost from £10K for 25 VUs to £66K for 1000 VUs. Also temporary VU days at £3.50 each per day (min 1000). This is not a quoted price.

FREE. No licensing limitations.


Support & consultancy

The support services available for the tool and some costs.

1.Load Runner

By M.I., approx 1/5 of initial cost per annum. Includes upgrades. MI & their partner companies also provide consultancy (including radiumtesting.blogspot.com).

2.OpenSTA

Various independent resources. etest associates charged from £50 per incident for remote tech support. Consultancy rates available on request.Numerous online resources including web and e-mail forums. Upgrades are free (approx every 3-6 months)


Training

The training services available for the tool.

1.Load Runner

MI has a range of courses from around £400 per day per person. Many partners also provide training.

2.OpenSTA

Specialist companies provide tailored training, prices vary.

System requirements

The operating systems required to host the tool. (Not the OS of the system under test).

1.Load Runner

MS windows 2000, NT4 (sp6a),XP-Pro (also 95 & 98 cannot run multi-threaded) - Load generators also support limited generator functionality on: Unix: HP, Solaris, Linux .

2.OpenSTA

MS windows 2000, NT4 (sp5+), XP-Pro.

Hardware requirements.

The hardware requirements to host the tool. (Not the hardware of the system under test).

1.Load Runner

Min: Pentium 350 & 128M - Load generators: Pentium 1GHz & 1 MB per VU.

2.OpenSTA

Min : Pentium 200 & 80MB RAM. Pref: Pentium 500MHz+ & 128MB+ RAM.


Access to source code

Availability of the source code of the tool itself.

1.Load Runner

Unavailable.

2.OpenSTA

Open Source GNU public license - ‘C++’ Language.

A users opinion.

Opinion of Internet performance engineers who have experience with both tools.

1.Load Runner

Has a very user friendly user interface and fantastic monitoring & results analysis. Automatic correlation and improved script recording facilities can aid productivity. Very flexible scripting functions & great help documents. Complexity of options and layout of controller are drawbacks. The availability (at a cost) of numerous protocols for recording is a significant strength.

2.OpenSTA

Easy to use interface and excellent scalability. Inbuilt results analysis is slightly limited in comparison to LoadRunner. The captured data is open and can also be exported to Excel in two clicks. Drag & drop scenario set-up and control is very intuitive and easy to interact with, simplifying creation of scenarios with modular scripts. Manual correlation can be a headache but is eased slightly by using third-party ‘Diff’ tools and the inbuilt GUI DOM addressing feature. A relative shortage of standard script language functions but enough to get most HTTP load testing jobs done. If not, with “Includes” support and the availability of the source code, it is very extensible