PATH - A Tool Box for Automated Testing
How To Lop A Zero Off Your Volume Test Costs
PATH is a proven set of tools that aid the automation of multi-user testing and the building of test environments. The tools provide
- Script capture
- Script engineering
- Database builds and expansions
- Multi-Test synchronisation over WANs
- SQL investigation
- Result Analysis and Visualisation
..... in addition to a generic script driver architecture which has enabled us to drive all the protocols we have yet encountered.
The Basic Idea
An automated test tool must
- Reproduce the traffic between the user and the system under test, in an application-specific and timely manner.
- Record meaningful response times whilst also detecting application error conditions.
Furthermore, scripts executed for each virtual test user must vary the data submitted in order to exercise any back end database in a realistic way.
In our early days we developed such a tool that we sold as a commercial software product for automated functional and performance testing.
Our design objectives were to
- Minimise the resources required by our user emulator to execute a test
- Avoid the need for a practitioner to have advanced programming skills.
Our software met its performance objectives, and achieved modest commercial success. We sold the PATH product to three customers, including the U.K. Ministry of Defence.
However, within 18 months our product was shelf-ware everywhere we had sold it.
As indeed, we suspect, are our competitors' products!
Nevertheless, we believe that our subsequent success over more than two decades as Consultants using our own software validates our minimalist approach.
- We consider test scripts to be Document Templates rather than Programs. In our model, Test Generation is more akin to Mail-merge than anything else.
- Our script drivers run on multiple platforms thus we easily carry out sychronised tests that are distributed across heterogeneous networks, hardware and software. They even run on Android smart-phones.
- From the beginning, we have had the ability to distribute our tests around a global network, and collect the results centrally. Cloud availability extends our capabilities by simplifying test provisioning, and reducing our costs, but doesn't otherwise affect us.
- Because we have minimised script driver resource usage, we can oft-times run the script drivers on the System Under Test (SUT) without invalidating the results, at least for initial testing.
For web-only tests, our approach is browser-neutral.
We can produce results most quickly by using our drivers' proxy capabilities to capture the scripts.
In other cases, the most flexible approach is to create script templates from network traffic (no matter what protocol) as it leaves the client PC, and to drive the scripts with a driver that effectively injects and processes network traffic. This approach assures us that we are capturing ALL the effects that the PC is having on its environment.
If either of these approaches is practicable we can drive
- Handfuls of concurrent simulated users with an Android Mobile Phone.
- Thousands of concurrent simulated users with an obsolete laptop.
- Millions of concurrent simulated users from a Public Cloud (if the Systems Under Test has sufficient internet bandwidth available to them!)
Over the years we have reverse engineered dozens of protocols from network traces. Apart from the trivial cases of HTTP and HTTPS, protocols that we have reverse engineered that are still commercially significant include .NET Remoting, Java RMI, ORACLE Forms Services, and the Documentum and Objective Enterprise Document Management Systems.
Second choice is use of a low-level client API. Most of our "Thick Client" tests follow this approach; the API might be ORACLE OCI, Sybase CT-Lib, Ingres, or something Microsoft-proprietary. It may mean that the number of simulated clients that can be driven from a given piece of hardware is reduced by a factor of 10.
Last choice is driving using mouseclicks and key-strokes. Nevertheless, if the application is Virtual Desktop Integration, this is what we must do.
Script capture and script driving do not need to take the same approach. For ORACLE OCI, we capture from the network, and drive using the API; for Citrix, we capture mouse-clicks and keystrokes, but again drive with an API.
Single users' workstations (traditionally PC's, but today almost anything) of course need to be tested and their contribution to end-user response understood but this is usually best handled independently of the testing of the multi-user components; the WAN, Firewalls, LAN, load-balancers, middle tier/application servers, web servers, database servers and all the rest.
Our minimalist approach extends to our Software Licencing Charges.
We don't make any.
We feel that our competitors' software hire fees alone can exceed the amount that it should be necessary to pay for an entire testing project, particularly if you need to test thousands of concurrent users, and the facilities thus purchased do nothing to speed up the rest of the testing project cycle, relative to a project based on PATH; rather the reverse.
The Benefits of PATH
PATH has a 23 year track record and has been used with the widest range of technologies.
PATH offers a flexible and versatile approach thus reducing costs and time scales.
The test drivers can be run from a mixture of operating systems and hardware platforms including all flavours of UNIX and Windows; anything for which a C compiler is available will do.
PATH can be used to intelligently drive any protocol and can thus be used to simulate data feeds or other interfaces in additional to any form of terminal/PC activity. For instance, IP Phones accessed via Computer/Telephony Interfaces, and Automated Teller Machines.
PATH script drivers can be distributed around a global network, tests run, results collected - all without leaving the central test control point.
PATH and E2 Systems represent the ultimate in Cost-effective Performance Testing. But don't just take our word for it. Have a look at the Web Version of our Workbench (we also have Green Screen and Thick Client versions) here, and see for yourself how you can lop a zero off the cost of your Volume Tests.