Data Services Stress Testing Framework
From Adobe Labs
| Table of contents |
Overview
The Data Services Stress Testing Framework helps developers using LiveCycle Data Services ES 2.6 load test the server-side implementation of their LCDS applications. This is a tool for putting load on the server and is not meant for stress testing an individual Flex/BlazeDS/LCDS application running in the Flash Player. This framework is not currently compatible with Blaze Data Services although BlazeDS support is planned for future versions of the tool.
LiveCycle Data Services ES 2.7 offers a number of data transport channels for communicating between a Flex client using LiveCycle Data Services ES and the LiveCycle Data Services ES server. A channel consists of a protocol, which is the format of the data being exchanged, and some communication behaviour, which is the means by which the data is exchanged. One of the channels offered by LCDS for example is the Streaming AMF channel. The protocol used by the Streaming AMF channel is AMF (specifically AMF3) and the communication behaviour is streaming. To provide the streaming behaviour, a persistent downstream connection between the LCDS server and the client is used for delivering messages to the client. This provides a real-time experience for the client application because as soon as the server receives a message intended for the client it can immediately be delivered over the persistent connection. For more information about the channels available in LCDS please refer to your LiveCycle Data Services ES 2.6 documentation. RTMP is another channel offered by LCDS. RTMP has it's own message format and provides simultaneous upstream and downstream communication over the same connection. With the terminology we have been using, RTMP is both a protocol and the behaviour with which messages are sent and received using that protocol.
Some commercial load testing tools have recently added support for protocols that are used by the LiveCycle Data Services channels such as AMF3 (AMF3 is the newer version of AMF used by LCDS). RTMP is a proprietary protocol used by the Flash player and currently not supported by commercial load testing tools.
As of the writing of this documentation, no commercial load testing tools support any of the LCDS channel communication behaviours besides simple request/response. Simple request/response communication can be used in LCDS for making a Remote Procedure Call (RPC) using the HTTP Service, Remoting Service or Web Service, where the client makes a request and the server immediately sends a response. Simple request/response communication does not work for Messaging and Data Management where a message may come from another client or be pushed by the server. In these scenarios the client doesn't make a request to the server and get a response. The client must poll the server for updates or have a persistent connection to the server over which to receive updates.
No commercial load testing tools currently support both the communication protocols and behaviours used by LiveCycle Data Services ES for Messaging and Data Management. The Data Services Stress Testing Framework allows LCDS developers to load test their applications that use these features without needing to use a commecial test tool.
The Data Services Stress Testing Framework runs a small socket server called the Browser Server on a number of client computers. The Browser Server is responsible for launching web browsers on each client computer that it runs on. Each web browser then loads a target Flex/LCDS test application written in MXML. This LCDS application then puts load on the server. To increase load, you set the Browser Server to launch more browser instances. When the resources of the client computer running the browser server are fully utilized, you can add additional client computers for the Browser Server to manage
A Flex/LCDS application called the DSStress Test Administrator runs the load test. The DSStress Test Administrator application communicates with the browser servers and with the Flex/LCDS test application running in each browser instance. The DSStress Test Administrator application communicates runtime configuration information to the browser servers, including how many browser instances to launch and which Flex/LCDS test application to run. The DSStress Test Administrator communicates directly with each instance of the Flex/LCDS test application that is launched by the browser server in order to start and stop the test run and collect test results. Starting and stopping test runs and collecting test results is managed using an LCDS Data Management Service destination (called Perf.Participants in the data-management-config.xml) to which the Test Administrator application and each Flex/LCDS test application instance subscribes. The Perf.Participants destination uses its own RTMP endpoint, so updating the load test results is not affected by the Flex/LCDS test application that sends and receives messages as part of the test.
Adobe is currently working with a number of testing tool vendors to implement all of the data transport protocols available in Flash. In addition to partnering with testing tool vendors, we are providing the following load testing framework to enable LiveCycle Data Services developers to load test their applications. The Data Services Stress Testing Framework is currently in beta form, but available for testing and evaluation. Please download and test the framework and provide us your feedback.
Download
To use the Data Services Stress Testing Framework with your LiveCycle Data Services ES 2.6 applications, follow these steps:
1. Download the Data Services Stress Testing Framework and view the license.
Note: Your use of this site including software downloads, submission of comments, ideas, feature requests and techniques on this and other Adobe maintained forums, as well as Adobe’s rights to use such materials, is governed by the Terms of Use.
2. Unzip dsstress.zip
3. Follow the detailed instructions in the Client computer setup and Server setup instructions below.
Client computer setup
An browser server is run on each client computer that will be used to generate load against the LiveCycle Data Services ES server. The browser server is responsible for launching multiple web browsers, each of which runs an instance a Flex/LCDS load test application. The load test applications then make requests to the server putting it under load.
Complete the following steps to set up client computers:
- Copy the browserServer directory from the dsstress_install_root\resources directory to each of your client computers.
- Start the browser server by running "java -jar browserServer.jar " in the browserServer directory on the client computer
- The command line should show the following information, which indicates that the browser server is ready:
Tue Jul 18 15:33:13 EDT 2006 : browserServer version 0.95
Tue Jul 18 15:33:14 EDT 2006 : starting server on port 7777
Tue Jul 18 15:33:14 EDT 2006 : server ready
The browser server's configuration can be changed using the browserServer.properties file that lives in the same directory as the browserServer.jar. Here is an example of the contents of the browserServer.propeties file.
ie="C:\\Program Files\\Internet Explorer\\iexplore.exe" firefox="C:\\Program Files\\Mozilla Firefox\\firefox.exe" default=ie serverPort=7777
You can change the port that a browser server listens on by changing the serverPort property in the properties file. When you use the DSStress Test Administrator application to run a load test, you must use the new serverPort value when entering connection information for the browser server.
The browserServer.properties file also contains a list of browser types that can be used on the computer running the browser server. To add support for a different browser type, create a property which is an alias name for the browser, for exaple "opera." Then, for the value of this property enter the path to the executable for the browser on the filesystem. When you use the TestAdmin application to run a load test, you specify the alias for the browser that you want to use that was defined in the browserServer.properties file. The default settings for Internet Explorer and Firefox should work on most Windows systems. On 64 bit Windows machines, however, you will need to change the path to the executable to point to the 32 bit version of Internet Explorer as the Flash Player is not supported on 64 bit browsers. Also, on non Windows systems you will need to change the path for the executables to be valid for your operating system.
Each browser server can launch only one type of browser at a time, but you can run more than one instance of the browser server on a single computer as long as each instance uses a different port. Each instance of the browser server can then launch a different browser. For example, one browser server instance on the computer can use Internet Explorer and the other can use Firefox. If you don't specify a browser to use when configuring the browser server in the DSStress Test Administrator application, it uses the browser specified by the default property in the browserServer.properties file.
Server setup
Complete the following steps to set up your LiveCycle Data Services ES web application to use the Data Services Stress Testing Framework. XML snippets containing the changes that need to be made to each of the configuration files can also be found in the dsstress_install_root\resources\config directory. A compiled version of the TestAdmin application used for running the load tests has been included in the dsstress_install_root\testadmin directory. This compiled swf assumes that your LCDS web application has a context root of "lcds". If your web application uses a different context root you will need to recompile the TestAdmin application. Instructions on how to recompile the TestAdmin application can be found at the end of this section of the documentation.
1. Add the following channel definition to the services-config.xml file in your LiveCycle Data Services ES web application. This is the separate RTMP endpoint that is used by the Flex/LCDS test applications so that they can update test results without interfering with publishing and receiving messages that are part of the test.
<channel-definition class="mx.messaging.channels.RTMPChannel" id="perf-rtmp-internal">
<endpoint class="flex.messaging.endpoints.RTMPEndpoint" url="rtmp://{server.name}:2152"></endpoint>
<properties>
<idle-timeout-minutes>0</idle-timeout-minutes>
<serialization>
<enable-small-messages>true</enable-small-messages>
</serialization>
<!-- for deployment on WebSphere, must be mapped to a WorkManager available in the web application's jndi context.
<websphere-workmanager-jndi-name>java:comp/env/wm/MessagingWorkManager</websphere-workmanager-jndi-name>
-->
</properties>
</channel-definition>
2. Add the following destination to the data-management-config.xml file in your LiveCycle Data Services ES web application. This is the destination that the Flex/LCDS test applications add their results to and the Test Administrator application gets results from.
<destination adapter="java-dao" channels="perf-rtmp-internal" id="Perf.Participants"> <properties> <source>adobe.dsstress.controller.lcds.ParticipantAssembler</source> <scope>application</scope> <use-transactions>false</use-transactions> <metadata> <identity property="id"></identity> </metadata> <network> <session-timeout>0</session-timeout> <paging enabled="false"></paging> </network> <server> <fill-method> <name>fill</name> </fill-method> <sync-method> <name>sync</name> </sync-method> </server> </properties> </destination>
3. Add the following destination to the remoting-config.xml file. This destination allows the Flex/LCDS test applications to log debug messages to System.out on the server.
<destination id="dsstress.Logger">
<channels>
<channel ref="perf-rtmp-internal"></channel>
</channels>
<properties>
<source>adobe.dsstress.Logger</source>
<scope>request</scope>
</properties>
</destination>
4. The Test Administrator application communicates with the browser server instances by using the DefaultHTTP destination of the HTTPService. The DefaultHTTP destination has a list of URL patterns, or dynamic URLs that act as a whitelist, that specifies which URLs the destination can access. For the Test Administrator application to access the browser server instances, you modify the <dynamic-url> settings for the DefaultHTTP destination.
To set up dynamic URLs for DefaultHTTP destination, open the proxy-config.xml file for your LiveCycle Data Services ES web application, and find the DefaultHTTP destination. The easiest way to give the Test Administrator application access to all of the browser server instances is by adding a <dynamic-url> entry with value of "*", as the following example shows. This lets the DefaultHTTP destination make a request to any URL. Although this is not recommended for a production environment, it is acceptable for testing behind a firewall. Alternatively, you can add <dynamic-url> entries with more restrictive file patterns that give access only to computers on which the browser server is running or the specific port on which the browser server is listening. For more information about configuring the HTTPService, see the LiveCycle Data Services ES documentation.
<destination id="DefaultHTTP">
<properties>
<dynamic-url>*</dynamic-url>
</properties>
</destination>
5. Copy the file dsstress.jar from the dsstress_install_root\resources\lib directory to the WEB-INF\lib directory of your LiveCycle Data Services ES web application.
6. If your LCDS web application uses a context root other than "lcds" you will need to recompile the TestAdmin MXML application prior to deploying it. The source for the TestAdmin can be found in the dsstress_install_root\resources\src directory. When you compile the TestAdmin.mxml application, the dsstress.swc file from the dstress_install_root\resources\frameworks\libs directory needs to be in the path of your compiler. This swc contains ActionScript classes used by TestAdmin. Make sure when you compile the TestAdmin that the images directory is in the same directory as the MXML file as the contents of this directory need to be compiled into the swf. If you are using a Flex Project in Flex Builder to compile the application you can add the dsstress.swc file to the libs directory in your project. Alternatively, you can add dsstress.swc to the frameworks\libs directory of your LiveCycle Data Services ES 2.6 SDK. For more information on compiling Flex applications that use LiveCycle Data Services ES 2.6 view the file lcds_install_root\resources\frameworks\libs\readme.html that came with your LiveCycle Data Services ES installation. Once you have compiled the TestAdmin.mxml file into a swf, copy the swf into the dsstress_install_root\testadmin directory and proceed to the next step.
7. Copy the testadmin directory from the dsstress_install_root to the root directory of your LiveCycle Data Services ES web application.
Writing a load test
You can use any Flex application as the basis for your load test. There are just a few things you add to a Flex application to use it for load testing. The following section assumes you are already familiar with compiling Flex applications that use LiveCycle Data Services ES 2.6. For more information on compiling Flex applications that use LiveCycle Data Services ES 2.6 view the file lcds_install_root\resources\frameworks\libs\readme.html that came with your LiveCycle Data Services ES installation.
The Data Services Stress Testing Framework requires that some ActionScript classes be compiled into your load test application. These ActionScript classes allow the TestAdmin application to start and stop the load test and to collect test results from each of the load test application instances. These ActionScript classes are contained in the dstress_install_root\resources\frameworks\libs\dsstress.swc file. When you compile your load test application this swc needs to be in path of your compiler. If you created a Flex Project in Flex Builder you can add the dsstress.swc file to the libs directory in your project. Alternatively, you can add dsstress.swc to the frameworks\libs directory of your LiveCycle Data Services ES 2.6 SDK.
The following code is an example of a simple load test. All this test does is increment a counter when a timer event is dispatched, which in this example is once every second. The number of times that the timer event is dispatched becomes the metric that we track for our load test.
<?xml version="1.0"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" creationComplete="run()">
<mx:Script>
<![CDATA[
import adobe.dsstress.client.LoadTestResult;
import adobe.dsstress.client.LCDSTestManager;
import adobe.dsstress.client.ITestManager;
import flash.utils.Timer;
import flash.events.TimerEvent;
private var requestTimer:Timer;
//counter to capture number of messages received
private var messageCount:int = 0;
//every stress test will need to create a TestManager
private var testManager:ITestManager;
public function run():void {
requestTimer = new Timer(1000, 0);
requestTimer.addEventListener("timer", requestHandler);
//the framework passes in the unique id of the TestManager at runtime
var id:String = Application.application.parameters.id;
//create a TestManager instance passing in the unique id.
testManager = new LCDSTestManager(id);
//setup event listeners for starting and stopping the test
testManager.addEventListener("startTest",startTest);
testManager.addEventListener("stopTest", stopTest);
}
//this is called by the test framework to start the test
private function startTest(e:Event):void {
requestTimer.start();
}
//this is called by the test framework to stop the test
private function stopTest(e:Event):void {
requestTimer.stop();
//Create a new LoadTestResult object for the test results
var results:LoadTestResult = new LoadTestResult();
//set the messageCount to the number of messages received
results.messageCount = messageCount;
//could set customCount to another metric you are tracking
//results.customCount = sendCount;
//could set failureCount to number of failures if you had any
//results.failureCount = failureCount;
//call addResults on the TestManager to send the results to the server
testManager.addResults(results);
}
private function requestHandler(event:TimerEvent):void {
//when the timer gets invoked increment the counter. if this test were using messaging would
//increment the counter when the message handler got invoked
messageCount++;
}
]]>
</mx:Script>
</mx:Application>
Every load test must define and create an instance of a class that implements the ITestManager interface. For testing using LiveCycle Data Services, the class that you create is the LCDSTestManager. This is currently the only implementation of ITestManager. In the future we may add a new test manager to work with BlazeDS. You can decide where to create a new test manager. In this example, a new instance of the LCDSTestManager class is created in a run() method that is called when the application finishes loading. When the LCDSTestManager instance is created, it is added to the data service, and the DSStress Test Administrator application can access it. The Test Administrator application lists that browser/application instance as being connected and assumes that it is then ready to be used in the load test. If there is any setup or initialization that you must do, you should do it before instantiating the LCDSTestManager class. An example of where you would want to do some additional setup is in a messaging load test. You would probably want to wait for the Consumer component to have successfully subscribed to the messaging destination before instantiating the LCDSTestManager. An example of a test that uses messaging can be found in the samples folder included with this distribution.
The constructor for the LCDSTestManager class takes a String that uniquely identifies the application/browser instance. When the Test Admin application calls the browser server, it passes the URL of the MXML test application to the browser server. The Test Admin application also passes a unique ID that identifies the application instance as a request parameter at the end of the URL. The following URL is an example of a URL passed to the browser server:
http://localhost:8400/lcds/DSStress/samples/simpleLoadTest.mxml?id=localhost:7777(0)
When the web browser loads the MXML test application, the request parameter "id" is passed into the compiled SWF file and can be found on the Application.application.parameters Object. The ID is retrieved from Application.application.parameters.id and then passed in the constructor for the LCDSTestManager.
Next, event listeners that are dispatched by the Test Admin application are added. The startTest event is dispatched when the Test Admin application starts the load test. The stopTest event is dispatched when the Test Admin application stops the test.
In your handler for the startTest event, you start running your test. In the sample test, when the load test starts all the start test method does is call start on a timer. If this was a messaging load test, you could start publishing messages here; or if this was a data service load test, you could call the fill() method on your DataService component or call the insert(), update(), or delete() method on your managed ArrayCollection, and so forth.
In the sample test, every time the timer event is dispatched, the messageCount counter is incremented. If this was a Message Service test, you might increment the counter in the message handler that is invoked when the Consumer component receives a message; if it was a Data Management Service test, you might increment the counter when the result handler is invoked from a call to a DataService fill() method. If your load test consists of a series of operations, you would increment the counter when the entire test scenario or series of operations is completed.
In the handler for the stopTest event, the sample test stops the timer. This is for clean-up purposes. If this was a Message Service test, you might also unsubscribe from the destination at this point.
Next, the test creates a new instance of adobe.dsstress.client.LoadTestResult to store the test results. The test sets the messageCount property to the value of the messageCount counter. There is also a failureCount property you can use to track the number of failures the test receives. For example if this was a Message Service test you might have a fault handler that listens for a MessageFaultEvent. In the MessageFaultHandler you could increment a failureCount counter and then assign this value to the failureCount property of the LoadTestResult instance. The LoadTestResult class also has a customCount property that can be used to track other metrics. For example if you had a Message Service test where you wanted to capture both the number of messages that were sent from the test application and received by the test application you could store the number of messages received in the messageCount property and the number of messages sent in the sendCount property.
The last step is to call the addResults method on the LCDSTestManager, passing in the LoadTestResult object. This causes the test results to get sent back to the Test Admin application which collects the results from all the test application instances.
This section contains the basic guidelines for writing a load test. For more examples, see the samples directory included with this distribution. When you have finished writing your load test, compile it and copy it to your LiveCycle Data Services ES web application.
Running a load test
Before starting to run a load test, ensure that the browser server is running on each client computer that will participate in the load test and that the correct version of Flash Player is installed on each of the computers. Next, with your application server running, open the testadmin/TestAdmin.html file you copied from the DSStress installation to your LCDS web application in a web browser. The url should be similar to http://localhost:8400/lcds/testadmin/TestAdmin.html.
The following image shows the TestAdmin application:
1. Enter information for the test run in the Test Settings section. A test run consists of one or more load tests. Each load test will be run in sequence, one after the other. For each load test enter the name of the test, the number of browsers to launch on each client machine where the browser server is running, and the url to the load test in your LCDS web application. The test name is an arbitrary name that will be used to identify the test in the test results and in the test report. For ramp up type testing you might have tests with different numbers of browsers but the same test url, so the test name could be something like simpleLoadTest_5_Browsers which will uniquely identify the test. The Browsers per server setting controls how many browsers are launched by each instance of the browser server. If you have 5 browser servers and Browsers per server is set to 5, the load test will run using 25 browsers. The test url is the url to the load test deployed in your LCDS web application for example http://localhost:8400/lcds/simpleLoadTest/simpleLoadTest.swf. After entering information for the load test, click the Add button which will add it to the list of load tests to be executed during the test run. If you need to remove a load test from the test run, select the test in the DataGrid and click the Remove Test button.
2. Set the length of time each test will be run for by entering the time in seconds in the Test time box. The default test length is 60 seconds. You can run an open ended load test which will run until you manually stop the test by entering -1 in the Test time field. To run an open ended load test the browser server timeout also needs to be set to -1 to stop the browser server from attempting to do cleanup and killing the browsers.
3. The TestAdmin application can ensure that each of the test application instances is using the correct version of the Flash Player. To enable this feature, enter the required Flash Player version in the Require Player version box. When the load test is starting, the Flash Player version is checked and if any of the test applications are using the wrong Flash Player version, the load test does not run.
4. Enter the Browser Server settings. For each browser server, in the Server address box, enter the IP address or hostname of the computer where the browser server is running. In the Port box, enter the port on which the browser server is listening. In the Browser box, enter the name of the browser. The name of the browser is an alias that is specified in the browserServer.properties file and maps to the executable for the browser. For example, "ie" is used by default for Internet Explorer. In the Browser timeout seconds box, enter a time in seconds after which you want the browser server to attempt to do cleanup and kill any remaining browser that did not shut down cleanly. At the end of the load test the TestAdmin sends a message to the browser server to shut down all the browsers it is managing, but if there is a problem and the browser server doesn't get this message it can be useful to have a browser server timeout which will kill any remaining browsers. Make sure that the browser server timeout value is greater than the test time so the browser instances are not shutdown prematurely. If you want to run an open ended load test, however, the browser server timeout needs to be set to -1 to stop the browser server from attempting to do cleanup. After you finish adding the information for the browser server, click the Add button. You should see a new entry for this browser server added to the DataGrid. If the TestAdmin application can communicate correctly with the browser server, the browser server shows a status of "connected" in the DataGrid. If you believe the status of a browser server has changed, for example if you forgot to start the browser server on the client machine before adding it in the TestAdmin, you can click the Check Servers button to update the status of the browser servers. You can use the Remove Server button to remove a server from the list of browser servers.
5. Now that the settings for the test run have been entered, and the browser servers have been added, click the Start Test button. This will launch the browsers which will load the test application in each of the browser instances. In the Clients section of the TestAdmin you should see the test application clients being added in the righthand DataGrid. As the test application client loads in each browser instance, the connected property for the client should change from "false" to "true." When all of the test application instances have connected, the TestAdmin will send the command to the test application clients to start running the test and the Status field in Test Results section of the TestAdmin will change from "Starting" to "Runnng".
6. When the test finishes, the Status of the test in the Test Results section of the TestAdmin changes to "Finished" and you can view the results. The Message count field shows the total number of messages, transactions, and so forth that completed in the given time period. The particular metric that is stored in this field was determined by you when you wrote the load test. The Failure count field is meant to show the total number of failures that occurred while the load test was running although what is stored in this field is also determined by what is in the load test application. The Custom count field is meant to allow you track another metric. If you are running a messaging test for example with a Producer and a Consumer in each test application and the Message count is the number of messages received by the Consumers, you might track the number of messages sent by the Producers in the Custom count field. All of these result fields show the cumulative total from all of the test application clients. The Messages per second property is the Message count divided by the time period.
7. Troubleshooting. Two problems you may run into are that the test application clients from a previous test run may not have shutdown correctly and a test application client may not have started up correctly. If the first occurs, when you run the test there will be more Connected clients listed in the Clients section of the TestAdmin than there should be. For example, if the number of Browsers per server for your test is 5 and there are 5 Browser servers, there should be 25 test application clients; but if there is a test application client hanging around from a previous test run this number may be 26. The test won't start running in this case because the number of clients is incorrect. To fix this problem click the Abort button in the TestAdmin. This will kill all the test application instances and reset the number of connected clients to 0. The next time you run the load test, the number of connected clients should be accurate. If a test application client doesn't startup correctly, the connected property will remain set to "false". You can try clicking the Abort button to abort the current test run and rerun the test. If this doesn't fix the problem you should try to debug the problem on the computer that is running the browser server to see why test application clients are not starting correctly on that machine.
The Abort button can be used at any time to abort a test run. It causes the TestAdmin to remove all of the test application clients that are currently connected and each of the browser server instances to shutdown all of the browsers they are managing. If there are more tests remaining in the test run they will not be executed. You will also not receive any results for the test that was aborted. If you want to stop a test run prematurely and still receive test results you can use the Stop button in the TestAdmin. The Stop button stops the test that is currently executing and collects the test results for the test before shutting it down. If there are more tests remaining in the test run they will not be executed.
After the test run has finished, the "Send Report" button can be clicked to do additional reporting on the test results. By default, this button will just dump the test results to the flashlog file. Note, you must be using the debug player for trace statements to be written to flashlog. Debug players for a variety of operating systems ship in the Flex SDK which is included in the LiveCycle Data Services ES distribution. If you want to do additional reporting you need to write your own reporting class. For example you might write a reporting class that sends the test results to a web service that writes the results to a database. For more information on writing reporting classes see the "Writing custom reports" section of this documentation.
There is a Logging/Debugging panel in the TestAdmin that shows some basic debugging information related to connecting to the browser server instances and launching browsers. To view the Logging/Debugging panel click the bar in the TestAdmin that says Logging/Debugging. If there is a problem connecting to a browser server instance, the information here can be useful, but you should also check the server console output to see if the LiveCycle Data Services ES server reported any errors.
Passing parameters to tests
You can pass parameters to each instance of your load test by appending the parameters to the end of the test url in the Data Services TestAdmin. Currently the only parameter names that are supported are "channel" and "dataSize" but in future versions of the framework parameters with any name should be supported. These parameters get sent to the browser server along with the test URL. The browser server then parses the URL and parameters from the request and builds a new URL string which it sends to the browser. In the TestAdmin, the parameters should be appended to the end of the test url using url query string syntax. This means the list of parameters should begin with a "?" and individual parameters should be separated by "&". In the following example, the two parameters channel and dataSize will be passed to each instance of the load test at runtime. The value of the channel property is "my-rtmp" and the value of dataSize is "5".
A good use of the channel parameter is to pass in the name of the channel to use in the load test at runtime. This will allow you to write a single test that can be run over a number of channels. You might want to run a test over different channels to see what the performance of the application will be with different channels. For example your application may use a Streaming AMF channel by default but fallback to AMF Polling. In that case you may want to see what the performance of the application will be in both cases. The following code snippet shows how you might use the channel parameter in your load test.
//create a new channelset with the channel we want to use in the test. var channelSet:ChannelSet = new ChannelSet(); var channel:Channel = ServerConfig.getChannel(Application.application.parameters.channel); channelSet.addChannel(channel); con.channelSet = channelSet; con.subscribe();
The example first creates a new ChannelSet. It then uses the ServerConfig.getChannel() method to retrieve a channel with the supplied name. The name that is supplied is value that you passed in which can be retrieved from the channel property on the Application.application.parameters object. After creating the channel, the example then adds it to the ChannelSet and then assigns this ChannelSet to a Consumer component names con. The Consumer then subscribes over the channel.
A good use of the dataSize parameter is to pass in the size of a dataset. This will allow you to write a single test that can run with different dataset sizes. You might want to run a test with different dataset sizes to see how the size of the data set in your application affects performance. The following code snippet shows how you might use the dataSize parameter in your load test.
var obj:Object = new Object();
obj.prop1 = "hello";
obj.prop2 = 12345;
for (var i:Number = 0; i < Application.application.parameters.dataSize; i++) {
ac.addItem(ObjectUtil.copy(obj));
}
The example creates a dynamic object and adds a couple properties to it. The example then uses a loop to add n copies of this object to an ArrayCollection with the name ac. The size of the loop is the value of the dataSize property that you passed in the URL parameter. The ArrayCollection can be used later in the test, for example it could be sent in the body of a message that is sent to a messaging destination. There are many ways you can use dataSize propery. In a Data Management load test for example you might just want to pass this parameter in a fill() request to do testing with managed collections of different sizes. An example of a test that uses parameters passed from the TestAdmin application can be found in the samples directory included with this distribution.
Using the TestAdmin config file
When the Data Services TestAdmin application is run, it reads in settings from a configuration file called config.xml. The config.xml file needs to be in the same directory as the TestAdmin.swf. The config.xml file can be used to configure your load test run without having to manually enter the settings. Here is an example config.xml file.
<?xml version="1.0" encoding="utf-8"?> <config> <serverAddress value="192.168.x"/> <testDelaySeconds value="5"/> <servers> <server ip="localhost" port="7777" browser="ie" timeout="-1"/> </servers> <tests> <test testName="SimpleMessagingTest" browsersPerServer="5" testURL="http://localhost:8400/lcds/simpleLoadTest/simpleLoadTest.swf"/> </tests> <reports> <report reportClass="adobe.dsstress.admin.reports.FlashlogCSVReport"/> </reports> </config>
The serverAddress property can be used to set a default Server address to be used in the Browser Server Settings. You might use this to save typing if all the client machines used in your load test are in the same subnet.
The testDelaySeconds property is the amount of time to wait in seconds after running a test before running the next test in a test run. This setting is used to make sure there is enough time for the previous test in the test run to get cleaned up before running the next test. If you are running tests with large numbers of test application clients it might take some time for the browser servers to shutdown all of the browser instances. The 5 second value that is used here should be good in most cases but if you see problems with a test starting before the previous test has finished cleaning up try increasing this value. Also note that this option is not exposed in the Data Services TestAdmin application so must be set through the config.xml file.
The servers property can be used to populate the browser servers DataGrid in the Browser Server Settings section of the TestAdmin. In addition to populating the DataGrid when the configuration information is loaded, the TestAdmin will try to connect to each of the browser servers. The servers property takes a list of server entries using the format shown above. For each server, supply the ip address or hostname, the port on which the browser server is listening, the browser alias and a timeout value just as you would if you were adding the browser server using the TestAdmin. See the "Running a load test" section of this documentation for more information about these settings.
The tests property can be used to populate the tests DataGrid in the Test Settings section of the TestAdmin. The servers property takes a list of test entries using the format shown above. For each test, supply the test name, the numbers of browsers to launch on each browser server, and the test URL just as you would if you were adding the test using the TestAdmin. See the "Writing a load test" section of this documentation for more information about these settings.
The reports property can be used to add custom report classes that will be used by the TestAdmin to do reporting. The reports property takes a list of report entries using the format shown above. For each report, supply the fully qualified classname of the report class. The report class must implement the IReport interface. For more information on writing reports see the "Writing custom reports" section of this documentation. Also note that this option is not exposed in the Data Services TestAdmin application so must set it through the config.xml file.
Writing custom reports
When you click the "Send Report" button in the TestAdmin, the TestAdmin will loop over all of the registered report classes that were added listed in the config.xml file. For each report class, the TestAdmin will instantiate the class and then call its createReport method. The createReport method will be passed an ArrayCollection with the load test results at which point reporting operations can be done on the results.
The reporting class that you write must implement the adobe.dsstress.admin.reports interface.
package adobe.dsstress.admin.reports
{
import mx.collections.ArrayCollection;
/**
* Interface which reporting classes in the TestAdmin app must
* implement. Users can add their own reporting classes to do custom
* reporting.
*/
public interface IReport
{
/**
* The createReport method receives an array of finished tests
* which it can then use for reporting purposes.
*
* @param tests An array of finished tests.
*/
function createReport(tests:ArrayCollection):void;
}
}
This interface has a single method called createReport which will be passed an ArrayCollection with the test results. It's up to you to determine what you do with the test results at this point. For example, you might send the test results to a WebService which loads them into a database. The ArrayCollection that gets passed to the createReport method contains objects of type adobe.dsstress.admin.LoadTestResult. Here is the implementation of the LoadTestResult class so you can see what properties are available to you when you write your custom report class.
package adobe.dsstress.admin
{
[Bindable]
public class LoadTestResult
{
////////////////////////////////////////////////////////////////////////////
//
// Variables
//
////////////////////////////////////////////////////////////////////////////
private var _status:String;
private var _testName:String;
private var _time:Number;
private var _browsers:Number;
private var _messageCount:Number = 0;
private var _failureCount:Number = 0;
private var _customCount:Number = 0;
private var _messagesPerSecond:Number = 0;
////////////////////////////////////////////////////////////////////////////
//
// Constructor
//
////////////////////////////////////////////////////////////////////////////
public function LoadTestResult()
{
}
public function get status():String
{
return _status;
}
public function set status(status:String):void
{
_status = status;
}
public function get testName():String
{
return _testName;
}
private function set testName(testName:String):void
{
_testName = testName;
}
public function get time():Number
{
return _time;
}
private function set time(time:Number):void
{
_time = time;
}
public function get browsers():Number
{
return _browsers;
}
private function set browsers(browsers:Number):void
{
_browsers = browsers;
}
public function get customCount():Number
{
return _customCount;
}
private function set customCount(customCount:Number):void
{
_customCount = customCount;
}
public function get messageCount():Number
{
return _messageCount;
}
private function set messageCount(messageCount:Number):void
{
_messageCount = messageCount;
}
public function get failureCount():Number
{
return _failureCount;
}
private function set failureCount(failureCount:Number):void
{
_failureCount = failureCount;
}
public function get messagesPerSecond():Number
{
return _messagesPerSecond;
}
private function set messagesPerSecond(messagesPerSecond:Number):void
{
_messagesPerSecond = messagesPerSecond;
}
}
}
Conclusion
The information in this article can help you start load testing your own LiveCycle Data Services ES applications. It should be fairly easy to build on the sample test case in this article to create more complicated Data Management Service and Message Service test scenarios.
