CloudTest Test Results Query API

Introduction

The Results Query API for CloudTest is a unified REST API that allows CloudTest customers to receive a JSON response with aggregate test data. Data for a given test can be grouped and aggregated by several levels of grouping and can also be filtered by time, location and other attributes.

The following API end points are available:

  • Summary: Queries for results by date range of the test start or result name or composition name. High level metrics for results are returned as well as a result id which will be necessary in order to get more detailed result data using the other end points.

  • Clip Element: Queries action related metrics (HTTP messages, database actions, etc.) from a result with grouping/aggregation by location, time in seconds or minutes, network profile, element type, and/or element name.

  • Collection: Queries collection related metrics (pages, transactions, clips) from a result with grouping/aggregation by location, time in seconds or minutes, network profile, element type, and/or element name

Authentication

For more information, see the Tokens API.

To use the REST APIs, callers must issue a request using the CloudTest URL below to get a security token. Note that the CloudTest API URL is case sensitive.

To get a security token with a username and password:

curl -X PUT -H "Content-type: application/json" --data-binary '{"userName":"<username>", "password":"xxxxx"}' "https://[host name]/concerto/services/rest/RepositoryService/v1/Tokens"

If Single Sign On (SSO) is enabled for the default tenant of the user, an apiToken must be specified instead:

curl -X PUT -H "Content-type: application/json" --data-binary '{"apiToken":"<token>"}' "https://[host name]/concerto/services/rest/RepositoryService/v1/Tokens"

Users who belong to multiple tenants and wish to obtain a security token for a tenant that is not their default, can add the “tenant” key to the JSON body:

curl -X PUT -H "Content-type: application/json" --data-binary '{"apiToken":"<token>", "tenant": "<tenant>"}' "https://[host]/concerto/services/rest/RepositoryService/v1/Tokens"

The REST JSON response will contain the security token needed for further calls: {"token":"da7be4d72030656559f3e41a98924a6b2a544730"}

Once a security token is acquired, it is used in all subsequent queries. The URLs for accessing the REST API use the token, the result id, and any optional parameters that go with the end point, and take the following format:

/concerto/services/rest/Results/v1/<id>/<query-type>?<optional parameters>

If curl is used, the command line looks like this:

curl -H "Authentication: <security token>" "https://[host name]/services/rest/Results/v1/<query type>?<optional parameters>"

Errors

Error Code Meaning
400 Bad Request – Input Error
401 Unauthorized – The Authorization Token is invalid
404 Not Found – The specified resource could not be found
405 Method Not Allowed – Invalid method used to access resource
429 Too many Requests – too many requests were sent within a short amount of time, for the same tenant
500 Internal Server Error – The request caused a server error.

Result Summary

To interact with a result, you will need to know the ID, which can be retrieved using the url below.

/concerto/services/rest/Results/v1?<optional parameters>

If the result ID is known, then summary information about that result can be obtained like this:

/concerto/services/rest/Results/v1/<id>

URL Query Parameters

Parameter Description
ID ID of the Result
date-start Start of the range of test start times to query for. Specified in UTC time since epoch.
date-end End of the range of test start times to query for. Specified in UTC time since epoch.
name Name of the result
composition Name and path of a composition to get results for
orderBy Field to sort by. May optionally be suffixed with “:asc” to specify ascending order. Defaults to descending.
limit Maximum rows to return

Valid orderBy fields

Parameter Description
startTime Result start time
endTime Result completion time
totalTime Total duration of the test
totalMessages Total number of http messages issued during the test
name Name of the result
resultSize Result size
resultsDatabase Results data service

Example URLs

To get details for the result with ID 42:

GET /concerto/services/rest/Results/v1/42

To get details for the result named “Result from Tue Aug 28 2017 09:17:54”:

GET /concerto/services/rest/Results/v1?name=”Result from Tue Aug 28 2017 09:17:54”

To get a list of results for composition “/LatestRuns/Compositions/Prod-100K-VU” sorted by test start time with the oldest result first:

GET /concerto/services/rest/Results/v1?composition=”/LatestRuns/Compositions/Prod-100K-VU”&orderBy=startTime:asc

To get a list of the results run in the last week ordered by result size with the largest first:

GET /concerto/services/rest/Results/v1?date-start=1512086400&date-end=1512691200&orderBy=resultSize:desc

Request

curl -X GET -H "X-Auth-Token: 1234-ABC" -H "Content-Type: application/json" http://[host name]/concerto/services/rest/Results/v1?name=”Result from Tue Aug 28 2017 09:17:54” 

Response

{ "Results":[
  {"id":42,
   "name":"Result from Tue Aug 28 2017 09:17:54", 
   "compositionName": "Prod-100K-VU",
   "compositionPath": "/LatestRuns/Compositions",
   "summary":"Final production test",
   "effectiveStartTime": 1511380336746,
   "endTime": 1511383939576,
   "startTimeAsISO": "2017-11-22T11:52:07.409-08:00",
   "effectiveStartTimeAsISO": "2017-11-22T11:52:16.746-08:00",
   "endTimeAsISO": "2017-11-22T12:52:19.576-08:00",
   "totalMessages": 2180120,
   "totalErrorMessages": 2424,
   "slaCount": 0,
   "slaWarningCount": 0,
   "slaFailureCount": 0,
   "resultSize": 261488640,
   "baselineResultID": 99, 
   "childResultIDs":[26,27,28]
}]}

Clip Element

To interact with a result, you will need to know the ID, which can be retrieved as shown above. The format of this end point is as follows:

/concerto/services/rest/Results/v1/<id>/clip-element?<optional parameters>

URL Query Parameters

Parameter Description
groupBy One or more items to group by. Metrics are aggregated by these groups. See valid values below. Defaults to “elementType”. Multiple items can be specified in two ways: 1) groupBy=location&groupBy=elementType or 2) groupBy=location,elementType
errorsOnly true to get only elements which had an error. Defaults to false.
test-range-start Start of the range to query for (the number of seconds since the effective start of the test). Elements which started in the range are included.
test-range-end End of the range to query for (the number of seconds since the effective start of the test).
annotation-start Name of the Annotation which defines the start of the range to query for. If a range type annotation then this may also specify the range end. Annotations are automatically generated by the system for ramping and steady state phases of a test. It is also possible to create user created annotations.
annotation-end Name of the Annotation which defines the end of the range to query for.
location Location filter. Multiple locations can be specified as follows: “location=Amazon%20EC2%20Mumbai&location=Amazon%20EC2%20US%20East”.
networkProfile NetworkProfile filter. Multiple profiles can be specified as follows: “networkProfile=XXXX&networkProfile=YYYYY”.
elementType Element type filter. Valid values are: “message”, “script”, “browserAction”, “appAction”, “databaseAction”.

Valid groupBy fields

Parameter Description
second Clip Elements which started in the same second
minute Clip Elements which started in the same minute
location Location of the load generator which issued the request
networkProfile Network Profile used for the request
error Clip Elements which share the same error
elementType Element type - eg. message, delay. Except when elementName is specified, elementType is automatically added to end of groupBy list.
elementName Name of the element
flattenedHierarchy Hierarchy expressed as a path. For example: “Shopper Clip/Cart Transaction/Cart Page”.
hierarchy Hierarchy where each node of the hierarchy is a node in the JSON and the JSON reflect a parent/child set of relationships.

Notes:

  1. Not all combinations of groupBy fields are permitted. In particular hierarchy, flattenedHierarchy, and elementName can not be combined with second or minute and can not be combined with time filtering.

  2. The ordering of groupBy fields specifies the hierarchy of nodes in the JSON. For example, groupBy=error,location will result in JSON that looks like this:

{"errors": [{
            "error": 404,
            "locations": [
                {"location": "US East","metrics": {....}}, 
                {"location": "Singapore","metrics": {....}}]
        },
        {
            "error": 500,
            "locations": [
                {"location": "US East","metrics": {....}}]
        .....

while groupBy=location,error will yield this:

{"locations": [{
        "location": "US East",
        "errors": [
            {"error": 404,"metrics": {....}}, 
            {"error": 500,"metrics": {....}}]
        }, {
            "location": "Singapore",
        "errors": [
            {"error": 404,"metrics": {....}}, 
            {"error": 500,"metrics": {....}}]
        ......

Example URLs

To get metrics for the result with ID 42 for each location and elementType combination:

GET /concerto/services/rest/Results/v1/42/clip-element?groupBy=location,elementType

To get metrics for the steady state period of the test:

GET /concerto/services/rest/Results/v1/42/clip-element?annotation-start="Steady%20State%201"&groupBy=elementType

To get metrics for each combination of error, flattenedHierarchy, and elementName and only for those elements which were in error:

GET /concerto/services/rest/Results/v1/42/clip-element?errorsOnly=true&groupBy=error&groupBy=flattenedHierarchy&groupBy=elementName

Request

curl -X GET -H "X-Auth-Token: 1234-ABC" -H "Content-Type: application/json" http://[host name]/concerto/services/rest/Results/v1/clip-element?groupBy=error,hierarchy,elementName 

Response

{  
    "errors": [{
        "error": "HTTP 404",
        "children": [{
            "name": "Scenerio Buyer Clip",
            "type": "Clip",
            "children": [{
                "name": "ShoppingCart Transaction",
                "type": "Transaction",
                "children": [{
                    "name": "Shopping Cart Page",
                    "type": "Page",
                    "children": [{
                        "name": "Message1",
                        "type": "message ",
                        "operation": "GET",
                        "metrics": {
                            "totalCount": 100,
                            "sendCount": 0,
                            "errorCount": 100,
                            "totalResponseTime": 0,
                            "maxResponseTime": 0,
                            "minResponseTime": 0,
                            "totalTime": 5000,
                            "maxTotalTime": 100,
                            "minTotalTime": 1,
                            "totalBytesSent": 0,
                            "totalBytesReceived": 0,
                            "totalTTFB": 0,
                            "totalTTLB": 0
                        }
                    }]
                }]
            }]
        }]
    }]
}

Collections

To interact with a result, you will need to know the ID, which can be retrieved as shown above. The format of this end point is as follows:

/concerto/services/rest/Results/v1/<id>/collection?<optional parameters>

URL Query Parameters

Parameter Description
groupBy One or more items to group by. Metrics are aggregated by these groups. See valid values below. Defaults to one row of metrics for the entire test. Multiple items can be specified in two ways: 1) groupBy=location&groupBy=elementType or 2) groupBy=location,elementType
errorsOnly true to get only collections which had an error. Defaults to false. Note that this is for collections which themselves were in error - not necessarily the items within the collection.
test-range-start Start of the range to query for (the number of seconds since the effective start of the test). Elements which started in the range are included.
test-range-end End of the range to query for (the number of seconds since the effective start of the test).
annotation-start Name of the Annotation which defines the start of the range to query for. If a range type annotation then this may also specify the range end. Annotations are automatically generated by the system for ramping and steady state phases of a test. It is also possible to create user created annotations.
annotation-end Name of the Annotation which defines the end of the range to query for.
location Location filter. Multiple locations can be specified as follows: “location=Amazon%20EC2%20Mumbai&location=Amazon%20EC2%20US%20East”.
networkProfile NetworkProfile filter. Multiple profiles can be specified as follows: “networkProfile=XXXX&networkProfile=YYYYY”.
elementType Element type filter. Valid values are: “clip”, “transaction”, “group”, “page”, “burst”.
percentile One or more percentile values to return. Defaults to 50th percentile. Multiple percentiles can be specified as: “percentile=50&percentile=85”.

Valid groupBy fields

Parameter Description
second Collections which started in the same second
minute Collections which started in the same minute
location Location of the load generator which ran the collection
networkProfile Network Profile used for the collection
error Collections which share the same error. (Note: This is not directly related to errors within the page/transaction/etc.)
elementType Element type - eg. transaction, page.
elementName Name of the element
flattenedHierarchy Hierarchy expressed as a path. For example: “Shopper Clip/Cart Transaction/Cart Page”.
hierarchy Hierarchy where each node of the hierarchy is a node in the JSON and the JSON reflect a parent/child set of relationships.

Notes:

  1. The ordering of groupBy fields specifies the hierarchy of nodes in the JSON. For example, groupBy=error,location will result in JSON that looks like this: {error:404, locations:[{location: US East, …},{location: Singapore,….}],{error: 500, locations …

Example URLs

To get metrics for the result with ID 42 for each location and elementType combination:

GET /concerto/services/rest/Results/v1/42/collection?groupBy=location,elementType

To get metrics for the steady state period of the test:

GET /concerto/services/rest/Results/v1/42/collection?annotation-start="Steady%20State%201"&groupBy=elementType

To get metrics for a 30 minute range that starts 2 minutes into the test for each collection:

GET /concerto/services/rest/Results/v1/42/collection?test-range-start=120&test-range-end=1920"&groupBy=elementName

To get metrics for each combination of flattenedHierarchy, and elementName:

GET /concerto/services/rest/Results/v1/42/collection?errorsOnly=true&groupBy=flattenedHierarchy&groupBy=elementName

To get metrics for the entire test including 50th and 80th percentile:

GET /concerto/services/rest/Results/v1/42/collection?percentile=50&percentile=80

Request

curl -X GET -H "X-Auth-Token: 1234-ABC" -H "Content-Type: application/json" http://ec2-54-165-89-178.compute-1.amazonaws.com/concerto/services/rest/Results/v1/collection?groupBy=hierarchy,elementType 

Response

{
    "children": [{
        "name": "Scenerio Buyer Clip",
        "type": "Clip",
        "metrics": {
            "countStarted": 1000,
            "countCompleted": 900,
            "errorCount": 100,
            "totalDuration": 567890,
            "maxDuration": 4567,
            "minDuration": 2,
            "totalEffectiveDuration": 567890,
            "maxEffectiveDuration": 4567,
            "minEffectiveDuration": 2,
            "percentiles": [{
                "percentile": 50,
                "value": 123
            }],
            "baseMessageResponseTime": 8888,
            "baseMessageResponseCount": 111,
            "baseMessageTTFB": 1234,
            "totalBytesSent": 12345,
            "totalBytesReceived": 67890
        },
        "children": [{
            "name": "ShoppingCart Transaction",
            "type": "Transaction",
            "metrics": {
                "countStarted": 1000,
                "countCompleted": 900,
                "errorCount": 100,
                "totalDuration": 567890,
                "maxDuration": 4567,
                "minDuration": 2,
                "totalEffectiveDuration": 567890,
                "maxEffectiveDuration": 4567,
                "minEffectiveDuration": 2,
                "percentiles": [{
                    "percentile": 50,
                    "value": 123
                }],
                "baseMessageResponseTime": 8888,
                "baseMessageResponseCount": 111,
                "baseMessageTTFB": 1234,
                "totalBytesSent": 12345,
                "totalBytesReceived": 67890
            },
            "children": [{
                "name": "Shopping Cart Page",
                "type": "Page",
                "metrics": {
                    "countStarted": 1000,
                    "countCompleted": 900,
                    "errorCount": 100,
                    "totalDuration": 567890,
                    "maxDuration": 4567,
                    "minDuration": 2,
                    "totalEffectiveDuration": 567890,
                    "maxEffectiveDuration": 4567,
                    "minEffectiveDuration": 2,
                    "percentiles": [{
                        "percentile": 50,
                        "value": 123
                    }],
                    "baseMessageResponseTime": 8888,
                    "baseMessageResponseCount": 111,
                    "baseMessageTTFB": 1234,
                    "totalBytesSent": 12345,
                    "totalBytesReceived": 67890
                }
            }]
        }]
    }]
}