CloudTest Test Results Query API

Introduction

The Results Query API for CloudTest is a unified REST API that allows CloudTest customers to receive a JSON response with aggregate test data. Data for a given test can be grouped and aggregated by several levels of grouping and can also be filtered by time, location and other attributes.
There is also an api for deleting results asynchronously.

The following API end points are available:

  • Summary: Queries for results by date range of the test start or result name or composition name. High level metrics for results are returned as well as a result id which will be necessary in order to get more detailed result data using the other end points.

  • Clip Element: Queries action related metrics (HTTP messages, database actions, etc.) from a result with grouping/aggregation by location, time in seconds or minutes, network profile, element type, and/or element name. This is what you would use to get the number of HTTP 404 errors or compute the average response time or determine which assets contained the largest number of bytes.

  • Collection: Queries collection related metrics (pages, transactions, clips) from a result with grouping/aggregation by location, time in seconds or minutes, network profile, element type, and/or element name. This is what you would use to determine page load time or transactions per second (tps) or get the 90th percentile for a transaction.

  • Management: Allows results to be deleted asynchronously to help with result organization and to free up disk space.

Authentication

For more information, see the Tokens API.

To use the REST APIs, callers must issue a request using the CloudTest URL below to get a security token. Note that the CloudTest API URL is case sensitive.

To get a security token with a username and password:

curl -X PUT -H "Content-type: application/json" --data-binary '{"userName":"<username>", "password":"xxxxx"}' "https://[host name]/concerto/services/rest/RepositoryService/v1/Tokens"

If Single Sign On (SSO) is enabled for the default tenant of the user, an apiToken must be specified instead:

curl -X PUT -H "Content-type: application/json" --data-binary '{"apiToken":"<token>"}' "https://[host name]/concerto/services/rest/RepositoryService/v1/Tokens"

Users who belong to multiple tenants and wish to obtain a security token for a tenant that is not their default, can add the “tenant” key to the JSON body:

curl -X PUT -H "Content-type: application/json" --data-binary '{"apiToken":"<token>", "tenant": "<tenant>"}' "https://[host]/concerto/services/rest/RepositoryService/v1/Tokens"

The REST JSON response will contain the security token needed for further calls: {"token":"da7be4d72030656559f3e41a98924a6b2a544730"}

Once a security token is acquired, it is used in all subsequent queries. The URLs for accessing the REST API use the token, the result id, and any optional parameters that go with the end point, and take the following format:

/concerto/services/rest/Results/v1/<id>/<query-type>?<optional parameters>

If curl is used, the command line looks like this:

curl -H "Authentication: <security token>" "https://[host name]/services/rest/Results/v1/<query type>?<optional parameters>"

Errors

Error Code Meaning
400 Bad Request – Input Error
401 Unauthorized – The Authorization Token is invalid
404 Not Found – The specified resource could not be found
405 Method Not Allowed – Invalid method used to access resource
409 Conflict – An existing request is in conflict with this request
429 Too many Requests – too many requests were sent within a short amount of time, for the same tenant
500 Internal Server Error – The request caused a server error.

Result Summary

To interact with a result, you will need to know the ID, which can be retrieved using the url below.

/concerto/services/rest/Results/v1?<optional parameters>

If the result ID is known, then summary information about that result can be obtained like this:

/concerto/services/rest/Results/v1/<id>

URL Query Parameters

Parameter Description
ID ID of the Result
date-start Start of the range of test start times to query for. Specified in UTC time since epoch.
date-end End of the range of test start times to query for. Specified in UTC time since epoch.
name Name of the result
composition Name and path of a composition to get results for
orderBy Field to sort by. May optionally be suffixed with “:asc” to specify ascending order. Defaults to descending.
limit Maximum rows to return

Valid orderBy fields

Parameter Description
startTime Result start time
endTime Result completion time
totalTime Total duration of the test
totalMessages Total number of http messages issued during the test
name Name of the result
resultSize Result size
resultsDatabase Results data service

Example URLs

To get details for the result with ID 42:

GET /concerto/services/rest/Results/v1/42

To get details for the result named “Result from Tue Aug 28 2017 09:17:54”:

GET /concerto/services/rest/Results/v1?name=”Result from Tue Aug 28 2017 09:17:54”

To get a list of results for composition “/LatestRuns/Compositions/Prod-100K-VU” sorted by test start time with the oldest result first:

GET /concerto/services/rest/Results/v1?composition=”/LatestRuns/Compositions/Prod-100K-VU”&orderBy=startTime:asc

To get a list of the results run in the last week ordered by result size with the largest first:

GET /concerto/services/rest/Results/v1?date-start=1512086400&date-end=1512691200&orderBy=resultSize:desc

Request

curl -X GET -H "X-Auth-Token: 1234-ABC" -H "Content-Type: application/json" http://[host name]/concerto/services/rest/Results/v1?name=”Result from Tue Aug 28 2017 09:17:54” 

Response

{ "Results":[
  {"id":42,
   "name":"Result from Tue Aug 28 2017 09:17:54", 
   "compositionName": "Prod-100K-VU",
   "compositionPath": "/LatestRuns/Compositions",
   "summary":"Final production test",
   "status":"Completed",
   "effectiveStartTime": 1511380336746,
   "endTime": 1511383939576,
   "startTimeAsISO": "2017-11-22T11:52:07.409-08:00",
   "effectiveStartTimeAsISO": "2017-11-22T11:52:16.746-08:00",
   "endTimeAsISO": "2017-11-22T12:52:19.576-08:00",
   "totalMessages": 2180120,
   "totalErrorMessages": 2424,
   "slaCount": 0,
   "slaWarningCount": 0,
   "slaFailureCount": 0,
   "resultSize": 261488640,
   "baselineResultID": 99, 
   "childResultIDs":[26,27,28]
}]}

Clip Element

To interact with a result, you will need to know the ID, which can be retrieved as shown above. The format of this end point is as follows:

/concerto/services/rest/Results/v1/<id>/clip-element?<optional parameters>

URL Query Parameters

Parameter Description
groupBy One or more items to group by. Metrics are aggregated by these groups. See valid values below. Defaults to “elementType”. Multiple items can be specified in two ways: 1) groupBy=location&groupBy=elementType or 2) groupBy=location,elementType
errorsOnly true to get only elements which had an error. Defaults to false.
test-range-start Start of the range to query for (the number of seconds since the effective start of the test). Elements which started in the range are included.
test-range-end End of the range to query for (the number of seconds since the effective start of the test).
annotation-start Name of the Annotation which defines the start of the range to query for. If a range type annotation then this may also specify the range end. Annotations are automatically generated by the system for ramping and steady state phases of a test. It is also possible to create user created annotations.
annotation-end Name of the Annotation which defines the end of the range to query for.
location Location filter. Multiple locations can be specified as follows: “location=Amazon%20EC2%20Mumbai&location=Amazon%20EC2%20US%20East”.
networkProfile NetworkProfile filter. Multiple profiles can be specified as follows: “networkProfile=XXXX&networkProfile=YYYYY”.
elementType Element type filter. Valid values are: “message”, “script”, “browserAction”, “appAction”, “databaseAction”.

Valid groupBy fields

Parameter Description
second Clip Elements which started in the same second
minute Clip Elements which started in the same minute
location Location of the load generator which issued the request
networkProfile Network Profile used for the request
error Clip Elements which share the same error
elementType Element type - eg. message, delay. Except when elementName is specified, elementType is automatically added to end of groupBy list.
elementName Name of the element
flattenedHierarchy Hierarchy expressed as a path. For example: “Shopper Clip/Cart Transaction/Cart Page”.
hierarchy Hierarchy where each node of the hierarchy is a node in the JSON and the JSON reflect a parent/child set of relationships.

Notes:

  1. Not all combinations of groupBy fields are permitted. In particular hierarchy, flattenedHierarchy, and elementName can not be combined with second or minute and can not be combined with time filtering.

  2. The ordering of groupBy fields specifies the hierarchy of nodes in the JSON. For example, groupBy=error,location will result in JSON that looks like this:

{"errors": [{
                "error": 404,
                "locations": [
                        {"location": "US East","metrics": {....}},
                        {"location": "Singapore","metrics": {....}}]
                },
                {
                "error": 500,
                "locations": [
                        {"location": "US East","metrics": {....}}]
                .....

while groupBy=location,error will yield this:

{"locations": [{
                "location": "US East",
                "errors": [
                        {"error": 404,"metrics": {....}},
                        {"error": 500,"metrics": {....}}]
                },
                {
                "location": "Singapore",
                "errors": [
                        {"error": 404,"metrics": {....}},
                        {"error": 500,"metrics": {....}}]
                ......

Example URLs

To get metrics for the result with ID 42 for each location and elementType combination:

GET /concerto/services/rest/Results/v1/42/clip-element?groupBy=location,elementType

To get metrics for the steady state period of the test:

GET /concerto/services/rest/Results/v1/42/clip-element?annotation-start="Steady%20State%201"&groupBy=elementType

To get metrics for each combination of error, flattenedHierarchy, and elementName and only for those elements which were in error:

GET /concerto/services/rest/Results/v1/42/clip-element?errorsOnly=true&groupBy=error&groupBy=flattenedHierarchy&groupBy=elementName

Request

curl -X GET -H "X-Auth-Token: 1234-ABC" -H "Content-Type: application/json" http://[host name]/concerto/services/rest/Results/v1/42/clip-element?groupBy=error,hierarchy,elementName 

Response

{
        "errors": [{
                "error": "HTTP 404",
                "children": [{
                        "name": "Scenerio Buyer Clip",
                        "containerType": "clip",
                        "children": [{
                                "name": "ShoppingCart Transaction",
                                "containerType": "transaction",
                                "children": [{
                                        "name": "Shopping Cart Page",
                                        "containerType": "page",
                                        "children": [{
                                                "name": "Message1",
                                                "elementType": "message ",
                                                "operation": "get",
                                                "metrics": {
                                                        "count": 100,
                                                        "sendCount": 0,
                                                        "errors": 100,
                                                        "responseTime": 0,
                                                        "maxResponseTime": 0,
                                                        "minResponseTime": 0,
                                                        "totalTime": 5000,
                                                        "maxTotalTime": 100,
                                                        "bytesSent": 0,
                                                        "bytesReceived": 0,
                                                        "totalTTFB": 0,
                                                        "totalTTLB": 0
                                                }
                                        }]
                                }]
                        }]
                }]
        }]
}

Collections

To interact with a result, you will need to know the ID, which can be retrieved as shown above. The format of this end point is as follows:

/concerto/services/rest/Results/v1/<id>/collection?<optional parameters>

URL Query Parameters

Parameter Description
groupBy One or more items to group by. Metrics are aggregated by these groups. See valid values below. Defaults to one row of metrics for the entire test. Multiple items can be specified in two ways: 1) groupBy=location&groupBy=elementType or 2) groupBy=location,elementType
errorsOnly true to get only collections which had an error. Defaults to false. Note that this is for collections which themselves were in error - not necessarily the items within the collection.
test-range-start Start of the range to query for (the number of seconds since the effective start of the test). Elements which started in the range are included.
test-range-end End of the range to query for (the number of seconds since the effective start of the test).
annotation-start Name of the Annotation which defines the start of the range to query for. If a range type annotation then this may also specify the range end. Annotations are automatically generated by the system for ramping and steady state phases of a test. It is also possible to create user created annotations.
annotation-end Name of the Annotation which defines the end of the range to query for.
location Location filter. Multiple locations can be specified as follows: “location=Amazon%20EC2%20Mumbai&location=Amazon%20EC2%20US%20East”.
networkProfile NetworkProfile filter. Multiple profiles can be specified as follows: “networkProfile=XXXX&networkProfile=YYYYY”.
elementType Element type filter. Valid values are: “clip”, “transaction”, “group”, “page”, “burst”.
percentile One or more percentile values to return. Defaults to 50th percentile. Multiple percentiles can be specified as: “percentile=50&percentile=85”.

Valid groupBy fields

Parameter Description
second Collections which started in the same second
minute Collections which started in the same minute
location Location of the load generator which ran the collection
networkProfile Network Profile used for the collection
error Collections which share the same error. (Note: This is not directly related to errors within the page/transaction/etc.)
elementType Element type - eg. transaction, page.
elementName Name of the element
flattenedHierarchy Hierarchy expressed as a path. For example: “Shopper Clip/Cart Transaction/Cart Page”.
hierarchy Hierarchy where each node of the hierarchy is a node in the JSON and the JSON reflect a parent/child set of relationships.

Notes:

  1. Not all combinations of groupBy fields are permitted. In particular hierarchy, flattenedHierarchy, and elementName can not be combined with second or minute and can not be combined with time filtering.

  2. The ordering of groupBy fields specifies the hierarchy of nodes in the JSON. For example, groupBy=error,location will result in JSON that looks like this: {error:404, locations:[{location: US East, …},{location: Singapore,….}],{error: 500, locations …

Example URLs

To get metrics for the result with ID 42 for each location and elementType combination:

GET /concerto/services/rest/Results/v1/42/collection?groupBy=location,elementType

To get metrics for the steady state period of the test:

GET /concerto/services/rest/Results/v1/42/collection?annotation-start="Steady%20State%201"&groupBy=elementType

To get metrics for a 30 minute range that starts 2 minutes into the test for each collection:

GET /concerto/services/rest/Results/v1/42/collection?test-range-start=120&test-range-end=1920&groupBy=elementName

To get metrics for each combination of flattenedHierarchy, and elementName:

GET /concerto/services/rest/Results/v1/42/collection?errorsOnly=true&groupBy=flattenedHierarchy&groupBy=elementName

To get metrics for the entire test including 50th and 80th percentile:

GET /concerto/services/rest/Results/v1/42/collection?percentile=50&percentile=80

To get metrics for every page in a test, grouped by location, and including those pages which failed or stopped early when the test stopped. For each page include the 85th and 95th percentiles.

GET /concerto/services/rest/Results/v1/42/collection?groupBy=location&includeStopped=true&includeFailed=true&percentile=85&percentile=90

Request

curl -X GET -H "X-Auth-Token: 1234-ABC" -H "Content-Type: application/json" http://ec2-54-165-89-178.compute-1.amazonaws.com/concerto/services/rest/Results/v1/42/collection?groupBy=hierarchy,elementType 

Response

{
   "children":[
      {
         "name":"Scenerio Buyer Clip",
         "containerType":"Clip",
         "elementTypes":[
            {
               "elementType":"clip",
               "metrics":{
                  "started":1000,
                  "fullDuration":567890,
                  "errors":100,
                  "effectiveDuration":567890,
                  "minEffectiveDuration":2,
                  "maxEffectiveDuration":4567,
                  "totalResponseCount":212,
                  "totalResponseTime":612565,
                  "minResponseTime":2,
                  "maxResponseTime":2387,
                  "averageResponseTime":687,
                  "percentiles": [
                    {
                      "percentile": 50,
                      "value":456988
                    }
                  ],
                  "effectiveDurationStandardOfDeviation":2.96570556239706,
                  "totalHtmlResponseCount":111,
                  "totalHtmlPageResponseTime":8888,
                  "totalHtmlPageTTFB":1234,
                  "bytesSent":12345,
                  "bytesReceived":67890,
                  "completed":900,
                  "tps":0.021957271150341434
               },
               "children":[
                  {
                     "name":"ShoppingCart Transaction",
                     "type":"Transaction",
                     "metrics":{
                        "started":1000,
                        "fullDuration":567890,
                        "errors":100,
                        "effectiveDuration":567890,
                        "minEffectiveDuration":2,
                        "maxEffectiveDuration":4567,
                        "totalResponseCount":212,
                        "totalResponseTime":612565,
                        "minResponseTime":2,
                        "maxResponseTime":2387,
                        "averageResponseTime":687,
                        "percentiles": [
                         {
                            "percentile": 50,
                            "value": 456988
                          }
                        ],
                        "effectiveDurationStandardOfDeviation":2.96570556239706,
                        "totalHtmlResponseCount":111,
                        "totalHtmlPageResponseTime":8888,
                        "totalHtmlPageTTFB":1234,
                        "bytesSent":12345,
                        "bytesReceived":67890,
                        "completed":900,
                        "tps":0.021957271150341434
                     },
                     "children":[
                        {
                           "name":"Shopping Cart Page",
                           "type":"Page",
                           "metrics":{
                              "started":1000,
                              "fullDuration":567890,
                              "errors":100,
                              "effectiveDuration":567890,
                              "minEffectiveDuration":2,
                              "maxEffectiveDuration":4567,
                              "totalResponseCount":212,
                              "totalResponseTime":612565,
                              "minResponseTime":2,
                              "maxResponseTime":2387,
                              "averageResponseTime":687,
                              "percentiles": [
                                {
                                   "percentile": 50,
                                   "value": 456988
                                }
                              ],
                              "effectiveDurationStandardOfDeviation":2.96570556239706,
                              "totalHtmlResponseCount":111,
                              "totalHtmlPageResponseTime":8888,
                              "totalHtmlPageTTFB":1234,
                              "bytesSent":12345,
                              "bytesReceived":67890,
                              "completed":900,
                              "tps":0.021957271150341434
                           }
                        }
                     ]
                  }
               ]
            }
         ]
      }
   ]
}

Results Management

To initiate a request to delete results asynchronously, you issue a POST which will create a background task to select and delete results. You can then poll using GET requests to see the status and results of the task. A DELETE command can be used to attempt to stop the task. The POST uses a json payload to specify which results should be deleted. Results which are marked “locked” in the UI are never deleted as well as any tests which are still in a running state. The POST is done with the following URL.

/concerto/services/rest/Results/management/v1

The POST command returns an HTTP 202 Accepted status code and JSON which contains two items: 1) The task id which was submitted, and 2) a URL which can be used to get the status and result of the task. Information about that task can be obtained like this:

/concerto/services/rest/Results/management/v1/<id>

By issuing a GET on this url you will get a response which contains the status, start time, and percentage complete as well as information on all results which have been deleted. Information about tasks remains available for approximately one hour after they complete.

JSON Payload

Parameter Description
status One of the following result completion status: Completed, Failed, Stopped, Aborted
preview True to not actually delete any results but will report back the results that it would have deleted. Defaults to true.
dateRangeStart Start of the range of test end times to query for. eg. “2019-04-28”, “2019-04-28T12:30:00”, “2019-04-28T12:30:00-0800”
dateRangeEnd End of the range of test end times to query for. See above for available formats.
resultsDatabaseName Name of a results database to limit the result selection. Defaults to all running databases.
olderThan Delete results for which it has been more than N days since they completed. Defaults to 0.
minResultSize Delete results which are greater than or equal to the specified size. Sizes can be specified in bytes, or KB/MB/GB. eg. 15.4 MB
maxResultSize Delete results which are less than or equal to the specified size. Sizes can be specified in bytes, or KB/MB/GB. eg. 1260KB
minDuration Delete results for test that ran greater than or equal to the specified duration (in seconds).
maxDuration Delete results for test that ran less than or equal to the specified duration (in seconds).
limit Maximum rows to delete. Defaults to 50 (which can be changed via the repository service setting “Results.Management.Deletion.Limit”)

Example URLs

To submit a delete request for all results which are older than 5 days:

POST /concerto/services/rest/Results/management/v1 --data-binary {"olderThan":5, "preview":false}

To submite a delete request for all results which completed in June, 2019 as measured by Pacific Standard Time.

POST /concerto/services/rest/Results/management/v1 --data-binary {"dateRangeStart":"2019-06-01T00:00:00-0800","dateRangeEnd":"2019-06-30T23:59:59-0800", "preview":false}

To get the current status and results of the above task (if the task id is 457):

GET /concerto/services/rest/Results/management/v1/457

To get a list of all tasks:

GET /concerto/services/rest/Results/management/v1

To attempt to stop a currently running task:

DELETE /concerto/services/rest/Results/management/v1/457

Stopping a task does not undo any deleted results. It just tells the task to stop as quickly as it can. Tasks which have been stopped have a status of “STOPPED”. A status of “COMPLETED” means that the task was completed before it received the stop command.

POST Request

curl -X POST -H "X-Auth-Token: 1234-ABC" -H "Content-Type: application/json" http://[host name]/concerto/services/rest/Results/management/v1  --data-binary \
{
  "olderThan" : 5,
  "resultsDatabaseName" : "Main",
  "dateRangeStart" : "2018-05-01"
}

Response

{
  "task": {
    "taskid": 1738473444,
    "uri": "/concerto/services/rest/Results/management/v1/1738473444",
    "status":"SCHEDULED"
  }
}

GET Request

curl -X GET -H "X-Auth-Token: 1234-ABC" http://[host name]/concerto/services/rest/Results/management/v1/1738473444 

Response

{
  "task": {
    "taskid": 1738473444,
    "uri": "/concerto/services/rest/Results/management/v1/1738473444",
    "status": "STARTED",
    "startTimeAsISO": "2019-05-13T15:47:12.182-07:00",
    "totalItems": 2,
    "completedItems": 0,
    "percent-complete": 0
  }
}

After a task has completed the response will also include the list of deleted results:

{
  "task": {
    "taskid": 1738473444,
    "uri": "/concerto/services/rest/Results/management/v1/1738473444",
    "status": "COMPLETED",
    "startTimeAsISO": "2019-05-13T15:47:12.182-07:00",
    "endTimeAsISO": "2019-05-13T15:47:23.182-07:00",
    "totalItems": 2,
    "completedItems": 2,
    "percent-complete": 100,
    "details": {
      "results": [
        {
          "id": 5,
          "name": "Result from Thu Dec 11 2018",
          "compositionPath": "/Results Service unit tests/Empty Composition",
          "summary": "Send of Message \"Message2\" failed. (Band \"Band 1\" Track \"Track 1\" Clip \"Static Page clip\" Message \"Message2\" Target \"mysite.com\")",
          "status": "Failed",
          "resultSize": "4063232"
        },
        {
          "id": 8,
          "name": "Result from Thu Dec 31 2018",
          "compositionPath": "/Regression Tests/Shopping Cart Checkout Test",
          "summary": "Composition stopped.",
          "status": "Stopped",
          "resultSize": "1523712"
        }
      ]
    }
  }
}