Setting Up Custom DataStream Endpoints

August 13, 2020 · by Achuthananda Padmanabhaiah ·

DataStream provides real-time access to application activity data, including aggregated metrics on complete request-response cycles as well as origin response times. It helps simplify web application monitoring, and includes applications deployed to cloud hosting or that leverage third-party services. 

DataStream works in two modes:

  1. A push model that pushes logs to Splunk, Datadog, S3, SumoLogic, or custom endpoints to the level of aggregation configured.

  2. A pull model that uses an application programming interface (API) to pull data from DataStream — for more details on this model visit

In this post, I will show you how to set up a custom endpoint for DataStream data. This is helpful if you don't have access to Splunk, Datadog, S3, or Sumologic and you want your data to be pushed rather than pulled proactively.


To create a custom endpoint for DataStream, make sure you have an HTTPS server up and running — I use a Google Cloud Instance for this example. Configure as follows:

  1. Set up an Apache server to receive all HTTP requests. I use Python for server-side programming, so I work with the Flask Python web app framework. If you don’t have your server up and running, this step will get you there.

  2. Make the server HTTPS-capable to create a custom endpoint for DataStream. You can create a self-signed certificate and deploy it, but since many browsers will not accept that approach, I generate a certificate with my own domain []. An easy way to obtain an HTTPS certificate is to use a Let’s Encrypt Certbot — this bot generates a certificate and deploys it to Apache.

Create a DataStream

As discussed, DataStream works in both a push and pull model. To pull the data from DataStream, you can make use of APIs. The push model is helpful because there is no manual intervention to see the data — at the intervals defined, the data keeps getting pushed to the endpoints created. To push the data, follow the steps below:

  1. Create a new stream and associate it with a property.



  2. Select the datasets by choosing the data you want to be logged — for example, Request Header Data, Response Header Data, or NetPerf Data.

    Data Sets


  3. Select the endpoint, then unselect the “Do not push to destination” box. Select the destination type and enter the endpoint URL — in my case, where I have configured a POST request handler to handle POST requests by DataStream.



Associate a Delivery Configuration

Associate the DataStream with a delivery configuration by adding a behavior called “DataStream” and entering the ID of the stream.


Create a POST Handler 

Now you need to set up a POST handler at the server side. Since I use Flask, the Python code follows — my implementation listens to POST requests and writes to a file.

code block

Verify the Code

To test that the custom DataStream endpoint works, follow the steps below:

  1. Generate the traffic on the hostname — mine generates a lot of traffic using curl commands.

  2. Verify the POST requests by checking “tail -f /var/log/apache2/modsec_audit.log”.

    code block

  3. Data.txt should contain the data posted.

    code block


About the Author

authorAchuthananda Padmanabhaiah is a Solutions Architect II at Akamai Technologies. Achuth is involved in implementing various solutions of Akamai ranging from web, media, and security. Achuth has also delivered many custom Devops Solutions to Akamai customers and believes that no repetitive task should be done manually and should embrace automation.