Pre-caching online content on the mobile device for offline use

 

TL;DR -  Mobile App Performance (MAP) SDK is being retired and we are using this post to share how our flagship feature (pre-caching) works under the hood with the aim of inspiring mobile app developers to do this on their own using other open-source/DIY tools available.

Problem

In the world of mobile applications, one of the main tasks of any mobile app is to communicate with the network. Every time the application is used, enormous network traffic is generated.

Timeouts, broken connections, huge amounts of downloaded images and data have a huge impact on the behavior and reception of the end-user of an application.

All these factors which may expose users to additional costs related to the data packet used, or other problems may have a key impact on the business.

It is easy to imagine a situation where we start a sale in our store, but loading our offer takes forever, and the user, discouraged by the behavior of the application, closes it instead of buying something.

It is the responsibility of every app developer to provide the best possible user experience while meeting the needs of the business. Combining these two aspects is often very difficult, but it guarantees the success of the mobile app.

 

Solution

Coming back to the problem described, the first and most effective solution to huge and costly network traffic is, of course, caching.

All popular network libraries offer caching mechanisms that are more or less configurable by the developer.

Oftentimes, these cache configurations are sufficient, but over time can become complex, and difficult to maintain.

In trying to solve this problem in Akamai, we managed to create a mechanism that connects caching with some “magic” AI packaged as pre-caching with Mobile App Performance (MAP) SDK. The effects we found across apps when pre-caching was applied were amazing!

Why is Pre-Caching such a cool feature?

Pre-cache helps Mobile App Performance (MAP) SDK learn the popular objects in demand across your user base for the mobile application and then starts downloading them to the device cache when the network is idle to enhance offload at the device. 

Pre-caching studies network requests for the last seven days the first time when it has been configured and then starts updating the pre-cache segment once every 24 hours to stay up-to-date with changing content requirements and patterns. Pre-caching only works for objects/responses which can be cached and are defined using the "Cache-Control" header with a valid "max-age" value.

When the device is offline, content responses for requests are served from the cache until the expiration date. However, if the application sets the “max-stale” value of the “Cache-Control” request header, MAP SDK will continue to serve a stale response even after the expiration date (“max-age”) from the cache.

Unfortunately, not everyone was excited about using an SDK and we had to make the hard decision to retire the product.

Through this article, we aim to share how pre-caching worked to help inspire fellow developer to build some of this logic on your own to enhance your mobile applications.

How does Pre-Caching work under the hood with Mobile App Performance (MAP) SDK?

When Mobile App Performance (MAP) SDK is correctly integrated with a customer app, it intercepts every network call (excluding blacklisted URLs). This is the point where it all begins.

BACKEND:

Each network call to any external resources is logged and saved in an internal database.

In addition to URL, values ​​such as event timestamp, hostname, content type, information whether a given resource is cacheable, and many others are saved. 

The saved data is anonymous, we do not store any sensitive values ​​such as user data or responses. This business logic allows us to store the “full history” of all calls which are made using mobile apps.

Every day, a job is launched to analyze the collected data. The frequency of queries for a given URL, specific time intervals, information on whether a given resource is cacheable is analyzed. Cacheable flag meaning is that response can be cached depending on response headers, file extensions, etc. The obtained result is combined with URLs that the client can enter on the portal, and it’s stored in our internal database.

Whenever the list with pre-cacheable URLs changes, the backend job updates the SDK configuration timestamp and refreshes it.

What are Blacklisted URLs? - Exclude URLs from being cached/pre-cached at the device.

An example database schema may look like this (the example below can be treated as the base one):

 

CREATE TABLE URL_EVENTS

(

   EVENT_ID    NUMBER(19)    NOT NULL,

   EVENT_HOSTNAME VARCHAR2(200 CHAR)   NOT NULL,

   EVENT_TIMESTAMP DATE NOT NULL,

   EVENT_URL VARCHAR2(200 CHAR)   NOT NULL,

   IS_CACHEBALE NUMBER(1) DEFAULT 0 NOT NULL,

   CONTENT_TYPE VARCHAR2(100 CHAR)   NOT NULL,

   -- ADDITIONAL DATA

);

 

IN THE MOBILE APP:

Each network call is saved on the device side as an event.

The SDK call for a new configuration is performed when certain conditions are met, including the application is restored from the background or the number of events saved in the database reaches the limit or other.

After receiving the configuration, the SDK compares the timestamp of the current configuration with that received from the backend.

If they differ, the SDK updates the configuration and saves the new settings.

The pre-cache list is also saved, each new link and content that is available under it (e.g. image, video, data, JSON) is saved on the device.

After saving, each time the pre-cached URL is called, the content will not be re-downloaded over the network but will be returned from the device.

As a result, the application will run faster and the user will be even more satisfied using the application more often.

 

About the authors

Michal

 

Michal Walawender is a Software Engineer II at Akamai Technologies where he has extensive knowledge about developing Mobile apps, and in Akamai he has great knowledge about MAP/mPulse SDK.His experience ranges from small startups to large multi-national corporations.