performance-testing-cloud

DATA ANALYSIS IN THE CLOUD

In this case study we present an overview of our process for creating a cloud based data analysis application for Foresight Factory. The application enables users to explore and chart trends in global consumer behaviours.

Background

We have been working with Foresight Factory over the past 2 years to design, develop and deliver a world class trends platform. With the core platform in place we more recently moved on to scoping the development of modern data analytics services and an intuitive and extendable user interface.

The initial challenge was to understand the data-set and discover the different views of the data that Foresight Factory’s customers wanted to see. This was quickly followed by unravelling the complexities of the large data store and planning a scalable architecture for the data analysis platform.

Approach

Working closely with the Foresight Factory product team we ran a number of workshops to refine requirements and to investigate and validate UX flows. These workshops included paper prototyping sessions, user conversations and testing as well as deep technical dives. Off the back of these sessions we started to develop a component and service based architecture that would enable us to build out the tools and interfaces required.

With scoping well underway, we developed software diagrams and wireframes that enabled team members to understand the overall system and the complex work flows of each component. This backbone was then used to drive a number of fast design and development iterations to create a prototype data platform and user interface.  We then set about validating and refining the system design prototype through extensive user testing.

Solution

The final solution combines a multi-cloud (Azure and AWS) infrastructure for managing data pipelines with a ReactJS/GraphQL based user interface to query and chart the data.

To solve the challenge of managing huge volumes of raw data we developed a series of infrastructure as code processes to create a system that is able to parallel-process large datasets quickly and cost-effectively. The data team can trigger the ELT (Extract, Load, Transform) process via a restful API. The API creates the cloud based environment and services required to perform the load and transformation of data. Once each step in the transformation pipeline has been run any unnecessary compute resources are automatically destroyed.

To deliver a fantastic user experience we designed and developed a ReactJS web application. The application interfaces with a number of supporting micro-services including a GraphQL end point. The GraphQL service provides access to research questions and meta information without the need for a middleman API. The end result is a fast and easy to use interface for users and a secure and scalable data platform for the Foresight Factory team.

data analysis tool
Image module
Image module

HAVE A DATA OR ANALYTICS PROJECT IN MIND?

If you would like to talk to us about your data and business intelligence workflows please drop us a line using the form or call us on +44(0) 117 933 2595.

We provide fully managed cloud services for all your hosting and support needs.