Big data technology has been working in the background in a wide range of sectors for more than a decade. Practically all activities of our daily lives have some sort of a digital footprint, organisations both public and private, are finding ways to extract wealth out of this data gold mine.
Areas of application of big data projects span a wide range of industries, such as healthcare, financial services, transport, logistics, and customer service, being typical examples.
But what does big data mean? In essence, big data means large sets of data originating from different sources and processed in separate storage for the sole purpose of analysis. The ultimate goal is that of reaping valuable and actionable information from this data that translates into value.
Big data means large sets of data originating from different sources and processed in separate storage for the sole purpose of analysis
To visualise this, one might take a simplistic example of a supermarket chain. The chain managers use analytics on the purchase data of thousands of customers across hundreds of stores, possibly uncovering hidden buying patterns. In turn, actions such as product placement on shelves might result in more sales.
As we shall see shortly, much of the data is generated in real-time and on a vast scale, as dictated by the sources involved. These may be multinational e-commerce websites with thousands of transactions per minute, or social media platforms with millions of users, with an aggregate of billions of posts a day.
Over the years, the concept of big data has evolved to include more characteristics in addition to the original three it was defined earlier on, resulting in five, eight or even more. For the purpose of this article, however, we will focus on five characteristics, also known as ‘Vs’ or 5V model. The 5V big data model is often depicted as a pyramid with volume at its base and value at its apex.
The first V is Volume. As individuals as well as businesses use digital tools in their activities the amounts of data they generate risse exponentially. The data collected could easily add up to billions of data points a day.
The second V is Velocity. Information has to flow as quickly as possible. One can visualise the importance of velocity, for instance, when consulting a traffic map that uses big data to make real-time recommendations on alternative routes as the traffic situation on the road evolves in real-time. An app with even a few minutes’ delay would be totally useless.
The third V of big data is variety, referring to many types of data streams and relative sources. Depending on the area of activity, many different sources are correlated to yield a complete picture. For instance, some countries have adopted a mobile app to track and flag potential exposure to COVID-19 of individuals by combining medical status with GPS location and other data from their smartphone.
The fourth V is veracity, referring to the quality of data, in the sense that it conveys an accurate representation of the particular real-life context.
The fifth V, which stands for value, sits at the apex of the pyramid. Here the insights are converted to decisions that in turn translate to tangible benefits and value. In addition to business intelligence analytics, the marriage of big data with AI technologies come into play. These provide deviation detection, probability of future outcome and pattern recognition functionality on the large volumes of data sets otherwise simply not possible by human intervention.
If we take applications that are already in existence, as well as what we can expect in the near future in two sectors – financial services and healthcare – respectively, these would be the following:
Financial services were able to leverage information about their customers, including transaction data, banking records, credit histories, spending patterns and financial assets, to be armed in order to expand their businesses to build new targeted services based on the knowledge acquired from big data about their customers.
In healthcare, examples are treatment improvement and outbreak progression. In the first, it translates to more effective treatment at lower cost. The second, it could predict outbreaks of future pandemics and identify clusters to reduce the impact of disease and human suffering.
The trend shows we can expect more organisations to recognise the value of the data they generate and thus embark on projects geared towards leveraging it. The resulting demand for big data experts and associated fields can only be expected to grow.
This article was prepared by collating various publicly available online sources.