We are in an era where digitization has touched almost every aspect of life. Data is generated at a phenomenal rate, and all this data hides insightful information waiting to be discovered. For a business entity, turning business data into insightful information that can help strategic decision making will be a priceless boon.
Large organizations already have seen the benefits of big data:
The On Road Integrated Optimization and Navigation (ORION) big data project rolled out by UPS has helped it save 8.4 million gallons of fuel by reducing 85 million miles in delivery route kilometres in one year. The project, largest of its kind in the area of operations research, relies on telematics sensor data located on its fleet of 46,000 delivery vehicles.
Macy’s, by using big data analytics approaches, has managed to reduce the cycle time for pricing optimization for its 73 million item strong inventory from 27 hours to just over an hour. This helps them re-price items based on changing demands of the retail marketplace almost by the hour!
In 2011, Wikibon, an open source knowledge sharing community, published a case study that compared the financial return of two analytics environments: (a) a high-speed data warehouse appliance employing traditional ETL and data provisioning processes, and (b) A big data set-up running on massively-parallel (MPP) hardware, and here are the findings on various economic parameters:
|
Parameter |
Big Data |
ETL Data Warehouse |
| Cumulative 3 year cash flow | $ 152 M | $ 53 M |
| Net Present Value | $ 138 M | $ 46 M |
| Internal Rate of Return | 524% | 74% |
| Breakeven | 4 months | 26 months |
Big data is clearly helping organizations see tangible benefits.
Big Data is different from other data paradigms that we are familiar with, because it is characterized by three ‘V’s:
Volume, whichrefers to the sheer storage capacity required to hold all the data that is generated. For instance, the daily subscriber location details of a typical communications service provider with 100 million subscribers is in the range of 50 terabytes. For a 100 day period, this could easily exceed 5 petabytes. Call Detail Record (CDR) data for the same service provider would exceed 5 billion records a day!
Velocity refers to the sheer amount of data that travels through data communication channels. It is estimated that by 2016, global mobile data is expected to reach 10.8 exa-bytes (1018 bytes!) per month, thanks to the increasing extent of users sharing pictures, audio and video files through Over-The-Top (OTT), mobile applications.
Variety refers to the multiple formats in which data is generated. Data is not just the structured rows and columns of a database that we are familiar with; it could be as varied as social media data (unstructured text), audio/video data, call logs, and multiple other unstructured formats.
The fundamental difference between the Business Intelligence/Analytics paradigm that we are familiar with and the Big Data paradigm gaining momentum now lies in the way data is processed. Current Business Intelligence/Analytics infrastructure handles data in a ‘store-and-analyse’ approach. In a typical data warehouse for instance, transactional data that was typically a day old was stored, and used for extracting business insights. In the Big Data analytics approach, data is analysed in near real time for predicting the future.
How has this Big Data paradigm shift been made possible? It represents the coming together of two main factors:
(a) Increasing processing power of commercial grade computers
(b) Development of robust, fault tolerant, distributed computing algorithms such as MapReduce that enable processing of massive amounts of data in real time
Businesses cannot afford to ignore the Big Data Analytics paradigm, since there are almost no domains which are as yet untouched by this paradigm in this internet enabled, information era.
Does the Big Data Analytics paradigm stand for the benefit of only large businesses with deep pockets? Not necessarily. In fact, businesses need not even invest on setting up all the infrastructure themselves, and can make use of cloud based big data analytics applications that suit their business needs.
Looking to build a social media based 360 degree profile of potential consumers who intend to buy your product? Intending to build a credit card fraud detection engine that compares real-time mobile location data during transactions? All these, and more, are possible through the Big Data analytics paradigm.