Data is a useless waste of storage capacity until it is turned into something useful. Yet the amount of data being collected and stored is quickly outstripping the ability of conventional storage and compute resources to actually make sense out of that data.
Now what? That's where the concept of big data comes in. Big data involves gleaning business intelligence from unstructured data, which often scales to multiple petabytes of capacity. Big data is not a simple query from a user that can be answered with a quick peek at a static database. Instead, because so much data is collected in real time and can provide information for decision-makers in real time, big data requires a whole new set of business intelligence tools that are only now coming to market.
The McKinsey Global Institute estimated that a retailer with the right big data tools could increase operating margins by more than 60 percent, and that the U.S. health-care industry could create more than $300 billion in value annually with big data.
Such savings would result from businesses being better able to unlock value by making information transparent and usable at higher frequencies, collecting more accurate and detailed information on everything from inventories to sick days, tailoring products and services to more narrow customer segments, and improving decision-making, according to the McKinsey Global Institute.
For example, imagine a retail store that continually collects sales, inventory, price and customer traffic data to dynamically adjust pricing and order more stock in real time. Now imagine a retail chain doing that for hundreds or thousands of locations around the country or around the world -- that's big data.
Technology for handling big data tasks is becoming available, most notably in the form of an open-source project named Hadoop that is increasingly being adopted by cloud technology and storage technology vendors.
But having technology to work with big data and having people able to actually make sense of the information gleaned from the technology are two different things.
There is a severe shortage of IT personnel with the ability to do the business intelligence functions related to big data. The McKinsey Global Institute estimated that, by 2018, the U.S. will need 440,000 to 490,000 people with deep analytical training, but even with immigration will be lucky to have 300,000 people with the right skills. That doesn't include the 1.5 million data-savvy managers businesses will also require to take advantage of all that big data.
