Big Data: The Big Misunderstanding

Just one moment is all that it takes for many of us to make some of life’s biggest decisions. Good or bad, from the point we make those decisions we, and those connected to our decisions, will have to live with them until new choices are made. So how does the presence of Big Data, coupled with tribal knowledge, affect our ability to make decisions, or even understand the impact of the choices we make? I recently attended the IEG Advanced Analytics Retail Forum held in Boston, where the unofficial goal was to get analysts, who usually are found in very small groups, to come together and share insight and knowledge gained from their experience with parent organizations.

What did we find? The retail space is one of the ultimate decision making laboratories in the world. Every day, billions of us make certain purchasing decisions, and retailers want to know why and how those purchases ended up in our shopping carts and what would complement them. Much of what we buy feels like a necessity by nature; it seems marketing has managed to increase the conversion rate by overtaking our instinctual reactions. As companies devote more resources to capturing data and creating business intelligence, the surreal quality of understanding what it all means becomes clouded. We come up with names to describe what we see in the data, such as “lumpy,” “fat tails,” “bimodal,” “cluster,” and try to base our conclusions on the resultant scatter plots – only to discover that all we really need is one good data point to make our case.

Big Datum is going to result in Big Understanding much more quickly and effectively than Big Data ever will. It is not that Big Data can’t be used to predict the future; it is just that the possible distribution of these results from these predictions is much narrower than what is really possible in the future. In many ways, Big Data and the accompanying analytics has become the drug of choice for those looking for justification in supporting their business cause. Most of the resulting actions will destroy value while a few, by almost luck, will create it – consider the success rate for new ventures after five to ten years.

Business Intelligence managers will have their work cut out for them as they treat data blindness, remove biases, begin real discovery, and start circulating it around the business. This last point is key to the future of Big Understanding. Too often, the access and ability to work with Big Data is held by a very narrow range of people within an organization. Impeccable data analysis notwithstanding, putting answers into the hands of your company’s members is the final, often neglected piece of an exceptional business intelligence program.

16082 views