Can You Reduce Big Data To A Simple Yes Or No?

Assuming you understand the question well, a simple yes or no answer isn’t very likely.

This is especially true in business where data analysis questions usually result in stories, not answers. The answers are derived from these stories by intelligent people. It’s the subject matter experts who can get at the answers.

There is confusion about data analysis. Often, people think an analytical system provides answers, which it does not.

All the time, effort, and money that go into designing, implementing, and using data analysis systems are only to make an order of things so that it’s easier for the experts to figure things out.

Graphs, charts, spreadsheets or what have you cannot tell you what to do. Ironically, I’ve been in meetings where people seemed almost frustrated with this fact.

The old adage that a tool is only as good as its user very much applies to data analysis systems such as what we call business intelligence, or big data, or predictive analytics, or something else.

It must be pretty obvious that big data cannot be reduced to a simple yes or no as the title of this article posits. But I chose the obvious to point out its absurdity. Big data doesn’t contain answers – you do, and it’s not a yes or no.

What you do with big data is as much an art form as it is science. If you can come up with a simple yes or no after analyzing heaps of information then you are a magician.

Recently, there’s been some chatter about artificial intelligence ”doing the thinking” for us. That’s another utopia. Even if such a thing were possible, why would we want to leave ourselves out of any part of the thought process?

The point of this article is that clients must be educated about what data can and cannot do. For example, data cannot answer questions or provide directions, only people can, and should, do that. However, data can enable people to answer questions better, easier, faster, etc.

It may again sound like I am pointing out the obvious here, but I was surprised at how many clients had unreasonable expectations of data analysis systems. Questions like ”why can’t it tell me if I should suspend sales of X to avoid product cannibalization?” came up in some meetings as well as other queries of similar nature where the misunderstanding was that ”it” somehow spits out ”what to do” instructions.

Implementations run a risk of failure if the users’ perception of the system is that ”it” doesn’t give them what they want, even if unreasonably so. This falls into expectation management and education, and it ought to be done right off the bat so that the title of this article doesn’t one day become a question in the meeting room.

Predictive Enough?

Back in early 2000s, Business Intelligence professionals were talking about the novelty of automated cross advertizing. An example could be advertizing greeting cards to someone who just purchased a gift, or vice versa. If you bought a present then you must need a greeting card, so how about some of the choices right there, presented to you before or after your check out. While this wasn’t predictive analytics, it was still based on past shopping behavior. Market baskets are a more sophisticated version of shopping behavior analysis.

Modern predictive analytics is more comprehensive – it is analysis of a pattern across a time span aimed at predicting what you are likely to do or need next. There is, probably, an infinite number of ways to analyze various data to try to predict human behavior – past shopping data, weather pattern data, seasonal data, income level data, credit score data. Creative analysis of combinations of these data and many more factors can uncover what’s about to happen before it happens.

However, many contemporary predictive analysis examples still amount to little more than simple cross advertizing that can, potentially, be achieved without the associated expense. It’s hardly necessary to go through extensive data analysis to “predict” that I am likely to need a greeting card to go with that gift I just purchased. It would be wise to understand the real business need for analytics. If the goal is to develop a competitive edge in the modern marketplace, then predictive analytics is likely the way to go, and it is a journey that takes methodology and patience.


Predictive analytics and advertising: the science behind knowing what women (and men) want

Competence & Honesty Is Hard Work

Hard as it may be to master data analysis, honest unbiased reporting is even more difficult.

We all make mistakes, and quite a few of us give in to the temptation to mislead others for personal gain. While mistakes are par for the course, deliberate manipulation is not.

Charts are one of the most common data visualization choices. It’s not uncommon for people to “distort” reality either by mistake or intentionally when presenting bar, line or other charts. In my career, I saw people spot and correct charting mistakes and make “clever” misrepresentations designed to help someone get ahead.

Labeling of X and Y axes can be misleading. The context of what’s being presented can be made unclear. Cherry picking is a common practice for making things look better than they really are. Misusing charting conventions can “trick” people into seeing something that isn’t there such as a pie chart seemingly showing a favorable picture overall when, really, the chart is focused only on a small subset without an accompanying explanation.

This brings me to the point I am trying to make – be extremely vigilant with analytics. It’s just as dangerous as it is helpful. If you know what you are doing or if you are not very good at it, the result can be a disaster just the same.


Is a chart lying to you? This video has some tips to figure it out. – Vox

Emotion is Also a Data Point

We must strike a balance between relying on data and emotion when making decisions. A glaring example of this is our last Presidential election here in America. Clinton relied on data and lost. Trump played on people’s emotions and won.


Tools for predictive analytics took a hit in presidential election