Can You Reduce Big Data To A Simple Yes Or No?

Assuming you understand the question well, a simple yes or no answer isn’t very likely.

This is especially true in business where data analysis questions usually result in stories, not answers. The answers are derived from these stories by intelligent people. It’s the subject matter experts who can get at the answers.

There is confusion about data analysis. Often, people think an analytical system provides answers, which it does not.

All the time, effort, and money that go into designing, implementing, and using data analysis systems are only to make an order of things so that it’s easier for the experts to figure things out.

Graphs, charts, spreadsheets or what have you cannot tell you what to do. Ironically, I’ve been in meetings where people seemed almost frustrated with this fact.

The old adage that a tool is only as good as its user very much applies to data analysis systems such as what we call business intelligence, or big data, or predictive analytics, or something else.

It must be pretty obvious that big data cannot be reduced to a simple yes or no as the title of this article posits. But I chose the obvious to point out its absurdity. Big data doesn’t contain answers – you do, and it’s not a yes or no.

What you do with big data is as much an art form as it is science. If you can come up with a simple yes or no after analyzing heaps of information then you are a magician.

Recently, there’s been some chatter about artificial intelligence ”doing the thinking” for us. That’s another utopia. Even if such a thing were possible, why would we want to leave ourselves out of any part of the thought process?

The point of this article is that clients must be educated about what data can and cannot do. For example, data cannot answer questions or provide directions, only people can, and should, do that. However, data can enable people to answer questions better, easier, faster, etc.

It may again sound like I am pointing out the obvious here, but I was surprised at how many clients had unreasonable expectations of data analysis systems. Questions like ”why can’t it tell me if I should suspend sales of X to avoid product cannibalization?” came up in some meetings as well as other queries of similar nature where the misunderstanding was that ”it” somehow spits out ”what to do” instructions.

Implementations run a risk of failure if the users’ perception of the system is that ”it” doesn’t give them what they want, even if unreasonably so. This falls into expectation management and education, and it ought to be done right off the bat so that the title of this article doesn’t one day become a question in the meeting room.

Competence & Honesty Is Hard Work

Hard as it may be to master data analysis, honest unbiased reporting is even more difficult.

We all make mistakes, and quite a few of us give in to the temptation to mislead others for personal gain. While mistakes are par for the course, deliberate manipulation is not.

Charts are one of the most common data visualization choices. It’s not uncommon for people to “distort” reality either by mistake or intentionally when presenting bar, line or other charts. In my career, I saw people spot and correct charting mistakes and make “clever” misrepresentations designed to help someone get ahead.

Labeling of X and Y axes can be misleading. The context of what’s being presented can be made unclear. Cherry picking is a common practice for making things look better than they really are. Misusing charting conventions can “trick” people into seeing something that isn’t there such as a pie chart seemingly showing a favorable picture overall when, really, the chart is focused only on a small subset without an accompanying explanation.

This brings me to the point I am trying to make – be extremely vigilant with analytics. It’s just as dangerous as it is helpful. If you know what you are doing or if you are not very good at it, the result can be a disaster just the same.


Is a chart lying to you? This video has some tips to figure it out. – Vox