Bigger is Not Necessarily Better

Mike Strange
Office Managing Vice President, Los Angeles Office

The term big data is popular, but perhaps creates the wrong connotation.  The value of big data is in the learnings that we can gain from investigation, not from the data itself, or even the tools.  And some of the most useful insight comes from targeted analysis, often with small amounts of information.  It is often true that patterns are more statistically relevant when considering large volumes of information, but volume is not the goal – insight is the goal. This is not always clear to our big data teams and we, as leaders, can help.

Pariveda has done a number of big data projects, and we see all shapes and sizes.  Some big data projects actually need small amounts of data to gain useful insight.  Others require massive amounts.  We recently completed two such projects, started with a hypothesis in both cases and, upon reflection, learned some valuable lessons.  In one case, we analyzed the effectiveness of marketing campaigns, correlating results with environmental and customer-oriented factors.  Using a shockingly small amount of “trial” data, we found several correlations – some obvious and a couple surprises.  We then leveraged machine learning concepts to demonstrate that it is possible to predict the impact of a marketing campaign within a few percentage points.  This is a huge finding – determined by prototyping a tightly-defined goal using a small subset of data.  Once you prove a concept like that, you can expand the volume of data – incorporating even more interesting correlations such as seasonality or longer-term economic cycles.

We often see large data stores, sometimes called data warehouses, which have all the hallmarks of a hard-to-watch episode of Hoarders – large amounts of overlapping, loosely-organized information collected for the purpose of collection.  Collecting was the goal – organization, analysis and output are envisioned as future extensions.  Some organizations spend years on “data in” and never get to “data out”.  Today’s expectations of rapid innovation demand more.

I think the lesson is this.  Many big data programs would benefit from defined targets -- value propositions – which establish the guideposts of progress.  In the example above, we sought to predict effectiveness through correlation of patterns in marketing programs.  Other big data efforts could focus on personalization by identifying patterns of consumer behavior, assessing cannibalization, determining price elasticity, assessing distribution channels, or attacking the elusive concept of determining consumer identity.  In any of these examples, it is more important to focus on the goal, and not the BIGness of your big data program.

On the other hand, depending on the situation, it may be constraining to apply the level of focus described above.  Sometimes we must collect, correlate, aggregate and analyze large amounts of data – applying research-style thinking to the investigation.  Sometimes we don’t know exactly what we are looking for.  For example, if we primarily applied goal-oriented thinking to medical research, we may never have discovered Penicillin.  But, in a commercial setting, we are often bounded by the realities of time, money and people.  In these settings, it may be more effective to show incremental progress through a more goal-oriented approach.

It actually sounds funny to say “we are driving a big data initiative, and intentionally starting small” – but sometimes that is the right way.

 
Mike Strange

About the Author

Mike Strange Office Managing Vice President, Los Angeles Office
Mr. Mike Strange is a passionate technology leader, with 20+ years of IT architecture and IT leadership experience. He has served in numerous IT leadership roles, including CTO and Senior Vice President. He is passionate about bringing clarity to the design and development of modern architectures and systems, and speaks at conferences regularly regarding IT trends. Mike is Vice President in our Los Angeles office.

Perspective
By Collins DeLoach
What does Cloud Transformation mean it IT?
Perspective
By Tim Hurst
Unlocking Marketing ROI Analytics
Perspective
By Sean McCall
The Missing Link in Advanced Analytics
Perspective
By Derrick Bowen
Stop Complaining About Changing Requirements
Perspective
by Phillip Manwaring
Let Serverless Solve the Tech Problems You Don't Have
Perspective
by Adrian Kosiak
Lessons from Dior on Becoming a Premium Brand
Perspective
By Marc Helberg
How AI Will Affect Your Patient’s Experience
Perspective
By Jeff Townes and Sean Beard
If You're Adopting the Cloud in Your Organization, You Don't have to Start from Scratch
Perspective
By Alan Henson
Artificial Intelligence Might Be Your Toughest Employee to Manage
Perspective
by Clayton Rothschild
Blockchain in an Enterprise Setting
Perspective
by Brian Duperrouzel
Hippos and Cows are Stopping Your Machine Learning Projects
Perspective
by Dave Winterhalter
Data in the Dugout
Perspective
by Mike Strange
Bigger is Not Necessarily Better
Perspective
By Ryan Gross
What Does It Really Mean to Operationalize a Predictive Model
Perspective
By Ryan McFarland
Machine Learning, Beyond the Stars!
Perspective
By Mike Strange
Untangling and Understanding DevOps
Perspective
by Mike Strange
DevOps a Practical Framework for Technology Process and Organizational Change
Perspective
by Julio Santos
Context as Currency
Perspective
by Oussama Housini
Why DevOps?
Perspective
By Jack Warner
Building Smart Deployment Pipeline
Perspective
By Marc Helberg
3 Ways You Can Begin to Take Patient Experience More Seriously
Perspective
By Margaret Rogers
Identifying the Value of Nonprofit Customer Experience
Perspective
By Margaret Rogers
Why an Agile Mindset is at the Root of an Excellent Guest Experience
Perspective
<p>Dimitrios Vilaetis and Sara Harshbarger</p>
Holistic Value Hunting
Perspective
by C<span>&eacute;</span>sar Giralt
Building your Alumni Network into a Value Creation Platform
Perspective
by Russell Clarkson
Mark Your Exits
Perspective
By Russell Clarkson
Is your Ecosystem a pipeline or a platform?
Perspective
by Susan Paul
Capabilities as Building Blocks
Perspective
By Mike Strange
4 Reasons to Leverage the Power of Small Teams
Perspective
By Susan Paul
When Corporate Culture Undermines IT Success
Perspective
By Alexandria Johnson
The Hottest Thing at SXSW You Learned it in Kindergarten
Perspective
By Bruce Ballengee
Developing the Individual
Perspective
by David Watson
Work Life Balance
Perspective
by Lori Dipprey
Why Performance Review are Here to Stay at Pariveda
Perspective
by Kerry Stover
I Know You Believe You Understood What You Think I Said
Perspective
by David Watson
4 Lessons Learned from Questions
Perspective
By Russell Clarkson
Stand Up for Good Presentations
Perspective
By Margaret Rogers
Failing Fast Is Fine — As Long As You’re Failing Well, Too
Perspective
By Carrie Beckner
Generation Z’s Members Are Entering the Workforce. Here’s How to Welcome Them.
Load More