Bigger is Not Necessarily Better

Mike Strange
Office Managing Vice President, Los Angeles Office

The term big data is popular, but perhaps creates the wrong connotation.  The value of big data is in the learnings that we can gain from investigation, not from the data itself, or even the tools.  And some of the most useful insight comes from targeted analysis, often with small amounts of information.  It is often true that patterns are more statistically relevant when considering large volumes of information, but volume is not the goal – insight is the goal. This is not always clear to our big data teams and we, as leaders, can help.

Pariveda has done a number of big data projects, and we see all shapes and sizes.  Some big data projects actually need small amounts of data to gain useful insight.  Others require massive amounts.  We recently completed two such projects, started with a hypothesis in both cases and, upon reflection, learned some valuable lessons.  In one case, we analyzed the effectiveness of marketing campaigns, correlating results with environmental and customer-oriented factors.  Using a shockingly small amount of “trial” data, we found several correlations – some obvious and a couple surprises.  We then leveraged machine learning concepts to demonstrate that it is possible to predict the impact of a marketing campaign within a few percentage points.  This is a huge finding – determined by prototyping a tightly-defined goal using a small subset of data.  Once you prove a concept like that, you can expand the volume of data – incorporating even more interesting correlations such as seasonality or longer-term economic cycles.

We often see large data stores, sometimes called data warehouses, which have all the hallmarks of a hard-to-watch episode of Hoarders – large amounts of overlapping, loosely-organized information collected for the purpose of collection.  Collecting was the goal – organization, analysis and output are envisioned as future extensions.  Some organizations spend years on “data in” and never get to “data out”.  Today’s expectations of rapid innovation demand more.

I think the lesson is this.  Many big data programs would benefit from defined targets -- value propositions – which establish the guideposts of progress.  In the example above, we sought to predict effectiveness through correlation of patterns in marketing programs.  Other big data efforts could focus on personalization by identifying patterns of consumer behavior, assessing cannibalization, determining price elasticity, assessing distribution channels, or attacking the elusive concept of determining consumer identity.  In any of these examples, it is more important to focus on the goal, and not the BIGness of your big data program.

On the other hand, depending on the situation, it may be constraining to apply the level of focus described above.  Sometimes we must collect, correlate, aggregate and analyze large amounts of data – applying research-style thinking to the investigation.  Sometimes we don’t know exactly what we are looking for.  For example, if we primarily applied goal-oriented thinking to medical research, we may never have discovered Penicillin.  But, in a commercial setting, we are often bounded by the realities of time, money and people.  In these settings, it may be more effective to show incremental progress through a more goal-oriented approach.

It actually sounds funny to say “we are driving a big data initiative, and intentionally starting small” – but sometimes that is the right way.

 
Mike Strange

About the Author

Mike Strange Office Managing Vice President, Los Angeles Office
Mr. Mike Strange is a passionate technology leader, with 20+ years of IT architecture and IT leadership experience. He has served in numerous IT leadership roles, including CTO and Senior Vice President. He is passionate about bringing clarity to the design and development of modern architectures and systems, and speaks at conferences regularly regarding IT trends. Mike is Vice President in our Los Angeles office.

More Perspectives

Perspective
by Bruce Ballengee
Teaching Roots Run Deep
Perspective
by Kent Norman and Kevin Moormann
I’ll be Home in Five Story-points
Perspective
By Russell Clarkson
Is your Ecosystem a pipeline or a platform?
Perspective
by Dimitrios Vilaetis
Business Capabilities: a Spotlight for Strategic Decision Making
Perspective
by Russell Clarkson
Mark Your Exits
Perspective
by Bruce Ballengee
Leadership and Company Culture
Perspective
by Kerry Stover
I Know You Believe You Understood What You Think I Said
Perspective
by Brian Duperrouzel
How to Handle Management Kidnapping
Perspective
By Russell Clarkson
What Chuck Yeager Taught Me About Disruption
Perspective
by Brian Duperrouzel
Lessons on Project and Team Management
Perspective
by Bruce Ballengee
Consulting that Focuses on the Individual
Perspective
By Bruce Ballengee
Developing the Individual
Perspective
by Lori Dipprey
Why Performance Reviews are Here to Stay at Pariveda
Perspective
By Nathan Hanks
What it Means to be "in the People Business"
Perspective
By Bruce Ballengee
Pariveda Fin-Centric Design
Perspective
By Bruce Ballengee
Unleashing the Power of Humility
Perspective
By Mike Strange
4 Reasons to Leverage the Power of Small Teams
Perspective
By David Watson
The Benefits of Working in Teams
Perspective
by Sean McCall
Forget Coffee- Energize Your Work Morning
Perspective
By Allison Esenkova
Wearing Heels to the Top
Perspective
by David Watson
Work-Life Balance
Perspective
by Allison Esenkova
Listen to the Still Small Voice
Perspective
By Kerry Stover
Must Read Books from the COO
Perspective
By Russell Clarkson
Why You Should take a Massive Open Online Course
Perspective
By Scott Hajer
The Importance of Making Space to Think
Perspective
by Sean McCall
4 Ways Sports Can Benefit Careers
Perspective
By Russell Clarkson
Stand Up for Good Presentations
Perspective
by Derrick Bowen
How Much of your Project Value is At Risk Due to Cognitive Bias
Perspective
by Susan Paul
Capabilities as Building Blocks
Perspective
By Susan Paul
When Corporate Culture Undermines IT Success
Perspective
by Kent Norman
Limit, to Win It
Perspective
By Brian Duperrouzel
The Case for Onshore Technology Teams
Perspective
By Tim Hurst
Unlocking Marketing ROI Analytics
Perspective
by Brian Duperrouzel
Hippos and Cows are Stopping Your Machine Learning Projects
Perspective
By Jack Warner
Building Smart Deployment Pipeline
Perspective
By Tom Cunningham
The Sound of the Future
Perspective
By Scott Hajer
You Need Only One Kind of Recruiting Technology
Point of View
by Ansley Galjour
Continuous Brand Engagement
Perspective
By Marc Helberg
3 Ways You Can Begin to Take Patient Experience More Seriously
Perspective
By Ryan Gross
What Does It Really Mean to Operationalize a Predictive Model
Perspective
by Sophie Zugnoni
Did You Catch Machine Learning Fever?
Perspective
By Collins DeLoach
What does Cloud Transformation mean it IT?
Perspective
By Mike Strange
Untangling and Understanding DevOps
Perspective
by Clayton Rothschild
Blockchain in an Enterprise Setting
Perspective
by Mike Strange
DevOps a Practical Framework for Technology Process and Organizational Change
Perspective
by Julio Santos - Finity Member
Context as Currency
Perspective
by Oussama Housini
Why DevOps?
Perspective
by Dave Winterhalter
Data in the Dugout
Perspective
by Brian Duperrouzell
3 Tips to get the Blockchain Ball Rolling
Perspective
by Victor Diaz
6 Things to Consider when Choosing a Data Visualization Library
Perspective
By Julio Santos and Jon Landers
How Customer Context and Smarter Algorithms will Power the Next Level of Experiences and Engagement
Perspective
by Mike Strange
Can We Predict the Future?
Perspective
By Brian Duperrouzel
No Estimates, No Approval, No Dice
Perspective
By Marc Helberg
How AI Will Affect Your Patient’s Experience
Perspective
By Sean Beard and Brian Orrell
Life After Mobile
Perspective
by Phillip Manwaring
Let Serverless Solve the Tech Problems You Don't Have
Perspective
by Mike Strange
Bigger is Not Necessarily Better
Perspective
By Marc Helberg
Patient Experience – Taking the Next Step Through Machine Learning
Load More