09 January 2013

Book Review: The Signal and The Noise


The Signal and The Noise:  Why So Many Predictions Fail -- but Some Don't
By Nate Silver
Penguin Press, 534 pages


Can you go to jail for promising your client great results?

Ask the Italian seismologists found guilty of manslaughter for underestimating the risk of what turned out to be a magnitude-6.3 earthquake that killed over 300 people.  They're one of the cases analyzed in The Signal and The Noise, a new book about successful and failed predictions.  Advertising isn't (usually) a life-or-death matter, but it is a major line item on your client's P&L.  You might not go to jail for a bad prediction, but you could get fired.  That's why I recommend this book:  It helps us understand the impact of so-called Big Data in marketing today.

The Signal and The Noise: Summary

In electrical engineering, “Signal to Noise" describes the relationship between signals that report a useful reading and random noise that makes signals hard to identify.  Similarly, as we drown in data, generating 2.5 quintillion bytes every day, it's harder to separate signal from noise.  Compounding the problem is our own subjectivity:  human beings, more than any other species, are wired to see patterns, and often in the data we see patterns that aren’t real.  Worse, we use those non-patterns to predict future events.  The solution is to embrace our subjectivity and test hypotheses, getting “closer and closer to the truth as we gather more evidence.”  Examples are drawn from pro baseball, politics, earthquakes, economics, epidemics, gambling, global warming, and terrorism.  The author, Nate Silver, knows whereof he speaks.  Years ago he built a reliable tool forecasting baseball player performance, and later gained wider fame for correctly predicting the state-by-state results of the last two presidential elections.

True to the topic, Silver’s analyses are sincere and (generally) objective.  It’s not the type of book, however, so common on the business shelf, that outlines 7 key findings or 10 ways to improve your predictive power.  In fact, buried on page 195 in one of the most hopeless cases – economics forecasting, which will destroy whatever confidence you had left in economists – are what I saw as his three keys to success:  (1) Improved computer power, (2) Better data collection, (3) Old-fashioned hard work.  Comically in a book that keeps reminding us that “to err is human”, there are some unfortunate typos like this one on page 379, quoting a NASA climate researcher:  “I finally realized the definition of rocket science is using relatively simple psychics to solve complex problems.”

Why Advertising people should read The Signal and The Noise

The book is relevant to marketing today because we have far more data than ever and, increasingly, the expectation that we can predict results.  If you think about it, our day-to-day decisions are predictions about what will succeed.  We launch that new product (and hope it isn’t in the 90% that fail this year).  We choose those three animatics for test (and pray that one of them scores).  We buy this medium over another (and look for which half of the ad budget we wasted). 

Silver points out that Prediction and Forecast are two different things.  A prediction is definitive, e.g., "this new product will achieve $60 million in Year I sales."  A forecast is probabilistic, e.g., BASES may tell you Year I sales within a +/- 20% range.  This once frustrated a CPG CEO who didn’t realize how his brand managers were jacking up the assumptions that went into the company’s BASES forecasts.  Of note, the U.S. Geological Survey explicitly states they can’t predict earthquakes – they work hard (and fruitlessly, to hear Silver tell it) to forecast earthquakes’ probability.  (Small comfort to Italian seismologists.)

Likewise there’s a difference between Risk and Uncertainty.  Risk is something you can put a price on, a calculable estimate.  Ipsos/ASI may report a persuasion score as having an 80% or 95% level of confidence.  That means there is a 20% or 5% risk the copy won’t be persuasive.  Uncertainty is risk that is harder to measure.  Silver’s example is the gross miscalculation by credit ratings agencies as to how risky collateralized debt obligations really were.  (The chapter on the 2008 financial meltdown, “A Catastrophic Failure of Prediction”, is worth a read if only to understand that fiasco in 28 simple pages.)

Three Lessons for Marketing and Advertising

All data is not created equal.  Silver admits that some things are easier to predict than others.  Baseball happens to have a rich set of data, whereas predicting earthquakes is virtually impossible because we can’t actually observe and record the subterranean shifting of tectonic plates.  The same lesson has historically separated direct response (did version A or version B have a higher response rate?) from advertising (which half of the budget am I wasting?).

Calibrate your crap detector.  The book is a treasure trove of ways we should not interpret data.  You’ll cringe at some of the mistakes – and realize you’ve made some of them yourself.  One of the more intriguing discussions is about “unknown unknowns” – what is it we don’t see because we would never dream of it?  Which leads to my last point.

Use your imagination.  We’re human and our subjective POV is inevitable, so why not use it? 

Silver’s personal template for prediction is called Bayes’s Theorem.  It’s essentially a way to apply the scientific method:  observe a phenomenon, develop a hypothesis to explain it, formulate a prediction from the hypothesis, and test the prediction.  To be clear, this is not a left-brain analysis that a computer could perform.  It requires human imagination.  Computers just help us calculate the possibilities.  

In other words:  It’s up to us to distinguish signal from noise.