The Signal and The Noise: Why So Many
Predictions Fail -- but Some Don't
By Nate Silver
Penguin Press, 534 pages
Can you go to jail for promising your client great
results?
Ask the Italian seismologists found guilty of
manslaughter for underestimating the risk of what turned out to be a
magnitude-6.3 earthquake that killed over 300 people. They're one of the
cases analyzed in The Signal and The Noise, a new book about successful and failed predictions. Advertising isn't (usually)
a life-or-death matter, but it is a major line item on your client's
P&L. You might not go to jail for a bad prediction, but you could get
fired. That's why I recommend this book: It helps us understand
the impact of so-called Big Data in marketing today.
The Signal and The Noise: Summary
In electrical engineering, “Signal to Noise" describes the relationship between signals that report a useful reading and
random noise that makes signals hard to identify. Similarly, as we drown in data, generating 2.5 quintillion bytes every day, it's harder to separate signal from noise. Compounding the problem is our own
subjectivity: human beings, more than any other species, are wired to see
patterns, and often in the data we see patterns that aren’t real. Worse,
we use those non-patterns to predict future events. The solution is to
embrace our subjectivity and test hypotheses, getting “closer and closer to the
truth as we gather more evidence.” Examples are drawn from pro baseball,
politics, earthquakes, economics, epidemics, gambling, global warming, and
terrorism. The author, Nate Silver, knows whereof he speaks. Years
ago he built a reliable tool forecasting baseball player performance, and later
gained wider fame for correctly predicting the state-by-state results of the
last two presidential elections.
True to the topic, Silver’s analyses are sincere and
(generally) objective. It’s not the type of book, however, so common on
the business shelf, that outlines 7 key findings or 10 ways to improve your
predictive power. In fact, buried on page 195 in one of the most hopeless
cases – economics forecasting, which will destroy whatever confidence you had
left in economists – are what I saw as his three keys to success: (1)
Improved computer power, (2) Better data collection, (3) Old-fashioned hard
work. Comically in a book that keeps reminding us that “to err is human”,
there are some unfortunate typos like this one on page 379, quoting a NASA
climate researcher: “I finally realized the definition of rocket science
is using relatively simple psychics to solve complex problems.”
Why Advertising people should read The Signal and The
Noise
The book is relevant to marketing today because we
have far more data than ever and, increasingly, the expectation that we can
predict results. If you think about it, our day-to-day decisions are
predictions about what will succeed. We launch that new product
(and hope it isn’t in the 90% that fail this year). We choose those
three animatics for test (and pray that one of them scores). We buy this
medium over another (and look for which half of the ad budget we wasted).
Silver points out that Prediction and Forecast are two
different things. A prediction is definitive, e.g., "this new product
will achieve $60 million in Year I sales." A forecast is probabilistic,
e.g., BASES may tell you Year I sales within a +/- 20% range. This once
frustrated a CPG CEO who didn’t realize how his brand managers were jacking up
the assumptions that went into the company’s BASES forecasts. Of note,
the U.S. Geological Survey explicitly states they can’t predict earthquakes –
they work hard (and fruitlessly, to hear Silver tell it) to forecast
earthquakes’ probability. (Small comfort to Italian seismologists.)
Likewise there’s a difference between Risk and
Uncertainty. Risk is something you can put a price on, a calculable
estimate. Ipsos/ASI may report a persuasion score as having an 80% or 95%
level of confidence. That means there is a 20% or 5% risk the copy won’t
be persuasive. Uncertainty is risk that is harder to measure.
Silver’s example is the gross miscalculation by credit ratings agencies as to
how risky collateralized debt obligations really were. (The chapter on
the 2008 financial meltdown, “A Catastrophic Failure of Prediction”, is worth a
read if only to understand that fiasco in 28 simple pages.)
Three Lessons for Marketing and Advertising
All data is not created equal. Silver admits
that some things are easier to predict than others. Baseball happens to
have a rich set of data, whereas predicting earthquakes is virtually impossible
because we can’t actually observe and record the subterranean shifting of tectonic plates. The
same lesson has historically separated direct response (did version A or version
B have a higher response rate?) from advertising (which half of the budget am I
wasting?).
Calibrate your crap detector. The book is a
treasure trove of ways we should not interpret data. You’ll cringe
at some of the mistakes – and realize you’ve made some of them yourself.
One of the more intriguing discussions is about “unknown unknowns” – what is it
we don’t see because we would never dream of it? Which leads to my last
point.
Use your imagination. We’re human and our
subjective POV is inevitable, so why not use it?
Silver’s personal template for prediction is called
Bayes’s Theorem. It’s essentially a way to apply the scientific
method: observe a phenomenon, develop a hypothesis to explain it,
formulate a prediction from the hypothesis, and test the prediction. To
be clear, this is not a left-brain analysis that a computer could
perform. It requires human imagination. Computers just help us
calculate the possibilities.
In other words: It’s up to us to distinguish signal from
noise.
