baseball, book review, book reviews, books, fivethirtyeight, nate silver, non-fiction, PECOTA, prediction models, predictions, social science, The Signal and the Noise, What's Making Me Happy This Week
The Signal and the Noise, by Nate Silver
This is “What’s Making Me Happy This Week,” a weekly feature inspired by the Pop Culture Happy Hour podcast. It’s pretty self-explanatory.
What’s Making Me Happy This Week is a book that I’ve been reading for the past ten days or so and will likely finish tonight, The Signal and the Noise: Why So Many Predictions Fail – But Some Don’t, by Nate Silver. I have long been a fan of Silver’s, who made his name with his uncanny 2008 U.S. election predictions at his blog FiveThirtyEight (making Time Magazine‘s 2009 list of the World’s 100 Most Influential People in the process). Before that Silver was the creator of PECOTA, a system for forecasting the performance and career development of Major League Baseball players. PECOTA – which was sold almost immediately to Baseball Prospectus, which also hired Silver to be a writer, manage and further develop PECOTA, and eventually be Managing Partner – is a relatively simple system for projecting player performance. It looks at a player’s career performance to date, adjusted for context such as ballpark effects and other extrinsic variables, and predicts future performance based on what comparable players have done over the course of their careers. I say it is “simple” not because it is easy to compute, but because the theory behind the system is so intuitive: Baseball has been played for over 100 years and, though it has evolved, the basics of the game haven’t changed much over that time. For that reason, there are many thousands of player-seasons to fill a database, and only the most extreme outlier of a player (think late-career-peaking Barry Bonds) exhibits a career path that hasn’t been seen before many times over. Silver then couched all of his predictions in probabilistic, rather than absolute terms. Think of it this way: If player A’s career to date looks like a lot like like player’s B through Z’s career through the same age, player A’s next season is almost certain to fall somewhere in the range of the best to worst seasons next seen for players B through Z, and more likely than not somewhere in the middle of the range. It’s one of those things that is really obvious in hindsight, but until Silver thought of it and made his predictions, no one else was doing it. (Now there are many prediction models for baseball players (and athletes in other sports) based on the same methodology, some of which are actually better than PECOTA.)
After conquering the world of predicting baseball and political elections, what was next for the boy wonder? In part, it was writing a book that would teach the world to think (and make predictions) like he does. Baseball happens to be an arena where there is a lot of “signal” and not as much “noise”, and it is relatively easy to distinguish between the two. There is also a staggering amount of data – accurate data – and immediate ability and motivation to measure the accuracy of predictions. Baseball is like a perfect storm of factors contributing to such accurate predictions that some feel have even made the game boring. Political elections exhibit some but not all of these factors, as Silver illustrates in his book. (For example, there are certain pundits who do not have an incentive to make accurate predictions; they are better served having a low batting average but hitting some home runs (i.e. making bold predictions that are often wrong but are noticed by the public only when they turn out right).) This makes political predictions less accurate, though they don’t have to be. Predictions of global climate change fall into a similar category. There is ample data and clarity to the science to enable scientists to make reasonable predictions (albeit with a fair amount of uncertainty) about global climate change. However, due to extrinsic (mostly politically-driven) factors, the science isn’t actually getting much better.
There are other areas where predictions are not reliable at all, mainly due to an excess of “noise” and the unwillingness of those making the predictions to see the limitations. The book jacket sums this up well: “Most predictions fail, often at great cost to society, because most of us have a poor understanding of probability and uncertainty. Both experts and laypeople mistake more confident predictions for more accurate ones. But overconfidence is often the reason for failure. If our appreciation of uncertainty improves, our predictions can get better too.” Earthquake predictions suffer from this. As do macro-economic predictions. And many health-care predictions, such as the expected spread and effect of viruses. These are all the subject of major interest and study – not to mention major consequence – and yet our prediction models are actually getting worse with time, as more data and computational power becomes available, due to the mistaking of noise for signal and the bull-headed refusal of many to recognize it.
If the world is in such a dire place on so many levels, why is this book making me happy? For one thing, I learned a lot. I can speak intelligently now on subjects as diverse as baseball, global warming, the housing bubble, poker, hurricanes, bird flu and chess. Second, I learned how to recognize when something is not known or unknowable, and to distinguish confident predictions from accurate ones. Third, at a basic level, it was a great read on a subject that I love – statistics / social science. Silver is smart without being arrogant and writes on a level that hits the sweet spot between sophistication and accessibility. Finally, there is hope, even in areas where progress is not being made. Silver notes, when writing about his meeting with Donald Rumsfeld to speak about the attacks on Pearl Harbor, “Worse than being unprepared, we had mistaken our ignorance for knowledge and made ourselves more vulnerable as a result.” As long as people like Silver are around to think about things critically and calling out the bad actors in the prediction game, society should be able to reach the point where it knows exactly what it is that it doesn’t know (like how to accurately predict an earthquake or a terrorist attack). Sometimes, knowing just that much is the difference between being prepared for the worst and being overconfident that the worst can’t happen.
Bottom line: This is a great book for people who are smart and curious. I like to think I’m a little of both, and this is a book I’ll keep proudly and prominently on my bookshelf for a long time.
And that’s What’s Making Me Happy This Week.
PS Thanks to PECOTA and copy-cat prognosticators I won my fantasy baseball league two consecutive seasons and was the reigning champion at the time it disbanded. I’m telling you … this stuff works!
From → Books & Literature