Book Review: Super Crunchers-The fight between experts, gut and data

I enjoyed reading Ian Ayres book. Let me say that again and right - I really enjoyed reading Ian Ayres book. For those who have not already read the book-it details how data driven number crunching algorithms work better than expert predictions and gut feelings and how super crunchers(read-statistically literate and number crunching savvy) individuals will have an edge in decision making in the future.

I liked the book because it lucidly illustrates trends that I have seen in the last decade-a better adoption of predictive models among businesses, more data generation and storage, an industry wide need for talented number crunchers and the conflict when data driven approaches come face to face with the resident expert or the manager who swears by his gut.

The case studies are very interesting and apt-it was amusing to read about the prediction of a vintage by an algorithm(I must pick up some wine based on the prediction soon). I could empathise with the story about a fellow economists frustration at waiting to get the final odds number on the Downs syndrome screening for his unborn child, and the inability of the technicians to apply the Bayes theorem(I've been there). As Ayres points out, I agree neural nets have a long way to go before they replace other mainstream techniques and it's not just due to the over fitting problem. Randomised trials still need to become mainstream among most marketers.

What really makes the book stand out is that data crunchers like me along with millions others 'get it'. I build predictive models that are elegant and simple and able to help clients make better decisions about their businesses. We constantly face sceptics about how predictive models can fare better than the resident experts knowledge of his market or brand or business. We sometimes pitch to client's who tell us their business problems cannot be put in an equation(it makes me squirm because I have a personal data project on which aims at predicting market prices for Indian contemporary art). After years in statistics, it's still difficult to help people understand standard deviation or 2SD.

Do I agree with the book's central premise-yes I do. In a data driven world, let numbers do the talking-stand aside experts and intuition.