17 results back to index
airport security, availability heuristic, Bayesian statistics, Benoit Mandelbrot, Berlin Wall, Bernie Madoff, big-box store, Black Swan, Broken windows theory, Carmen Reinhart, Claude Shannon: information theory, Climategate, Climatic Research Unit, cognitive dissonance, collapse of Lehman Brothers, collateralized debt obligation, complexity theory, computer age, correlation does not imply causation, Credit Default Swap, credit default swaps / collateralized debt obligations, cuban missile crisis, Daniel Kahneman / Amos Tversky, diversification, Donald Trump, Edmond Halley, Edward Lorenz: Chaos theory, en.wikipedia.org, equity premium, Eugene Fama: efficient market hypothesis, everywhere but in the productivity statistics, fear of failure, Fellow of the Royal Society, Freestyle chess, fudge factor, George Akerlof, haute cuisine, Henri Poincaré, high batting average, housing crisis, income per capita, index fund, Intergovernmental Panel on Climate Change (IPCC), Internet Archive, invention of the printing press, invisible hand, Isaac Newton, James Watt: steam engine, John Nash: game theory, John von Neumann, Kenneth Rogoff, knowledge economy, locking in a profit, Loma Prieta earthquake, market bubble, Mikhail Gorbachev, Moneyball by Michael Lewis explains big data, Monroe Doctrine, mortgage debt, Nate Silver, negative equity, new economy, Norbert Wiener, PageRank, pattern recognition, pets.com, Pierre-Simon Laplace, prediction markets, Productivity paradox, random walk, Richard Thaler, Robert Shiller, Robert Shiller, Rodney Brooks, Ronald Reagan, Saturday Night Live, savings glut, security theater, short selling, Skype, statistical model, Steven Pinker, The Great Moderation, The Market for Lemons, the scientific method, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, Thomas Bayes, Thomas Kuhn: the structure of scientific revolutions, too big to fail, transaction costs, transfer pricing, University of East Anglia, Watson beat the top human players on Jeopardy!, wikimedia commons
Stephen’s Green, Dublin 2, Ireland (a division of Penguin Books Ltd) • Penguin Books Australia Ltd, 250 Camberwell Road, Camberwell, Victoria 3124, Australia (a division of Pearson Australia Group Pty Ltd) • Penguin Books India Pvt Ltd, 11 Community Centre, Panchsheel Park, New Delhi – 110 017, India • Penguin Group (NZ), 67 Apollo Drive, Rosedale, Auckland 0632, New Zealand (a division of Pearson New Zealand Ltd) • Penguin Books (South Africa) (Pty) Ltd, 24 Sturdee Avenue, Rosebank, Johannesburg 2196, South Africa Penguin Books Ltd, Registered Offices: 80 Strand, London WC2R 0RL, England First published in 2012 by The Penguin Press, a member of Penguin Group (USA) Inc. Copyright © Nate Silver, 2012 All rights reserved Illustration credits Figure 4-2: Courtesy of Dr. Tim Parker, University of Oxford Figure 7-1: From “1918 Influenza: The Mother of All Pandemics” by Jeffery Taubenberger and David Morens, Emerging Infectious Disease Journal, vol. 12, no. 1, January 2006, Centers for Disease Control and Prevention Figures 9-2, 9-3A, 9-3C, 9-4, 9-5, 9-6 and 9-7: By Cburnett, Wikimedia Commons Figure 12-2: Courtesy of Dr. J. Scott Armstrong, The Wharton School, University of Pennsylvania LIBRARY OF CONGRESS CATALOGING IN PUBLICATION DATA Silver, Nate. The signal and the noise : why most predictions fail but some don’t / Nate Silver. p. cm. Includes bibliographical references and index.
“Election Results: House Big Board,” New York Times, November 2, 2010. http://elections.nytimes.com/2010/results/house/big-board. 26. Nate Silver, “A Warning on the Accuracy of Primary Polls,” FiveThirtyEight, New York Times, March 1, 2012. http://fivethirtyeight.blogs.nytimes.com/2012/03/01/a-warning-on-the-accuracy-of-primary-polls/. 27. Nate Silver, “Bill Buckner Strikes Again,” FiveThirtyEight, New York Times; September 29, 2011. http://fivethirtyeight.blogs.nytimes.com/2011/09/29/bill-buckner-strikes-again/. 28. Otherwise, you should have assigned the congressman a 100 percent chance of victory instead. 29. Matthew Dickinson, “Nate Silver Is Not a Political Scientist,” in Presidential Power: A Nonpartisan Analysis of Presidential Power, Blogs Dot Middlebury, November 1, 2010. http://blogs.middlebury.edu/presidentialpower/2010/11/01/nate-silver-is-not-a-political-scientist/. 30.
Alan Schwarz, “The Great Debate,” Baseball America, January. 7, 2005. http://www.baseballamerica.com/today/features/050107debate.html. 20. Per interview with Billy Beane. 21. Nate Silver, “What Tim Geithner Can Learn from Baseball,” Esquire, March 11, 2009. http://www.esquire.com/features/data/mlb-player-salaries-0409. 22. As a result of my original agreement in 2003 and a subsequent agreement in 2009, Baseball Prospectus now fully owns and operates PECOTA. Beginning with the 2010 season, the PECOTA forecasts reflect certain changes, improvements, and departures from my original methodology. The methods I describe herein apply to the 2003–2009 version of PECOTA specifically. 23. Nate Silver, “PECOTA Takes on the Field,” Baseball Prospectus, January 16, 2004. http://www.baseballprospectus.com/article.php?articleid=2515. 24. Nate Silver, “Lies, Damned Lies: Projection Reflection,” Baseball Prospectus, October 11, 2006. http://www.baseballprospectus.com/article.php?
Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are by Seth Stephens-Davidowitz
affirmative action, AltaVista, Amazon Mechanical Turk, Asian financial crisis, Bernie Sanders, big data - Walmart - Pop Tarts, Cass Sunstein, computer vision, correlation does not imply causation, crowdsourcing, Daniel Kahneman / Amos Tversky, desegregation, Donald Trump, Edward Glaeser, Filter Bubble, game design, happiness index / gross national happiness, income inequality, Jeff Bezos, John Snow's cholera map, Mark Zuckerberg, Nate Silver, peer-to-peer lending, Peter Thiel, price discrimination, quantitative hedge fund, Ronald Reagan, Rosa Parks, sentiment analysis, Silicon Valley, statistical model, Steve Jobs, Steven Levy, Steven Pinker, TaskRabbit, The Signal and the Noise by Nate Silver, working poor
Did it inform her about the world? Did it make her laugh? Or consider baseball’s data revolution in the 1990s. Many teams began using increasingly intricate statistics—rather than relying on old-fashioned human scouts—to make decisions. It was easy to measure offense and pitching but not fielding, so some organizations ended up underestimating the importance of defense. In fact, in his book The Signal and the Noise, Nate Silver estimates that the Oakland A’s, a data-driven organization profiled in Moneyball, were giving up eight to ten wins per year in the mid-nineties because of their lousy defense. The solution is not always more Big Data. A special sauce is often necessary to help Big Data work best: the judgment of humans and small surveys, what we might call small data. In an interview with Silver, Billy Beane, the A’s then general manager and the main character in Moneyball, said that he actually had begun increasing his scouting budget.
Dahl, “Family Violence and Football: The Effect of Unexpected Emotional Cues on Violent Behavior,” Quarterly Journal of Economics 126, no. 1 (2011). 197 Here’s how Bill Simmons: Bill Simmons, “It’s Hard to Say Goodbye to David Ortiz,” ESPN.com, June 2, 2009, http://www.espn.com/espnmag/story?id=4223584. 198 how can we predict how a baseball player will perform in the future: This is discussed in Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t (New York: Penguin, 2012). 199 “beefy sluggers” indeed do, on average, peak early: Ryan Campbell, “How Will Prince Fielder Age?” October 28, 2011, http://www.fangraphs.com/blogs/how-will-prince-fielder-age/. 199 Ortiz’s doppelgangers’: This data was kindly provided to me by Rob McQuown of Baseball Prospectus. 204 Kohane asks: I interviewed Isaac Kohane by phone on June 15, 2015. 205 James Heywood is an entrepreneur: I interviewed James Heywood by phone on August 17, 2015.
See also bias; discrimination; race/racism self-employed people, and taxes, 178–80 sentiment analysis, 87–92, 247–48 sex as addiction, 219 and benefits of digital truth serum, 158–59, 161 and childhood experiences, 50–52 condoms and, 5, 122 and digital revolution, 274, 279 and dimensions of sexuality, 279 during marriage, 5–6 and fetishes, 120 and Freud, 45–52 Google searches about, 5–6, 51–52, 114, 115, 117, 118, 122–24, 126, 127–28 and handling the truth, 158–59, 161 and Harvard Crimson editorial about Zuckerberg, 155 how much, 122–23, 124–25, 127 in India, 19 new information about, 19 oral, 128 and physical appearance, 120, 120n, 125–26, 127 and power of Big Data, 53 pregancy and having, 189 Rolling Stones song about, 278 and sex organs, 123–24 Stephens-Davidowitz’s first New York Times column about, 282 and traditional research methods, 274 truth about, 5–6, 112–28, 114n, 117 and typing errors, 48–50 and women’s genitals, 126–27 See also incest; penis; pornography; rape; vagina Shadow (app), 47 Shakespeare, William, 89–90 Shapiro, Jesse, 74–76, 93–97, 141–44, 235, 273 “Shattered” (Rolling Stones song), 278 shopping habits, predictions about, 71–74 The Signal and the Noise (Silver), 254 Silver, Nate, 10, 12–13, 133, 199, 200, 254, 255 Simmons, Bill, 197–98 Singapore, pregnancy in, 190 Siroker, Dan, 211–12 sleep and digital revolution, 279 Jawbone and, 276–77 and pregnancy, 189 “Slutload,” 58 small data, 255–56 smiles, and pictures as data, 99 Smith, Michael D., 224 Snow, John, 275 Sochi, Russia, gays in, 119 social media bias of data from, 150–53 doppelganger hunting on, 201–3 and wives descriptions of husbands, 160–61, 160–61n See also specific site or topic social science, 272–74, 276, 279 social security, and words as data, 93 socioeconomic background and predicting success in basketball, 34–41 See also pedigrees sociology, 273, 274 Soltas, Evan, 130, 162, 266–67 South Africa, pregnancy in, 189 Southern Poverty Law Center, 137 Spain, pregnancy in, 190 Spartanburg Herald-Journal (South Carolina), and words as data, 96 specialization, extreme, 186 speed, for transmitting data, 56–59 “Spider Solitaire,” 58 Stephens-Davidowitz, Noah, 165–66, 165–66n, 169, 206, 263 Stephens-Davidowitz, Seth ambitions of, 33 lying by, 282n mate choice for, 25–26, 271 motivations of, 2 obsessiveness of, 282, 282n professional background of, 14 and writing conclusions, 271–72, 279, 280–84 Stern, Howard, 157 stock market data for, 55–56 and examples of Big Data searches, 22 Summers-Stephens-Davidowitz attempt to predict the, 245–48, 251–52 Stone, Oliver, 185 Stoneham, James, 266, 269 Storegard, Adam, 99–101 stories categories/types of, 91–92 viral, 22, 92 and zooming in, 205–6 See also specific story Stormfront (website), 7, 14, 18, 137–40 stretch marks, and pregnancy, 188–89 Stuyvesant High School (New York City), 231–37, 238, 240 suburban areas, and origins of notable Americans, 183–84 successful/notable Americans factors that drive, 185–86 zooming in on, 180–86 suffering, and benefits of digital truth serum, 161 suicide, and danger of empowered government, 266, 267–68 Summers, Lawrence and Obama-racism study, 243–44 and predicting the stock market, 245, 246, 251–52 Stephens-Davidowitz’s meeting with, 243–45 Sunstein, Cass, 140 Super Bowl games, advertising during, 221–25, 239 Super Crunchers (Gnau), 264 Supreme Court, and abortion, 147 Surowiecki, James, 203 surveys in-person, 108 internet, 108 and lying, 105–7, 108, 108n and pictures as data, 97 skepticism about, 171 telephone, 108 and truth about sex, 113, 116 and zooming in on hours and minutes, 193 See also specific survey or topic Syrian refugees, 131 Taleb, Nassim, 17 Tartt, Donna, 283 TaskRabbit, 212 taxes cheating on, 22, 178–80, 206 and examples of Big Data searches, 22 and lying, 180 and self-employed people, 178–80 and words as data, 93–95 zooming in on, 172–73, 178–80, 206 teachers, using tests to judge, 253–54 teenagers adopted, 108n as gay, 114, 116 lying by, 108n and origins of political preferences, 169 and truth about sex, 114, 116 See also children television and A/B testing, 222 advertising on, 221–26 Terabyte, 264 terrorism, 18, 129–31 tests/testing of high school students, 231–37, 253–54 and judging teacher, 253–54 and obsessive infatuations with numbers, 253–54 online behavior as supplement to, 278 and small data, 255–56 See also specific test or study Thiel, Peter, 155 Think Progress (website), 130 Thinking, Fast and Slow (Kahneman), 283 Thome, Jim, 200 Tourangeau, Roger, 107, 108 towns, zooming in on, 172–90 Toy Story (movie), 192 Trump, Donald elections of 2012 and, 7 and ignoring what people tell you, 157 and immigration, 184 issues propagated by, 7 and origins of notable Americans, 184 polls about, 1 predictions about, 11–14 and racism, 8, 9, 11, 12, 14, 133, 139 See also elections, 2016 truth benefits of knowing, 158–63 handling the, 158–63 See also digital truth serum; lying; specific topic Tuskegee University, 183 Twentieth Century Fox, 221–22 Twitter, 151–52, 160–61n, 201–3 typing errors by searchers, 48–50 The Unbearable Lightness of Being (Kundera), 233 Uncharted (Aiden and Michel), 78–79 unemployment and child abuse, 145–47 data about, 56–57, 58–59 unintended consequences, 197 United States and Civil War, 79 as united or divided, 78–79 University of California, Berkeley, racism in 2008 election study at, 2 University of Maryland, survey of graduates of, 106–7 urban areas and life expectancy, 177 and origins of notable Americans, 183–84, 186 vagina, smells of, 19, 126–27, 161 Varian, Hal, 57–58, 224 Vikingmaiden88, 136–37, 140–41, 145 violence and real science, 273 zooming in on, 190–97 See also murder voter registration, 106 voter turnout, 9–10, 109–10 voting behavior, and lying, 106, 107, 109–10 Vox, 202 Walmart, 71–72 Washington Post, and words as data, 75, 94 Washington Times, and words as data, 75, 94–95 wealth and life expectancy, 176–77 See also income distribution weather, and predictions about wine, 73–74 Weil, David N., 99–101 Weiner, Anthony, 234n white nationalism, 137–40, 145.
Albert Einstein, algorithmic trading, Amazon Mechanical Turk, Apple's 1984 Super Bowl advert, backtesting, Black Swan, book scanning, bounce rate, business intelligence, business process, call centre, commoditize, computer age, conceptual framework, correlation does not imply causation, crowdsourcing, dark matter, data is the new oil, en.wikipedia.org, Erik Brynjolfsson, Everything should be made as simple as possible, experimental subject, Google Glasses, happiness index / gross national happiness, job satisfaction, Johann Wolfgang von Goethe, lifelogging, Machine translation of "The spirit is willing, but the flesh is weak." to Russian and back, mass immigration, Moneyball by Michael Lewis explains big data, Nate Silver, natural language processing, Netflix Prize, Network effects, Norbert Wiener, personalized medicine, placebo effect, prediction markets, Ray Kurzweil, recommendation engine, risk-adjusted returns, Ronald Coase, Search for Extraterrestrial Intelligence, self-driving car, sentiment analysis, software as a service, speech recognition, statistical model, Steven Levy, text mining, the scientific method, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, Thomas Bayes, Turing test, Watson beat the top human players on Jeopardy!, X Prize, Yogi Berra, zero-sum game
Michael Scherer, “Inside the Secret World of the Data Crunchers Who Helped Obama Win.” TIME Magazine, November 07, 2012. http://swampland.time.com/2012/11/07/inside-the-secret-world-of-quants-and-data-crunchers-who-helped-obama-win/. Colbert Nation, www.colbertnation.com. Stephen Colbert interviews Nate Silver, New York Times blogger about his book, The Signal and the Noise: Why So Many Predictions Fail—but Some Don’t. http://www.colbertnation.com/the-colbert-report-videos/420765/november-05-2012/nate-silver. Peggy Noonan, “They’ve Lost That Lovin’ Feeling.” Wall Street Journal, July 30, 2011. http://online.wsj.com/article/SB10001424053111904800304576474620336602248.html. Jack Gillum, “Mitt Romney Uses Secretive Data Mining To Identify Wealthy Donors.” Huffington Post, August 24, 2012. www.huffingtonpost.com/2012/08/24/mitt-romney-data-mining_n_1827318.html.
The bad news is that it’s actually more than half; the good news is that PA can learn to do better. A Faulty Oracle Everyone Loves The first step toward predicting the future is admitting you can’t. —Stephen Dubner, Freakonomics Radio, March 30, 2011 The “prediction paradox”: The more humility we have about our ability to make predictions, the more successful we can be in planning for the future. —Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—but Some Don’t Half of what we will teach you in medical school will, by the time you are done practicing, be proved wrong. —Dr. Mehmet Oz Your resident “oracle,” PA, tells you which customers are most likely to respond. It earmarks a quarter of the entire list and says, “These folks are three times more likely to respond than average!” So now you have a short list of 250,000 customers of which 3 percent will respond—7,500 responses.
However, they kept at it, squeezing every drop of potential out of their brainshare and data, right up until the final weeks before the big match. Confidence without Overconfidence Both experts and laypeople mistake more confident predictions for more accurate ones. But overconfidence is often the reason for failure. If our appreciation of uncertainty improves, our predictions can get better too. —Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—but Some Don’t The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt. —Bertrand Russell You got to know when to hold ‘em, know when to fold ‘em. —Don Schlitz, “The Gambler” (sung by Kenny Rogers) Jeopardy! wasn’t built for players with no self-doubt. —Chris Jones, Esquire Magazine Besides answering questions, there’s a second skill each Jeopardy!
Superforecasting: The Art and Science of Prediction by Philip Tetlock, Dan Gardner
Affordable Care Act / Obamacare, Any sufficiently advanced technology is indistinguishable from magic, availability heuristic, Black Swan, butterfly effect, cloud computing, cuban missile crisis, Daniel Kahneman / Amos Tversky, desegregation, drone strike, Edward Lorenz: Chaos theory, forward guidance, Freestyle chess, fundamental attribution error, germ theory of disease, hindsight bias, index fund, Jane Jacobs, Jeff Bezos, Kenneth Arrow, Mikhail Gorbachev, Mohammed Bouazizi, Nash equilibrium, Nate Silver, obamacare, pattern recognition, performance metric, Pierre-Simon Laplace, place-making, placebo effect, prediction markets, quantitative easing, random walk, randomized controlled trial, Richard Feynman, Richard Feynman, Richard Thaler, Robert Shiller, Robert Shiller, Ronald Reagan, Saturday Night Live, Silicon Valley, Skype, statistical model, stem cell, Steve Ballmer, Steve Jobs, Steven Pinker, the scientific method, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, Thomas Bayes, Watson beat the top human players on Jeopardy!
A mere two years after it was published the Arab Spring turned the Middle East topsy-turvy, but I can’t find it in Friedman’s book, which casts some doubt on his forecasts for the remaining ninety-eight years. Friedman is also the author of the 1991 book The Coming War with Japan—that’s the coming American war with Japan—which has yet to prove its prescience. 7. For islands of professionalism in a sea of malpractice, see the forecasting concepts and tools reviewed in Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—but Some Don’t (New York: Penguin Press, 2012); J. Scott Armstrong, ed., Principles of Forecasting: A Handbook for Researchers and Practitioners (Boston: Kluwer, 2001); and Bruce Bueno de Mesquita, The Predictioneer’s Game (New York: Random House, 2009). Expanding these islands has proven hard. There is often little transfer of classroom statistical concepts, like regression toward the mean, to problems that students later encounter in life.
They have no idea how good their forecasts are in the short, medium, or long term—and no idea how good their forecasts could become. At best, they have vague hunches. That’s because the forecast-measure-revise procedure operates only within the rarefied confines of high-tech forecasting, such as the work of macroeconomists at central banks or marketing and financial professionals in big companies or opinion poll analysts like Nate Silver.7 More often forecasts are made and then … nothing. Accuracy is seldom determined after the fact and is almost never done with sufficient regularity and rigor that conclusions can be drawn. The reason? Mostly it’s a demand-side problem: The consumers of forecasting—governments, business, and the public—don’t demand evidence of accuracy. So there is no measurement. Which means no revision. And without revision, there can be no improvement.
Consider the weather in Phoenix, Arizona. Each June, it gets very hot and sunny. A forecaster who followed a mindless rule like, “always assign 100% to hot and sunny” could get a Brier score close to 0, leaving 0.2 in the dust. Here, the right test of skill would be whether a forecaster can do better than mindlessly predicting no change. This is an underappreciated point. For example, after the 2012 presidential election, Nate Silver, Princeton’s Sam Wang, and other poll aggregators were hailed for correctly predicting all fifty state outcomes, but almost no one noted that a crude, across-the-board prediction of “no change”—if a state went Democratic or Republican in 2008, it will do the same in 2012—would have scored forty-eight out of fifty, which suggests that the many excited exclamations of “He called all fifty states!”
Stuffocation by James Wallman
3D printing, Airbnb, back-to-the-land, Berlin Wall, big-box store, Black Swan, BRICs, carbon footprint, Cass Sunstein, clean water, collaborative consumption, commoditize, creative destruction, crowdsourcing, David Brooks, Fall of the Berlin Wall, happiness index / gross national happiness, high net worth, income inequality, Intergovernmental Panel on Climate Change (IPCC), James Hargreaves, Joseph Schumpeter, Kitchen Debate, Martin Wolf, mass immigration, McMansion, means of production, Nate Silver, Occupy movement, Paul Samuelson, post-industrial society, Post-materialism, post-materialism, Richard Florida, Richard Thaler, sharing economy, Silicon Valley, Simon Kuznets, Skype, spinning jenny, The Signal and the Noise by Nate Silver, Thorstein Veblen, Tyler Cowen: Great Stagnation, World Values Survey, Zipcar
Using the Past to Tell the Future I am indebted to three sources for this section: Peter N Stearns, “Why Study History?”, American Historical Association, 1998; Nate Silver, The Signal and the Noise (New York: Allen Lane, 2012); and Rob Hyndman, “Why are some things easier to forecast than others?”, 18 September 2012, on his blog, Hyndsight (www.robjhyndman.com/hyndsight). “In the 1970s, the high temperature forecasts were wrong, on average, by about six degrees. Today they are only wrong by half that amount, three degrees. When hurricane forecasters predicted where a hurricane would hit land in the 1980s they were usually out by 350 miles. Today, their predictions are only wrong by 100 miles.” If you’re not ready – yet – to take on all of Nate Silver’s The Signal and the Noise, read Nate Silver, “The Weatherman Is Not a Moron”, New York Times, 7 September 2012. The Farm Where the Corn Did Not Grow Tall For Everett Rogers’s version of his life, read Everett M Rogers, The Fourteenth Paw (Singapore: Asian Media Information and Communication Centre, 2008).
3D printing, Albert Einstein, Amazon Mechanical Turk, Arthur Eddington, basic income, Bayesian statistics, Benoit Mandelbrot, bioinformatics, Black Swan, Brownian motion, cellular automata, Claude Shannon: information theory, combinatorial explosion, computer vision, constrained optimization, correlation does not imply causation, creative destruction, crowdsourcing, Danny Hillis, data is the new oil, double helix, Douglas Hofstadter, Erik Brynjolfsson, experimental subject, Filter Bubble, future of work, global village, Google Glasses, Gödel, Escher, Bach, information retrieval, job automation, John Markoff, John Snow's cholera map, John von Neumann, Joseph Schumpeter, Kevin Kelly, lone genius, mandelbrot fractal, Mark Zuckerberg, Moneyball by Michael Lewis explains big data, Narrative Science, Nate Silver, natural language processing, Netflix Prize, Network effects, NP-complete, off grid, P = NP, PageRank, pattern recognition, phenotype, planetary scale, pre–internet, random walk, Ray Kurzweil, recommendation engine, Richard Feynman, Richard Feynman, Second Machine Age, self-driving car, Silicon Valley, speech recognition, statistical model, Stephen Hawking, Steven Levy, Steven Pinker, superintelligent machines, the scientific method, The Signal and the Noise by Nate Silver, theory of mind, Thomas Bayes, transaction costs, Turing machine, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!, white flight, zero-sum game
Adam, the robot scientist, is described in “The automation of science,” by Ross King et al. (Science, 2009). Sasha Issenberg’s The Victory Lab (Broadway Books, 2012) dissects the use of data analysis in politics. “How President Obama’s campaign used big data to rally individual votes,” by the same author (MIT Technology Review, 2013), tells the story of its greatest success to date. Nate Silver’s The Signal and the Noise (Penguin Press, 2012) has a chapter on his poll aggregation method. Robot warfare is the theme of P. W. Singer’s Wired for War (Penguin, 2009). Cyber War, by Richard Clarke and Robert Knake (Ecco, 2012), sounds the alarm on cyberwar. My work on combining machine learning with game theory to defeat adversaries, which started as a class project, is described in “Adversarial classification,”* by Nilesh Dalvi et al.
David Wolpert derives his “no free lunch” theorem for induction in “The lack of a priori distinctions between learning algorithms”* (Neural Computation, 1996). I discuss the importance of prior knowledge in machine learning in “Toward knowledge-rich data mining”* (Data Mining and Knowledge Discovery, 2007) and misinterpretations of Occam’s razor in “The role of Occam’s razor in knowledge discovery”* (Data Mining and Knowledge Discovery, 1999). Overfitting is one of the main themes of The Signal and the Noise, by Nate Silver (Penguin Press, 2012), who calls it “the most important scientific problem you’ve never heard of.” “Why most published research findings are false,”* by John Ioannidis (PLoS Medicine, 2005), discusses the problem of mistaking chance findings for true ones in science. Yoav Benjamini and Yosef Hochberg propose a way to combat it in “Controlling the false discovery rate: A practical and powerful approach to multiple testing”* (Journal of the Royal Statistical Society, Series B, 1995).
In politics, as in business and war, there is nothing worse than seeing your opponent make moves that you don’t understand and don’t know what to do about until it’s too late. That’s what happened to the Romney campaign. They could see the other side buying ads in particular cable stations in particular towns but couldn’t tell why; their crystal ball was too fuzzy. In the end, Obama won every battleground state save North Carolina and by larger margins than even the most accurate pollsters had predicted. The most accurate pollsters, in turn, were the ones (like Nate Silver) who used the most sophisticated prediction techniques; they were less accurate than the Obama campaign because they had fewer resources. But they were a lot more accurate than the traditional pundits, whose predictions were based on their expertise. You might think the 2012 election was a fluke: most elections are not close enough for machine learning to be the deciding factor. But machine learning will cause more elections to be close in the future.
Big Data: A Revolution That Will Transform How We Live, Work, and Think by Viktor Mayer-Schonberger, Kenneth Cukier
23andMe, Affordable Care Act / Obamacare, airport security, AltaVista, barriers to entry, Berlin Wall, big data - Walmart - Pop Tarts, Black Swan, book scanning, business intelligence, business process, call centre, cloud computing, computer age, correlation does not imply causation, dark matter, double entry bookkeeping, Eratosthenes, Erik Brynjolfsson, game design, IBM and the Holocaust, index card, informal economy, intangible asset, Internet of things, invention of the printing press, Jeff Bezos, lifelogging, Louis Pasteur, Mark Zuckerberg, Menlo Park, Moneyball by Michael Lewis explains big data, Nate Silver, natural language processing, Netflix Prize, Network effects, obamacare, optical character recognition, PageRank, performance metric, Peter Thiel, Post-materialism, post-materialism, random walk, recommendation engine, self-driving car, sentiment analysis, Silicon Valley, Silicon Valley startup, smart grid, smart meter, social graph, speech recognition, Steve Jobs, Steven Levy, the scientific method, The Signal and the Noise by Nate Silver, The Wealth of Nations by Adam Smith, Turing test, Watson beat the top human players on Jeopardy!
Neyman’s famous paper is Jerzy Neyman, “On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection,” Journal of the Royal Statistical Society 97, no. 4 (1934), pp. 558–625. A sample of 1,100 observations is sufficient—Earl Babbie, Practice of Social Research (12th ed. 2010), pp. 204–207. [>] The cellphone effect—“Estimating the Cellphone Effect,” September 20, 2008 (http://www.fivethirtyeight.com/2008/09/estimating-cellphone-effect-22-points.html); for more on polling biases and other statistical insights see Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t (Penguin, 2012). [>] Steve Jobs’s gene sequencing—Walter Isaacson, Steve Jobs (Simon and Schuster, 2011), pp. 550–551. [>] Google Flu Trends predicting to city level—Dugas et al., “Google Flu Trends.” Etzioni on temporal data—Interview by Cukier, October 2011. [>] John Kunze quotation—Jonathan Rosenthal, “Special Report: International Banking,” The Economist, May 19, 2012, pp. 7–8.
Wall Street Journal, November 19, 2010 (http://online.wsj.com/article/SB10001424052748704648604575620750998072986.html). Scott, James. Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed. Yale University Press, 1998. Seltzer, William, and Margo Anderson. “The Dark Side of Numbers: The Role of Population Data Systems in Human Rights Abuses.” Social Research 68 (2001) pp. 481–513. Silver, Nate. The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t. Penguin, 2012. Singel, Ryan. “Netflix Spilled Your Brokeback Mountain Secret, Lawsuit Claims.” Wired, December 17, 2009 (http://www.wired.com/threatlevel/2009/12/netflix-privacy-lawsuit/). Smith, Adam. The Wealth of Nations (1776). Reprinted Bantam Classics, 2003. A free electronic version is available (http://www2.hn.psu.edu/faculty/jmanis/adam-smith/Wealth-Nations.pdf).
Its accuracy depends on ensuring randomness when collecting the sample data, but achieving such randomness is tricky. Systematic biases in the way the data is collected can lead to the extrapolated results being very wrong. There are echoes of such problems in election polling using landline phones. The sample is biased against people who only use cell-phones (who are younger and more liberal), as the statistician Nate Silver has pointed out. This has resulted in incorrect election predictions. In the 2008 presidential election between Barack Obama and John McCain, the major polling organizations of Gallup, Pew, and ABC/Washington Post found differences of between one and three percentage points when they polled with and without adjusting for cellphone users—a hefty margin considering the tightness of the race. Most troublingly, random sampling doesn’t scale easily to include subcategories, as breaking the results down into smaller and smaller subgroups increases the possibility of erroneous predictions.
Deep Thinking: Where Machine Intelligence Ends and Human Creativity Begins by Garry Kasparov
3D printing, Ada Lovelace, AI winter, Albert Einstein, AltaVista, barriers to entry, Berlin Wall, business process, call centre, clean water, computer age, Daniel Kahneman / Amos Tversky, David Brooks, Donald Trump, Douglas Hofstadter, Drosophila, Elon Musk, Erik Brynjolfsson, factory automation, Freestyle chess, Gödel, Escher, Bach, job automation, Leonard Kleinrock, Mikhail Gorbachev, Nate Silver, Norbert Wiener, packet switching, pattern recognition, Ray Kurzweil, Richard Feynman, Richard Feynman, rising living standards, rolodex, Second Machine Age, self-driving car, Silicon Valley, Silicon Valley startup, Skype, speech recognition, stem cell, Stephen Hawking, Steven Pinker, technological singularity, The Coming Technological Singularity, The Signal and the Noise by Nate Silver, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!, zero-sum game
This hypothesis was proposed by Murray Campbell at least as early as Monty Newborn’s 2002 book on Deep Blue. The punch line to his theory was that Deep Blue’s mysterious move wasn’t profound at all; it was a blunder and the result of a yet another bug. Per Campbell and Hsu, the move was “random,” the result of a known bug they had failed to kill before the match began. This tale acquired new life when election analyst Nate Silver used it as the centerpiece for an entire chapter of his 2012 book, The Signal and the Noise. The narrative suggested by Frederic and spread by Campbell was irresistible: Kasparov lost to Deep Blue because of a bug! Writes Silver, “The bug was anything but unfortunate for Deep Blue: it was likely what allowed the computer to beat Kasparov.” TIME, Wired, and other outlets ran with breathless variations on this theme, each story containing more errors about chess and more silly assumptions about my mental state than the last.
I will not repeat here the stream of profanities in Russian, English, and languages not yet invented that escaped my lips when I first read that paragraph. What in the hell was this? Two paragraphs after Illescas says IBM had hired Russian speakers to spy on me, he says the team entered this critical line into Deep Blue’s book that morning? An obscure variation that I had only discussed with my team in the privacy of our suite at the Plaza Hotel that week in New York? I’m no Nate Silver, but the odds of winning the lottery are quite attractive in comparison to those of the Deep Blue team entering a specific variation I had never played before in my life into the computer’s book on the very same day it appeared on the board in the final game. And not only preparing the machine for the 4..Nd7 Caro-Kann—even during my brief dalliance with the Caro-Kann as a fifteen-year-old I played the 4..Bf5 line exclusively—but also forcing it to play 8.Nxe6 and doing this despite generally giving Deep Blue “a lot of freedom to play,” in Illescas’s own words.
23andMe, 3D printing, Affordable Care Act / Obamacare, Anne Wojcicki, Atul Gawande, augmented reality, bioinformatics, call centre, Clayton Christensen, clean water, cloud computing, commoditize, computer vision, conceptual framework, connected car, correlation does not imply causation, creative destruction, crowdsourcing, dark matter, data acquisition, disintermediation, don't be evil, Edward Snowden, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, Firefox, global village, Google Glasses, Google X / Alphabet X, Ignaz Semmelweis: hand washing, information asymmetry, interchangeable parts, Internet of things, Isaac Newton, job automation, Joseph Schumpeter, Julian Assange, Kevin Kelly, license plate recognition, lifelogging, Lyft, Mark Zuckerberg, Marshall McLuhan, meta analysis, meta-analysis, microbiome, Nate Silver, natural language processing, Network effects, Nicholas Carr, obamacare, pattern recognition, personalized medicine, phenotype, placebo effect, RAND corporation, randomized controlled trial, Second Machine Age, self-driving car, Silicon Valley, Skype, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, Snapchat, social graph, speech recognition, stealth mode startup, Steve Jobs, the scientific method, The Signal and the Noise by Nate Silver, The Wealth of Nations by Adam Smith, Turing test, Uber for X, Watson beat the top human players on Jeopardy!, X Prize
Although Eisenstein stopped short of claiming that the first industrial revolution was an outgrowth of the printing press, many others have claimed this. Marshall McLuhan, in The Gutenberg Galaxy, wrote: “The invention of typography confirmed and extended the new visual stress of applied knowledge, providing the first uniformly repeatable commodity, the first assembly-line, and the first mass-production.”11 More recently, Nate Silver, in The Signal and the Noise, asserted that the industrial revolution of 1775 was sparked by the printing press, whereby the economic growth rate that was stagnant at 0.1 percent per year then grew faster than the growth rate of the population.12 But I prefer to principally assess the Gutenberg transformative effects by the specific attributes that they induced or cultivated instead of as a precursor for subsequent momentous periods in history.
Matthew, “The World’s Most Expensive Book Just Sold For Over $14 Million,” Business Insider, November 26, 2013, http://www.businessinsider.com/worlds-most-expensive-book-sells-for-14-million-2013-11. 5. Eisenstein, The Printing Press as an Agent of Change, 152. 6. Ibid., 159. 7. N. Silver, The Signal and the Noise (New York, NY: Penguin, 2012), 2. 8. N. Carr, The Shallows: What the Internet Is Doing to Our Brains (New York, NY: W.W. Norton, 2010), 69. 9. Silver, The Signal and the Noise, 12. 10. Eisenstein, The Printing Press as an Agent of Change, 41. 11. McLuhan, The Gutenberg Galaxy, 124. 12. Silver, The Signal and the Noise, 7. 13. Ibid., 3. 14. Carr, The Shallows: What the Internet Is Doing to Our Brains, 71. 15. “The Book of Jobs: Hope, Hype, and Apple’s iPad,” The Economist, January 30–February 5, 2010. 16. Eisenstein, The Printing Press as an Agent of Change, 75. 17.
Standage, “Social Networking in the 1600s,” New York Times, June 23, 2013, http://www.nytimes.com/2013/06/23/opinion/sunday/social-networking-in-the-1600s.html?pagewanted=all. 22. Eisenstein, The Printing Press as an Agent of Change, 53. 23. V. Goel, “Our Daily Cup of Facebook,” New York Times, August 13, 2013, http://bits.blogs.nytimes.com/2013/08/13/our-daily-cup-of-facebook/?ref=technology&_r=0&pagewanted=print. 24. N. Silver, The Signal and the Noise, 2. 25. “The March of Protest,” The Economist, June 29, 2013, http://www.economist.com/printedition/2013-06-29. 26. W. Ghonim, Revolution 2.0: The Power of the People Is Greater Than the People in Power (New York: Houghton Mifflin Harcourt, 2012). 27. Eisenstein, The Printing Press as an Agent of Change, 129. 28. M. B. Hall, The Scientific Renaissance 1450–1630 (New York, NY: Harper & Brothers, 1962), 130. 29.
Misbehaving: The Making of Behavioral Economics by Richard H. Thaler
3Com Palm IPO, Albert Einstein, Alvin Roth, Amazon Mechanical Turk, Andrei Shleifer, Apple's 1984 Super Bowl advert, Atul Gawande, Berlin Wall, Bernie Madoff, Black-Scholes formula, capital asset pricing model, Cass Sunstein, Checklist Manifesto, choice architecture, clean water, cognitive dissonance, conceptual framework, constrained optimization, Daniel Kahneman / Amos Tversky, delayed gratification, diversification, diversified portfolio, Edward Glaeser, endowment effect, equity premium, Eugene Fama: efficient market hypothesis, experimental economics, Fall of the Berlin Wall, George Akerlof, hindsight bias, Home mortgage interest deduction, impulse control, index fund, information asymmetry, invisible hand, Jean Tirole, John Nash: game theory, John von Neumann, Kenneth Arrow, late fees, law of one price, libertarian paternalism, Long Term Capital Management, loss aversion, market clearing, Mason jar, mental accounting, meta analysis, meta-analysis, money market fund, More Guns, Less Crime, mortgage debt, Myron Scholes, Nash equilibrium, Nate Silver, New Journalism, nudge unit, Paul Samuelson, payday loans, Ponzi scheme, presumed consent, pre–internet, principal–agent problem, prisoner's dilemma, profit maximization, random walk, randomized controlled trial, Richard Thaler, Robert Shiller, Robert Shiller, Ronald Coase, Silicon Valley, South Sea Bubble, statistical model, Steve Jobs, technology bubble, The Chicago School, The Myth of the Rational Market, The Signal and the Noise by Nate Silver, The Wealth of Nations by Adam Smith, Thomas Kuhn: the structure of scientific revolutions, transaction costs, ultimatum game, Vilfredo Pareto, Walter Mischel, zero-sum game
So what effect has this research plus a free app had on the behavior of football coaches? Essentially none. Since Romer wrote his paper, the frequency of going for it on fourth down has marginally gone down, meaning that teams have gotten dumber! (Similarly, there has been no noticeable change in teams’ draft strategy since our paper came out.) Nate Silver, the ex–sports analytics junkie who became famous for his political forecasts and for the excellent book The Signal and the Noise, estimates that bad fourth-down decisions cost a football team an average of half a win per season. The Times analysts estimate it to be closer to two-thirds of a win per year. That may not seem like a lot, but the season is only sixteen games. A team can win an extra game every other year just by making the smart decision two or three times a game, one they can even check online if they need help.¶ Of course, coaches are Humans.
Chapter 29: Football 277 “Division of labor strongly attenuates”: Stewart (1997). 278 football paper: Massey and Thaler (2013). 280 The winner’s curse: For a review, see my “Anomalies” column on the subject (Thaler, 1988a). 280 The false consensus effect: Ross, Greene, and House (1977). 284 If a team is paying a high draft pick a lot of money: Camerer and Weber (1999). 292 teams don’t go for it: Romer (2006). 292 New York Times used his model: For an example of Brian Burke’s work, see http://www.advancedfootballanalytics.com/. 292 “New York Times 4th Down Bot”: The bot’s recommendations can be found at http://nyt4thdownbot.com/. For a comparison between coaches and the NYT Bot’s performances, see Burk and Quealy (2014). 292 The Signal and the Noise: Silver (2012). 293 Peter Principle: Peter and Hull (1969). Chapter 30: Game Shows 296 They asked me if I would like to join the team: Post et al. (2008). 299 my paper with Eric Johnson: Thaler and Johnson (1990). 300 an experiment to measure the difference between public and private decisions: Baltussen, van den Assem, and van Dolder (2015). 301 more risk averse in front of the crowd: This lines up with findings that investors take more risks online than in front of others.
. ———. 1984. “Stock Prices and Social Dynamics.” Brookings Papers on Economic Activity 2: 457–510. ———. 1986. “Comments on Miller and on Kleidon.” Journal of Business 59, no. 4, part 2: S501–5. ———. 2000. Irrational Exuberance. Princeton: Princeton University Press. Shleifer, Andrei, and Robert W. Vishny. 1997. “The Limits of Arbitrage.” Journal of Finance 52, no. 1: 35–55. Silver, Nate. 2012. The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t. New York: Penguin. Simon, Herbert A. 1957. Models of Man, Social and Rational: Mathematical Essays on Rational Human Behavior in a Social Setting. Oxford: Wiley. Sloman, Steven A. 1996. “The Empirical Case for Two Systems of Reasoning.” Psychological Bulletin 119, no. 1: 3. Slonim, Robert L., and Alvin E. Roth. 1998. “Learning in High Stakes Ultimatum Games: An Experiment in the Slovak Republic.”
Warnings by Richard A. Clarke
active measures, Albert Einstein, algorithmic trading, anti-communist, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, Bernie Madoff, cognitive bias, collateralized debt obligation, complexity theory, corporate governance, cuban missile crisis, data acquisition, discovery of penicillin, double helix, Elon Musk, failed state, financial thriller, fixed income, Flash crash, forensic accounting, friendly AI, Intergovernmental Panel on Climate Change (IPCC), Internet of things, James Watt: steam engine, Jeff Bezos, John Maynard Keynes: Economic Possibilities for our Grandchildren, knowledge worker, Maui Hawaii, megacity, Mikhail Gorbachev, money market fund, mouse model, Nate Silver, new economy, Nicholas Carr, nuclear winter, pattern recognition, personalized medicine, phenotype, Ponzi scheme, Ray Kurzweil, Richard Feynman, Richard Feynman, Richard Feynman: Challenger O-ring, risk tolerance, Ronald Reagan, Search for Extraterrestrial Intelligence, self-driving car, Silicon Valley, smart grid, statistical model, Stephen Hawking, Stuxnet, technological singularity, The Future of Employment, the scientific method, The Signal and the Noise by Nate Silver, Tunguska event, uranium enrichment, Vernor Vinge, Watson beat the top human players on Jeopardy!, women in the workforce, Y2K
Later books have looked at Tetlock’s foundational results in some additional detail. Dan Gardner’s 2012 Future Babble draws on recent research in psychology, neuroscience, and behavioral economics to detail the biases and other cognitive processes that skew our judgment when we try to make predictions about the future. And building on a successful career in sports and political forecasting, Nate Silver discusses in his book, The Signal and the Noise, how thinking more probabilistically can help us distill more accurate predictions from a sea of raw data. Fundamentally, these books all identify the difficulties inherent in trying to see into the future. Predicting natural phenomena is stymied by the chaotic nature of the universe: natural processes are nonlinear systems driven by feedback loops that are often inherently unpredictable themselves.
However, such a response does little to help prepare for future disaster or to ensure that Cassandra’s warnings do not again go unheeded. As it seemed we weren’t the only ones who had noticed Cassandras in our midst, we then wondered if there existed any scholarly research on the topic of predictions. In fact, prediction is something that academics have spent a lot of time studying and considering. The statistician Nate Silver has taken a highly quantitative approach to prediction, one that works for a certain class of event. The jurist Richard Posner examined the phenomenon of catastrophes in the years after 9/11. Psychologists like Dan Ariely and Tsachi Ein-Dor have probed the way our brains work (and don’t) through empirical observation and the study of warnings. Unquestionably one of the foundational works in this area, predictions within the social sciences, is Philip Tetlock’s Expert Political Judgment.
3D printing, algorithmic trading, Any sufficiently advanced technology is indistinguishable from magic, augmented reality, big data - Walmart - Pop Tarts, call centre, Cass Sunstein, Clayton Christensen, commoditize, computer age, death of newspapers, deferred acceptance, Edward Lorenz: Chaos theory, Erik Brynjolfsson, Filter Bubble, Flash crash, Florence Nightingale: pie chart, Frank Levy and Richard Murnane: The New Division of Labor, Google Earth, Google Glasses, High speed trading, Internet Archive, Isaac Newton, Jaron Lanier, Jeff Bezos, job automation, John Markoff, Kevin Kelly, Kodak vs Instagram, lifelogging, Marshall McLuhan, means of production, Nate Silver, natural language processing, Netflix Prize, pattern recognition, price discrimination, recommendation engine, Richard Thaler, Rosa Parks, self-driving car, sentiment analysis, Silicon Valley, Silicon Valley startup, Slavoj Žižek, social graph, speech recognition, Steve Jobs, Steven Levy, Steven Pinker, Stewart Brand, the scientific method, The Signal and the Noise by Nate Silver, upwardly mobile, Wall-E, Watson beat the top human players on Jeopardy!, Y Combinator
While I was first coming up with formulas at college, trying to mathematically determine whether we should go to the library to get some work done, deep down in the recesses of our dorky ids I think that what we were saying is that life is uncertain and we were trying to make it more certain. I’m not as disturbed by numbers providing answers as I am by the potential that there might not be answers.” What is it about the modern world that makes us demand easy answers? Is it that we are naturally pattern-seeking creatures, as the statistician Nate Silver argues in The Signal and the Noise? Or is there something about the effects of the march of technology that demands the kind of answers only an algorithm can provide? “[The algorithm does] seem to be a key metaphor for what matters now in terms of organizing the world,” acknowledges McKenzie Wark, a media theorist who has written about digital technologies for the last 20 years. “If one thinks of algorithms as processes which terminate and generate a result, there’s a moment when the process ceases and you have your answer.
Simple Rules: How to Thrive in a Complex World by Donald Sull, Kathleen M. Eisenhardt
Affordable Care Act / Obamacare, Airbnb, asset allocation, Atul Gawande, barriers to entry, Basel III, Berlin Wall, carbon footprint, Checklist Manifesto, complexity theory, Craig Reynolds: boids flock, Credit Default Swap, Daniel Kahneman / Amos Tversky, diversification, drone strike, en.wikipedia.org, European colonialism, Exxon Valdez, facts on the ground, Fall of the Berlin Wall, haute cuisine, invention of the printing press, Isaac Newton, Kickstarter, late fees, Lean Startup, Louis Pasteur, Lyft, Moneyball by Michael Lewis explains big data, Nate Silver, Network effects, obamacare, Paul Graham, performance metric, price anchoring, RAND corporation, risk/return, Saturday Night Live, sharing economy, Silicon Valley, Startup school, statistical model, Steve Jobs, TaskRabbit, The Signal and the Noise by Nate Silver, transportation-network company, two-sided market, Wall-E, web application, Y Combinator, Zipcar
., “Health on Impulse: When Low Self-Control Promotes Healthy Food Choices,” Health Psychology 33, no. 2 (2013): 103–9, http://www.medscape.com/medline/abstract/2347758. [>] In contrast, people: Brian Wansink, David R. Rust, and Collin R. Payne, “Mindless Eating and Healthy Heuristics for the Irrational,” American Economic Review: Papers and Proceedings 99, no. 2 (2009): 165–69. [>] Meteorologists make: Nate Silver, The Signal and the Noise (New York: Penguin, 2012), 126–27. [>] Japanese honeybees: Atsushi Ugajin et al., “Detection of Neural Activity in the Brains of Japanese Honeybee Workers During the Formation of a ‘Hot Defensive Bee Ball,’” PLoS One 7, no. 3 (2012), available at the website of the National Center for Biotechnology Information, http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3303784/. [>] As an example of: Our account of the bees’ choice of new nest is based on the research of Thomas Seeley, especially Thomas D.
Affordable Care Act / Obamacare, Black Swan, business intelligence, Carmen Reinhart, cognitive bias, correlation does not imply causation, Daniel Kahneman / Amos Tversky, Donald Trump, en.wikipedia.org, Kenneth Rogoff, labor-force participation, lake wobegon effect, Long Term Capital Management, Mercator projection, Mercator projection distort size, especially Greenland and Africa, meta analysis, meta-analysis, Nate Silver, obamacare, p-value, PageRank, pattern recognition, publication bias, QR code, randomized controlled trial, risk-adjusted returns, Ronald Reagan, selection bias, statistical model, The Signal and the Noise by Nate Silver, Thomas Bayes, Tim Cook: Apple, wikimedia commons, Yogi Berra
“There is no scientifically plausible way of predicting the occurrence of a particular earthquake,” they note, adding that “prediction, as people expect it, requires predicting the magnitude, timing, and location of the future earthquake, which is not currently possible.”17 We simply don’t have the data, nor do we have the technology, to accurately predict quakes at this time. That said, the USGS does describe the places “most likely to produce earthquakes in the long term.” They call this forecasting, when they estimate the likelihood of a seismic event occurring over a period of time. This brings us to the distinction—or lack thereof—between a prediction and a forecast. As Nate Silver notes in The Signal and the Noise, the terms are used differently by some (most notably seismologists, who study earthquakes) but interchangeably by others. Some would argue that predictions are binary—something will or won’t happen—while forecasts are more probabilistic—there’s an X percent chance that something will happen. (To further complicate the issue, an estimate may be used when talking about past, current, or future data.)
The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies by Erik Brynjolfsson, Andrew McAfee
2013 Report for America's Infrastructure - American Society of Civil Engineers - 19 March 2013, 3D printing, access to a mobile phone, additive manufacturing, Airbnb, Albert Einstein, Amazon Mechanical Turk, Amazon Web Services, American Society of Civil Engineers: Report Card, Any sufficiently advanced technology is indistinguishable from magic, autonomous vehicles, barriers to entry, basic income, Baxter: Rethink Robotics, British Empire, business intelligence, business process, call centre, Chuck Templeton: OpenTable, clean water, combinatorial explosion, computer age, computer vision, congestion charging, corporate governance, creative destruction, crowdsourcing, David Ricardo: comparative advantage, digital map, employer provided health coverage, en.wikipedia.org, Erik Brynjolfsson, factory automation, falling living standards, Filter Bubble, first square of the chessboard / second half of the chessboard, Frank Levy and Richard Murnane: The New Division of Labor, Freestyle chess, full employment, game design, global village, happiness index / gross national happiness, illegal immigration, immigration reform, income inequality, income per capita, indoor plumbing, industrial robot, informal economy, intangible asset, inventory management, James Watt: steam engine, Jeff Bezos, jimmy wales, job automation, John Markoff, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, Joseph Schumpeter, Kevin Kelly, Khan Academy, knowledge worker, Kodak vs Instagram, law of one price, low skilled workers, Lyft, Mahatma Gandhi, manufacturing employment, Marc Andreessen, Mark Zuckerberg, Mars Rover, mass immigration, means of production, Narrative Science, Nate Silver, natural language processing, Network effects, new economy, New Urbanism, Nicholas Carr, Occupy movement, oil shale / tar sands, oil shock, pattern recognition, Paul Samuelson, payday loans, price stability, Productivity paradox, profit maximization, Ralph Nader, Ray Kurzweil, recommendation engine, Report Card for America’s Infrastructure, Robert Gordon, Rodney Brooks, Ronald Reagan, Second Machine Age, self-driving car, sharing economy, Silicon Valley, Simon Kuznets, six sigma, Skype, software patent, sovereign wealth fund, speech recognition, statistical model, Steve Jobs, Steven Pinker, Stuxnet, supply-chain management, TaskRabbit, technological singularity, telepresence, The Bell Curve by Richard Herrnstein and Charles Murray, The Signal and the Noise by Nate Silver, The Wealth of Nations by Adam Smith, total factor productivity, transaction costs, Tyler Cowen: Great Stagnation, Vernor Vinge, Watson beat the top human players on Jeopardy!, winner-take-all economy, Y2K
., “Grading Student Loans,” Liberty Street Economics blog, Federal Reserve Bank of New York, March 5, 2012, http://libertystreeteconomics.newyorkfed.org/2012/03/grading-student-loans.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+LibertyStreetEconomics+(Liberty+Street+Economics). 21. Tim Hornyak, “Towel-folding Robot Won’t Do the Dishes,” CNET, March 31, 2010, http://news.cnet.com/8301-17938_105-10471898-1.html. 22. Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t, 1st ed. (New York: Penguin, 2012). Chapter 13 POLICY RECOMMENDATIONS 1. “Employment Level,” Economic Research—Federal Reserve Bank of St. Louis (U.S. Department of Labor, Bureau of Labor Statistics, August 2, 2013), http://research.stlouisfed.org/fred2/series/LNU02000000. 2. Claudia Goldin and Lawrence F. Katz, The Race Between Education and Technology (Cambridge, MA: Belknap Press of Harvard University Press, 2010). 3.
Bayesian statistics, business intelligence, business process, cellular automata, Celtic Tiger, cloud computing, collateralized debt obligation, conceptual framework, congestion charging, corporate governance, correlation does not imply causation, crowdsourcing, discrete time, George Gilder, Google Earth, Infrastructure as a Service, Internet Archive, Internet of things, invisible hand, knowledge economy, late capitalism, lifelogging, linked data, Masdar, means of production, Nate Silver, natural language processing, openstreetmap, pattern recognition, platform as a service, recommendation engine, RFID, semantic web, sentiment analysis, slashdot, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, smart grid, smart meter, software as a service, statistical model, supply-chain management, the scientific method, The Signal and the Noise by Nate Silver, transaction costs
Sicular, S. (2013) ‘Big data is falling into the trough of disillusionment’, Gartner, 22 January, http://blogs.gartner.com/svetlana-sicular/big-data-is-falling-into-the-trough-of-disillusionment/ (last accessed 26 February 2013). Siegel, E. (2013) Predictive Analytics. Wiley, Hoboken, NJ. Sigala, M. (2005) ‘Integrating customer relationship management in hotel operations: managerial and operations implications’, International Journal of Hospitality Management, 24(3): 391–413. Silver, N. (2012) The Signal and the Noise: The Art and Science of Prediction. Penguin, London. Singer, N. (2012a) ‘You for sale: mapping, and sharing, the consumer genome’, New York Times, 17 June, http://www.nytimes.com/2012/06/17/technology/acxiom-the-quiet-giant-of-consumerdatabase-marketing.html (last accessed 11 October 2013). Singer, N. (2012b) ‘F.T.C. opens an inquiry into data brokers’, New York Times, 18 December, http://www.nytimes.com/2012/12/19/technology/ftc-opens-an-inquiry-into-data-brokers.html (last accessed 11 October 2013).
Rubinstein, I.S. (2013) ‘Big data: the end of privacy or a new beginning?’, International Data Privacy Law, online first, http://idpl.oxfordjournals.org/content/early/2013/01/24/idpl.ips036.short (last accessed 15 July 2013). Ruppert, E. (2012) ‘The governmental topologies of database devices’, Theory, Culture Society, 29: 116–36. Ruppert, E. (2013) ‘Rethinking empirical social sciences’, Dialogues in Human Geography, 3(3): 268–73. Salmon, F. (2014) ‘Why the Nate Silvers of the world don’t know everything’, Wired, 7 January, http://www.wired.com/business/2014/01/quants-dont-know-everything/ (last accessed 8 January 2014). Salus, P. (1995) Casting the Net: From Arpanet to Internet and Beyond. Addison Wesley, Reading, MA. Sawyer, S. (2008) ‘Data wealth, data poverty, science and cyberinfrastructure’, Prometheus: Critical Studies in Innovation, 26(4): 355–71. Schnapp, J. and Presner, P. (2009) Digital Humanities Manifesto 2.0. http://www.humanitiesblast.com/manifesto/Manifesto_V2.pdf (last accessed 13 March 2013).
Stress Test: Reflections on Financial Crises by Timothy F. Geithner
Affordable Care Act / Obamacare, asset-backed security, Atul Gawande, bank run, banking crisis, Basel III, Bernie Madoff, Bernie Sanders, break the buck, Buckminster Fuller, Carmen Reinhart, central bank independence, collateralized debt obligation, correlation does not imply causation, creative destruction, credit crunch, Credit Default Swap, credit default swaps / collateralized debt obligations, currency manipulation / currency intervention, David Brooks, Doomsday Book, eurozone crisis, financial innovation, Flash crash, Goldman Sachs: Vampire Squid, housing crisis, Hyman Minsky, illegal immigration, implied volatility, London Interbank Offered Rate, Long Term Capital Management, margin call, market fundamentalism, Martin Wolf, McMansion, Mexican peso crisis / tequila crisis, money market fund, moral hazard, mortgage debt, Nate Silver, negative equity, Northern Rock, obamacare, paradox of thrift, pets.com, price stability, profit maximization, pushing on a string, quantitative easing, race to the bottom, RAND corporation, regulatory arbitrage, reserve currency, Saturday Night Live, savings glut, selection bias, short selling, sovereign wealth fund, The Great Moderation, The Signal and the Noise by Nate Silver, Tobin tax, too big to fail, working poor
That seemed highly unlikely, so Merrill usually kept the super-seniors on its balance sheet. Their modest returns were still more than the cost of financing them, and they seemed almost bulletproof. Standard & Poor’s estimated a mere 0.12 percent chance that one of its AAA-rated CDOs would fail to pay out over five years—and super-seniors were considered safer than typical AAAs. But as Nate Silver noted in The Signal and the Noise, his excellent book about why many predictions fail, the actual default rate for AAA-rated tranches of CDOs would be 28 percent, more than two hundred times higher than S&P had predicted. Their perceived safety rested on all kinds of flawed assumptions, starting with the notion that housing prices would never fall simultaneously across the country. CDOs were often spliced together from geographically diverse piles of subprime mortgages, which was supposed to mitigate the effects of a housing slump in any one region.