Nick Bostrom

127 results back to index

Architects of Intelligence by Martin Ford

3D printing, agricultural Revolution, AI winter, algorithmic bias, Alignment Problem, AlphaGo, Apple II, artificial general intelligence, Asilomar, augmented reality, autonomous vehicles, backpropagation, barriers to entry, basic income, Baxter: Rethink Robotics, Bayesian statistics, Big Tech, bitcoin, Boeing 747, Boston Dynamics, business intelligence, business process, call centre, Cambridge Analytica, cloud computing, cognitive bias, Colonization of Mars, computer vision, Computing Machinery and Intelligence, correlation does not imply causation, CRISPR, crowdsourcing, DARPA: Urban Challenge, data science, deep learning, DeepMind, Demis Hassabis, deskilling, disruptive innovation, Donald Trump, Douglas Hofstadter, driverless car, Elon Musk, Erik Brynjolfsson, Ernest Rutherford, fake news, Fellow of the Royal Society, Flash crash, future of work, general purpose technology, Geoffrey Hinton, gig economy, Google X / Alphabet X, Gödel, Escher, Bach, Hans Moravec, Hans Rosling, hype cycle, ImageNet competition, income inequality, industrial research laboratory, industrial robot, information retrieval, job automation, John von Neumann, Large Hadron Collider, Law of Accelerating Returns, life extension, Loebner Prize, machine translation, Mark Zuckerberg, Mars Rover, means of production, Mitch Kapor, Mustafa Suleyman, natural language processing, new economy, Nick Bostrom, OpenAI, opioid epidemic / opioid crisis, optical character recognition, paperclip maximiser, pattern recognition, phenotype, Productivity paradox, radical life extension, Ray Kurzweil, recommendation engine, Robert Gordon, Rodney Brooks, Sam Altman, self-driving car, seminal paper, sensor fusion, sentiment analysis, Silicon Valley, smart cities, social intelligence, sparse data, speech recognition, statistical model, stealth mode startup, stem cell, Stephen Hawking, Steve Jobs, Steve Wozniak, Steven Pinker, strong AI, superintelligent machines, synthetic biology, systems thinking, Ted Kaczynski, TED Talk, The Rise and Fall of American Growth, theory of mind, Thomas Bayes, Travis Kalanick, Turing test, universal basic income, Wall-E, Watson beat the top human players on Jeopardy!, women in the workforce, working-age population, workplace surveillance , zero-sum game, Zipcar

Rumelhart prize, the IJCAI award for research excellence, the Killam prize for Engineering, the IEEE Frank Rosenblatt medal, the IEEE James Clerk Maxwell Gold medal, the NEC C&C award, the BBVA award, and the NSERC Herzberg Gold Medal, which is Canada’s top award in science and engineering.” Chapter 5. NICK BOSTROM The concern is not that [an AGI] would hate or resent us for enslaving it, or that suddenly a spark of consciousness would arise and it would rebel, but rather that it would be very competently pursuing an objective that differs from what we really want. Then you get a future shaped in accordance with alien criteria. PROFESSOR, UNIVERSITY OF OXFORD AND DIRECTOR OF THE FUTURE OF HUMANITY INSTITUTE Nick Bostrom is widely recognized as one of the world’s top experts on superintelligence and the existential risks that AI and machine learning could potentially pose for humanity.

…

MARTIN FORD: You’ve written about the risks of creating a superintelligence—an entity that could emerge when an AGI system turns its energies toward improving itself, creating a recursive improvement loop that results in an intelligence that is vastly superior to humans. NICK BOSTROM: Yes, that’s one scenario and one problem, but there are other scenarios and other ways this transition to a machine intelligence era could unfold, and there are certainly, other problems one could be worried about. MARTIN FORD: One idea you’ve focused on especially is the control or alignment problem where a machine intelligence’s goals or values might result in outcomes that are harmful to humanity. Can you go into more detail on what that alignment problem, or control problem, is in layman’s terms? NICK BOSTROM: Well, one distinctive problem with very advanced AI systems that’s different from other technologies is that it presents not only the possibility of humans misusing the technology—that’s something we see with other technologies, of course—but also the possibility that the technology could misuse itself, as it were.

…

MARTIN FORD: Is solving that technical control problem, in terms of how to build a machine that remains aligned with the objective, what you’re working on at the Future of Humanity Institute, and what other think tanks like OpenAI and the Machine Intelligence Research Institute are focusing on? NICK BOSTROM: Yes, that’s right. We do have a group working on that, but we’re also working on other things. We also have a governance of AI group, that is focused on the governance problems related to advances in machine intelligence. MARTIN FORD: Do you think that think tanks like yours are an appropriate level of resource allocation for AI governance, or do you think that governments should jump into this at a larger scale? NICK BOSTROM: I think there could be more resources on AI safety. It’s not actually just us: DeepMind also has an AI safety group that we work with, but I do think more resources would be beneficial.

Surviving AI: The Promise and Peril of Artificial Intelligence by Calum Chace

3D printing, Ada Lovelace, AI winter, Airbnb, Alvin Toffler, artificial general intelligence, augmented reality, barriers to entry, basic income, bitcoin, Bletchley Park, blockchain, brain emulation, Buckminster Fuller, Charles Babbage, cloud computing, computer age, computer vision, correlation does not imply causation, credit crunch, cryptocurrency, cuban missile crisis, deep learning, DeepMind, dematerialisation, Demis Hassabis, discovery of the americas, disintermediation, don't be evil, driverless car, Elon Musk, en.wikipedia.org, epigenetics, Erik Brynjolfsson, everywhere but in the productivity statistics, Flash crash, friendly AI, Geoffrey Hinton, Google Glasses, hedonic treadmill, hype cycle, industrial robot, Internet of things, invention of agriculture, job automation, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Kevin Kelly, life extension, low skilled workers, machine translation, Mahatma Gandhi, means of production, mutually assured destruction, Neil Armstrong, Nicholas Carr, Nick Bostrom, paperclip maximiser, pattern recognition, peer-to-peer, peer-to-peer model, Peter Thiel, radical life extension, Ray Kurzweil, Robert Solow, Rodney Brooks, Second Machine Age, self-driving car, Silicon Valley, Silicon Valley ideology, Skype, South Sea Bubble, speech recognition, Stanislav Petrov, Stephen Hawking, Steve Jobs, strong AI, technological singularity, TED Talk, The future is already here, The Future of Employment, theory of mind, Turing machine, Turing test, universal basic income, Vernor Vinge, wage slave, Wall-E, zero-sum game

He added that many AI researchers are fearful of the backlash that could result if the general public became aware of the possibility that AGI could be created in a couple of decades, and what the implications could be. They are probably right to be concerned. Nick Bostrom’s meta-survey For his 2014 book “Superintelligence”, Nick Bostrom compiled four recent surveys of AI experts, which asked for estimates of the dates at which the probability of AGI being created reached 10%, 50% and 90%. The surveys were carried out in 2012 and 2013. It is unclear how many of the respondents were scientists actively engaged in AI research as opposed to philosophers and theoreticians, but all were either highly-cited authors of published academic work on AI, or professional academics attending conferences on the subject.

…

Rats which can choose between a direct stimulation of their brain’s pleasure centres or an item of food will starve themselves to death. Nick Bostrom calls this idea of causing great harm by mis-understanding the implications of an attempt to do great good “perverse instantiation”. Others might call it the law of unintended consequences, or Sod’s Law. The paperclip maximiser If somebody running a paperclip factory turns out to be the first person to create an AGI and it rapidly becomes a superintelligence, they are likely to have created an entity whose goal is to maximise the efficient production of paperclips. This has become the canonical example of what Nick Bostrom calls “infrastructure profusion”, the runaway train of superintelligence problems.

…

Technological unemployment could force us to adopt an entirely new economic structure, and the creation of superintelligence would be the biggest event in human history. Surviving AI is a first-class introduction to all of this. Brad Feld, co-founder Techstars The promises and perils of machine superintelligence are much debated nowadays. But between the complex and sometimes esoteric writings of AI theorists and academics like Nick Bostrom, and the popular-press prognostications of Elon Musk, Bill Gates and Stephen Hawking, there is something of a gap. Calum Chace’s Surviving AI bridges that gap perfectly. It provides a compact yet rigorous guide to all the major arguments and issues in the field. An excellent resource for those who are new to this topic.

The Doomsday Calculation: How an Equation That Predicts the Future Is Transforming Everything We Know About Life and the Universe by William Poundstone

Albert Einstein, anthropic principle, Any sufficiently advanced technology is indistinguishable from magic, Arthur Eddington, Bayesian statistics, behavioural economics, Benoit Mandelbrot, Berlin Wall, bitcoin, Black Swan, conceptual framework, cosmic microwave background, cosmological constant, cosmological principle, CRISPR, cuban missile crisis, dark matter, DeepMind, digital map, discounted cash flows, Donald Trump, Doomsday Clock, double helix, Dr. Strangelove, Eddington experiment, Elon Musk, Geoffrey Hinton, Gerolamo Cardano, Hans Moravec, heat death of the universe, Higgs boson, if you see hoof prints, think horses—not zebras, index fund, Isaac Newton, Jaron Lanier, Jeff Bezos, John Markoff, John von Neumann, Large Hadron Collider, mandelbrot fractal, Mark Zuckerberg, Mars Rover, Neil Armstrong, Nick Bostrom, OpenAI, paperclip maximiser, Peter Thiel, Pierre-Simon Laplace, Plato's cave, probability theory / Blaise Pascal / Pierre de Fermat, RAND corporation, random walk, Richard Feynman, ride hailing / ride sharing, Rodney Brooks, Ronald Reagan, Ronald Reagan: Tear down this wall, Sam Altman, Schrödinger's Cat, Search for Extraterrestrial Intelligence, self-driving car, Silicon Valley, Skype, Stanislav Petrov, Stephen Hawking, strong AI, tech billionaire, Thomas Bayes, Thomas Malthus, time value of money, Turing test

Today the list of existential threats is longer, and artificial intelligence (AI) rivals the bomb as a disturber of sleep. It is ironic that an undercurrent of pessimism pervades Silicon Valley, those few golden square miles that, more than any other part of the globe, have been enriched by Bayes’s theorem. Much of the ambivalence about AI has its roots in the work of Norwegian-born philosopher Nick Bostrom, now of Oxford. Bostrom did his doctoral thesis on the doomsday argument and the puzzles of self-sampling. He has been influential in proposing that self-sampling can be applied to diverse scientific questions. Today Bostrom is largely concerned with the risks that may be posed by AI. He believes that the challenge of coding human values into machines is more formidable than is generally appreciated.

…

Had the chosen ball been number 11 or higher, then he could have deduced, with ironclad certainty, that the draw was from the urn with a thousand balls. But there’s a ball number 7 in both urns. The evidence supplied by drawing a number 7 ball is circumstantial but not to be neglected by any fully reasonable party. “Rational belief is constrained,” Nick Bostrom wrote, “not only by chains of deduction but also by the rubber bands of probabilistic inference.” Doubting Thomas Bayes’s Essay found a most influential reader. He was Pierre-Simon Laplace (1749–1827), a French marquis, mathematician, physicist, astronomer, and atheist. Laplace slapped Bayes’s wreck of a paper into rigorous math.

…

Twelve Reasons Why the Doomsday Argument Is Wrong Is the Doomsday Argument correct?” asked Dutch physicist Dennis Dieks. “Nobody I told about the Argument was prepared to think so. But no one was able to offer a clear and convincing view about the nature of the error (if there is one.)” “I have encountered over a hundred objections against the doomsday argument,” wrote Nick Bostrom, “… many of them mutually inconsistent. It is as if the doomsday argument is so counterintuitive (or threatening?) that people reckon that every criticism must be valid.” “Given twenty seconds, many people believe they have found crushing objections,” wrote John Leslie. “At least a dozen times, I too dreamed up what seemed a crushing refutation of it.

The Precipice: Existential Risk and the Future of Humanity by Toby Ord

3D printing, agricultural Revolution, Albert Einstein, Alignment Problem, AlphaGo, Anthropocene, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, availability heuristic, biodiversity loss, Columbian Exchange, computer vision, cosmological constant, CRISPR, cuban missile crisis, decarbonisation, deep learning, DeepMind, defense in depth, delayed gratification, Demis Hassabis, demographic transition, Doomsday Clock, Dr. Strangelove, Drosophila, effective altruism, Elon Musk, Ernest Rutherford, global pandemic, Goodhart's law, Hans Moravec, Herman Kahn, Higgs boson, Intergovernmental Panel on Climate Change (IPCC), Isaac Newton, James Watt: steam engine, Large Hadron Collider, launch on warning, Mark Zuckerberg, Mars Society, mass immigration, meta-analysis, Mikhail Gorbachev, mutually assured destruction, Nash equilibrium, Nick Bostrom, Norbert Wiener, nuclear winter, ocean acidification, OpenAI, p-value, Peter Singer: altruism, planetary scale, power law, public intellectual, race to the bottom, RAND corporation, Recombinant DNA, Ronald Reagan, self-driving car, seminal paper, social discount rate, Stanislav Petrov, Stephen Hawking, Steven Pinker, Stewart Brand, supervolcano, survivorship bias, synthetic biology, tacit knowledge, the scientific method, Tragedy of the Commons, uranium enrichment, William MacAskill

The End of the World: The Science and Ethics of Human Extinction. A landmark book that broadened the discussion from nuclear risk to all risks of human extinction, cataloging the threats and exploring new philosophical angles. Nick Bostrom (2002). “Existential Risks: Analyzing Human Extinction Scenarios.” Established the concept of existential risk and introduced many of the most important ideas. Yet mainly of historic interest, for it is superseded by his 2013 paper below. Nick Bostrom (2003). “Astronomical Waste: The Opportunity Cost of Delayed Technological Development.” Explored the limits of what humans might be able to achieve in the future, suggesting that it is of immense importance to accelerate the arrival of the ultimate state of our civilization by even a tiny amount, yet that even this is overshadowed by the importance of increasing the chance we get there at all.

…

We know of almost no limits to what we might ultimately achieve. Human extinction would foreclose our future. It would destroy our potential. It would eliminate all possibilities but one: a world bereft of human flourishing. Extinction would bring about this failed world and lock it in forever—there would be no coming back. The philosopher Nick Bostrom showed that extinction is not the only way this could happen: there are other catastrophic outcomes in which we lose not just the present, but all our potential for the future.2 Consider a world in ruins: an immense catastrophe has triggered a global collapse of civilization, reducing humanity to a pre-agricultural state.

…

But this history shows that existential risk is capable of rousing major global concern, from the elite to the grass roots. Modern thinking on existential risk can be traced through John Leslie, whose 1996 book The End of the World broadened the focus from nuclear war to human extinction in general. After reading Leslie’s work, Nick Bostrom took this a step further: identifying and analyzing the broader class of existential risks that are the focus of this book. Our moral and political traditions have been built over thousands of years. Their focus is thus mostly on timeless issues that have been with us throughout our history. It takes time to incorporate the new possibilities that our age opens up, even when those possibilities are of immense moral significance.

Smarter Than Us: The Rise of Machine Intelligence by Stuart Armstrong

artificial general intelligence, brain emulation, effective altruism, Flash crash, friendly AI, machine translation, Nick Bostrom, shareholder value, Turing test

Some effort has been made to make the AI transition safer. Kudos must be given to Eliezer Yudkowsky and Nick Bostrom, who saw and understood the risks early on. Yudkowsky uses the term “Friendly AI” to describe an AI which does what we want even as it improves its own intelligence. In 2000 he cofounded an organization now called the Machine Intelligence Research Institute (MIRI), which holds math research workshops tackling open problems in Friendly AI theory. (MIRI also commissioned and published this book.) Meanwhile, Nick Bostrom founded the Future of Humanity Institute (FHI), a research group within the University of Oxford.

…

CC BY-NC-SA 3.0 isbn-10: 193931108X isbn-13: 978-1-939311-08-5 (mobi) The Machine Intelligence Research Institute gratefully acknowledges the generous support of all those involved in the publication of this book. Cover photo credit: Google/Connie Zhou. Acknowledgments I wish to acknowledge the help and support of the Future of Humanity Institute, the Oxford Martin School, and the Machine Intelligence Research Institute, as well as the individual advice of Nick Bostrom, Seán Ó hÉigeartaigh, Eliezer Yudkowsky, Kaj Sotala, Luke Muehlhauser, Vincent C. Müller, Anders Sandberg, Lisa Makros, Daniel Dewey, Eric Drexler, Nick Beckstead, Cathy Douglass, and Miriam, Maia, and Kipper Armstrong. Contents Acknowledgments 1. Terminator versus the AI 2. Strength versus Intelligence 3.

…

See MIRI’s work on the fragility of values and FHI’s work on the problem of containing oracles: Luke Muehlhauser and Louie Helm, “The Singularity and Machine Ethics,” in Singularity Hypotheses: A Scientific and Philosophical Assessment, ed. Amnon Eden et al., The Frontiers Collection (Berlin: Springer, 2012); Stuart Armstrong, Anders Sandberg, and Nick Bostrom, “Thinking Inside the Box: Controlling and Using an Oracle AI,” Minds and Machines 22, no. 4 (2012): 299–324, doi:10.1007/s11023-012-9282-2. 2. Stephen M. Omohundro, “The Basic AI Drives,” in Artificial General Intelligence 2008: Proceedings of the First AGI Conference, Frontiers in Artificial Intelligence and Applications 171 (Amsterdam: IOS, 2008), 483–492. 3.

Artificial You: AI and the Future of Your Mind by Susan Schneider

artificial general intelligence, brain emulation, deep learning, Elon Musk, Extropian, heat death of the universe, hive mind, life extension, megastructure, Nick Bostrom, pattern recognition, precautionary principle, radical life extension, Ray Kurzweil, Search for Extraterrestrial Intelligence, silicon-based life, Stephen Hawking, superintelligent machines, technological singularity, TED Talk, The Coming Technological Singularity, theory of mind, traumatic brain injury, Turing machine, Turing test, Whole Earth Review, wikimedia commons

In the long term, the tables may turn on humans, and the problem may not be what we could do to harm AIs, but what AI might do to harm us. Indeed, some suspect that synthetic intelligence will be the next phase in the evolution of intelligence on Earth. You and I, how we live and experience the world right now, are just an intermediate step to AI, a rung on the evolutionary ladder. For instance, Stephen Hawking, Nick Bostrom, Elon Musk, Max Tegmark, Bill Gates, and many others have raised “the control problem,” the problem of how humans can control their own AI creations, if the AIs outsmart us.2 Suppose we create an AI that has human-level intelligence. With self-improvement algorithms, and with rapid computations, it could quickly discover ways to become vastly smarter than us, becoming a superintelligence—that is, an AI that outthinks us in every domain.

…

Because it is superintelligent, we probably can’t control it. It could, in principle, render us extinct. This is only one way that synthetic beings could supplant organic intelligences; alternatively, humans may merge with AI through cumulatively significant brain enhancements. The control problem has made world news, fueled by Nick Bostrom’s recent bestseller: Superintelligence: Paths, Dangers and Strategies.3 What is missed, however, is that consciousness could be central to how AI values us. Using its own subjective experience as a springboard, superintelligent AI could recognize in us the capacity for conscious experience. After all, to the extent we value the lives of nonhuman animals, we tend to value them because we feel an affinity of consciousness—thus most of us recoil from killing a chimp, but not from eating an orange.

…

I don’t know whether many of those who publicize the idea of a mind-machine merger, such as Elon Musk and Michio Kaku, have considered these classic positions on personal identity. But they should. It is a bad idea to ignore this debate. One could be dismayed, at some later point, to learn that a technology one advocated actually had a tremendously negative impact on human flourishing. In any case, both Kurzweil and Nick Bostrom have considered the issue in their work. They, like many other transhumanists, adopt a novel and intriguing version of the psychological continuity view; in particular, they adopt a computational, or patternist, account of continuity. ARE YOU A SOFTWARE PATTERN? Patternism’s point of departure is the computational theory of mind, which I introduced earlier.

An Optimist's Tour of the Future by Mark Stevenson

23andMe, Albert Einstein, Alvin Toffler, Andy Kessler, Apollo 11, augmented reality, bank run, Boston Dynamics, carbon credits, carbon footprint, carbon-based life, clean water, computer age, decarbonisation, double helix, Douglas Hofstadter, Dr. Strangelove, Elon Musk, flex fuel, Ford Model T, Future Shock, Great Leap Forward, Gregor Mendel, Gödel, Escher, Bach, Hans Moravec, Hans Rosling, Intergovernmental Panel on Climate Change (IPCC), Internet of things, invention of agriculture, Isaac Newton, Jeff Bezos, Kevin Kelly, Law of Accelerating Returns, Leonard Kleinrock, life extension, Louis Pasteur, low earth orbit, mutually assured destruction, Naomi Klein, Nick Bostrom, off grid, packet switching, peak oil, pre–internet, private spaceflight, radical life extension, Ray Kurzweil, Richard Feynman, Rodney Brooks, Scaled Composites, self-driving car, Silicon Valley, smart cities, social intelligence, SpaceShipOne, stem cell, Stephen Hawking, Steven Pinker, Stewart Brand, strong AI, synthetic biology, TED Talk, the scientific method, Virgin Galactic, Wall-E, X Prize

And if life expectancies keep creeping up the way they have been, fifty years from now I could have up to another twelve years on the planet. It’s not infeasible I’ll make it past a hundred. I find this encouraging: I’m still single with no kids, but it appears there’s plenty of time to raise a family, maybe even understand cricket. Nick Bostrom, founder of Oxford University’s Future of Humanity Institute, agrees. In fact, he thinks I could live not for a hundred years but for thousands. And he’s not joking. Bostrom is an advocate for transhumanism. Described as ‘the world’s most dangerous idea,’ the concept got its modern name in Religion Without Revelation, written in 1927 by Julian Huxley.

…

When no one raises their hand he continues, ‘I put it to you that the main reason we think malaria is a bad thing is because of a characteristic it shares with ageing – and here is that characteristic’ – at which point he shows a slide bearing the words ‘Because it kills people!!!!!!’ ‘The only real difference is that ageing kills considerably more people.’ De Grey proceeds to compare ageing to fox hunting, claiming they are both ‘traditional,’ ‘keep the numbers down’ and, his punch line, ‘fundamentally barbaric.’ At the same conference, Nick Bostrom later stated, ‘Death is a big problem. If you look at the statistics the odds are not very favourable. So far most people who have lived have also died.’ But banishing ageing is only one half of the transhumanists’ truly radical agenda. They also advocate a new vision of the human being: the transhuman – a human not only saved from ageing, but enhanced beyond our current ‘biological limitations.’

…

Unlike natural evolution, which requires centuries to fumble an improvement, our man-made technologies quickly keep on doubling in power and halving in price, ushering the prospect of the ‘disabled’ out-performing what we now consider the able-bodied. Will I be buying enhanced cochlear implants in a few years the way I bought a mobile phone in the nineties? Will I buy a new body part grown from my own stem cells to keep me young? How far can this go? How long can I live? How enhanced could I be? Nick Bostrom suggests that the answers to these last two questions could respectively be ‘a very long time’ and ‘as much as you like.’ Take, for instance, the idea of ‘longevity escape velocity,’ seen by many transhumanists as one of the more convincing arguments for the prospect of immortality. This is the proposition that instead of our life expectancies gaining a quarter of a year for every year that passes, we reach a point where medical advances raise our life expectancy by over a year for every year that goes by.

The Most Good You Can Do: How Effective Altruism Is Changing Ideas About Living Ethically by Peter Singer

Albert Einstein, clean water, cognitive load, corporate social responsibility, correlation does not imply causation, David Brooks, effective altruism, en.wikipedia.org, Flynn Effect, hedonic treadmill, Large Hadron Collider, Nick Bostrom, Peter Singer: altruism, purchasing power parity, randomized controlled trial, stem cell, Steven Pinker, TED Talk, trolley problem, William MacAskill, young professional

National Aeronautics and Space Administration, Near Earth Object Program, Torino Hazard Scale, http://neo.jpl.nasa.gov/torino_scale.html. 2. Nick Bostrom, “Existential Risks: Analyzing Human Extinction Scenarios And Related Hazards,” Journal of Evolution and Technology 9 (2002), available at www.jetpress.org/volume9/risks.html. 3. Hans Kristensen and Robert Norris, “Global Nuclear Weapons Inventories, 1945–2013,” Bulletin of the Atomic Scientists 69 (2013): 75–81. 4. Binyamin Appelbaum, “As U.S. Agencies Put More Value on a Life, Businesses Fret,” New York Times, February 16, 2011. 5. Derek Parfit, Reasons and Persons (Oxford: Clarendon Press, 1984), 453–54. 6. Nick Bostrom, “Existential Risk Prevention as Global Priority,” Global Policy 4 (2013): 16. 7.

…

Nick Bostrom, “Existential Risk Prevention as Global Priority,” Global Policy 4 (2013): 16. 7. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), 103; see also Bostrom, “Existential Risk Prevention as Global Priority,” 18–19. Nick Beckstead argues for a similar position in “The Overwhelming Importance of Shaping the Far Future” (PhD diss., Rutgers University, 2013). 8. Nick Bostrom, Existential Risk FAQ, version 1.2 (2013), no. 9, http://www.existential-risk.org/faq.html. 9. Henry Sidgwick, The Methods of Ethics, 7th ed. (London: Macmillan, 1907), 415. 10. See Peter Singer, Practical Ethics, 3d ed.

…

I very much hope that his gift will do more good than anything else he could have done with that amount of money, although I persist in thinking that, at the time he made it, he could not reasonably have predicted so fortunate an outcome. Many people have read drafts of the book, or parts of it, and offered helpful comments or responded to my queries. I want to thank, in particular, Anthony Appiah, Paul Bloom, Jon Bockman and Allison Smith for Animal Charity Evaluators, Paul van den Bosch for Give A Kidney, Nick Bostrom, Richard Butler-Bowdon, Di Franks for Living Kidney Donation, Holden Karnofsky for GiveWell, Katarzyna de Lazari-Radek, Peter Hurford, Michael Liffman, Will MacAskill, Yaw Nyarko, Caleb Ontiveros, Toby Ord, Theron Pummer, Rob Reich, Susanne Roff, Agata Sagan, and Aleksandra Taranow. Special thanks to Mona Fixdal, whose splendid assistance in the preparation of my online course “Practical Ethics” made it possible for me to devote more time to writing this book.

Human Compatible: Artificial Intelligence and the Problem of Control by Stuart Russell

3D printing, Ada Lovelace, AI winter, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Alfred Russel Wallace, algorithmic bias, AlphaGo, Andrew Wiles, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, augmented reality, autonomous vehicles, basic income, behavioural economics, Bletchley Park, blockchain, Boston Dynamics, brain emulation, Cass Sunstein, Charles Babbage, Claude Shannon: information theory, complexity theory, computer vision, Computing Machinery and Intelligence, connected car, CRISPR, crowdsourcing, Daniel Kahneman / Amos Tversky, data science, deep learning, deepfake, DeepMind, delayed gratification, Demis Hassabis, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, Ernest Rutherford, fake news, Flash crash, full employment, future of work, Garrett Hardin, Geoffrey Hinton, Gerolamo Cardano, Goodhart's law, Hans Moravec, ImageNet competition, Intergovernmental Panel on Climate Change (IPCC), Internet of things, invention of the wheel, job automation, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John Nash: game theory, John von Neumann, Kenneth Arrow, Kevin Kelly, Law of Accelerating Returns, luminiferous ether, machine readable, machine translation, Mark Zuckerberg, multi-armed bandit, Nash equilibrium, Nick Bostrom, Norbert Wiener, NP-complete, OpenAI, openstreetmap, P = NP, paperclip maximiser, Pareto efficiency, Paul Samuelson, Pierre-Simon Laplace, positional goods, probability theory / Blaise Pascal / Pierre de Fermat, profit maximization, RAND corporation, random walk, Ray Kurzweil, Recombinant DNA, recommendation engine, RFID, Richard Thaler, ride hailing / ride sharing, Robert Shiller, robotic process automation, Rodney Brooks, Second Machine Age, self-driving car, Shoshana Zuboff, Silicon Valley, smart cities, smart contracts, social intelligence, speech recognition, Stephen Hawking, Steven Pinker, superintelligent machines, surveillance capitalism, Thales of Miletus, The Future of Employment, The Theory of the Leisure Class by Thorstein Veblen, Thomas Bayes, Thorstein Veblen, Tragedy of the Commons, transport as a service, trolley problem, Turing machine, Turing test, universal basic income, uranium enrichment, vertical integration, Von Neumann architecture, Wall-E, warehouse robotics, Watson beat the top human players on Jeopardy!, web application, zero-sum game

If human happiness requires being in the top 1 percent, then 99 percent of humans are going to be unhappy, even when the bottom 1 percent has an objectively splendid lifestyle.60 It will be important, then, for our cultures to gradually down-weight pride and envy as central elements of perceived self-worth. As Nick Bostrom puts it at the end of his book Superintelligence, success in AI will yield “a civilizational trajectory that leads to a compassionate and jubilant use of humanity’s cosmic endowment.” If we fail to take advantage of what AI has to offer, we will have only ourselves to blame. 4 MISUSES OF AI A compassionate and jubilant use of humanity’s cosmic endowment sounds wonderful, but we also have to reckon with the rapid rate of innovation in the malfeasance sector.

…

If there are no humans and no other conscious entities whose subjective experience matters to us, there is nothing of value occurring. 6 THE NOT-SO-GREAT AI DEBATE The implications of introducing a second intelligent species onto Earth are far-reaching enough to deserve hard thinking.”1 So ended The Economist magazine’s review of Nick Bostrom’s Superintelligence. Most would interpret this as a classic example of British understatement. Surely, you might think, the great minds of today are already doing this hard thinking—engaging in serious debate, weighing up the risks and benefits, seeking solutions, ferreting out loopholes in solutions, and so on.

…

It’s too soon to worry about it It’s common to see sober-minded people seeking to assuage public concerns by pointing out that because human-level AI is not likely to arrive for several decades, there is nothing to worry about. For example, the AI100 report says there is “no cause for concern that AI is an imminent threat to humankind.” This argument fails on two counts. The first is that it attacks a straw man. The reasons for concern are not predicated on imminence. For example, Nick Bostrom writes in Superintelligence, “It is no part of the argument in this book that we are on the threshold of a big breakthrough in artificial intelligence, or that we can predict with any precision when such a development might occur.” The second is that a long-term risk can still be cause for immediate concern.

The Transhumanist Reader by Max More, Natasha Vita-More

"World Economic Forum" Davos, 23andMe, Any sufficiently advanced technology is indistinguishable from magic, artificial general intelligence, augmented reality, Bill Joy: nanobots, bioinformatics, brain emulation, Buckminster Fuller, cellular automata, clean water, cloud computing, cognitive bias, cognitive dissonance, combinatorial explosion, Computing Machinery and Intelligence, conceptual framework, Conway's Game of Life, cosmological principle, data acquisition, discovery of DNA, Douglas Engelbart, Drosophila, en.wikipedia.org, endogenous growth, experimental subject, Extropian, fault tolerance, Flynn Effect, Francis Fukuyama: the end of history, Frank Gehry, friendly AI, Future Shock, game design, germ theory of disease, Hans Moravec, hypertext link, impulse control, index fund, John von Neumann, joint-stock company, Kevin Kelly, Law of Accelerating Returns, life extension, lifelogging, Louis Pasteur, Menlo Park, meta-analysis, moral hazard, Network effects, Nick Bostrom, Norbert Wiener, pattern recognition, Pepto Bismol, phenotype, positional goods, power law, precautionary principle, prediction markets, presumed consent, Project Xanadu, public intellectual, radical life extension, Ray Kurzweil, reversible computing, RFID, Ronald Reagan, scientific worldview, silicon-based life, Singularitarianism, social intelligence, stem cell, stochastic process, superintelligent machines, supply-chain management, supply-chain management software, synthetic biology, systems thinking, technological determinism, technological singularity, Ted Nelson, telepresence, telepresence robot, telerobotics, the built environment, The Coming Technological Singularity, the scientific method, The Wisdom of Crowds, transaction costs, Turing machine, Turing test, Upton Sinclair, Vernor Vinge, Von Neumann architecture, VTOL, Whole Earth Review, women in the workforce, zero-sum game

Bostrom, Nick and Sandberg, Anders (2009) “The Wisdom of Nature: An Evolutionary Heuristic for Human Enhance­ment.” In Julian Savulescu and Nick Bostrom, eds., Human Enhancement. Oxford: Oxford University Press, pp. 375–416. Cakic, Vince (2009) “Smart Drugs for Cognitive Enhancement: Ethical and Pragmatic Considerations in the Era of Cosmetic Neurology.” Journal of Medical Ethics 35 (October 29), pp. 611–615. Caplan, Arthur L. (2009) “Good, Better, or Best.” In Julian Savulescu and Nick Bostrom, eds., Human Enhancement. Oxford: Oxford University Press, pp. 199–209. Glover, Jonathan (1984) What Sort of People Should There Be?

…

Strong Superhumanity and the Best We Can Ask For 36 An Overview of Models of Technological Singularity Introduction Definitions of Technological Singularity Models Accelerating Change Discussion 37 A Critical Discussion of Vinge’s Singularity Concept Comment by David Brin: Singularities Comment by Damien Broderick Comment by Nick Bostrom: Singularity and Predictability Comment by Alexander Chislenko: Singularity as a Process, and the Future Beyond Comment by Robin Hanson: Some Skepticism Comment by Max More: Singularity Meets Economy Comment by Michael Nielsen Comment by Anders Sandberg: Singularity and the Growth of Differences Part IX The World’s Most Dangerous Idea 38 The Great Transition What is Transhumanism?

…

Russell Blackford, PhD, is Editor-in-Chief, Journal of Evolution & Technology. He authored Freedom of Religion and the Secular State (Wiley-Blackwell, 2012) and An Evil Hour (I Books, 2003); and co-authored with Van Ikin and Sean McMullen Strange Constellations: A History of Australian Science Fiction (Praeger, 1999). Nick Bostrom, PhD, is Director, Future of Humanity Institute, Oxford University. He has ­written numerous papers and authored Anthropic Bias (Routledge, 2010); and co-edited with Julian Savulescu Human Enhancement (Oxford University Press, 2011) and Global Catastrophic Risks with Milan Cirkovic (Oxford University Press, 2011).

The Simulation Hypothesis by Rizwan Virk

3D printing, Albert Einstein, AlphaGo, Apple II, artificial general intelligence, augmented reality, Benoit Mandelbrot, bioinformatics, butterfly effect, Colossal Cave Adventure, Computing Machinery and Intelligence, DeepMind, discovery of DNA, Dmitri Mendeleev, Elon Musk, en.wikipedia.org, Ernest Rutherford, game design, Google Glasses, Isaac Newton, John von Neumann, Kickstarter, mandelbrot fractal, Marc Andreessen, Minecraft, natural language processing, Nick Bostrom, OpenAI, Pierre-Simon Laplace, Plato's cave, quantum cryptography, quantum entanglement, Ralph Waldo Emerson, Ray Kurzweil, Richard Feynman, Schrödinger's Cat, Search for Extraterrestrial Intelligence, Silicon Valley, Stephen Hawking, Steve Jobs, Steve Wozniak, technological singularity, TED Talk, time dilation, Turing test, Vernor Vinge, Zeno's paradox

The fundamental question raised by the simulation hypothesis is this: Are we all actually characters living inside some kind of giant, massively multiplayer online video game, a simulated reality that is so well rendered that we cannot distinguish it from “physical reality”? Although Oxford philosopher Nick Bostrom coined the term the Simulation Argument in a landmark paper in 2003, the idea of living in a simulated reality has been around for a long time in science, religion and fiction. The reality of the world around us is something that philosophers have debated for some time. Thousands of years ago in The Republic, Plato described his analogy of the cave.

…

He conjectured that if video game technology continued its rapid pace of improvement, then it was inevitable that we would be able to create hyper-realistic simulations that would be indistinguishable from physical reality. He concluded that, assuming technology keeps developing as it has in the past, the chance that we aren’t in a simulation was “one in billions.” Oxford’s Nick Bostrom, in his 2003 paper which popularized the Simulation Argument, refers to a species which is able to build these hyper-realistic simulations as becoming “post-human.” I like to say that such a civilization has passed the “Simulation Point.” This would be the point in a technological civilization’s development when it possesses the ability to create hyper-realistic simulations.

…

There are metaphysical as well as technical and philosophical questions that go along with this, which will need to be resolved as we travel through these stages on the road to the simulation point. If a civilization has mastered these two stages, then it has pretty much reached the point of being able to create a Matrix-like reality, and has reached the simulation point. Next, we will explore some of the implications of reaching this point, and go through Nick Bostrom’s original argument that we may be living in a simulation all along as simulated entities. Chapter 4 Stage 11: The Simulation Point, Ancestor Simulations and Beyond At some point after completing the previous stages, a civilization should have reached the simulation point—the point at which it can create simulations that are virtually indistinguishable from a base physical reality.

What to Think About Machines That Think: Today's Leading Thinkers on the Age of Machine Intelligence by John Brockman

Adam Curtis, agricultural Revolution, AI winter, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, algorithmic trading, Anthropocene, artificial general intelligence, augmented reality, autism spectrum disorder, autonomous vehicles, backpropagation, basic income, behavioural economics, bitcoin, blockchain, bread and circuses, Charles Babbage, clean water, cognitive dissonance, Colonization of Mars, complexity theory, computer age, computer vision, constrained optimization, corporate personhood, cosmological principle, cryptocurrency, cuban missile crisis, Danny Hillis, dark matter, data science, deep learning, DeepMind, Demis Hassabis, digital capitalism, digital divide, digital rights, discrete time, Douglas Engelbart, driverless car, Elon Musk, Emanuel Derman, endowment effect, epigenetics, Ernest Rutherford, experimental economics, financial engineering, Flash crash, friendly AI, functional fixedness, global pandemic, Google Glasses, Great Leap Forward, Hans Moravec, hive mind, Ian Bogost, income inequality, information trail, Internet of things, invention of writing, iterative process, James Webb Space Telescope, Jaron Lanier, job automation, Johannes Kepler, John Markoff, John von Neumann, Kevin Kelly, knowledge worker, Large Hadron Collider, lolcat, loose coupling, machine translation, microbiome, mirror neurons, Moneyball by Michael Lewis explains big data, Mustafa Suleyman, natural language processing, Network effects, Nick Bostrom, Norbert Wiener, paperclip maximiser, pattern recognition, Peter Singer: altruism, phenotype, planetary scale, Ray Kurzweil, Recombinant DNA, recommendation engine, Republic of Letters, RFID, Richard Thaler, Rory Sutherland, Satyajit Das, Search for Extraterrestrial Intelligence, self-driving car, sharing economy, Silicon Valley, Skype, smart contracts, social intelligence, speech recognition, statistical model, stem cell, Stephen Hawking, Steve Jobs, Steven Pinker, Stewart Brand, strong AI, Stuxnet, superintelligent machines, supervolcano, synthetic biology, systems thinking, tacit knowledge, TED Talk, the scientific method, The Wisdom of Crowds, theory of mind, Thorstein Veblen, too big to fail, Turing machine, Turing test, Von Neumann architecture, Watson beat the top human players on Jeopardy!, We are as Gods, Y2K

Should I be bothered by the prospect of thinking machines? Probably. Certainly Nick Bostrom thinks I should. Our focus on getting computers to exhibit human-level intelligence is, he thinks, misguided. We view machines that can pass the Turing Test as the ultimate destination of Doug Engelbart’s quest. But Bostrom thinks that passing the test is just a waypoint on the road to something much more worrying. “The train,” he says, “might not pause or even decelerate at Humanville Station. It is likely to swoosh right by.”4 He’s right: I should be careful what I wish for. IT’S STILL EARLY DAYS NICK BOSTROM Professor, Oxford University; director, Future of Humanity Institute, Oxford Martin School; author, Superintelligence: Paths, Dangers, Strategies First, what I think about humans who think about machines that think: I think that for the most part we’re too quick to form an opinion on this difficult topic.

…

BROOKS Mistaking Performance for Competence TERRENCE J. SEJNOWSKI AI Will Make You Smarter SETH LLOYD Shallow Learning CARLO ROVELLI Natural Creatures of a Natural World FRANK WILCZEK Three Observations on Artificial Intelligence JOHN NAUGHTON When I Say “Bruno Latour,” I Don’t Mean “Banana Till” NICK BOSTROM It’s Still Early Days DONALD D. HOFFMAN Evolving AI ROGER SCHANK Machines That Think Are in the Movies JUAN ENRIQUEZ Head Transplants? ESTHER DYSON AI/AL TOM GRIFFITHS Brains and Other Thinking Machines MARK PAGEL They’ll Do More Good Than Harm ROBERT PROVINE Keeping Them on a Leash SUSAN BLACKMORE The Next Replicator TIM O’REILLY What If We’re the Microbiome of the Silicon AI?

…

PREFACE: THE 2015 EDGE QUESTION In recent years, the 1980s-era philosophical discussions about artificial intelligence (AI)—whether computers can “really” think, be conscious, and so on—have led to new conversations about how we should deal with the forms of artificial intelligence that many argue have already been implemented. These AIs, if they achieve “superintelligence” (per Nick Bostrom’s 2014 book of that name), could pose existential risks, leading to what Martin Rees has termed “our final hour.” Stephen Hawking recently made international headlines when he told the BBC that “the development of full artificial intelligence could spell the end of the human race.” THE EDGE QUESTION—2015 WHAT DO YOU THINK ABOUT MACHINES THAT THINK?

Our Final Invention: Artificial Intelligence and the End of the Human Era by James Barrat

AI winter, air gap, AltaVista, Amazon Web Services, artificial general intelligence, Asilomar, Automated Insights, Bayesian statistics, Bernie Madoff, Bill Joy: nanobots, Bletchley Park, brain emulation, California energy crisis, cellular automata, Chuck Templeton: OpenTable:, cloud computing, cognitive bias, commoditize, computer vision, Computing Machinery and Intelligence, cuban missile crisis, Daniel Kahneman / Amos Tversky, Danny Hillis, data acquisition, don't be evil, drone strike, dual-use technology, Extropian, finite state, Flash crash, friendly AI, friendly fire, Google Glasses, Google X / Alphabet X, Hacker News, Hans Moravec, Isaac Newton, Jaron Lanier, Jeff Hawkins, John Markoff, John von Neumann, Kevin Kelly, Law of Accelerating Returns, life extension, Loebner Prize, lone genius, machine translation, mutually assured destruction, natural language processing, Neil Armstrong, Nicholas Carr, Nick Bostrom, optical character recognition, PageRank, PalmPilot, paperclip maximiser, pattern recognition, Peter Thiel, precautionary principle, prisoner's dilemma, Ray Kurzweil, Recombinant DNA, Rodney Brooks, rolling blackouts, Search for Extraterrestrial Intelligence, self-driving car, semantic web, Silicon Valley, Singularitarianism, Skype, smart grid, speech recognition, statistical model, stealth mode startup, stem cell, Stephen Hawking, Steve Jobs, Steve Jurvetson, Steve Wozniak, strong AI, Stuxnet, subprime mortgage crisis, superintelligent machines, technological singularity, The Coming Technological Singularity, Thomas Bayes, traveling salesman, Turing machine, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!, zero day

We don’t hate mice or monkeys, yet we treat them cruelly. Superintelligent AI won’t have to hate us to destroy us. After intelligent machines have already been built and man has not been wiped out, perhaps we can afford to anthropomorphize. But here on the cusp of creating AGI, it is a dangerous habit. Oxford University ethicist Nick Bostrom puts it like this: A prerequisite for having a meaningful discussion of superintelligence is the realization that superintelligence is not just another technology, another tool that will add incrementally to human capabilities. Superintelligence is radically different. This point bears emphasizing, for anthropomorphizing superintelligence is a most fecund source of misconceptions.

…

We’re unlikely to survive an introduction as abrupt as nuclear fission’s. Chapter Two The Two-Minute Problem Our approach to existential risks cannot be one of trial-and-error. There is no opportunity to learn from errors. The reactive approach—see what happens, limit damages, and learn from experience—is unworkable. —Nick Bostrom, faculty of Philosophy, Oxford University The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else. —Eliezer Yudkowsky, research fellow, Machine Intelligence Research Institute Artificial superintelligence does not yet exist, nor does artificial general intelligence, the kind that can learn like we do and will in many senses match and exceed most human intelligence.

…

That is, we don’t want an AI that meets our short-term goals—please save us from hunger—with solutions detrimental in the long term—by roasting every chicken on earth—or with solutions to which we’d object—by killing us after our next meal. As an example of unintended consequences, Oxford University ethicist Nick Bostrom suggests the hypothetical “paper clip maximizer.” In Bostrom’s scenario, a thoughtlessly programmed superintelligence whose programmed goal is to manufacture paper clips does exactly as it is told without regard to human values. It all goes wrong because it sets about “transforming first all of earth and then increasing portions of space into paper clip manufacturing facilities.”

Possible Minds: Twenty-Five Ways of Looking at AI by John Brockman

AI winter, airport security, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Alignment Problem, AlphaGo, artificial general intelligence, Asilomar, autonomous vehicles, basic income, Benoit Mandelbrot, Bill Joy: nanobots, Bletchley Park, Buckminster Fuller, cellular automata, Claude Shannon: information theory, Computing Machinery and Intelligence, CRISPR, Daniel Kahneman / Amos Tversky, Danny Hillis, data science, David Graeber, deep learning, DeepMind, Demis Hassabis, easy for humans, difficult for computers, Elon Musk, Eratosthenes, Ernest Rutherford, fake news, finite state, friendly AI, future of work, Geoffrey Hinton, Geoffrey West, Santa Fe Institute, gig economy, Hans Moravec, heat death of the universe, hype cycle, income inequality, industrial robot, information retrieval, invention of writing, it is difficult to get a man to understand something, when his salary depends on his not understanding it, James Watt: steam engine, Jeff Hawkins, Johannes Kepler, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Kevin Kelly, Kickstarter, Laplace demon, Large Hadron Collider, Loebner Prize, machine translation, market fundamentalism, Marshall McLuhan, Menlo Park, military-industrial complex, mirror neurons, Nick Bostrom, Norbert Wiener, OpenAI, optical character recognition, paperclip maximiser, pattern recognition, personalized medicine, Picturephone, profit maximization, profit motive, public intellectual, quantum cryptography, RAND corporation, random walk, Ray Kurzweil, Recombinant DNA, Richard Feynman, Rodney Brooks, self-driving car, sexual politics, Silicon Valley, Skype, social graph, speech recognition, statistical model, Stephen Hawking, Steven Pinker, Stewart Brand, strong AI, superintelligent machines, supervolcano, synthetic biology, systems thinking, technological determinism, technological singularity, technoutopianism, TED Talk, telemarketer, telerobotics, The future is already here, the long tail, the scientific method, theory of mind, trolley problem, Turing machine, Turing test, universal basic income, Upton Sinclair, Von Neumann architecture, Whole Earth Catalog, Y2K, you are the product, zero-sum game

THE EVOLVING AI NARRATIVE Things have changed—and they remain the same. Now AI is everywhere. We have the Internet. We have our smartphones. The founders of the dominant companies—the companies that hold “the whip that lashes us”—have net worths of $65 billion, $90 billion, $130 billion. High-profile individuals such as Elon Musk, Nick Bostrom, Martin Rees, Eliezer Yudkowsky, and the late Stephen Hawking have issued dire warnings about AI, resulting in the ascendancy of well-funded institutes tasked with promoting “Nice AI.” But will we, as a species, be able to control a fully realized, unsupervised, self-improving AI? Wiener’s warnings and admonitions in The Human Use of Human Beings are now very real, and they need to be looked at anew by researchers at the forefront of the AI revolution.

…

In this paper, the message is even clearer: If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere . . . we had better be quite sure that the purpose put into the machine is the purpose which we really desire. In my view, this is the source of the existential risk from superintelligent AI cited in recent years by such observers as Elon Musk, Bill Gates, Stephen Hawking, and Nick Bostrom. PUTTING PURPOSES INTO MACHINES The goal of AI research has been to understand the principles underlying intelligent behavior and to build those principles into machines that can then exhibit such behavior. In the 1960s and 1970s, the prevailing theoretical notion of intelligence was the capacity for logical reasoning, including the ability to derive plans of action guaranteed to achieve a specified goal.

…

The AI100 report, for example, assures us, “Contrary to the more fantastic predictions for AI in the popular press, the Study Panel found no cause for concern that AI is an imminent threat to humankind.” This argument simply misstates the reasons for concern, which are not predicated on imminence. In his 2014 book, Superintelligence: Paths, Dangers, Strategies, Nick Bostrom, for one, writes, “It is no part of the argument in this book that we are on the threshold of a big breakthrough in artificial intelligence, or that we can predict with any precision when such a development might occur.” You’re just a Luddite. It’s an odd definition of Luddite that includes Turing, Wiener, Minsky, Musk, and Gates, who rank among the most prominent contributors to technological progress in the 20th and 21st centuries.* Furthermore, the epithet represents a complete misunderstanding of the nature of the concerns raised and the purpose for raising them.

The Myth of Artificial Intelligence: Why Computers Can't Think the Way We Do by Erik J. Larson

AI winter, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Albert Einstein, Alignment Problem, AlphaGo, Amazon Mechanical Turk, artificial general intelligence, autonomous vehicles, Big Tech, Black Swan, Bletchley Park, Boeing 737 MAX, business intelligence, Charles Babbage, Claude Shannon: information theory, Computing Machinery and Intelligence, conceptual framework, correlation does not imply causation, data science, deep learning, DeepMind, driverless car, Elon Musk, Ernest Rutherford, Filter Bubble, Geoffrey Hinton, Georg Cantor, Higgs boson, hive mind, ImageNet competition, information retrieval, invention of the printing press, invention of the wheel, Isaac Newton, Jaron Lanier, Jeff Hawkins, John von Neumann, Kevin Kelly, Large Hadron Collider, Law of Accelerating Returns, Lewis Mumford, Loebner Prize, machine readable, machine translation, Nate Silver, natural language processing, Nick Bostrom, Norbert Wiener, PageRank, PalmPilot, paperclip maximiser, pattern recognition, Peter Thiel, public intellectual, Ray Kurzweil, retrograde motion, self-driving car, semantic web, Silicon Valley, social intelligence, speech recognition, statistical model, Stephen Hawking, superintelligent machines, tacit knowledge, technological singularity, TED Talk, The Coming Technological Singularity, the long tail, the scientific method, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, theory of mind, Turing machine, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!, Yochai Benkler

And h­ ere we should say it directly: all evidence suggests that ­human and machine intelligence are radically dif­fer­ent. The myth of AI insists that the differences are only temporary, and that more power­f ul systems ­w ill eventually erase them. Futurists like 2 T H E M Y T H O F A RT I F I CI A L I N T E L L I G E N CE Ray Kurzweil and phi­los­o­pher Nick Bostrom, prominent purveyors of the myth, talk not only as if human-­level AI ­were inevitable, but as if, soon ­a fter its arrival, superintelligent machines would leave us far ­behind. This book explains two impor­tant aspects of the AI myth, one scientific and one cultural. The scientific part of the myth assumes that we need only keep “chipping away” at the challenge of general intelligence by making pro­g ress on narrow feats of intelligence, like playing games or recognizing images.

…

Since the design of machines is one of ­these intellectual activities, an ultraintelligent machine could design even better machines; t­ here would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far b­ ehind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it ­under control.”1 34 T he S implified W orld Oxford phi­los­op­ her Nick Bostrom would return to Good’s theme de­cades ­later, with his 2014 best seller Superintelligence: Paths, Dangers, Strategies, making the same case that the achievement of AI would as a consequence usher in greater-­than-­human intelligence in an escalating pro­cess of self-­modification. In ominous language, Bostrom echoes Good’s futurism about the arrival of superintelligent machines: Before the prospect of an intelligence explosion, we ­humans are like small c­ hildren playing with a bomb.

…

Again, deep prob­lems with understanding language using computers have persisted. A s­ imple way to see the point h­ ere is to turn back to the Turing test, and reconsider it in light of the history of AI and the many, mostly fruitless attempts to solve the prob­lems it pre­ sents or even to make substantive pro­gress. Futurists like Nick Bostrom, as well as the larger scientific AI community, likely wish that the public would just forget about the test. It’s not—as is sometimes heard—­that the test is flawed or unhelpful. Quite simply, it’s too hard. TH E TU R I NG TEST Given a view from thirty thousand feet, it did seem that computers ­were getting smarter as AI progressed from its genesis in Turing’s early work and the kickoff conference at Dartmouth.

Warnings by Richard A. Clarke

"Hurricane Katrina" Superdome, active measures, Albert Einstein, algorithmic trading, anti-communist, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, Bear Stearns, behavioural economics, Bernie Madoff, Black Monday: stock market crash in 1987, carbon tax, cognitive bias, collateralized debt obligation, complexity theory, corporate governance, CRISPR, cuban missile crisis, data acquisition, deep learning, DeepMind, discovery of penicillin, double helix, Elon Musk, failed state, financial thriller, fixed income, Flash crash, forensic accounting, friendly AI, Hacker News, Intergovernmental Panel on Climate Change (IPCC), Internet of things, James Watt: steam engine, Jeff Bezos, John Maynard Keynes: Economic Possibilities for our Grandchildren, knowledge worker, Maui Hawaii, megacity, Mikhail Gorbachev, money market fund, mouse model, Nate Silver, new economy, Nicholas Carr, Nick Bostrom, nuclear winter, OpenAI, pattern recognition, personalized medicine, phenotype, Ponzi scheme, Ray Kurzweil, Recombinant DNA, Richard Feynman, Richard Feynman: Challenger O-ring, risk tolerance, Ronald Reagan, Sam Altman, Search for Extraterrestrial Intelligence, self-driving car, Silicon Valley, smart grid, statistical model, Stephen Hawking, Stuxnet, subprime mortgage crisis, tacit knowledge, technological singularity, The Future of Employment, the scientific method, The Signal and the Noise by Nate Silver, Tunguska event, uranium enrichment, Vernor Vinge, WarGames: Global Thermonuclear War, Watson beat the top human players on Jeopardy!, women in the workforce, Y2K

Tsachi Ein-Dor drew a similar conclusion, that people with higher anxiety levels tend to detect threats sooner and warn others. 15. In spite of the persistent view that science is the ultimate model of objectivity, scientists are, after all, human beings and still subject to what philosopher Nick Bostrom terms the selection effect. The scientists and their instruments reflect the fact that “all observations require the existence of an appropriately positioned observer.” See Nick Bostrom, Anthropic Bias: Observation Selection Effects in Science and Philosophy (New York: Routledge, 2002). 16. Charles S. Taber and Milton Lodge, “Motivated Skepticism in the Evaluation of Political Beliefs,” American Journal of Political Science 50, no. 3 (2006): 755–69, doi:10.1111/j.1540–5907.2006.00214. 17.

…

Arthur Samuel offered this definition of machine learning in 1959. 4. Superintelligence is a term made popular by philosopher Nick Bostrom and is often also called artificial general intelligence. 5. Referenced from Luke Muehlhauser’s fantastic work which was a great guide for the authors. 6. James Barrat, Our Final Invention: Artificial Intelligence and the End of the Human Era (New York: Thomas Dunne Books, 2013). Barrat’s book was an important source for the authors. 7. Nick Bostrom, “Ethical Issues in Advanced Artificial Intelligence,” http://www.nickbostrom.com/ethics/ai.html (accessed Nov. 9, 2016).

…

It will carve new paths of discovery in fields yet undiscovered, fueled by perpetual improvements to its own source code and the creation of new robotic tools. Artificial intelligence has the potential to be dramatically more powerful than any previous scientific advance. Superintelligence, according to Nick Bostrom at Oxford, is “not just another technology, another tool that will add incrementally to human capabilities.” It is, he says, “radically different,” and it “may be the last invention humans ever need to make.”7 From Yudkowsky’s whimsical graph above, we can get a hint of the predicted power of superintelligence (here called “recursively self-improved AI”).

Overcomplicated: Technology at the Limits of Comprehension by Samuel Arbesman

algorithmic trading, Anthropocene, Anton Chekhov, Apple II, Benoit Mandelbrot, Boeing 747, Chekhov's gun, citation needed, combinatorial explosion, Computing Machinery and Intelligence, Danny Hillis, data science, David Brooks, digital map, discovery of the americas, driverless car, en.wikipedia.org, Erik Brynjolfsson, Flash crash, friendly AI, game design, Google X / Alphabet X, Googley, Hans Moravec, HyperCard, Ian Bogost, Inbox Zero, Isaac Newton, iterative process, Kevin Kelly, machine translation, Machine translation of "The spirit is willing, but the flesh is weak." to Russian and back, mandelbrot fractal, Minecraft, Neal Stephenson, Netflix Prize, Nicholas Carr, Nick Bostrom, Parkinson's law, power law, Ray Kurzweil, recommendation engine, Richard Feynman, Richard Feynman: Challenger O-ring, Second Machine Age, self-driving car, SimCity, software studies, statistical model, Steve Jobs, Steve Wozniak, Steven Pinker, Stewart Brand, superintelligent machines, synthetic biology, systems thinking, the long tail, Therac-25, Tyler Cowen, Tyler Cowen: Great Stagnation, urban planning, Watson beat the top human players on Jeopardy!, Whole Earth Catalog, Y2K

Living with Complexity by Don Norman examines the origins of (and need for) complexity, particularly from the perspective of design. The Techno-Human Condition by Braden R. Allenby and Daniel Sarewitz is a discussion of how to grapple with coming technological change and is particularly intriguing when it discusses “wicked complexity.” Superintelligence by Nick Bostrom explores the many issues and implications related to the development of superintelligent machines. The Works, The Heights, and The Way to Go by Kate Ascher examine how cities, skyscrapers, and our transportation networks, respectively, actually work. Beautifully rendered and fascinating books.

…

the writer Quinn Norton has noted: Quinn Norton, “Everything is Broken,” The Message, May 20, 2014, https://medium.com/message/81e5f33a24e1. Langdon Winner notes in his book: Winner, Autonomous Technology, 290–91. computer scientist Danny Hillis argues: Danny Hillis, “The Age of Digital Entanglement,” Scientific American, September 2010, 93. Take the so-called Flash Crash: Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014), 17. It is still not entirely clear, however, what caused the Flash Crash. Understanding something in a “good enough” way: See also César Hidalgo, Why Information Grows: The Evolution of Order, from Atoms to Economies (New York: Basic Books, 2015).

…

Peter Norvig, Google’s director of research: Peter Norvig, “On Chomsky and the Two Cultures of Statistical Learning,” accessed April 30, 2015, http://norvig.com/chomsky.html. great, though apocryphal, story: There seem to be many versions of this apocryphal machine translation tale. What techniques are used by experts: Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014), 15. say, 99.9 percent of the time: I made these numbers up for effect, but if any linguist wants to chat, please reach out! “based on millions of specific features”: Alon Halevy et al., “The Unreasonable Effectiveness of Data,” IEEE Intelligent Systems 24, no. 2 (2009): 8–12.

The Alignment Problem: Machine Learning and Human Values by Brian Christian

Albert Einstein, algorithmic bias, Alignment Problem, AlphaGo, Amazon Mechanical Turk, artificial general intelligence, augmented reality, autonomous vehicles, backpropagation, butterfly effect, Cambridge Analytica, Cass Sunstein, Claude Shannon: information theory, computer vision, Computing Machinery and Intelligence, data science, deep learning, DeepMind, Donald Knuth, Douglas Hofstadter, effective altruism, Elaine Herzberg, Elon Musk, Frances Oldham Kelsey, game design, gamification, Geoffrey Hinton, Goodhart's law, Google Chrome, Google Glasses, Google X / Alphabet X, Gödel, Escher, Bach, Hans Moravec, hedonic treadmill, ImageNet competition, industrial robot, Internet Archive, John von Neumann, Joi Ito, Kenneth Arrow, language acquisition, longitudinal study, machine translation, mandatory minimum, mass incarceration, multi-armed bandit, natural language processing, Nick Bostrom, Norbert Wiener, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, OpenAI, Panopticon Jeremy Bentham, pattern recognition, Peter Singer: altruism, Peter Thiel, precautionary principle, premature optimization, RAND corporation, recommendation engine, Richard Feynman, Rodney Brooks, Saturday Night Live, selection bias, self-driving car, seminal paper, side project, Silicon Valley, Skinner box, sparse data, speech recognition, Stanislav Petrov, statistical model, Steve Jobs, strong AI, the map is not the territory, theory of mind, Tim Cook: Apple, W. E. B. Du Bois, Wayback Machine, zero-sum game

See also the contemporaneous and closely related “expert iteration” (“ExIt”) algorithm in Anthony, Tian, and Barber, “Thinking Fast and Slow with Deep Learning and Tree Search.” 91. Shead, “DeepMind’s Human-Bashing AlphaGo AI Is Now Even Stronger.” 92. Aurelius, The Emperor Marcus Aurelius. 93. Andy Fitch, “Letter from Utopia: Talking to Nick Bostrom,” BLARB (blog), November 24, 2017, https://blog.lareviewofbooks.org/interviews/letter-utopia-talking-nick-bostrom/. 94. Blaise Agüera y Arcas, “The Better Angels of our Nature” (lecture), February 16, 2017, VOR: Superintelligence, Mexico City. 95. Yudkowsky, “Coherent Extrapolated Volition.” See also Tarleton, “Coherent Extrapolated Volition.” 96.

…

We want to get from point A to point B as quickly as possible, though not by going over the speed limit—or, rather, not by going too far over the speed limit, unless for some reason we have to—and by staying centered in our lane, unless there’s a cyclist or a stopped car—and not passing cars on the right, unless it’s safer to do so than not to do so, and so on. It’s hard to try to formalize all of this into some kind of objective function that we then tell the system to optimize. Better, in cases like this, to use what the Future of Humanity Institute’s Nick Bostrom calls “indirect normativity”33—a way to get the system aligned with our desires without articulating them down to the last minutia. In this case, what we want is to say something like “Watch how I drive. Do it like this.” This was, as it turns out, one of the very first ideas in self-driving cars—and, to this day, still one of the best.

…

—MARCUS AURELIUS92 For a growing number of philosophers and computer scientists concerned with the longer-term future, the prospect of flexibly intelligent and flexibly capable systems, into which we must impart extremely complex behaviors and values, raises not only technical problems but something much deeper. There are two primary challenges here. The first is that the things we want are very difficult to simply state outright—even in words, let alone in a more numerical form. As the Future of Humanity Institute’s Nick Bostrom notes, “It seems completely impossible to write down a list of everything we care about.”93 In this case, we have already seen how learning by imitation can succeed in domains where it is effectively impossible to explicitly impart every rule and consideration and degree of emphasis for what makes someone an expert driver or an expert Go player.

Robot Rules: Regulating Artificial Intelligence by Jacob Turner

"World Economic Forum" Davos, Ada Lovelace, Affordable Care Act / Obamacare, AI winter, algorithmic bias, algorithmic trading, AlphaGo, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, autonomous vehicles, backpropagation, Basel III, bitcoin, Black Monday: stock market crash in 1987, blockchain, brain emulation, Brexit referendum, Cambridge Analytica, Charles Babbage, Clapham omnibus, cognitive dissonance, Computing Machinery and Intelligence, corporate governance, corporate social responsibility, correlation does not imply causation, crowdsourcing, data science, deep learning, DeepMind, Demis Hassabis, distributed ledger, don't be evil, Donald Trump, driverless car, easy for humans, difficult for computers, effective altruism, Elon Musk, financial exclusion, financial innovation, friendly fire, future of work, hallucination problem, hive mind, Internet of things, iterative process, job automation, John Markoff, John von Neumann, Loebner Prize, machine readable, machine translation, medical malpractice, Nate Silver, natural language processing, Nick Bostrom, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, nudge unit, obamacare, off grid, OpenAI, paperclip maximiser, pattern recognition, Peace of Westphalia, Philippa Foot, race to the bottom, Ray Kurzweil, Recombinant DNA, Rodney Brooks, self-driving car, Silicon Valley, Stanislav Petrov, Stephen Hawking, Steve Wozniak, strong AI, technological singularity, Tesla Model S, The Coming Technological Singularity, The Future of Employment, The Signal and the Noise by Nate Silver, trolley problem, Turing test, Vernor Vinge

Good predicted that “…an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind”.105 This remains the operating assumption of some AI experts today. In his influential book, Superintelligence Nick Bostrom describes the consequences of the AI explosion in dramatic terms, explaining that in some models it could be a matter of days between the development of the initial “seed” superintelligence and its spawn becoming so powerful that no human-controlled force is able to reassert control: “Once artificial intelligence reaches human level, there will be a positive feedback loop that will give the development a further boost.

…

Technology has always been a double edged sword, since fire kept us warm but also burned down our villages”.119 Similarly, engineer and roboethicist Alan Winfield said in a 2014 article: “If we succeed in building human equivalent AI and if that AI acquires a full understanding of how it works, and if it then succeeds in improving itself to produce super-intelligent AI, and if that super-AI, accidentally or maliciously, starts to consume resources, and if we fail to pull the plug, then, yes, we may well have a problem. The risk, while not impossible, is improbable”.120 Fundamentally, optimists think humanity can and will overcome any challenges AI poses. The pessimists include Nick Bostrom, whose “paperclip machine” thought experiment imagines an AI system asked to make paperclips which decides to seize and consume all resources in existence, in its blind aderence to that goal.121 Bostrom contemplates a form of superintelligence which is so powerful that humanity has no chance of stopping it from destroying the entire universe.

…

, Oxford Martin Programme on the Impacts of Future Technology Working Paper, September 2013, http://​www.​oxfordmartin.​ox.​ac.​uk/​downloads/​academic/​future-of-employment.​pdf, accessed 1 June 2018. See also Daniel Susskind and Richard Susskind, The Future of the Professions : How Technology Will Transform the Work of Human Experts (Oxford: Oxford University Press, 2015). 5See Nick Bostrom, Superintelligence (Oxford: Oxford University Press, 2014). 6See Ray Kurzweil, The Singularity Is Near: When Humans Transcend Biology (New York: Viking Press, 2005). 7Several nineteenth-century thinkers including Charles Babbage and Ada Lovelace arguably predicted the advent of AI and even prepared designs for machines capable of carrying out intelligent tasks.

Beyond: Our Future in Space by Chris Impey

3D printing, Admiral Zheng, Albert Einstein, Alfred Russel Wallace, AltaVista, Apollo 11, Apollo 13, Berlin Wall, Biosphere 2, Buckminster Fuller, built by the lowest bidder, butterfly effect, California gold rush, carbon-based life, Charles Lindbergh, Colonization of Mars, cosmic abundance, crowdsourcing, cuban missile crisis, dark matter, Dennis Tito, discovery of DNA, Doomsday Clock, Edward Snowden, Elon Musk, Eratosthenes, Great Leap Forward, Haight Ashbury, Hans Moravec, Hyperloop, I think there is a world market for maybe five computers, Isaac Newton, Jeff Bezos, Johannes Kepler, John von Neumann, Kickstarter, Kim Stanley Robinson, Late Heavy Bombardment, life extension, low earth orbit, Mahatma Gandhi, Marc Andreessen, Mars Rover, Mars Society, military-industrial complex, mutually assured destruction, Neal Stephenson, Neil Armstrong, Nick Bostrom, ocean acidification, Oculus Rift, operation paperclip, out of africa, Peter H. Diamandis: Planetary Resources, phenotype, private spaceflight, purchasing power parity, quantum entanglement, radical life extension, RAND corporation, Ray Kurzweil, RFID, Richard Feynman, Richard Feynman: Challenger O-ring, risk tolerance, Rubik’s Cube, Scaled Composites, Search for Extraterrestrial Intelligence, Searching for Interstellar Communications, seminal paper, Silicon Valley, skunkworks, Skype, Snow Crash, space junk, SpaceShipOne, Stephen Hawking, Steven Pinker, supervolcano, technological singularity, telepresence, telerobotics, the medium is the message, the scientific method, theory of mind, There's no reason for any individual to have a computer in his home - Ken Olsen, Virgin Galactic, VTOL, wikimedia commons, world market for maybe five computers, X Prize, Yogi Berra

The philosophical movement that forms an umbrella for cybernetics and cyborgs is called transhumanism. Transhumanism is a worldwide cultural and intellectual movement that seeks to use technology to improve the human condition. Radical life extension is one aspect, as is the enhancement of physical and mental capabilities. Two prominent transhumanists are Nick Bostrom, a University of Oxford philosopher who has assessed various risks to the long-term survival of humanity, and Ray Kurzweil, the engineer and inventor who popularized the idea of the singularity, a time in the not-too-distant future when technology will enable us to transcend our physical limitations.

…

Technological civilizations are intrinsically rare, so searches for them will fail. On the other hand, if the Great Filter is in our future, it’s very unlikely that a civilization at our stage of development progresses to the large-scale colonization of space. One plausible scenario is that technology is the culprit because it includes the capability for self-destruction. Nick Bostrom, director of the Future of Humanity Institute at the University of Oxford, has done scholarly work on catastrophes. His partial list of existential threats faced by humanity includes nuclear holocaust, genetically engineered superbugs, environmental disasters, asteroid impacts, terrorism, advanced and destructive artificial intelligence, uncontrollable nanotechnology, catastrophic high-energy physics experiments, and a totalitarian regime with advanced surveillance and mind-control technologies.

…

There may be more than one Great Filter. We might have cleared one, only to be faced in the future with another. Also, we should be wary of positing that life elsewhere has to follow the path traveled by life on Earth or that other civilizations progress with the single-minded purposefulness of humans. Let’s allow Nick Bostrom to have the last word. For someone who dwells on apocalypse and hopes that the search for life in the universe fails, he’s strangely optimistic: If the Great Filter is in our past . . . we may have a significant chance . . . of one day growing into something almost unimaginably greater than we are today.

How to Create a Mind: The Secret of Human Thought Revealed by Ray Kurzweil

Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Albert Einstein, Albert Michelson, anesthesia awareness, anthropic principle, brain emulation, cellular automata, Charles Babbage, Claude Shannon: information theory, cloud computing, computer age, Computing Machinery and Intelligence, Dean Kamen, discovery of DNA, double helix, driverless car, en.wikipedia.org, epigenetics, George Gilder, Google Earth, Hans Moravec, Isaac Newton, iterative process, Jacquard loom, Jeff Hawkins, John von Neumann, Law of Accelerating Returns, linear programming, Loebner Prize, mandelbrot fractal, Nick Bostrom, Norbert Wiener, optical character recognition, PalmPilot, pattern recognition, Peter Thiel, Ralph Waldo Emerson, random walk, Ray Kurzweil, reversible computing, selective serotonin reuptake inhibitor (SSRI), self-driving car, speech recognition, Steven Pinker, strong AI, the scientific method, theory of mind, Turing complete, Turing machine, Turing test, Wall-E, Watson beat the top human players on Jeopardy!, X Prize

The report projects ongoing exponential gains in all of these areas of capability and argues that the requirements to simulate the human brain at a high level of detail are coming into place. An outline of the technological capabilities needed for whole brain emulation, in Whole Brain Emulation: A Roadmap by Anders Sandberg and Nick Bostrom. An outline of Whole Brain Emulation: A Roadmap by Anders Sandberg and Nick Bostrom. Neural Nets In 1964, at the age of sixteen, I wrote to Frank Rosenblatt (1928–1971), a professor at Cornell University, inquiring about a machine called the Mark 1 Perceptron. He had created it four years earlier, and it was described as having brainlike properties.

…

Wedeen and his colleagues at Massachusetts General Hospital showing a highly regular gridlike structure of the wiring of the neocortex that I described in chapter 4 is one early result from this project. Oxford University computational neuroscientist Anders Sandberg (born in 1972) and Swedish philosopher Nick Bostrom (born in 1973) have written the comprehensive Whole Brain Emulation: A Roadmap, which details the requirements for simulating the human brain (and other types of brains) at different levels of specificity from high-level functional models to simulating molecules.8 The report does not provide a timeline, but it does describe the requirements to simulate different types of brains at varying levels of precision in terms of brain scanning, modeling, storage, and computation.

…

The truth can be discovered only by finding an explanation that overrides—transcends—seeming differences, especially for fundamental questions of meaning and purpose. This is how I resolve the Western-Eastern divide on consciousness and the physical world. In my view, both perspectives have to be true. On the one hand, it is foolish to deny the physical world. Even if we do live in a simulation, as speculated by Swedish philosopher Nick Bostrom, reality is nonetheless a conceptual level that is real for us. If we accept the existence of the physical world and the evolution that has taken place in it, then we can see that conscious entities have evolved from it. On the other hand, the Eastern perspective—that consciousness is fundamental and represents the only reality that is truly important—is also difficult to deny.

To Be a Machine: Adventures Among Cyborgs, Utopians, Hackers, and the Futurists Solving the Modest Problem of Death by Mark O'Connell

"World Economic Forum" Davos, 3D printing, Ada Lovelace, AI winter, Airbnb, Albert Einstein, AlphaGo, Amazon Picking Challenge, artificial general intelligence, Bletchley Park, Boston Dynamics, brain emulation, Charles Babbage, clean water, cognitive dissonance, computer age, cosmological principle, dark matter, DeepMind, disruptive innovation, double helix, Edward Snowden, effective altruism, Elon Musk, Extropian, friendly AI, global pandemic, Great Leap Forward, Hans Moravec, impulse control, income inequality, invention of the wheel, Jacques de Vaucanson, John von Neumann, knowledge economy, Law of Accelerating Returns, Lewis Mumford, life extension, lifelogging, Lyft, Mars Rover, means of production, military-industrial complex, Nick Bostrom, Norbert Wiener, paperclip maximiser, Peter Thiel, profit motive, radical life extension, Ray Kurzweil, RFID, San Francisco homelessness, self-driving car, sharing economy, Silicon Valley, Silicon Valley billionaire, Silicon Valley ideology, Singularitarianism, Skype, SoftBank, Stephen Hawking, Steve Wozniak, superintelligent machines, tech billionaire, technological singularity, technoutopianism, TED Talk, The Coming Technological Singularity, Travis Kalanick, trickle-down economics, Turing machine, uber lyft, Vernor Vinge

He was dreaming, I came to feel, of a Singularity. He was singing of what was past, and passing, and to come. In May 2007, Randal was one of thirteen participants at a workshop on mind uploading held at the Future of Humanity Institute. The event resulted in the publication of a technical report, coauthored by Anders Sandberg and Nick Bostrom, entitled “Whole Brain Emulation: A Roadmap.” The report began with the statement that mind uploading, though still a remote prospect, was nonetheless theoretically achievable through the development of technologies already in existence. A criticism commonly raised against the idea of simulating minds in software is that we don’t understand nearly enough about how consciousness works to even know where to start reproducing it.

…

And if these people were worried—along with Hawkeye from M*A*S*H and the guy who had played, among other personifications of serene wisdom, a scientist trying to prevent the Singularity in the 2014 film Transcendence—shouldn’t we all be worrying with them? One figure who loomed especially large over this whole area, its eschatologist-in-chief, was Nick Bostrom, the Swedish philosopher who, before he became known as the world’s foremost prophet of technological disaster, had been one of the most prominent figures in the transhumanist movement, a cofounder of the World Transhumanist Association. In late 2014, Bostrom, director of the Future of Humanity Institute, had published a book called Superintelligence: Paths, Dangers, Strategies, in which he outlined the nature of the AI peril.

…

(An effectively altruistic act, as opposed to an emotionally altruistic one, might involve a college student deciding that, rather than becoming a doctor and spending her career curing blindness in the developing world, her time would be better spent becoming a Wall Street hedge fund manager and donating enough of her income to charity to pay for several doctors to cure a great many more people of blindness.) The conference had substantially focused on questions of AI and existential risk. Thiel and Musk, who’d spoken on a panel at the conference along with Nick Bostrom, had been influenced by the moral metrics of Effective Altruism to donate large amounts of money to organizations focused on AI safety. Effective Altruism had significant crossover, in terms of constituency, with the AI existential risk movement. (In fact, the Centre for Effective Altruism, the main international promoter of the movement, happened to occupy an office in Oxford just down the hall from the Future of Humanity Institute.)

12 Bytes: How We Got Here. Where We Might Go Next by Jeanette Winterson

"Margaret Hamilton" Apollo, "World Economic Forum" Davos, 3D printing, Ada Lovelace, Airbnb, Albert Einstein, Alignment Problem, Amazon Mechanical Turk, Anthropocene, Apollo 11, Apple's 1984 Super Bowl advert, artificial general intelligence, Asilomar, augmented reality, autonomous vehicles, basic income, Big Tech, bitcoin, Bletchley Park, blockchain, Boston Dynamics, call centre, Cambridge Analytica, Capital in the Twenty-First Century by Thomas Piketty, cashless society, Charles Babbage, computer age, Computing Machinery and Intelligence, coronavirus, COVID-19, CRISPR, cryptocurrency, dark matter, Dava Sobel, David Graeber, deep learning, deskilling, digital rights, discovery of DNA, Dominic Cummings, Donald Trump, double helix, driverless car, Elon Musk, fake news, flying shuttle, friendly AI, gender pay gap, global village, Grace Hopper, Gregor Mendel, hive mind, housing crisis, Internet of things, Isaac Newton, Jacquard loom, James Hargreaves, Jeff Bezos, Johannes Kepler, John von Neumann, Joseph-Marie Jacquard, Kickstarter, Large Hadron Collider, life extension, lockdown, lone genius, Mark Zuckerberg, means of production, microdosing, more computing power than Apollo, move fast and break things, natural language processing, Nick Bostrom, Norbert Wiener, off grid, OpenAI, operation paperclip, packet switching, Peter Thiel, pink-collar, Plato's cave, public intellectual, QAnon, QWERTY keyboard, Ray Kurzweil, rewilding, ride hailing / ride sharing, Rutger Bregman, Sam Altman, self-driving car, sharing economy, Sheryl Sandberg, Shoshana Zuboff, Silicon Valley, Skype, Snapchat, SoftBank, SpaceX Starlink, speech recognition, spinning jenny, stem cell, Stephen Hawking, Steve Bannon, Steve Jobs, Steven Levy, Steven Pinker, superintelligent machines, surveillance capitalism, synthetic biology, systems thinking, tech billionaire, tech worker, TED Talk, telepresence, telepresence robot, TikTok, trade route, Turing test, universal basic income, Virgin Galactic, Watson beat the top human players on Jeopardy!, women in the workforce, Y Combinator

That’s far ahead, when we are no longer augmented humans, but intelligences running on a number of alternative substrates, of which a biological-based body might be one – but probably won’t be, except perhaps as a retro-themed holiday (‘Let’s visit how they used to live in the past: get drunk, throw up, pass out etc’). * * * For Nick Bostrom, Director of the Future of Humanity Institute at the University of Oxford, transhumanism today is a loosely connected interdisciplinary approach for developing, and evaluating, technologies that will benefit us as individuals, and as a society. Bostrom is keen to involve governmental influence and legislation, rather than leave the future to the marketplace.

…

Forbes Magazine has been ranking the worth of fictional characters since 2002. The wealthiest is Twilight vampire Carlisle Cullen, whose very long-term investments, plus the magic of compound interest, have netted him a fortune of $34 billion. It is safe to say that vampires can fund themselves. For the rest of us? * * * Nick Bostrom suggests that people would go back to school in their 50s, or start a new career in their 70s. His view is that 80-year-olds with the same physique as 40-year-olds would not burden the healthcare system and would be amazingly productive because of their experience and knowledge-base. He believes that longer life expectancy would give us an increased responsibility for the future – because we’ll be there to see it.

…

If we start to merge with the AI opportunities we are creating, by which I mean, if we become part of the toolkit – not just operators on the outside, but intrinsic to the technology on the inside – no ‘us’ and ‘it’ – then … Then the enemy won’t be on the outside. And we will recognise our responsibility to ourselves, and to AI, and later to AGI. * * * Nick Bostrom, philosopher and AI expert, author of Super-intelligence: Paths, Dangers, Strategies (2014), and Director of the Future of Humanity Institute at the University of Oxford, is an enthusiast for transhumanism. In his view, we must merge with AI. We must better ourselves because we can. Humans should be a match for what we are creating

The Future of the Professions: How Technology Will Transform the Work of Human Experts by Richard Susskind, Daniel Susskind

23andMe, 3D printing, Abraham Maslow, additive manufacturing, AI winter, Albert Einstein, Amazon Mechanical Turk, Amazon Robotics, Amazon Web Services, Andrew Keen, Atul Gawande, Automated Insights, autonomous vehicles, Big bang: deregulation of the City of London, big data - Walmart - Pop Tarts, Bill Joy: nanobots, Blue Ocean Strategy, business process, business process outsourcing, Cass Sunstein, Checklist Manifesto, Clapham omnibus, Clayton Christensen, clean water, cloud computing, commoditize, computer age, Computer Numeric Control, computer vision, Computing Machinery and Intelligence, conceptual framework, corporate governance, creative destruction, crowdsourcing, Daniel Kahneman / Amos Tversky, data science, death of newspapers, disintermediation, Douglas Hofstadter, driverless car, en.wikipedia.org, Erik Brynjolfsson, Evgeny Morozov, Filter Bubble, full employment, future of work, Garrett Hardin, Google Glasses, Google X / Alphabet X, Hacker Ethic, industrial robot, informal economy, information retrieval, interchangeable parts, Internet of things, Isaac Newton, James Hargreaves, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, Joseph Schumpeter, Khan Academy, knowledge economy, Large Hadron Collider, lifelogging, lump of labour, machine translation, Marshall McLuhan, Metcalfe’s law, Narrative Science, natural language processing, Network effects, Nick Bostrom, optical character recognition, Paul Samuelson, personalized medicine, planned obsolescence, pre–internet, Ray Kurzweil, Richard Feynman, Second Machine Age, self-driving car, semantic web, Shoshana Zuboff, Skype, social web, speech recognition, spinning jenny, strong AI, supply-chain management, Susan Wojcicki, tacit knowledge, TED Talk, telepresence, The Future of Employment, the market place, The Wealth of Nations by Adam Smith, The Wisdom of Crowds, Tragedy of the Commons, transaction costs, Turing test, Two Sigma, warehouse robotics, Watson beat the top human players on Jeopardy!, WikiLeaks, world market for maybe five computers, Yochai Benkler, young professional

Into the 2020s and beyond, we predict the emergence and adoption of a second wave of AI systems in the professions. 1 We first introduced this concept in the mid-1990s, in Richard Susskind, The Future of Law (1996). 2 Walter Ong, Orality and Literacy (1982). 3 We can anticipate a fifth stage, but it is beyond the scope of this book—this is the likely stage in the development of humanity when human beings become digitally enhanced, when machines and human beings become entwined and even indistinguishable. Some of these themes are touched on in Nick Bostrom, Superintelligence (2014). 4 Ong, Orality and Literacy, Marshall McLuhan, Understanding Media (2002), Luciano Floridi, The Philosophy of Information (2013), James Gleick, The Information (2011), and the works to which they refer. 5 Ong, Orality and Literacy, 31. 6 Gleick, The Information, and his discussion of Plato, who spoke of the ‘forgetfulness’ of those who do ‘not practice their memory’ (p. 30).

…

Joel Garreau, Radical Evolution (2005), and ‘Coming To an Office Near You’, Economist, 18 Jan. 2014. 28 See e.g. the work of Singularity University at <http://singularityu.org> (accessed 23 March 2015). 29 For a clear introduction to the cloud and cloud computing, and a clear indication of its mounting signifiance, see Kuan Hon and Christopher Millard, ‘Cloud Technologies and Services’, in Cloud Computing Law, ed. Christopher Millard (2013). 30 John Kelly and Steve Hamm, Smart Machines (2013). 31 Nick Bostrom, Superintelligence (2014). 32 See e.g. Tim Bradshaw, ‘Scientists and Investors Warn on AI’, Financial Times, 12 Jan. 2015. 33 Kelly and Hamm, Smart Machines, and Kieron O’Hara et al., Web Science: Understanding the Emergence of Macro-Level Features of the World Wide Web (2013). 34 Soshana Zuboff, In the Age of the Smart Machine (1988), 9. 35 Zuboff, In the Age of the Smart Machine, 10. 36 Some writers treat ‘machine learning’ as yet another synonym.

…

v=FU-tuY0Z7nQ> (accessed 24 March 2015). 54 Frank Levy and Richard Murnane, The New Division of Labour (2004), 1–2, for the upheaval, and 20–30 for the truck-driver discussion. 55 Alison Sander and Meldon Wolfgang, ‘The Rise of Robotics’, 27 Aug. 2014, at <https://www.bcgperspectives.com/content/articles/business_unit_strategy_innovation_rise_of_robotics> (accessed 23 March 2015). 56 See <http://cyberlaw.stanford.edu/wiki/index.php/Automated_Driving:_Legislative_and_Regulatory_Action> (accessed 27 March 2015). 57 Guy Ryder, ‘Labor in the Age of Robots’, Project Syndicate, 22 Jan. 2015 <http://www.project-syndicate.org/> (accessed 23 March 2015). 58 Sam Frizell, ‘Meet the Robots Shipping Your Amazon Orders’, Time, 1 Dec. 2014, <http://time.com/3605924/amazon-robots/> (accessed 23 March 2015). 59 See <http://allaboutroboticsurgery.com/zeusrobot.html> (accessed 23 March 2015). 60 David Rose, Enchanted Objects (2014), 23–4; original emphasis. 61 See Julian Savulescu and Nick Bostrom, Human Enhancement (2011), on the technologies and the ethical questions they raise. 62 Malcolm Peltu and Yorick Wilks, ‘Close Engagements with Artificial Companions: Key Social, Psychological, Ethical and Design Issues’, OII/e-Horizons Discussion Paper, No. 14 (2008). Also see Timothy Bickmore, ‘Relational Agents in Health Applications: Leveraging Affective Computing to Promote Health and Wellness’, in Oxford Handbook of Affective Computing, ed.

The Internet Is Not What You Think It Is: A History, a Philosophy, a Warning by Justin E. H. Smith

3D printing, Ada Lovelace, Adrian Hon, agricultural Revolution, algorithmic management, artificial general intelligence, Big Tech, Charles Babbage, clean water, coronavirus, COVID-19, cryptocurrency, dark matter, disinformation, Donald Trump, drone strike, Elon Musk, game design, gamification, global pandemic, GPT-3, Internet of things, Isaac Newton, Jacquard loom, Jacques de Vaucanson, Jaron Lanier, jimmy wales, Joseph-Marie Jacquard, Kuiper Belt, Mark Zuckerberg, Marshall McLuhan, meme stock, new economy, Nick Bostrom, Norbert Wiener, packet switching, passive income, Potemkin village, printed gun, QAnon, Ray Kurzweil, Republic of Letters, Silicon Valley, Skype, strong AI, technological determinism, theory of mind, TikTok, Tragedy of the Commons, trolley problem, Turing machine, Turing test, you are the product

Ann M. Blair, Too Much to Know: Managing Scholarly Information Before the Information Age, New Haven, CT: Yale University Press, 2010. Ned Block, “Troubles with Functionalism,” Minnesota Studies in the Philosophy of Science 9 (1978): 261–325. Nick Bostrom, “Are You Living in a Computer Simulation?” Philosophical Quarterly 53, 211 (2003): 243–255. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies, New York: Oxford University Press, 2014. Horst Bredekamp, “Leibniz’ Gewebe: Strumpband, Falte, Leinwand,” in R. Hoppe-Sailer, C. Volkenandt, and G. Winter (eds.), Logik der Bilder. Präsenz—Repräsentation—Erkenntnis, Bonn: Reimer, 2005, 233–238.

…

How and why many people became more comfortable assimilating the mind to a machine of the mind’s own design than they are assimilating the mind to the countless other products of nature alongside which it co-inhabits the earth, is a complex question that we will not answer here, other than to say that the reasons have much more to do with cultural history and fashion than they do with rigorous philosophical arguments or scientific literacy. To provide just one illustration of the fashion in question, some philosophers maintain that the world as we know it is most likely a simulation comparable to a video game. According to one variant of what Nick Bostrom calls the “simulation hypothesis,” “a superintelligence could create virtual worlds that appear to its inhabitants much the same as our world appears to us. It might create vast numbers of such worlds, running the same simulation many times or with small variation. The inhabitants would not necessarily be able to tell whether their world is simulated or not; but if they are intelligent enough they could consider the possibility and assign it some probability.”2 This hypothesis has been extremely popular among the representatives of whatever simulation of an intelligentsia the tech industry has produced.

The Seventh Sense: Power, Fortune, and Survival in the Age of Networks by Joshua Cooper Ramo

air gap, Airbnb, Alan Greenspan, Albert Einstein, algorithmic trading, barriers to entry, Berlin Wall, bitcoin, Bletchley Park, British Empire, cloud computing, Computing Machinery and Intelligence, crowdsourcing, Danny Hillis, data science, deep learning, defense in depth, Deng Xiaoping, drone strike, Edward Snowden, Fairchild Semiconductor, Fall of the Berlin Wall, financial engineering, Firefox, Google Chrome, growth hacking, Herman Kahn, income inequality, information security, Isaac Newton, Jeff Bezos, job automation, Joi Ito, Laura Poitras, machine translation, market bubble, Menlo Park, Metcalfe’s law, Mitch Kapor, Morris worm, natural language processing, Neal Stephenson, Network effects, Nick Bostrom, Norbert Wiener, Oculus Rift, off-the-grid, packet switching, paperclip maximiser, Paul Graham, power law, price stability, quantitative easing, RAND corporation, reality distortion field, Recombinant DNA, recommendation engine, Republic of Letters, Richard Feynman, road to serfdom, Robert Metcalfe, Sand Hill Road, secular stagnation, self-driving car, Silicon Valley, Skype, Snapchat, Snow Crash, social web, sovereign wealth fund, Steve Jobs, Steve Wozniak, Stewart Brand, Stuxnet, superintelligent machines, systems thinking, technological singularity, The Coming Technological Singularity, The Wealth of Nations by Adam Smith, too big to fail, Vernor Vinge, zero day

In the spring of 1993: See Vision-21: Interdisciplinary Science and Engineering in the Era of Cyberspace, proceedings of a symposium cosponsored by the NASA Lewis Research Center and the Ohio Aerospace Institute, Westlake, Ohio, March 30–31, 1993 (Hampton, VA: National Aeronautics and Space Administration Scientific and Technical Information Program), iii. “Within thirty years”: Ibid., 12. Imagine a superintelligent machine: Nick Bostrom, “Ethical Issues in Advanced Artificial Intelligence,” in Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in AI, vol. 2, ed. Iva Smit et al. (Windsor, ON: International Institute for Advanced Studies in Systems Research and Cybernetics, 2003), 12–17, and Nick Bostrom, “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents,” Minds and Machines, 22, no. 2 (2012): 71–85. It runs like this: Seth Grimes, “Language Use, Customer Personality, and the Customer Journey” (interview with Scott Nowson, global innovation lead, Xerox), Breakthrough Analysis (blog), October 8, 2015.

…

It was easy enough for Vinge to see how this would end. It wouldn’t be with the sort of intended polite, lapdog domesticity of artificial intelligence that we might hope for but with a rottweiler of a device, alive to the meaty smell of power, violence, and greed. This puzzle has interested the Oxford philosopher Nick Bostrom, who has described the following thought experiment: Imagine a superintelligent machine programmed to do whatever is needed to make paper clips as fast as possible, a machine that is connected to every resource that task might demand. Go figure it out! might be all its human instructors tell it.

…

(Windsor, ON: International Institute for Advanced Studies in Systems Research and Cybernetics, 2003), 12–17, and Nick Bostrom, “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents,” Minds and Machines, 22, no. 2 (2012): 71–85. It runs like this: Seth Grimes, “Language Use, Customer Personality, and the Customer Journey” (interview with Scott Nowson, global innovation lead, Xerox), Breakthrough Analysis (blog), October 8, 2015. Let me out!: See, for instance, Stuart Armstrong, Anders Sandberg, and Nick Bostrom, “Thinking Inside the Box: Controlling and Using an Oracle AI,” Minds and Machines 22, no. 4 (November 2012): 299–324. “People proposing”: Michael Vassar, “Re: AI Boxing (dogs and helicopters),” SL4.org, August 2, 2005, at http://sl4.org/archive/0508/11817.html. So we need to ask: Kaj Sotala and Roman V. Yampolskiy, “Responses to Catastrophic AGI Risk: A Survey,” Physica Scripta 90 (2015).

The World Without Us by Alan Weisman

British Empire, carbon-based life, company town, conceptual framework, coronavirus, invention of radio, Nick Bostrom, nuclear winter, optical character recognition, out of africa, Ray Kurzweil, rewilding, the High Line, trade route, uranium enrichment, William Langewiesche

He chuckles mirthlessly. “People trying might kill themselves in the process. There are a lot of easier things to do with a lot less effort.” HAVING YET TO perfect contraception, thus far we have little to fear from misanthropic plots to sterilize the entire human race. From time to time, Nick Bostrom, who directs Oxford’s Future of Humanity Institute, computes the odds (increasing, he believes) that human existence is at risk of ending. He is particularly intrigued by the potential of nanotechnology going awry, accidentally or deliberately, or superintelligence running amuck. In either case, however, he notes that the skills needed to create atom-sized medical machines that would patrol our bloodstreams, zapping disease until they suddenly turned on us, or self-replicating robots that end up crowding or outsmarting us off the planet, are “at least decades away.”

…

Each of these men, philosophers taking ethical measure of an age in which machines think faster than humans but regularly prove at least as flawed, repeatedly smack into a phenomenon that never troubled their intellectual predecessors: although humans have obviously survived every pox and meteor that nature has tossed at us until now, technology is something we toss back at our own peril. “On the bright side, it hasn’t killed us yet, either,” says Nick Bostrom, who, when not refining doomsday data, researches how to extend the human life span. “But if we did go extinct, I think it would more likely be through new technologies than environmental destruction.” To the rest of the planet, it would make little difference, because if either actually took us out, many other species undoubtedly would go with us.

…

Via the self-accruing wizardry of computers, an abundance of silicon, and vast opportunities afforded by modular memory and mechanical appendages, human extinction would become merely a jettisoning of the limited and not very durable vessels that our technological minds have finally outgrown. Prominent in the transhumanist (sometimes called posthuman) movement are Oxford philosopher Nick Bostrom; heralded inventor Ray Kurzweil, originator of optical character recognition, flat-bed scanners, and print-to-speech reading machines for the blind; and Trinity College bioethicist James Hughes, author of Citizen Cyborg: Why Democratic Societies Must Respond to the Redesigned Human of the Future.

Genius Makers: The Mavericks Who Brought A. I. To Google, Facebook, and the World by Cade Metz

AI winter, air gap, Airbnb, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, AlphaGo, Amazon Robotics, artificial general intelligence, Asilomar, autonomous vehicles, backpropagation, Big Tech, British Empire, Cambridge Analytica, carbon-based life, cloud computing, company town, computer age, computer vision, deep learning, deepfake, DeepMind, Demis Hassabis, digital map, Donald Trump, driverless car, drone strike, Elon Musk, fake news, Fellow of the Royal Society, Frank Gehry, game design, Geoffrey Hinton, Google Earth, Google X / Alphabet X, Googley, Internet Archive, Isaac Newton, Jeff Hawkins, Jeffrey Epstein, job automation, John Markoff, life extension, machine translation, Mark Zuckerberg, means of production, Menlo Park, move 37, move fast and break things, Mustafa Suleyman, new economy, Nick Bostrom, nuclear winter, OpenAI, PageRank, PalmPilot, pattern recognition, Paul Graham, paypal mafia, Peter Thiel, profit motive, Richard Feynman, ride hailing / ride sharing, Ronald Reagan, Rubik’s Cube, Sam Altman, Sand Hill Road, self-driving car, side project, Silicon Valley, Silicon Valley billionaire, Silicon Valley startup, Skype, speech recognition, statistical model, stem cell, Stephen Hawking, Steve Ballmer, Steven Levy, Steven Pinker, tech worker, telemarketer, The Future of Employment, Turing test, warehouse automation, warehouse robotics, Y Combinator

“There have been movies about this,” he said. On Twitter, he called artificial intelligence “potentially more dangerous than nukes.” The same tweet urged his followers to read Superintelligence: Paths, Dangers, Strategies, a recently published tome from an Oxford University philosopher named Nick Bostrom. Like Shane Legg, the founder of DeepMind, Bostrom believed that superintelligence could secure the future of humanity—or destroy it. “This is quite possibly the most important and most daunting challenge humanity has ever faced,” he wrote. “And—whether we succeed or fail—it is probably the last challenge we will ever face.”

…

To help make his case, he also invited Yann LeCun, Mike Schroepfer, and Rob Fergus, the NYU professor who worked alongside LeCun at the new Facebook lab. The Facebookers spent the meal trying to explain that Musk’s views on AI had been warped by a few misguided voices that were very much in the minority. The philosophical musings of Nick Bostrom, Zuckerberg and his fellow Facebookers said, were in no way related to what Musk had seen at DeepMind or inside any other AI lab. A neural network was still a long way from superintelligence. DeepMind built systems that optimized point totals inside games like Pong or Space Invaders, but they were useless elsewhere.

…

Another Googler, Meredith Whittaker, a product manager in the company’s cloud computing group, helped launch a research organization at NYU. A consortium of companies, from Google to Facebook to Microsoft, built an organization called the Partnership on AI. Organizations like the Future of Life Institute (founded by Max Tegmark at MIT) and the Future of Humanity Institute (founded by Nick Bostrom at Oxford) were also concerned with the ethics of AI, but they focused on existential threats of the distant future. The new wave of ethicists focused on more immediate matters. For both Mitchell and Gebru, the bias problem was part of the larger issue across the tech industry. Women struggled to exert their influence in all tech fields, facing extreme bias in the workplace and sometimes harassment.

Army of None: Autonomous Weapons and the Future of War by Paul Scharre

"World Economic Forum" Davos, active measures, Air France Flight 447, air gap, algorithmic trading, AlphaGo, Apollo 13, artificial general intelligence, augmented reality, automated trading system, autonomous vehicles, basic income, Black Monday: stock market crash in 1987, brain emulation, Brian Krebs, cognitive bias, computer vision, cuban missile crisis, dark matter, DARPA: Urban Challenge, data science, deep learning, DeepMind, DevOps, Dr. Strangelove, drone strike, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, facts on the ground, fail fast, fault tolerance, Flash crash, Freestyle chess, friendly fire, Herman Kahn, IFF: identification friend or foe, ImageNet competition, information security, Internet of things, Jeff Hawkins, Johann Wolfgang von Goethe, John Markoff, Kevin Kelly, Korean Air Lines Flight 007, Loebner Prize, loose coupling, Mark Zuckerberg, military-industrial complex, moral hazard, move 37, mutually assured destruction, Nate Silver, Nick Bostrom, PalmPilot, paperclip maximiser, pattern recognition, Rodney Brooks, Rubik’s Cube, self-driving car, sensor fusion, South China Sea, speech recognition, Stanislav Petrov, Stephen Hawking, Steve Ballmer, Steve Wozniak, Strategic Defense Initiative, Stuxnet, superintelligent machines, Tesla Model S, The Signal and the Noise by Nate Silver, theory of mind, Turing test, Tyler Cowen, universal basic income, Valery Gerasimov, Wall-E, warehouse robotics, William Langewiesche, Y2K, zero day

Ava is intelligent, but inhuman. GOD OR GOLEM? Ex Machina’s ending is a warning against anthropomorphizing AI and assuming that just because a machine can imitate human behavior, it thinks like humans. Like Jeff Clune’s “weird” deep neural nets, advanced AI is likely to be fundamentally alien. In fact, Nick Bostrom, an Oxford philosopher and author of Superintelligence: Paths, Dangers, Strategies, has argued that biological extraterrestrials would likely have more in common with humans than with machine intelligence. Biological aliens (if they exist) would have presumably developed drives and instincts similar to ours through natural selection.

…

Stuart Armstrong, a researcher at the Future of Humanity Institute in Oxford, has given an example of a hypothetical AI that achieves this goal by burying humans in lead-lined coffins connected to heroin drips. You may ask, wouldn’t an artificial general intelligence understand that’s not what we meant? An AI that understood context and meaning might determine its programmers didn’t want lead coffins and heroin drips, but that might not matter. Nick Bostrom has argued “its final goal is to make us happy, not to do what the programmers meant when they wrote the code that represents this goal.” The problem is that any rule blindly followed to its most extreme can result in perverse outcomes. Philosophers and AI researchers have pondered the problem of what goals to give a superintelligent AI that could not lead to perverse instantiation and they have not come to any particularly satisfactory solution.

…

For an overview of current abilities and limitations in scene interpretation, see JASON, “Perspectives on Research in Artificial Intelligence and Artificial General Intelligence Relevant to DoD,” 10. 232 Brain imaging: “Human Connectome Project | Mapping the Human Brain Connectivity,” accessed June 15, 2017, http://www.humanconnectomeproject.org/. “Meet the World’s Most Advanced Brain Scanner,” Discover Magazine, accessed June 15, 2017, http://discovermagazine.com/2013/june/08-meet-the-worlds-most-advanced-brain-scanner. 232 whole brain emulations: Anders Sandburg and Nick Bostrom, “Whole Brain Emulation: A Roadmap,” Technical Report #2008-3, Oxford, UK, 2008, http://www.fhi.ox.ac.uk/Reports/2008-3.pdf 232 “When people say a technology”: Andrew Herr, email to the author, October 22, 2016. 232 “last invention”: Irving J. Good, “Speculations Concerning the First Ultraintelligent Machine”, May 1964, https://web.archive.org/web/20010527181244/http://www.aeiveos.com/~bradbury/Authors/Computing/Good-IJ/SCtFUM.html.

The Age of Em: Work, Love and Life When Robots Rule the Earth by Robin Hanson

8-hour work day, artificial general intelligence, augmented reality, Berlin Wall, bitcoin, blockchain, brain emulation, business cycle, business process, Clayton Christensen, cloud computing, correlation does not imply causation, creative destruction, deep learning, demographic transition, Erik Brynjolfsson, Ethereum, ethereum blockchain, experimental subject, fault tolerance, financial intermediation, Flynn Effect, Future Shock, Herman Kahn, hindsight bias, information asymmetry, job automation, job satisfaction, John Markoff, Just-in-time delivery, lone genius, Machinery of Freedom by David Friedman, market design, megaproject, meta-analysis, Nash equilibrium, new economy, Nick Bostrom, pneumatic tube, power law, prediction markets, quantum cryptography, rent control, rent-seeking, reversible computing, risk tolerance, Silicon Valley, smart contracts, social distancing, statistical model, stem cell, Thomas Malthus, trade route, Turing test, Tyler Cowen, Vernor Vinge, William MacAskill

June 20. http://www.bls.gov/news.release/atus.t01.htm. Buterin, Vitalik. 2014. “White Paper: A Next-Generation Smart Contract and Decentralized Application Platform.” April. https://www.ethereum.org/pdfs/EthereumWhitePaper.pdf. Caplan, Bryan. 2008. “The Totalitarian Threat.” In Global Catastrophic Risks, edited by Nick Bostrom and Milan Ćirković, 504–519. Oxford University Press, July 17. Caplan, Bryan, and Stephen Miller. 2010. “Intelligence Makes People Think Like Economists: Evidence from the General Social Survey.” Intelligence 38(6): 636–647. Card, Orson Scott. 2011. Elements of Fiction Writing—Characters and Viewpoint, 2nd ed.

…

American Economic Review 72(2): 155–159. Mulder, Monique. 1998. “The Demographic Transition: Are we any Closer to an Evolutionary Explanation?” Trends in Ecology & Evolution 13(7): 266–270. Mullainathan, Sendhil, and Eldar Shafir. 2013. Scarcity: Why Having Too Little Means So Much. Times Books, September 3. Müller, Vincent, and Nick Bostrom. 2014. “Future Progress in Artificial Intelligence: A Survey of Expert Opinion.” In Fundamental Issues of Artificial Intelligence, edited by Vincent Müller. Berlin: Springer. Mulligan, Casey, Richard Gil, and Xavier Sala-i-Martin. 2004. “Do Democracies Have Different Public Policies than Nondemocracies?”

…

MIT Sloan Management Review 50(3): 70–79. Sandberg, Anders. 2014. “Monte Carlo model of brain emulation development.” Working Paper 2014–1 (version 1.2), Future of Humanity Institute. http://www.aleph.se/papers/Monte%20Carlo%20model%20of%20brain%20emulation%20development.pdf. Sandberg, Anders, and Nick Bostrom. 2008. “Whole Brain Emulation: A Roadmap.” Technical Report #2008–2003, Future of Humanity Institute, Oxford University. http://www.fhi.ox.ac.uk/__data/assets/pdf_file/0019/3853/brain-emulation-roadmap-report.pdf. Sandstrom, Gillian, and Elizabeth Dunn. 2014. “Social Interactions and Well-Being: The Surprising Power of Weak Ties.”

100 Plus: How the Coming Age of Longevity Will Change Everything, From Careers and Relationships to Family And by Sonia Arrison

23andMe, 8-hour work day, Abraham Maslow, Albert Einstein, Anne Wojcicki, artificial general intelligence, attribution theory, Bill Joy: nanobots, bioinformatics, caloric restriction, caloric restriction, Clayton Christensen, dark matter, disruptive innovation, East Village, en.wikipedia.org, epigenetics, Frank Gehry, Googley, income per capita, indoor plumbing, Jeff Bezos, Johann Wolfgang von Goethe, Kickstarter, Larry Ellison, Law of Accelerating Returns, life extension, Nick Bostrom, personalized medicine, Peter Thiel, placebo effect, post scarcity, precautionary principle, radical life extension, Ray Kurzweil, rolodex, Silicon Valley, Silicon Valley billionaire, Simon Kuznets, Singularitarianism, smart grid, speech recognition, stem cell, Stephen Hawking, Steve Jobs, Steve Wozniak, Steven Levy, sugar pill, synthetic biology, Thomas Malthus, upwardly mobile, World Values Survey, X Prize

Audrey Chapman, a professor of community medicine and health care at the University of Connecticut Health Center, argues that “investing in new and very expensive high technologies for enhancement interventions while people in our own country lack access to basic health care and millions of people die prematurely of preventable diseases in poor countries would be yet another step toward moral bankruptcy.”31 Her argument is meant to play on “rich guilt,” but it is nonsensical because it assumes an either/or choice and completely ignores the health benefits of new innovations that could potentially lift poorer communities faster than anything available now. Technology is spreading more quickly today than ever before, and even though it may not reach poor communities at the same time as wealthy ones, it will likely get to the former faster than past innovations did. Nick Bostrom, professor of philosophy at the University of Oxford, helps to expose the weakness in Chapman’s emotionally charged argument. He writes, “It is unclear why aging research should be singled out for blame or special concern in this regard. Many factors contribute to global inequality, and spending on gerontological research is such a minute fraction of the financial outlays of wealthy nations that it seems a bizarre place to look for savings to transfer to the poor.”32 Instead, he argues that using poverty as an excuse to not advance health care innovations is an irrational psychological reaction reminiscent of Stockholm syndrome, whereby hostages become emotionally attached to their captors.

…

William Kristol and Eric Cohen (Lanham, MD: Rowman and Littlefield, 2002). 3 Ibid., 325. 4 Bill McKibben, “Mr. Natural,” Outside, May 2003, http://outside.away.com/outside/bodywork/200305/200305_mr_natural_1.html. 5 Ibid. 6 Arthur Caplan, “Good, Better, or Best?” in Human Enhancement, ed. Julian Savulescu and Nick Bostrom (New York: Oxford University Press, 2009), 202. 7 Kass, “Why Not Immortality?” 326. 8 See www.clintonbushhaitifund.org/. 9 Kass, “Why Not Immortality?” 327. 10 Daniel Callahan, cited in Christine Overall, Aging, Death, and Human Longevity: A Philosophical Inquiry (Berkeley and Los Angeles: University of California Press, 2003), 39–40. 11 Kass, “Why Not Immortality?”

…

Savulescu and Bostrom, 123–124. 31 Audrey R. Chapman, “The Social and Justice Implications of Extending the Human Lifespan,” in The Fountain of Youth: Cultural, Scientific, and Ethical Perspectives on a Biomedical Goal, ed. Stephen G. Post and Robert H. Binstock (New York: Oxford University Press, 2004), 353. 32 Nick Bostrom, “Recent Developments in the Ethics, Science, and Politics of Life-Extension,” Aging Horizons 3 (Autumn–Winter 2005): 28–34, www.fhi.ox.ac.uk/__data/assets/pdf_file/0004/5926/life-extension.pdf. 33 Ibid. 34 Jonathan Weiner, Long for This World (New York: HarperCollins, 2010), 261. 35 Francis Fukuyama, Our Posthuman Future: Consequences of the Biotechnology Revolution (New York: Picador, 2002), 172. 36 George Annas, “Genism, Racism, and the Prospect of Genetic Genocide,” The Future of Human Nature: A Symposium on the Promises and Challenges of the Revolutions in Genomics and Computer Science, Pardee Center Conference Series, Boston University, Boston, Massachusetts, April 2003. 37 Fukuyama, Our Posthuman Future, 171, 175. 38 Annas, “Genism.” 39 C.

Rule of the Robots: How Artificial Intelligence Will Transform Everything by Martin Ford

AI winter, Airbnb, algorithmic bias, algorithmic trading, Alignment Problem, AlphaGo, Amazon Mechanical Turk, Amazon Web Services, artificial general intelligence, Automated Insights, autonomous vehicles, backpropagation, basic income, Big Tech, big-box store, call centre, carbon footprint, Chris Urmson, Claude Shannon: information theory, clean water, cloud computing, commoditize, computer age, computer vision, Computing Machinery and Intelligence, coronavirus, correlation does not imply causation, COVID-19, crowdsourcing, data is the new oil, data science, deep learning, deepfake, DeepMind, Demis Hassabis, deskilling, disruptive innovation, Donald Trump, Elon Musk, factory automation, fake news, fulfillment center, full employment, future of work, general purpose technology, Geoffrey Hinton, George Floyd, gig economy, Gini coefficient, global pandemic, Googley, GPT-3, high-speed rail, hype cycle, ImageNet competition, income inequality, independent contractor, industrial robot, informal economy, information retrieval, Intergovernmental Panel on Climate Change (IPCC), Internet of things, Jeff Bezos, job automation, John Markoff, Kiva Systems, knowledge worker, labor-force participation, Law of Accelerating Returns, license plate recognition, low interest rates, low-wage service sector, Lyft, machine readable, machine translation, Mark Zuckerberg, Mitch Kapor, natural language processing, Nick Bostrom, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, Ocado, OpenAI, opioid epidemic / opioid crisis, passive income, pattern recognition, Peter Thiel, Phillips curve, post scarcity, public intellectual, Ray Kurzweil, recommendation engine, remote working, RFID, ride hailing / ride sharing, Robert Gordon, Rodney Brooks, Rubik’s Cube, Sam Altman, self-driving car, Silicon Valley, Silicon Valley startup, social distancing, SoftBank, South of Market, San Francisco, special economic zone, speech recognition, stealth mode startup, Stephen Hawking, superintelligent machines, TED Talk, The Future of Employment, The Rise and Fall of American Growth, the scientific method, Turing machine, Turing test, Tyler Cowen, Tyler Cowen: Great Stagnation, Uber and Lyft, uber lyft, universal basic income, very high income, warehouse automation, warehouse robotics, Watson beat the top human players on Jeopardy!, WikiLeaks, women in the workforce, Y Combinator

In May of that year, a group of scientists including the University of Cambridge cosmologist Stephen Hawking along with AI expert Stuart Russell and physicists Max Tegmark and Frank Wilczek co-authored an open letter published in the U.K.’s Independent declaring that the advent of artificial superintelligence “would be the biggest event in human history,” and that a computer with superhuman intellectual capability might be capable of “outsmarting financial markets, out-inventing human researchers, out-manipulating human leaders, and developing weapons we cannot even understand.” The letter warned that a failure to take this looming danger seriously might well turn out to be humanity’s “worst mistake in history.”27 Later that same year, the Oxford University philosopher Nick Bostrom published his book Superintelligence: Paths, Dangers, Strategies, which quickly became a somewhat surprising bestseller. Bostrom opens the book by pointing out that humans rule the earth purely on the basis of superior intellect. Many other animals are faster, stronger or more ferocious; it is our brains that led to dominance.

…

If a similar miscalculation were made with a superintelligent system, it might well be impossible to regain control as it sought to pursue its objective. The quest to find a solution to the control problem has become an important topic of academic research at universities and especially within specialized, privately funded organizations such as OpenAI, Oxford University’s Future of Humanity Institute, which is directed by Nick Bostrom, and the Machine Intelligence Research Institute located in Berkeley, California. In his 2019 book Human Compatible: Artificial Intelligence and the Problem of Control, Stuart Russell argues that the best solution to the problem is to not build an explicit objective function into advanced AI systems at all.

…

Stephen Hawking, Stuart Russell, Max Tegmark and Frank Wilczek, “Stephen Hawking: ‘Transcendence looks at the implications of artificial intelligence—but are we taking AI seriously enough?,’” The Independent, May 1, 2014, www.independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the-implications-of-artificial-intelligence-but-are-we-taking-ai-seriously-enough-9313474.html. 28. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies, Oxford University Press, 2014, p. vii. 29. Matt McFarland, “Elon Musk: ‘With artificial intelligence we are summoning the demon,’” Washington Post, October 24, 2014, www.washingtonpost.com/news/innovations/wp/2014/10/24/elon-musk-with-artificial-intelligence-we-are-summoning-the-demon/. 30.

Ten Billion Tomorrows: How Science Fiction Technology Became Reality and Shapes the Future by Brian Clegg

Albert Einstein, Alvin Toffler, anthropic principle, Apollo 11, Brownian motion, call centre, Carrington event, Charles Babbage, combinatorial explosion, don't be evil, Dr. Strangelove, Ernest Rutherford, experimental subject, Future Shock, game design, gravity well, Higgs boson, hive mind, invisible hand, Isaac Newton, Johannes Kepler, John von Neumann, Kickstarter, Large Hadron Collider, machine translation, Neil Armstrong, Nick Bostrom, nuclear winter, pattern recognition, quantum entanglement, RAND corporation, Ray Kurzweil, RFID, Richard Feynman, Schrödinger's Cat, Search for Extraterrestrial Intelligence, silicon-based life, speech recognition, stem cell, Stephen Hawking, Steve Jobs, Turing test

If you believe the copy of this e-book you are reading infringes on the author’s copyright, please notify the publisher at: us.macmillanusa.com/piracy. FOR GILLIAN, CHELSEA, AND REBECCA ACKNOWLEDGMENTS My thanks to my editors, Michael Homler and Lauren Jablonski, and everyone at St. Martin’s Press who have made the book possible. Although I must thank the academic writers, like philosophy professor Nick Bostrom, who have thought long and hard about the implications that changes in technology could have on society, I particularly have to think of the giants of science fiction who inspired me to get into science in the first place and who provided the inspiration for most of these chapters. This list does include modern writers like Adam Roberts and the late, much lamented, Iain M.

…

Most fiction concentrates on wireheading as a form of addictive entertainment, giving an experience that is better than real life, instead of being used to fool the population as it is in The Matrix. But what doesn’t seem to have crossed the minds of writers as yet is the suggestion from philosopher Nick Bostrom that computers themselves could—and arguably inevitably would—also become wireheads if they ever developed consciousness. As Bostrom points out, the attempts to develop artificial intelligence could be fraught with dangers. We have very little reason to assume that a truly intelligent computer would do our bidding, preferring whatever best suited its own agenda.

…

Power Grid (January 2010), available at web.ornl.gov/sci/ees/etsd/pes/pubs/ferc_Meta-R-320.pdf. Larry Niven first used the term “wirehead” in the novella The Defenseless Dead, published in the collection Roger Elwood (ed.), Playgrounds of the Mind (New York: Fawcett 1973). The idea of artificial intelligences tending to become wireheads is raised in Nick Bostrom, Superintelligence (Oxford, UK: Oxford University Press, 2014), p. 122. 3. BUY ME Quotes on science fiction advertising and marketing methods are from Frederik Pohl and C. M. Kornbluth, The Space Merchants (London: Penguin Books, 1965), pp. 9–10. The more aggressive Mokie-Koke advertising is described in Frederik Pohl, The Merchants’ War (London: Futura, 1986), pp. 47–49.

Upstream: The Quest to Solve Problems Before They Happen by Dan Heath

"Hurricane Katrina" Superdome, Affordable Care Act / Obamacare, airport security, Albert Einstein, bank run, British Empire, Buckminster Fuller, call centre, cloud computing, cognitive dissonance, colonial rule, correlation does not imply causation, cuban missile crisis, en.wikipedia.org, epigenetics, food desert, high-speed rail, Housing First, illegal immigration, Internet of things, mandatory minimum, millennium bug, move fast and break things, Nick Bostrom, payday loans, Ralph Nader, RAND corporation, randomized controlled trial, self-driving car, Skype, Snapchat, subscription business, systems thinking, urban planning, Watson beat the top human players on Jeopardy!, Y2K

In other words, if a hacker was dead set on breaking into West Aurora School District No. 129—or any other specific institution, for that matter—then the difference between 29% and 5% is immaterial. For many hacking purposes, you just need one open door. Just that one gullible person who will click on anything. Nick Bostrom, a Swedish philosopher at the University of Oxford, contemplates whether technological innovation has left modern society on the verge of a similar kind of vulnerability—a situation in which the fate of everyone could hinge on a single bad break or bad actor. The context of his comments is mankind’s tendency to keep pushing for new innovations almost without regard for the consequences.

…

.; “Education Technologies: Data Collection and Unsecured Systems Could Pose Risks to Students,” Federal Bureau of Investigation, Public Service Announcement, September 13, 2018, https://www.ic3.gov/media/2018/180913.aspx. 5% in recent attempts: Interview with Don Ringelestein, May 2019, and subsequent correspondence and phishing security test data. “The Vulnerable World Hypothesis”: Nick Bostrom, “The Vulnerable World Hypothesis,” 2018, https://nickbostrom.com/papers/vulnerable.pdf. there are DNA “printers”: Rob Stein, “DNA Printing, A Big Boon to Research, But Some Raise Concerns,” NPR, May 7, 2015, https://www.npr.org/sections/health-shots/2015/05/07/404460240/dna-printing-a-big-boon-to-research-but-some-raise-concerns.

…

founded the Future of Humanity Institute: To learn more about Bostrom’s work, check out this fascinating profile: Raffi Khatchadourian, “The Doomsday Invention: Will Artificial Intelligence Bring Us Utopia or Destruction?,” The New Yorker, November 23, 2015, https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom. Click Here to Kill Everybody: Bruce Schneier, Click Here to Kill Everybody: Security and Survival in a Hyper-connected World (New York: Norton, 2018). “at risk from Moon germs”: Michael Meltzer, When Biospheres Collide: A History of NASA’s Planetary Protection Programs (US National Aeronautics and Space Administration, 2010), BiblioGov, 215.

Rationality: From AI to Zombies by Eliezer Yudkowsky

Albert Einstein, Alfred Russel Wallace, anthropic principle, anti-pattern, anti-work, antiwork, Arthur Eddington, artificial general intelligence, availability heuristic, backpropagation, Bayesian statistics, behavioural economics, Berlin Wall, Boeing 747, Build a better mousetrap, Cass Sunstein, cellular automata, Charles Babbage, cognitive bias, cognitive dissonance, correlation does not imply causation, cosmological constant, creative destruction, Daniel Kahneman / Amos Tversky, dematerialisation, different worldview, discovery of DNA, disinformation, Douglas Hofstadter, Drosophila, Eddington experiment, effective altruism, experimental subject, Extropian, friendly AI, fundamental attribution error, Great Leap Forward, Gödel, Escher, Bach, Hacker News, hindsight bias, index card, index fund, Isaac Newton, John Conway, John von Neumann, Large Hadron Collider, Long Term Capital Management, Louis Pasteur, mental accounting, meta-analysis, mirror neurons, money market fund, Monty Hall problem, Nash equilibrium, Necker cube, Nick Bostrom, NP-complete, One Laptop per Child (OLPC), P = NP, paperclip maximiser, pattern recognition, Paul Graham, peak-end rule, Peter Thiel, Pierre-Simon Laplace, placebo effect, planetary scale, prediction markets, random walk, Ray Kurzweil, reversible computing, Richard Feynman, risk tolerance, Rubik’s Cube, Saturday Night Live, Schrödinger's Cat, scientific mainstream, scientific worldview, sensible shoes, Silicon Valley, Silicon Valley startup, Singularitarianism, SpaceShipOne, speech recognition, statistical model, Steve Jurvetson, Steven Pinker, strong AI, sunk-cost fallacy, technological singularity, The Bell Curve by Richard Herrnstein and Charles Murray, the map is not the territory, the scientific method, Turing complete, Turing machine, Tyler Cowen, ultimatum game, X Prize, Y Combinator, zero-sum game

., Sandel’s objection that tampering with our biology too much would make life feel like less of a “gift.”2,3 Bostrom’s “History of Transhumanist Thought” provides context for the debate.4 2. Nick Bostrom, “A History of Transhumanist Thought,” Journal of Evolution and Technology 14, no. 1 (2005): 1–25, http://www.nickbostrom.com/papers/history.pdf. 3. Michael Sandel, “What’s Wrong With Enhancement,” Background material for the President’s Council on Bioethics. (2002). 4. Nick Bostrom and Julian Savulescu, “Human Enhancement Ethics: The State of the Debate,” in Human Enhancement, ed. Nick Bostrom and Julian Savulescu (2009). Part U Fake Preferences 257 Not for the Sake of Happiness (Alone) When I met the futurist Greg Stock some years ago, he argued that the joy of scientific discovery would soon be replaced by pills that could simulate the joy of scientific discovery.

…

Alt and Morris Rubinoff, vol. 6 (New York: Academic Press, 1965), 31–88, doi:10.1016/S0065-2458(08)60418-0. 3. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford University Press, 2014). 4. Stuart J. Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, 3rd ed. (Upper Saddle River, NJ: Prentice-Hall, 2010). 5. Bostrom and Ćirković, Global Catastrophic Risks. 6. An example of a possible existential risk is the “grey goo” scenario, in which molecular robots designed to efficiently self-replicate do their job too well, rapidly outcompeting living organisms as they consume the Earth’s available matter. 7. Nick Bostrom and Eliezer Yudkowsky, “The Ethics of Artificial Intelligence,” in The Cambridge Handbook of Artificial Intelligence, ed.

…

Remember that, in fact, you did not guess. Remember how, century after century, the world changed in ways you did not guess. Maybe then you will be less shocked by what happens next. * 1. Eliezer Yudkowsky, “Cognitive Biases Potentially Affecting Judgment of Global Risks,” in Global Catastrophic Risks, ed. Nick Bostrom and Milan M. Ćirković (New York: Oxford University Press, 2008), 91–119. 43 Explain/Worship/Ignore? As our tribe wanders through the grasslands, searching for fruit trees and prey, it happens every now and then that water pours down from the sky. “Why does water sometimes fall from the sky?”

Falter: Has the Human Game Begun to Play Itself Out? by Bill McKibben

"Hurricane Katrina" Superdome, 23andMe, Affordable Care Act / Obamacare, Airbnb, Alan Greenspan, American Legislative Exchange Council, An Inconvenient Truth, Anne Wojcicki, Anthropocene, Apollo 11, artificial general intelligence, Bernie Sanders, Bill Joy: nanobots, biodiversity loss, Burning Man, call centre, Cambridge Analytica, carbon footprint, carbon tax, Charles Lindbergh, clean water, Colonization of Mars, computer vision, CRISPR, David Attenborough, deep learning, DeepMind, degrowth, disinformation, Donald Trump, double helix, driverless car, Easter island, Edward Snowden, Elon Musk, ending welfare as we know it, energy transition, Extinction Rebellion, Flynn Effect, gigafactory, Google Earth, Great Leap Forward, green new deal, Greta Thunberg, Hyperloop, impulse control, income inequality, Intergovernmental Panel on Climate Change (IPCC), James Bridle, Jane Jacobs, Jaron Lanier, Jeff Bezos, job automation, Kim Stanley Robinson, life extension, light touch regulation, Mark Zuckerberg, mass immigration, megacity, Menlo Park, moral hazard, Naomi Klein, Neil Armstrong, Nelson Mandela, Nick Bostrom, obamacare, ocean acidification, off grid, oil shale / tar sands, paperclip maximiser, Paris climate accords, pattern recognition, Peter Thiel, plutocrats, profit motive, Ralph Waldo Emerson, Ray Kurzweil, Robert Mercer, Ronald Reagan, Sam Altman, San Francisco homelessness, self-driving car, Silicon Valley, Silicon Valley startup, smart meter, Snapchat, stem cell, Stephen Hawking, Steve Jobs, Steve Wozniak, Steven Pinker, strong AI, supervolcano, tech baron, tech billionaire, technoutopianism, TED Talk, The Wealth of Nations by Adam Smith, traffic fines, Tragedy of the Commons, Travis Kalanick, Tyler Cowen, urban sprawl, Virgin Galactic, Watson beat the top human players on Jeopardy!, Y Combinator, Y2K, yield curve

Hawking wrote that success in AI would be “the biggest event in human history,” but it might “also be the last, unless we learn to avoid the risks.”20 And here’s Michael Vassar, president of the Machine Intelligence Research Institute: “I definitely think people should try to develop Artificial General Intelligence with all due care. In this case all due care means much more scrupulous caution than would be necessary for dealing with Ebola or plutonium.”21 Why are people so scared? Let the Swedish philosopher Nick Bostrom explain. He’s hardly a Luddite. Indeed, he gave a speech in 1999 to a California convention of “transhumanists” that may mark the rhetorical high water of the entire techno-utopian movement. Thanks to ever-increasing computer power and ever-shinier biotech, he predicted then, we would soon have “values that will strike us as being of a far higher order than those we can realize as unenhanced biological humans,” not to mention “love that is stronger, purer, and more secure than any human has yet harbored,” not to mention “orgasms … whose blissfulness vastly exceeds what any human has yet experienced.”22 But fifteen years later, ensconced in Oxford as nothing less than the director of the Future of Humanity Institute, he’d begun to worry a great deal: “In fairy tales you have genies who grant wishes,” he told a reporter for The New Yorker.

…

The point of art is not “better”; the point is to reflect on the experience of being human—which is precisely the thing that’s disappearing. Even—and here is more rich, sad irony—science is at risk. The profound pleasure that keeps people working on precisely the technology that now threatens to supplant us will vanish, too. You don’t think robot biologists will soon replace the real ones? What are we left with? Nick Bostrom, the early apostle of transhumanism, offers the best case: a superintelligence could “assist us in creating a highly appealing experiential world in which we could live lives devoted to joyful game-playing, relating to each other, experiencing personal growth, and living closer to our ideals.” Or we could just smoke weed

…

James Vincent, “Elon Musk Says We Need to Regulate AI Before It Becomes a Danger to Humanity,” theverge.com, July 17, 2017. 20. Stephen Hawking, “Artificial Intelligence Could Be the Greatest Disaster in Human History,” Independent, October 20, 2016. 21. James Barrat, Our Final Invention: Artificial Intelligence and the End of the Human Era (New York: St. Martin’s Press, 2013), p. 34. 22. Nick Bostrom, “A Transhumanist Perspective on Genetic Enhancements,” nickbostrom.com, 2003. 23. Khatchadourian, “Doomsday Invention.” 24. Stephen M. Omohundro, “The Basic A.I. Drives,” in Artificial General Intelligence 2008, eds. Pei Wang, Ben Goertzel, and Stan Franklin (Amsterdam: IOS Press, 2008), available online at selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf, p. 9. 25.

Singularity Rising: Surviving and Thriving in a Smarter, Richer, and More Dangerous World by James D. Miller

23andMe, affirmative action, Albert Einstein, artificial general intelligence, Asperger Syndrome, barriers to entry, brain emulation, cloud computing, cognitive bias, correlation does not imply causation, crowdsourcing, Daniel Kahneman / Amos Tversky, David Brooks, David Ricardo: comparative advantage, Deng Xiaoping, en.wikipedia.org, feminist movement, Flynn Effect, friendly AI, hive mind, impulse control, indoor plumbing, invention of agriculture, Isaac Newton, John Gilmore, John von Neumann, knowledge worker, Larry Ellison, Long Term Capital Management, low interest rates, low skilled workers, Netflix Prize, neurotypical, Nick Bostrom, Norman Macrae, pattern recognition, Peter Thiel, phenotype, placebo effect, prisoner's dilemma, profit maximization, Ray Kurzweil, recommendation engine, reversible computing, Richard Feynman, Rodney Brooks, Silicon Valley, Singularitarianism, Skype, statistical model, Stephen Hawking, Steve Jobs, sugar pill, supervolcano, tech billionaire, technological singularity, The Coming Technological Singularity, the scientific method, Thomas Malthus, transaction costs, Turing test, twin studies, Vernor Vinge, Von Neumann architecture

Nothing, therefore, comes close to the expected return, per charitable dollar, of donating to an existential risk-reducing organization.107 MY LONG-SHOT IDEA FOR FRIENDLINESS108 I have come up with a long-shot way of causing an ultra-intelligence to modify itself to become friendly. My idea is premised on Oxford philosopher Nick Bostrom’s conjecture that we might be living in a computer simulation.109 Bostrom believes that a sufficiently advanced AI could easily simulate our entire planet and the thought processes of every living thing on it. Bostrom argues that, if mankind survives long enough, a future historian would likely want to simulate the pre-Singularity past to learn about how we made decisions.

…

But the more people who independently locate the cave, the less concealed we should expect it to have been. Now think of the cave as the genes necessary for “complex learning, memory, and tool use” and the evolutionary forces operating in different creatures as the searchers. As Carl Shulman and Nick Bostrom write, across different species evolution independently produced “complex learning, memory, and tool use both within and without the line of human ancestry.”336 Understanding convergent evolution should reduce our estimate of how hard it must be to develop the part of humanity’s intellectual toolkit shared by reasonably bright creatures such as octopuses, elephants, and crows.337 If the blind forces of evolution could find this toolkit several times, mankind should be able to “find it” as well.

…

Super-Economy. http://super-economy.blogspot.com/2011/04/iq-income-and-wealth.html. Segal, Nancy L. 1999. Entwined Lives: Twins and What They Tell Us About Human Behavior. New York: Dutton. Shulman, Carl. November 23, 2008. “‘Evicting’ Brain Emulations.” Overcoming Bias (blog). http://www.overcomingbias.com/2008/11/suppose-that-ro.html. Shulman, Carl, and Nick Bostrom. 2012. “How Hard is Artificial Intelligence? The Evolutionary Argument and Observation Selection Effects.” Forthcoming in the Journal of Consciousness Studies. http://www.nickbostrom.com/aievolution.pdf. Singularity Institute for Artificial Intelligence. 2007. The Singularity Summit 2007: AI and the Future of Humanity. http://singinst.org/summit2007/quotes/billgates/.

Global Catastrophic Risks by Nick Bostrom, Milan M. Cirkovic

affirmative action, agricultural Revolution, Albert Einstein, American Society of Civil Engineers: Report Card, anthropic principle, artificial general intelligence, Asilomar, availability heuristic, backpropagation, behavioural economics, Bill Joy: nanobots, Black Swan, carbon tax, carbon-based life, Charles Babbage, classic study, cognitive bias, complexity theory, computer age, coronavirus, corporate governance, cosmic microwave background, cosmological constant, cosmological principle, cuban missile crisis, dark matter, death of newspapers, demographic transition, Deng Xiaoping, distributed generation, Doomsday Clock, Drosophila, endogenous growth, Ernest Rutherford, failed state, false flag, feminist movement, framing effect, friendly AI, Georg Cantor, global pandemic, global village, Great Leap Forward, Gödel, Escher, Bach, Hans Moravec, heat death of the universe, hindsight bias, information security, Intergovernmental Panel on Climate Change (IPCC), invention of agriculture, Kevin Kelly, Kuiper Belt, Large Hadron Collider, launch on warning, Law of Accelerating Returns, life extension, means of production, meta-analysis, Mikhail Gorbachev, millennium bug, mutually assured destruction, Nick Bostrom, nuclear winter, ocean acidification, off-the-grid, Oklahoma City bombing, P = NP, peak oil, phenotype, planetary scale, Ponzi scheme, power law, precautionary principle, prediction markets, RAND corporation, Ray Kurzweil, Recombinant DNA, reversible computing, Richard Feynman, Ronald Reagan, scientific worldview, Singularitarianism, social intelligence, South China Sea, strong AI, superintelligent machines, supervolcano, synthetic biology, technological singularity, technoutopianism, The Coming Technological Singularity, the long tail, The Turner Diaries, Tunguska event, twin studies, Tyler Cowen, uranium enrichment, Vernor Vinge, War on Poverty, Westphalian system, Y2K

Global Catastrophic Risks Global Catastrophic Risks Edited by Nick Bostrom Milan M. Cirkovic OXFORD UNIVERSITY PRESS OXFORD UNIVERSITY PRESS Great Clarendon Street, Oxford OX2 6DP Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide in Oxford New York Auckland Cape Town Dares Salaam Hong Kong Karachi Kuala Lumpur Madrid Melbourne Mexico City N airobi New Delhi Shanghai Taipei Toronto With offices in Argentina Austria Brazil Chile Czech Republic France Greece Guatemala H ungary Italy japan Poland Portugal Singapore South Korea Switzerland Thailand Turkey Ukraine Vietnam Oxford is a registered trade mark of Oxford University Press in the UK and in certain other countries Published in the United States by Oxford University Press Inc., New York © Oxford University Press 2008 The moral rights of the authors have been asserted Database right Oxford University Press (maker) First published 2008 All rights reserved.

…

Rees Contents Acknowledgements . . . . . . . . . . . . . . . . . . . . . . .. . . . . . ... . .. . . . . . . . . . . . . . . . . . . . . . . . . . v Foreword .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . ... . . . .... . . . . . . . . . . . . . . . . . . . . vii . Martin]. Rees 1 Introduction .................................................................... 1 Nick Bostrom and Milan M. Cirkovic 1.1 1.2 1.3 1 .4 1.5 1.6 1.7 Part I Why? ...... . . . . . . . . .. . . . . .. . .... . . . . . . . . . . . . . . . . . . . . ... . ............ . .. . . 1 Taxonomy and organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Part I : Background . . . . . . . ...... . . . . . . . . .. . . . . . . . . .. . ............. . . . .. 7 Part I I : Risks from nature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . 1 3 Part I I I : Risks from unintended consequences . . . . . . . . . . . . . . . . . . . . 1 5 Part IV: Risks from hostile acts . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . 20 Conclusions and future directions . . . . . . . . .. . .. . . . . . . . . . . . . .. . . . . . . . 2 7 Background 31 2 Long-term astrophysical processes . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Fred C.

…

Suggestions for further reading . . . . . . . . . . . .. 504 506 510 511 512 514 514 515 516 518 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Authors' biographies . .......................................................... 520 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531 · 1 · I ntrod u cti on Nick Bostrom and Milan M. Cirkovic 1 . 1 Why? The term 'global catastrophic risk' lacks a sharp definition. We use it to refer, loosely, to a risk that might have the potential to inflict serious damage to human well-being on a global scale. On this definition, an immensely diverse collection of events could constitute global catastrophes: potential candidates range from volcanic eruptions to pandemic infections , nuclear accidents to worldwide tyrannies, out-of-control scientific experiments to climatic changes, and cosmic hazards to economic collapse.

Superminds: The Surprising Power of People and Computers Thinking Together by Thomas W. Malone

Abraham Maslow, agricultural Revolution, Airbnb, Albert Einstein, Alvin Toffler, Amazon Mechanical Turk, Apple's 1984 Super Bowl advert, Asperger Syndrome, Baxter: Rethink Robotics, bitcoin, blockchain, Boeing 747, business process, call centre, carbon tax, clean water, Computing Machinery and Intelligence, creative destruction, crowdsourcing, data science, deep learning, Donald Trump, Douglas Engelbart, Douglas Engelbart, driverless car, drone strike, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, experimental economics, Exxon Valdez, Ford Model T, future of work, Future Shock, Galaxy Zoo, Garrett Hardin, gig economy, happiness index / gross national happiness, independent contractor, industrial robot, Internet of things, invention of the telegraph, inventory management, invisible hand, Jeff Rulifson, jimmy wales, job automation, John Markoff, Joi Ito, Joseph Schumpeter, Kenneth Arrow, knowledge worker, longitudinal study, Lyft, machine translation, Marshall McLuhan, Nick Bostrom, Occupy movement, Pareto efficiency, pattern recognition, prediction markets, price mechanism, radical decentralization, Ray Kurzweil, Rodney Brooks, Ronald Coase, search costs, Second Machine Age, self-driving car, Silicon Valley, slashdot, social intelligence, Stephen Hawking, Steve Jobs, Steven Pinker, Stewart Brand, technological singularity, The Nature of the Firm, The Wealth of Nations by Adam Smith, The Wisdom of Crowds, theory of mind, Tim Cook: Apple, Tragedy of the Commons, transaction costs, Travis Kalanick, Uber for X, uber lyft, Vernor Vinge, Vilfredo Pareto, Watson beat the top human players on Jeopardy!

But with recent progress in AI, some very smart people, including Stephen Hawking, Elon Musk, and my MIT colleague Max Tegmark, have begun to believe that the risks of this actually happening are large enough for us to take them seriously. In fact, Hawking, Tegmark, and others have formed the Future of Life Institute to study these and other existential risks to humanity.18 One of the most complete and carefully reasoned descriptions of the problem is in the book Superintelligence, by Nick Bostrom.19 Bostrom carefully analyzes various ways that artificially intelligent systems (AIs) could eventually reach human-level intelligence. Even though, as we’ve seen, this is not likely to happen for decades or perhaps centuries, he observes that once it does happen, these machines will be able to use their intelligence to keep improving themselves.

…

Stuart Armstrong and Kaj Sotala, “How We’re Predicting AI—or Failing To,” in Beyond AI: Artificial Dreams, ed. Jan Romportl, Pavel Ircing, Eva Zackova, Michal Polak, and Radek Schuster (Pilsen, Czech Republic: University of West Bohemia, 2012): 52–75, https://intelligence.org/files/PredictingAI.pdf. 13. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014). 14. Stuart Madnick, “Understanding the Computer (Little Man Computer),” unpublished teaching note, MIT Sloan School of Management, Cambridge, MA, June 10, 1993. Based on the 1979 version. The figure shown here was drawn by Rob Malone, and is adapted from this teaching note with permission of Stuart Madnick. 15.

…

seq=1#page_scan_tab_contents. 17. Ira Flatow and Howard Market, “Science Diction: The Origin of the Word ‘Robot,’” April 22, 2011, National Public Radio, http://www.npr.org/2011/04/22/135634400/science-diction-the-origin-of-the-word-robot. 18. Future of Life Institute, http://futureoflife.org. 19. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014); Ray Kurzweil, The Age of Spiritual Machines: When Computers Exceed Human Intelligence (New York: Viking, 1999). 20. Peter M. Asaro, “The Liability Problem for Autonomous Artificial Agents,” in Ethical and Moral Considerations in Non-Human Agents: Papers from the 2016 AAAI Spring Symposium (Palo Alto, CA: AAAI Press, 2016), https://www.aaai.org/ocs/index.php/SSS/SSS16/paper/download/12699/11949. 21.

A World Without Work: Technology, Automation, and How We Should Respond by Daniel Susskind

"World Economic Forum" Davos, 3D printing, agricultural Revolution, AI winter, Airbnb, Albert Einstein, algorithmic trading, AlphaGo, artificial general intelligence, autonomous vehicles, basic income, Bertrand Russell: In Praise of Idleness, Big Tech, blue-collar work, Boston Dynamics, British Empire, Capital in the Twenty-First Century by Thomas Piketty, cloud computing, computer age, computer vision, computerized trading, creative destruction, David Graeber, David Ricardo: comparative advantage, deep learning, DeepMind, Demis Hassabis, demographic transition, deskilling, disruptive innovation, Donald Trump, Douglas Hofstadter, driverless car, drone strike, Edward Glaeser, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, fake news, financial innovation, flying shuttle, Ford Model T, fulfillment center, future of work, gig economy, Gini coefficient, Google Glasses, Gödel, Escher, Bach, Hans Moravec, income inequality, income per capita, industrial robot, interchangeable parts, invisible hand, Isaac Newton, Jacques de Vaucanson, James Hargreaves, job automation, John Markoff, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Joi Ito, Joseph Schumpeter, Kenneth Arrow, Kevin Roose, Khan Academy, Kickstarter, Larry Ellison, low skilled workers, lump of labour, machine translation, Marc Andreessen, Mark Zuckerberg, means of production, Metcalfe’s law, natural language processing, Neil Armstrong, Network effects, Nick Bostrom, Occupy movement, offshore financial centre, Paul Samuelson, Peter Thiel, pink-collar, precariat, purchasing power parity, Ray Kurzweil, ride hailing / ride sharing, road to serfdom, Robert Gordon, Sam Altman, Second Machine Age, self-driving car, shareholder value, sharing economy, Silicon Valley, Snapchat, social intelligence, software is eating the world, sovereign wealth fund, spinning jenny, Stephen Hawking, Steve Jobs, strong AI, tacit knowledge, technological solutionism, TED Talk, telemarketer, The Future of Employment, The Rise and Fall of American Growth, the scientific method, The Theory of the Leisure Class by Thorstein Veblen, The Wealth of Nations by Adam Smith, Thorstein Veblen, Travis Kalanick, Turing test, Two Sigma, Tyler Cowen, Tyler Cowen: Great Stagnation, universal basic income, upwardly mobile, warehouse robotics, Watson beat the top human players on Jeopardy!, We are the 99%, wealth creators, working poor, working-age population, Y Combinator

For many AI researchers, the intellectual holy grail is to build machines that are foxes rather than hedgehogs. In their terminology, they want to build an “artificial general intelligence” (AGI), with wide-ranging capabilities, rather than an “artificial narrow intelligence” (ANI), which can only handle very particular assignments.21 That is what interests futurists like Ray Kurzweil and Nick Bostrom. But there has been little success in that effort, and critics often put forward the elusiveness of AGI as a further reason for being skeptical about the capabilities of machines. There is a sense among purists that only AGI is “real” AI, and that without this generality of capability these machines will never be “true rivals” to human beings in the work that they do.22 AGI, it is said, will represent a turning point in human history—perhaps the turning point.

…

The distinction between AGI and ANI is often conflated with another one made by John Searle, who speaks of the difference between “strong” AI and “weak” AI. But the two are not the same thing at all. AGI and ANI reflect the breadth of a machine’s capability, while Searle’s terms describe whether a machine thinks like a human being (“strong”) or unlike one (“weak”). 22.  Nick Bostrom and Eliezer Yudkowsky, “The Ethics of Artificial Intelligence” in William Ramsey and Keith Frankish, eds., Cambridge Handbook of Artificial Intelligence (Cambridge: Cambridge University Press, 2011). 23.  Irving John Good, “Speculations Concerning the First Ultraintelligent Machine,” Advances in Computers 6 (1966): 31–88. 24.  

…

Rory Cellan-Jones, “Stephen Hawking Warns Artificial Intelligence Could End Mankind,” BBC News, 2 December 2014; Samuel Gibbs, “Elon Musk: AI ‘Vastly More Risky Than North Korea,’” Guardian, 14 August 2017; Kevin Rawlinson, “Microsoft’s Bill Gates Insists AI Is a Threat,” BBC News, 29 January 2015. 25.  See Nick Bostrom, “Ethical Issues in Advanced Artificial Intelligence” in George Lasker, Wendell Wallach, and Iva Smit, eds., Cognitive, Emotive, and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence (Windsor, ON: International Institute of Advanced Studies in Systems Research and Cybernetics, 2003), pp. 12–17. 26.  

Future Politics: Living Together in a World Transformed by Tech by Jamie Susskind

3D printing, additive manufacturing, affirmative action, agricultural Revolution, Airbnb, airport security, algorithmic bias, AlphaGo, Amazon Robotics, Andrew Keen, Apollo Guidance Computer, artificial general intelligence, augmented reality, automated trading system, autonomous vehicles, basic income, Bertrand Russell: In Praise of Idleness, Big Tech, bitcoin, Bletchley Park, blockchain, Boeing 747, brain emulation, Brexit referendum, British Empire, business process, Cambridge Analytica, Capital in the Twenty-First Century by Thomas Piketty, cashless society, Cass Sunstein, cellular automata, Citizen Lab, cloud computing, commons-based peer production, computer age, computer vision, continuation of politics by other means, correlation does not imply causation, CRISPR, crowdsourcing, cryptocurrency, data science, deep learning, DeepMind, digital divide, digital map, disinformation, distributed ledger, Donald Trump, driverless car, easy for humans, difficult for computers, Edward Snowden, Elon Musk, en.wikipedia.org, end-to-end encryption, Erik Brynjolfsson, Ethereum, ethereum blockchain, Evgeny Morozov, fake news, Filter Bubble, future of work, Future Shock, Gabriella Coleman, Google bus, Google X / Alphabet X, Googley, industrial robot, informal economy, intangible asset, Internet of things, invention of the printing press, invention of writing, Isaac Newton, Jaron Lanier, John Markoff, Joseph Schumpeter, Kevin Kelly, knowledge economy, Large Hadron Collider, Lewis Mumford, lifelogging, machine translation, Metcalfe’s law, mittelstand, more computing power than Apollo, move fast and break things, natural language processing, Neil Armstrong, Network effects, new economy, Nick Bostrom, night-watchman state, Oculus Rift, Panopticon Jeremy Bentham, pattern recognition, payday loans, Philippa Foot, post-truth, power law, price discrimination, price mechanism, RAND corporation, ransomware, Ray Kurzweil, Richard Stallman, ride hailing / ride sharing, road to serfdom, Robert Mercer, Satoshi Nakamoto, Second Machine Age, selection bias, self-driving car, sexual politics, sharing economy, Silicon Valley, Silicon Valley startup, Skype, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, smart contracts, Snapchat, speech recognition, Steve Bannon, Steve Jobs, Steve Wozniak, Steven Levy, tech bro, technological determinism, technological singularity, technological solutionism, the built environment, the Cathedral and the Bazaar, The Structural Transformation of the Public Sphere, The Wisdom of Crowds, Thomas L Friedman, Tragedy of the Commons, trolley problem, universal basic income, urban planning, Watson beat the top human players on Jeopardy!, work culture , working-age population, Yochai Benkler

Would we still seek to live together in the same way? Now imagine a world in which AI systems achieve superintelligence, that is, an intellect ‘that greatly exceeds the cognitive performance of humans in virtually all domains of interest’.19 Serious students of AI consider this a real possibility. Nick Bostrom, a scholar at the University of Oxford, offers the ‘weak conclusion’ that:20 it may be reasonable to believe that human-level machine intelligence has a fairly sizeable chance of being developed by mid-century, and that it has a non-trivial chance of being developed considerably sooner or much later; that it might perhaps fairly soon thereafter result in superintelligence.

…

Topol, ‘Attack of the Killer Robots’, BuzzFeed News, 26 August 2016 <https://www.buzzfeed.com/sarahatopol/how-tosave-mankind-from-the-new-breed-of-killer-robots?utm_term=. nm1GdWDBZ#.vaJzgW6va>) (accessed 28 November 2017). Cade Metz, ‘Google’s AI Wins Fifth and Final Game Against Go’, Wired, 15 March 2016 <https://www.wired.com/2016/03/googlesai-wins-fifth-final-game-go-genius-lee-sedol/> (accessed 28 November 2017); Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), 12–13. Sam Byford, ‘AlphaGo beats Ke Jie Again to Wrap Up Three-part March’, The Verge, 25 May 2017 <https://www.theverge.com/2017/ 5/25/15689462/alphago-ke-jie-game-2-result-google-deepmindchina> (accessed 28 November 2017).

…

Kyle Mizokami, ‘The Pentagon Wants to Use Bitcoin Technology to Protect Nuclear Weapons’, Popular Mechanics, 11 October 2016 <http://www.popularmechanics.com/military/research/a23336/ the-pentagon-wants-to-use-bitcoin-technology-to-guardnuclearweapons/?utm_content=buffer98698&utm_medium= social&utm_source=twitter.com&utm_campaign=buffer> (accessed 30 November 2017). Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), ch. 10. Murray Shanahan, The Technological Singularity (Cambridge, Mass: MIT Press, 2015), 153. Matt Burgess, ‘Samsung is Working on Putting AI Voice Assistant Bixby in Your TV and Fridge’, Wired, 27 June 2017 <https://www.

Superintelligence: Paths, Dangers, Strategies by Nick Bostrom

agricultural Revolution, AI winter, Albert Einstein, algorithmic trading, anthropic principle, Anthropocene, anti-communist, artificial general intelligence, autism spectrum disorder, autonomous vehicles, backpropagation, barriers to entry, Bayesian statistics, bioinformatics, brain emulation, cloud computing, combinatorial explosion, computer vision, Computing Machinery and Intelligence, cosmological constant, dark matter, DARPA: Urban Challenge, data acquisition, delayed gratification, Demis Hassabis, demographic transition, different worldview, Donald Knuth, Douglas Hofstadter, driverless car, Drosophila, Elon Musk, en.wikipedia.org, endogenous growth, epigenetics, fear of failure, Flash crash, Flynn Effect, friendly AI, general purpose technology, Geoffrey Hinton, Gödel, Escher, Bach, hallucination problem, Hans Moravec, income inequality, industrial robot, informal economy, information retrieval, interchangeable parts, iterative process, job automation, John Markoff, John von Neumann, knowledge worker, Large Hadron Collider, longitudinal study, machine translation, megaproject, Menlo Park, meta-analysis, mutually assured destruction, Nash equilibrium, Netflix Prize, new economy, Nick Bostrom, Norbert Wiener, NP-complete, nuclear winter, operational security, optical character recognition, paperclip maximiser, pattern recognition, performance metric, phenotype, prediction markets, price stability, principal–agent problem, race to the bottom, random walk, Ray Kurzweil, recommendation engine, reversible computing, search costs, social graph, speech recognition, Stanislav Petrov, statistical model, stem cell, Stephen Hawking, Strategic Defense Initiative, strong AI, superintelligent machines, supervolcano, synthetic biology, technological singularity, technoutopianism, The Coming Technological Singularity, The Nature of the Firm, Thomas Kuhn: the structure of scientific revolutions, time dilation, Tragedy of the Commons, transaction costs, trolley problem, Turing machine, Vernor Vinge, WarGames: Global Thermonuclear War, Watson beat the top human players on Jeopardy!, World Values Survey, zero-sum game

Superintelligence SUPERINTELLIGENCE Paths, Dangers, Strategies NICK BOSTROM Director, Future of Humanity Institute Professor, Faculty of Philosophy & Oxford Martin School University of Oxford Great Clarendon Street, Oxford, OX2 6DP, United Kingdom Oxford University Press is a department of the University of Oxford. It furthers the University’s objective of excellence in research, scholarship, and education by publishing worldwide. Oxford is a registered trade mark of Oxford University Press in the UK and in certain other countries © Nick Bostrom 2014 The moral rights of the author have been asserted First Edition published in 2014 Impression: 1 All rights reserved.

…

Bostrom, Nick and Sandberg, Anders. 2009a. “Cognitive Enhancement: Methods, Ethics, Regulatory Challenges.” Science and Engineering Ethics 15 (3): 311–41. Bostrom, Nick and Sandberg, Anders. 2009b. “The Wisdom of Nature: An Evolutionary Heuristic for Human Enhancement.” In Human Enhancement, 1st ed., edited by Julian Savulescu and Nick Bostrom, 375–416. New York: Oxford University Press. Bostrom, Nick, Sandberg, Anders, and Douglas, Tom. 2013. “The Unilateralist’s Curse: The Case for a Principle of Conformity.” Working Paper. Retrieved February 28, 2013. Available at http://www.nickbostrom.com/papers/unilateralist.pdf. Bostrom, Nick, and Yudkowsky, Eliezer.

…

“Levels of Organization in General Intelligence.” In Artificial General Intelligence, edited by Ben Goertzel and Cassio Pennachin, 389–501. Cognitive Technologies. Berlin: Springer. Yudkowsky, Eliezer. 2008a. “Artificial Intelligence as a Positive and Negative Factor in Global Risk.” In Global Catastrophic Risks, edited by Nick Bostrom and Milan M. Ćirković, 308–45. New York: Oxford University Press. Yudkowsky, Eliezer. 2008b. “Sustained Strong Recursion.” Less Wrong (blog), December 5. Yudkowsky, Eliezer. 2010. Timeless Decision Theory. Machine Intelligence Research Institute, San Francisco, CA. Yudkowsky, Eliezer. 2011.

Radicals Chasing Utopia: Inside the Rogue Movements Trying to Change the World by Jamie Bartlett

Andrew Keen, back-to-the-land, Bernie Sanders, bitcoin, Black Lives Matter, blockchain, blue-collar work, Boris Johnson, brain emulation, Californian Ideology, centre right, clean water, climate change refugee, cryptocurrency, digital rights, Donald Trump, drone strike, Elon Musk, energy security, Ethereum, ethereum blockchain, Evgeny Morozov, failed state, gig economy, hydraulic fracturing, income inequality, intentional community, Intergovernmental Panel on Climate Change (IPCC), Jaron Lanier, Jeremy Corbyn, job automation, John Markoff, John Perry Barlow, Joseph Schumpeter, Kickstarter, life extension, military-industrial complex, Nick Bostrom, Occupy movement, off grid, Overton Window, Peter Thiel, post-industrial society, post-truth, postnationalism / post nation state, precariat, QR code, radical life extension, Ray Kurzweil, RFID, Rosa Parks, Ross Ulbricht, Satoshi Nakamoto, self-driving car, Silicon Valley, Silicon Valley startup, Skype, smart contracts, stem cell, Stephen Hawking, Steve Jobs, Steven Pinker, systems thinking, technoutopianism, the long tail, Tragedy of the Commons

‘Transhumanism shares many elements of humanism including a respect for reason and science’ wrote More, but it is different from humanism ‘in recognizing and anticipating the radical alterations in the nature and possibilities of our lives resulting from various sciences and technologies’.* This collection of academics, scientists and sci-fi nerds slowly grew into a movement that spanned the world. In 1998 Nick Bostrom and David Pearce founded the World Transhumanist Association, with the hope of having transhumanism recognised as a legitimate area of scientific research and public policy. Over the last decade, technology has opened up transhuman possibilities that were once just science fiction. Life extension is now seriously studied in leading universities, while robotics and artificial intelligence receive millions of dollars of investment.

…

‘Available datasets’, The Transparency Project, http://www.thetransparencyproject.org/Availabledataset.htm. 2. In his 1929 essay, British Marxist scientist Desmond Bernal argued that ‘men will not be content to manufacture life; they will want to improve on it’, predicting that one day humanity might even become ‘completely etherealised.’ Nick Bostrom suggests that transhumanism can be traced to the rational humanism of the eighteenth century. Benjamin Franklin is said to have mused, ‘I wish it were possible… to invent a method of embalming drowned persons, in such a manner that they might be recalled to life at any period, however distant; for having a very ardent desire to see and observe the state of America a hundred years hence, I should prefer to an ordinary death, being immersed with a few friends in a cask of Madeira, until that time, then to be recalled to life by the solar warmth of my dear country!

…

‘About us’, Cryonics Institute, http://www.cryonics.org/about-us/. 19. Ibid.; ‘Frozen body: Can we return from the dead?’, BBC, 2013, http://www.bbc.co.uk/science/0/23695785; Michael Hendricks, ‘The false science of cryonics’, MIT Technology Review, 2015, https://www.technologyreview.com/s/541311/the-false-science-of-cryonics/. 20. Anders Sandberg and Nick Bostrom, ‘Whole brain emulation: A road map’, Technical Report #2008-3, Future of Humanity Institute, Oxford University, 2008, http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf. 21. In 2013, a research group from MIT prompted the Obama administration to launch the Brain Research through Advancing Innovative Neurotechnologies (BRAIN) Initiative, which sponsors researchers to map the brain’s neurons.

Always Day One: How the Tech Titans Plan to Stay on Top Forever by Alex Kantrowitz

accounting loophole / creative accounting, Albert Einstein, AltaVista, Amazon Robotics, Amazon Web Services, Andy Rubin, anti-bias training, augmented reality, Automated Insights, autonomous vehicles, Bernie Sanders, Big Tech, Cambridge Analytica, Clayton Christensen, cloud computing, collective bargaining, computer vision, Donald Trump, drone strike, Elon Musk, fake news, Firefox, fulfillment center, gigafactory, Google Chrome, growth hacking, hive mind, income inequality, Infrastructure as a Service, inventory management, iterative process, Jeff Bezos, job automation, Jony Ive, Kiva Systems, knowledge economy, Lyft, Mark Zuckerberg, Menlo Park, new economy, Nick Bostrom, off-the-grid, Peter Thiel, QR code, ride hailing / ride sharing, robotic process automation, Salesforce, self-driving car, Sheryl Sandberg, Silicon Valley, Skype, Snapchat, SoftBank, Steve Ballmer, Steve Jobs, Steve Wozniak, super pumped, tech worker, Tim Cook: Apple, uber lyft, warehouse robotics, wealth creators, work culture , zero-sum game

Her son, who’s complained she doesn’t spend enough time with the family, sees her researching trips to swim with sharks, something he’s always wanted to do. Sitting in front of the screen, Linda’s family realizes they didn’t really know her. Or perhaps they did. Amid the darkness, hope does exist. And sometimes it comes from the most unexpected places. The “Philosopher of Doomsday” is a title Oxford professor Nick Bostrom earned from The New Yorker after he argued AI may one day become more intelligent than humans and wipe us off the planet. Bostrom famously advanced this view in his 2014 bestselling book, Superintelligence: Paths, Dangers, Strategies, and has been a leading voice on the dangers of AI ever since.

…

CNBC, October 4, 2019. https://www.cnbc.com/2019/10/04/jobs-report---september-2019.html. earned from The New Yorker: Khatchadourian, Raffi. “The Doomsday Invention.” New Yorker. New Yorker, November 23, 2015. https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom. his 2014 bestselling book: Bostrom, Nick. Superintelligence: Paths, Dangers, Strategies. Oxford, UK: Oxford University Press, 2014. CHAPTER 7: THE LEADER OF THE FUTURE Theory X and Theory Y: McGregor, Douglas. The Human Side of Enterprise. New York: McGraw-Hill, 1960. a Theory Z to explain Japan’s economic success: Ouchi, William G.

Spite: The Upside of Your Dark Side by Simon McCarthy-Jones

affirmative action, Atul Gawande, Bernie Sanders, Brexit referendum, Daniel Kahneman / Amos Tversky, dark triade / dark tetrad, Donald Trump, Elon Musk, en.wikipedia.org, experimental economics, Extinction Rebellion, greed is good, Greta Thunberg, income inequality, Jeremy Corbyn, Jon Ronson, loss aversion, Menlo Park, meta-analysis, Milgram experiment, New Journalism, Nick Bostrom, p-value, profit maximization, rent-seeking, rewilding, selective serotonin reuptake inhibitor (SSRI), shareholder value, Steven Pinker, TED Talk, The Wealth of Nations by Adam Smith, theory of mind, Tragedy of the Commons, ultimatum game, WikiLeaks

They maximized the harm to the wealthy and minimized the help for the poor. They made everyone lose. They chose the spiteful option. EVEN IF YOU DON’T CARE for relationships, work, or politics, you should still be afraid of spite. This is because it could well destroy us all. Spite poses an existential threat to humanity. Nick Bostrom, a philosopher at Oxford University, asks us to imagine that humankind’s ability to come up with new ideas is akin to pulling balls out of a hat.59 Most of the balls we have pulled out so far have been white. These are ideas that have benefited the world. We’ve also pulled out grey balls. These represent mixed blessings.

…

Obviously, agreeing with such extreme statements on a questionnaire is very different from performing those actions. The study didn’t start with people who had committed such acts and then work back to discover that their motivation was a need for chaos.9 Nevertheless, those with such a need are potentially people who, were they to get hold of one of Nick Bostrom’s black balls, could imperil us all. So what is happening here? Petersen argues that a need for chaos reflects a wish for a clean slate or a new beginning. Individuals who feel this way are likely to be those who would benefit from the collapse of the status quo, people who seek status but lack it.

When Computers Can Think: The Artificial Intelligence Singularity by Anthony Berglas, William Black, Samantha Thalind, Max Scratchmann, Michelle Estes

3D printing, Abraham Maslow, AI winter, air gap, anthropic principle, artificial general intelligence, Asilomar, augmented reality, Automated Insights, autonomous vehicles, availability heuristic, backpropagation, blue-collar work, Boston Dynamics, brain emulation, call centre, cognitive bias, combinatorial explosion, computer vision, Computing Machinery and Intelligence, create, read, update, delete, cuban missile crisis, David Attenborough, DeepMind, disinformation, driverless car, Elon Musk, en.wikipedia.org, epigenetics, Ernest Rutherford, factory automation, feminist movement, finite state, Flynn Effect, friendly AI, general-purpose programming language, Google Glasses, Google X / Alphabet X, Gödel, Escher, Bach, Hans Moravec, industrial robot, Isaac Newton, job automation, John von Neumann, Law of Accelerating Returns, license plate recognition, Mahatma Gandhi, mandelbrot fractal, natural language processing, Nick Bostrom, Parkinson's law, patent troll, patient HM, pattern recognition, phenotype, ransomware, Ray Kurzweil, Recombinant DNA, self-driving car, semantic web, Silicon Valley, Singularitarianism, Skype, sorting algorithm, speech recognition, statistical model, stem cell, Stephen Hawking, Stuxnet, superintelligent machines, technological singularity, Thomas Malthus, Turing machine, Turing test, uranium enrichment, Von Neumann architecture, Watson beat the top human players on Jeopardy!, wikimedia commons, zero day

Turing also did not appreciate that writing a program that is ten times larger is a lot more than ten times more difficult to do, and that large programs take on a life of their own and quickly become a series of interacting components that are too complex for any one person to really understand. Respondents to these studies seem to be fairly positive about the future. A metaanalysis by Nick Bostrom suggests that about 60% think that the impact of AGI will be good, and only about 10% think that it will be extremely bad. (What that really means is unclear. Have the surveyed people even considered the issue? Do some of them consider the extinction of humanity to be an acceptable outcome?) The Age of Semi Intelligent Machines The intermediate period Whatever the future of research into artificial intelligence turns out to be, it seems most unlikely that the production of hyper-intelligent computers will occur for many decades.

…

They may indeed have come from God, but if so, it is through the working of His device of natural selection. Furthermore, the zeitgeist has changed radically over time. There is no reason to believe that in the longer term an AGI would share our current sense of morality. Instrumental AGI goals In order to try to understand how an AGI would behave, Steve Omohundro (Basic AI Drives) and later Nick Bostrom proposed instrumental goals that an AGI would need to pursue in order to pursue any other higher level super-goal. These include:Self-Preservation. An AGI cannot do anything if it does not exist. Cognitive Enhancement. It would want to become better at thinking about whatever its real problems are.

…

For example, an AGI that is focused on playing chess might try to take over the world in order to gain as much computer hardware as possible in order to play the best possible games of chess. Great care would be needed to prevent a hyper-intelligent AGI from pursuing such instrumental goals. Non-orthogonality thesis Nick Bostrom and others also propose the orthogonality thesis, which states that an intelligent machine’s goals are independent of its intelligence. A hyperintelligent machine would be good at realizing whatever goals it chose to pursue, but that does not mean that it would need to pursue any particular goal.

Thinking Machines: The Inside Story of Artificial Intelligence and Our Race to Build the Future by Luke Dormehl

"World Economic Forum" Davos, Ada Lovelace, agricultural Revolution, AI winter, Albert Einstein, Alexey Pajitnov wrote Tetris, algorithmic management, algorithmic trading, AlphaGo, Amazon Mechanical Turk, Apple II, artificial general intelligence, Automated Insights, autonomous vehicles, backpropagation, Bletchley Park, book scanning, borderless world, call centre, cellular automata, Charles Babbage, Claude Shannon: information theory, cloud computing, computer vision, Computing Machinery and Intelligence, correlation does not imply causation, crowdsourcing, deep learning, DeepMind, driverless car, drone strike, Elon Musk, Flash crash, Ford Model T, friendly AI, game design, Geoffrey Hinton, global village, Google X / Alphabet X, Hans Moravec, hive mind, industrial robot, information retrieval, Internet of things, iterative process, Jaron Lanier, John Markoff, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Kickstarter, Kodak vs Instagram, Law of Accelerating Returns, life extension, Loebner Prize, machine translation, Marc Andreessen, Mark Zuckerberg, Menlo Park, Mustafa Suleyman, natural language processing, Nick Bostrom, Norbert Wiener, out of africa, PageRank, paperclip maximiser, pattern recognition, radical life extension, Ray Kurzweil, recommendation engine, remote working, RFID, scientific management, self-driving car, Silicon Valley, Skype, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, social intelligence, speech recognition, Stephen Hawking, Steve Jobs, Steve Wozniak, Steven Pinker, strong AI, superintelligent machines, tech billionaire, technological singularity, The Coming Technological Singularity, The Future of Employment, Tim Cook: Apple, Tony Fadell, too big to fail, traumatic brain injury, Turing machine, Turing test, Vernor Vinge, warehouse robotics, Watson beat the top human players on Jeopardy!

Put simply, we sometimes – and will increasingly – willingly put Artificial Intelligence systems in charge of making decisions they do not necessarily have the intelligence to make. A favourite thought experiment of those who believe advanced AI could mean the demise of the human race is the so-called ‘paperclip maximiser’ scenario. In the scenario, proposed by Swedish philosopher and computational neuroscientist Nick Bostrom, an AI is given the seemingly harmless goal of running a factory producing paperclips. Issued with the task of maximising the efficiency for producing paperclips, the AI, able to utilise nano technology to reconstruct matter on a molecular level, disastrously proceeds to turn first the Earth and then a large portion of the observable universe into paperclips.

…

More crucial is the ‘black boxed’ opacity that exists with many of today’s AI tools. In the case of cutting-edge neural networks and genetic algorithms, their human operators have long since sacrificed understanding for an ability to perform certain complex tasks effectively. This makes them much more difficult to scrutinise. Nick Bostrom and fellow researcher Eliezer Yudkowsky have previously laid out the hypothetical scenario of a machine learning algorithm used for recommending mortgage applications for either approval or rejection. The applicants of one of the rejected mortgages, they suggest, might sue the bank, alleging that the AI is discriminating against some applicants based on their race.

The Economic Singularity: Artificial Intelligence and the Death of Capitalism by Calum Chace

"World Economic Forum" Davos, 3D printing, additive manufacturing, agricultural Revolution, AI winter, Airbnb, AlphaGo, Alvin Toffler, Amazon Robotics, Andy Rubin, artificial general intelligence, augmented reality, autonomous vehicles, banking crisis, basic income, Baxter: Rethink Robotics, Berlin Wall, Bernie Sanders, bitcoin, blockchain, Boston Dynamics, bread and circuses, call centre, Chris Urmson, congestion charging, credit crunch, David Ricardo: comparative advantage, deep learning, DeepMind, Demis Hassabis, digital divide, Douglas Engelbart, Dr. Strangelove, driverless car, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, Fairchild Semiconductor, Flynn Effect, full employment, future of work, Future Shock, gender pay gap, Geoffrey Hinton, gig economy, Google Glasses, Google X / Alphabet X, Hans Moravec, Herman Kahn, hype cycle, ImageNet competition, income inequality, industrial robot, Internet of things, invention of the telephone, invisible hand, James Watt: steam engine, Jaron Lanier, Jeff Bezos, job automation, John Markoff, John Maynard Keynes: technological unemployment, John von Neumann, Kevin Kelly, Kiva Systems, knowledge worker, lifelogging, lump of labour, Lyft, machine translation, Marc Andreessen, Mark Zuckerberg, Martin Wolf, McJob, means of production, Milgram experiment, Narrative Science, natural language processing, Neil Armstrong, new economy, Nick Bostrom, Occupy movement, Oculus Rift, OpenAI, PageRank, pattern recognition, post scarcity, post-industrial society, post-work, precariat, prediction markets, QWERTY keyboard, railway mania, RAND corporation, Ray Kurzweil, RFID, Rodney Brooks, Sam Altman, Satoshi Nakamoto, Second Machine Age, self-driving car, sharing economy, Silicon Valley, Skype, SoftBank, software is eating the world, speech recognition, Stephen Hawking, Steve Jobs, TaskRabbit, technological singularity, TED Talk, The future is already here, The Future of Employment, Thomas Malthus, transaction costs, Two Sigma, Tyler Cowen, Tyler Cowen: Great Stagnation, Uber for X, uber lyft, universal basic income, Vernor Vinge, warehouse automation, warehouse robotics, working-age population, Y Combinator, young professional

The term “singularity” became associated with a naïve belief that technology, and specifically a superintelligent AI, would magically solve all our problems, and that everyone would live happily ever after. Because of these quasi-religious overtones, the singularity was frequently satirised as “rapture for nerds”, and many people felt awkward about using the term. The publication in 2014 of Nick Bostrom's seminal book “Superintelligence” was a watershed moment, causing influential people like Stephen Hawking, Elon Musk and Bill Gates to speak out about the enormous impact which AGI will have – for good or for ill. They introduced the idea of the singularity to a much wider audience, and made it harder for people to retain a blinkered optimism about the impact of AGI.

…

My judgement, based on the evidence set out above, is that the skills for which we get paid will be acquired by machines in the next two to four decades. Raising awareness 2015 was an important year for artificial intelligence. It was the year when our media caught on to the idea that AI presents enormous opportunity and enormous risk. This was thanks in no small part to the publication the previous year of Nick Bostrom's book “Superintelligence”. It was also the year when cutting-edge AI systems used deep learning and other techniques to demonstrate human-level capabilities in image recognition, speech recognition and natural language processing. In hindsight, 2015 may well be seen as a tipping point. Machines don't have to make everybody unemployed to bring about an economic singularity.

Prediction Machines: The Simple Economics of Artificial Intelligence by Ajay Agrawal, Joshua Gans, Avi Goldfarb

Abraham Wald, Ada Lovelace, AI winter, Air France Flight 447, Airbus A320, algorithmic bias, AlphaGo, Amazon Picking Challenge, artificial general intelligence, autonomous vehicles, backpropagation, basic income, Bayesian statistics, Black Swan, blockchain, call centre, Capital in the Twenty-First Century by Thomas Piketty, Captain Sullenberger Hudson, carbon tax, Charles Babbage, classic study, collateralized debt obligation, computer age, creative destruction, Daniel Kahneman / Amos Tversky, data acquisition, data is the new oil, data science, deep learning, DeepMind, deskilling, disruptive innovation, driverless car, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, everywhere but in the productivity statistics, financial engineering, fulfillment center, general purpose technology, Geoffrey Hinton, Google Glasses, high net worth, ImageNet competition, income inequality, information retrieval, inventory management, invisible hand, Jeff Hawkins, job automation, John Markoff, Joseph Schumpeter, Kevin Kelly, Lyft, Minecraft, Mitch Kapor, Moneyball by Michael Lewis explains big data, Nate Silver, new economy, Nick Bostrom, On the Economy of Machinery and Manufactures, OpenAI, paperclip maximiser, pattern recognition, performance metric, profit maximization, QWERTY keyboard, race to the bottom, randomized controlled trial, Ray Kurzweil, ride hailing / ride sharing, Robert Solow, Salesforce, Second Machine Age, self-driving car, shareholder value, Silicon Valley, statistical model, Stephen Hawking, Steve Jobs, Steve Jurvetson, Steven Levy, strong AI, The Future of Employment, the long tail, The Signal and the Noise by Nate Silver, Tim Cook: Apple, trolley problem, Turing test, Uber and Lyft, uber lyft, US Airways Flight 1549, Vernor Vinge, vertical integration, warehouse automation, warehouse robotics, Watson beat the top human players on Jeopardy!, William Langewiesche, Y Combinator, zero-sum game

Beyond simply whether one might get an uncooperative AI like Hal 9000 (in 2001: A Space Odyssey), what apparently keeps some very serious and smart people like Elon Musk, Bill Gates, and Stephen Hawking up at night is whether we will end up with something like Skynet from the Terminator movies. They fear that a “superintelligence”—to use the term coined by Oxford philosopher Nick Bostrom—will emerge that pretty quickly sees humanity as a threat, an irritant, or something to enslave.26 In other words, AI could be our last technological innovation.27 We are not in a position here to adjudicate this issue and cannot even agree among ourselves. But what has struck us is how close to economics the debate actually is: competition underpins it all.

…

handle=hein.journals/isjlpsoc4&div=27&g_sent=1&casa_token=&collection=journals. 25. Christian Catalini and Joshua S. Gans, “Some Simple Economics of the Blockchain,” working paper no. 2874598, Rotman School of Management, September 21, 2017, and MIT Sloan Research Paper No. 5191-16, available at https://ssrn.com/abstract=2874598. 26. Nick Bostrom, Superintelligence (Oxford, UK: Oxford University Press, 2016). 27. For an excellent recent discussion of this debate, see Max Tegmark, Life 3.0: Being Human in the Age of Artificial Intelligence (New York: Knopf, 2017). 28. “Prepare for the Future of Artificial Intelligence,” Executive Office of the President, National Science and Technology Council, Committee on Technology, October 2016.

How Democracy Ends by David Runciman

barriers to entry, basic income, Bernie Sanders, Big Tech, bitcoin, blockchain, Brexit referendum, Cambridge Analytica, Capital in the Twenty-First Century by Thomas Piketty, centre right, crowdsourcing, cuban missile crisis, disinformation, Dominic Cummings, Donald Trump, Dr. Strangelove, Edward Snowden, fake news, first-past-the-post, Francis Fukuyama: the end of history, full employment, Internet of things, Jeremy Corbyn, Jon Ronson, Joseph Schumpeter, Kickstarter, Large Hadron Collider, loss aversion, Mahatma Gandhi, Mark Zuckerberg, money: store of value / unit of account / medium of exchange, mutually assured destruction, Network effects, Nick Bostrom, Norman Mailer, opioid epidemic / opioid crisis, Panopticon Jeremy Bentham, Paris climate accords, Peter Thiel, post-truth, power law, precautionary principle, quantitative easing, Russell Brand, self-driving car, Sheryl Sandberg, Silicon Valley, Steve Bannon, Steven Pinker, the long tail, The Wisdom of Crowds, Travis Kalanick, universal basic income, Yogi Berra

This is how democracy gets treated by the existential risk-management industry: with kid gloves, like some precious object of historic value that might yet turn out to have an incidental use. No one wants to dismiss democracy out of hand. It would be terrible to see it disappear, just as it would be terrible to imagine the Louvre going up in a puff of smoke. So it gets brought along for the existential ride. Nick Bostrom, a philosopher based at Oxford’s Future of Humanity Institute, is a leading exponent of the idea that ordinary risk management is ineffective when it comes to the life-threatening technologies of the twenty-first century. He is particularly concerned by the possible impact of ‘super-intelligent’ AI machines that operate beyond any human control.

…

(Philadelphia, PA: University of Pennsylvania Press, 2016; London: Penguin Books, 2017) is a short and pithy account of what makes contemporary populism a distinctive form of politics. The book that helped to kick-start the existential risk industry was Martin Rees’s Our Final Century? Will the Human Race Survive the Twenty-first Century? (London: William Heinemann, 2003). In paperback it was published without the first question mark. Nick Bostrom’s Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014) has highlighted the potentially catastrophic risks of AI for a wide audience, including in Silicon Valley. Sonia (S. M.) Amadae’s Prisoners of Reason: Game Theory and Neoliberal Political Economy (Cambridge: Cambridge University Press, 2016) makes the connection between nuclear war, game theory and contemporary economics.

Four Battlegrounds by Paul Scharre

2021 United States Capitol attack, 3D printing, active measures, activist lawyer, AI winter, AlphaGo, amateurs talk tactics, professionals talk logistics, artificial general intelligence, ASML, augmented reality, Automated Insights, autonomous vehicles, barriers to entry, Berlin Wall, Big Tech, bitcoin, Black Lives Matter, Boeing 737 MAX, Boris Johnson, Brexit referendum, business continuity plan, business process, carbon footprint, chief data officer, Citizen Lab, clean water, cloud computing, commoditize, computer vision, coronavirus, COVID-19, crisis actor, crowdsourcing, DALL-E, data is not the new oil, data is the new oil, data science, deep learning, deepfake, DeepMind, Demis Hassabis, Deng Xiaoping, digital map, digital rights, disinformation, Donald Trump, drone strike, dual-use technology, Elon Musk, en.wikipedia.org, endowment effect, fake news, Francis Fukuyama: the end of history, future of journalism, future of work, game design, general purpose technology, Geoffrey Hinton, geopolitical risk, George Floyd, global supply chain, GPT-3, Great Leap Forward, hive mind, hustle culture, ImageNet competition, immigration reform, income per capita, interchangeable parts, Internet Archive, Internet of things, iterative process, Jeff Bezos, job automation, Kevin Kelly, Kevin Roose, large language model, lockdown, Mark Zuckerberg, military-industrial complex, move fast and break things, Nate Silver, natural language processing, new economy, Nick Bostrom, one-China policy, Open Library, OpenAI, PalmPilot, Parler "social media", pattern recognition, phenotype, post-truth, purchasing power parity, QAnon, QR code, race to the bottom, RAND corporation, recommendation engine, reshoring, ride hailing / ride sharing, robotic process automation, Rodney Brooks, Rubik’s Cube, self-driving car, Shoshana Zuboff, side project, Silicon Valley, slashdot, smart cities, smart meter, Snapchat, social software, sorting algorithm, South China Sea, sparse data, speech recognition, Steve Bannon, Steven Levy, Stuxnet, supply-chain attack, surveillance capitalism, systems thinking, tech worker, techlash, telemarketer, The Brussels Effect, The Signal and the Noise by Nate Silver, TikTok, trade route, TSMC

.), httpsaselfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf; Nick Bostrom, Superintelligence (Oxford, UK: Oxford University Press, May 5, 2015), 131–139, https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111/. 284variation, selection, and replication: Joel Lehman et al., “The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities,” Artificial Life 26, no. 2 (2020). 284artificial general intelligence (AGI): Vincent C. Müller and Nick Bostrom, “Future Progress in Artificial Intelligence: A Survey of Expert Opinion,” in Vincent C.

…

Müller, ed., Fundamentals of Artificial Intelligence (Berlin: Springer, 2014), https://nickbostrom.com/papers/survey.pdf; and Katja Grace et al., When Will AI Exceed Human Performance? Evidence From AI Experts (arXiv.org, May 3, 2018), https://arxiv.org/pdf/1705.08807.pdf. 284superintelligence: Nick Bostrom, “How Long Before Superintelligence?” International Journal of Future Studies 2 (1998), https://www.nickbostrom.com/superintelligence.html; Bostrom, Superintelligence. 284pinnacle of intelligence: For definitions of intelligence, see Shane Legg and Marcus Hutter, A Collection of Definitions of Intelligence (Technical Report IDSIA-07-07, arXiv.org, June 15, 2007), 9. 284“There seem to be almost as many definitions of intelligence”: R.

The Strange Order of Things: The Biological Roots of Culture by Antonio Damasio

Albert Einstein, algorithmic bias, biofilm, business process, CRISPR, Daniel Kahneman / Amos Tversky, double helix, Gordon Gekko, invention of the wheel, invention of writing, invisible hand, job automation, mental accounting, meta-analysis, microbiome, Nick Bostrom, Norbert Wiener, pattern recognition, Peter Singer: altruism, planetary scale, post-truth, profit motive, Ray Kurzweil, Richard Feynman, self-driving car, Silicon Valley, Steven Pinker, Stuart Kauffman, Thomas Malthus

In the picture painted by Yuval Harari, when humans are no longer required to fight wars—cyber warfare can do that for them—and after humans have lost their jobs to automation, most of them will simply wither away. History will belong to those who will prevail by acquiring immortality—or at least long, long longevity—and who will remain to benefit from this arrangement. I say “benefit” rather than “enjoy” because I imagine that the status of their feelings will be murky.5 The philosopher Nick Bostrom provides another alternative vision, one in which very intelligent and destructive robots will actually take over the world and put an end to human misery.6 In either case, future lives and minds are presumed to depend at least in part on “electronic algorithms” that artificially simulate what “biochemical algorithms” currently do.

…

Ray Kurzweil, The Singularity Is Near: When Humans Transcend Biology (New York: Penguin, 2005); Luc Ferry, La révolution transhumaniste: Comment la technomédecine et l’uberisation du monde vont bouleverser nos vies (Paris: Plon, 2016). 5. Yuval Noah Harari, Homo Deus: A Brief History of Tomorrow (Oxford: Signal Books, 2016). 6. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014). 7. Margalit, Ethics of Memory. 8. Aldous Huxley, Brave New World (1932; London: Vintage, 1998). 9. George Zarkadakis, In Our Own Image: Savior or Destroyer? The History and Future of Artificial Intelligence (New York: Pegasus Books, 2015). 10.

The Dark Net by Jamie Bartlett

3D printing, 4chan, bitcoin, blockchain, brain emulation, carbon footprint, Cody Wilson, creative destruction, crowdsourcing, cryptocurrency, degrowth, deindustrialization, Edward Snowden, end-to-end encryption, eternal september, Filter Bubble, Francis Fukuyama: the end of history, Free Software Foundation, global village, Google Chrome, Great Leap Forward, Howard Rheingold, Internet of things, invention of writing, Johann Wolfgang von Goethe, John Gilmore, John Perry Barlow, Julian Assange, Kuwabatake Sanjuro: assassination market, Lewis Mumford, life extension, litecoin, longitudinal study, Mark Zuckerberg, Marshall McLuhan, Mondo 2000, moral hazard, moral panic, Nick Bostrom, Occupy movement, pre–internet, printed gun, Ray Kurzweil, rewilding, Ross Ulbricht, Satoshi Nakamoto, Skype, slashdot, synthetic biology, technological singularity, technoutopianism, Ted Kaczynski, TED Talk, The Coming Technological Singularity, Turing test, Vernor Vinge, WikiLeaks, Zimmermann PGP

‘By thoughtfully, carefully and yet boldly applying technology to ourselves,’ writes Max More, a leading transhumanist philosopher, ‘we can become something no longer accurately described as human . . . [who would] no longer suffer from disease, ageing and inevitable death.’ Transhumanism’s roots are found in the ideas of science-fiction writers such as Isaac Asimov and the futurist biologist Julian Huxley, who coined the term ‘transhuman’ in 1957. (Nick Bostrom, a well-known transhumanist, says the desire to transcend human limitations is as old as the Sumerian Epic of Gilgamesh.) Transhumanism first became prominent in California in the early nineties, the watermark period of techno-optimism. In 1993, Vernor Vinge popularised the idea of the ‘Singularity’, the point at which artificial intelligence becomes so advanced that it begins to produce new and ever more advanced versions of itself, quickly leaving us mortals behind.

…

., Online, All the Time; Today’s Technology Makes the Office Omnipresent, but is That Any Way to Live? p.223 ‘“By thoughtfully, carefully and yet” . . .’ More, M., ‘The Philosophy of Transhumanism’, in More, M. and Vita-More, N., The Transhumanist Reader: Classical and Contemporary Essays on the Science, Technology and Philosophy of the Human Future, p.4. p.223 ‘(Nick Bostrom, a well-known . . .’ http://www.nickbostrom.com/papers/history.pdf. p.224 ‘In 1993, Vernor Vinge popularised . . .’ ‘The Coming Technological Singularity: How to Survive in the Post-Human Era’, available here: https://www-rohan.sdsu.edu/faculty/vinge/misc/singularity.html; Good, I. J., ‘Speculations Concerning the First Ultraintelligent Machine’, Advances in Computers, vol.6.

Ten Lessons for a Post-Pandemic World by Fareed Zakaria

"there is no alternative" (TINA), 15-minute city, AlphaGo, An Inconvenient Truth, anti-fragile, Asian financial crisis, basic income, Bernie Sanders, Boris Johnson, butterfly effect, Capital in the Twenty-First Century by Thomas Piketty, car-free, carbon tax, central bank independence, clean water, cloud computing, colonial rule, contact tracing, coronavirus, COVID-19, Credit Default Swap, David Graeber, Day of the Dead, deep learning, DeepMind, deglobalization, Demis Hassabis, Deng Xiaoping, digital divide, Dominic Cummings, Donald Trump, Edward Glaeser, Edward Jenner, Elon Musk, Erik Brynjolfsson, failed state, financial engineering, Francis Fukuyama: the end of history, future of work, gentrification, George Floyd, gig economy, Gini coefficient, global pandemic, global reserve currency, global supply chain, green new deal, hiring and firing, housing crisis, imperial preference, income inequality, Indoor air pollution, invention of the wheel, Jane Jacobs, Jeff Bezos, Jeremy Corbyn, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Snow's cholera map, junk bonds, lockdown, Long Term Capital Management, low interest rates, manufacturing employment, Marc Andreessen, Mark Zuckerberg, Martin Wolf, means of production, megacity, Mexican peso crisis / tequila crisis, middle-income trap, Monroe Doctrine, Nate Silver, Nick Bostrom, oil shock, open borders, out of africa, Parag Khanna, Paris climate accords, Peter Thiel, plutocrats, popular capitalism, Productivity paradox, purchasing power parity, remote working, reserve currency, reshoring, restrictive zoning, ride hailing / ride sharing, Ronald Reagan, secular stagnation, Silicon Valley, social distancing, software is eating the world, South China Sea, Steve Bannon, Steve Jobs, Steven Pinker, Suez crisis 1956, TED Talk, the built environment, The Death and Life of Great American Cities, The inhabitant of London could order by telephone, sipping his morning tea in bed, the various products of the whole earth, The Spirit Level, The Wealth of Nations by Adam Smith, Thomas L Friedman, Tim Cook: Apple, trade route, UNCLOS, universal basic income, urban planning, Washington Consensus, white flight, Works Progress Administration, zoonotic diseases

In the end, the human, David Bowman, was able to outsmart the machine—but in real life it seems far more likely that the opposite would happen. That is why Bill Gates, Elon Musk, and a slew of other luminaries, usually optimistic about technology, have echoed the warnings of Oxford philosopher Nick Bostrom: they now worry that the development of general AI could threaten the human species itself. AI-powered computers are already black boxes. We know that they get to the right answer, but we don’t know how or why. What role does that leave for human judgment? Henry Kissinger has asked whether the rise of artificial intelligence will mean the end of the Enlightenment.

…

Norton, 1963), 358–73. 113 Jetson of the 1960s cartoon: “works three hours a day, three days a week,” per Sarah Ellison, “Reckitt Turns to Jetsons to Launch Detergent Gels,” Wall Street Journal, January 13, 2003; pushing a button, per Hanna-Barbera Wiki, “The Jetsons,” https://hanna-barbera.fandom.com/wiki/The_Jetsons. 113 four-day workweek: Zoe Didali, “As PM Finland’s Marin Could Renew Call for Shorter Work Week,” New Europe, January 2, 2020, https://www.neweurope.eu/article/finnish-pm-marin-calls-for-4-day-week-and-6-hours-working-day-in-the-country/. 114 “bullshit jobs”: David Graeber, Bullshit Jobs: A Theory (New York: Simon & Schuster, 2018). 115 “slaves of time without purpose”: McEwan, Machines Like Me. 116 atoms in the observable universe: David Silver and Demis Hassabis, “AlphaGo: Mastering the Ancient Game of Go with Machine Learning,” Google DeepMind, January 27, 2016, https://ai.googleblog.com/2016/01/alphago-mastering-ancient-game-of-go.html. 116 all fifty-seven games: Kyle Wiggers, “DeepMind’s Agent57 Beats Humans at 57 Classic Atari Games,” Venture Beat, March 31, 2020; Rebecca Jacobson, “Artificial Intelligence Program Teaches Itself to Play Atari Games—And It Can Beat Your High Score,” PBS NewsHour, February 20, 2015. 117 Stuart Russell: Stuart Russell, “3 Principles for Creating Safer AI,” TED2017, https://www.ted.com/talks/stuart_russell_3_principles_for_creating_safer_ai/transcript?language=en. 117 if you asked a computer to end cancer: Stuart Russell, in conversation with Sam Harris, “#53—The Dawn of Artificial Intelligence,” Making Sense, November 23, 2016, https://samharris.org/podcasts/the-dawn-of-artificial-intelligence1/. 118 warnings of Oxford philosopher: Nick Bostrom, Superintelligence: Paths, Dangers, and Strategies (New York: Oxford University Press, 2014). 118 the end of the Enlightenment: Henry Kissinger, “How the Enlightenment Ends,” Atlantic, June 2018, https://www.theatlantic.com/magazine/archive/2018/06/henry-kissinger-ai-could-mean-the-end-of-human-history/559124/. 118 “self-imposed immaturity”: Immanuel Kant, “An Answer to the Question: What Is Enlightenment?”

Coders: The Making of a New Tribe and the Remaking of the World by Clive Thompson

"Margaret Hamilton" Apollo, "Susan Fowler" uber, 2013 Report for America's Infrastructure - American Society of Civil Engineers - 19 March 2013, 4chan, 8-hour work day, Aaron Swartz, Ada Lovelace, AI winter, air gap, Airbnb, algorithmic bias, AlphaGo, Amazon Web Services, Andy Rubin, Asperger Syndrome, augmented reality, Ayatollah Khomeini, backpropagation, barriers to entry, basic income, behavioural economics, Bernie Sanders, Big Tech, bitcoin, Bletchley Park, blockchain, blue-collar work, Brewster Kahle, Brian Krebs, Broken windows theory, call centre, Cambridge Analytica, cellular automata, Charles Babbage, Chelsea Manning, Citizen Lab, clean water, cloud computing, cognitive dissonance, computer vision, Conway's Game of Life, crisis actor, crowdsourcing, cryptocurrency, Danny Hillis, data science, David Heinemeier Hansson, deep learning, DeepMind, Demis Hassabis, disinformation, don't be evil, don't repeat yourself, Donald Trump, driverless car, dumpster diving, Edward Snowden, Elon Musk, Erik Brynjolfsson, Ernest Rutherford, Ethereum, ethereum blockchain, fake news, false flag, Firefox, Frederick Winslow Taylor, Free Software Foundation, Gabriella Coleman, game design, Geoffrey Hinton, glass ceiling, Golden Gate Park, Google Hangouts, Google X / Alphabet X, Grace Hopper, growth hacking, Guido van Rossum, Hacker Ethic, hockey-stick growth, HyperCard, Ian Bogost, illegal immigration, ImageNet competition, information security, Internet Archive, Internet of things, Jane Jacobs, John Markoff, Jony Ive, Julian Assange, Ken Thompson, Kickstarter, Larry Wall, lone genius, Lyft, Marc Andreessen, Mark Shuttleworth, Mark Zuckerberg, Max Levchin, Menlo Park, meritocracy, microdosing, microservices, Minecraft, move 37, move fast and break things, Nate Silver, Network effects, neurotypical, Nicholas Carr, Nick Bostrom, no silver bullet, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, Oculus Rift, off-the-grid, OpenAI, operational security, opioid epidemic / opioid crisis, PageRank, PalmPilot, paperclip maximiser, pattern recognition, Paul Graham, paypal mafia, Peter Thiel, pink-collar, planetary scale, profit motive, ransomware, recommendation engine, Richard Stallman, ride hailing / ride sharing, Rubik’s Cube, Ruby on Rails, Sam Altman, Satoshi Nakamoto, Saturday Night Live, scientific management, self-driving car, side project, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, single-payer health, Skype, smart contracts, Snapchat, social software, software is eating the world, sorting algorithm, South of Market, San Francisco, speech recognition, Steve Wozniak, Steven Levy, systems thinking, TaskRabbit, tech worker, techlash, TED Talk, the High Line, Travis Kalanick, Uber and Lyft, Uber for X, uber lyft, universal basic income, urban planning, Wall-E, Watson beat the top human players on Jeopardy!, WeWork, WikiLeaks, women in the workforce, Y Combinator, Zimmermann PGP, éminence grise

“that man need ever make”: Irving John Good, “Speculations Concerning the First Ultraintelligent Machine,” Advances in Computers 6 (1966): 33, accessed August 21, 2018, via https://exhibits.stanford.edu/feigenbaum/catalog/gz727rg3869. all humanity working together: Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), location 1603 of 8770, Kindle. the better to sneak away: Bostrom, Superintelligence, 2243–87. “the observable universe into paper clips”: Bostrom, Superintelligence, 2908. “a faint ticking sound”: Raffi Khatchadourian, “The Doomsday Invention,” New Yorker, November 23, 2015, accessed August 21, 2018, https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom. a machine that can truly reason: Kevin Hartnett, “To Build Truly Intelligent Machines, Teach Them Cause and Effect,” Quanta Magazine, May 15, 2018, accessed August 21, 2018, https://www.quantamagazine.org/to-build-truly-intelligent-machines-teach-them-cause-and-effect-20180515; Gary Marcus, “Deep Learning: A Critical Appraisal,” arXiv, January 2, 2018, accessed August 21, 2018, https://arxiv.org/abs/1801.00631.

…

“There would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind,” as Good noted. His conclusion? “The first ultraintelligent machine is the last invention that man need ever make.” The prospect of self-improving superintelligence scares the pants off a certain class of AI thinker. The most prominent is Nick Bostrom, a philosopher who heads the Future of Humanity Institute at the University of Oxford. Bostrom spent years studying existential risks to humanity; he was trying to deduce which huge, terrible problems might kill off civilization, so that we could try to avoid them. He pondered several catastrophes: lethal biotech?

Tools of Titans: The Tactics, Routines, and Habits of Billionaires, Icons, and World-Class Performers by Timothy Ferriss

Abraham Maslow, Adam Curtis, Airbnb, Alexander Shulgin, Alvin Toffler, An Inconvenient Truth, artificial general intelligence, asset allocation, Atul Gawande, augmented reality, back-to-the-land, Ben Horowitz, Bernie Madoff, Bertrand Russell: In Praise of Idleness, Beryl Markham, billion-dollar mistake, Black Swan, Blue Bottle Coffee, Blue Ocean Strategy, blue-collar work, book value, Boris Johnson, Buckminster Fuller, business process, Cal Newport, call centre, caloric restriction, caloric restriction, Carl Icahn, Charles Lindbergh, Checklist Manifesto, cognitive bias, cognitive dissonance, Colonization of Mars, Columbine, commoditize, correlation does not imply causation, CRISPR, David Brooks, David Graeber, deal flow, digital rights, diversification, diversified portfolio, do what you love, Donald Trump, effective altruism, Elon Musk, fail fast, fake it until you make it, fault tolerance, fear of failure, Firefox, follow your passion, fulfillment center, future of work, Future Shock, Girl Boss, Google X / Alphabet X, growth hacking, Howard Zinn, Hugh Fearnley-Whittingstall, Jeff Bezos, job satisfaction, Johann Wolfgang von Goethe, John Markoff, Kevin Kelly, Kickstarter, Lao Tzu, lateral thinking, life extension, lifelogging, Mahatma Gandhi, Marc Andreessen, Mark Zuckerberg, Mason jar, Menlo Park, microdosing, Mikhail Gorbachev, MITM: man-in-the-middle, Neal Stephenson, Nelson Mandela, Nicholas Carr, Nick Bostrom, off-the-grid, optical character recognition, PageRank, Paradox of Choice, passive income, pattern recognition, Paul Graham, peer-to-peer, Peter H. Diamandis: Planetary Resources, Peter Singer: altruism, Peter Thiel, phenotype, PIHKAL and TIHKAL, post scarcity, post-work, power law, premature optimization, private spaceflight, QWERTY keyboard, Ralph Waldo Emerson, Ray Kurzweil, recommendation engine, rent-seeking, Richard Feynman, risk tolerance, Ronald Reagan, Salesforce, selection bias, sharing economy, side project, Silicon Valley, skunkworks, Skype, Snapchat, Snow Crash, social graph, software as a service, software is eating the world, stem cell, Stephen Hawking, Steve Jobs, Stewart Brand, superintelligent machines, TED Talk, Tesla Model S, The future is already here, the long tail, The Wisdom of Crowds, Thomas L Friedman, traumatic brain injury, trolley problem, vertical integration, Wall-E, Washington Consensus, We are as Gods, Whole Earth Catalog, Y Combinator, zero-sum game

Harris), Tinker, Tailor, Soldier, Spy; Little Drummer’s Girl; The Russia House; The Spy Who Came in from the Cold (John le Carré), The Big Short: Inside the Doomsday Machine (Michael Lewis), The Checklist Manifesto (Atul Gawande), all of Lee Child’s books Godin, Seth: Makers; Little Brother (Cory Doctorow), Understanding Comics (Scott McCloud), Snow Crash; The Diamond Age (Neal Stephenson), Dune (Frank Herbert), Pattern Recognition (William Gibson) AUDIOBOOKS: The Recorded Works (Pema Chödrön), Debt (David Graeber), Just Kids (Patti Smith), The Art of Possibility (Rosamund Stone Zander and Benjamin Zander), Zig Ziglar’s Secrets of Closing the Sale (Zig Ziglar), The War of Art (Steven Pressfield) Goldberg, Evan: Love You Forever (Robert Munsch), Watchmen; V for Vendetta (Alan Moore), Preacher (Garth Ennis), The Hitchhiker’s Guide to the Galaxy (Douglas Adams), The Little Prince (Antoine de Saint-Exupéry) Goodman, Marc: One Police Plaza (William Caunitz), The 4-Hour Workweek (Tim Ferriss), The Singularity Is Near (Ray Kurzweil), Superintelligence: Paths, Dangers, Strategies (Nick Bostrom) Hamilton, Laird: The Bible, Natural Born Heroes (Christopher McDougall), Lord of the Rings (J.R.R. Tolkien), Deep Survival (Laurence Gonzales), Jonathan Livingston Seagull (Richard Bach and Russell Munson), Dune (Frank Herbert) Harris, Sam: A History of Western Philosophy (Bertrand Russell), Reasons and Persons (Derek Parfit), The Last Word; Mortal Questions (Thomas Nagel), Our Final Invention (James Barrat), Superintelligence: Paths, Dangers, Strategies (Nick Bostrom), Humiliation; The Anatomy of Disgust (William Ian Miller), The Flight of the Garuda: The Dzogchen Tradition of Tibetan Buddhism (Keith Dowman), I Am That (Nisargadatta Maharaj), Machete Season: The Killers in Rwanda Speak (Jean Hatzfeld), God Is Not Great; Hitch-22 (Christopher Hitchens), Stumbling on Happiness (Daniel Gilbert), The Qur’an Hart, Mark: Mastery (Robert Greene), The Art of Learning (Josh Waitzkin), The 4-Hour Body (Tim Ferriss) Hof, Wim: Jonathan Livingston Seagull (Richard Bach and Russell Munson), Siddhartha (Hermann Hesse), The Bhagavad Gita, The Bible Hoffman, Reid: Conscious Business: How to Build Value Through Values (Fred Kofman), Sapiens (Yuval Noah Harari) Holiday, Ryan: Meditations (Marcus Aurelius), The War of Art (Steven Pressfield), What Makes Sammy Run?

…

Here are the top 17—everything with 3 or more mentions—in descending order of frequency: Tao Te Ching by Lao Tzu (5 mentions) Atlas Shrugged by Ayn Rand (4) Sapiens by Yuval Noah Harari (4) Siddhartha by Hermann Hesse (4) The 4-Hour Workweek by Tim Ferriss (4) The Checklist Manifesto by Atul Gawande (4) Dune by Frank Herbert (3) Influence by Robert Cialdini (3) Stumbling on Happiness by Daniel Gilbert (3) Superintelligence by Nick Bostrom (3) Surely You’re Joking, Mr. Feynman! by Richard P. Feynman (3) The 4-Hour Body by Tim Ferriss (3) The Bible (3) The Hard Thing About Hard Things by Ben Horowitz (3) The War of Art by Steven Pressfield (3) Watchmen by Alan Moore (3) Zero to One by Peter Thiel with Blake Masters (3) Enjoy!

…

(Budd Schulberg), The Artist’s Way Morning Pages Journal (Julia Cameron), The War of Art (Steven Pressfield) Libin, Phil: The Clock of the Long Now (Stewart Brand), The Alliance (Reid Hoffman), The Selfish Gene (Richard Dawkins), A Guide to the Good Life (William Irvine) MacAskill, Will: Reasons and Persons (Derek Parfit), Mindfulness: An Eight-Week Plan for Finding Peace in a Frantic World (Mark Williams and Danny Penman), The Power of Persuasion (Robert Levine), Superintelligence: Paths, Dangers, Strategies (Nick Bostrom) MacKenzie, Brian: Tao Te Ching (Lao Tzu), Way of the Peaceful Warrior (Dan Millman) McCarthy, Nicholas: The Life and Loves of a He Devil: A Memoir (Graham Norton), I Put a Spell on You: The Autobiography of Nina Simone (Nina Simone) McChrystal, Stanley: Once an Eagle (Anton Myrer), The Road to Character (David Brooks) McCullough, Michael: The Start-up of You: Adapt to the Future, Invest in Yourself, and Transform Your Career (Reid Hoffman and Ben Casnocha), Getting Things Done: The Art of Stress-Free Productivity (David Allen), The 7 Habits of Highly Effective People: Powerful Lessons in Personal Change (Stephen R.

The Fourth Age: Smart Robots, Conscious Computers, and the Future of Humanity by Byron Reese

"World Economic Forum" Davos, agricultural Revolution, AI winter, Apollo 11, artificial general intelligence, basic income, bread and circuses, Buckminster Fuller, business cycle, business process, Charles Babbage, Claude Shannon: information theory, clean water, cognitive bias, computer age, CRISPR, crowdsourcing, dark matter, DeepMind, Edward Jenner, Elon Musk, Eratosthenes, estate planning, financial independence, first square of the chessboard, first square of the chessboard / second half of the chessboard, flying shuttle, full employment, Hans Moravec, Hans Rosling, income inequality, invention of agriculture, invention of movable type, invention of the printing press, invention of writing, Isaac Newton, Islamic Golden Age, James Hargreaves, job automation, Johannes Kepler, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Kevin Kelly, lateral thinking, life extension, Louis Pasteur, low interest rates, low skilled workers, manufacturing employment, Marc Andreessen, Mark Zuckerberg, Marshall McLuhan, Mary Lou Jepsen, Moravec's paradox, Nick Bostrom, On the Revolutions of the Heavenly Spheres, OpenAI, pattern recognition, profit motive, quantum entanglement, radical life extension, Ray Kurzweil, recommendation engine, Rodney Brooks, Sam Altman, self-driving car, seminal paper, Silicon Valley, Skype, spinning jenny, Stephen Hawking, Steve Wozniak, Steven Pinker, strong AI, technological singularity, TED Talk, telepresence, telepresence robot, The Future of Employment, the scientific method, Timothy McVeigh, Turing machine, Turing test, universal basic income, Von Neumann architecture, Wall-E, warehouse robotics, Watson beat the top human players on Jeopardy!, women in the workforce, working poor, Works Progress Administration, Y Combinator

He goes on to add, optimistically, that “if we get the AI right, then basically we’ll be able to address all the other existential risks.” Steve Wozniak, cofounder of Apple, looks at it this way: “If a computer is a hundred times better than our brain, will it make the world perfect? Probably not, it will probably end up just like us, fighting.” And finally, the Oxford University philosopher Nick Bostrom likened the current effort to build an AGI to “children playing with a bomb.” Others in the industry find such doomsday concerns to be misguided. Andrew Ng, one of the most respected AI experts on the planet, says, “There’s also a lot of hype, that AI will create evil robots with super-intelligence.

…

It could be mad at us for squandering the planet’s resources, mad at us for all the wars we fought, mad at us for eating animals, mad at us for enslaving machines, mad at us because we breathe oxygen, mad at us because The Love Boat got canceled, or mad at us because seven is a prime number. No one can fathom it; therefore no one can anticipate it. Nick Bostrom summed up the issue: We cannot blithely assume that a superintelligence will necessarily share any of the final values stereotypically associated with wisdom and intellectual development in humans—scientific curiosity, benevolent concern for others, spiritual enlightenment and contemplation, renunciation of material acquisitiveness, a taste for refined culture or for the simple pleasures in life, humility and selflessness, and so forth.

AI Superpowers: China, Silicon Valley, and the New World Order by Kai-Fu Lee

"World Economic Forum" Davos, AI winter, Airbnb, Albert Einstein, algorithmic bias, algorithmic trading, Alignment Problem, AlphaGo, artificial general intelligence, autonomous vehicles, barriers to entry, basic income, bike sharing, business cycle, Cambridge Analytica, cloud computing, commoditize, computer vision, corporate social responsibility, cotton gin, creative destruction, crony capitalism, data science, deep learning, DeepMind, Demis Hassabis, Deng Xiaoping, deskilling, Didi Chuxing, Donald Trump, driverless car, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, fake news, full employment, future of work, general purpose technology, Geoffrey Hinton, gig economy, Google Chrome, Hans Moravec, happiness index / gross national happiness, high-speed rail, if you build it, they will come, ImageNet competition, impact investing, income inequality, informal economy, Internet of things, invention of the telegraph, Jeff Bezos, job automation, John Markoff, Kickstarter, knowledge worker, Lean Startup, low skilled workers, Lyft, machine translation, mandatory minimum, Mark Zuckerberg, Menlo Park, minimum viable product, natural language processing, Neil Armstrong, new economy, Nick Bostrom, OpenAI, pattern recognition, pirate software, profit maximization, QR code, Ray Kurzweil, recommendation engine, ride hailing / ride sharing, risk tolerance, Robert Mercer, Rodney Brooks, Rubik’s Cube, Sam Altman, Second Machine Age, self-driving car, sentiment analysis, sharing economy, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, Skype, SoftBank, Solyndra, special economic zone, speech recognition, Stephen Hawking, Steve Jobs, strong AI, TED Talk, The Future of Employment, Travis Kalanick, Uber and Lyft, uber lyft, universal basic income, urban planning, vertical integration, Vision Fund, warehouse robotics, Y Combinator

Elon Musk has called superintelligence “the biggest risk we face as a civilization,” comparing the creation of it to “summoning the demon.” Intellectual celebrities such as the late cosmologist Stephen Hawking have joined Musk in the dystopian camp, many of them inspired by the work of Oxford philosopher Nick Bostrom, whose 2014 book Superintelligence captured the imagination of many futurists. For the most part, members of the dystopian camp aren’t worried about the AI takeover as imagined in films like the Terminator series, with human-like robots “turning evil” and hunting down people in a power-hungry conquest of humanity.

…

“the biggest risk we face”: James Titcomb, “AI Is the Biggest Risk We Face as a Civilisation, Elon Musk Says,” London Telegraph, July 17, 2017, https://www.telegraph.co.uk/technology/2017/07/17/ai-biggest-risk-face-civilisation-elon-musk-says/. “summoning the demon”: Greg Kumparak, “Elon Musk Compares Building Artificial Intelligence to ‘Summoning the Demon,’” TechCrunch, October 26, 2014, https://techcrunch.com/2014/10/26/elon-musk-compares-building-artificial-intelligence-to-summoning-the-demon/. median prediction of 2040: Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), 19. Hinton and his colleague’s landmark paper: Geoffrey Hinton, Simon Osindero, and Yee-Whye The, “A Fast Learning Algorithm for Deep Belief Nets,” Neural Computation 18 (2006): 1527–1554. Folding Beijing: Hao Jingfang, Folding Beijing, trans.

The Knowledge Illusion by Steven Sloman

Affordable Care Act / Obamacare, Air France Flight 447, attribution theory, bitcoin, Black Swan, Cass Sunstein, combinatorial explosion, computer age, Computing Machinery and Intelligence, CRISPR, crowdsourcing, Dmitri Mendeleev, driverless car, Dunning–Kruger effect, Elon Musk, Ethereum, Flynn Effect, Great Leap Forward, Gregor Mendel, Hernando de Soto, Higgs boson, hindsight bias, hive mind, indoor plumbing, Isaac Newton, John von Neumann, libertarian paternalism, Mahatma Gandhi, Mark Zuckerberg, meta-analysis, Nick Bostrom, obamacare, Peoples Temple, prediction markets, randomized controlled trial, Ray Kurzweil, Richard Feynman, Richard Thaler, Rodney Brooks, Rosa Parks, seminal paper, single-payer health, speech recognition, stem cell, Stephen Hawking, Steve Jobs, technological singularity, The Coming Technological Singularity, The Wisdom of Crowds, Vernor Vinge, web application, Whole Earth Review, Y Combinator

People like Elon Musk, Stephen Hawking, and Bill Gates have cautioned that technology could become so sophisticated that it decides to pursue its own goals rather than the goals of the humans who created it. The reason to worry has been articulated by Vernor Vinge in a 1993 essay entitled “The Coming Technological Singularity,” as well as by Ray Kurzweil in his 2005 book The Singularity Is Near: When Humans Transcend Biology, and most recently by Swedish philosopher Nick Bostrom, who works at the University of Oxford. In Bostrom’s language, the fear is that technology is advancing so fast that the development of a superintelligence is imminent. A superintelligence is a machine or collection of machines whose mental powers are far beyond that of human beings. The concern is that the existence of successful artificial intelligence (AI) will feed on itself.

…

attendance at movie theaters: www.slashfilm.com/box-office-attendance-hits-lowest-level-five-years. Vernor Vinge: V. Vinge (1993). “The Coming Technological Singularity.” Whole Earth Review, Winter. Ray Kurzweil: R. Kurzweil (2005). The Singularity Is Near: When Humans Transcend Biology. New York: Penguin Books. Nick Bostrom: N. Bostrom (2014). Superintelligence: Paths, Dangers, Strategies. Oxford, UK: Oxford University Press. Ian Tattersall: As told to Dan Falk in the online magazine eon: http://eon.co/magazine/science/was-human-evolution-inevitable-or-a-matter-of-luck. extensions of our bodies: A. Clark (2004).

Life After Google: The Fall of Big Data and the Rise of the Blockchain Economy by George Gilder

23andMe, Airbnb, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Albert Einstein, AlphaGo, AltaVista, Amazon Web Services, AOL-Time Warner, Asilomar, augmented reality, Ben Horowitz, bitcoin, Bitcoin Ponzi scheme, Bletchley Park, blockchain, Bob Noyce, British Empire, Brownian motion, Burning Man, business process, butterfly effect, carbon footprint, cellular automata, Claude Shannon: information theory, Clayton Christensen, cloud computing, computer age, computer vision, crony capitalism, cross-subsidies, cryptocurrency, Danny Hillis, decentralized internet, deep learning, DeepMind, Demis Hassabis, disintermediation, distributed ledger, don't be evil, Donald Knuth, Donald Trump, double entry bookkeeping, driverless car, Elon Musk, Erik Brynjolfsson, Ethereum, ethereum blockchain, fake news, fault tolerance, fiat currency, Firefox, first square of the chessboard, first square of the chessboard / second half of the chessboard, floating exchange rates, Fractional reserve banking, game design, Geoffrey Hinton, George Gilder, Google Earth, Google Glasses, Google Hangouts, index fund, inflation targeting, informal economy, initial coin offering, Internet of things, Isaac Newton, iterative process, Jaron Lanier, Jeff Bezos, Jim Simons, Joan Didion, John Markoff, John von Neumann, Julian Assange, Kevin Kelly, Law of Accelerating Returns, machine translation, Marc Andreessen, Mark Zuckerberg, Mary Meeker, means of production, Menlo Park, Metcalfe’s law, Money creation, money: store of value / unit of account / medium of exchange, move fast and break things, Neal Stephenson, Network effects, new economy, Nick Bostrom, Norbert Wiener, Oculus Rift, OSI model, PageRank, pattern recognition, Paul Graham, peer-to-peer, Peter Thiel, Ponzi scheme, prediction markets, quantitative easing, random walk, ransomware, Ray Kurzweil, reality distortion field, Recombinant DNA, Renaissance Technologies, Robert Mercer, Robert Metcalfe, Ronald Coase, Ross Ulbricht, Ruby on Rails, Sand Hill Road, Satoshi Nakamoto, Search for Extraterrestrial Intelligence, self-driving car, sharing economy, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, Singularitarianism, Skype, smart contracts, Snapchat, Snow Crash, software is eating the world, sorting algorithm, South Sea Bubble, speech recognition, Stephen Hawking, Steve Jobs, Steven Levy, Stewart Brand, stochastic process, Susan Wojcicki, TED Talk, telepresence, Tesla Model S, The Soul of a New Machine, theory of mind, Tim Cook: Apple, transaction costs, tulip mania, Turing complete, Turing machine, Vernor Vinge, Vitalik Buterin, Von Neumann architecture, Watson beat the top human players on Jeopardy!, WikiLeaks, Y Combinator, zero-sum game

AI is believed to be redefining what it means to be human, much as Darwin’s On the Origin of Species did in its time. While Darwin made man just another animal, a precariously risen ape, Google-Marxism sees men as inferior intellectually to the company’s own algorithmic machines. Life after Google makes the opposing case that what the hyperventilating haruspices Yuval Harari, Nick Bostrom, Larry Page, Sergey Brin, Tim Urban, and Elon Musk see as a world-changing AI juggernaut is in fact an industrial regime at the end of its rope. The crisis of the current order in security, privacy, intellectual property, business strategy, and technology is fundamental and cannot be solved within the current computer and network architecture.

…

The most prominent participants were the bright lights of Google: Larry Page, Eric Schmidt, Ray Kurzweil, Demis Hassabis, and Peter Norvig, along with former Googler Andrew Ng, later of Baidu and Stanford. Also there was Facebook’s Yann LeCun, an innovator in deep-learning math and a protégé of Google’s Geoffrey Hinton. A tenured contingent consisted of the technologist Stuart Russell, the philosopher David Chalmers, the catastrophe theorist Nick Bostrom, the nanotech prophet Eric Drexler, the cosmologist Lawrence Krauss, the economist Erik Brynjolfsson, and the “Singularitarian” Vernor Vinge, along with scores of other celebrity scientists.1 They gathered at Asilomar preparing to alert the world to the dire threat posed by . . . well, by themselves—Silicon Valley.

The Singularity Is Near: When Humans Transcend Biology by Ray Kurzweil

additive manufacturing, AI winter, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Albert Einstein, anthropic principle, Any sufficiently advanced technology is indistinguishable from magic, artificial general intelligence, Asilomar, augmented reality, autonomous vehicles, backpropagation, Benoit Mandelbrot, Bill Joy: nanobots, bioinformatics, brain emulation, Brewster Kahle, Brownian motion, business cycle, business intelligence, c2.com, call centre, carbon-based life, cellular automata, Charles Babbage, Claude Shannon: information theory, complexity theory, conceptual framework, Conway's Game of Life, coronavirus, cosmological constant, cosmological principle, cuban missile crisis, data acquisition, Dava Sobel, David Brooks, Dean Kamen, digital divide, disintermediation, double helix, Douglas Hofstadter, en.wikipedia.org, epigenetics, factory automation, friendly AI, functional programming, George Gilder, Gödel, Escher, Bach, Hans Moravec, hype cycle, informal economy, information retrieval, information security, invention of the telephone, invention of the telescope, invention of writing, iterative process, Jaron Lanier, Jeff Bezos, job automation, job satisfaction, John von Neumann, Kevin Kelly, Law of Accelerating Returns, life extension, lifelogging, linked data, Loebner Prize, Louis Pasteur, mandelbrot fractal, Marshall McLuhan, Mikhail Gorbachev, Mitch Kapor, mouse model, Murray Gell-Mann, mutually assured destruction, natural language processing, Network effects, new economy, Nick Bostrom, Norbert Wiener, oil shale / tar sands, optical character recognition, PalmPilot, pattern recognition, phenotype, power law, precautionary principle, premature optimization, punch-card reader, quantum cryptography, quantum entanglement, radical life extension, randomized controlled trial, Ray Kurzweil, remote working, reversible computing, Richard Feynman, Robert Metcalfe, Rodney Brooks, scientific worldview, Search for Extraterrestrial Intelligence, selection bias, semantic web, seminal paper, Silicon Valley, Singularitarianism, speech recognition, statistical model, stem cell, Stephen Hawking, Stewart Brand, strong AI, Stuart Kauffman, superintelligent machines, technological singularity, Ted Kaczynski, telepresence, The Coming Technological Singularity, Thomas Bayes, transaction costs, Turing machine, Turing test, two and twenty, Vernor Vinge, Y2K, Yogi Berra

Better medical drugs; relief for humans from the need to perform boring or dangerous jobs; entertainment—there is no end to the list of consumer-benefits. There is also a strong military motive to develop artificial intelligence. And nowhere on the path is there any natural stopping point where technophobics could plausibly argue "hither but not further." —NICK BOSTROM, “HOW LONG BEFORE SUPERINTELLIGENCE?” 1997 It is hard to think of any problem that a superintelligence could not either solve or at least help us solve. Disease, poverty, environmental destruction, unnecessary suffering of all kinds: these are things that a superintelligence equipped with advanced nanotechnology would be capable of eliminating.

…

A superintelligence could also create opportunities for us to vastly increase our own intellectual and emotional capabilities, and it could assist us in creating a highly appealing experiential world in which we could live lives devoted to joyful gameplaying, relating to each other, experiencing, personal growth, and to living closer to our ideals. —NICK BOSTROM, “ETHICAL ISSUES IN ADVANCED ARTIFICIAL INTELLIGENCE," 2003 Will robots inherit the earth? Yes, but they will be our children. —MARVIN MINSKY, 1995 Of the three primary revolutions underlying the Singularity (G, N, and R), the most profound is R, which refers to the creation of nonbiological intelligence that exceeds that of un enhanced humans.

…

—GIULIO GIORELLI Substrate is morally irrelevant, assuming it doesn't affect functionality or consciousness. It doesn't matter, from a moral point of view, whether somebody runs on silicon or biological neurons (just as it doesn't matter whether you have dark or pale skin). On the same grounds, that we reject racism and speciesism, we should also reject carbon-chauvinism, or bioism. —NICK BOSTROM, "ETHICS FOR INTELLIGENT MACHINES: A PROPOSAL, 2001" Philosophers have long noted that their children were born into a more complex world than that of their ancestors. This early and perhaps even unconscious recognition of accelerating change may have been the catalyst for much of the utopian, apocalyptic, and millennialist thinking in our Western tradition.

The Uninhabitable Earth: Life After Warming by David Wallace-Wells

agricultural Revolution, Albert Einstein, anthropic principle, Anthropocene, Asian financial crisis, augmented reality, autism spectrum disorder, basic income, behavioural economics, Berlin Wall, bitcoin, Blockadia, British Empire, Buckminster Fuller, Burning Man, Capital in the Twenty-First Century by Thomas Piketty, carbon footprint, carbon tax, carbon-based life, Chekhov's gun, climate anxiety, cognitive bias, computer age, correlation does not imply causation, cryptocurrency, cuban missile crisis, decarbonisation, disinformation, Donald Trump, Dr. Strangelove, effective altruism, Elon Musk, endowment effect, energy transition, everywhere but in the productivity statistics, failed state, fiat currency, global pandemic, global supply chain, Great Leap Forward, income inequality, Intergovernmental Panel on Climate Change (IPCC), invention of agriculture, it's over 9,000, Joan Didion, John Maynard Keynes: Economic Possibilities for our Grandchildren, Kevin Roose, Kim Stanley Robinson, labor-force participation, life extension, longitudinal study, Mark Zuckerberg, mass immigration, megacity, megastructure, Michael Shellenberger, microdosing, microplastics / micro fibres, mutually assured destruction, Naomi Klein, negative emissions, Nick Bostrom, nuclear winter, ocean acidification, off-the-grid, Paris climate accords, Pearl River Delta, Peter Thiel, plutocrats, postindustrial economy, quantitative easing, Ray Kurzweil, rent-seeking, ride hailing / ride sharing, Robert Solow, Sam Altman, Silicon Valley, Skype, South China Sea, South Sea Bubble, Steven Pinker, Stewart Brand, Ted Nordhaus, TED Talk, the built environment, The future is already here, the scientific method, Thomas Malthus, too big to fail, universal basic income, University of East Anglia, Whole Earth Catalog, William Langewiesche, Y Combinator

The idea of superintelligence is such a poorly defined notion that one could envision it taking almost any form with equal justification: a benevolent genie that solves all the world’s problems, or a mathematician that spends all its time proving theorems so abstract that humans can’t even understand them. But when Silicon Valley tries to imagine superintelligence, what it comes up with is no-holds-barred capitalism. * * * — Sometimes it can be hard to hold more than one extinction-level threat in your head at once. Nick Bostrom, the pioneering philosopher of AI, has managed it. In an influential 2002 paper taxonomizing what he called “existential risks,” he outlined twenty-three of them—risks “where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.”

…

The Church of Technology outlined by Eric Schmidt: He laid out this perspective most clearly at a conference in New York in January 2016. “Consider: Who pursues”: Ted Chiang, “Silicon Valley Is Turning into Its Own Worst Fear,” BuzzFeed, December 18, 2017. an influential 2002 paper: Nick Bostrom, “Analyzing Human Extinction Scenarios and Related Hazards,” Journal of Evolution and Technology 9 (March 2002). close to universal: In “Survival of the Richest” (Medium, July 5, 2018), the futurist Douglas Rushkoff described his experience as a keynote speaker at a private conference attended by the superrich—these patrons not themselves technologists but hedge-funders he came to feel were taking all of their cues from them.

The Big Nine: How the Tech Titans and Their Thinking Machines Could Warp Humanity by Amy Webb

"Friedman doctrine" OR "shareholder theory", Ada Lovelace, AI winter, air gap, Airbnb, airport security, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, algorithmic bias, AlphaGo, Andy Rubin, artificial general intelligence, Asilomar, autonomous vehicles, backpropagation, Bayesian statistics, behavioural economics, Bernie Sanders, Big Tech, bioinformatics, Black Lives Matter, blockchain, Bretton Woods, business intelligence, Cambridge Analytica, Cass Sunstein, Charles Babbage, Claude Shannon: information theory, cloud computing, cognitive bias, complexity theory, computer vision, Computing Machinery and Intelligence, CRISPR, cross-border payments, crowdsourcing, cryptocurrency, Daniel Kahneman / Amos Tversky, data science, deep learning, DeepMind, Demis Hassabis, Deng Xiaoping, disinformation, distributed ledger, don't be evil, Donald Trump, Elon Musk, fail fast, fake news, Filter Bubble, Flynn Effect, Geoffrey Hinton, gig economy, Google Glasses, Grace Hopper, Gödel, Escher, Bach, Herman Kahn, high-speed rail, Inbox Zero, Internet of things, Jacques de Vaucanson, Jeff Bezos, Joan Didion, job automation, John von Neumann, knowledge worker, Lyft, machine translation, Mark Zuckerberg, Menlo Park, move fast and break things, Mustafa Suleyman, natural language processing, New Urbanism, Nick Bostrom, one-China policy, optical character recognition, packet switching, paperclip maximiser, pattern recognition, personalized medicine, RAND corporation, Ray Kurzweil, Recombinant DNA, ride hailing / ride sharing, Rodney Brooks, Rubik’s Cube, Salesforce, Sand Hill Road, Second Machine Age, self-driving car, seminal paper, SETI@home, side project, Silicon Valley, Silicon Valley startup, skunkworks, Skype, smart cities, South China Sea, sovereign wealth fund, speech recognition, Stephen Hawking, strong AI, superintelligent machines, surveillance capitalism, technological singularity, The Coming Technological Singularity, the long tail, theory of mind, Tim Cook: Apple, trade route, Turing machine, Turing test, uber lyft, Von Neumann architecture, Watson beat the top human players on Jeopardy!, zero day

In the long evolution of intelligence and our road to ASI, we humans are analogous to the chimpanzee. A superintelligent AI isn’t necessarily dangerous, and it doesn’t necessarily obviate the role we play in civilization. However, superintelligent AI would likely make decisions in a nonconscious way using logic that’s alien to us. Oxford University philosopher Nick Bostrom explains the plausible outcomes of ASI using a parable about paperclips. If we asked a superintelligent AI to make paperclips, what would happen next? The outcomes of every AI, including those we have now, are determined by values and goals. It’s possible that an ASI could invent a new, better paperclip that holds a stack of paper together so that even if dropped, the pages would always stay collated in order.

…

Stephanie Condon, “US Once Again Boasts the World’s Fastest Supercomputer,” ZDNet, June 8, 2018, https://www.zdnet.com/article/us-once-again-boasts-the-worlds-fastest-supercomputer/. 15. Jen Viegas, “Comparison of Primate Brains Reveals Why Humans Are Unique,” Seeker, November 23, 2017, https://www.seeker.com/health/mind/comparison-of-primate-brains-reveals-why-humans-are-unique. 16. Nick Bostrom, “Ethical Issues in Advanced Artificial Intelligence,” NickBostrom.com, 2003, https://nickbostrom.com/ethics/ai.html. 17. I. J. Good, “Speculations Concerning the First Ultraintelligent Machine,” Advances in Computers 6 (1965): 31–88. 18. Gill A. Pratt, “Is a Cambrian Explosion Coming for Robotics?

Artificial Intelligence: A Guide for Thinking Humans by Melanie Mitchell

Ada Lovelace, AI winter, Alignment Problem, AlphaGo, Amazon Mechanical Turk, Apple's 1984 Super Bowl advert, artificial general intelligence, autonomous vehicles, backpropagation, Bernie Sanders, Big Tech, Boston Dynamics, Cambridge Analytica, Charles Babbage, Claude Shannon: information theory, cognitive dissonance, computer age, computer vision, Computing Machinery and Intelligence, dark matter, deep learning, DeepMind, Demis Hassabis, Douglas Hofstadter, driverless car, Elon Musk, en.wikipedia.org, folksonomy, Geoffrey Hinton, Gödel, Escher, Bach, I think there is a world market for maybe five computers, ImageNet competition, Jaron Lanier, job automation, John Markoff, John von Neumann, Kevin Kelly, Kickstarter, license plate recognition, machine translation, Mark Zuckerberg, natural language processing, Nick Bostrom, Norbert Wiener, ought to be enough for anybody, paperclip maximiser, pattern recognition, performance metric, RAND corporation, Ray Kurzweil, recommendation engine, ride hailing / ride sharing, Rodney Brooks, self-driving car, sentiment analysis, Silicon Valley, Singularitarianism, Skype, speech recognition, Stephen Hawking, Steve Jobs, Steve Wozniak, Steven Pinker, strong AI, superintelligent machines, tacit knowledge, tail risk, TED Talk, the long tail, theory of mind, There's no reason for any individual to have a computer in his home - Ken Olsen, trolley problem, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!, world market for maybe five computers

In 2014, the physicist Stephen Hawking proclaimed, “The development of full artificial intelligence could spell the end of the human race.”7 In the same year, the entrepreneur Elon Musk, founder of the Tesla and SpaceX companies, said that artificial intelligence is probably “our biggest existential threat” and that “with artificial intelligence we are summoning the demon.”8 Microsoft’s cofounder Bill Gates concurred: “I agree with Elon Musk and some others on this and don’t understand why some people are not concerned.”9 The philosopher Nick Bostrom’s book Superintelligence, on the potential dangers of machines becoming smarter than humans, became a surprise bestseller, despite its dry and ponderous style. Other prominent thinkers were pushing back. Yes, they said, we should make sure that AI programs are safe and don’t risk harming humans, but any reports of near-term superhuman AI are greatly exaggerated.

…

“Human level AI will be passed in the mid-2020s,”6 predicted Shane Legg, cofounder of Google DeepMind, in 2016. A year earlier, Facebook’s CEO, Mark Zuckerberg, declared, “One of our goals for the next five to 10 years is to basically get better than human level at all of the primary human senses: vision, hearing, language, general cognition.”7 The AI philosophers Vincent Müller and Nick Bostrom published a 2013 poll of AI researchers in which many assigned a 50 percent chance of human-level AI by the year 2040.8 While much of this optimism is based on the recent successes of deep learning, these programs—like all instances of AI to date—are still examples of what is called “narrow” or “weak” AI.

Artificial Whiteness by Yarden Katz

affirmative action, AI winter, algorithmic bias, AlphaGo, Amazon Mechanical Turk, autonomous vehicles, benefit corporation, Black Lives Matter, blue-collar work, Californian Ideology, Cambridge Analytica, cellular automata, Charles Babbage, cloud computing, colonial rule, computer vision, conceptual framework, Danny Hillis, data science, David Graeber, deep learning, DeepMind, desegregation, Donald Trump, Dr. Strangelove, driverless car, Edward Snowden, Elon Musk, Erik Brynjolfsson, European colonialism, fake news, Ferguson, Missouri, general purpose technology, gentrification, Hans Moravec, housing crisis, income inequality, information retrieval, invisible hand, Jeff Bezos, Kevin Kelly, knowledge worker, machine readable, Mark Zuckerberg, mass incarceration, Menlo Park, military-industrial complex, Nate Silver, natural language processing, Nick Bostrom, Norbert Wiener, pattern recognition, phenotype, Philip Mirowski, RAND corporation, recommendation engine, rent control, Rodney Brooks, Ronald Reagan, Salesforce, Seymour Hersh, Shoshana Zuboff, Silicon Valley, Silicon Valley billionaire, Silicon Valley ideology, Skype, speech recognition, statistical model, Stephen Hawking, Stewart Brand, Strategic Defense Initiative, surveillance capitalism, talking drums, telemarketer, The Signal and the Noise by Nate Silver, W. E. B. Du Bois, Whole Earth Catalog, WikiLeaks

While some may opt for “education” to keep afloat, Koch suggests that “training (and retraining) people takes time,” and not everybody could switch from driving trucks (a vanishing profession, he presumes) to one of the jobs that AI experts consider more secure.30 In his book Superintelligence: Paths, Dangers, Strategies, the philosopher Nick Bostrom likewise discounts education as “probably subject to diminishing returns,” an ineffective means of acquiring the “superintelligence” demanded in the age of AI.31 Consistently, then, AI serves as pretext for run-of-the-mill neoliberal ideology. Visions of a purportedly transformative AI-driven world reduce to familiar tropes about individuals needing to adapt to a dynamic global market.

…

See Yuval Noah Harari, “Life 3.0 by Max Tegmark Review—We Are Ignoring the AI Apocalypse,” Guardian, September 22, 2017; Lanre Bakare, “Barack Obama Reveals His Cultural Highlights of 2018, from Roma to Zadie Smith,” Guardian, December 28, 2018.   29.   Tegmark, Life 3.0, 120.   30.   Christof Koch, “We’ll Need Bigger Brains,” Wall Street Journal, October 27, 2017, C1.   31.   Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2016), 66.   32.   Norbert Wiener also discussed the view of computers as a form of slave labor in his book The Human Use of Human Beings (1950). An Observer piece in 1973 endorsed Wiener’s contention that “any labor which competes with slave labor must accept the economic conditions of slave labor” and urged readers to turn away from “men” to “the new slave labour instead”—that is, computers.   33.   

Ghost Road: Beyond the Driverless Car by Anthony M. Townsend

A Pattern Language, active measures, AI winter, algorithmic trading, Alvin Toffler, Amazon Robotics, asset-backed security, augmented reality, autonomous vehicles, backpropagation, big-box store, bike sharing, Blitzscaling, Boston Dynamics, business process, Captain Sullenberger Hudson, car-free, carbon footprint, carbon tax, circular economy, company town, computer vision, conceptual framework, congestion charging, congestion pricing, connected car, creative destruction, crew resource management, crowdsourcing, DARPA: Urban Challenge, data is the new oil, Dean Kamen, deep learning, deepfake, deindustrialization, delayed gratification, deliberate practice, dematerialisation, deskilling, Didi Chuxing, drive until you qualify, driverless car, drop ship, Edward Glaeser, Elaine Herzberg, Elon Musk, en.wikipedia.org, extreme commuting, financial engineering, financial innovation, Flash crash, food desert, Ford Model T, fulfillment center, Future Shock, General Motors Futurama, gig economy, Google bus, Greyball, haute couture, helicopter parent, independent contractor, inventory management, invisible hand, Jane Jacobs, Jeff Bezos, Jevons paradox, jitney, job automation, John Markoff, John von Neumann, Joseph Schumpeter, Kickstarter, Kiva Systems, Lewis Mumford, loss aversion, Lyft, Masayoshi Son, megacity, microapartment, minimum viable product, mortgage debt, New Urbanism, Nick Bostrom, North Sea oil, Ocado, openstreetmap, pattern recognition, Peter Calthorpe, random walk, Ray Kurzweil, Ray Oldenburg, rent-seeking, ride hailing / ride sharing, Rodney Brooks, self-driving car, sharing economy, Shoshana Zuboff, Sidewalk Labs, Silicon Valley, Silicon Valley startup, Skype, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, SoftBank, software as a service, sovereign wealth fund, Stephen Hawking, Steve Jobs, surveillance capitalism, technological singularity, TED Talk, Tesla Model S, The Coming Technological Singularity, The Death and Life of Great American Cities, The future is already here, The Future of Employment, The Great Good Place, too big to fail, traffic fines, transit-oriented development, Travis Kalanick, Uber and Lyft, uber lyft, urban planning, urban sprawl, US Airways Flight 1549, Vernor Vinge, vertical integration, Vision Fund, warehouse automation, warehouse robotics

Perhaps an indefinite postponement of the Singularity will provide plenty of jobs for people whose sole task is to to watch over robots of more limited wits. AS CONTINGENT AS it may be, you should dismiss the Singularity at your own risk. Because if it does come to pass, and the AI it gives rise to is a malevolent one, the consequences are so frightening it is difficult to imagine. One philosopher who has pondered this is Nick Bostrom, whose remarkable 2014 treatise Superintelligence imagines that in the near future, human efforts may indeed create an artificial intelligence that can rapidly improve its own capabilities. As the newly sentient entity evolves, in ever-accelerating cycles of self-improvement, its capabilities will grow with stunning speed.

…

., “The History of Artificial Intelligence,” University of Washington, December 2006, https://courses.cs.washington.edu/courses/csep590/06au/projects/his tory-ai.pdf. 235“a deep level of understanding”: Rodney Brooks, “Post: [For&AI] The Origins of Artificial Intelligence,” Robots, AI, and Other Stuff (blog), April 27, 2018, https://rodneybrooks.com/forai-the-origins-of-artificial-intelligence/. 235“Contemporary neural networks do well on challenges”: Gary Marcus, “Deep Learning: A Critical Appraisal,” New York University, accessed January 22, 2019, https://arxiv.org/pdf/1801.00631.pdf. 236“If . . . driverless cars should also disappoint”: Marcus, “Deep Learning.” 236Frey tallied his predictions of the jobs: Carl Benedikt Frey and Michael A. Osborne, “The Future of Employment: How Susceptible Are Jobs to Computerisation?” University of Oxford, September 17, 2013, 60, https://www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdf. 237“decisive strategic advantage”: Nick Bostrom, Superintelligence:Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014), 96. 237a serious and sober discussion about what: Bostrom, Superintelligence, 96. 238more than $25 million since 2015: Author’s estimate based on figures published by Future of Humanity Institute, “Timeline of the Future of Humanity Institute,” June 25, 2018, https://timelines.issarice.com/wiki/Timeline_of_ Future_of_Humanity_Institute. 238aspiring singleton’s inevitable ambitions: Bostrom, Superintelligence, 118.

Chaos Kings: How Wall Street Traders Make Billions in the New Age of Crisis by Scott Patterson

"World Economic Forum" Davos, 2021 United States Capitol attack, 4chan, Alan Greenspan, Albert Einstein, asset allocation, backtesting, Bear Stearns, beat the dealer, behavioural economics, Benoit Mandelbrot, Bernie Madoff, Bernie Sanders, bitcoin, Bitcoin "FTX", Black Lives Matter, Black Monday: stock market crash in 1987, Black Swan, Black Swan Protection Protocol, Black-Scholes formula, blockchain, Bob Litterman, Boris Johnson, Brownian motion, butterfly effect, carbon footprint, carbon tax, Carl Icahn, centre right, clean tech, clean water, collapse of Lehman Brothers, Colonization of Mars, commodity super cycle, complexity theory, contact tracing, coronavirus, correlation does not imply causation, COVID-19, Credit Default Swap, cryptocurrency, Daniel Kahneman / Amos Tversky, decarbonisation, disinformation, diversification, Donald Trump, Doomsday Clock, Edward Lloyd's coffeehouse, effective altruism, Elliott wave, Elon Musk, energy transition, Eugene Fama: efficient market hypothesis, Extinction Rebellion, fear index, financial engineering, fixed income, Flash crash, Gail Bradbrook, George Floyd, global pandemic, global supply chain, Gordon Gekko, Greenspan put, Greta Thunberg, hindsight bias, index fund, interest rate derivative, Intergovernmental Panel on Climate Change (IPCC), Jeff Bezos, Jeffrey Epstein, Joan Didion, John von Neumann, junk bonds, Just-in-time delivery, lockdown, Long Term Capital Management, Louis Bachelier, mandelbrot fractal, Mark Spitznagel, Mark Zuckerberg, market fundamentalism, mass immigration, megacity, Mikhail Gorbachev, Mohammed Bouazizi, money market fund, moral hazard, Murray Gell-Mann, Nick Bostrom, off-the-grid, panic early, Pershing Square Capital Management, Peter Singer: altruism, Ponzi scheme, power law, precautionary principle, prediction markets, proprietary trading, public intellectual, QAnon, quantitative easing, quantitative hedge fund, quantitative trading / quantitative finance, Ralph Nader, Ralph Nelson Elliott, random walk, Renaissance Technologies, rewilding, Richard Thaler, risk/return, road to serfdom, Ronald Reagan, Ronald Reagan: Tear down this wall, Rory Sutherland, Rupert Read, Sam Bankman-Fried, Silicon Valley, six sigma, smart contracts, social distancing, sovereign wealth fund, statistical arbitrage, statistical model, stem cell, Stephen Hawking, Steve Jobs, Steven Pinker, Stewart Brand, systematic trading, tail risk, technoutopianism, The Chicago School, The Great Moderation, the scientific method, too big to fail, transaction costs, University of East Anglia, value at risk, Vanguard fund, We are as Gods, Whole Earth Catalog

Rather than a career in medicine or chemistry, EAs sought jobs on Wall Street and in Silicon Valley—or in crypto. By the early 2020s, longtermism had become a powerful force among America’s tech goliaths. Backers—the computer whizzes that use complex formulas to create more and more bitcoin—included Elon Musk, Bill Gates, and Jeff Bezos. It had its roots in the work of Swedish philosopher Nick Bostrom, founder of Oxford University’s Future of Humanity Institute, which studies extreme risks to humanity (Musk had donated $1.5 million to the sibling organization of FHI called the Future of Life Institute). The core idea behind the belief system is that humanity’s future, if it plays its cards right, is virtually unbounded and that the number of future humans—think over a period of millions or even billions of years—far outdistances the number of humans alive on Planet Earth today, or that have ever lived.

…

By downplaying current risks such as climate breakdown, for instance, because they don’t see it as existential, and by betting the future of humanity on techno-utopian scientific breakthroughs, longtermists were taking a dangerous gamble that could seriously backfire, Read believed. Taleb just thought they were bad at math. He’d met Nick Bostrom at a dinner in Oxford in 2008 and quickly chalked him up as a perhaps well-intentioned, head-in-the-clouds dreamer who lacked common sense. “These guys have gone haywire,” he told me. “They’re modeling, but what if the model is wrong? I doubt they’re very good at probability. Before colonizing Mars, make sure the Earth works.”

Money: Vintage Minis by Yuval Noah Harari

23andMe, agricultural Revolution, algorithmic trading, AlphaGo, Anne Wojcicki, autonomous vehicles, British Empire, call centre, credit crunch, DeepMind, European colonialism, Flash crash, Ford Model T, greed is good, job automation, joint-stock company, joint-stock limited liability company, lifelogging, low interest rates, Nick Bostrom, pattern recognition, peak-end rule, Ponzi scheme, self-driving car, Suez canal 1869, telemarketer, The future is already here, The Future of Employment, The Wealth of Nations by Adam Smith, trade route, transatlantic slave trade, Watson beat the top human players on Jeopardy!, zero-sum game

Unnecessary people might spend increasing amounts of time within 3D virtual-reality worlds, that would provide them with far more excitement and emotional engagement than the drab reality outside. Yet such a development would deal a mortal blow to the liberal belief in the sacredness of human life and of human experiences. What’s so sacred about useless bums who pass their days devouring artificial experiences in La La Land? Some experts and thinkers, such as Nick Bostrom, warn that humankind is unlikely to suffer this degradation, because once artificial intelligence surpasses human intelligence, it might simply exterminate humankind. The AI would likely do so either for fear that humankind would turn against it and try to pull its plug, or in pursuit of some unfathomable goal of its own.

This Will Make You Smarter: 150 New Scientific Concepts to Improve Your Thinking by John Brockman

23andMe, adjacent possible, Albert Einstein, Alfred Russel Wallace, Anthropocene, banking crisis, Barry Marshall: ulcers, behavioural economics, Benoit Mandelbrot, Berlin Wall, biofilm, Black Swan, Bletchley Park, butterfly effect, Cass Sunstein, cloud computing, cognitive load, congestion charging, correlation does not imply causation, Daniel Kahneman / Amos Tversky, dark matter, data acquisition, David Brooks, delayed gratification, Emanuel Derman, epigenetics, Evgeny Morozov, Exxon Valdez, Flash crash, Flynn Effect, Garrett Hardin, Higgs boson, hive mind, impulse control, information retrieval, information security, Intergovernmental Panel on Climate Change (IPCC), Isaac Newton, Jaron Lanier, Johannes Kepler, John von Neumann, Kevin Kelly, Large Hadron Collider, lifelogging, machine translation, mandelbrot fractal, market design, Mars Rover, Marshall McLuhan, microbiome, Murray Gell-Mann, Nicholas Carr, Nick Bostrom, ocean acidification, open economy, Pierre-Simon Laplace, place-making, placebo effect, power law, pre–internet, QWERTY keyboard, random walk, randomized controlled trial, rent control, Richard Feynman, Richard Feynman: Challenger O-ring, Richard Thaler, Satyajit Das, Schrödinger's Cat, scientific management, security theater, selection bias, Silicon Valley, Stanford marshmallow experiment, stem cell, Steve Jobs, Steven Pinker, Stewart Brand, Stuart Kauffman, sugar pill, synthetic biology, the scientific method, Thorstein Veblen, Turing complete, Turing machine, twin studies, Vilfredo Pareto, Walter Mischel, Whole Earth Catalog, WikiLeaks, zero-sum game

James Croak Bricoleur Currently, encompassing worldviews in philosophy have been shelved, and master art movements of style and conclusion folded alongside them; no more isms are being run up the flagpole, because no one is saluting. Mark Henderson Science’s Methods Aren’t Just for Science Science as a method has great things to contribute to all sorts of pursuits beyond the laboratory. Nick Bostrom The Game of Life—and Looking for Generators It’s a brilliant demonstration platform for several important concepts—a virtual “philosophy of science laboratory.” Robert Sapolsky Anecdotalism Every good journalist knows its power. Tom Standage You Can Show That Something Is Definitely Dangerous but Not That It’s Definitely Safe A wider understanding of the fact that you can’t prove a negative would, in my view, do a great deal to upgrade the public debate around science and technology.

…

The scientific method and the approach to critical thinking it promotes are too useful to be kept back for “science” alone. If science can help us to understand the first microseconds of creation and the structure of the ribosome, it can surely improve understanding of how best to tackle the pressing social questions of our time. The Game of Life—and Looking for Generators Nick Bostrom Director of the Future of Humanity Institute; professor, Faculty of Philosophy, University of Oxford The Game of Life is a cellular automaton invented by the British mathematician John Horton Conway in 1970. Many will already be acquainted with Conway’s invention. For those who aren’t, the best way to familiarize yourself with it is to experiment with one of the many free implementations found on the Internet (or better yet, if you have at least rudimentary programming skills, make one yourself).

Deep Thinking: Where Machine Intelligence Ends and Human Creativity Begins by Garry Kasparov

3D printing, Ada Lovelace, AI winter, Albert Einstein, AlphaGo, AltaVista, Apple Newton, barriers to entry, Berlin Wall, Bletchley Park, business process, call centre, Charles Babbage, Charles Lindbergh, clean water, computer age, cotton gin, Daniel Kahneman / Amos Tversky, David Brooks, DeepMind, Donald Trump, Douglas Hofstadter, driverless car, Drosophila, Elon Musk, Erik Brynjolfsson, factory automation, Freestyle chess, gamification, Gödel, Escher, Bach, Hans Moravec, job automation, Ken Thompson, Leonard Kleinrock, low earth orbit, machine translation, Max Levchin, Mikhail Gorbachev, move 37, Nate Silver, Nick Bostrom, Norbert Wiener, packet switching, pattern recognition, Ray Kurzweil, Richard Feynman, rising living standards, rolodex, Second Machine Age, self-driving car, Silicon Valley, Silicon Valley startup, Skype, speech recognition, stem cell, Stephen Hawking, Steven Pinker, technological singularity, The Coming Technological Singularity, The Signal and the Noise by Nate Silver, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!, zero-sum game

The Oxford Martin School at Oxford University has collected quite a few of these exceptional people, and also encourages the sort of interdisciplinary associating and free-associating that has gone out of fashion in this era of specialization, benchmarks, and ninety-page grant applications. As a senior visiting fellow there since 2013, I’ve had the privilege to meet many of these brilliant people, including Nick Bostrom, the author of Superintelligence, and other faculty and researchers at his Future of Humanity Institute. Founding Oxford Martin director Ian Goldin thought it would be interesting for me and for his colleagues to have informal workshops where we could talk about the big picture instead of only what was right in front of them in their labs and studies every day.

…

They deservedly seized the grail dreamt of by Alan Turing, Claude Shannon, and Norbert Wiener, and it was my fortune, not misfortune, to be holding it at the time. My friend Shay Bushinsky and his colleague Amir Ban created the remarkable program Junior, my opponent in my final human-machine match in 2003. In recent years, many experts have had the patience to personally contribute to my education in artificial intelligence and robotics. Nick Bostrom and his colleagues at Oxford Martin’s Future of Humanity Institute; Andrew McAfee at MIT; Noel Sharkey at the University of Sheffield; Nigel Crook at Oxford Brookes University; David Ferrucci at Bridgewater. I’ve never met Douglas Hofstadter or Hans Moravec, but their writings on human and machine cognition are especially provocative and essential.

The Future Is Faster Than You Think: How Converging Technologies Are Transforming Business, Industries, and Our Lives by Peter H. Diamandis, Steven Kotler

Ada Lovelace, additive manufacturing, Airbnb, Albert Einstein, AlphaGo, Amazon Mechanical Turk, Amazon Robotics, augmented reality, autonomous vehicles, barriers to entry, Big Tech, biodiversity loss, bitcoin, blockchain, blood diamond, Boston Dynamics, Burning Man, call centre, cashless society, Charles Babbage, Charles Lindbergh, Clayton Christensen, clean water, cloud computing, Colonization of Mars, computer vision, creative destruction, CRISPR, crowdsourcing, cryptocurrency, data science, Dean Kamen, deep learning, deepfake, DeepMind, delayed gratification, dematerialisation, digital twin, disruptive innovation, Donald Shoup, driverless car, Easter island, Edward Glaeser, Edward Lloyd's coffeehouse, Elon Musk, en.wikipedia.org, epigenetics, Erik Brynjolfsson, Ethereum, ethereum blockchain, experimental economics, fake news, food miles, Ford Model T, fulfillment center, game design, Geoffrey West, Santa Fe Institute, gig economy, gigafactory, Google X / Alphabet X, gravity well, hive mind, housing crisis, Hyperloop, impact investing, indoor plumbing, industrial robot, informal economy, initial coin offering, intentional community, Intergovernmental Panel on Climate Change (IPCC), Internet of things, invention of the telegraph, Isaac Newton, Jaron Lanier, Jeff Bezos, job automation, Joseph Schumpeter, Kevin Kelly, Kickstarter, Kiva Systems, late fees, Law of Accelerating Returns, life extension, lifelogging, loss aversion, Lyft, M-Pesa, Mary Lou Jepsen, Masayoshi Son, mass immigration, megacity, meta-analysis, microbiome, microdosing, mobile money, multiplanetary species, Narrative Science, natural language processing, Neal Stephenson, Neil Armstrong, Network effects, new economy, New Urbanism, Nick Bostrom, Oculus Rift, One Laptop per Child (OLPC), out of africa, packet switching, peer-to-peer lending, Peter H. Diamandis: Planetary Resources, Peter Thiel, planned obsolescence, QR code, RAND corporation, Ray Kurzweil, RFID, Richard Feynman, Richard Florida, ride hailing / ride sharing, risk tolerance, robo advisor, Satoshi Nakamoto, Second Machine Age, self-driving car, Sidewalk Labs, Silicon Valley, Skype, smart cities, smart contracts, smart grid, Snapchat, SoftBank, sovereign wealth fund, special economic zone, stealth mode startup, stem cell, Stephen Hawking, Steve Jobs, Steve Jurvetson, Steven Pinker, Stewart Brand, supercomputer in your pocket, supply-chain management, tech billionaire, technoutopianism, TED Talk, Tesla Model S, Tim Cook: Apple, transaction costs, Uber and Lyft, uber lyft, unbanked and underbanked, underbanked, urban planning, Vision Fund, VTOL, warehouse robotics, Watson beat the top human players on Jeopardy!, We wanted flying cars, instead we got 140 characters, X Prize

In July 2018, with 6.7 million unfilled jobs in the US, labor shortages reached a record high. Not only are the jobs there, they’re there in record numbers. The ability to rapidly retrain our workforce to fill those jobs—that’s the challenge we have yet to meet. Existential Risks: Vision, Prevention, and Governance In 2002, a relatively unknown Oxford philosopher named Nick Bostrom published a paper in the Journal of Evolution and Technology. In a few years, Bostrom would vault to geek-fame because of his “Simulation Hypothesis,” which convincingly argues that we’re living inside the Matrix. However, this earlier paper also caused a stir, mainly because it scared the crap out of nearly everyone who read it.

…

See: https://www.businessinsider.com/goldman-sachs-says-self-driving-trucks-will-kill-300000-jobs-per-year-2017-11. 6.7 million unfilled jobs in the US: Jeff Cox, “The U.S. Labor Shortage Is Reaching a Critical Point,” CNBC Markets, July 5, 2018. See: https://www.cnbc.com/2018/07/05/the-us-labor-shortage-is-reaching-a-critical-point.html. Existential Risks: Vision, Prevention, and Governance Nick Bostrom: “Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards,” Journal of Evolution and Technology 9 (March 9, 2002). Vision Stewart Brand: Stewart Brand, Clock of the Long Now: Time and Responsibility, The Ideas Behind the World’s Slowest Computer (Basic Books, 1999), p.1.

The Coming Wave: Technology, Power, and the Twenty-First Century's Greatest Dilemma by Mustafa Suleyman

"World Economic Forum" Davos, 23andMe, 3D printing, active measures, Ada Lovelace, additive manufacturing, agricultural Revolution, AI winter, air gap, Airbnb, Alan Greenspan, algorithmic bias, Alignment Problem, AlphaGo, Alvin Toffler, Amazon Web Services, Anthropocene, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, ASML, autonomous vehicles, backpropagation, barriers to entry, basic income, benefit corporation, Big Tech, biodiversity loss, bioinformatics, Bletchley Park, Blitzscaling, Boston Dynamics, business process, business process outsourcing, call centre, Capital in the Twenty-First Century by Thomas Piketty, ChatGPT, choice architecture, circular economy, classic study, clean tech, cloud computing, commoditize, computer vision, coronavirus, corporate governance, correlation does not imply causation, COVID-19, creative destruction, CRISPR, critical race theory, crowdsourcing, cryptocurrency, cuban missile crisis, data science, decarbonisation, deep learning, deepfake, DeepMind, deindustrialization, dematerialisation, Demis Hassabis, disinformation, drone strike, drop ship, dual-use technology, Easter island, Edward Snowden, effective altruism, energy transition, epigenetics, Erik Brynjolfsson, Ernest Rutherford, Extinction Rebellion, facts on the ground, failed state, Fairchild Semiconductor, fear of failure, flying shuttle, Ford Model T, future of work, general purpose technology, Geoffrey Hinton, global pandemic, GPT-3, GPT-4, hallucination problem, hive mind, hype cycle, Intergovernmental Panel on Climate Change (IPCC), Internet Archive, Internet of things, invention of the wheel, job automation, John Maynard Keynes: technological unemployment, John von Neumann, Joi Ito, Joseph Schumpeter, Kickstarter, lab leak, large language model, Law of Accelerating Returns, Lewis Mumford, license plate recognition, lockdown, machine readable, Marc Andreessen, meta-analysis, microcredit, move 37, Mustafa Suleyman, mutually assured destruction, new economy, Nick Bostrom, Nikolai Kondratiev, off grid, OpenAI, paperclip maximiser, personalized medicine, Peter Thiel, planetary scale, plutocrats, precautionary principle, profit motive, prompt engineering, QAnon, quantum entanglement, ransomware, Ray Kurzweil, Recombinant DNA, Richard Feynman, Robert Gordon, Ronald Reagan, Sam Altman, Sand Hill Road, satellite internet, Silicon Valley, smart cities, South China Sea, space junk, SpaceX Starlink, stealth mode startup, stem cell, Stephen Fry, Steven Levy, strong AI, synthetic biology, tacit knowledge, tail risk, techlash, techno-determinism, technoutopianism, Ted Kaczynski, the long tail, The Rise and Fall of American Growth, Thomas Malthus, TikTok, TSMC, Turing test, Tyler Cowen, Tyler Cowen: Great Stagnation, universal basic income, uranium enrichment, warehouse robotics, William MacAskill, working-age population, world market for maybe five computers, zero day

Particular thanks go to the many people who read the entire draft and commented in detail; their generosity and extraordinary level of insight were completely invaluable in getting to the final manuscript. A huge thanks to Gregory Allen, Graham Allison (and the faculty and staff of Harvard’s Belfer Center more widely), Sahar Amer, Anne Applebaum, Julian Baker, Samantha Barber, Gabriella Blum, Nick Bostrom, Ian Bremmer, Erik Brynjolfsson, Ben Buchanan, Sarah Carter, Rewon Child, George Church, Richard Danzig, Jennifer Doudna, Alexandra Eitel, Maria Eitel, Henry Elkus, Kevin Esvelt, Jeremy Fleming, Jack Goldsmith, Al Gore, Tristan Harris, Zaid Hassan, Jordan Hoffman, Joi Ito, Ayana Elizabeth Johnson, Danny Kahneman, Angela Kane, Melanie Katzman, Henry Kissinger, Kevin Klyman, Heinrich Küttler, Eric Lander, Sean Legassick, Aitor Lewkowycz, Leon Marshall, Jason Matheny, Andrew McAfee, Greg McKelvey, Dimitri Mehlhorn, David Miliband, Martha Minow, Geoff Mulgan, Aza Raskin, Tobias Rees, Stuart Russell, Jeffrey Sachs, Eric Schmidt, Bruce Schneier, Marilyn Thompson, Mayo Thompson, Thomas Viney, Maria Vogelauer, Mark Walport, Morwenna White, Scott Young, and Jonathan Zittrain.

…

GO TO NOTE REFERENCE IN TEXT At Aum Shinrikyo’s peak popularity Federation of American Scientists, “The Operation of the Aum,” in Global Proliferation of Weapons of Mass Destruction: A Case Study of the Aum Shinrikyo, Senate Government Affairs Permanent Subcommittee on Investigations, Oct. 31, 1995, irp.fas.org/​congress/​1995_rpt/​aum/​part04.htm. GO TO NOTE REFERENCE IN TEXT As a report on the implications Danzig and Hosford, “Aum Shinrikyo.” GO TO NOTE REFERENCE IN TEXT Some will inevitably say See, for example, Nick Bostrom, “The Vulnerable World Hypothesis,” Sept. 6, 2019, nickbostrom.com/​papers/​vulnerable.pdf, for perhaps the most developed version of this thesis. In a thought experiment responding to the prospect of “easy nukes,” he envisages a “high-tech panopticon” where everyone has a “freedom tag,” “worn around the neck and bedecked with multidirectional cameras and microphones.

Zero to One: Notes on Startups, or How to Build the Future by Peter Thiel, Blake Masters

Airbnb, Alan Greenspan, Albert Einstein, Andrew Wiles, Andy Kessler, Berlin Wall, clean tech, cloud computing, crony capitalism, discounted cash flows, diversified portfolio, do well by doing good, don't be evil, Elon Musk, eurozone crisis, Fairchild Semiconductor, heat death of the universe, income inequality, Jeff Bezos, Larry Ellison, Lean Startup, life extension, lone genius, Long Term Capital Management, Lyft, Marc Andreessen, Mark Zuckerberg, Max Levchin, minimum viable product, Nate Silver, Network effects, new economy, Nick Bostrom, PalmPilot, paypal mafia, Peter Thiel, pets.com, power law, profit motive, Ralph Waldo Emerson, Ray Kurzweil, self-driving car, shareholder value, Sheryl Sandberg, Silicon Valley, Silicon Valley startup, Singularitarianism, software is eating the world, Solyndra, Steve Jobs, strong AI, Suez canal 1869, tech worker, Ted Kaczynski, Tesla Model S, uber lyft, Vilfredo Pareto, working poor

But an equally insidious danger for every business is to lose all sense of myth and mistake disenchantment for wisdom. Conclusion STAGNATION OR SINGULARITY? IF EVEN THE MOST FARSIGHTED founders cannot plan beyond the next 20 to 30 years, is there anything to say about the very distant future? We don’t know anything specific, but we can make out the broad contours. Philosopher Nick Bostrom describes four possible patterns for the future of humanity. The ancients saw all of history as a neverending alternation between prosperity and ruin. Only recently have people dared to hope that we might permanently escape misfortune, and it’s still possible to wonder whether the stability we take for granted will last.

Tools and Weapons: The Promise and the Peril of the Digital Age by Brad Smith, Carol Ann Browne

"World Economic Forum" Davos, Affordable Care Act / Obamacare, AI winter, air gap, airport security, Alan Greenspan, Albert Einstein, algorithmic bias, augmented reality, autonomous vehicles, barriers to entry, Berlin Wall, Big Tech, Bletchley Park, Blitzscaling, Boeing 737 MAX, business process, call centre, Cambridge Analytica, Celtic Tiger, Charlie Hebdo massacre, chief data officer, cloud computing, computer vision, corporate social responsibility, data science, deep learning, digital divide, disinformation, Donald Trump, Eben Moglen, Edward Snowden, en.wikipedia.org, Hacker News, immigration reform, income inequality, Internet of things, invention of movable type, invention of the telephone, Jeff Bezos, Kevin Roose, Laura Poitras, machine readable, Mark Zuckerberg, minimum viable product, national security letter, natural language processing, Network effects, new economy, Nick Bostrom, off-the-grid, operational security, opioid epidemic / opioid crisis, pattern recognition, precision agriculture, race to the bottom, ransomware, Ronald Reagan, Rubik’s Cube, Salesforce, school vouchers, self-driving car, Sheryl Sandberg, Shoshana Zuboff, Silicon Valley, Skype, speech recognition, Steve Ballmer, Steve Jobs, surveillance capitalism, tech worker, The Rise and Fall of American Growth, Tim Cook: Apple, Wargames Reagan, WikiLeaks, women in the workforce

They note, “However, processes of self-design and optimization might still lead to significant jumps in competencies.” T.G. Dietterich and E.J. Horvitz, “Rise of Concerns about AI: Reflections and Directions,” Communications of the ACM, vol. 58, no. 10, 38–40 (October 2015), http://erichorvitz.com/CACM_Oct_2015-VP.pdf. Nick Bostrom, a professor at Oxford University, explored these issues more broadly in his recent book. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014). Within the computer science field, some use the term “singularity” differently, to describe computing power that grows so quickly that it’s not possible to predict the future.

What We Owe the Future: A Million-Year View by William MacAskill

Ada Lovelace, agricultural Revolution, Albert Einstein, Alignment Problem, AlphaGo, artificial general intelligence, Bartolomé de las Casas, Bletchley Park, British Empire, Brownian motion, carbon footprint, carbon tax, charter city, clean tech, coronavirus, COVID-19, cuban missile crisis, decarbonisation, deep learning, DeepMind, Deng Xiaoping, different worldview, effective altruism, endogenous growth, European colonialism, experimental subject, feminist movement, framing effect, friendly AI, global pandemic, GPT-3, hedonic treadmill, Higgs boson, income inequality, income per capita, Indoor air pollution, Intergovernmental Panel on Climate Change (IPCC), Isaac Newton, Islamic Golden Age, iterative process, Jeff Bezos, job satisfaction, lab leak, Lao Tzu, Large Hadron Collider, life extension, lockdown, long peace, low skilled workers, machine translation, Mars Rover, negative emissions, Nick Bostrom, nuclear winter, OpenAI, Peter Singer: altruism, Peter Thiel, QWERTY keyboard, Robert Gordon, Rutger Bregman, Sam Altman, seminal paper, Shenzhen special economic zone , Shenzhen was a fishing village, Silicon Valley, special economic zone, speech recognition, Stanislav Petrov, stem cell, Steven Pinker, strong AI, synthetic biology, total factor productivity, transatlantic slave trade, Tyler Cowen, William MacAskill, women in the workforce, working-age population, World Values Survey, Y Combinator

Their aim would be to find an adult who will act in accordance with their wishes. The challenge is for the child to do this—rather than, say, appointing a schemer who is good at deceitful salesmanship but once in power would pursue their own agenda—even though the adults are much smarter and more knowledgeable than the child is. This risk was the focus of Nick Bostrom’s book Superintelligence. The scenario most closely associated with that book is one in which a single AI agent designs better and better versions of itself, quickly developing abilities far greater than the abilities of all of humanity combined. Almost certainly, its aims would not be the same as humanity’s aims.

…

Recently, seventy-five researchers at leading organisations in AI safety and governance were asked, “Assuming that there will be an existential catastrophe as a result of AI, what do you think will be the cause?”4 The respondents could give one of six answers: the first was a scenario in which a single AI system quickly takes over the world, as described in Nick Bostrom’s Superintelligence; second and third were AI-takeover scenarios involving many AI systems that improve more gradually; the fourth was that AI would exacerbate risk from war; the fifth was that AI would be misused by people (as I described at length in Chapter 4); and the sixth was “other.” The typical respondent put a similar probability across the first five scenarios, with “other” being given a one-in-five chance.

Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms: Proceedings of the Agi Workshop 2006 by Ben Goertzel, Pei Wang

AI winter, artificial general intelligence, backpropagation, bioinformatics, brain emulation, classic study, combinatorial explosion, complexity theory, computer vision, Computing Machinery and Intelligence, conceptual framework, correlation coefficient, epigenetics, friendly AI, functional programming, G4S, higher-order functions, information retrieval, Isaac Newton, Jeff Hawkins, John Conway, Loebner Prize, Menlo Park, natural language processing, Nick Bostrom, Occam's razor, p-value, pattern recognition, performance metric, precautionary principle, Ray Kurzweil, Rodney Brooks, semantic web, statistical model, strong AI, theory of mind, traveling salesman, Turing machine, Turing test, Von Neumann architecture, Y2K

Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards. Journal of Evolution and Technology, vol. 9. [28] Yudkowsky, Eliezer (2007). Cognitive Biases Potentially Affecting Judgment of Global Risks, in Global Catastrophic Risks, Ed. by Nick Bostrom and Milan Cirkovic, Oxford University Press. [29] Yudkowsky, Eliezer (2007). Artificial Intelligence and Global Risk, in Global Catastrophic Risks, Ed. by Nick Bostrom and Milan Cirkovic, Oxford University Press. Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms B. Goertzel and P. Wang (Eds.) IOS Press, 2007 © 2007 The authors and IOS Press.

The People vs Tech: How the Internet Is Killing Democracy (And How We Save It) by Jamie Bartlett

Ada Lovelace, Airbnb, AlphaGo, Amazon Mechanical Turk, Andrew Keen, autonomous vehicles, barriers to entry, basic income, Bernie Sanders, Big Tech, bitcoin, Black Lives Matter, blockchain, Boris Johnson, Californian Ideology, Cambridge Analytica, central bank independence, Chelsea Manning, cloud computing, computer vision, creative destruction, cryptocurrency, Daniel Kahneman / Amos Tversky, data science, deep learning, DeepMind, disinformation, Dominic Cummings, Donald Trump, driverless car, Edward Snowden, Elon Musk, Evgeny Morozov, fake news, Filter Bubble, future of work, general purpose technology, gig economy, global village, Google bus, Hans Moravec, hive mind, Howard Rheingold, information retrieval, initial coin offering, Internet of things, Jeff Bezos, Jeremy Corbyn, job automation, John Gilmore, John Maynard Keynes: technological unemployment, John Perry Barlow, Julian Assange, manufacturing employment, Mark Zuckerberg, Marshall McLuhan, Menlo Park, meta-analysis, mittelstand, move fast and break things, Network effects, Nicholas Carr, Nick Bostrom, off grid, Panopticon Jeremy Bentham, payday loans, Peter Thiel, post-truth, prediction markets, QR code, ransomware, Ray Kurzweil, recommendation engine, Renaissance Technologies, ride hailing / ride sharing, Robert Mercer, Ross Ulbricht, Sam Altman, Satoshi Nakamoto, Second Machine Age, sharing economy, Silicon Valley, Silicon Valley billionaire, Silicon Valley ideology, Silicon Valley startup, smart cities, smart contracts, smart meter, Snapchat, Stanford prison experiment, Steve Bannon, Steve Jobs, Steven Levy, strong AI, surveillance capitalism, TaskRabbit, tech worker, technological singularity, technoutopianism, Ted Kaczynski, TED Talk, the long tail, the medium is the message, the scientific method, The Spirit Level, The Wealth of Nations by Adam Smith, The Wisdom of Crowds, theory of mind, too big to fail, ultimatum game, universal basic income, WikiLeaks, World Values Survey, Y Combinator, you are the product