undersea cable

89 results back to index

Tubes: A Journey to the Center of the Internet by Andrew Blum

air freight, cable laying ship, call centre, digital divide, Donald Davies, global village, Hibernia Atlantic: Project Express, if you build it, they will come, inflight wifi, invisible hand, John Markoff, Kevin Kelly, Leonard Kleinrock, Marc Andreessen, Mark Zuckerberg, Menlo Park, Mercator projection, messenger bag, Neal Stephenson, Network effects, New Urbanism, packet switching, Ralph Waldo Emerson, RAND corporation, satellite internet, side project, Silicon Valley, Skype, South of Market, San Francisco, Steve Crocker, Steve Jobs, Steven Levy, undersea cable, urban planning, UUNET, WikiLeaks, zero-sum game

Our encounters with this kind of geography typically come with more familiar images, like a ribbon of interstate, a length of train track, or a 747 parked expectantly at the airport gate. But undersea cables are invisible. They feel more like rivers than paths, containing a continuous flow of energy rather than the occasional passing conveyance. If the first step in visiting the Internet was to imagine it, then undersea cables always struck me as its most magical places. And only more so when I realized their paths were often ancient. With few exceptions, undersea cables land in or near classic port cities, places like Lisbon, Marseille, Hong Kong, Singapore, New York, Alexandria, Mumbai, Cyprus, or Mombasa.

…

But for Alston the acceleration came with the arrival of an unfathomably long and skinny thing, a singular path across the bottom of the sea. Undersea cables are the ultimate totems of our physical connections. If the Internet is a global phenomenon, it’s because there are tubes underneath the ocean. They are the fundamental medium of the global village. The fiber-optic technology is fantastically complex and dependent on the latest materials and computing technology. Yet the basic principle of the cables is shockingly simple: light goes in on one shore of the ocean and comes out on the other. Undersea cables are straightforward containers for light, as a subway tunnel is for trains.

…

Not every customer wants a link from Kenya to London, but once you can do it, and do it consistently and do it well, people begin to think about things like call centers, which are constantly hunting for the place with the lowest cost services. The demand springs up.” Undersea cables link people—in rich nations, first—but the earth itself always stands in the way. To determine the route of an undersea cable requires navigating a maze of economics, geopolitics, and topography. For example, the curvature of the planet makes the shortest distance between Japan and the United States a northern arc paralleling the coast of Alaska and landing near Seattle.

The Idea Factory: Bell Labs and the Great Age of American Innovation by Jon Gertner

Albert Einstein, back-to-the-land, Black Swan, business climate, Charles Babbage, Claude Shannon: information theory, Clayton Christensen, complexity theory, corporate governance, cuban missile crisis, Dennis Ritchie, Edward Thorp, Fairchild Semiconductor, Henry Singleton, horn antenna, Hush-A-Phone, information retrieval, invention of the telephone, James Watt: steam engine, Karl Jansky, Ken Thompson, knowledge economy, Leonard Kleinrock, machine readable, Metcalfe’s law, Nicholas Carr, Norbert Wiener, Picturephone, Richard Feynman, Robert Metcalfe, Russell Ohl, Sand Hill Road, Silicon Valley, Skype, space junk, Steve Jobs, Telecommunications Act of 1996, Teledyne, traveling salesman, undersea cable, uranium enrichment, vertical integration, William Shockley: the traitorous eight

When the Labs’ chemists needed to help design impermeable underwater cables, for example, one possibility was gutta-percha. But gutta-percha had drawbacks, including its extreme expense. For an undersea cable to Catalina Island, off the coast of California, the Labs’ chemists began looking for an alternative. Natural rubber was considered too soft in its pure state, and when it was treated with sulfur to toughen it—that is, to “vulcanize” it—the men had to address the problem that sulfur corrodes copper and would undoubtedly degrade the vital wires within the undersea cable. Only after the chemists determined that they could purify the rubber in a complex manner and then create fine silica flour as an insulator could the cable go into production.

…

At the same time, he noted, there was an obvious and unique answer: Figure out a way to make an undersea cable with repeaters spaced every forty miles or so, and then figure out a way to make it work for twenty years without leaks or interruption. To a large degree, then, the cable was a challenge of engineering rather than science, but even Kelly seemed humbled by “the magnitude of the task, its tremendous scope, the necessity for establishing that the designs of each and every item meet not only functional requirements but also the twenty-year no-failure requirement.”5 Fortunately, by the time Kelly began planning an undersea cable with British phone engineers in 1953, a small library already existed about what would and would not work.

…

A number of multinational corporations—RCA, General Electric, ITT, among others—had already made clear their intentions, as Fortune magazine put it in July 1961, to install “the great cable in space.” Satellites offered an alternative to the increasingly burdened underseas cables; more important, they could carry live television, which the current underseas cables could not. Within a decade, some economists predicted, orbiting relays could be a billion-dollar-a-year business.31 “We think the best practical system would use about 50 satellites in space,” Kappel explained to U.S. News & World Report. Would AT&T spend $25 million to do it?

Seriously Curious: The Facts and Figures That Turn Our World Upside Down by Tom Standage

"World Economic Forum" Davos, agricultural Revolution, augmented reality, autonomous vehicles, Big Tech, blood diamond, business logic, corporate governance, CRISPR, deep learning, Deng Xiaoping, Donald Trump, Dr. Strangelove, driverless car, Elon Musk, failed state, financial independence, gender pay gap, gig economy, Gini coefficient, high net worth, high-speed rail, income inequality, index fund, industrial robot, Internet of things, invisible hand, it's over 9,000, job-hopping, Julian Assange, life extension, Lyft, M-Pesa, Mahatma Gandhi, manufacturing employment, mega-rich, megacity, Minecraft, mobile money, natural language processing, Nelson Mandela, plutocrats, post-truth, price mechanism, private spaceflight, prosperity theology / prosperity gospel / gospel of success, purchasing power parity, ransomware, reshoring, ride hailing / ride sharing, Ronald Coase, self-driving car, Silicon Valley, Snapchat, South China Sea, speech recognition, stem cell, supply-chain management, transaction costs, Uber and Lyft, uber lyft, undersea cable, US Airways Flight 1549, WikiLeaks, zoonotic diseases

Why 5G might be both faster and slower than previous wireless technologies Mobile phones are more common than electricity in much of sub-Saharan Africa Why self-driving cars will mostly be shared, not owned How ride-hailing apps reduce drink-driving What is augmented reality? Why we’re still waiting for the space elevator How astronomers spotted the first interstellar asteroid Why drones could pose a greater risk to aircraft than birds What is the point of spam e-mail? Why the police should wear body cameras Why tech giants are laying their own undersea cables Game theory: sport and leisure Why tennis players grunt Why board games are so popular in Nigeria How drones can keep beaches safe from sharks How football transfers work How St Louis became America’s chess capital What does “digitally remastering” a film really mean? How bookmakers deal with winning customers The world’s highest-earning football clubs Speaking my language: words and wisdom Why emoji have beneficial linguistic side-effects How the letters of the alphabet got their names Why Papua New Guinea has so many languages Is Serbo-Croatian one language or four?

…

Questions will then arise as to how long such data should be kept and in what circumstances recordings should be released to the public. Such worries are not insignificant. But the evidence is mounting that the usefulness of such cameras outweighs the drawbacks. Why tech giants are laying their own undersea cables In September 2017 Microsoft and Facebook announced the completion of a 6,600km (4,100-mile) cable stretching from Virginia Beach, Virginia, to Bilbao, Spain. Dubbed Marea, Spanish for “tide”, the bundle of eight fibre-optic threads, roughly the size of a garden hose, is the highest-capacity connection across the Atlantic Ocean.

…

Firms such as Google, Facebook and Microsoft used to lease all their international bandwidth from carriers such as BT or AT&T. Now they need so much network capacity to synchronise data across their networks of data centres around the world that it makes more sense to lay their own dedicated cables. This has led to a boom in new undersea cable systems. The Submarine Telecoms Forum, an industry body, reckons that 100,000km of submarine cable was laid in 2016, up from just 16,000km in 2015. TeleGeography, a market-research firm, reckons that $9.2bn will have been spent on such cable projects between 2016 and 2018, five times as much as in the previous three years.

Your Computer Is on Fire by Thomas S. Mullaney, Benjamin Peters, Mar Hicks, Kavita Philip

"Susan Fowler" uber, 2013 Report for America's Infrastructure - American Society of Civil Engineers - 19 March 2013, A Declaration of the Independence of Cyberspace, affirmative action, Airbnb, algorithmic bias, AlphaGo, AltaVista, Amazon Mechanical Turk, Amazon Web Services, American Society of Civil Engineers: Report Card, An Inconvenient Truth, Asilomar, autonomous vehicles, Big Tech, bitcoin, Bletchley Park, blockchain, Boeing 737 MAX, book value, British Empire, business cycle, business process, Californian Ideology, call centre, Cambridge Analytica, carbon footprint, Charles Babbage, cloud computing, collective bargaining, computer age, computer vision, connected car, corporate governance, corporate social responsibility, COVID-19, creative destruction, cryptocurrency, dark matter, data science, Dennis Ritchie, deskilling, digital divide, digital map, don't be evil, Donald Davies, Donald Trump, Edward Snowden, en.wikipedia.org, European colonialism, fake news, financial innovation, Ford Model T, fulfillment center, game design, gentrification, George Floyd, glass ceiling, global pandemic, global supply chain, Grace Hopper, hiring and firing, IBM and the Holocaust, industrial robot, informal economy, Internet Archive, Internet of things, Jeff Bezos, job automation, John Perry Barlow, Julian Assange, Ken Thompson, Kevin Kelly, Kickstarter, knowledge economy, Landlord’s Game, Lewis Mumford, low-wage service sector, M-Pesa, Mark Zuckerberg, mass incarceration, Menlo Park, meta-analysis, mobile money, moral panic, move fast and break things, Multics, mutually assured destruction, natural language processing, Neal Stephenson, new economy, Norbert Wiener, off-the-grid, old-boy network, On the Economy of Machinery and Manufactures, One Laptop per Child (OLPC), packet switching, pattern recognition, Paul Graham, pink-collar, pneumatic tube, postindustrial economy, profit motive, public intellectual, QWERTY keyboard, Ray Kurzweil, Reflections on Trusting Trust, Report Card for America’s Infrastructure, Salesforce, sentiment analysis, Sheryl Sandberg, Silicon Valley, Silicon Valley ideology, smart cities, Snapchat, speech recognition, SQL injection, statistical model, Steve Jobs, Stewart Brand, tacit knowledge, tech worker, techlash, technoutopianism, telepresence, the built environment, the map is not the territory, Thomas L Friedman, TikTok, Triangle Shirtwaist Factory, undersea cable, union organizing, vertical integration, warehouse robotics, WikiLeaks, wikimedia commons, women in the workforce, Y2K

So, too, were the years after Edward Snowden’s 2013 disclosures about NSA surveillance, which spurred many nations to start ocean-spanning cable projects, hoping to circumvent US networks. Undersea cable technology is not new; in the mid-nineteenth century, a global telegraph network depended on them. In the 1850s, undersea cables were made of copper, iron, and gutta-percha (a Malaysian tree latex introduced to the West by a colonial officer of the Indian medical service). A century later, they were coaxial cables with vacuum tube amplifier repeaters. By the late twentieth century, they were fiber optics. By 2015, 99 percent of international data traveled over undersea cables, moving information eight times faster than satellite transmission.19 Communication infrastructure has looked rather like tubes for a century and a half.

…

See Paul N. Edwards, “Platforms Are Infrastructures on Fire,” ch. 15 in this volume. 8. The essays in this volume address a wide variety of subject matter and case studies, ranging from human-in-the-loop content moderation to Bitcoin mining, from mobile banking platforms to game design, and from undersea cable networks to keyboard interfaces, among many other examples. At the same time, the volume does not aspire to be a “conspectus” on computing and new media. There are noteworthy omissions, such as high-frequency trading, 5G and the question of cellphone radiation, disability studies, biohacking, election hacking, and a great many other subjects.

…

Whom does it all serve? It should not surprise us that no other class of actor has benefited more from the enormous global reach, speed, and capacity of the internet than the organizations that prop it up. Not the users. Not the visionary network designers. Other networked organizations—from the undersea cable owners to service providers to big data brokers to data aggregators—have gained the most. While surely not all organizations intend to exploit people, it remains the historical rule that at least a few will, and in so prospering just a few bad actors may normalize exploitative behavior that serves networked organizations in general: networks now let us exchange data but at the cost of rendering ourselves products on the network subject to network surveillance, discrimination, gatekeeping, platforms rigged to reward addictive behavior and groupthink, toxic attention economies, and much else.

The Wires of War: Technology and the Global Struggle for Power by Jacob Helberg

"World Economic Forum" Davos, 2021 United States Capitol attack, A Declaration of the Independence of Cyberspace, active measures, Affordable Care Act / Obamacare, air gap, Airbnb, algorithmic management, augmented reality, autonomous vehicles, Berlin Wall, Bernie Sanders, Big Tech, bike sharing, Black Lives Matter, blockchain, Boris Johnson, Brexit referendum, cable laying ship, call centre, Cambridge Analytica, Cass Sunstein, cloud computing, coronavirus, COVID-19, creative destruction, crisis actor, data is the new oil, data science, decentralized internet, deep learning, deepfake, deglobalization, deindustrialization, Deng Xiaoping, deplatforming, digital nomad, disinformation, don't be evil, Donald Trump, dual-use technology, Edward Snowden, Elon Musk, en.wikipedia.org, end-to-end encryption, fail fast, fake news, Filter Bubble, Francis Fukuyama: the end of history, geopolitical risk, glass ceiling, global pandemic, global supply chain, Google bus, Google Chrome, GPT-3, green new deal, information security, Internet of things, Jeff Bezos, Jeffrey Epstein, John Markoff, John Perry Barlow, knowledge economy, Larry Ellison, lockdown, Loma Prieta earthquake, low earth orbit, low skilled workers, Lyft, manufacturing employment, Marc Andreessen, Mark Zuckerberg, Mary Meeker, Mikhail Gorbachev, military-industrial complex, Mohammed Bouazizi, move fast and break things, Nate Silver, natural language processing, Network effects, new economy, one-China policy, open economy, OpenAI, Parler "social media", Peter Thiel, QAnon, QR code, race to the bottom, Ralph Nader, RAND corporation, reshoring, ride hailing / ride sharing, Ronald Reagan, Russian election interference, Salesforce, Sam Altman, satellite internet, self-driving car, Sheryl Sandberg, side project, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, Skype, smart grid, SoftBank, Solyndra, South China Sea, SpaceX Starlink, Steve Jobs, Steven Levy, Stuxnet, supply-chain attack, Susan Wojcicki, tech worker, techlash, technoutopianism, TikTok, Tim Cook: Apple, trade route, TSMC, Twitter Arab Spring, uber lyft, undersea cable, Unsafe at Any Speed, Valery Gerasimov, vertical integration, Wargames Reagan, Westphalian system, white picket fence, WikiLeaks, Y Combinator, zero-sum game

As Priscilla Moriuchi, the director of strategic threat development at a Massachusetts cybersecurity company, has warned, “It’s the technologies of the future and technologies of future security systems that could be the biggest security threat in the Belt and Road project.”32 All of these efforts amount to a bewildering mix of corporate deal-making and state-sponsored extortion. They traverse a tangle of undersea cables, next-generation wireless networks, and global supply chains. But broadly speaking, we can break China’s back-end strategy for dominating the Internet into four main components: controlling the production, controlling the pipes, controlling the protocols, and controlling the post-4G future. To help make sense of how these four puzzle pieces fit together, consider the many steps involved in sending a simple email.

…

“It’s in the ocean.”65 Roughly a century and a half after that first copper connection under the English Channel, there are around 400 underwater cables crisscrossing nearly 750,000 miles of ocean floor.66 The longest runs some 24,000 miles, stopping at nearly 40 different landing points along the way—beginning in Germany, hooking around Spain and Portugal, traveling through the Mediterranean and the Red Sea, then hugging the coast of the Indian subcontinent on its way to twin terminuses in Korea and Australia.67 All together, these cables carry 95 percent of the world’s communications.68 And China is well on its way to controlling many of them. * * * The British company that laid that first undersea cable in 1850 still exists. In 2008, its successor entity, Global Marine, formed a joint venture with a company that will sound very familiar—a subsidiary called Huawei Marine. For decades, American companies (and a few infrastructure giants like India’s Tata) were the only firms with the resources to manufacture, install, and maintain the infrastructure of a giant telecommunications system.

…

In fact, content providers now control over half the bandwidth on these underwater cables.69 Facebook and Google, for instance, partnered with several other firms to design and build the first underwater cable in two decades to connect the United States and northern Europe.III It’s part of the nearly $40 billion that Facebook and Google invested in network infrastructure like undersea cables in 2018 alone.70 But as China’s wealth and influence have grown, Chinese companies have joined the undersea game. Over the past decade, Huawei Marine has taken on roughly ninety projects to construct or modernize subsea cables, some connecting to key American allies.71 Huawei has completed a 3,750-mile cable between Cameroon and Brazil.

The Cable by Gillian Cookson

British Empire, cable laying ship, financial engineering, it's over 9,000, joint-stock company, Monroe Doctrine, undersea cable

In one respect, though, the Americans could not be self-sufficient. England was the only possible source of submarine cables. British dominance in this area was partly a result of their monopoly on the insulating material, gutta percha. More importantly, much of the expertise accumulated during the short life of undersea cables was concentrated in Britain. Cyrus Field therefore set out again to London, to inspect samples of cable made for him by Kuper & Co. using copper cores from the Gutta Percha Co. His only other contact there was John Watkins Brett, who had become head of the Magnetic Telegraph Co. Gisborne, by then on travels around the United States, had furnished Field with a letter of introduction to Brett.

…

Once applied as an insulator to submarine cables, it needed to be stored in water to retain its remarkable properties. Its supremacy lasted for a century, until the advent of polyethylene-based synthetics, handing Britain, which had a monopoly on gutta percha, a long-lasting stranglehold on the production of undersea cables. While the material itself was seen as a godsend, there was still some way to go in improving the way it was applied to the cable. With land telegraphs, most of which were carried on overhead lines, insulation was hardly an issue. The cable was insulated by glass or earthenware holders at points where it met a telegraph pole.

…

For a submarine line, though, the insulation had to be of consistently high quality to prevent any contact between the copper conductor and the water all around. The insulation problem turned out to be much more than a matter of waterproofing. Even when the quality of the cable was excellent, and the insulation perfectly sound, undersea cables simply did not function as expected. Messages would not pass down the line at anything like an acceptable speed without breaking down into a chaotic jumble. This electrical phenomenon came to be called ‘retardation of the signals’, sometimes known as ‘induction’. Retardation was at the root of problems with long cables, and until it could be understood and its effects overcome, the Atlantic cable could never work.

Some Remarks by Neal Stephenson

airport security, augmented reality, barriers to entry, Bletchley Park, British Empire, cable laying ship, call centre, cellular automata, edge city, Eratosthenes, Fellow of the Royal Society, Hacker Ethic, high-speed rail, impulse control, Iridium satellite, Isaac Newton, Jaron Lanier, John von Neumann, Just-in-time delivery, Kevin Kelly, Kim Stanley Robinson, megaproject, music of the spheres, Neal Stephenson, Neil Armstrong, Norbert Wiener, offshore financial centre, oil shock, packet switching, pirate software, Richard Feynman, Saturday Night Live, shareholder value, Shenzhen special economic zone , Silicon Valley, Skype, slashdot, Snow Crash, social web, Socratic dialogue, South China Sea, SpaceShipOne, special economic zone, Stephen Hawking, the scientific method, trade route, Turing machine, undersea cable, uranium enrichment, Vernor Vinge, X Prize

The cyberspace-warping power of wires, therefore, changes the geometry of the world of commerce and politics and ideas that we live in. The financial districts of New York, London, and Tokyo, linked by thousands of wires, are much closer to each other than, say, the Bronx is to Manhattan. Today this is all quite familiar, but in the 19th century, when the first feeble bits struggled down the first undersea cable joining the Old World to the New, it must have made people’s hair stand up on end in more than just the purely electrical sense—it must have seemed supernatural. Perhaps this sort of feeling explains why when Samuel Morse stretched a wire between Washington and Baltimore in 1844, the first message he sent with his code was “What hath God wrought!”

…

The only things that have changed since then are that the stakes have gotten smaller, the process more bureaucratized, and the personalities less interesting. Those early cables were eventually made to work, albeit not without founding whole new fields of scientific inquiry and generating many lucrative patents. Undersea cables, and long-distance communications in general, became the highest of high tech, with many of the same connotations as rocket science or nuclear physics or brain surgery would acquire in later decades. Some countries and companies (the distinction between countries and companies is hazy in the telco world) became very good at it, and some didn’t.

…

Because the protocol laid down a standard way for people to interact, which was clearly set out and could be understood by anyone, it attracted smart, adaptable, ambitious people from all over the place, and at a certain point it flew completely out of control and turned into something that no one had ever envisioned: something thriving, colorful, wildly diverse, essentially peaceful, and plagued only by the congestion of its own success. JARING’s link to the global Internet is over an undersea cable that connects it to the United States. This is typical of many Southeast Asian countries, which are far better connected to the US than they are to one another. But in late June of 1996, a barge called the Elbe appeared off the coast of Penang. Divers and boats came ashore, braving an infestation of sea snakes, and floated in a segment of armored cable that will become Malaysia’s link to FLAG.

Black Code: Inside the Battle for Cyberspace by Ronald J. Deibert

4chan, air gap, Any sufficiently advanced technology is indistinguishable from magic, Brian Krebs, call centre, citizen journalism, Citizen Lab, cloud computing, connected car, corporate social responsibility, crowdsourcing, cuban missile crisis, data acquisition, digital divide, disinformation, end-to-end encryption, escalation ladder, Evgeny Morozov, failed state, Firefox, Gabriella Coleman, global supply chain, global village, Google Hangouts, Hacker Ethic, Herman Kahn, informal economy, information security, invention of writing, Iridium satellite, jimmy wales, John Gilmore, John Markoff, Kibera, Kickstarter, knowledge economy, Lewis Mumford, low earth orbit, Marshall McLuhan, military-industrial complex, MITM: man-in-the-middle, mobile money, mutually assured destruction, Naomi Klein, new economy, Occupy movement, off-the-grid, Panopticon Jeremy Bentham, planetary scale, rent-seeking, Ronald Reagan, Ronald Reagan: Tear down this wall, Silicon Valley, Silicon Valley startup, Skype, smart grid, South China Sea, Steven Levy, Streisand effect, Stuxnet, Ted Kaczynski, the medium is the message, Turing test, Twitter Arab Spring, undersea cable, unit 8200, We are Anonymous. We are Legion, WikiLeaks, Yochai Benkler, zero day

It may come as a surprise that the same type of cables that Shakarian accidentally unearthed traverse the world’s lakes and oceans, and bind cyberspace together in a very material sense. Undersea cables are one of the links that connect today’s cyberspace to the late Industrial Revolution. The first such cables were laid in the late nineteenth century to facilitate telegraph traffic over long distances. Early designs were prone to failure and barely allowed the clicks of a telegraph exchange to be discerned across small bodies of water like the English Channel, but over time innovations in electronics and protective cable sheathings allowed the undersea cable industry to flourish. (This growth led to a dramatic increase in international telephone calls, and a new market for the sap of gutta-percha trees, which was used to coat and protect the cables until the mid-twentieth century.)

…

Although international telecommunications have been supplemented with microwave and satellite transmissions, a surprisingly large volume of data still traverses the world through cables crossing the Atlantic and Pacific oceans, and major bodies of water like the Mediterranean Sea. Due to the staggering costs involved, companies often share the same undersea cable trenches and sometimes competing companies even share the same protective sheathing. This makes those trenches highly vulnerable to major disruption. In a May 2012 article published on the website Gizmodo, provocatively titled “How to Destroy the Internet,” the author details the physical elements of the Internet that could be easily targeted.

…

But a review of video surveillance taken of the harbour during the outage period showed no ship traffic in the area of the severed cable. Others suggested it could have been a minor earthquake, causing a shift in the ocean floor, but seismic data didn’t support this conjecture. Whatever the cause, such cuts to cables are fairly routine: Even in their trenches, undersea cables are pushed to and fro by currents and constantly rub against a rough seafloor. In the case of the 2008 Mediterranean incident, the damage was severe: there were disruptions to 70 percent of Internet traffic in Egypt and 60 percent in India, and outages in Afghanistan, Bahrain, Bangladesh, Kuwait, the Maldives, Pakistan, Qatar, Saudi Arabia, and the United Arab Emirates.

The Emperor's New Road: How China's New Silk Road Is Remaking the World by Jonathan Hillman

"World Economic Forum" Davos, British Empire, cable laying ship, capital controls, colonial rule, coronavirus, COVID-19, Deng Xiaoping, Donald Trump, drone strike, energy security, facts on the ground, high-speed rail, intermodal, joint-stock company, Just-in-time delivery, land reform, low interest rates, M-Pesa, Malacca Straits, megaproject, moral hazard, offshore financial centre, rent-seeking, Scramble for Africa, Silicon Valley, smart cities, South China Sea, special economic zone, Suez canal 1869, Suez crisis 1956, supply-chain management, trade route, transcontinental railway, undersea cable, union organizing, Washington Consensus

Mackinder and Mahan, for all their wisdom, did not live in a world of airpower, let alone cyber power. While Western capitals debate the risks of allowing Chinese technology into their next-generation wireless networks, Chinese firms are rapidly connecting the world’s next-generation markets, as Chapter 9 explains. In East Africa, Chinese firms are laying underseas cables, equipping cities with surveillance cameras, and building wireless networks. On Maslow’s hierarchy of digital needs, network security is secondary to cost, positioning China’s tech champions to grow globally even if their Western presence withers. New connections, physical and digital, also produce unintended consequences.

…

Building south to Gwadar would require upgrading an existing line, for which feasibility studies have only begun.109 Altogether, the two extensions are estimated to cost $16 billion, and each will take at least five years to build.110 Officially, the southern component is slated to begin construction in 2025, and the northern component will begin in 2030, close enough to allow officials to speak about the project with a straight face and far enough away to allow them to comfortably delay it.111 While the railway and pipeline have not moved forward, the Chinese telecom giant Huawei has laid a fiber-optic cable across the Khunjerab Pass. At $44 million, it is less expensive and more practical.112 Huawei representatives point out that it will allow Pakistanis to access Chinese content more quickly, and by avoiding undersea cables, the route makes espionage from neighboring countries other than China more difficult.113 Unlike the proposed pipeline, the fiber-optic cable is less vulnerable to attack and cheaper to repair. Unlike the proposed railway, it cannot carry people. It offers the type of connectivity that is most comfortable to both governments.

…

Not to be left out, Russia and Saudi Arabia have expressed interest in setting up bases. One of Djibouti’s greatest assets is beneath the waves. It is a major landing point for underseas fiber-optic cables and a critical stop on China’s “digital silk road.” Globally, there are about 380 active underseas cables, which carry the vast majority of international data. Data from cell phones, for example, is sent to a tower and then onward across terrestrial and submarine cables. These cables will only become more important with the arrival of 5G and other services that will increase the speed and volume of data that is transferred.

2034: A Novel of the Next World War by Elliot Ackerman, James Admiral Stavridis

coronavirus, COVID-19, creative destruction, cuban missile crisis, digital map, loose coupling, mutually assured destruction, South China Sea, sovereign wealth fund, undersea cable

“Perfect,” said Kolchak as he stared at their thrashing dorsal fins. As if sensing Farshad’s confusion, he explained himself. “Those sharks are heading in the direction of the 10G undersea cables. They’re attracted to the electromagnetic energy. Those cables connect to the United States, and sharks have been known to chew through them. Their presence will give us deniability.” Destroying a few of the undersea cables would send a powerful message to the Americans, slowing internet across the country by as much as 60 percent, or so Farshad had been told by Kolchak. This might be enough to de-escalate the crisis, to bring everyone to their senses.

…

Equally obvious was the opportunity presented by its absence. No fewer than five hundred fiber-optic cables, which accounted for 90 percent of North America’s 10G internet access, crisscrossed these icy depths. “If the Americans detonate a nuclear weapon,” said Kolchak, “I don’t think the world will much care if we tamper with a few undersea cables.” He held Farshad in his gaze. “I also don’t think the world would say much if our troops seized a sliver of Poland, to unite Kaliningrad to the Russian mainland.” Kolchak pointed to a map on the wall. He traced out a corridor with his finger, which would give Russia direct overland access to its one Baltic port.

…

He would imagine the positions taken by Trent Wisecarver, by the various service chiefs and cabinet secretaries, the tabulations of arguments for or against what they were about to do—what they had all committed themselves to do when the president had put down her “red line” and dared her counterparts in Beijing to cross it. Which was what it seemed Beijing had now done, though not in the way anyone had anticipated. The cutting of the undersea cables and the resulting plunge into darkness was the demonstrable fact that, when discussed around the conference table, proved Beijing had crossed the red line. The question was the response. And even that was settled in remarkably short order. Chowdhury envisioned the scene—a disquisition of US interests by Wisecarver, followed by a range of options (or lack thereof) presented by the Joint Chiefs, and then formal nuclear authorizations being granted by the president herself.

Time Lord: Sir Sandford Fleming and the Creation of Standard Time by Clark Blaise

British Empire, Cornelius Vanderbilt, creative destruction, Dava Sobel, digital divide, James Watt: steam engine, John Harrison: Longitude, junk bonds, Khartoum Gordon, Robert Gordon, scientific management, Silicon Valley, transcontinental railway, traveling salesman, undersea cable, Upton Sinclair

In the 1870s and early 1880s he orchestrated world support for standard time, and in 1884 he succeeded. Still later, and at great personal cost, he bravely rallied the informal commonwealth of overseas dominions against British communications monopolies, and succeeded in laying the worldwide, undersea cable. A thousand miles east of Toronto, in 1851, the Railway Minister of the Nova Scotia colony, Joseph Howe, ventured a prophecy that is almost eerie in its accuracy. Fleming heard it as well, and filed it for later use. Fleming and Howe would not meet for another thirteen years, by which time Howe’s obvious capacity for leadership would carry him to the top of Nova Scotia’s parliamentary ranks.

…

What did those new “social” sciences prove but that man behaved irrationally, and that one of the vaunted Victorian articles of faith, “character,” derived as much from secret drives, repressions, or unconscious influences as from sturdy moral and social exempla? After the standardtime issues had been settled, Fleming spent the next twenty years perfecting a world-circling undersea cable that would link England instantaneously with all her distant colonies. He believed (and how could he not?) that intimate and immediate contact would spawn greater knowledge, loyalty, and affection between the mother of parliaments and her satellites. The better we know our government, the deeper our respect.

…

Rivers were crossed, ships’ designs turned from wood and sail to steel and iron, hold capacities and passenger cabins expanded a hundredfold, with a need to fill their holds with thousands of tons of coal for ocean passage. Meticulous planning and astronomical start-up costs entered the calculation for any new enterprise, and London and Continental banks oversaw bond issues for projects that might have seemed fanciful only a generation earlier: undersea cables, new shipyards, new steel mills, new mining equipment; transcontinental railroads spanning Canada, South Africa, America, India; telegraphs down the African coast. The new technology ran on coal, on more coal than traditional methods could ever extract. Workers had to be at least semiskilled just to handle the demands of the new technology, and, in the end, many grew sufficiently confident to challenge ancient wisdom, or to suggest shortcuts to greater efficiency.

The Perfect Weapon: War, Sabotage, and Fear in the Cyber Age by David E. Sanger

active measures, air gap, autonomous vehicles, Bernie Sanders, Big Tech, bitcoin, Black Lives Matter, Bletchley Park, British Empire, call centre, Cambridge Analytica, Cass Sunstein, Chelsea Manning, computer age, cryptocurrency, cuban missile crisis, disinformation, Donald Trump, drone strike, Edward Snowden, fake news, Google Chrome, Google Earth, information security, Jacob Appelbaum, John Markoff, Kevin Roose, Laura Poitras, Mark Zuckerberg, MITM: man-in-the-middle, mutually assured destruction, off-the-grid, RAND corporation, ransomware, Sand Hill Road, Sheryl Sandberg, Silicon Valley, Silicon Valley ideology, Skype, South China Sea, Steve Bannon, Steve Jobs, Steven Levy, Stuxnet, Tim Cook: Apple, too big to fail, Twitter Arab Spring, undersea cable, unit 8200, uranium enrichment, Valery Gerasimov, WikiLeaks, zero day

The diagram didn’t specify exactly how the NSA was planning on getting between those servers, but there were only a few possible options. The NSA would have to hack in remotely from one of its bases around the world, physically tap the undersea cables themselves, or get cooperation from a foreign partner, such as the British. The most likely method was physically tapping the termination points in a country where the undersea cables came ashore. And since Google had not gotten around to encrypting the data that was “in transit” through these cables, merely getting into the network itself was the price of admission to the data. When the Washington Post first published the slide, on October 30, 2013, over four months after the first Snowden revelations, the reaction inside the Googleplex in Mountain View was immediate and predictable.

…

The slides explaining Shotgiant in the Snowden trove gave a sense of the NSA’s thinking: “If we can determine the company’s plans and intentions,” an analyst wrote, “we hope that this will lead us back to the plans and intentions of the PRC.” The NSA saw an additional opportunity: As Huawei invested in new technology and laid undersea cables to connect its networking empire, the agency was interested in tunneling into key Chinese customers, including “high priority targets—Iran, Afghanistan, Pakistan, Kenya, Cuba.” In short, eager as the NSA was to figure out whether Huawei was the PLA’s puppet, it was more interested in putting its own back doors into Huawei networks.

…

In the open cubicles that looked out on the old air base, Moffett Field, in the heart of what is now Silicon Valley—the remnant of a pre–World War II age when the airplane was remaking global power—Grosse and his team of engineers were working day and night to NSA-proof the Google systems. The project, Grosse said, had actually begun long before they saw the smiley-face diagram. As early as 2008, Google had been investing in consortiums that laid undersea cables. But sharing had its risks: The company was not fully in control of who else had access to the cables, and there was always the risk that Google’s traffic could be thrown off the lines during an emergency, leaving its users without access to their data. Less than a decade later, Google had undertaken a multibillion-dollar effort to put down its own fiber-optic lines across the Atlantic and the Pacific so that it could control the speed and reliability of information that flowed between its servers and its users.

Internet for the People: The Fight for Our Digital Future by Ben Tarnoff

4chan, A Declaration of the Independence of Cyberspace, accounting loophole / creative accounting, Alan Greenspan, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, algorithmic management, AltaVista, Amazon Web Services, barriers to entry, Bernie Sanders, Big Tech, Black Lives Matter, blue-collar work, business logic, call centre, Charles Babbage, cloud computing, computer vision, coronavirus, COVID-19, decentralized internet, deep learning, defund the police, deindustrialization, desegregation, digital divide, disinformation, Edward Snowden, electricity market, fake news, Filter Bubble, financial intermediation, future of work, gamification, General Magic , gig economy, God and Mammon, green new deal, independent contractor, information asymmetry, Internet of things, Jeff Bezos, Jessica Bruder, John Markoff, John Perry Barlow, Kevin Roose, Kickstarter, Leo Hollis, lockdown, lone genius, low interest rates, Lyft, Mark Zuckerberg, means of production, Menlo Park, natural language processing, Network effects, Nicholas Carr, packet switching, PageRank, pattern recognition, pets.com, profit maximization, profit motive, QAnon, recommendation engine, rent-seeking, ride hailing / ride sharing, Sheryl Sandberg, Shoshana Zuboff, side project, Silicon Valley, single-payer health, smart grid, social distancing, Steven Levy, stock buybacks, supply-chain management, surveillance capitalism, techlash, Telecommunications Act of 1996, TikTok, transportation-network company, Travis Kalanick, Uber and Lyft, Uber for X, uber lyft, undersea cable, UUNET, vertical integration, Victor Gruen, web application, working poor, Yochai Benkler

India shuts down the internet more than any other country; see Marc Daniel Davies, “Internet Shutdowns Plunged Millions into ‘Digital Darkness’ Last Year,” Bloomberg, March 3, 2021. 65, But state spying, censorship, and shutdowns … NSA tapping fiber-optic cables: Craig Timberg, “NSA Slide Shows Surveillance of Undersea Cables,” Washington Post, July 10, 2013, and Olga Khazan, “The Creepy, Long-Standing Practice of Undersea Cable Tapping,” The Atlantic, July 16, 2013. 65, The post office was the first … Postal confidentiality: Anuj C. Desai, “Wiretapping before the Wires: The Post Office and the Birth of Communications Privacy,” Stanford Law Review 60, no. 2 (2007): 553–94.

…

I do the same with Facebook, which in late 2021 rebranded as Meta. 29, Finding a more direct … The interconnection of small and medium-sized networks is known as “donut peering”; the “hole” in the donut consists of the larger networks that are being bypassed. Google Fiber: David Anders, “Whatever Happened to Google Fiber?,” CNET, March 5, 2021. 29, Content providers like Google … Own or lease more than half of undersea bandwidth: Adam Satariano, “How the Internet Travels Across Oceans,” New York Times, March 10, 2019. Google building its own undersea cables: Jameson Zimmer, “Google Owns 63,605 Miles and 8.5% of Submarine Cables Worldwide,” Broadband-Now, September 12, 2018. 30, The proponents of a … “Innovation, expanded …”: The “Framework for Global Electronic Commerce” is available at clinton whitehouse4.archives.gov. 30, How well does … Internet costs higher in US than in Europe or Asia: Becky Chao and Claire Park, “The Cost of Connectivity 2020,” New America, July 2020.

Kitten Clone: Inside Alcatel-Lucent by Douglas Coupland

"World Economic Forum" Davos, British Empire, cable laying ship, Claude Shannon: information theory, cosmic microwave background, Downton Abbey, Golden arches theory, Great Leap Forward, Hibernia Atlantic: Project Express, hiring and firing, industrial research laboratory, Isaac Newton, Jeff Bezos, Marshall McLuhan, messenger bag, military-industrial complex, Neal Stephenson, oil shale / tar sands, pre–internet, quantum entanglement, Richard Feynman, Silicon Valley, Skype, Steve Jobs, tech worker, technological determinism, TED Talk, Turing machine, undersea cable, upwardly mobile, urban planning, UUNET, Wall-E

Eight men, working six months on/six months off, walking in clockwise circles again and again and again, each successive layer only two thirds of an inch higher than the one below it. It’s about as entrancing as watching insolent teenagers chew gum, but completely necessary. The Île de Bréhat’s undersea cable will be deployed on the bottom of the Atlantic Ocean at a rate of up to seven kilometres per hour, and it will take seven months for all of it to be laid. The ocean is as hilly and mountainous as the world above the water, and a one-kilometre length of cable might cover only seven hundred metres on a map.

…

Marine cable names, when viewed together, somewhat resemble a catalogue of extremely high-end yachting gear and sonar equipment: ACS Alaska-Oregon Network (AKORN) Alonso de Ojeda ALPAL-2 American Samoa-Hawaii (ASH) Antillas 1 Asia-America Gateway Australia-Japan Cable Bahamas Domestic Submarine Network Baltica Batam-Rengit Cable System Canada-United States 1 Cayman-Jamaica Fiber System Challenger Bermuda-1 (CB-1) Colombia-Florida Subsea Fiber (CFX-1) Denmark-Sweden 17 East African Marine System Emerald Bridge Estepona-Tetouan Fehmarn Bält Fiber Optic Gulf Fibralink Finland Estonia Connection Gemini Bermuda Geo-Eirgrid GO-1 Mediterranean Cable System Gondwana-1 Greenland Connect Guam Okinama Kyushu Incheon (GOKI) Hibernia Atlantic Hokkaido-Sakhalin Cable System Kattegat 1 Kodiak Kenai Fiber Link Latin American Nautilus Latvia-Sweden 1 (LV-SE 1) Lower Indian Ocean Network (LION) North Asia Loop Pacific Fibre Pangea Baltic Ring Pangea North Saudi Arabia-Sudan-2 (SAS-2) SeaMeWe-4 Svalbard Undersea Cable System Tangerine Tasman-2 Tata TGN-Pacific Turcyos-2 Ulysses Yellow/Atlantic Crossing-2 (AC-2) Near land, most marine cables are laid inside deep trenches cut by deep-water plows. Once on land, the cables travel as unobtrusively as possible to the bland, anonymous-looking routing centres mentioned earlier, mostly in London, New York, Frankfurt, Amsterdam, suburban Virginia, Palo Alto, California and Singapore.

The Metaverse: And How It Will Revolutionize Everything by Matthew Ball

"hyperreality Baudrillard"~20 OR "Baudrillard hyperreality", 3D printing, Airbnb, Albert Einstein, Amazon Web Services, Apple Newton, augmented reality, Big Tech, bitcoin, blockchain, business process, call centre, cloud computing, commoditize, computer vision, COVID-19, cryptocurrency, deepfake, digital divide, digital twin, disintermediation, don't be evil, Elon Musk, en.wikipedia.org, Ethereum, ethereum blockchain, game design, gig economy, Google Chrome, Google Earth, Google Glasses, hype cycle, intermodal, Internet Archive, Internet of things, iterative process, Jeff Bezos, John Gruber, Kevin Roose, Kickstarter, lockdown, Mark Zuckerberg, Metcalfe’s law, Minecraft, minimum viable product, Neal Stephenson, Network effects, new economy, non-fungible token, open economy, openstreetmap, pattern recognition, peer-to-peer, peer-to-peer model, Planet Labs, pre–internet, QR code, recommendation engine, rent control, rent-seeking, ride hailing / ride sharing, Robinhood: mobile stock trading app, satellite internet, self-driving car, SETI@home, Silicon Valley, skeuomorphism, Skype, smart contracts, Snapchat, Snow Crash, social graph, social web, SpaceX Starlink, Steve Ballmer, Steve Jobs, thinkpad, TikTok, Tim Cook: Apple, TSMC, undersea cable, Vannevar Bush, vertical integration, Vitalik Buterin, Wayback Machine, Y2K

In addition, most internet cables are not laid as the crow flies—they must navigate international rights, geographic impediments, and cost/benefit analyses. As a result, many countries and major cities lack a direct connection. NYC has a direct undersea cable to France, but not to Portugal. Traffic from the United States can go directly to Tokyo, but reaching India requires jumping from one undersea cable to another on the Asian or Oceanian continent. A single cable could be laid from the United States to India, but it would need to navigate through or around Thailand—adding hundreds or even thousands of miles—and that only solves shore-to-shore transmission.

…

Laying (or replacing) cables means working around extensive transportation infrastructure (freeways and railways), various population centers (each with their own political processes, constituencies, and incentives), and protected parks and lands. Laying a cable over a seamount in international waters is simple compared to laying a cable over a private-public mountain range. Image 1. Undersea Cables A map of the nearly 500 submarine cables and 1,250 landing stations that enable the global internet. TeleGeography The phrase “internet backbone” might bring to mind a largely planned out, and partly federated, network of cables. In truth, the internet backbone is really a loose federation of private networks.

…

See augmented reality (AR); virtual reality (VR) “mobile internet era,” 12, 35, 63, 64, 240, 242–44, 269, 291 Mojang, 114 Monsters University, 36, 89–90 Moore’s Law, 100, 161 “most favored nations” (MFN) clauses, 180–81, 183 Motorola, 158, 212, 213 multiplayer games battle royale games, 32–33, 91–93, 98, 114–15, 117n, 146–47, 268, 275 computing and, 75 “massively multiplayer” online games, 55 matchmaking, 81, 176, 178 the problem of concurrent users (CCUs), 54–55, 90–92, 122, 146, 234, 245, 261, 268, 283 see also specific games and gaming platforms Multi-User Dungeons (MUDs) and related topics, 7–8, 12, 13–14, 30 Musk, Elon Neuralink, 154 SpaceX, xi, 4, 87 Starlink, 87 Tesla, 101, 166, 271 X.com, 61 MySpace, 34, 274 Nadella, Satya, xii, 18, 141, 239, 279–80 Naked Sun, The, 5 Namco, 173 NASA, ix NASDAQ, 309 National Advisory Committee for Aeronautics, ix National Basketball Association (NBA), 218, 259 National Football League (NFL), 139 National Science Foundation, x native apps, 26, 194–95 near-field communication (NFC), 142, 189, 199–200, 203 Nest Labs, 158 “netcode” solutions, 81–82 NetEase, 19–20 Netflix, 19, 49–53, 96, 100, 111, 187, 194, 196–98, 244, 270, 276n Netherlands, 203 Network Attached Storage (NAS) drive, 74 networking, 71–88, 85 acceptability thresholds in, 80–81 bandwidth and, 27, 41, 48–51, 64, 72–79, 80, 96, 100, 305 broadband operators, 15–16, 38, 49, 271 fiber optic networks, 27, 84, 87, 128 4G networks, 81, 87, 244, 245, 249 5G networks, 87, 243 the internet as a “network of networks,” 16, 23–24, 62 latency and, 27, 48–53, 64, 79–88, 95–96, 99–100, 230, 243, 248, 271, 305 network gateways, 130 rent-seeking and, 15, 299 3G networks, 243 undersea cables, 84–85, 85 Neuralink, 154 Neuromancer, 5–6, 8 Newsweek, 308 New World, 277 New York Times, xv, 4–5, 73, 224, 256 as an app, 149, 185, 194, 196, 202 Live Election feed, 49 Nexon, 105 NextVR, 144 Niantic, 115, 144, 275 Nike, 121, 139, 189–90, 208, 248, 264 Nintendo, 11, 104, 151–52, 173, 303 Nintendo Entertainment System (NES), 32, 173 Nintendo Switch, 30–32, 75, 97, 134, 174–76 Nintendo Wii, 132 Nokia, 136–37 non-fungible tokens (NFTs), 140, 216–22 avatars and identity, 218, 229, 293–94 fractionalization into fungible tokens/shares, 202 marketplaces, 202, 301 “minting” in, 217 speculation, 128, 220, 231 as useless, 220 nonprofit organizations, 227, 231 Nouns DAO, 229 Nuance Communication, 212 Nvidia, xii, 66, 97–98, 282 Android and, 213 GeForce Now, 131, 282, 286 market capitalization of, 166 NVM software development kit, 175 Omniverse, 136–37, 282 see also graphics processing units (GPUs); Huang, Jensen Nye, David, 242 “Oasis, The,” 22, 289 OBJ file format, 299 occlusion, 97 Oculus VR, xi, 21, 57, 143, 153, 160, 274, 276 Horizon Worlds, 115, 204, 277 Oculus Quest 2, 143, 145–47, 161–62 Population: One, 146–47, 268 Office of Scientific Research and Development (OSRD), ix–x OKCupid, 19, 261 Omniverse from Nvidia, 136–37, 282 OnLive!

New Dark Age: Technology and the End of the Future by James Bridle

AI winter, Airbnb, Alfred Russel Wallace, AlphaGo, Anthropocene, Automated Insights, autonomous vehicles, back-to-the-land, Benoit Mandelbrot, Bernie Sanders, bitcoin, Boeing 747, British Empire, Brownian motion, Buckminster Fuller, Cambridge Analytica, Capital in the Twenty-First Century by Thomas Piketty, carbon footprint, coastline paradox / Richardson effect, cognitive bias, cognitive dissonance, combinatorial explosion, computer vision, congestion charging, cryptocurrency, data is the new oil, disinformation, Donald Trump, Douglas Engelbart, Douglas Engelbart, Douglas Hofstadter, Dr. Strangelove, drone strike, Edward Snowden, Eyjafjallajökull, Fairchild Semiconductor, fake news, fear of failure, Flash crash, fulfillment center, Google Earth, Greyball, Haber-Bosch Process, Higgs boson, hive mind, income inequality, informal economy, Internet of things, Isaac Newton, ITER tokamak, James Bridle, John von Neumann, Julian Assange, Kickstarter, Kim Stanley Robinson, Large Hadron Collider, late capitalism, Laura Poitras, Leo Hollis, lone genius, machine translation, mandelbrot fractal, meta-analysis, Minecraft, mutually assured destruction, natural language processing, Network effects, oil shock, p-value, pattern recognition, peak oil, recommendation engine, road to serfdom, Robert Mercer, Ronald Reagan, security theater, self-driving car, Seymour Hersh, Silicon Valley, Silicon Valley ideology, Skype, social graph, sorting algorithm, South China Sea, speech recognition, Spread Networks laid a new fibre optics cable between New York and Chicago, stem cell, Stuxnet, technoutopianism, the built environment, the scientific method, Uber for X, undersea cable, University of East Anglia, uranium enrichment, Vannevar Bush, warehouse robotics, WikiLeaks

The cloud shapes itself to geographies of power and influence, and it serves to reinforce them. The cloud is a power relationship, and most people are not on top of it. These are valid criticisms, and one way of interrogating the cloud is to look where its shadow falls: to investigate the sites of data centres and undersea cables and see what they tell us about the real disposition of power at work today. We can seed the cloud, condense it, and force it to give up some of its stories. As it fades, certain secrets may be revealed. By understanding the way the figure of the cloud is used to obscure the real operation of technology, we can start to understand the many ways in which technology itself hides its own agency – through opaque machines and inscrutable code, as well as physical distance and legal constructs.

…

6 The class and social conflicts, the historical hierarchies and injustices, that Woolf alludes to in her processions and ceremonies have in no measure abated today, but some of the places to think them may have changed. The crowds that in 1938 lined London’s Lord Mayor’s and coronation parades are now distributed through the network, and the galleries and places of worship have likewise migrated into data centres and undersea cables. We cannot unthink the network; we can only think through and within it. And we can listen to it, when it tries to speak to us in an emergency. Nothing here is an argument against technology: to do so would be to argue against ourselves. Rather, it is an argument for a more thoughtful engagement with technology, coupled with a radically different understanding of what it is possible to think and know about the world.

…

Prometheus – pro-metheus – is foresight, but without the wisdom we might take to accompany it. It’s anticipation. It’s the white heat of scientific and technological discovery, and that desire for the oncoming rush of the future, the head-down drive of forward movement. It’s resource extraction, fossil fuels, undersea cables, server farms, air conditioning, on-demand delivery, giant robots, and meat under pressure. It’s scale and subjugation, the pushing back of the darkness with little thought for what’s beyond – for who already lives there or who gets crushed along the way. The illusion of knowledge and the anticipation of mastery combine to impel the timeline of progress, but they obfuscate the absence of understanding at its articulation point: the zero mark, the dark present, where we see and comprehend nothing beyond movement and efficiency, where our only possible act is to accelerate the existing order.

Krakatoa: The Day the World Exploded by Simon Winchester

Alfred Russel Wallace, British Empire, cable laying ship, company town, Easter island, global village, God and Mammon, Isaac Newton, joint-stock company, lateral thinking, Marshall McLuhan, mass immigration, Maui Hawaii, seminal paper, South China Sea, spice trade, Suez canal 1869, trade route, undersea cable

And yet, as it happens, their observations, painstaking and precise as science demanded, collided head on with a most discomfiting reality: that while in 1883 the world was becoming ever more scientifically advanced, it was in part because of these same advances that its people found themselves in a strangely febrile and delicately balanced condition, which an event like Krakatoa did much to unsettle. The communications technology of the time, for example – the advances of telegraphy, the building of undersea cables, the flourishing of news agencies – ensured that the world's more advanced peoples learned about the eruption within moments of its happening. But at the time the limited, and only slowly unfolding, geological knowledge then to hand did not give the audience an explanation of the events that was sufficient to soothe their fears about all they were learning.

…

But if the operator marked the message Via Eastern, then it went swiftly and securely, beneath the sea. Thirty-five years before, the first insulated cable had been dropped from a ship called the Princess Clementine moored in Folkestone Harbour and connected to a boat two miles away, with a message successfully sent between the two. Since that time the undersea cable had become fixed in the public consciousness. Alfred, Lord Tennyson had written a hymn to the romance of the idea of coded voices hurrying along the ocean floor; and so had Rudyard Kipling, whose brief poem ‘The Deep-Sea Cables’ remains among his best loved: The wrecks dissolve above us; their dust drops down from afar – Down to the dark, to the utter dark, where the blind white sea-snakes are.

…

Here in the womb of the world – here on the tie-ribs of earth Words, and the words of men, flicker and flutter and beat – Warning, sorrow, and gain, salutation and mirth – For a Power troubles the Still that has neither voice nor feet. They have wakened the timeless Things; they have killed their father Time; Joining hands in the gloom, a league from the last of the sun. Hush! Men talk to-day o'er the waste of the ultimate slime, And a new Word runs between: whispering, ‘Let us be one!’ The undersea cable connecting Singapore with London, after passing first by land in those days to Penang, crossed the Bay of Bengal to Madras, hopscotched across India and then plunged into the Arabian Sea on the long passage from Bombay to Aden. It was then put ashore briefly at Port Sudan, nosed up to its receiving stations at Suez and Alexandria, crossed the southern Mediterranean to the island of Malta, travelled through the northern part of the Mediterranean to Gibraltar, between the Pillars of Hercules to Carcavelos on an Atlantic headland ten miles west of Lisbon, up to windswept Vigo on Spain's western The worldwide network of telegraph connections used by Reuter's agency for collecting and disseminating this most valuable commodity – news.

@War: The Rise of the Military-Internet Complex by Shane Harris

air gap, Amazon Web Services, barriers to entry, Berlin Wall, Brian Krebs, centralized clearinghouse, Citizen Lab, clean water, computer age, crowdsourcing, data acquisition, don't be evil, Edward Snowden, end-to-end encryption, failed state, Firefox, information security, John Markoff, Julian Assange, military-industrial complex, mutually assured destruction, peer-to-peer, Silicon Valley, Silicon Valley startup, Skype, Stuxnet, systems thinking, undersea cable, uranium enrichment, WikiLeaks, zero day

In exchange for the federal government granting the company a license to operate in the United States, they had to sign a contract that guaranteed US intelligence agencies uninterrupted access to the networks, so that phone calls could be logged and recorded. One agreement, with Level 3 Communications, even included a “kill switch” provision, stating that if ever directed by the US government, the company must be able to immediately sever all communications traveling through its undersea cables into the United States. This was a protective measure, meant to block the network from delivering malicious software or traffic in the event of a cyber attack. In some cases a foreign communication could be intercepted from inside the United States. (In fact, this kind of capture was routine for e-mail traffic, much of which flowed through United States–based cables and routers.)

…

And with access to the systems of major e-mail and Internet companies, the NSA could gather more intelligence about its adversaries and craft messages that looked trustworthy but were actually loaded with viruses and other malware. The Internet was a battlefield, and the new law gave the NSA more ways to enter it. As the NSA’s powers grew, it cast its net wider, tapping into the undersea cables that carry communications between continents. The agency started filtering the content of all e-mails going in and out of the United States, scanning them for the names, phone numbers, or e-mail addresses of suspected terrorists. And it managed to penetrate the defenses of Google and Yahoo, stealing communications as they traveled between the companies’ overseas private data centers and the public Internet.

…

See Tailored Access Operations Target, [>], [>], [>] TASC, [>]–[>] Team Themis, [>]–[>] telecommunications companies, [>]–[>], [>], [>], [>]–[>], [>], [>], [>]–[>], [>]. See also specific companies telecommunications networks: DOD, [>]; fiber-optic trunk lines, [>], [>], [>]; Google, [>]; kill switch provision, [>]; right to spy in exchange for US license, [>]; Skype, [>]; undersea cables, [>], [>] Telvent, [>]–[>], [>] terrorism. See counterterrorism; 9/11 attacks threat signature, [>], [>], [>], [>], [>], [>]–[>], [>] thumb drives, [>], [>], [>], [>], [>]–[>], [>], [>] Tier [>], [>] tippers, within communications data, [>] Tiversa, [>]–[>] Tor, [>]–[>], [>], [>], [>] Tranche 2 plan, [>]–[>], [>] Transportation Security Administration, [>] transportation system.

An Empire of Wealth: Rise of American Economy Power 1607-2000 by John Steele Gordon

accounting loophole / creative accounting, Alan Greenspan, bank run, banking crisis, Bretton Woods, British Empire, business cycle, buttonwood tree, California gold rush, Charles Babbage, clean water, collective bargaining, Corn Laws, Cornelius Vanderbilt, corporate governance, cotton gin, cuban missile crisis, disintermediation, double entry bookkeeping, failed state, Fairchild Semiconductor, financial independence, flying shuttle, Ford Model T, Frederick Winslow Taylor, full employment, Glass-Steagall Act, global village, Ida Tarbell, imperial preference, industrial research laboratory, informal economy, interchangeable parts, invisible hand, Isaac Newton, it's over 9,000, Jacquard loom, James Hargreaves, James Watt: steam engine, joint-stock company, joint-stock limited liability company, junk bonds, lone genius, Louis Pasteur, low interest rates, margin call, Marshall McLuhan, means of production, megaproject, Menlo Park, Mikhail Gorbachev, Money creation, money market fund, money: store of value / unit of account / medium of exchange, moral hazard, new economy, New Urbanism, postindustrial economy, price mechanism, Ralph Waldo Emerson, RAND corporation, rent control, rent-seeking, reserve currency, rolodex, Ronald Reagan, Savings and loan crisis, spinning jenny, Suez canal 1869, The Wealth of Nations by Adam Smith, three-masted sailing ship, trade route, transaction costs, transcontinental railway, undersea cable, vertical integration, Yom Kippur War

Dickens, born in 1812, the most popular novelist of his time, never mentioned the railroads and telegraph that so characterized the new economic world he lived in. Chapter Nine CHAINING THE LIGHTNING OF HEAVEN THE SPEED OF TRAVEL was not the only thing accelerating in the early nineteenth century. So was the speed of communications. It is hard to imagine today, when satellites and undersea cables keep every part of the globe in instant communication with every other part, just how slowly news spread in the eighteenth century. The battles of Lexington and Concord, the opening events of the American Revolution, occurred on Wednesday, April 19, 1775. But news of the events reached New York only on Sunday, April 23, and Philadelphia on April 24.

…

The next day volume on the New York Stock Exchange reached 1.3 million shares, its highest since the panic of 1907, and many leading stocks fell more than 20 percent. On Friday, July 31, the London Stock Exchange closed for the first time in its history, and New York was the only major exchange scheduled to open the next day. It had no real choice except to close as well. With the world’s markets now tied together with a cat’s cradle of undersea cables, sellers converged on New York, and sell orders began piling up in mountains awaiting the Saturday opening. (The New York Stock Exchange would have a Saturday morning session until after the Second World War.) The governors voted to close the exchange, and the president of the exchange consulted J.

…

Geopositioning satellites have made it possible to determine, with the help of a simple device, one’s location within a few feet. The devices are now appearing in many automobiles, giving directions through the use of synthetic voices, a technology that would have seemed utterly miraculous only a couple of decades ago. Communications satellites, together with an ever-increasing number of undersea cables, have helped greatly to lower the cost of long-distance telephony, leading to an astonishing upsurge in its use. In 1950 about a million overseas phone calls were initiated in the United States. By 1970 the number had grown to twenty-three million; by 1980 to two hundred million. By 2001, as the cost plummeted, the number was 6.3 billion and rising fast.

Tesla: Inventor of the Electrical Age by W. Bernard Carlson

1960s counterculture, air gap, Albert Einstein, Charles Babbage, Clayton Christensen, creative destruction, disruptive innovation, en.wikipedia.org, Ford Model T, Henri Poincaré, invention of radio, Isaac Newton, James Watt: steam engine, Joseph Schumpeter, Menlo Park, packet switching, Plato's cave, popular electronics, Robert Gordon, Ronald Reagan, Steve Jobs, Steve Wozniak, Strategic Defense Initiative, undersea cable, yellow journalism

Unable to make his case fully in person, Tesla followed up each encounter with letters. In the first, Tesla built a case for developing his wireless technology by pointing out that it could compete effectively with the transatlantic telegraph cables. Knowing that Morgan relied on messages sent between his New York and London offices by undersea cable, Tesla reported that he was now able to manipulate electrical pressures of a hundred million volts and hundreds of thousands of horsepower of electrical energy so that it was no longer necessary to rely on those “long and expensive cables” to send messages. Implicitly citing what he had learned by observing the stationary waves set up by thunderstorms in Colorado, Tesla assured Morgan that “Long practical experience with apparatus of this kind and exact measurements embracing a range of nearly seven hundred miles, enable me to construct plants for telegraphic communication across the Atlantic and, if need be, across the Pacific Ocean, with the fullest measure of success.”

…

Marconi outlined the achievements of his system so far—seventy ships equipped with wireless and twenty shore stations in England. He described his experiments in Newfoundland, including the problems he had faced with flying kites in the wintry weather. After mentioning that he hoped that wireless telegraphy would allow for cheaper messages than the existing undersea cables, Marconi concluded by raising a glass and toasting the institute. Marconi’s speech was followed by remarks from Thomson and Professor Michael Pupin of Columbia University. Both men emphasized that although the proof for Marconi’s achievement was limited, they accepted Marconi at his word since they knew and trusted him.

…

., 134, 440n15 dynamos: Anthony and, 103–4; arc lighting and, 41; electricity and, 35, 41–45, 63–71, 81–82, 85–88, 95–96, 103, 110, 115, 120, 168, 304, 320–21, 328, 431n42, 433n23; Mary-Ann, 70; power transmission and, 41–42; Weston and, 81; (see also generators) Earth: as compressible fluid, 362–64; electrical potential of, 269–74; high-frequency phenomena and, 138–42; Kennelly-Heaviside layer and, 209, 335; oscillators and, 138–42; resonance and, 211, 248, 269–74, 329, 363; response to electricity pumped into, 362–64 earthquakes, artificial, 187, 446n26 Eastman, George, 101 Eastman Kodak, 101 economic issues: arc lighting and, 72–73, 76; Astor and, 214, 255–61, 264, 303, 305, 313–15, 353, 361, 371, 417; bankruptcy, 311, 374–76, 399, 446n24; copper costs and, 179, 257–58; creativity and, 401–2; disruptive innovation and, 401–3; funding and, 5–6, 130–31, 206, 255–56, 282, 302–5, 316, 318, 343–45, 352–58, 368–69, 386, 388, 393, 402, 445n6; innovation and, 10, 57–59 (see also innovation); investors and, 102, 128, 130–32, 163, 179, 206–7, 218, 263, 294–95, 305, 310, 332, 344–45, 348–61, 374, 407, 465n63; laboratory fire and, 216–17; licensing and, 94, 101–3, 145, 153, 155, 205–6, 218, 255, 347, 378; Light-House Board and, 283–84; manufacturing and, 339; Morgan deal details and, 357–58; navy and, 282–84; patent-promote-sell strategy and, 102–3, 132, 371, 397; power transmission and, 328 (see also power transmission); price of coal and, 165; royalties and, 94, 101, 111, 113, 130–32, 145, 305, 349, 361, 378, 467n19; Schumpeter and, 10, 57–59, 401, 404, 411; selling patents and, 107–13 (see also patents); Simpson and Crawford and, 256, 264; speculative bubbles and, 346–51; undersea cables and, 336; Vanderbilt and, 214, 230; Wall Street and, 2, 78–79, 130, 164, 206, 304, 311, 345, 349–50, 353, 357, 359; Wardenclyffe tower and, 323, 326; wireless communication and, 350 eddy currents, 53–54 Edison, Thomas Alva, 2; arc lighting and, 71–73, 431n40; Brown and, 78–79; Brush Electric Light Company and, 72; corporate conspiracy and, 398; direct current (DC) and, 4, 165; French lighting organization of, 63–66; General Electric and, 101–2, 129, 160, 162, 166, 224, 311, 317, 352, 357, 373, 397, 452n62; incandescent lighting and, 4, 63–67, 72–73, 102, 140, 206, 219; Johnson and, 197–98; Martin and, 195; Mary-Ann dynamos and, 70; mathematics and, 64; Menlo Park and, 49, 194–95, 198; Nobel Prize and, 375; Orton and, 406; patents and, 49, 63, 72, 161; Peck and, 79; phonograph and, 49, 107, 195, 301; physics and, 375; Puskás and, 49–50; pyromagnetic generator and, 76; reporter coverage of, 194–95; S.

Wall Street Meat by Andy Kessler

accounting loophole / creative accounting, Alan Greenspan, Andy Kessler, automated trading system, banking crisis, Bob Noyce, George Gilder, index fund, Jeff Bezos, John Bogle, junk bonds, market bubble, Mary Meeker, Menlo Park, Michael Milken, Pepto Bismol, pets.com, Robert Metcalfe, rolodex, Salesforce, Sand Hill Road, Silicon Valley, Small Order Execution System, Steve Jobs, technology bubble, undersea cable, Y2K

Jack was generating revenue, and it was increasingly easy to justify his piece of the action. But the ax stuff, as I mentioned, is amazingly intoxicating. It is almost impossible not to get drunk on the glory of moving stocks and of raising billions. Jack became a kingmaker. Gary Winnick had started Global Crossing to sell undersea cable and take market share from AT&T and others. His first undersea cable, Atlantic Crossing 1, cost a few hundred million and was soon generating $100 million a month. Jack was just the guy to help raise billions to duplicate the effort, one more time in the Atlantic and then under the Pacific. · · · The dotcoms’ meltdown started in March and April 2000.

The Four: How Amazon, Apple, Facebook, and Google Divided and Conquered the World by Scott Galloway

"Susan Fowler" uber, activist fund / activist shareholder / activist investor, additive manufacturing, Affordable Care Act / Obamacare, Airbnb, Amazon Robotics, Amazon Web Services, Apple II, autonomous vehicles, barriers to entry, Ben Horowitz, Bernie Sanders, Big Tech, big-box store, Bob Noyce, Brewster Kahle, business intelligence, California gold rush, Cambridge Analytica, cloud computing, Comet Ping Pong, commoditize, cuban missile crisis, David Brooks, Didi Chuxing, digital divide, disintermediation, don't be evil, Donald Trump, Elon Musk, fake news, follow your passion, fulfillment center, future of journalism, future of work, global supply chain, Google Earth, Google Glasses, Google X / Alphabet X, Hacker Conference 1984, Internet Archive, invisible hand, Jeff Bezos, Jony Ive, Khan Academy, Kiva Systems, longitudinal study, Lyft, Mark Zuckerberg, meta-analysis, Network effects, new economy, obamacare, Oculus Rift, offshore financial centre, passive income, Peter Thiel, profit motive, race to the bottom, RAND corporation, ride hailing / ride sharing, risk tolerance, Robert Mercer, Robert Shiller, Search for Extraterrestrial Intelligence, self-driving car, sentiment analysis, shareholder value, Sheryl Sandberg, Silicon Valley, Snapchat, software is eating the world, speech recognition, Stephen Hawking, Steve Ballmer, Steve Bannon, Steve Jobs, Steve Wozniak, Stewart Brand, supercomputer in your pocket, Tesla Model S, the long tail, Tim Cook: Apple, Travis Kalanick, Uber and Lyft, Uber for X, uber lyft, undersea cable, vertical integration, warehouse automation, warehouse robotics, Wayback Machine, Whole Earth Catalog, winner-take-all economy, working poor, you are the product, young professional

“Google’s Next Cloud Product: Google Blimps to Bring Wireless Internet to Africa.” Fortune. June 5, 2013. https://www.forbes.com/sites/matthewstibbe/2013/06/05/googles-next-cloud-product-google-blimps-to-bring-wireless-internet-to-africa/#4439e478449b. 48. Weise, Elizabeth. “Microsoft, Facebook to lay massive undersea cable.” USA Today. May 26, 2016. https://www.usatoday.com/story/experience/2016/05/26/microsoft-facebook-undersea-cable-google-marea-amazon/84984882/. 49. “The Nokia effect.” Economist. August 25, 2012. http://www.economist.com/node/21560867. 50. Downie, Ryan. “Behind Nokia’s 70% Drop in 10 Years (NOK).” Investopedia. September 8, 2016. http://www.investopedia.com/articles/credit-loans-mortgages/090816/behind-nokias-70-drop-10-years-nok.asp.

Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World by Bruce Schneier

23andMe, Airbnb, airport security, AltaVista, Anne Wojcicki, AOL-Time Warner, augmented reality, behavioural economics, Benjamin Mako Hill, Black Swan, Boris Johnson, Brewster Kahle, Brian Krebs, call centre, Cass Sunstein, Chelsea Manning, citizen journalism, Citizen Lab, cloud computing, congestion charging, data science, digital rights, disintermediation, drone strike, Eben Moglen, Edward Snowden, end-to-end encryption, Evgeny Morozov, experimental subject, failed state, fault tolerance, Ferguson, Missouri, Filter Bubble, Firefox, friendly fire, Google Chrome, Google Glasses, heat death of the universe, hindsight bias, informal economy, information security, Internet Archive, Internet of things, Jacob Appelbaum, James Bridle, Jaron Lanier, John Gilmore, John Markoff, Julian Assange, Kevin Kelly, Laura Poitras, license plate recognition, lifelogging, linked data, Lyft, Mark Zuckerberg, moral panic, Nash equilibrium, Nate Silver, national security letter, Network effects, Occupy movement, operational security, Panopticon Jeremy Bentham, payday loans, pre–internet, price discrimination, profit motive, race to the bottom, RAND corporation, real-name policy, recommendation engine, RFID, Ross Ulbricht, satellite internet, self-driving car, Shoshana Zuboff, Silicon Valley, Skype, smart cities, smart grid, Snapchat, social graph, software as a service, South China Sea, sparse data, stealth mode startup, Steven Levy, Stuxnet, TaskRabbit, technological determinism, telemarketer, Tim Cook: Apple, transaction costs, Uber and Lyft, uber lyft, undersea cable, unit 8200, urban planning, Wayback Machine, WikiLeaks, workplace surveillance , Yochai Benkler, yottabyte, zero day

A Russian system was only used for Russian communications. If the NSA tapped an undersea cable between Petropavlovsk and Vladivostok, it didn’t have to worry about accidentally intercepting phone calls between Detroit and Cleveland. The Internet works differently. Everyone’s communications are mixed up on the same networks. Terrorists use the same e-mail providers as everyone else. The same circuits that carry Russian, Iranian, and Cuban government communications could also carry your Twitter feed. Internet phone calls between New York and Los Angeles might end up on Russian undersea cables. Communications between Rio de Janeiro and Lisbon might be routed through Florida.

…

GOVERNMENT HACKS Electronic espionage is different today from what it was in the pre-Internet days of the Cold War. Before the Internet, when surveillance consisted largely of government-on-government espionage, agencies like the NSA would target specific communications circuits: that Soviet undersea cable between Petropavlovsk and Vladivostok, a military communications satellite, a microwave network. This was for the most part passive, requiring large antenna farms in nearby countries. Modern targeted surveillance is likely to involve actively breaking into an adversary’s computer network and installing malicious software designed to take over that network and “exfiltrate” data—that’s NSA talk for stealing it.

Reset by Ronald J. Deibert

23andMe, active measures, air gap, Airbnb, Amazon Web Services, Anthropocene, augmented reality, availability heuristic, behavioural economics, Bellingcat, Big Tech, bitcoin, blockchain, blood diamond, Brexit referendum, Buckminster Fuller, business intelligence, Cal Newport, call centre, Cambridge Analytica, carbon footprint, cashless society, Citizen Lab, clean water, cloud computing, computer vision, confounding variable, contact tracing, contact tracing app, content marketing, coronavirus, corporate social responsibility, COVID-19, crowdsourcing, data acquisition, data is the new oil, decarbonisation, deep learning, deepfake, Deng Xiaoping, disinformation, Donald Trump, Doomsday Clock, dual-use technology, Edward Snowden, Elon Musk, en.wikipedia.org, end-to-end encryption, Evgeny Morozov, failed state, fake news, Future Shock, game design, gig economy, global pandemic, global supply chain, global village, Google Hangouts, Great Leap Forward, high-speed rail, income inequality, information retrieval, information security, Internet of things, Jaron Lanier, Jeff Bezos, John Markoff, Lewis Mumford, liberal capitalism, license plate recognition, lockdown, longitudinal study, Mark Zuckerberg, Marshall McLuhan, mass immigration, megastructure, meta-analysis, military-industrial complex, move fast and break things, Naomi Klein, natural language processing, New Journalism, NSO Group, off-the-grid, Peter Thiel, planetary scale, planned obsolescence, post-truth, proprietary trading, QAnon, ransomware, Robert Mercer, Sheryl Sandberg, Shoshana Zuboff, Silicon Valley, single source of truth, Skype, Snapchat, social distancing, sorting algorithm, source of truth, sovereign wealth fund, sparse data, speech recognition, Steve Bannon, Steve Jobs, Stuxnet, surveillance capitalism, techlash, technological solutionism, the long tail, the medium is the message, The Structural Transformation of the Public Sphere, TikTok, TSMC, undersea cable, unit 8200, Vannevar Bush, WikiLeaks, zero day, zero-sum game

Those devices send electronic signals through radio waves or cables that are transmitted through a physical infrastructure of routers, servers, cell towers, and data farms, in some cases spread throughout multiple countries. Each of these elements is operated by numerous businesses, which can include internet service providers (ISPs), cable companies, cell service providers, satellite services, undersea cable providers, and telecommunications firms as well as the various hardware and software manufacturers supporting them all. Which companies operate the various components of this ecosystem, and according to whose rules, matters enormously for users’ security and privacy. For example, policymakers and analysts have raised concerns that China-based communications routing equipment manufacturers Huawei and ZTE may have designed secret “back doors” in their technology that would provide China’s security agencies with an intelligence toehold in 5G networks.16 However valid, these concerns are not unique to China or China-based companies.

…

A program code-named HAPPYFOOT showed that the NSA and its U.K. partner, the Government Communications Headquarters, were exploiting the location data “leaked” by popular mobile apps.27 To help locate targets of interest, the NSA collected hundreds of thousands of contacts from instant messaging and email applications.28 On and on the stories went, following the same pattern: an astounding digital vacuum-cleaning operation up and down the entire infrastructure of global communications networks owned and operated by the private sector. To be sure, the NSA and its partners did still “tap into” and have “direct access” to a lot of networking equipment and infrastructure. Their hubris was remarkable. By partnering with or otherwise enlisting the cooperation of undersea cable providers, telecommunications companies, and cell service operators, they hoovered up massive volumes of data flowing directly through the arteries of the global communications ecosystem, storing it in giant data warehouses for later fusion (the process of integrating disparate data sources together) and analysis.

The Costs of Connection: How Data Is Colonizing Human Life and Appropriating It for Capitalism by Nick Couldry, Ulises A. Mejias

"World Economic Forum" Davos, 23andMe, Airbnb, Amazon Mechanical Turk, Amazon Web Services, behavioural economics, Big Tech, British Empire, call centre, Cambridge Analytica, Cass Sunstein, choice architecture, cloud computing, colonial rule, computer vision, corporate governance, dark matter, data acquisition, data is the new oil, data science, deep learning, different worldview, digital capitalism, digital divide, discovery of the americas, disinformation, diversification, driverless car, Edward Snowden, emotional labour, en.wikipedia.org, European colonialism, Evgeny Morozov, extractivism, fake news, Gabriella Coleman, gamification, gig economy, global supply chain, Google Chrome, Google Earth, hiring and firing, income inequality, independent contractor, information asymmetry, Infrastructure as a Service, intangible asset, Internet of things, Jaron Lanier, job automation, Kevin Kelly, late capitalism, lifelogging, linked data, machine readable, Marc Andreessen, Mark Zuckerberg, means of production, military-industrial complex, move fast and break things, multi-sided market, Naomi Klein, Network effects, new economy, New Urbanism, PageRank, pattern recognition, payday loans, Philip Mirowski, profit maximization, Ray Kurzweil, RFID, Richard Stallman, Richard Thaler, Salesforce, scientific management, Scientific racism, Second Machine Age, sharing economy, Shoshana Zuboff, side hustle, Sidewalk Labs, Silicon Valley, Slavoj Žižek, smart cities, Snapchat, social graph, social intelligence, software studies, sovereign wealth fund, surveillance capitalism, techlash, The Future of Employment, the scientific method, Thomas Davenport, Tim Cook: Apple, trade liberalization, trade route, undersea cable, urban planning, W. E. B. Du Bois, wages for housework, work culture , workplace surveillance

On a regular day, Netflix and YouTube account for half of all internet traffic in the United States, with massive implications for their ability to “buy” and distribute content for money or otherwise.21 The one-to-many model of mass media dissemination has simply been replaced by a many-to-one model, not the democratic many-to-many model we once hoped for.22 The monopoly-monopsony hybrids we are describing represent massive concentrations of power. These companies currently exercise a hold over the infrastructure that makes social quantification possible; they control everything from undersea cables to satellites to the “last mile” architecture that delivers internet service to individuals. They also control the environments or platforms in which data is being generated, the design and production of the devices through which data is collected, and the computing capacity necessary to analyze the data, including through increasingly sophisticated machine learning methods that do not require explicit human intervention.

…

Cultural Studies 31, nos. 2–3 (2017): 185–204. Mignolo, Walter D. The Darker Side of Western Modernity. Durham, NC: Duke University Press, 2011. . “Delinking: The Rhetoric of Modernity, the Logic of Coloniality and the Grammar of De-Coloniality.” Cultural Studies 21, nos. 2–3 (2007): 449–514. Miller, Ron. “Google’s Latest Undersea Cable Project Will Connect Japan to Australia.” Techcrunch, April 4, 2018. Mirowski, Philip. Machine Dreams. Cambridge, UK: Cambridge University Press, 2002. Mitchell, Stacy. “Amazon Is Trying to Control the Underlying Infrastructure of Our Economy.” Motherboard, June 25, 2017. https://motherboard.vice.com/en_us/article/7xpgvx/amazons-is-trying-to-control-the-underlying-infrastructure-of-our-economy.

The Fifth Domain: Defending Our Country, Our Companies, and Ourselves in the Age of Cyber Threats by Richard A. Clarke, Robert K. Knake

"World Economic Forum" Davos, A Declaration of the Independence of Cyberspace, Affordable Care Act / Obamacare, air gap, Airbnb, Albert Einstein, Amazon Web Services, autonomous vehicles, barriers to entry, bitcoin, Black Lives Matter, Black Swan, blockchain, Boeing 737 MAX, borderless world, Boston Dynamics, business cycle, business intelligence, call centre, Cass Sunstein, cloud computing, cognitive bias, commoditize, computer vision, corporate governance, cryptocurrency, data acquisition, data science, deep learning, DevOps, disinformation, don't be evil, Donald Trump, Dr. Strangelove, driverless car, Edward Snowden, Exxon Valdez, false flag, geopolitical risk, global village, immigration reform, information security, Infrastructure as a Service, Internet of things, Jeff Bezos, John Perry Barlow, Julian Assange, Kubernetes, machine readable, Marc Benioff, Mark Zuckerberg, Metcalfe’s law, MITM: man-in-the-middle, Morris worm, move fast and break things, Network effects, open borders, platform as a service, Ponzi scheme, quantum cryptography, ransomware, Richard Thaler, Salesforce, Sand Hill Road, Schrödinger's Cat, self-driving car, shareholder value, Silicon Valley, Silicon Valley startup, Skype, smart cities, Snapchat, software as a service, Steven Levy, Stuxnet, technoutopianism, The future is already here, Tim Cook: Apple, undersea cable, unit 8200, WikiLeaks, Y2K, zero day

In order to “block” this traffic, the U.S. government would have had to be sitting in between the attacking computers and the target computers. While blocking the attacks sounds appealing, the reality is that the United States has open borders in cyberspace. No agency of the federal government sits at the internet exchange points, where the undersea cables come up onto land, to inspect each packet of internet traffic. Without such a capability, the U.S. government is simply not positioned to block malicious traffic to protect banks or any other companies. Nor should we want such a system to be built. While China has a Great Firewall, a vast system of traffic inspection and interception deployed at the borders of China’s internet and throughout the country, calling the system a firewall suggests, erroneously, that it has value for cybersecurity when it is in fact a tool for censorship and surveillance.

…

, but whereas in the early days those networks were all open and packets of data flowed across them based on the optimal route, now many of those networks have erected barriers around themselves. Still holding it all together is the United States. Most global internet traffic still travels through the United States. An email going from, say, Budapest to Hanoi would head across the Atlantic through undersea cables, traverse the Lower 48 states, and plunge into the Pacific before hitting Japan and being dispersed in the direction of Vietnam. The United States continues to be the loudest voice in the various internet forums for maintaining the global internet. ICANN, the Internet Corporation for Assigned Names and Numbers, which manages the global Domain Name System, is still very much a U.S. entity (though it is no longer controlled by the U.S. government) and it promotes the original U.S. view of the internet.

Building Secure and Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems by Heather Adkins, Betsy Beyer, Paul Blankinship, Ana Oprea, Piotr Lewandowski, Adam Stubblefield

air gap, anti-pattern, barriers to entry, bash_history, behavioural economics, business continuity plan, business logic, business process, Cass Sunstein, cloud computing, cognitive load, continuous integration, correlation does not imply causation, create, read, update, delete, cryptocurrency, cyber-physical system, database schema, Debian, defense in depth, DevOps, Edward Snowden, end-to-end encryption, exponential backoff, fault tolerance, fear of failure, general-purpose programming language, Google Chrome, if you see hoof prints, think horses—not zebras, information security, Internet of things, Kubernetes, load shedding, margin call, microservices, MITM: man-in-the-middle, NSO Group, nudge theory, operational security, performance metric, pull request, ransomware, reproducible builds, revision control, Richard Thaler, risk tolerance, self-driving car, single source of truth, Skype, slashdot, software as a service, source of truth, SQL injection, Stuxnet, the long tail, Turing test, undersea cable, uranium enrichment, Valgrind, web application, Y2K, zero day

When planning your recovery, you should address the one system that’s known to be compromised, but also consider two other factors: whether the other 20 servers are also compromised, and how you will mitigate the effects of the vulnerability for all of these machines in the future. Repeating Risks Imagine that a large ship needs to hold its position at sea. The ship drops anchor and inadvertently hooks onto an undersea cable, which isn’t strong enough and breaks. While the risk of a vessel snagging an undersea cable is small, the effects can be catastrophic, having a huge impact on intercontinental network capacity—and this risk exists for all the undersea cables in the world, not just the one that happened to have been hooked by that one errant ship. A network outage of this type should trigger an analysis of the broader category of risk.

Taming the Sun: Innovations to Harness Solar Energy and Power the Planet by Varun Sivaram

"World Economic Forum" Davos, accelerated depreciation, addicted to oil, Albert Einstein, An Inconvenient Truth, asset light, asset-backed security, autonomous vehicles, bitcoin, blockchain, carbon footprint, carbon tax, clean tech, collateralized debt obligation, Colonization of Mars, currency risk, decarbonisation, deep learning, demand response, disruptive innovation, distributed generation, diversified portfolio, Donald Trump, electricity market, Elon Musk, energy security, energy transition, financial engineering, financial innovation, fixed income, gigafactory, global supply chain, global village, Google Earth, hive mind, hydrogen economy, index fund, Indoor air pollution, Intergovernmental Panel on Climate Change (IPCC), Internet of things, low interest rates, M-Pesa, market clearing, market design, Masayoshi Son, mass immigration, megacity, Michael Shellenberger, mobile money, Negawatt, ocean acidification, off grid, off-the-grid, oil shock, peer-to-peer lending, performance metric, renewable energy transition, Richard Feynman, ride hailing / ride sharing, rolling blackouts, Ronald Reagan, Silicon Valley, Silicon Valley startup, smart grid, smart meter, SoftBank, Solyndra, sovereign wealth fund, Ted Nordhaus, Tesla Model S, time value of money, undersea cable, vertical integration, wikimedia commons

Next, countries will link up regionally, in line both with Masa’s early vision of an Asian supergrid and other ambitious proposals, such as a grid to link sunny North Africa with windy Europe. Then, the grand third stage will bring together the regional supergrids to form a single global supergrid, linked by transoceanic undersea cables and electrical superstations where lines would converge (figure 8.1). Figure 8.1 A global supergrid. This map combines various proposed regional supergrids (for example, in the Asia-Pacific region, Europe and the Middle East/North Africa, and North America). Black lines represent long-distance HVDC electricity transmission lines.

…

Such tensions might prove the biggest obstacle to countries trusting one another enough to willingly increase their dependence on traded energy. Therefore, Liu’s eventual goal of going beyond regional supergrids to achieve a global one is pretty implausible, requiring more international trust, far more money, and further advances in such technologies as undersea cables. Nevertheless, that ultimate vision might function as an inspirational, if perhaps impossible, target toward which even partial progress—such as the establishment of regional supergrids—would constitute a big step forward for the world. Indeed, bigger grids will be essential for tapping the world’s best solar and other renewable resources.

Victorian Internet by Tom Standage

British Empire, Charles Babbage, disinformation, financial independence, global village, invention of the telegraph, invention of the telephone, Jacquard loom, paper trading, pneumatic tube, QWERTY keyboard, technoutopianism, undersea cable

His life's work, the transatlantic cable, was hailed as "the most wonderful achievement of our civilization." The cables were so profitable that Field was able to pay off all his debts in 1867. That year, when one of the two cables got crushed by an iceberg and stopped working, it was repaired within weeks. Before long, the recovery and repair of undersea cables was regarded as commonplace. Another banquet was held for Morse at Delmonico's in New York in December 1868, where he was toasted for having "annihilated both space and time in the transmission of intelligence. The breadth of the Atlantic, with all its waves, is as nothing." Echoing the sentiments expressed on the completion of the 1858 cable, a toast proposed by Edward Thornton, the British ambassador, emphasized the peacemaking potential of the telegraph.

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann

active measures, Amazon Web Services, billion-dollar mistake, bitcoin, blockchain, business intelligence, business logic, business process, c2.com, cloud computing, collaborative editing, commoditize, conceptual framework, cryptocurrency, data science, database schema, deep learning, DevOps, distributed ledger, Donald Knuth, Edward Snowden, end-to-end encryption, Ethereum, ethereum blockchain, exponential backoff, fake news, fault tolerance, finite state, Flash crash, Free Software Foundation, full text search, functional programming, general-purpose programming language, Hacker News, informal economy, information retrieval, Internet of things, iterative process, John von Neumann, Ken Thompson, Kubernetes, Large Hadron Collider, level 1 cache, loose coupling, machine readable, machine translation, Marc Andreessen, microservices, natural language processing, Network effects, no silver bullet, operational security, packet switching, peer-to-peer, performance metric, place-making, premature optimization, recommendation engine, Richard Feynman, self-driving car, semantic web, Shoshana Zuboff, social graph, social web, software as a service, software is eating the world, sorting algorithm, source of truth, SPARQL, speech recognition, SQL injection, statistical model, surveillance capitalism, systematic bias, systems thinking, Tragedy of the Commons, undersea cable, web application, WebSocket, wikimedia commons

Public cloud services such as EC2 are notorious for having frequent transient net‐ work glitches [14], and well-managed private datacenter networks can be stabler environments. Nevertheless, nobody is immune from network problems: for exam‐ ple, a problem during a software upgrade for a switch could trigger a network topol‐ ogy reconfiguration, during which network packets could be delayed for more than a minute [17]. Sharks might bite undersea cables and damage them [18]. Other surpris‐ ing faults include a network interface that sometimes drops all inbound packets but sends outbound packets successfully [19]: just because a network link works in one direction doesn’t guarantee it’s also working in the opposite direction. Network partitions When one part of the network is cut off from the rest due to a net‐ work fault, that is sometimes called a network partition or netsplit.

…

The opposite of bounded. 558 | Glossary Index A aborts (transactions), 222, 224 in two-phase commit, 356 performance of optimistic concurrency con‐ trol, 266 retrying aborted transactions, 231 abstraction, 21, 27, 222, 266, 321 access path (in network model), 37, 60 accidental complexity, removing, 21 accountability, 535 ACID properties (transactions), 90, 223 atomicity, 223, 228 consistency, 224, 529 durability, 226 isolation, 225, 228 acknowledgements (messaging), 445 active/active replication (see multi-leader repli‐ cation) active/passive replication (see leader-based rep‐ lication) ActiveMQ (messaging), 137, 444 distributed transaction support, 361 ActiveRecord (object-relational mapper), 30, 232 actor model, 138 (see also message-passing) comparison to Pregel model, 425 comparison to stream processing, 468 Advanced Message Queuing Protocol (see AMQP) aerospace systems, 6, 10, 305, 372 aggregation data cubes and materialized views, 101 in batch processes, 406 in stream processes, 466 aggregation pipeline query language, 48 Agile, 22 minimizing irreversibility, 414, 497 moving faster with confidence, 532 Unix philosophy, 394 agreement, 365 (see also consensus) Airflow (workflow scheduler), 402 Ajax, 131 Akka (actor framework), 139 algorithms algorithm correctness, 308 B-trees, 79-83 for distributed systems, 306 hash indexes, 72-75 mergesort, 76, 402, 405 red-black trees, 78 SSTables and LSM-trees, 76-79 all-to-all replication topologies, 175 AllegroGraph (database), 50 ALTER TABLE statement (SQL), 40, 111 Amazon Dynamo (database), 177 Amazon Web Services (AWS), 8 Kinesis Streams (messaging), 448 network reliability, 279 postmortems, 9 RedShift (database), 93 S3 (object storage), 398 checking data integrity, 530 amplification of bias, 534 of failures, 364, 495 Index | 559 of tail latency, 16, 207 write amplification, 84 AMQP (Advanced Message Queuing Protocol), 444 (see also messaging systems) comparison to log-based messaging, 448, 451 message ordering, 446 analytics, 90 comparison to transaction processing, 91 data warehousing (see data warehousing) parallel query execution in MPP databases, 415 predictive (see predictive analytics) relation to batch processing, 411 schemas for, 93-95 snapshot isolation for queries, 238 stream analytics, 466 using MapReduce, analysis of user activity events (example), 404 anti-caching (in-memory databases), 89 anti-entropy, 178 Apache ActiveMQ (see ActiveMQ) Apache Avro (see Avro) Apache Beam (see Beam) Apache BookKeeper (see BookKeeper) Apache Cassandra (see Cassandra) Apache CouchDB (see CouchDB) Apache Curator (see Curator) Apache Drill (see Drill) Apache Flink (see Flink) Apache Giraph (see Giraph) Apache Hadoop (see Hadoop) Apache HAWQ (see HAWQ) Apache HBase (see HBase) Apache Helix (see Helix) Apache Hive (see Hive) Apache Impala (see Impala) Apache Jena (see Jena) Apache Kafka (see Kafka) Apache Lucene (see Lucene) Apache MADlib (see MADlib) Apache Mahout (see Mahout) Apache Oozie (see Oozie) Apache Parquet (see Parquet) Apache Qpid (see Qpid) Apache Samza (see Samza) Apache Solr (see Solr) Apache Spark (see Spark) 560 | Index Apache Storm (see Storm) Apache Tajo (see Tajo) Apache Tez (see Tez) Apache Thrift (see Thrift) Apache ZooKeeper (see ZooKeeper) Apama (stream analytics), 466 append-only B-trees, 82, 242 append-only files (see logs) Application Programming Interfaces (APIs), 5, 27 for batch processing, 403 for change streams, 456 for distributed transactions, 361 for graph processing, 425 for services, 131-136 (see also services) evolvability, 136 RESTful, 133 SOAP, 133 application state (see state) approximate search (see similarity search) archival storage, data from databases, 131 arcs (see edges) arithmetic mean, 14 ASCII text, 119, 395 ASN.1 (schema language), 127 asynchronous networks, 278, 553 comparison to synchronous networks, 284 formal model, 307 asynchronous replication, 154, 553 conflict detection, 172 data loss on failover, 157 reads from asynchronous follower, 162 Asynchronous Transfer Mode (ATM), 285 atomic broadcast (see total order broadcast) atomic clocks (caesium clocks), 294, 295 (see also clocks) atomicity (concurrency), 553 atomic increment-and-get, 351 compare-and-set, 245, 327 (see also compare-and-set operations) replicated operations, 246 write operations, 243 atomicity (transactions), 223, 228, 553 atomic commit, 353 avoiding, 523, 528 blocking and nonblocking, 359 in stream processing, 360, 477 maintaining derived data, 453 for multi-object transactions, 229 for single-object writes, 230 auditability, 528-533 designing for, 531 self-auditing systems, 530 through immutability, 460 tools for auditable data systems, 532 availability, 8 (see also fault tolerance) in CAP theorem, 337 in service level agreements (SLAs), 15 Avro (data format), 122-127 code generation, 127 dynamically generated schemas, 126 object container files, 125, 131, 414 reader determining writer’s schema, 125 schema evolution, 123 use in Hadoop, 414 awk (Unix tool), 391 AWS (see Amazon Web Services) Azure (see Microsoft) B B-trees (indexes), 79-83 append-only/copy-on-write variants, 82, 242 branching factor, 81 comparison to LSM-trees, 83-85 crash recovery, 82 growing by splitting a page, 81 optimizations, 82 similarity to dynamic partitioning, 212 backpressure, 441, 553 in TCP, 282 backups database snapshot for replication, 156 integrity of, 530 snapshot isolation for, 238 use for ETL processes, 405 backward compatibility, 112 BASE, contrast to ACID, 223 bash shell (Unix), 70, 395, 503 batch processing, 28, 389-431, 553 combining with stream processing lambda architecture, 497 unifying technologies, 498 comparison to MPP databases, 414-418 comparison to stream processing, 464 comparison to Unix, 413-414 dataflow engines, 421-423 fault tolerance, 406, 414, 422, 442 for data integration, 494-498 graphs and iterative processing, 424-426 high-level APIs and languages, 403, 426-429 log-based messaging and, 451 maintaining derived state, 495 MapReduce and distributed filesystems, 397-413 (see also MapReduce) measuring performance, 13, 390 outputs, 411-413 key-value stores, 412 search indexes, 411 using Unix tools (example), 391-394 Bayou (database), 522 Beam (dataflow library), 498 bias, 534 big ball of mud, 20 Bigtable data model, 41, 99 binary data encodings, 115-128 Avro, 122-127 MessagePack, 116-117 Thrift and Protocol Buffers, 117-121 binary encoding based on schemas, 127 by network drivers, 128 binary strings, lack of support in JSON and XML, 114 BinaryProtocol encoding (Thrift), 118 Bitcask (storage engine), 72 crash recovery, 74 Bitcoin (cryptocurrency), 532 Byzantine fault tolerance, 305 concurrency bugs in exchanges, 233 bitmap indexes, 97 blockchains, 532 Byzantine fault tolerance, 305 blocking atomic commit, 359 Bloom (programming language), 504 Bloom filter (algorithm), 79, 466 BookKeeper (replicated log), 372 Bottled Water (change data capture), 455 bounded datasets, 430, 439, 553 (see also batch processing) bounded delays, 553 in networks, 285 process pauses, 298 broadcast hash joins, 409 Index | 561 brokerless messaging, 442 Brubeck (metrics aggregator), 442 BTM (transaction coordinator), 356 bulk synchronous parallel (BSP) model, 425 bursty network traffic patterns, 285 business data processing, 28, 90, 390 byte sequence, encoding data in, 112 Byzantine faults, 304-306, 307, 553 Byzantine fault-tolerant systems, 305, 532 Byzantine Generals Problem, 304 consensus algorithms and, 366 C caches, 89, 553 and materialized views, 101 as derived data, 386, 499-504 database as cache of transaction log, 460 in CPUs, 99, 338, 428 invalidation and maintenance, 452, 467 linearizability, 324 CAP theorem, 336-338, 554 Cascading (batch processing), 419, 427 hash joins, 409 workflows, 403 cascading failures, 9, 214, 281 Cascalog (batch processing), 60 Cassandra (database) column-family data model, 41, 99 compaction strategy, 79 compound primary key, 204 gossip protocol, 216 hash partitioning, 203-205 last-write-wins conflict resolution, 186, 292 leaderless replication, 177 linearizability, lack of, 335 log-structured storage, 78 multi-datacenter support, 184 partitioning scheme, 213 secondary indexes, 207 sloppy quorums, 184 cat (Unix tool), 391 causal context, 191 (see also causal dependencies) causal dependencies, 186-191 capturing, 191, 342, 494, 514 by total ordering, 493 causal ordering, 339 in transactions, 262 sending message to friends (example), 494 562 | Index causality, 554 causal ordering, 339-343 linearizability and, 342 total order consistent with, 344, 345 consistency with, 344-347 consistent snapshots, 340 happens-before relationship, 186 in serializable transactions, 262-265 mismatch with clocks, 292 ordering events to capture, 493 violations of, 165, 176, 292, 340 with synchronized clocks, 294 CEP (see complex event processing) certificate transparency, 532 chain replication, 155 linearizable reads, 351 change data capture, 160, 454 API support for change streams, 456 comparison to event sourcing, 457 implementing, 454 initial snapshot, 455 log compaction, 456 changelogs, 460 change data capture, 454 for operator state, 479 generating with triggers, 455 in stream joins, 474 log compaction, 456 maintaining derived state, 452 Chaos Monkey, 7, 280 checkpointing in batch processors, 422, 426 in high-performance computing, 275 in stream processors, 477, 523 chronicle data model, 458 circuit-switched networks, 284 circular buffers, 450 circular replication topologies, 175 clickstream data, analysis of, 404 clients calling services, 131 pushing state changes to, 512 request routing, 214 stateful and offline-capable, 170, 511 clocks, 287-299 atomic (caesium) clocks, 294, 295 confidence interval, 293-295 for global snapshots, 294 logical (see logical clocks) skew, 291-294, 334 slewing, 289 synchronization and accuracy, 289-291 synchronization using GPS, 287, 290, 294, 295 time-of-day versus monotonic clocks, 288 timestamping events, 471 cloud computing, 146, 275 need for service discovery, 372 network glitches, 279 shared resources, 284 single-machine reliability, 8 Cloudera Impala (see Impala) clustered indexes, 86 CODASYL model, 36 (see also network model) code generation with Avro, 127 with Thrift and Protocol Buffers, 118 with WSDL, 133 collaborative editing multi-leader replication and, 170 column families (Bigtable), 41, 99 column-oriented storage, 95-101 column compression, 97 distinction between column families and, 99 in batch processors, 428 Parquet, 96, 131, 414 sort order in, 99-100 vectorized processing, 99, 428 writing to, 101 comma-separated values (see CSV) command query responsibility segregation (CQRS), 462 commands (event sourcing), 459 commits (transactions), 222 atomic commit, 354-355 (see also atomicity; transactions) read committed isolation, 234 three-phase commit (3PC), 359 two-phase commit (2PC), 355-359 commutative operations, 246 compaction of changelogs, 456 (see also log compaction) for stream operator state, 479 of log-structured storage, 73 issues with, 84 size-tiered and leveled approaches, 79 CompactProtocol encoding (Thrift), 119 compare-and-set operations, 245, 327 implementing locks, 370 implementing uniqueness constraints, 331 implementing with total order broadcast, 350 relation to consensus, 335, 350, 352, 374 relation to transactions, 230 compatibility, 112, 128 calling services, 136 properties of encoding formats, 139 using databases, 129-131 using message-passing, 138 compensating transactions, 355, 461, 526 complex event processing (CEP), 465 complexity distilling in theoretical models, 310 hiding using abstraction, 27 of software systems, managing, 20 composing data systems (see unbundling data‐ bases) compute-intensive applications, 3, 275 concatenated indexes, 87 in Cassandra, 204 Concord (stream processor), 466 concurrency actor programming model, 138, 468 (see also message-passing) bugs from weak transaction isolation, 233 conflict resolution, 171, 174 detecting concurrent writes, 184-191 dual writes, problems with, 453 happens-before relationship, 186 in replicated systems, 161-191, 324-338 lost updates, 243 multi-version concurrency control (MVCC), 239 optimistic concurrency control, 261 ordering of operations, 326, 341 reducing, through event logs, 351, 462, 507 time and relativity, 187 transaction isolation, 225 write skew (transaction isolation), 246-251 conflict-free replicated datatypes (CRDTs), 174 conflicts conflict detection, 172 causal dependencies, 186, 342 in consensus algorithms, 368 in leaderless replication, 184 Index | 563 in log-based systems, 351, 521 in nonlinearizable systems, 343 in serializable snapshot isolation (SSI), 264 in two-phase commit, 357, 364 conflict resolution automatic conflict resolution, 174 by aborting transactions, 261 by apologizing, 527 convergence, 172-174 in leaderless systems, 190 last write wins (LWW), 186, 292 using atomic operations, 246 using custom logic, 173 determining what is a conflict, 174, 522 in multi-leader replication, 171-175 avoiding conflicts, 172 lost updates, 242-246 materializing, 251 relation to operation ordering, 339 write skew (transaction isolation), 246-251 congestion (networks) avoidance, 282 limiting accuracy of clocks, 293 queueing delays, 282 consensus, 321, 364-375, 554 algorithms, 366-368 preventing split brain, 367 safety and liveness properties, 365 using linearizable operations, 351 cost of, 369 distributed transactions, 352-375 in practice, 360-364 two-phase commit, 354-359 XA transactions, 361-364 impossibility of, 353 membership and coordination services, 370-373 relation to compare-and-set, 335, 350, 352, 374 relation to replication, 155, 349 relation to uniqueness constraints, 521 consistency, 224, 524 across different databases, 157, 452, 462, 492 causal, 339-348, 493 consistent prefix reads, 165-167 consistent snapshots, 156, 237-242, 294, 455, 500 (see also snapshots) 564 | Index crash recovery, 82 enforcing constraints (see constraints) eventual, 162, 322 (see also eventual consistency) in ACID transactions, 224, 529 in CAP theorem, 337 linearizability, 324-338 meanings of, 224 monotonic reads, 164-165 of secondary indexes, 231, 241, 354, 491, 500 ordering guarantees, 339-352 read-after-write, 162-164 sequential, 351 strong (see linearizability) timeliness and integrity, 524 using quorums, 181, 334 consistent hashing, 204 consistent prefix reads, 165 constraints (databases), 225, 248 asynchronously checked, 526 coordination avoidance, 527 ensuring idempotence, 519 in log-based systems, 521-524 across multiple partitions, 522 in two-phase commit, 355, 357 relation to consensus, 374, 521 relation to event ordering, 347 requiring linearizability, 330 Consul (service discovery), 372 consumers (message streams), 137, 440 backpressure, 441 consumer offsets in logs, 449 failures, 445, 449 fan-out, 11, 445, 448 load balancing, 444, 448 not keeping up with producers, 441, 450, 502 context switches, 14, 297 convergence (conflict resolution), 172-174, 322 coordination avoidance, 527 cross-datacenter, 168, 493 cross-partition ordering, 256, 294, 348, 523 services, 330, 370-373 coordinator (in 2PC), 356 failure, 358 in XA transactions, 361-364 recovery, 363 copy-on-write (B-trees), 82, 242 CORBA (Common Object Request Broker Architecture), 134 correctness, 6 auditability, 528-533 Byzantine fault tolerance, 305, 532 dealing with partial failures, 274 in log-based systems, 521-524 of algorithm within system model, 308 of compensating transactions, 355 of consensus, 368 of derived data, 497, 531 of immutable data, 461 of personal data, 535, 540 of time, 176, 289-295 of transactions, 225, 515, 529 timeliness and integrity, 524-528 corruption of data detecting, 519, 530-533 due to pathological memory access, 529 due to radiation, 305 due to split brain, 158, 302 due to weak transaction isolation, 233 formalization in consensus, 366 integrity as absence of, 524 network packets, 306 on disks, 227 preventing using write-ahead logs, 82 recovering from, 414, 460 Couchbase (database) durability, 89 hash partitioning, 203-204, 211 rebalancing, 213 request routing, 216 CouchDB (database) B-tree storage, 242 change feed, 456 document data model, 31 join support, 34 MapReduce support, 46, 400 replication, 170, 173 covering indexes, 86 CPUs cache coherence and memory barriers, 338 caching and pipelining, 99, 428 increasing parallelism, 43 CRDTs (see conflict-free replicated datatypes) CREATE INDEX statement (SQL), 85, 500 credit rating agencies, 535 Crunch (batch processing), 419, 427 hash joins, 409 sharded joins, 408 workflows, 403 cryptography defense against attackers, 306 end-to-end encryption and authentication, 519, 543 proving integrity of data, 532 CSS (Cascading Style Sheets), 44 CSV (comma-separated values), 70, 114, 396 Curator (ZooKeeper recipes), 330, 371 curl (Unix tool), 135, 397 cursor stability, 243 Cypher (query language), 52 comparison to SPARQL, 59 D data corruption (see corruption of data) data cubes, 102 data formats (see encoding) data integration, 490-498, 543 batch and stream processing, 494-498 lambda architecture, 497 maintaining derived state, 495 reprocessing data, 496 unifying, 498 by unbundling databases, 499-515 comparison to federated databases, 501 combining tools by deriving data, 490-494 derived data versus distributed transac‐ tions, 492 limits of total ordering, 493 ordering events to capture causality, 493 reasoning about dataflows, 491 need for, 385 data lakes, 415 data locality (see locality) data models, 27-64 graph-like models, 49-63 Datalog language, 60-63 property graphs, 50 RDF and triple-stores, 55-59 query languages, 42-48 relational model versus document model, 28-42 data protection regulations, 542 data systems, 3 about, 4 Index | 565 concerns when designing, 5 future of, 489-544 correctness, constraints, and integrity, 515-533 data integration, 490-498 unbundling databases, 499-515 heterogeneous, keeping in sync, 452 maintainability, 18-22 possible faults in, 221 reliability, 6-10 hardware faults, 7 human errors, 9 importance of, 10 software errors, 8 scalability, 10-18 unreliable clocks, 287-299 data warehousing, 91-95, 554 comparison to data lakes, 415 ETL (extract-transform-load), 92, 416, 452 keeping data systems in sync, 452 schema design, 93 slowly changing dimension (SCD), 476 data-intensive applications, 3 database triggers (see triggers) database-internal distributed transactions, 360, 364, 477 databases archival storage, 131 comparison of message brokers to, 443 dataflow through, 129 end-to-end argument for, 519-520 checking integrity, 531 inside-out, 504 (see also unbundling databases) output from batch workflows, 412 relation to event streams, 451-464 (see also changelogs) API support for change streams, 456, 506 change data capture, 454-457 event sourcing, 457-459 keeping systems in sync, 452-453 philosophy of immutable events, 459-464 unbundling, 499-515 composing data storage technologies, 499-504 designing applications around dataflow, 504-509 566 | Index observing derived state, 509-515 datacenters geographically distributed, 145, 164, 278, 493 multi-tenancy and shared resources, 284 network architecture, 276 network faults, 279 replication across multiple, 169 leaderless replication, 184 multi-leader replication, 168, 335 dataflow, 128-139, 504-509 correctness of dataflow systems, 525 differential, 504 message-passing, 136-139 reasoning about, 491 through databases, 129 through services, 131-136 dataflow engines, 421-423 comparison to stream processing, 464 directed acyclic graphs (DAG), 424 partitioning, approach to, 429 support for declarative queries, 427 Datalog (query language), 60-63 datatypes binary strings in XML and JSON, 114 conflict-free, 174 in Avro encodings, 122 in Thrift and Protocol Buffers, 121 numbers in XML and JSON, 114 Datomic (database) B-tree storage, 242 data model, 50, 57 Datalog query language, 60 excision (deleting data), 463 languages for transactions, 255 serial execution of transactions, 253 deadlocks detection, in two-phase commit (2PC), 364 in two-phase locking (2PL), 258 Debezium (change data capture), 455 declarative languages, 42, 554 Bloom, 504 CSS and XSL, 44 Cypher, 52 Datalog, 60 for batch processing, 427 recursive SQL queries, 53 relational algebra and SQL, 42 SPARQL, 59 delays bounded network delays, 285 bounded process pauses, 298 unbounded network delays, 282 unbounded process pauses, 296 deleting data, 463 denormalization (data representation), 34, 554 costs, 39 in derived data systems, 386 materialized views, 101 updating derived data, 228, 231, 490 versus normalization, 462 derived data, 386, 439, 554 from change data capture, 454 in event sourcing, 458-458 maintaining derived state through logs, 452-457, 459-463 observing, by subscribing to streams, 512 outputs of batch and stream processing, 495 through application code, 505 versus distributed transactions, 492 deterministic operations, 255, 274, 554 accidental nondeterminism, 423 and fault tolerance, 423, 426 and idempotence, 478, 492 computing derived data, 495, 526, 531 in state machine replication, 349, 452, 458 joins, 476 DevOps, 394 differential dataflow, 504 dimension tables, 94 dimensional modeling (see star schemas) directed acyclic graphs (DAGs), 424 dirty reads (transaction isolation), 234 dirty writes (transaction isolation), 235 discrimination, 534 disks (see hard disks) distributed actor frameworks, 138 distributed filesystems, 398-399 decoupling from query engines, 417 indiscriminately dumping data into, 415 use by MapReduce, 402 distributed systems, 273-312, 554 Byzantine faults, 304-306 cloud versus supercomputing, 275 detecting network faults, 280 faults and partial failures, 274-277 formalization of consensus, 365 impossibility results, 338, 353 issues with failover, 157 limitations of distributed transactions, 363 multi-datacenter, 169, 335 network problems, 277-286 quorums, relying on, 301 reasons for using, 145, 151 synchronized clocks, relying on, 291-295 system models, 306-310 use of clocks and time, 287 distributed transactions (see transactions) Django (web framework), 232 DNS (Domain Name System), 216, 372 Docker (container manager), 506 document data model, 30-42 comparison to relational model, 38-42 document references, 38, 403 document-oriented databases, 31 many-to-many relationships and joins, 36 multi-object transactions, need for, 231 versus relational model convergence of models, 41 data locality, 41 document-partitioned indexes, 206, 217, 411 domain-driven design (DDD), 457 DRBD (Distributed Replicated Block Device), 153 drift (clocks), 289 Drill (query engine), 93 Druid (database), 461 Dryad (dataflow engine), 421 dual writes, problems with, 452, 507 duplicates, suppression of, 517 (see also idempotence) using a unique ID, 518, 522 durability (transactions), 226, 554 duration (time), 287 measurement with monotonic clocks, 288 dynamic partitioning, 212 dynamically typed languages analogy to schema-on-read, 40 code generation and, 127 Dynamo-style databases (see leaderless replica‐ tion) E edges (in graphs), 49, 403 property graph model, 50 edit distance (full-text search), 88 effectively-once semantics, 476, 516 Index | 567 (see also exactly-once semantics) preservation of integrity, 525 elastic systems, 17 Elasticsearch (search server) document-partitioned indexes, 207 partition rebalancing, 211 percolator (stream search), 467 usage example, 4 use of Lucene, 79 ElephantDB (database), 413 Elm (programming language), 504, 512 encodings (data formats), 111-128 Avro, 122-127 binary variants of JSON and XML, 115 compatibility, 112 calling services, 136 using databases, 129-131 using message-passing, 138 defined, 113 JSON, XML, and CSV, 114 language-specific formats, 113 merits of schemas, 127 representations of data, 112 Thrift and Protocol Buffers, 117-121 end-to-end argument, 277, 519-520 checking integrity, 531 publish/subscribe streams, 512 enrichment (stream), 473 Enterprise JavaBeans (EJB), 134 entities (see vertices) epoch (consensus algorithms), 368 epoch (Unix timestamps), 288 equi-joins, 403 erasure coding (error correction), 398 Erlang OTP (actor framework), 139 error handling for network faults, 280 in transactions, 231 error-correcting codes, 277, 398 Esper (CEP engine), 466 etcd (coordination service), 370-373 linearizable operations, 333 locks and leader election, 330 quorum reads, 351 service discovery, 372 use of Raft algorithm, 349, 353 Ethereum (blockchain), 532 Ethernet (networks), 276, 278, 285 packet checksums, 306, 519 568 | Index Etherpad (collaborative editor), 170 ethics, 533-543 code of ethics and professional practice, 533 legislation and self-regulation, 542 predictive analytics, 533-536 amplifying bias, 534 feedback loops, 536 privacy and tracking, 536-543 consent and freedom of choice, 538 data as assets and power, 540 meaning of privacy, 539 surveillance, 537 respect, dignity, and agency, 543, 544 unintended consequences, 533, 536 ETL (extract-transform-load), 92, 405, 452, 554 use of Hadoop for, 416 event sourcing, 457-459 commands and events, 459 comparison to change data capture, 457 comparison to lambda architecture, 497 deriving current state from event log, 458 immutability and auditability, 459, 531 large, reliable data systems, 519, 526 Event Store (database), 458 event streams (see streams) events, 440 deciding on total order of, 493 deriving views from event log, 461 difference to commands, 459 event time versus processing time, 469, 477, 498 immutable, advantages of, 460, 531 ordering to capture causality, 493 reads as, 513 stragglers, 470, 498 timestamp of, in stream processing, 471 EventSource (browser API), 512 eventual consistency, 152, 162, 308, 322 (see also conflicts) and perpetual inconsistency, 525 evolvability, 21, 111 calling services, 136 graph-structured data, 52 of databases, 40, 129-131, 461, 497 of message-passing, 138 reprocessing data, 496, 498 schema evolution in Avro, 123 schema evolution in Thrift and Protocol Buffers, 120 schema-on-read, 39, 111, 128 exactly-once semantics, 360, 476, 516 parity with batch processors, 498 preservation of integrity, 525 exclusive mode (locks), 258 eXtended Architecture transactions (see XA transactions) extract-transform-load (see ETL) F Facebook Presto (query engine), 93 React, Flux, and Redux (user interface libra‐ ries), 512 social graphs, 49 Wormhole (change data capture), 455 fact tables, 93 failover, 157, 554 (see also leader-based replication) in leaderless replication, absence of, 178 leader election, 301, 348, 352 potential problems, 157 failures amplification by distributed transactions, 364, 495 failure detection, 280 automatic rebalancing causing cascading failures, 214 perfect failure detectors, 359 timeouts and unbounded delays, 282, 284 using ZooKeeper, 371 faults versus, 7 partial failures in distributed systems, 275-277, 310 fan-out (messaging systems), 11, 445 fault tolerance, 6-10, 555 abstractions for, 321 formalization in consensus, 365-369 use of replication, 367 human fault tolerance, 414 in batch processing, 406, 414, 422, 425 in log-based systems, 520, 524-526 in stream processing, 476-479 atomic commit, 477 idempotence, 478 maintaining derived state, 495 microbatching and checkpointing, 477 rebuilding state after a failure, 478 of distributed transactions, 362-364 transaction atomicity, 223, 354-361 faults, 6 Byzantine faults, 304-306 failures versus, 7 handled by transactions, 221 handling in supercomputers and cloud computing, 275 hardware, 7 in batch processing versus distributed data‐ bases, 417 in distributed systems, 274-277 introducing deliberately, 7, 280 network faults, 279-281 asymmetric faults, 300 detecting, 280 tolerance of, in multi-leader replication, 169 software errors, 8 tolerating (see fault tolerance) federated databases, 501 fence (CPU instruction), 338 fencing (preventing split brain), 158, 302-304 generating fencing tokens, 349, 370 properties of fencing tokens, 308 stream processors writing to databases, 478, 517 Fibre Channel (networks), 398 field tags (Thrift and Protocol Buffers), 119-121 file descriptors (Unix), 395 financial data, 460 Firebase (database), 456 Flink (processing framework), 421-423 dataflow APIs, 427 fault tolerance, 422, 477, 479 Gelly API (graph processing), 425 integration of batch and stream processing, 495, 498 machine learning, 428 query optimizer, 427 stream processing, 466 flow control, 282, 441, 555 FLP result (on consensus), 353 FlumeJava (dataflow library), 403, 427 followers, 152, 555 (see also leader-based replication) foreign keys, 38, 403 forward compatibility, 112 forward decay (algorithm), 16 Index | 569 Fossil (version control system), 463 shunning (deleting data), 463 FoundationDB (database) serializable transactions, 261, 265, 364 fractal trees, 83 full table scans, 403 full-text search, 555 and fuzzy indexes, 88 building search indexes, 411 Lucene storage engine, 79 functional reactive programming (FRP), 504 functional requirements, 22 futures (asynchronous operations), 135 fuzzy search (see similarity search) G garbage collection immutability and, 463 process pauses for, 14, 296-299, 301 (see also process pauses) genome analysis, 63, 429 geographically distributed datacenters, 145, 164, 278, 493 geospatial indexes, 87 Giraph (graph processing), 425 Git (version control system), 174, 342, 463 GitHub, postmortems, 157, 158, 309 global indexes (see term-partitioned indexes) GlusterFS (distributed filesystem), 398 GNU Coreutils (Linux), 394 GoldenGate (change data capture), 161, 170, 455 (see also Oracle) Google Bigtable (database) data model (see Bigtable data model) partitioning scheme, 199, 202 storage layout, 78 Chubby (lock service), 370 Cloud Dataflow (stream processor), 466, 477, 498 (see also Beam) Cloud Pub/Sub (messaging), 444, 448 Docs (collaborative editor), 170 Dremel (query engine), 93, 96 FlumeJava (dataflow library), 403, 427 GFS (distributed file system), 398 gRPC (RPC framework), 135 MapReduce (batch processing), 390 570 | Index (see also MapReduce) building search indexes, 411 task preemption, 418 Pregel (graph processing), 425 Spanner (see Spanner) TrueTime (clock API), 294 gossip protocol, 216 government use of data, 541 GPS (Global Positioning System) use for clock synchronization, 287, 290, 294, 295 GraphChi (graph processing), 426 graphs, 555 as data models, 49-63 example of graph-structured data, 49 property graphs, 50 RDF and triple-stores, 55-59 versus the network model, 60 processing and analysis, 424-426 fault tolerance, 425 Pregel processing model, 425 query languages Cypher, 52 Datalog, 60-63 recursive SQL queries, 53 SPARQL, 59-59 Gremlin (graph query language), 50 grep (Unix tool), 392 GROUP BY clause (SQL), 406 grouping records in MapReduce, 406 handling skew, 407 H Hadoop (data infrastructure) comparison to distributed databases, 390 comparison to MPP databases, 414-418 comparison to Unix, 413-414, 499 diverse processing models in ecosystem, 417 HDFS distributed filesystem (see HDFS) higher-level tools, 403 join algorithms, 403-410 (see also MapReduce) MapReduce (see MapReduce) YARN (see YARN) happens-before relationship, 340 capturing, 187 concurrency and, 186 hard disks access patterns, 84 detecting corruption, 519, 530 faults in, 7, 227 sequential write throughput, 75, 450 hardware faults, 7 hash indexes, 72-75 broadcast hash joins, 409 partitioned hash joins, 409 hash partitioning, 203-205, 217 consistent hashing, 204 problems with hash mod N, 210 range queries, 204 suitable hash functions, 203 with fixed number of partitions, 210 HAWQ (database), 428 HBase (database) bug due to lack of fencing, 302 bulk loading, 413 column-family data model, 41, 99 dynamic partitioning, 212 key-range partitioning, 202 log-structured storage, 78 request routing, 216 size-tiered compaction, 79 use of HDFS, 417 use of ZooKeeper, 370 HDFS (Hadoop Distributed File System), 398-399 (see also distributed filesystems) checking data integrity, 530 decoupling from query engines, 417 indiscriminately dumping data into, 415 metadata about datasets, 410 NameNode, 398 use by Flink, 479 use by HBase, 212 use by MapReduce, 402 HdrHistogram (numerical library), 16 head (Unix tool), 392 head vertex (property graphs), 51 head-of-line blocking, 15 heap files (databases), 86 Helix (cluster manager), 216 heterogeneous distributed transactions, 360, 364 heuristic decisions (in 2PC), 363 Hibernate (object-relational mapper), 30 hierarchical model, 36 high availability (see fault tolerance) high-frequency trading, 290, 299 high-performance computing (HPC), 275 hinted handoff, 183 histograms, 16 Hive (query engine), 419, 427 for data warehouses, 93 HCatalog and metastore, 410 map-side joins, 409 query optimizer, 427 skewed joins, 408 workflows, 403 Hollerith machines, 390 hopping windows (stream processing), 472 (see also windows) horizontal scaling (see scaling out) HornetQ (messaging), 137, 444 distributed transaction support, 361 hot spots, 201 due to celebrities, 205 for time-series data, 203 in batch processing, 407 relieving, 205 hot standbys (see leader-based replication) HTTP, use in APIs (see services) human errors, 9, 279, 414 HyperDex (database), 88 HyperLogLog (algorithm), 466 I I/O operations, waiting for, 297 IBM DB2 (database) distributed transaction support, 361 recursive query support, 54 serializable isolation, 242, 257 XML and JSON support, 30, 42 electromechanical card-sorting machines, 390 IMS (database), 36 imperative query APIs, 46 InfoSphere Streams (CEP engine), 466 MQ (messaging), 444 distributed transaction support, 361 System R (database), 222 WebSphere (messaging), 137 idempotence, 134, 478, 555 by giving operations unique IDs, 518, 522 idempotent operations, 517 immutability advantages of, 460, 531 Index | 571 deriving state from event log, 459-464 for crash recovery, 75 in B-trees, 82, 242 in event sourcing, 457 inputs to Unix commands, 397 limitations of, 463 Impala (query engine) for data warehouses, 93 hash joins, 409 native code generation, 428 use of HDFS, 417 impedance mismatch, 29 imperative languages, 42 setting element styles (example), 45 in doubt (transaction status), 358 holding locks, 362 orphaned transactions, 363 in-memory databases, 88 durability, 227 serial transaction execution, 253 incidents cascading failures, 9 crashes due to leap seconds, 290 data corruption and financial losses due to concurrency bugs, 233 data corruption on hard disks, 227 data loss due to last-write-wins, 173, 292 data on disks unreadable, 309 deleted items reappearing, 174 disclosure of sensitive data due to primary key reuse, 157 errors in transaction serializability, 529 gigabit network interface with 1 Kb/s throughput, 311 network faults, 279 network interface dropping only inbound packets, 279 network partitions and whole-datacenter failures, 275 poor handling of network faults, 280 sending message to ex-partner, 494 sharks biting undersea cables, 279 split brain due to 1-minute packet delay, 158, 279 vibrations in server rack, 14 violation of uniqueness constraint, 529 indexes, 71, 555 and snapshot isolation, 241 as derived data, 386, 499-504 572 | Index B-trees, 79-83 building in batch processes, 411 clustered, 86 comparison of B-trees and LSM-trees, 83-85 concatenated, 87 covering (with included columns), 86 creating, 500 full-text search, 88 geospatial, 87 hash, 72-75 index-range locking, 260 multi-column, 87 partitioning and secondary indexes, 206-209, 217 secondary, 85 (see also secondary indexes) problems with dual writes, 452, 491 SSTables and LSM-trees, 76-79 updating when data changes, 452, 467 Industrial Revolution, 541 InfiniBand (networks), 285 InfiniteGraph (database), 50 InnoDB (storage engine) clustered index on primary key, 86 not preventing lost updates, 245 preventing write skew, 248, 257 serializable isolation, 257 snapshot isolation support, 239 inside-out databases, 504 (see also unbundling databases) integrating different data systems (see data integration) integrity, 524 coordination-avoiding data systems, 528 correctness of dataflow systems, 525 in consensus formalization, 365 integrity checks, 530 (see also auditing) end-to-end, 519, 531 use of snapshot isolation, 238 maintaining despite software bugs, 529 Interface Definition Language (IDL), 117, 122 intermediate state, materialization of, 420-423 internet services, systems for implementing, 275 invariants, 225 (see also constraints) inversion of control, 396 IP (Internet Protocol) unreliability of, 277 ISDN (Integrated Services Digital Network), 284 isolation (in transactions), 225, 228, 555 correctness and, 515 for single-object writes, 230 serializability, 251-266 actual serial execution, 252-256 serializable snapshot isolation (SSI), 261-266 two-phase locking (2PL), 257-261 violating, 228 weak isolation levels, 233-251 preventing lost updates, 242-246 read committed, 234-237 snapshot isolation, 237-242 iterative processing, 424-426 J Java Database Connectivity (JDBC) distributed transaction support, 361 network drivers, 128 Java Enterprise Edition (EE), 134, 356, 361 Java Message Service (JMS), 444 (see also messaging systems) comparison to log-based messaging, 448, 451 distributed transaction support, 361 message ordering, 446 Java Transaction API (JTA), 355, 361 Java Virtual Machine (JVM) bytecode generation, 428 garbage collection pauses, 296 process reuse in batch processors, 422 JavaScript in MapReduce querying, 46 setting element styles (example), 45 use in advanced queries, 48 Jena (RDF framework), 57 Jepsen (fault tolerance testing), 515 jitter (network delay), 284 joins, 555 by index lookup, 403 expressing as relational operators, 427 in relational and document databases, 34 MapReduce map-side joins, 408-410 broadcast hash joins, 409 merge joins, 410 partitioned hash joins, 409 MapReduce reduce-side joins, 403-408 handling skew, 407 sort-merge joins, 405 parallel execution of, 415 secondary indexes and, 85 stream joins, 472-476 stream-stream join, 473 stream-table join, 473 table-table join, 474 time-dependence of, 475 support in document databases, 42 JOTM (transaction coordinator), 356 JSON Avro schema representation, 122 binary variants, 115 for application data, issues with, 114 in relational databases, 30, 42 representing a résumé (example), 31 Juttle (query language), 504 K k-nearest neighbors, 429 Kafka (messaging), 137, 448 Kafka Connect (database integration), 457, 461 Kafka Streams (stream processor), 466, 467 fault tolerance, 479 leader-based replication, 153 log compaction, 456, 467 message offsets, 447, 478 request routing, 216 transaction support, 477 usage example, 4 Ketama (partitioning library), 213 key-value stores, 70 as batch process output, 412 hash indexes, 72-75 in-memory, 89 partitioning, 201-205 by hash of key, 203, 217 by key range, 202, 217 dynamic partitioning, 212 skew and hot spots, 205 Kryo (Java), 113 Kubernetes (cluster manager), 418, 506 L lambda architecture, 497 Lamport timestamps, 345 Index | 573 Large Hadron Collider (LHC), 64 last write wins (LWW), 173, 334 discarding concurrent writes, 186 problems with, 292 prone to lost updates, 246 late binding, 396 latency instability under two-phase locking, 259 network latency and resource utilization, 286 response time versus, 14 tail latency, 15, 207 leader-based replication, 152-161 (see also replication) failover, 157, 301 handling node outages, 156 implementation of replication logs change data capture, 454-457 (see also changelogs) statement-based, 158 trigger-based replication, 161 write-ahead log (WAL) shipping, 159 linearizability of operations, 333 locking and leader election, 330 log sequence number, 156, 449 read-scaling architecture, 161 relation to consensus, 367 setting up new followers, 155 synchronous versus asynchronous, 153-155 leaderless replication, 177-191 (see also replication) detecting concurrent writes, 184-191 capturing happens-before relationship, 187 happens-before relationship and concur‐ rency, 186 last write wins, 186 merging concurrently written values, 190 version vectors, 191 multi-datacenter, 184 quorums, 179-182 consistency limitations, 181-183, 334 sloppy quorums and hinted handoff, 183 read repair and anti-entropy, 178 leap seconds, 8, 290 in time-of-day clocks, 288 leases, 295 implementation with ZooKeeper, 370 574 | Index need for fencing, 302 ledgers, 460 distributed ledger technologies, 532 legacy systems, maintenance of, 18 less (Unix tool), 397 LevelDB (storage engine), 78 leveled compaction, 79 Levenshtein automata, 88 limping (partial failure), 311 linearizability, 324-338, 555 cost of, 335-338 CAP theorem, 336 memory on multi-core CPUs, 338 definition, 325-329 implementing with total order broadcast, 350 in ZooKeeper, 370 of derived data systems, 492, 524 avoiding coordination, 527 of different replication methods, 332-335 using quorums, 334 relying on, 330-332 constraints and uniqueness, 330 cross-channel timing dependencies, 331 locking and leader election, 330 stronger than causal consistency, 342 using to implement total order broadcast, 351 versus serializability, 329 LinkedIn Azkaban (workflow scheduler), 402 Databus (change data capture), 161, 455 Espresso (database), 31, 126, 130, 153, 216 Helix (cluster manager) (see Helix) profile (example), 30 reference to company entity (example), 34 Rest.li (RPC framework), 135 Voldemort (database) (see Voldemort) Linux, leap second bug, 8, 290 liveness properties, 308 LMDB (storage engine), 82, 242 load approaches to coping with, 17 describing, 11 load testing, 16 load balancing (messaging), 444 local indexes (see document-partitioned indexes) locality (data access), 32, 41, 555 in batch processing, 400, 405, 421 in stateful clients, 170, 511 in stream processing, 474, 478, 508, 522 location transparency, 134 in the actor model, 138 locks, 556 deadlock, 258 distributed locking, 301-304, 330 fencing tokens, 303 implementation with ZooKeeper, 370 relation to consensus, 374 for transaction isolation in snapshot isolation, 239 in two-phase locking (2PL), 257-261 making operations atomic, 243 performance, 258 preventing dirty writes, 236 preventing phantoms with index-range locks, 260, 265 read locks (shared mode), 236, 258 shared mode and exclusive mode, 258 in two-phase commit (2PC) deadlock detection, 364 in-doubt transactions holding locks, 362 materializing conflicts with, 251 preventing lost updates by explicit locking, 244 log sequence number, 156, 449 logic programming languages, 504 logical clocks, 293, 343, 494 for read-after-write consistency, 164 logical logs, 160 logs (data structure), 71, 556 advantages of immutability, 460 compaction, 73, 79, 456, 460 for stream operator state, 479 creating using total order broadcast, 349 implementing uniqueness constraints, 522 log-based messaging, 446-451 comparison to traditional messaging, 448, 451 consumer offsets, 449 disk space usage, 450 replaying old messages, 451, 496, 498 slow consumers, 450 using logs for message storage, 447 log-structured storage, 71-79 log-structured merge tree (see LSMtrees) replication, 152, 158-161 change data capture, 454-457 (see also changelogs) coordination with snapshot, 156 logical (row-based) replication, 160 statement-based replication, 158 trigger-based replication, 161 write-ahead log (WAL) shipping, 159 scalability limits, 493 loose coupling, 396, 419, 502 lost updates (see updates) LSM-trees (indexes), 78-79 comparison to B-trees, 83-85 Lucene (storage engine), 79 building indexes in batch processes, 411 similarity search, 88 Luigi (workflow scheduler), 402 LWW (see last write wins) M machine learning ethical considerations, 534 (see also ethics) iterative processing, 424 models derived from training data, 505 statistical and numerical algorithms, 428 MADlib (machine learning toolkit), 428 magic scaling sauce, 18 Mahout (machine learning toolkit), 428 maintainability, 18-22, 489 defined, 23 design principles for software systems, 19 evolvability (see evolvability) operability, 19 simplicity and managing complexity, 20 many-to-many relationships in document model versus relational model, 39 modeling as graphs, 49 many-to-one and many-to-many relationships, 33-36 many-to-one relationships, 34 MapReduce (batch processing), 390, 399-400 accessing external services within job, 404, 412 comparison to distributed databases designing for frequent faults, 417 diversity of processing models, 416 diversity of storage, 415 Index | 575 comparison to stream processing, 464 comparison to Unix, 413-414 disadvantages and limitations of, 419 fault tolerance, 406, 414, 422 higher-level tools, 403, 426 implementation in Hadoop, 400-403 the shuffle, 402 implementation in MongoDB, 46-48 machine learning, 428 map-side processing, 408-410 broadcast hash joins, 409 merge joins, 410 partitioned hash joins, 409 mapper and reducer functions, 399 materialization of intermediate state, 419-423 output of batch workflows, 411-413 building search indexes, 411 key-value stores, 412 reduce-side processing, 403-408 analysis of user activity events (exam‐ ple), 404 grouping records by same key, 406 handling skew, 407 sort-merge joins, 405 workflows, 402 marshalling (see encoding) massively parallel processing (MPP), 216 comparison to composing storage technolo‐ gies, 502 comparison to Hadoop, 414-418, 428 master-master replication (see multi-leader replication) master-slave replication (see leader-based repli‐ cation) materialization, 556 aggregate values, 101 conflicts, 251 intermediate state (batch processing), 420-423 materialized views, 101 as derived data, 386, 499-504 maintaining, using stream processing, 467, 475 Maven (Java build tool), 428 Maxwell (change data capture), 455 mean, 14 media monitoring, 467 median, 14 576 | Index meeting room booking (example), 249, 259, 521 membership services, 372 Memcached (caching server), 4, 89 memory in-memory databases, 88 durability, 227 serial transaction execution, 253 in-memory representation of data, 112 random bit-flips in, 529 use by indexes, 72, 77 memory barrier (CPU instruction), 338 MemSQL (database) in-memory storage, 89 read committed isolation, 236 memtable (in LSM-trees), 78 Mercurial (version control system), 463 merge joins, MapReduce map-side, 410 mergeable persistent data structures, 174 merging sorted files, 76, 402, 405 Merkle trees, 532 Mesos (cluster manager), 418, 506 message brokers (see messaging systems) message-passing, 136-139 advantages over direct RPC, 137 distributed actor frameworks, 138 evolvability, 138 MessagePack (encoding format), 116 messages exactly-once semantics, 360, 476 loss of, 442 using total order broadcast, 348 messaging systems, 440-451 (see also streams) backpressure, buffering, or dropping mes‐ sages, 441 brokerless messaging, 442 event logs, 446-451 comparison to traditional messaging, 448, 451 consumer offsets, 449 replaying old messages, 451, 496, 498 slow consumers, 450 message brokers, 443-446 acknowledgements and redelivery, 445 comparison to event logs, 448, 451 multiple consumers of same topic, 444 reliability, 442 uniqueness in log-based messaging, 522 Meteor (web framework), 456 microbatching, 477, 495 microservices, 132 (see also services) causal dependencies across services, 493 loose coupling, 502 relation to batch/stream processors, 389, 508 Microsoft Azure Service Bus (messaging), 444 Azure Storage, 155, 398 Azure Stream Analytics, 466 DCOM (Distributed Component Object Model), 134 MSDTC (transaction coordinator), 356 Orleans (see Orleans) SQL Server (see SQL Server) migrating (rewriting) data, 40, 130, 461, 497 modulus operator (%), 210 MongoDB (database) aggregation pipeline, 48 atomic operations, 243 BSON, 41 document data model, 31 hash partitioning (sharding), 203-204 key-range partitioning, 202 lack of join support, 34, 42 leader-based replication, 153 MapReduce support, 46, 400 oplog parsing, 455, 456 partition splitting, 212 request routing, 216 secondary indexes, 207 Mongoriver (change data capture), 455 monitoring, 10, 19 monotonic clocks, 288 monotonic reads, 164 MPP (see massively parallel processing) MSMQ (messaging), 361 multi-column indexes, 87 multi-leader replication, 168-177 (see also replication) handling write conflicts, 171 conflict avoidance, 172 converging toward a consistent state, 172 custom conflict resolution logic, 173 determining what is a conflict, 174 linearizability, lack of, 333 replication topologies, 175-177 use cases, 168 clients with offline operation, 170 collaborative editing, 170 multi-datacenter replication, 168, 335 multi-object transactions, 228 need for, 231 Multi-Paxos (total order broadcast), 367 multi-table index cluster tables (Oracle), 41 multi-tenancy, 284 multi-version concurrency control (MVCC), 239, 266 detecting stale MVCC reads, 263 indexes and snapshot isolation, 241 mutual exclusion, 261 (see also locks) MySQL (database) binlog coordinates, 156 binlog parsing for change data capture, 455 circular replication topology, 175 consistent snapshots, 156 distributed transaction support, 361 InnoDB storage engine (see InnoDB) JSON support, 30, 42 leader-based replication, 153 performance of XA transactions, 360 row-based replication, 160 schema changes in, 40 snapshot isolation support, 242 (see also InnoDB) statement-based replication, 159 Tungsten Replicator (multi-leader replica‐ tion), 170 conflict detection, 177 N nanomsg (messaging library), 442 Narayana (transaction coordinator), 356 NATS (messaging), 137 near-real-time (nearline) processing, 390 (see also stream processing) Neo4j (database) Cypher query language, 52 graph data model, 50 Nephele (dataflow engine), 421 netcat (Unix tool), 397 Netflix Chaos Monkey, 7, 280 Network Attached Storage (NAS), 146, 398 network model, 36 Index | 577 graph databases versus, 60 imperative query APIs, 46 Network Time Protocol (see NTP) networks congestion and queueing, 282 datacenter network topologies, 276 faults (see faults) linearizability and network delays, 338 network partitions, 279, 337 timeouts and unbounded delays, 281 next-key locking, 260 nodes (in graphs) (see vertices) nodes (processes), 556 handling outages in leader-based replica‐ tion, 156 system models for failure, 307 noisy neighbors, 284 nonblocking atomic commit, 359 nondeterministic operations accidental nondeterminism, 423 partial failures in distributed systems, 275 nonfunctional requirements, 22 nonrepeatable reads, 238 (see also read skew) normalization (data representation), 33, 556 executing joins, 39, 42, 403 foreign key references, 231 in systems of record, 386 versus denormalization, 462 NoSQL, 29, 499 transactions and, 223 Notation3 (N3), 56 npm (package manager), 428 NTP (Network Time Protocol), 287 accuracy, 289, 293 adjustments to monotonic clocks, 289 multiple server addresses, 306 numbers, in XML and JSON encodings, 114 O object-relational mapping (ORM) frameworks, 30 error handling and aborted transactions, 232 unsafe read-modify-write cycle code, 244 object-relational mismatch, 29 observer pattern, 506 offline systems, 390 (see also batch processing) 578 | Index stateful, offline-capable clients, 170, 511 offline-first applications, 511 offsets consumer offsets in partitioned logs, 449 messages in partitioned logs, 447 OLAP (online analytic processing), 91, 556 data cubes, 102 OLTP (online transaction processing), 90, 556 analytics queries versus, 411 workload characteristics, 253 one-to-many relationships, 30 JSON representation, 32 online systems, 389 (see also services) Oozie (workflow scheduler), 402 OpenAPI (service definition format), 133 OpenStack Nova (cloud infrastructure) use of ZooKeeper, 370 Swift (object storage), 398 operability, 19 operating systems versus databases, 499 operation identifiers, 518, 522 operational transformation, 174 operators, 421 flow of data between, 424 in stream processing, 464 optimistic concurrency control, 261 Oracle (database) distributed transaction support, 361 GoldenGate (change data capture), 161, 170, 455 lack of serializability, 226 leader-based replication, 153 multi-table index cluster tables, 41 not preventing write skew, 248 partitioned indexes, 209 PL/SQL language, 255 preventing lost updates, 245 read committed isolation, 236 Real Application Clusters (RAC), 330 recursive query support, 54 snapshot isolation support, 239, 242 TimesTen (in-memory database), 89 WAL-based replication, 160 XML support, 30 ordering, 339-352 by sequence numbers, 343-348 causal ordering, 339-343 partial order, 341 limits of total ordering, 493 total order broadcast, 348-352 Orleans (actor framework), 139 outliers (response time), 14 Oz (programming language), 504 P package managers, 428, 505 packet switching, 285 packets corruption of, 306 sending via UDP, 442 PageRank (algorithm), 49, 424 paging (see virtual memory) ParAccel (database), 93 parallel databases (see massively parallel pro‐ cessing) parallel execution of graph analysis algorithms, 426 queries in MPP databases, 216 Parquet (data format), 96, 131 (see also column-oriented storage) use in Hadoop, 414 partial failures, 275, 310 limping, 311 partial order, 341 partitioning, 199-218, 556 and replication, 200 in batch processing, 429 multi-partition operations, 514 enforcing constraints, 522 secondary index maintenance, 495 of key-value data, 201-205 by key range, 202 skew and hot spots, 205 rebalancing partitions, 209-214 automatic or manual rebalancing, 213 problems with hash mod N, 210 using dynamic partitioning, 212 using fixed number of partitions, 210 using N partitions per node, 212 replication and, 147 request routing, 214-216 secondary indexes, 206-209 document-based partitioning, 206 term-based partitioning, 208 serial execution of transactions and, 255 Paxos (consensus algorithm), 366 ballot number, 368 Multi-Paxos (total order broadcast), 367 percentiles, 14, 556 calculating efficiently, 16 importance of high percentiles, 16 use in service level agreements (SLAs), 15 Percona XtraBackup (MySQL tool), 156 performance describing, 13 of distributed transactions, 360 of in-memory databases, 89 of linearizability, 338 of multi-leader replication, 169 perpetual inconsistency, 525 pessimistic concurrency control, 261 phantoms (transaction isolation), 250 materializing conflicts, 251 preventing, in serializability, 259 physical clocks (see clocks) pickle (Python), 113 Pig (dataflow language), 419, 427 replicated joins, 409 skewed joins, 407 workflows, 403 Pinball (workflow scheduler), 402 pipelined execution, 423 in Unix, 394 point in time, 287 polyglot persistence, 29 polystores, 501 PostgreSQL (database) BDR (multi-leader replication), 170 causal ordering of writes, 177 Bottled Water (change data capture), 455 Bucardo (trigger-based replication), 161, 173 distributed transaction support, 361 foreign data wrappers, 501 full text search support, 490 leader-based replication, 153 log sequence number, 156 MVCC implementation, 239, 241 PL/pgSQL language, 255 PostGIS geospatial indexes, 87 preventing lost updates, 245 preventing write skew, 248, 261 read committed isolation, 236 recursive query support, 54 representing graphs, 51 Index | 579 serializable snapshot isolation (SSI), 261 snapshot isolation support, 239, 242 WAL-based replication, 160 XML and JSON support, 30, 42 pre-splitting, 212 Precision Time Protocol (PTP), 290 predicate locks, 259 predictive analytics, 533-536 amplifying bias, 534 ethics of (see ethics) feedback loops, 536 preemption of datacenter resources, 418 of threads, 298 Pregel processing model, 425 primary keys, 85, 556 compound primary key (Cassandra), 204 primary-secondary replication (see leaderbased replication) privacy, 536-543 consent and freedom of choice, 538 data as assets and power, 540 deleting data, 463 ethical considerations (see ethics) legislation and self-regulation, 542 meaning of, 539 surveillance, 537 tracking behavioral data, 536 probabilistic algorithms, 16, 466 process pauses, 295-299 processing time (of events), 469 producers (message streams), 440 programming languages dataflow languages, 504 for stored procedures, 255 functional reactive programming (FRP), 504 logic programming, 504 Prolog (language), 61 (see also Datalog) promises (asynchronous operations), 135 property graphs, 50 Cypher query language, 52 Protocol Buffers (data format), 117-121 field tags and schema evolution, 120 provenance of data, 531 publish/subscribe model, 441 publishers (message streams), 440 punch card tabulating machines, 390 580 | Index pure functions, 48 putting computation near data, 400 Q Qpid (messaging), 444 quality of service (QoS), 285 Quantcast File System (distributed filesystem), 398 query languages, 42-48 aggregation pipeline, 48 CSS and XSL, 44 Cypher, 52 Datalog, 60 Juttle, 504 MapReduce querying, 46-48 recursive SQL queries, 53 relational algebra and SQL, 42 SPARQL, 59 query optimizers, 37, 427 queueing delays (networks), 282 head-of-line blocking, 15 latency and response time, 14 queues (messaging), 137 quorums, 179-182, 556 for leaderless replication, 179 in consensus algorithms, 368 limitations of consistency, 181-183, 334 making decisions in distributed systems, 301 monitoring staleness, 182 multi-datacenter replication, 184 relying on durability, 309 sloppy quorums and hinted handoff, 183 R R-trees (indexes), 87 RabbitMQ (messaging), 137, 444 leader-based replication, 153 race conditions, 225 (see also concurrency) avoiding with linearizability, 331 caused by dual writes, 452 dirty writes, 235 in counter increments, 235 lost updates, 242-246 preventing with event logs, 462, 507 preventing with serializable isolation, 252 write skew, 246-251 Raft (consensus algorithm), 366 sensitivity to network problems, 369 term number, 368 use in etcd, 353 RAID (Redundant Array of Independent Disks), 7, 398 railways, schema migration on, 496 RAMCloud (in-memory storage), 89 ranking algorithms, 424 RDF (Resource Description Framework), 57 querying with SPARQL, 59 RDMA (Remote Direct Memory Access), 276 read committed isolation level, 234-237 implementing, 236 multi-version concurrency control (MVCC), 239 no dirty reads, 234 no dirty writes, 235 read path (derived data), 509 read repair (leaderless replication), 178 for linearizability, 335 read replicas (see leader-based replication) read skew (transaction isolation), 238, 266 as violation of causality, 340 read-after-write consistency, 163, 524 cross-device, 164 read-modify-write cycle, 243 read-scaling architecture, 161 reads as events, 513 real-time collaborative editing, 170 near-real-time processing, 390 (see also stream processing) publish/subscribe dataflow, 513 response time guarantees, 298 time-of-day clocks, 288 rebalancing partitions, 209-214, 556 (see also partitioning) automatic or manual rebalancing, 213 dynamic partitioning, 212 fixed number of partitions, 210 fixed number of partitions per node, 212 problems with hash mod N, 210 recency guarantee, 324 recommendation engines batch process outputs, 412 batch workflows, 403, 420 iterative processing, 424 statistical and numerical algorithms, 428 records, 399 events in stream processing, 440 recursive common table expressions (SQL), 54 redelivery (messaging), 445 Redis (database) atomic operations, 243 durability, 89 Lua scripting, 255 single-threaded execution, 253 usage example, 4 redundancy hardware components, 7 of derived data, 386 (see also derived data) Reed–Solomon codes (error correction), 398 refactoring, 22 (see also evolvability) regions (partitioning), 199 register (data structure), 325 relational data model, 28-42 comparison to document model, 38-42 graph queries in SQL, 53 in-memory databases with, 89 many-to-one and many-to-many relation‐ ships, 33 multi-object transactions, need for, 231 NoSQL as alternative to, 29 object-relational mismatch, 29 relational algebra and SQL, 42 versus document model convergence of models, 41 data locality, 41 relational databases eventual consistency, 162 history, 28 leader-based replication, 153 logical logs, 160 philosophy compared to Unix, 499, 501 schema changes, 40, 111, 130 statement-based replication, 158 use of B-tree indexes, 80 relationships (see edges) reliability, 6-10, 489 building a reliable system from unreliable components, 276 defined, 6, 22 hardware faults, 7 human errors, 9 importance of, 10 of messaging systems, 442 Index | 581 software errors, 8 Remote Method Invocation (Java RMI), 134 remote procedure calls (RPCs), 134-136 (see also services) based on futures, 135 data encoding and evolution, 136 issues with, 134 using Avro, 126, 135 using Thrift, 135 versus message brokers, 137 repeatable reads (transaction isolation), 242 replicas, 152 replication, 151-193, 556 and durability, 227 chain replication, 155 conflict resolution and, 246 consistency properties, 161-167 consistent prefix reads, 165 monotonic reads, 164 reading your own writes, 162 in distributed filesystems, 398 leaderless, 177-191 detecting concurrent writes, 184-191 limitations of quorum consistency, 181-183, 334 sloppy quorums and hinted handoff, 183 monitoring staleness, 182 multi-leader, 168-177 across multiple datacenters, 168, 335 handling write conflicts, 171-175 replication topologies, 175-177 partitioning and, 147, 200 reasons for using, 145, 151 single-leader, 152-161 failover, 157 implementation of replication logs, 158-161 relation to consensus, 367 setting up new followers, 155 synchronous versus asynchronous, 153-155 state machine replication, 349, 452 using erasure coding, 398 with heterogeneous data systems, 453 replication logs (see logs) reprocessing data, 496, 498 (see also evolvability) from log-based messaging, 451 request routing, 214-216 582 | Index approaches to, 214 parallel query execution, 216 resilient systems, 6 (see also fault tolerance) response time as performance metric for services, 13, 389 guarantees on, 298 latency versus, 14 mean and percentiles, 14 user experience, 15 responsibility and accountability, 535 REST (Representational State Transfer), 133 (see also services) RethinkDB (database) document data model, 31 dynamic partitioning, 212 join support, 34, 42 key-range partitioning, 202 leader-based replication, 153 subscribing to changes, 456 Riak (database) Bitcask storage engine, 72 CRDTs, 174, 191 dotted version vectors, 191 gossip protocol, 216 hash partitioning, 203-204, 211 last-write-wins conflict resolution, 186 leaderless replication, 177 LevelDB storage engine, 78 linearizability, lack of, 335 multi-datacenter support, 184 preventing lost updates across replicas, 246 rebalancing, 213 search feature, 209 secondary indexes, 207 siblings (concurrently written values), 190 sloppy quorums, 184 ring buffers, 450 Ripple (cryptocurrency), 532 rockets, 10, 36, 305 RocksDB (storage engine), 78 leveled compaction, 79 rollbacks (transactions), 222 rolling upgrades, 8, 112 routing (see request routing) row-oriented storage, 96 row-based replication, 160 rowhammer (memory corruption), 529 RPCs (see remote procedure calls) Rubygems (package manager), 428 rules (Datalog), 61 S safety and liveness properties, 308 in consensus algorithms, 366 in transactions, 222 sagas (see compensating transactions) Samza (stream processor), 466, 467 fault tolerance, 479 streaming SQL support, 466 sandboxes, 9 SAP HANA (database), 93 scalability, 10-18, 489 approaches for coping with load, 17 defined, 22 describing load, 11 describing performance, 13 partitioning and, 199 replication and, 161 scaling up versus scaling out, 146 scaling out, 17, 146 (see also shared-nothing architecture) scaling up, 17, 146 scatter/gather approach, querying partitioned databases, 207 SCD (slowly changing dimension), 476 schema-on-read, 39 comparison to evolvable schema, 128 in distributed filesystems, 415 schema-on-write, 39 schemaless databases (see schema-on-read) schemas, 557 Avro, 122-127 reader determining writer’s schema, 125 schema evolution, 123 dynamically generated, 126 evolution of, 496 affecting application code, 111 compatibility checking, 126 in databases, 129-131 in message-passing, 138 in service calls, 136 flexibility in document model, 39 for analytics, 93-95 for JSON and XML, 115 merits of, 127 schema migration on railways, 496 Thrift and Protocol Buffers, 117-121 schema evolution, 120 traditional approach to design, fallacy in, 462 searches building search indexes in batch processes, 411 k-nearest neighbors, 429 on streams, 467 partitioned secondary indexes, 206 secondaries (see leader-based replication) secondary indexes, 85, 557 partitioning, 206-209, 217 document-partitioned, 206 index maintenance, 495 term-partitioned, 208 problems with dual writes, 452, 491 updating, transaction isolation and, 231 secondary sorts, 405 sed (Unix tool), 392 self-describing files, 127 self-joins, 480 self-validating systems, 530 semantic web, 57 semi-synchronous replication, 154 sequence number ordering, 343-348 generators, 294, 344 insufficiency for enforcing constraints, 347 Lamport timestamps, 345 use of timestamps, 291, 295, 345 sequential consistency, 351 serializability, 225, 233, 251-266, 557 linearizability versus, 329 pessimistic versus optimistic concurrency control, 261 serial execution, 252-256 partitioning, 255 using stored procedures, 253, 349 serializable snapshot isolation (SSI), 261-266 detecting stale MVCC reads, 263 detecting writes that affect prior reads, 264 distributed execution, 265, 364 performance of SSI, 265 preventing write skew, 262-265 two-phase locking (2PL), 257-261 index-range locks, 260 performance, 258 Serializable (Java), 113 Index | 583 serialization, 113 (see also encoding) service discovery, 135, 214, 372 using DNS, 216, 372 service level agreements (SLAs), 15 service-oriented architecture (SOA), 132 (see also services) services, 131-136 microservices, 132 causal dependencies across services, 493 loose coupling, 502 relation to batch/stream processors, 389, 508 remote procedure calls (RPCs), 134-136 issues with, 134 similarity to databases, 132 web services, 132, 135 session windows (stream processing), 472 (see also windows) sessionization, 407 sharding (see partitioning) shared mode (locks), 258 shared-disk architecture, 146, 398 shared-memory architecture, 146 shared-nothing architecture, 17, 146-147, 557 (see also replication) distributed filesystems, 398 (see also distributed filesystems) partitioning, 199 use of network, 277 sharks biting undersea cables, 279 counting (example), 46-48 finding (example), 42 website about (example), 44 shredding (in relational model), 38 siblings (concurrent values), 190, 246 (see also conflicts) similarity search edit distance, 88 genome data, 63 k-nearest neighbors, 429 single-leader replication (see leader-based rep‐ lication) single-threaded execution, 243, 252 in batch processing, 406, 421, 426 in stream processing, 448, 463, 522 size-tiered compaction, 79 skew, 557 584 | Index clock skew, 291-294, 334 in transaction isolation read skew, 238, 266 write skew, 246-251, 262-265 (see also write skew) meanings of, 238 unbalanced workload, 201 compensating for, 205 due to celebrities, 205 for time-series data, 203 in batch processing, 407 slaves (see leader-based replication) sliding windows (stream processing), 472 (see also windows) sloppy quorums, 183 (see also quorums) lack of linearizability, 334 slowly changing dimension (data warehouses), 476 smearing (leap seconds adjustments), 290 snapshots (databases) causal consistency, 340 computing derived data, 500 in change data capture, 455 serializable snapshot isolation (SSI), 261-266, 329 setting up a new replica, 156 snapshot isolation and repeatable read, 237-242 implementing with MVCC, 239 indexes and MVCC, 241 visibility rules, 240 synchronized clocks for global snapshots, 294 snowflake schemas, 95 SOAP, 133 (see also services) evolvability, 136 software bugs, 8 maintaining integrity, 529 solid state drives (SSDs) access patterns, 84 detecting corruption, 519, 530 faults in, 227 sequential write throughput, 75 Solr (search server) building indexes in batch processes, 411 document-partitioned indexes, 207 request routing, 216 usage example, 4 use of Lucene, 79 sort (Unix tool), 392, 394, 395 sort-merge joins (MapReduce), 405 Sorted String Tables (see SSTables) sorting sort order in column storage, 99 source of truth (see systems of record) Spanner (database) data locality, 41 snapshot isolation using clocks, 295 TrueTime API, 294 Spark (processing framework), 421-423 bytecode generation, 428 dataflow APIs, 427 fault tolerance, 422 for data warehouses, 93 GraphX API (graph processing), 425 machine learning, 428 query optimizer, 427 Spark Streaming, 466 microbatching, 477 stream processing on top of batch process‐ ing, 495 SPARQL (query language), 59 spatial algorithms, 429 split brain, 158, 557 in consensus algorithms, 352, 367 preventing, 322, 333 using fencing tokens to avoid, 302-304 spreadsheets, dataflow programming capabili‐ ties, 504 SQL (Structured Query Language), 21, 28, 43 advantages and limitations of, 416 distributed query execution, 48 graph queries in, 53 isolation levels standard, issues with, 242 query execution on Hadoop, 416 résumé (example), 30 SQL injection vulnerability, 305 SQL on Hadoop, 93 statement-based replication, 158 stored procedures, 255 SQL Server (database) data warehousing support, 93 distributed transaction support, 361 leader-based replication, 153 preventing lost updates, 245 preventing write skew, 248, 257 read committed isolation, 236 recursive query support, 54 serializable isolation, 257 snapshot isolation support, 239 T-SQL language, 255 XML support, 30 SQLstream (stream analytics), 466 SSDs (see solid state drives) SSTables (storage format), 76-79 advantages over hash indexes, 76 concatenated index, 204 constructing and maintaining, 78 making LSM-Tree from, 78 staleness (old data), 162 cross-channel timing dependencies, 331 in leaderless databases, 178 in multi-version concurrency control, 263 monitoring for, 182 of client state, 512 versus linearizability, 324 versus timeliness, 524 standbys (see leader-based replication) star replication topologies, 175 star schemas, 93-95 similarity to event sourcing, 458 Star Wars analogy (event time versus process‐ ing time), 469 state derived from log of immutable events, 459 deriving current state from the event log, 458 interplay between state changes and appli‐ cation code, 507 maintaining derived state, 495 maintenance by stream processor in streamstream joins, 473 observing derived state, 509-515 rebuilding after stream processor failure, 478 separation of application code and, 505 state machine replication, 349, 452 statement-based replication, 158 statically typed languages analogy to schema-on-write, 40 code generation and, 127 statistical and numerical algorithms, 428 StatsD (metrics aggregator), 442 stdin, stdout, 395, 396 Stellar (cryptocurrency), 532 Index | 585 stock market feeds, 442 STONITH (Shoot The Other Node In The Head), 158 stop-the-world (see garbage collection) storage composing data storage technologies, 499-504 diversity of, in MapReduce, 415 Storage Area Network (SAN), 146, 398 storage engines, 69-104 column-oriented, 95-101 column compression, 97-99 defined, 96 distinction between column families and, 99 Parquet, 96, 131 sort order in, 99-100 writing to, 101 comparing requirements for transaction processing and analytics, 90-96 in-memory storage, 88 durability, 227 row-oriented, 70-90 B-trees, 79-83 comparing B-trees and LSM-trees, 83-85 defined, 96 log-structured, 72-79 stored procedures, 161, 253-255, 557 and total order broadcast, 349 pros and cons of, 255 similarity to stream processors, 505 Storm (stream processor), 466 distributed RPC, 468, 514 Trident state handling, 478 straggler events, 470, 498 stream processing, 464-481, 557 accessing external services within job, 474, 477, 478, 517 combining with batch processing lambda architecture, 497 unifying technologies, 498 comparison to batch processing, 464 complex event processing (CEP), 465 fault tolerance, 476-479 atomic commit, 477 idempotence, 478 microbatching and checkpointing, 477 rebuilding state after a failure, 478 for data integration, 494-498 586 | Index maintaining derived state, 495 maintenance of materialized views, 467 messaging systems (see messaging systems) reasoning about time, 468-472 event time versus processing time, 469, 477, 498 knowing when window is ready, 470 types of windows, 472 relation to databases (see streams) relation to services, 508 search on streams, 467 single-threaded execution, 448, 463 stream analytics, 466 stream joins, 472-476 stream-stream join, 473 stream-table join, 473 table-table join, 474 time-dependence of, 475 streams, 440-451 end-to-end, pushing events to clients, 512 messaging systems (see messaging systems) processing (see stream processing) relation to databases, 451-464 (see also changelogs) API support for change streams, 456 change data capture, 454-457 derivative of state by time, 460 event sourcing, 457-459 keeping systems in sync, 452-453 philosophy of immutable events, 459-464 topics, 440 strict serializability, 329 strong consistency (see linearizability) strong one-copy serializability, 329 subjects, predicates, and objects (in triplestores), 55 subscribers (message streams), 440 (see also consumers) supercomputers, 275 surveillance, 537 (see also privacy) Swagger (service definition format), 133 swapping to disk (see virtual memory) synchronous networks, 285, 557 comparison to asynchronous networks, 284 formal model, 307 synchronous replication, 154, 557 chain replication, 155 conflict detection, 172 system models, 300, 306-310 assumptions in, 528 correctness of algorithms, 308 mapping to the real world, 309 safety and liveness, 308 systems of record, 386, 557 change data capture, 454, 491 treating event log as, 460 systems thinking, 536 T t-digest (algorithm), 16 table-table joins, 474 Tableau (data visualization software), 416 tail (Unix tool), 447 tail vertex (property graphs), 51 Tajo (query engine), 93 Tandem NonStop SQL (database), 200 TCP (Transmission Control Protocol), 277 comparison to circuit switching, 285 comparison to UDP, 283 connection failures, 280 flow control, 282, 441 packet checksums, 306, 519, 529 reliability and duplicate suppression, 517 retransmission timeouts, 284 use for transaction sessions, 229 telemetry (see monitoring) Teradata (database), 93, 200 term-partitioned indexes, 208, 217 termination (consensus), 365 Terrapin (database), 413 Tez (dataflow engine), 421-423 fault tolerance, 422 support by higher-level tools, 427 thrashing (out of memory), 297 threads (concurrency) actor model, 138, 468 (see also message-passing) atomic operations, 223 background threads, 73, 85 execution pauses, 286, 296-298 memory barriers, 338 preemption, 298 single (see single-threaded execution) three-phase commit, 359 Thrift (data format), 117-121 BinaryProtocol, 118 CompactProtocol, 119 field tags and schema evolution, 120 throughput, 13, 390 TIBCO, 137 Enterprise Message Service, 444 StreamBase (stream analytics), 466 time concurrency and, 187 cross-channel timing dependencies, 331 in distributed systems, 287-299 (see also clocks) clock synchronization and accuracy, 289 relying on synchronized clocks, 291-295 process pauses, 295-299 reasoning about, in stream processors, 468-472 event time versus processing time, 469, 477, 498 knowing when window is ready, 470 timestamp of events, 471 types of windows, 472 system models for distributed systems, 307 time-dependence in stream joins, 475 time-of-day clocks, 288 timeliness, 524 coordination-avoiding data systems, 528 correctness of dataflow systems, 525 timeouts, 279, 557 dynamic configuration of, 284 for failover, 158 length of, 281 timestamps, 343 assigning to events in stream processing, 471 for read-after-write consistency, 163 for transaction ordering, 295 insufficiency for enforcing constraints, 347 key range partitioning by, 203 Lamport, 345 logical, 494 ordering events, 291, 345 Titan (database), 50 tombstones, 74, 191, 456 topics (messaging), 137, 440 total order, 341, 557 limits of, 493 sequence numbers or timestamps, 344 total order broadcast, 348-352, 493, 522 consensus algorithms and, 366-368 Index | 587 implementation in ZooKeeper and etcd, 370 implementing with linearizable storage, 351 using, 349 using to implement linearizable storage, 350 tracking behavioral data, 536 (see also privacy) transaction coordinator (see coordinator) transaction manager (see coordinator) transaction processing, 28, 90-95 comparison to analytics, 91 comparison to data warehousing, 93 transactions, 221-267, 558 ACID properties of, 223 atomicity, 223 consistency, 224 durability, 226 isolation, 225 compensating (see compensating transac‐ tions) concept of, 222 distributed transactions, 352-364 avoiding, 492, 502, 521-528 failure amplification, 364, 495 in doubt/uncertain status, 358, 362 two-phase commit, 354-359 use of, 360-361 XA transactions, 361-364 OLTP versus analytics queries, 411 purpose of, 222 serializability, 251-266 actual serial execution, 252-256 pessimistic versus optimistic concur‐ rency control, 261 serializable snapshot isolation (SSI), 261-266 two-phase locking (2PL), 257-261 single-object and multi-object, 228-232 handling errors and aborts, 231 need for multi-object transactions, 231 single-object writes, 230 snapshot isolation (see snapshots) weak isolation levels, 233-251 preventing lost updates, 242-246 read committed, 234-238 transitive closure (graph algorithm), 424 trie (data structure), 88 triggers (databases), 161, 441 implementing change data capture, 455 implementing replication, 161 588 | Index triple-stores, 55-59 SPARQL query language, 59 tumbling windows (stream processing), 472 (see also windows) in microbatching, 477 tuple spaces (programming model), 507 Turtle (RDF data format), 56 Twitter constructing home timelines (example), 11, 462, 474, 511 DistributedLog (event log), 448 Finagle (RPC framework), 135 Snowflake (sequence number generator), 294 Summingbird (processing library), 497 two-phase commit (2PC), 353, 355-359, 558 confusion with two-phase locking, 356 coordinator failure, 358 coordinator recovery, 363 how it works, 357 issues in practice, 363 performance cost, 360 transactions holding locks, 362 two-phase locking (2PL), 257-261, 329, 558 confusion with two-phase commit, 356 index-range locks, 260 performance of, 258 type checking, dynamic versus static, 40 U UDP (User Datagram Protocol) comparison to TCP, 283 multicast, 442 unbounded datasets, 439, 558 (see also streams) unbounded delays, 558 in networks, 282 process pauses, 296 unbundling databases, 499-515 composing data storage technologies, 499-504 federation versus unbundling, 501 need for high-level language, 503 designing applications around dataflow, 504-509 observing derived state, 509-515 materialized views and caching, 510 multi-partition data processing, 514 pushing state changes to clients, 512 uncertain (transaction status) (see in doubt) uniform consensus, 365 (see also consensus) uniform interfaces, 395 union type (in Avro), 125 uniq (Unix tool), 392 uniqueness constraints asynchronously checked, 526 requiring consensus, 521 requiring linearizability, 330 uniqueness in log-based messaging, 522 Unix philosophy, 394-397 command-line batch processing, 391-394 Unix pipes versus dataflow engines, 423 comparison to Hadoop, 413-414 comparison to relational databases, 499, 501 comparison to stream processing, 464 composability and uniform interfaces, 395 loose coupling, 396 pipes, 394 relation to Hadoop, 499 UPDATE statement (SQL), 40 updates preventing lost updates, 242-246 atomic write operations, 243 automatically detecting lost updates, 245 compare-and-set operations, 245 conflict resolution and replication, 246 using explicit locking, 244 preventing write skew, 246-251 V validity (consensus), 365 vBuckets (partitioning), 199 vector clocks, 191 (see also version vectors) vectorized processing, 99, 428 verification, 528-533 avoiding blind trust, 530 culture of, 530 designing for auditability, 531 end-to-end integrity checks, 531 tools for auditable data systems, 532 version control systems, reliance on immutable data, 463 version vectors, 177, 191 capturing causal dependencies, 343 versus vector clocks, 191 Vertica (database), 93 handling writes, 101 replicas using different sort orders, 100 vertical scaling (see scaling up) vertices (in graphs), 49 property graph model, 50 Viewstamped Replication (consensus algo‐ rithm), 366 view number, 368 virtual machines, 146 (see also cloud computing) context switches, 297 network performance, 282 noisy neighbors, 284 reliability in cloud services, 8 virtualized clocks in, 290 virtual memory process pauses due to page faults, 14, 297 versus memory management by databases, 89 VisiCalc (spreadsheets), 504 vnodes (partitioning), 199 Voice over IP (VoIP), 283 Voldemort (database) building read-only stores in batch processes, 413 hash partitioning, 203-204, 211 leaderless replication, 177 multi-datacenter support, 184 rebalancing, 213 reliance on read repair, 179 sloppy quorums, 184 VoltDB (database) cross-partition serializability, 256 deterministic stored procedures, 255 in-memory storage, 89 output streams, 456 secondary indexes, 207 serial execution of transactions, 253 statement-based replication, 159, 479 transactions in stream processing, 477 W WAL (write-ahead log), 82 web services (see services) Web Services Description Language (WSDL), 133 webhooks, 443 webMethods (messaging), 137 WebSocket (protocol), 512 Index | 589 windows (stream processing), 466, 468-472 infinite windows for changelogs, 467, 474 knowing when all events have arrived, 470 stream joins within a window, 473 types of windows, 472 winners (conflict resolution), 173 WITH RECURSIVE syntax (SQL), 54 workflows (MapReduce), 402 outputs, 411-414 key-value stores, 412 search indexes, 411 with map-side joins, 410 working set, 393 write amplification, 84 write path (derived data), 509 write skew (transaction isolation), 246-251 characterizing, 246-251, 262 examples of, 247, 249 materializing conflicts, 251 occurrence in practice, 529 phantoms, 250 preventing in snapshot isolation, 262-265 in two-phase locking, 259-261 options for, 248 write-ahead log (WAL), 82, 159 writes (database) atomic write operations, 243 detecting writes affecting prior reads, 264 preventing dirty writes with read commit‐ ted, 235 WS-* framework, 133 (see also services) WS-AtomicTransaction (2PC), 355 590 | Index X XA transactions, 355, 361-364 heuristic decisions, 363 limitations of, 363 xargs (Unix tool), 392, 396 XML binary variants, 115 encoding RDF data, 57 for application data, issues with, 114 in relational databases, 30, 41 XSL/XPath, 45 Y Yahoo!

…

The opposite of bounded. 558 | Glossary Index A aborts (transactions), 222, 224 in two-phase commit, 356 performance of optimistic concurrency con‐ trol, 266 retrying aborted transactions, 231 abstraction, 21, 27, 222, 266, 321 access path (in network model), 37, 60 accidental complexity, removing, 21 accountability, 535 ACID properties (transactions), 90, 223 atomicity, 223, 228 consistency, 224, 529 durability, 226 isolation, 225, 228 acknowledgements (messaging), 445 active/active replication (see multi-leader repli‐ cation) active/passive replication (see leader-based rep‐ lication) ActiveMQ (messaging), 137, 444 distributed transaction support, 361 ActiveRecord (object-relational mapper), 30, 232 actor model, 138 (see also message-passing) comparison to Pregel model, 425 comparison to stream processing, 468 Advanced Message Queuing Protocol (see AMQP) aerospace systems, 6, 10, 305, 372 aggregation data cubes and materialized views, 101 in batch processes, 406 in stream processes, 466 aggregation pipeline query language, 48 Agile, 22 minimizing irreversibility, 414, 497 moving faster with confidence, 532 Unix philosophy, 394 agreement, 365 (see also consensus) Airflow (workflow scheduler), 402 Ajax, 131 Akka (actor framework), 139 algorithms algorithm correctness, 308 B-trees, 79-83 for distributed systems, 306 hash indexes, 72-75 mergesort, 76, 402, 405 red-black trees, 78 SSTables and LSM-trees, 76-79 all-to-all replication topologies, 175 AllegroGraph (database), 50 ALTER TABLE statement (SQL), 40, 111 Amazon Dynamo (database), 177 Amazon Web Services (AWS), 8 Kinesis Streams (messaging), 448 network reliability, 279 postmortems, 9 RedShift (database), 93 S3 (object storage), 398 checking data integrity, 530 amplification of bias, 534 of failures, 364, 495 Index | 559 of tail latency, 16, 207 write amplification, 84 AMQP (Advanced Message Queuing Protocol), 444 (see also messaging systems) comparison to log-based messaging, 448, 451 message ordering, 446 analytics, 90 comparison to transaction processing, 91 data warehousing (see data warehousing) parallel query execution in MPP databases, 415 predictive (see predictive analytics) relation to batch processing, 411 schemas for, 93-95 snapshot isolation for queries, 238 stream analytics, 466 using MapReduce, analysis of user activity events (example), 404 anti-caching (in-memory databases), 89 anti-entropy, 178 Apache ActiveMQ (see ActiveMQ) Apache Avro (see Avro) Apache Beam (see Beam) Apache BookKeeper (see BookKeeper) Apache Cassandra (see Cassandra) Apache CouchDB (see CouchDB) Apache Curator (see Curator) Apache Drill (see Drill) Apache Flink (see Flink) Apache Giraph (see Giraph) Apache Hadoop (see Hadoop) Apache HAWQ (see HAWQ) Apache HBase (see HBase) Apache Helix (see Helix) Apache Hive (see Hive) Apache Impala (see Impala) Apache Jena (see Jena) Apache Kafka (see Kafka) Apache Lucene (see Lucene) Apache MADlib (see MADlib) Apache Mahout (see Mahout) Apache Oozie (see Oozie) Apache Parquet (see Parquet) Apache Qpid (see Qpid) Apache Samza (see Samza) Apache Solr (see Solr) Apache Spark (see Spark) 560 | Index Apache Storm (see Storm) Apache Tajo (see Tajo) Apache Tez (see Tez) Apache Thrift (see Thrift) Apache ZooKeeper (see ZooKeeper) Apama (stream analytics), 466 append-only B-trees, 82, 242 append-only files (see logs) Application Programming Interfaces (APIs), 5, 27 for batch processing, 403 for change streams, 456 for distributed transactions, 361 for graph processing, 425 for services, 131-136 (see also services) evolvability, 136 RESTful, 133 SOAP, 133 application state (see state) approximate search (see similarity search) archival storage, data from databases, 131 arcs (see edges) arithmetic mean, 14 ASCII text, 119, 395 ASN.1 (schema language), 127 asynchronous networks, 278, 553 comparison to synchronous networks, 284 formal model, 307 asynchronous replication, 154, 553 conflict detection, 172 data loss on failover, 157 reads from asynchronous follower, 162 Asynchronous Transfer Mode (ATM), 285 atomic broadcast (see total order broadcast) atomic clocks (caesium clocks), 294, 295 (see also clocks) atomicity (concurrency), 553 atomic increment-and-get, 351 compare-and-set, 245, 327 (see also compare-and-set operations) replicated operations, 246 write operations, 243 atomicity (transactions), 223, 228, 553 atomic commit, 353 avoiding, 523, 528 blocking and nonblocking, 359 in stream processing, 360, 477 maintaining derived data, 453 for multi-object transactions, 229 for single-object writes, 230 auditability, 528-533 designing for, 531 self-auditing systems, 530 through immutability, 460 tools for auditable data systems, 532 availability, 8 (see also fault tolerance) in CAP theorem, 337 in service level agreements (SLAs), 15 Avro (data format), 122-127 code generation, 127 dynamically generated schemas, 126 object container files, 125, 131, 414 reader determining writer’s schema, 125 schema evolution, 123 use in Hadoop, 414 awk (Unix tool), 391 AWS (see Amazon Web Services) Azure (see Microsoft) B B-trees (indexes), 79-83 append-only/copy-on-write variants, 82, 242 branching factor, 81 comparison to LSM-trees, 83-85 crash recovery, 82 growing by splitting a page, 81 optimizations, 82 similarity to dynamic partitioning, 212 backpressure, 441, 553 in TCP, 282 backups database snapshot for replication, 156 integrity of, 530 snapshot isolation for, 238 use for ETL processes, 405 backward compatibility, 112 BASE, contrast to ACID, 223 bash shell (Unix), 70, 395, 503 batch processing, 28, 389-431, 553 combining with stream processing lambda architecture, 497 unifying technologies, 498 comparison to MPP databases, 414-418 comparison to stream processing, 464 comparison to Unix, 413-414 dataflow engines, 421-423 fault tolerance, 406, 414, 422, 442 for data integration, 494-498 graphs and iterative processing, 424-426 high-level APIs and languages, 403, 426-429 log-based messaging and, 451 maintaining derived state, 495 MapReduce and distributed filesystems, 397-413 (see also MapReduce) measuring performance, 13, 390 outputs, 411-413 key-value stores, 412 search indexes, 411 using Unix tools (example), 391-394 Bayou (database), 522 Beam (dataflow library), 498 bias, 534 big ball of mud, 20 Bigtable data model, 41, 99 binary data encodings, 115-128 Avro, 122-127 MessagePack, 116-117 Thrift and Protocol Buffers, 117-121 binary encoding based on schemas, 127 by network drivers, 128 binary strings, lack of support in JSON and XML, 114 BinaryProtocol encoding (Thrift), 118 Bitcask (storage engine), 72 crash recovery, 74 Bitcoin (cryptocurrency), 532 Byzantine fault tolerance, 305 concurrency bugs in exchanges, 233 bitmap indexes, 97 blockchains, 532 Byzantine fault tolerance, 305 blocking atomic commit, 359 Bloom (programming language), 504 Bloom filter (algorithm), 79, 466 BookKeeper (replicated log), 372 Bottled Water (change data capture), 455 bounded datasets, 430, 439, 553 (see also batch processing) bounded delays, 553 in networks, 285 process pauses, 298 broadcast hash joins, 409 Index | 561 brokerless messaging, 442 Brubeck (metrics aggregator), 442 BTM (transaction coordinator), 356 bulk synchronous parallel (BSP) model, 425 bursty network traffic patterns, 285 business data processing, 28, 90, 390 byte sequence, encoding data in, 112 Byzantine faults, 304-306, 307, 553 Byzantine fault-tolerant systems, 305, 532 Byzantine Generals Problem, 304 consensus algorithms and, 366 C caches, 89, 553 and materialized views, 101 as derived data, 386, 499-504 database as cache of transaction log, 460 in CPUs, 99, 338, 428 invalidation and maintenance, 452, 467 linearizability, 324 CAP theorem, 336-338, 554 Cascading (batch processing), 419, 427 hash joins, 409 workflows, 403 cascading failures, 9, 214, 281 Cascalog (batch processing), 60 Cassandra (database) column-family data model, 41, 99 compaction strategy, 79 compound primary key, 204 gossip protocol, 216 hash partitioning, 203-205 last-write-wins conflict resolution, 186, 292 leaderless replication, 177 linearizability, lack of, 335 log-structured storage, 78 multi-datacenter support, 184 partitioning scheme, 213 secondary indexes, 207 sloppy quorums, 184 cat (Unix tool), 391 causal context, 191 (see also causal dependencies) causal dependencies, 186-191 capturing, 191, 342, 494, 514 by total ordering, 493 causal ordering, 339 in transactions, 262 sending message to friends (example), 494 562 | Index causality, 554 causal ordering, 339-343 linearizability and, 342 total order consistent with, 344, 345 consistency with, 344-347 consistent snapshots, 340 happens-before relationship, 186 in serializable transactions, 262-265 mismatch with clocks, 292 ordering events to capture, 493 violations of, 165, 176, 292, 340 with synchronized clocks, 294 CEP (see complex event processing) certificate transparency, 532 chain replication, 155 linearizable reads, 351 change data capture, 160, 454 API support for change streams, 456 comparison to event sourcing, 457 implementing, 454 initial snapshot, 455 log compaction, 456 changelogs, 460 change data capture, 454 for operator state, 479 generating with triggers, 455 in stream joins, 474 log compaction, 456 maintaining derived state, 452 Chaos Monkey, 7, 280 checkpointing in batch processors, 422, 426 in high-performance computing, 275 in stream processors, 477, 523 chronicle data model, 458 circuit-switched networks, 284 circular buffers, 450 circular replication topologies, 175 clickstream data, analysis of, 404 clients calling services, 131 pushing state changes to, 512 request routing, 214 stateful and offline-capable, 170, 511 clocks, 287-299 atomic (caesium) clocks, 294, 295 confidence interval, 293-295 for global snapshots, 294 logical (see logical clocks) skew, 291-294, 334 slewing, 289 synchronization and accuracy, 289-291 synchronization using GPS, 287, 290, 294, 295 time-of-day versus monotonic clocks, 288 timestamping events, 471 cloud computing, 146, 275 need for service discovery, 372 network glitches, 279 shared resources, 284 single-machine reliability, 8 Cloudera Impala (see Impala) clustered indexes, 86 CODASYL model, 36 (see also network model) code generation with Avro, 127 with Thrift and Protocol Buffers, 118 with WSDL, 133 collaborative editing multi-leader replication and, 170 column families (Bigtable), 41, 99 column-oriented storage, 95-101 column compression, 97 distinction between column families and, 99 in batch processors, 428 Parquet, 96, 131, 414 sort order in, 99-100 vectorized processing, 99, 428 writing to, 101 comma-separated values (see CSV) command query responsibility segregation (CQRS), 462 commands (event sourcing), 459 commits (transactions), 222 atomic commit, 354-355 (see also atomicity; transactions) read committed isolation, 234 three-phase commit (3PC), 359 two-phase commit (2PC), 355-359 commutative operations, 246 compaction of changelogs, 456 (see also log compaction) for stream operator state, 479 of log-structured storage, 73 issues with, 84 size-tiered and leveled approaches, 79 CompactProtocol encoding (Thrift), 119 compare-and-set operations, 245, 327 implementing locks, 370 implementing uniqueness constraints, 331 implementing with total order broadcast, 350 relation to consensus, 335, 350, 352, 374 relation to transactions, 230 compatibility, 112, 128 calling services, 136 properties of encoding formats, 139 using databases, 129-131 using message-passing, 138 compensating transactions, 355, 461, 526 complex event processing (CEP), 465 complexity distilling in theoretical models, 310 hiding using abstraction, 27 of software systems, managing, 20 composing data systems (see unbundling data‐ bases) compute-intensive applications, 3, 275 concatenated indexes, 87 in Cassandra, 204 Concord (stream processor), 466 concurrency actor programming model, 138, 468 (see also message-passing) bugs from weak transaction isolation, 233 conflict resolution, 171, 174 detecting concurrent writes, 184-191 dual writes, problems with, 453 happens-before relationship, 186 in replicated systems, 161-191, 324-338 lost updates, 243 multi-version concurrency control (MVCC), 239 optimistic concurrency control, 261 ordering of operations, 326, 341 reducing, through event logs, 351, 462, 507 time and relativity, 187 transaction isolation, 225 write skew (transaction isolation), 246-251 conflict-free replicated datatypes (CRDTs), 174 conflicts conflict detection, 172 causal dependencies, 186, 342 in consensus algorithms, 368 in leaderless replication, 184 Index | 563 in log-based systems, 351, 521 in nonlinearizable systems, 343 in serializable snapshot isolation (SSI), 264 in two-phase commit, 357, 364 conflict resolution automatic conflict resolution, 174 by aborting transactions, 261 by apologizing, 527 convergence, 172-174 in leaderless systems, 190 last write wins (LWW), 186, 292 using atomic operations, 246 using custom logic, 173 determining what is a conflict, 174, 522 in multi-leader replication, 171-175 avoiding conflicts, 172 lost updates, 242-246 materializing, 251 relation to operation ordering, 339 write skew (transaction isolation), 246-251 congestion (networks) avoidance, 282 limiting accuracy of clocks, 293 queueing delays, 282 consensus, 321, 364-375, 554 algorithms, 366-368 preventing split brain, 367 safety and liveness properties, 365 using linearizable operations, 351 cost of, 369 distributed transactions, 352-375 in practice, 360-364 two-phase commit, 354-359 XA transactions, 361-364 impossibility of, 353 membership and coordination services, 370-373 relation to compare-and-set, 335, 350, 352, 374 relation to replication, 155, 349 relation to uniqueness constraints, 521 consistency, 224, 524 across different databases, 157, 452, 462, 492 causal, 339-348, 493 consistent prefix reads, 165-167 consistent snapshots, 156, 237-242, 294, 455, 500 (see also snapshots) 564 | Index crash recovery, 82 enforcing constraints (see constraints) eventual, 162, 322 (see also eventual consistency) in ACID transactions, 224, 529 in CAP theorem, 337 linearizability, 324-338 meanings of, 224 monotonic reads, 164-165 of secondary indexes, 231, 241, 354, 491, 500 ordering guarantees, 339-352 read-after-write, 162-164 sequential, 351 strong (see linearizability) timeliness and integrity, 524 using quorums, 181, 334 consistent hashing, 204 consistent prefix reads, 165 constraints (databases), 225, 248 asynchronously checked, 526 coordination avoidance, 527 ensuring idempotence, 519 in log-based systems, 521-524 across multiple partitions, 522 in two-phase commit, 355, 357 relation to consensus, 374, 521 relation to event ordering, 347 requiring linearizability, 330 Consul (service discovery), 372 consumers (message streams), 137, 440 backpressure, 441 consumer offsets in logs, 449 failures, 445, 449 fan-out, 11, 445, 448 load balancing, 444, 448 not keeping up with producers, 441, 450, 502 context switches, 14, 297 convergence (conflict resolution), 172-174, 322 coordination avoidance, 527 cross-datacenter, 168, 493 cross-partition ordering, 256, 294, 348, 523 services, 330, 370-373 coordinator (in 2PC), 356 failure, 358 in XA transactions, 361-364 recovery, 363 copy-on-write (B-trees), 82, 242 CORBA (Common Object Request Broker Architecture), 134 correctness, 6 auditability, 528-533 Byzantine fault tolerance, 305, 532 dealing with partial failures, 274 in log-based systems, 521-524 of algorithm within system model, 308 of compensating transactions, 355 of consensus, 368 of derived data, 497, 531 of immutable data, 461 of personal data, 535, 540 of time, 176, 289-295 of transactions, 225, 515, 529 timeliness and integrity, 524-528 corruption of data detecting, 519, 530-533 due to pathological memory access, 529 due to radiation, 305 due to split brain, 158, 302 due to weak transaction isolation, 233 formalization in consensus, 366 integrity as absence of, 524 network packets, 306 on disks, 227 preventing using write-ahead logs, 82 recovering from, 414, 460 Couchbase (database) durability, 89 hash partitioning, 203-204, 211 rebalancing, 213 request routing, 216 CouchDB (database) B-tree storage, 242 change feed, 456 document data model, 31 join support, 34 MapReduce support, 46, 400 replication, 170, 173 covering indexes, 86 CPUs cache coherence and memory barriers, 338 caching and pipelining, 99, 428 increasing parallelism, 43 CRDTs (see conflict-free replicated datatypes) CREATE INDEX statement (SQL), 85, 500 credit rating agencies, 535 Crunch (batch processing), 419, 427 hash joins, 409 sharded joins, 408 workflows, 403 cryptography defense against attackers, 306 end-to-end encryption and authentication, 519, 543 proving integrity of data, 532 CSS (Cascading Style Sheets), 44 CSV (comma-separated values), 70, 114, 396 Curator (ZooKeeper recipes), 330, 371 curl (Unix tool), 135, 397 cursor stability, 243 Cypher (query language), 52 comparison to SPARQL, 59 D data corruption (see corruption of data) data cubes, 102 data formats (see encoding) data integration, 490-498, 543 batch and stream processing, 494-498 lambda architecture, 497 maintaining derived state, 495 reprocessing data, 496 unifying, 498 by unbundling databases, 499-515 comparison to federated databases, 501 combining tools by deriving data, 490-494 derived data versus distributed transac‐ tions, 492 limits of total ordering, 493 ordering events to capture causality, 493 reasoning about dataflows, 491 need for, 385 data lakes, 415 data locality (see locality) data models, 27-64 graph-like models, 49-63 Datalog language, 60-63 property graphs, 50 RDF and triple-stores, 55-59 query languages, 42-48 relational model versus document model, 28-42 data protection regulations, 542 data systems, 3 about, 4 Index | 565 concerns when designing, 5 future of, 489-544 correctness, constraints, and integrity, 515-533 data integration, 490-498 unbundling databases, 499-515 heterogeneous, keeping in sync, 452 maintainability, 18-22 possible faults in, 221 reliability, 6-10 hardware faults, 7 human errors, 9 importance of, 10 software errors, 8 scalability, 10-18 unreliable clocks, 287-299 data warehousing, 91-95, 554 comparison to data lakes, 415 ETL (extract-transform-load), 92, 416, 452 keeping data systems in sync, 452 schema design, 93 slowly changing dimension (SCD), 476 data-intensive applications, 3 database triggers (see triggers) database-internal distributed transactions, 360, 364, 477 databases archival storage, 131 comparison of message brokers to, 443 dataflow through, 129 end-to-end argument for, 519-520 checking integrity, 531 inside-out, 504 (see also unbundling databases) output from batch workflows, 412 relation to event streams, 451-464 (see also changelogs) API support for change streams, 456, 506 change data capture, 454-457 event sourcing, 457-459 keeping systems in sync, 452-453 philosophy of immutable events, 459-464 unbundling, 499-515 composing data storage technologies, 499-504 designing applications around dataflow, 504-509 566 | Index observing derived state, 509-515 datacenters geographically distributed, 145, 164, 278, 493 multi-tenancy and shared resources, 284 network architecture, 276 network faults, 279 replication across multiple, 169 leaderless replication, 184 multi-leader replication, 168, 335 dataflow, 128-139, 504-509 correctness of dataflow systems, 525 differential, 504 message-passing, 136-139 reasoning about, 491 through databases, 129 through services, 131-136 dataflow engines, 421-423 comparison to stream processing, 464 directed acyclic graphs (DAG), 424 partitioning, approach to, 429 support for declarative queries, 427 Datalog (query language), 60-63 datatypes binary strings in XML and JSON, 114 conflict-free, 174 in Avro encodings, 122 in Thrift and Protocol Buffers, 121 numbers in XML and JSON, 114 Datomic (database) B-tree storage, 242 data model, 50, 57 Datalog query language, 60 excision (deleting data), 463 languages for transactions, 255 serial execution of transactions, 253 deadlocks detection, in two-phase commit (2PC), 364 in two-phase locking (2PL), 258 Debezium (change data capture), 455 declarative languages, 42, 554 Bloom, 504 CSS and XSL, 44 Cypher, 52 Datalog, 60 for batch processing, 427 recursive SQL queries, 53 relational algebra and SQL, 42 SPARQL, 59 delays bounded network delays, 285 bounded process pauses, 298 unbounded network delays, 282 unbounded process pauses, 296 deleting data, 463 denormalization (data representation), 34, 554 costs, 39 in derived data systems, 386 materialized views, 101 updating derived data, 228, 231, 490 versus normalization, 462 derived data, 386, 439, 554 from change data capture, 454 in event sourcing, 458-458 maintaining derived state through logs, 452-457, 459-463 observing, by subscribing to streams, 512 outputs of batch and stream processing, 495 through application code, 505 versus distributed transactions, 492 deterministic operations, 255, 274, 554 accidental nondeterminism, 423 and fault tolerance, 423, 426 and idempotence, 478, 492 computing derived data, 495, 526, 531 in state machine replication, 349, 452, 458 joins, 476 DevOps, 394 differential dataflow, 504 dimension tables, 94 dimensional modeling (see star schemas) directed acyclic graphs (DAGs), 424 dirty reads (transaction isolation), 234 dirty writes (transaction isolation), 235 discrimination, 534 disks (see hard disks) distributed actor frameworks, 138 distributed filesystems, 398-399 decoupling from query engines, 417 indiscriminately dumping data into, 415 use by MapReduce, 402 distributed systems, 273-312, 554 Byzantine faults, 304-306 cloud versus supercomputing, 275 detecting network faults, 280 faults and partial failures, 274-277 formalization of consensus, 365 impossibility results, 338, 353 issues with failover, 157 limitations of distributed transactions, 363 multi-datacenter, 169, 335 network problems, 277-286 quorums, relying on, 301 reasons for using, 145, 151 synchronized clocks, relying on, 291-295 system models, 306-310 use of clocks and time, 287 distributed transactions (see transactions) Django (web framework), 232 DNS (Domain Name System), 216, 372 Docker (container manager), 506 document data model, 30-42 comparison to relational model, 38-42 document references, 38, 403 document-oriented databases, 31 many-to-many relationships and joins, 36 multi-object transactions, need for, 231 versus relational model convergence of models, 41 data locality, 41 document-partitioned indexes, 206, 217, 411 domain-driven design (DDD), 457 DRBD (Distributed Replicated Block Device), 153 drift (clocks), 289 Drill (query engine), 93 Druid (database), 461 Dryad (dataflow engine), 421 dual writes, problems with, 452, 507 duplicates, suppression of, 517 (see also idempotence) using a unique ID, 518, 522 durability (transactions), 226, 554 duration (time), 287 measurement with monotonic clocks, 288 dynamic partitioning, 212 dynamically typed languages analogy to schema-on-read, 40 code generation and, 127 Dynamo-style databases (see leaderless replica‐ tion) E edges (in graphs), 49, 403 property graph model, 50 edit distance (full-text search), 88 effectively-once semantics, 476, 516 Index | 567 (see also exactly-once semantics) preservation of integrity, 525 elastic systems, 17 Elasticsearch (search server) document-partitioned indexes, 207 partition rebalancing, 211 percolator (stream search), 467 usage example, 4 use of Lucene, 79 ElephantDB (database), 413 Elm (programming language), 504, 512 encodings (data formats), 111-128 Avro, 122-127 binary variants of JSON and XML, 115 compatibility, 112 calling services, 136 using databases, 129-131 using message-passing, 138 defined, 113 JSON, XML, and CSV, 114 language-specific formats, 113 merits of schemas, 127 representations of data, 112 Thrift and Protocol Buffers, 117-121 end-to-end argument, 277, 519-520 checking integrity, 531 publish/subscribe streams, 512 enrichment (stream), 473 Enterprise JavaBeans (EJB), 134 entities (see vertices) epoch (consensus algorithms), 368 epoch (Unix timestamps), 288 equi-joins, 403 erasure coding (error correction), 398 Erlang OTP (actor framework), 139 error handling for network faults, 280 in transactions, 231 error-correcting codes, 277, 398 Esper (CEP engine), 466 etcd (coordination service), 370-373 linearizable operations, 333 locks and leader election, 330 quorum reads, 351 service discovery, 372 use of Raft algorithm, 349, 353 Ethereum (blockchain), 532 Ethernet (networks), 276, 278, 285 packet checksums, 306, 519 568 | Index Etherpad (collaborative editor), 170 ethics, 533-543 code of ethics and professional practice, 533 legislation and self-regulation, 542 predictive analytics, 533-536 amplifying bias, 534 feedback loops, 536 privacy and tracking, 536-543 consent and freedom of choice, 538 data as assets and power, 540 meaning of privacy, 539 surveillance, 537 respect, dignity, and agency, 543, 544 unintended consequences, 533, 536 ETL (extract-transform-load), 92, 405, 452, 554 use of Hadoop for, 416 event sourcing, 457-459 commands and events, 459 comparison to change data capture, 457 comparison to lambda architecture, 497 deriving current state from event log, 458 immutability and auditability, 459, 531 large, reliable data systems, 519, 526 Event Store (database), 458 event streams (see streams) events, 440 deciding on total order of, 493 deriving views from event log, 461 difference to commands, 459 event time versus processing time, 469, 477, 498 immutable, advantages of, 460, 531 ordering to capture causality, 493 reads as, 513 stragglers, 470, 498 timestamp of, in stream processing, 471 EventSource (browser API), 512 eventual consistency, 152, 162, 308, 322 (see also conflicts) and perpetual inconsistency, 525 evolvability, 21, 111 calling services, 136 graph-structured data, 52 of databases, 40, 129-131, 461, 497 of message-passing, 138 reprocessing data, 496, 498 schema evolution in Avro, 123 schema evolution in Thrift and Protocol Buffers, 120 schema-on-read, 39, 111, 128 exactly-once semantics, 360, 476, 516 parity with batch processors, 498 preservation of integrity, 525 exclusive mode (locks), 258 eXtended Architecture transactions (see XA transactions) extract-transform-load (see ETL) F Facebook Presto (query engine), 93 React, Flux, and Redux (user interface libra‐ ries), 512 social graphs, 49 Wormhole (change data capture), 455 fact tables, 93 failover, 157, 554 (see also leader-based replication) in leaderless replication, absence of, 178 leader election, 301, 348, 352 potential problems, 157 failures amplification by distributed transactions, 364, 495 failure detection, 280 automatic rebalancing causing cascading failures, 214 perfect failure detectors, 359 timeouts and unbounded delays, 282, 284 using ZooKeeper, 371 faults versus, 7 partial failures in distributed systems, 275-277, 310 fan-out (messaging systems), 11, 445 fault tolerance, 6-10, 555 abstractions for, 321 formalization in consensus, 365-369 use of replication, 367 human fault tolerance, 414 in batch processing, 406, 414, 422, 425 in log-based systems, 520, 524-526 in stream processing, 476-479 atomic commit, 477 idempotence, 478 maintaining derived state, 495 microbatching and checkpointing, 477 rebuilding state after a failure, 478 of distributed transactions, 362-364 transaction atomicity, 223, 354-361 faults, 6 Byzantine faults, 304-306 failures versus, 7 handled by transactions, 221 handling in supercomputers and cloud computing, 275 hardware, 7 in batch processing versus distributed data‐ bases, 417 in distributed systems, 274-277 introducing deliberately, 7, 280 network faults, 279-281 asymmetric faults, 300 detecting, 280 tolerance of, in multi-leader replication, 169 software errors, 8 tolerating (see fault tolerance) federated databases, 501 fence (CPU instruction), 338 fencing (preventing split brain), 158, 302-304 generating fencing tokens, 349, 370 properties of fencing tokens, 308 stream processors writing to databases, 478, 517 Fibre Channel (networks), 398 field tags (Thrift and Protocol Buffers), 119-121 file descriptors (Unix), 395 financial data, 460 Firebase (database), 456 Flink (processing framework), 421-423 dataflow APIs, 427 fault tolerance, 422, 477, 479 Gelly API (graph processing), 425 integration of batch and stream processing, 495, 498 machine learning, 428 query optimizer, 427 stream processing, 466 flow control, 282, 441, 555 FLP result (on consensus), 353 FlumeJava (dataflow library), 403, 427 followers, 152, 555 (see also leader-based replication) foreign keys, 38, 403 forward compatibility, 112 forward decay (algorithm), 16 Index | 569 Fossil (version control system), 463 shunning (deleting data), 463 FoundationDB (database) serializable transactions, 261, 265, 364 fractal trees, 83 full table scans, 403 full-text search, 555 and fuzzy indexes, 88 building search indexes, 411 Lucene storage engine, 79 functional reactive programming (FRP), 504 functional requirements, 22 futures (asynchronous operations), 135 fuzzy search (see similarity search) G garbage collection immutability and, 463 process pauses for, 14, 296-299, 301 (see also process pauses) genome analysis, 63, 429 geographically distributed datacenters, 145, 164, 278, 493 geospatial indexes, 87 Giraph (graph processing), 425 Git (version control system), 174, 342, 463 GitHub, postmortems, 157, 158, 309 global indexes (see term-partitioned indexes) GlusterFS (distributed filesystem), 398 GNU Coreutils (Linux), 394 GoldenGate (change data capture), 161, 170, 455 (see also Oracle) Google Bigtable (database) data model (see Bigtable data model) partitioning scheme, 199, 202 storage layout, 78 Chubby (lock service), 370 Cloud Dataflow (stream processor), 466, 477, 498 (see also Beam) Cloud Pub/Sub (messaging), 444, 448 Docs (collaborative editor), 170 Dremel (query engine), 93, 96 FlumeJava (dataflow library), 403, 427 GFS (distributed file system), 398 gRPC (RPC framework), 135 MapReduce (batch processing), 390 570 | Index (see also MapReduce) building search indexes, 411 task preemption, 418 Pregel (graph processing), 425 Spanner (see Spanner) TrueTime (clock API), 294 gossip protocol, 216 government use of data, 541 GPS (Global Positioning System) use for clock synchronization, 287, 290, 294, 295 GraphChi (graph processing), 426 graphs, 555 as data models, 49-63 example of graph-structured data, 49 property graphs, 50 RDF and triple-stores, 55-59 versus the network model, 60 processing and analysis, 424-426 fault tolerance, 425 Pregel processing model, 425 query languages Cypher, 52 Datalog, 60-63 recursive SQL queries, 53 SPARQL, 59-59 Gremlin (graph query language), 50 grep (Unix tool), 392 GROUP BY clause (SQL), 406 grouping records in MapReduce, 406 handling skew, 407 H Hadoop (data infrastructure) comparison to distributed databases, 390 comparison to MPP databases, 414-418 comparison to Unix, 413-414, 499 diverse processing models in ecosystem, 417 HDFS distributed filesystem (see HDFS) higher-level tools, 403 join algorithms, 403-410 (see also MapReduce) MapReduce (see MapReduce) YARN (see YARN) happens-before relationship, 340 capturing, 187 concurrency and, 186 hard disks access patterns, 84 detecting corruption, 519, 530 faults in, 7, 227 sequential write throughput, 75, 450 hardware faults, 7 hash indexes, 72-75 broadcast hash joins, 409 partitioned hash joins, 409 hash partitioning, 203-205, 217 consistent hashing, 204 problems with hash mod N, 210 range queries, 204 suitable hash functions, 203 with fixed number of partitions, 210 HAWQ (database), 428 HBase (database) bug due to lack of fencing, 302 bulk loading, 413 column-family data model, 41, 99 dynamic partitioning, 212 key-range partitioning, 202 log-structured storage, 78 request routing, 216 size-tiered compaction, 79 use of HDFS, 417 use of ZooKeeper, 370 HDFS (Hadoop Distributed File System), 398-399 (see also distributed filesystems) checking data integrity, 530 decoupling from query engines, 417 indiscriminately dumping data into, 415 metadata about datasets, 410 NameNode, 398 use by Flink, 479 use by HBase, 212 use by MapReduce, 402 HdrHistogram (numerical library), 16 head (Unix tool), 392 head vertex (property graphs), 51 head-of-line blocking, 15 heap files (databases), 86 Helix (cluster manager), 216 heterogeneous distributed transactions, 360, 364 heuristic decisions (in 2PC), 363 Hibernate (object-relational mapper), 30 hierarchical model, 36 high availability (see fault tolerance) high-frequency trading, 290, 299 high-performance computing (HPC), 275 hinted handoff, 183 histograms, 16 Hive (query engine), 419, 427 for data warehouses, 93 HCatalog and metastore, 410 map-side joins, 409 query optimizer, 427 skewed joins, 408 workflows, 403 Hollerith machines, 390 hopping windows (stream processing), 472 (see also windows) horizontal scaling (see scaling out) HornetQ (messaging), 137, 444 distributed transaction support, 361 hot spots, 201 due to celebrities, 205 for time-series data, 203 in batch processing, 407 relieving, 205 hot standbys (see leader-based replication) HTTP, use in APIs (see services) human errors, 9, 279, 414 HyperDex (database), 88 HyperLogLog (algorithm), 466 I I/O operations, waiting for, 297 IBM DB2 (database) distributed transaction support, 361 recursive query support, 54 serializable isolation, 242, 257 XML and JSON support, 30, 42 electromechanical card-sorting machines, 390 IMS (database), 36 imperative query APIs, 46 InfoSphere Streams (CEP engine), 466 MQ (messaging), 444 distributed transaction support, 361 System R (database), 222 WebSphere (messaging), 137 idempotence, 134, 478, 555 by giving operations unique IDs, 518, 522 idempotent operations, 517 immutability advantages of, 460, 531 Index | 571 deriving state from event log, 459-464 for crash recovery, 75 in B-trees, 82, 242 in event sourcing, 457 inputs to Unix commands, 397 limitations of, 463 Impala (query engine) for data warehouses, 93 hash joins, 409 native code generation, 428 use of HDFS, 417 impedance mismatch, 29 imperative languages, 42 setting element styles (example), 45 in doubt (transaction status), 358 holding locks, 362 orphaned transactions, 363 in-memory databases, 88 durability, 227 serial transaction execution, 253 incidents cascading failures, 9 crashes due to leap seconds, 290 data corruption and financial losses due to concurrency bugs, 233 data corruption on hard disks, 227 data loss due to last-write-wins, 173, 292 data on disks unreadable, 309 deleted items reappearing, 174 disclosure of sensitive data due to primary key reuse, 157 errors in transaction serializability, 529 gigabit network interface with 1 Kb/s throughput, 311 network faults, 279 network interface dropping only inbound packets, 279 network partitions and whole-datacenter failures, 275 poor handling of network faults, 280 sending message to ex-partner, 494 sharks biting undersea cables, 279 split brain due to 1-minute packet delay, 158, 279 vibrations in server rack, 14 violation of uniqueness constraint, 529 indexes, 71, 555 and snapshot isolation, 241 as derived data, 386, 499-504 572 | Index B-trees, 79-83 building in batch processes, 411 clustered, 86 comparison of B-trees and LSM-trees, 83-85 concatenated, 87 covering (with included columns), 86 creating, 500 full-text search, 88 geospatial, 87 hash, 72-75 index-range locking, 260 multi-column, 87 partitioning and secondary indexes, 206-209, 217 secondary, 85 (see also secondary indexes) problems with dual writes, 452, 491 SSTables and LSM-trees, 76-79 updating when data changes, 452, 467 Industrial Revolution, 541 InfiniBand (networks), 285 InfiniteGraph (database), 50 InnoDB (storage engine) clustered index on primary key, 86 not preventing lost updates, 245 preventing write skew, 248, 257 serializable isolation, 257 snapshot isolation support, 239 inside-out databases, 504 (see also unbundling databases) integrating different data systems (see data integration) integrity, 524 coordination-avoiding data systems, 528 correctness of dataflow systems, 525 in consensus formalization, 365 integrity checks, 530 (see also auditing) end-to-end, 519, 531 use of snapshot isolation, 238 maintaining despite software bugs, 529 Interface Definition Language (IDL), 117, 122 intermediate state, materialization of, 420-423 internet services, systems for implementing, 275 invariants, 225 (see also constraints) inversion of control, 396 IP (Internet Protocol) unreliability of, 277 ISDN (Integrated Services Digital Network), 284 isolation (in transactions), 225, 228, 555 correctness and, 515 for single-object writes, 230 serializability, 251-266 actual serial execution, 252-256 serializable snapshot isolation (SSI), 261-266 two-phase locking (2PL), 257-261 violating, 228 weak isolation levels, 233-251 preventing lost updates, 242-246 read committed, 234-237 snapshot isolation, 237-242 iterative processing, 424-426 J Java Database Connectivity (JDBC) distributed transaction support, 361 network drivers, 128 Java Enterprise Edition (EE), 134, 356, 361 Java Message Service (JMS), 444 (see also messaging systems) comparison to log-based messaging, 448, 451 distributed transaction support, 361 message ordering, 446 Java Transaction API (JTA), 355, 361 Java Virtual Machine (JVM) bytecode generation, 428 garbage collection pauses, 296 process reuse in batch processors, 422 JavaScript in MapReduce querying, 46 setting element styles (example), 45 use in advanced queries, 48 Jena (RDF framework), 57 Jepsen (fault tolerance testing), 515 jitter (network delay), 284 joins, 555 by index lookup, 403 expressing as relational operators, 427 in relational and document databases, 34 MapReduce map-side joins, 408-410 broadcast hash joins, 409 merge joins, 410 partitioned hash joins, 409 MapReduce reduce-side joins, 403-408 handling skew, 407 sort-merge joins, 405 parallel execution of, 415 secondary indexes and, 85 stream joins, 472-476 stream-stream join, 473 stream-table join, 473 table-table join, 474 time-dependence of, 475 support in document databases, 42 JOTM (transaction coordinator), 356 JSON Avro schema representation, 122 binary variants, 115 for application data, issues with, 114 in relational databases, 30, 42 representing a résumé (example), 31 Juttle (query language), 504 K k-nearest neighbors, 429 Kafka (messaging), 137, 448 Kafka Connect (database integration), 457, 461 Kafka Streams (stream processor), 466, 467 fault tolerance, 479 leader-based replication, 153 log compaction, 456, 467 message offsets, 447, 478 request routing, 216 transaction support, 477 usage example, 4 Ketama (partitioning library), 213 key-value stores, 70 as batch process output, 412 hash indexes, 72-75 in-memory, 89 partitioning, 201-205 by hash of key, 203, 217 by key range, 202, 217 dynamic partitioning, 212 skew and hot spots, 205 Kryo (Java), 113 Kubernetes (cluster manager), 418, 506 L lambda architecture, 497 Lamport timestamps, 345 Index | 573 Large Hadron Collider (LHC), 64 last write wins (LWW), 173, 334 discarding concurrent writes, 186 problems with, 292 prone to lost updates, 246 late binding, 396 latency instability under two-phase locking, 259 network latency and resource utilization, 286 response time versus, 14 tail latency, 15, 207 leader-based replication, 152-161 (see also replication) failover, 157, 301 handling node outages, 156 implementation of replication logs change data capture, 454-457 (see also changelogs) statement-based, 158 trigger-based replication, 161 write-ahead log (WAL) shipping, 159 linearizability of operations, 333 locking and leader election, 330 log sequence number, 156, 449 read-scaling architecture, 161 relation to consensus, 367 setting up new followers, 155 synchronous versus asynchronous, 153-155 leaderless replication, 177-191 (see also replication) detecting concurrent writes, 184-191 capturing happens-before relationship, 187 happens-before relationship and concur‐ rency, 186 last write wins, 186 merging concurrently written values, 190 version vectors, 191 multi-datacenter, 184 quorums, 179-182 consistency limitations, 181-183, 334 sloppy quorums and hinted handoff, 183 read repair and anti-entropy, 178 leap seconds, 8, 290 in time-of-day clocks, 288 leases, 295 implementation with ZooKeeper, 370 574 | Index need for fencing, 302 ledgers, 460 distributed ledger technologies, 532 legacy systems, maintenance of, 18 less (Unix tool), 397 LevelDB (storage engine), 78 leveled compaction, 79 Levenshtein automata, 88 limping (partial failure), 311 linearizability, 324-338, 555 cost of, 335-338 CAP theorem, 336 memory on multi-core CPUs, 338 definition, 325-329 implementing with total order broadcast, 350 in ZooKeeper, 370 of derived data systems, 492, 524 avoiding coordination, 527 of different replication methods, 332-335 using quorums, 334 relying on, 330-332 constraints and uniqueness, 330 cross-channel timing dependencies, 331 locking and leader election, 330 stronger than causal consistency, 342 using to implement total order broadcast, 351 versus serializability, 329 LinkedIn Azkaban (workflow scheduler), 402 Databus (change data capture), 161, 455 Espresso (database), 31, 126, 130, 153, 216 Helix (cluster manager) (see Helix) profile (example), 30 reference to company entity (example), 34 Rest.li (RPC framework), 135 Voldemort (database) (see Voldemort) Linux, leap second bug, 8, 290 liveness properties, 308 LMDB (storage engine), 82, 242 load approaches to coping with, 17 describing, 11 load testing, 16 load balancing (messaging), 444 local indexes (see document-partitioned indexes) locality (data access), 32, 41, 555 in batch processing, 400, 405, 421 in stateful clients, 170, 511 in stream processing, 474, 478, 508, 522 location transparency, 134 in the actor model, 138 locks, 556 deadlock, 258 distributed locking, 301-304, 330 fencing tokens, 303 implementation with ZooKeeper, 370 relation to consensus, 374 for transaction isolation in snapshot isolation, 239 in two-phase locking (2PL), 257-261 making operations atomic, 243 performance, 258 preventing dirty writes, 236 preventing phantoms with index-range locks, 260, 265 read locks (shared mode), 236, 258 shared mode and exclusive mode, 258 in two-phase commit (2PC) deadlock detection, 364 in-doubt transactions holding locks, 362 materializing conflicts with, 251 preventing lost updates by explicit locking, 244 log sequence number, 156, 449 logic programming languages, 504 logical clocks, 293, 343, 494 for read-after-write consistency, 164 logical logs, 160 logs (data structure), 71, 556 advantages of immutability, 460 compaction, 73, 79, 456, 460 for stream operator state, 479 creating using total order broadcast, 349 implementing uniqueness constraints, 522 log-based messaging, 446-451 comparison to traditional messaging, 448, 451 consumer offsets, 449 disk space usage, 450 replaying old messages, 451, 496, 498 slow consumers, 450 using logs for message storage, 447 log-structured storage, 71-79 log-structured merge tree (see LSMtrees) replication, 152, 158-161 change data capture, 454-457 (see also changelogs) coordination with snapshot, 156 logical (row-based) replication, 160 statement-based replication, 158 trigger-based replication, 161 write-ahead log (WAL) shipping, 159 scalability limits, 493 loose coupling, 396, 419, 502 lost updates (see updates) LSM-trees (indexes), 78-79 comparison to B-trees, 83-85 Lucene (storage engine), 79 building indexes in batch processes, 411 similarity search, 88 Luigi (workflow scheduler), 402 LWW (see last write wins) M machine learning ethical considerations, 534 (see also ethics) iterative processing, 424 models derived from training data, 505 statistical and numerical algorithms, 428 MADlib (machine learning toolkit), 428 magic scaling sauce, 18 Mahout (machine learning toolkit), 428 maintainability, 18-22, 489 defined, 23 design principles for software systems, 19 evolvability (see evolvability) operability, 19 simplicity and managing complexity, 20 many-to-many relationships in document model versus relational model, 39 modeling as graphs, 49 many-to-one and many-to-many relationships, 33-36 many-to-one relationships, 34 MapReduce (batch processing), 390, 399-400 accessing external services within job, 404, 412 comparison to distributed databases designing for frequent faults, 417 diversity of processing models, 416 diversity of storage, 415 Index | 575 comparison to stream processing, 464 comparison to Unix, 413-414 disadvantages and limitations of, 419 fault tolerance, 406, 414, 422 higher-level tools, 403, 426 implementation in Hadoop, 400-403 the shuffle, 402 implementation in MongoDB, 46-48 machine learning, 428 map-side processing, 408-410 broadcast hash joins, 409 merge joins, 410 partitioned hash joins, 409 mapper and reducer functions, 399 materialization of intermediate state, 419-423 output of batch workflows, 411-413 building search indexes, 411 key-value stores, 412 reduce-side processing, 403-408 analysis of user activity events (exam‐ ple), 404 grouping records by same key, 406 handling skew, 407 sort-merge joins, 405 workflows, 402 marshalling (see encoding) massively parallel processing (MPP), 216 comparison to composing storage technolo‐ gies, 502 comparison to Hadoop, 414-418, 428 master-master replication (see multi-leader replication) master-slave replication (see leader-based repli‐ cation) materialization, 556 aggregate values, 101 conflicts, 251 intermediate state (batch processing), 420-423 materialized views, 101 as derived data, 386, 499-504 maintaining, using stream processing, 467, 475 Maven (Java build tool), 428 Maxwell (change data capture), 455 mean, 14 media monitoring, 467 median, 14 576 | Index meeting room booking (example), 249, 259, 521 membership services, 372 Memcached (caching server), 4, 89 memory in-memory databases, 88 durability, 227 serial transaction execution, 253 in-memory representation of data, 112 random bit-flips in, 529 use by indexes, 72, 77 memory barrier (CPU instruction), 338 MemSQL (database) in-memory storage, 89 read committed isolation, 236 memtable (in LSM-trees), 78 Mercurial (version control system), 463 merge joins, MapReduce map-side, 410 mergeable persistent data structures, 174 merging sorted files, 76, 402, 405 Merkle trees, 532 Mesos (cluster manager), 418, 506 message brokers (see messaging systems) message-passing, 136-139 advantages over direct RPC, 137 distributed actor frameworks, 138 evolvability, 138 MessagePack (encoding format), 116 messages exactly-once semantics, 360, 476 loss of, 442 using total order broadcast, 348 messaging systems, 440-451 (see also streams) backpressure, buffering, or dropping mes‐ sages, 441 brokerless messaging, 442 event logs, 446-451 comparison to traditional messaging, 448, 451 consumer offsets, 449 replaying old messages, 451, 496, 498 slow consumers, 450 message brokers, 443-446 acknowledgements and redelivery, 445 comparison to event logs, 448, 451 multiple consumers of same topic, 444 reliability, 442 uniqueness in log-based messaging, 522 Meteor (web framework), 456 microbatching, 477, 495 microservices, 132 (see also services) causal dependencies across services, 493 loose coupling, 502 relation to batch/stream processors, 389, 508 Microsoft Azure Service Bus (messaging), 444 Azure Storage, 155, 398 Azure Stream Analytics, 466 DCOM (Distributed Component Object Model), 134 MSDTC (transaction coordinator), 356 Orleans (see Orleans) SQL Server (see SQL Server) migrating (rewriting) data, 40, 130, 461, 497 modulus operator (%), 210 MongoDB (database) aggregation pipeline, 48 atomic operations, 243 BSON, 41 document data model, 31 hash partitioning (sharding), 203-204 key-range partitioning, 202 lack of join support, 34, 42 leader-based replication, 153 MapReduce support, 46, 400 oplog parsing, 455, 456 partition splitting, 212 request routing, 216 secondary indexes, 207 Mongoriver (change data capture), 455 monitoring, 10, 19 monotonic clocks, 288 monotonic reads, 164 MPP (see massively parallel processing) MSMQ (messaging), 361 multi-column indexes, 87 multi-leader replication, 168-177 (see also replication) handling write conflicts, 171 conflict avoidance, 172 converging toward a consistent state, 172 custom conflict resolution logic, 173 determining what is a conflict, 174 linearizability, lack of, 333 replication topologies, 175-177 use cases, 168 clients with offline operation, 170 collaborative editing, 170 multi-datacenter replication, 168, 335 multi-object transactions, 228 need for, 231 Multi-Paxos (total order broadcast), 367 multi-table index cluster tables (Oracle), 41 multi-tenancy, 284 multi-version concurrency control (MVCC), 239, 266 detecting stale MVCC reads, 263 indexes and snapshot isolation, 241 mutual exclusion, 261 (see also locks) MySQL (database) binlog coordinates, 156 binlog parsing for change data capture, 455 circular replication topology, 175 consistent snapshots, 156 distributed transaction support, 361 InnoDB storage engine (see InnoDB) JSON support, 30, 42 leader-based replication, 153 performance of XA transactions, 360 row-based replication, 160 schema changes in, 40 snapshot isolation support, 242 (see also InnoDB) statement-based replication, 159 Tungsten Replicator (multi-leader replica‐ tion), 170 conflict detection, 177 N nanomsg (messaging library), 442 Narayana (transaction coordinator), 356 NATS (messaging), 137 near-real-time (nearline) processing, 390 (see also stream processing) Neo4j (database) Cypher query language, 52 graph data model, 50 Nephele (dataflow engine), 421 netcat (Unix tool), 397 Netflix Chaos Monkey, 7, 280 Network Attached Storage (NAS), 146, 398 network model, 36 Index | 577 graph databases versus, 60 imperative query APIs, 46 Network Time Protocol (see NTP) networks congestion and queueing, 282 datacenter network topologies, 276 faults (see faults) linearizability and network delays, 338 network partitions, 279, 337 timeouts and unbounded delays, 281 next-key locking, 260 nodes (in graphs) (see vertices) nodes (processes), 556 handling outages in leader-based replica‐ tion, 156 system models for failure, 307 noisy neighbors, 284 nonblocking atomic commit, 359 nondeterministic operations accidental nondeterminism, 423 partial failures in distributed systems, 275 nonfunctional requirements, 22 nonrepeatable reads, 238 (see also read skew) normalization (data representation), 33, 556 executing joins, 39, 42, 403 foreign key references, 231 in systems of record, 386 versus denormalization, 462 NoSQL, 29, 499 transactions and, 223 Notation3 (N3), 56 npm (package manager), 428 NTP (Network Time Protocol), 287 accuracy, 289, 293 adjustments to monotonic clocks, 289 multiple server addresses, 306 numbers, in XML and JSON encodings, 114 O object-relational mapping (ORM) frameworks, 30 error handling and aborted transactions, 232 unsafe read-modify-write cycle code, 244 object-relational mismatch, 29 observer pattern, 506 offline systems, 390 (see also batch processing) 578 | Index stateful, offline-capable clients, 170, 511 offline-first applications, 511 offsets consumer offsets in partitioned logs, 449 messages in partitioned logs, 447 OLAP (online analytic processing), 91, 556 data cubes, 102 OLTP (online transaction processing), 90, 556 analytics queries versus, 411 workload characteristics, 253 one-to-many relationships, 30 JSON representation, 32 online systems, 389 (see also services) Oozie (workflow scheduler), 402 OpenAPI (service definition format), 133 OpenStack Nova (cloud infrastructure) use of ZooKeeper, 370 Swift (object storage), 398 operability, 19 operating systems versus databases, 499 operation identifiers, 518, 522 operational transformation, 174 operators, 421 flow of data between, 424 in stream processing, 464 optimistic concurrency control, 261 Oracle (database) distributed transaction support, 361 GoldenGate (change data capture), 161, 170, 455 lack of serializability, 226 leader-based replication, 153 multi-table index cluster tables, 41 not preventing write skew, 248 partitioned indexes, 209 PL/SQL language, 255 preventing lost updates, 245 read committed isolation, 236 Real Application Clusters (RAC), 330 recursive query support, 54 snapshot isolation support, 239, 242 TimesTen (in-memory database), 89 WAL-based replication, 160 XML support, 30 ordering, 339-352 by sequence numbers, 343-348 causal ordering, 339-343 partial order, 341 limits of total ordering, 493 total order broadcast, 348-352 Orleans (actor framework), 139 outliers (response time), 14 Oz (programming language), 504 P package managers, 428, 505 packet switching, 285 packets corruption of, 306 sending via UDP, 442 PageRank (algorithm), 49, 424 paging (see virtual memory) ParAccel (database), 93 parallel databases (see massively parallel pro‐ cessing) parallel execution of graph analysis algorithms, 426 queries in MPP databases, 216 Parquet (data format), 96, 131 (see also column-oriented storage) use in Hadoop, 414 partial failures, 275, 310 limping, 311 partial order, 341 partitioning, 199-218, 556 and replication, 200 in batch processing, 429 multi-partition operations, 514 enforcing constraints, 522 secondary index maintenance, 495 of key-value data, 201-205 by key range, 202 skew and hot spots, 205 rebalancing partitions, 209-214 automatic or manual rebalancing, 213 problems with hash mod N, 210 using dynamic partitioning, 212 using fixed number of partitions, 210 using N partitions per node, 212 replication and, 147 request routing, 214-216 secondary indexes, 206-209 document-based partitioning, 206 term-based partitioning, 208 serial execution of transactions and, 255 Paxos (consensus algorithm), 366 ballot number, 368 Multi-Paxos (total order broadcast), 367 percentiles, 14, 556 calculating efficiently, 16 importance of high percentiles, 16 use in service level agreements (SLAs), 15 Percona XtraBackup (MySQL tool), 156 performance describing, 13 of distributed transactions, 360 of in-memory databases, 89 of linearizability, 338 of multi-leader replication, 169 perpetual inconsistency, 525 pessimistic concurrency control, 261 phantoms (transaction isolation), 250 materializing conflicts, 251 preventing, in serializability, 259 physical clocks (see clocks) pickle (Python), 113 Pig (dataflow language), 419, 427 replicated joins, 409 skewed joins, 407 workflows, 403 Pinball (workflow scheduler), 402 pipelined execution, 423 in Unix, 394 point in time, 287 polyglot persistence, 29 polystores, 501 PostgreSQL (database) BDR (multi-leader replication), 170 causal ordering of writes, 177 Bottled Water (change data capture), 455 Bucardo (trigger-based replication), 161, 173 distributed transaction support, 361 foreign data wrappers, 501 full text search support, 490 leader-based replication, 153 log sequence number, 156 MVCC implementation, 239, 241 PL/pgSQL language, 255 PostGIS geospatial indexes, 87 preventing lost updates, 245 preventing write skew, 248, 261 read committed isolation, 236 recursive query support, 54 representing graphs, 51 Index | 579 serializable snapshot isolation (SSI), 261 snapshot isolation support, 239, 242 WAL-based replication, 160 XML and JSON support, 30, 42 pre-splitting, 212 Precision Time Protocol (PTP), 290 predicate locks, 259 predictive analytics, 533-536 amplifying bias, 534 ethics of (see ethics) feedback loops, 536 preemption of datacenter resources, 418 of threads, 298 Pregel processing model, 425 primary keys, 85, 556 compound primary key (Cassandra), 204 primary-secondary replication (see leaderbased replication) privacy, 536-543 consent and freedom of choice, 538 data as assets and power, 540 deleting data, 463 ethical considerations (see ethics) legislation and self-regulation, 542 meaning of, 539 surveillance, 537 tracking behavioral data, 536 probabilistic algorithms, 16, 466 process pauses, 295-299 processing time (of events), 469 producers (message streams), 440 programming languages dataflow languages, 504 for stored procedures, 255 functional reactive programming (FRP), 504 logic programming, 504 Prolog (language), 61 (see also Datalog) promises (asynchronous operations), 135 property graphs, 50 Cypher query language, 52 Protocol Buffers (data format), 117-121 field tags and schema evolution, 120 provenance of data, 531 publish/subscribe model, 441 publishers (message streams), 440 punch card tabulating machines, 390 580 | Index pure functions, 48 putting computation near data, 400 Q Qpid (messaging), 444 quality of service (QoS), 285 Quantcast File System (distributed filesystem), 398 query languages, 42-48 aggregation pipeline, 48 CSS and XSL, 44 Cypher, 52 Datalog, 60 Juttle, 504 MapReduce querying, 46-48 recursive SQL queries, 53 relational algebra and SQL, 42 SPARQL, 59 query optimizers, 37, 427 queueing delays (networks), 282 head-of-line blocking, 15 latency and response time, 14 queues (messaging), 137 quorums, 179-182, 556 for leaderless replication, 179 in consensus algorithms, 368 limitations of consistency, 181-183, 334 making decisions in distributed systems, 301 monitoring staleness, 182 multi-datacenter replication, 184 relying on durability, 309 sloppy quorums and hinted handoff, 183 R R-trees (indexes), 87 RabbitMQ (messaging), 137, 444 leader-based replication, 153 race conditions, 225 (see also concurrency) avoiding with linearizability, 331 caused by dual writes, 452 dirty writes, 235 in counter increments, 235 lost updates, 242-246 preventing with event logs, 462, 507 preventing with serializable isolation, 252 write skew, 246-251 Raft (consensus algorithm), 366 sensitivity to network problems, 369 term number, 368 use in etcd, 353 RAID (Redundant Array of Independent Disks), 7, 398 railways, schema migration on, 496 RAMCloud (in-memory storage), 89 ranking algorithms, 424 RDF (Resource Description Framework), 57 querying with SPARQL, 59 RDMA (Remote Direct Memory Access), 276 read committed isolation level, 234-237 implementing, 236 multi-version concurrency control (MVCC), 239 no dirty reads, 234 no dirty writes, 235 read path (derived data), 509 read repair (leaderless replication), 178 for linearizability, 335 read replicas (see leader-based replication) read skew (transaction isolation), 238, 266 as violation of causality, 340 read-after-write consistency, 163, 524 cross-device, 164 read-modify-write cycle, 243 read-scaling architecture, 161 reads as events, 513 real-time collaborative editing, 170 near-real-time processing, 390 (see also stream processing) publish/subscribe dataflow, 513 response time guarantees, 298 time-of-day clocks, 288 rebalancing partitions, 209-214, 556 (see also partitioning) automatic or manual rebalancing, 213 dynamic partitioning, 212 fixed number of partitions, 210 fixed number of partitions per node, 212 problems with hash mod N, 210 recency guarantee, 324 recommendation engines batch process outputs, 412 batch workflows, 403, 420 iterative processing, 424 statistical and numerical algorithms, 428 records, 399 events in stream processing, 440 recursive common table expressions (SQL), 54 redelivery (messaging), 445 Redis (database) atomic operations, 243 durability, 89 Lua scripting, 255 single-threaded execution, 253 usage example, 4 redundancy hardware components, 7 of derived data, 386 (see also derived data) Reed–Solomon codes (error correction), 398 refactoring, 22 (see also evolvability) regions (partitioning), 199 register (data structure), 325 relational data model, 28-42 comparison to document model, 38-42 graph queries in SQL, 53 in-memory databases with, 89 many-to-one and many-to-many relation‐ ships, 33 multi-object transactions, need for, 231 NoSQL as alternative to, 29 object-relational mismatch, 29 relational algebra and SQL, 42 versus document model convergence of models, 41 data locality, 41 relational databases eventual consistency, 162 history, 28 leader-based replication, 153 logical logs, 160 philosophy compared to Unix, 499, 501 schema changes, 40, 111, 130 statement-based replication, 158 use of B-tree indexes, 80 relationships (see edges) reliability, 6-10, 489 building a reliable system from unreliable components, 276 defined, 6, 22 hardware faults, 7 human errors, 9 importance of, 10 of messaging systems, 442 Index | 581 software errors, 8 Remote Method Invocation (Java RMI), 134 remote procedure calls (RPCs), 134-136 (see also services) based on futures, 135 data encoding and evolution, 136 issues with, 134 using Avro, 126, 135 using Thrift, 135 versus message brokers, 137 repeatable reads (transaction isolation), 242 replicas, 152 replication, 151-193, 556 and durability, 227 chain replication, 155 conflict resolution and, 246 consistency properties, 161-167 consistent prefix reads, 165 monotonic reads, 164 reading your own writes, 162 in distributed filesystems, 398 leaderless, 177-191 detecting concurrent writes, 184-191 limitations of quorum consistency, 181-183, 334 sloppy quorums and hinted handoff, 183 monitoring staleness, 182 multi-leader, 168-177 across multiple datacenters, 168, 335 handling write conflicts, 171-175 replication topologies, 175-177 partitioning and, 147, 200 reasons for using, 145, 151 single-leader, 152-161 failover, 157 implementation of replication logs, 158-161 relation to consensus, 367 setting up new followers, 155 synchronous versus asynchronous, 153-155 state machine replication, 349, 452 using erasure coding, 398 with heterogeneous data systems, 453 replication logs (see logs) reprocessing data, 496, 498 (see also evolvability) from log-based messaging, 451 request routing, 214-216 582 | Index approaches to, 214 parallel query execution, 216 resilient systems, 6 (see also fault tolerance) response time as performance metric for services, 13, 389 guarantees on, 298 latency versus, 14 mean and percentiles, 14 user experience, 15 responsibility and accountability, 535 REST (Representational State Transfer), 133 (see also services) RethinkDB (database) document data model, 31 dynamic partitioning, 212 join support, 34, 42 key-range partitioning, 202 leader-based replication, 153 subscribing to changes, 456 Riak (database) Bitcask storage engine, 72 CRDTs, 174, 191 dotted version vectors, 191 gossip protocol, 216 hash partitioning, 203-204, 211 last-write-wins conflict resolution, 186 leaderless replication, 177 LevelDB storage engine, 78 linearizability, lack of, 335 multi-datacenter support, 184 preventing lost updates across replicas, 246 rebalancing, 213 search feature, 209 secondary indexes, 207 siblings (concurrently written values), 190 sloppy quorums, 184 ring buffers, 450 Ripple (cryptocurrency), 532 rockets, 10, 36, 305 RocksDB (storage engine), 78 leveled compaction, 79 rollbacks (transactions), 222 rolling upgrades, 8, 112 routing (see request routing) row-oriented storage, 96 row-based replication, 160 rowhammer (memory corruption), 529 RPCs (see remote procedure calls) Rubygems (package manager), 428 rules (Datalog), 61 S safety and liveness properties, 308 in consensus algorithms, 366 in transactions, 222 sagas (see compensating transactions) Samza (stream processor), 466, 467 fault tolerance, 479 streaming SQL support, 466 sandboxes, 9 SAP HANA (database), 93 scalability, 10-18, 489 approaches for coping with load, 17 defined, 22 describing load, 11 describing performance, 13 partitioning and, 199 replication and, 161 scaling up versus scaling out, 146 scaling out, 17, 146 (see also shared-nothing architecture) scaling up, 17, 146 scatter/gather approach, querying partitioned databases, 207 SCD (slowly changing dimension), 476 schema-on-read, 39 comparison to evolvable schema, 128 in distributed filesystems, 415 schema-on-write, 39 schemaless databases (see schema-on-read) schemas, 557 Avro, 122-127 reader determining writer’s schema, 125 schema evolution, 123 dynamically generated, 126 evolution of, 496 affecting application code, 111 compatibility checking, 126 in databases, 129-131 in message-passing, 138 in service calls, 136 flexibility in document model, 39 for analytics, 93-95 for JSON and XML, 115 merits of, 127 schema migration on railways, 496 Thrift and Protocol Buffers, 117-121 schema evolution, 120 traditional approach to design, fallacy in, 462 searches building search indexes in batch processes, 411 k-nearest neighbors, 429 on streams, 467 partitioned secondary indexes, 206 secondaries (see leader-based replication) secondary indexes, 85, 557 partitioning, 206-209, 217 document-partitioned, 206 index maintenance, 495 term-partitioned, 208 problems with dual writes, 452, 491 updating, transaction isolation and, 231 secondary sorts, 405 sed (Unix tool), 392 self-describing files, 127 self-joins, 480 self-validating systems, 530 semantic web, 57 semi-synchronous replication, 154 sequence number ordering, 343-348 generators, 294, 344 insufficiency for enforcing constraints, 347 Lamport timestamps, 345 use of timestamps, 291, 295, 345 sequential consistency, 351 serializability, 225, 233, 251-266, 557 linearizability versus, 329 pessimistic versus optimistic concurrency control, 261 serial execution, 252-256 partitioning, 255 using stored procedures, 253, 349 serializable snapshot isolation (SSI), 261-266 detecting stale MVCC reads, 263 detecting writes that affect prior reads, 264 distributed execution, 265, 364 performance of SSI, 265 preventing write skew, 262-265 two-phase locking (2PL), 257-261 index-range locks, 260 performance, 258 Serializable (Java), 113 Index | 583 serialization, 113 (see also encoding) service discovery, 135, 214, 372 using DNS, 216, 372 service level agreements (SLAs), 15 service-oriented architecture (SOA), 132 (see also services) services, 131-136 microservices, 132 causal dependencies across services, 493 loose coupling, 502 relation to batch/stream processors, 389, 508 remote procedure calls (RPCs), 134-136 issues with, 134 similarity to databases, 132 web services, 132, 135 session windows (stream processing), 472 (see also windows) sessionization, 407 sharding (see partitioning) shared mode (locks), 258 shared-disk architecture, 146, 398 shared-memory architecture, 146 shared-nothing architecture, 17, 146-147, 557 (see also replication) distributed filesystems, 398 (see also distributed filesystems) partitioning, 199 use of network, 277 sharks biting undersea cables, 279 counting (example), 46-48 finding (example), 42 website about (example), 44 shredding (in relational model), 38 siblings (concurrent values), 190, 246 (see also conflicts) similarity search edit distance, 88 genome data, 63 k-nearest neighbors, 429 single-leader replication (see leader-based rep‐ lication) single-threaded execution, 243, 252 in batch processing, 406, 421, 426 in stream processing, 448, 463, 522 size-tiered compaction, 79 skew, 557 584 | Index clock skew, 291-294, 334 in transaction isolation read skew, 238, 266 write skew, 246-251, 262-265 (see also write skew) meanings of, 238 unbalanced workload, 201 compensating for, 205 due to celebrities, 205 for time-series data, 203 in batch processing, 407 slaves (see leader-based replication) sliding windows (stream processing), 472 (see also windows) sloppy quorums, 183 (see also quorums) lack of linearizability, 334 slowly changing dimension (data warehouses), 476 smearing (leap seconds adjustments), 290 snapshots (databases) causal consistency, 340 computing derived data, 500 in change data capture, 455 serializable snapshot isolation (SSI), 261-266, 329 setting up a new replica, 156 snapshot isolation and repeatable read, 237-242 implementing with MVCC, 239 indexes and MVCC, 241 visibility rules, 240 synchronized clocks for global snapshots, 294 snowflake schemas, 95 SOAP, 133 (see also services) evolvability, 136 software bugs, 8 maintaining integrity, 529 solid state drives (SSDs) access patterns, 84 detecting corruption, 519, 530 faults in, 227 sequential write throughput, 75 Solr (search server) building indexes in batch processes, 411 document-partitioned indexes, 207 request routing, 216 usage example, 4 use of Lucene, 79 sort (Unix tool), 392, 394, 395 sort-merge joins (MapReduce), 405 Sorted String Tables (see SSTables) sorting sort order in column storage, 99 source of truth (see systems of record) Spanner (database) data locality, 41 snapshot isolation using clocks, 295 TrueTime API, 294 Spark (processing framework), 421-423 bytecode generation, 428 dataflow APIs, 427 fault tolerance, 422 for data warehouses, 93 GraphX API (graph processing), 425 machine learning, 428 query optimizer, 427 Spark Streaming, 466 microbatching, 477 stream processing on top of batch process‐ ing, 495 SPARQL (query language), 59 spatial algorithms, 429 split brain, 158, 557 in consensus algorithms, 352, 367 preventing, 322, 333 using fencing tokens to avoid, 302-304 spreadsheets, dataflow programming capabili‐ ties, 504 SQL (Structured Query Language), 21, 28, 43 advantages and limitations of, 416 distributed query execution, 48 graph queries in, 53 isolation levels standard, issues with, 242 query execution on Hadoop, 416 résumé (example), 30 SQL injection vulnerability, 305 SQL on Hadoop, 93 statement-based replication, 158 stored procedures, 255 SQL Server (database) data warehousing support, 93 distributed transaction support, 361 leader-based replication, 153 preventing lost updates, 245 preventing write skew, 248, 257 read committed isolation, 236 recursive query support, 54 serializable isolation, 257 snapshot isolation support, 239 T-SQL language, 255 XML support, 30 SQLstream (stream analytics), 466 SSDs (see solid state drives) SSTables (storage format), 76-79 advantages over hash indexes, 76 concatenated index, 204 constructing and maintaining, 78 making LSM-Tree from, 78 staleness (old data), 162 cross-channel timing dependencies, 331 in leaderless databases, 178 in multi-version concurrency control, 263 monitoring for, 182 of client state, 512 versus linearizability, 324 versus timeliness, 524 standbys (see leader-based replication) star replication topologies, 175 star schemas, 93-95 similarity to event sourcing, 458 Star Wars analogy (event time versus process‐ ing time), 469 state derived from log of immutable events, 459 deriving current state from the event log, 458 interplay between state changes and appli‐ cation code, 507 maintaining derived state, 495 maintenance by stream processor in streamstream joins, 473 observing derived state, 509-515 rebuilding after stream processor failure, 478 separation of application code and, 505 state machine replication, 349, 452 statement-based replication, 158 statically typed languages analogy to schema-on-write, 40 code generation and, 127 statistical and numerical algorithms, 428 StatsD (metrics aggregator), 442 stdin, stdout, 395, 396 Stellar (cryptocurrency), 532 Index | 585 stock market feeds, 442 STONITH (Shoot The Other Node In The Head), 158 stop-the-world (see garbage collection) storage composing data storage technologies, 499-504 diversity of, in MapReduce, 415 Storage Area Network (SAN), 146, 398 storage engines, 69-104 column-oriented, 95-101 column compression, 97-99 defined, 96 distinction between column families and, 99 Parquet, 96, 131 sort order in, 99-100 writing to, 101 comparing requirements for transaction processing and analytics, 90-96 in-memory storage, 88 durability, 227 row-oriented, 70-90 B-trees, 79-83 comparing B-trees and LSM-trees, 83-85 defined, 96 log-structured, 72-79 stored procedures, 161, 253-255, 557 and total order broadcast, 349 pros and cons of, 255 similarity to stream processors, 505 Storm (stream processor), 466 distributed RPC, 468, 514 Trident state handling, 478 straggler events, 470, 498 stream processing, 464-481, 557 accessing external services within job, 474, 477, 478, 517 combining with batch processing lambda architecture, 497 unifying technologies, 498 comparison to batch processing, 464 complex event processing (CEP), 465 fault tolerance, 476-479 atomic commit, 477 idempotence, 478 microbatching and checkpointing, 477 rebuilding state after a failure, 478 for data integration, 494-498 586 | Index maintaining derived state, 495 maintenance of materialized views, 467 messaging systems (see messaging systems) reasoning about time, 468-472 event time versus processing time, 469, 477, 498 knowing when window is ready, 470 types of windows, 472 relation to databases (see streams) relation to services, 508 search on streams, 467 single-threaded execution, 448, 463 stream analytics, 466 stream joins, 472-476 stream-stream join, 473 stream-table join, 473 table-table join, 474 time-dependence of, 475 streams, 440-451 end-to-end, pushing events to clients, 512 messaging systems (see messaging systems) processing (see stream processing) relation to databases, 451-464 (see also changelogs) API support for change streams, 456 change data capture, 454-457 derivative of state by time, 460 event sourcing, 457-459 keeping systems in sync, 452-453 philosophy of immutable events, 459-464 topics, 440 strict serializability, 329 strong consistency (see linearizability) strong one-copy serializability, 329 subjects, predicates, and objects (in triplestores), 55 subscribers (message streams), 440 (see also consumers) supercomputers, 275 surveillance, 537 (see also privacy) Swagger (service definition format), 133 swapping to disk (see virtual memory) synchronous networks, 285, 557 comparison to asynchronous networks, 284 formal model, 307 synchronous replication, 154, 557 chain replication, 155 conflict detection, 172 system models, 300, 306-310 assumptions in, 528 correctness of algorithms, 308 mapping to the real world, 309 safety and liveness, 308 systems of record, 386, 557 change data capture, 454, 491 treating event log as, 460 systems thinking, 536 T t-digest (algorithm), 16 table-table joins, 474 Tableau (data visualization software), 416 tail (Unix tool), 447 tail vertex (property graphs), 51 Tajo (query engine), 93 Tandem NonStop SQL (database), 200 TCP (Transmission Control Protocol), 277 comparison to circuit switching, 285 comparison to UDP, 283 connection failures, 280 flow control, 282, 441 packet checksums, 306, 519, 529 reliability and duplicate suppression, 517 retransmission timeouts, 284 use for transaction sessions, 229 telemetry (see monitoring) Teradata (database), 93, 200 term-partitioned indexes, 208, 217 termination (consensus), 365 Terrapin (database), 413 Tez (dataflow engine), 421-423 fault tolerance, 422 support by higher-level tools, 427 thrashing (out of memory), 297 threads (concurrency) actor model, 138, 468 (see also message-passing) atomic operations, 223 background threads, 73, 85 execution pauses, 286, 296-298 memory barriers, 338 preemption, 298 single (see single-threaded execution) three-phase commit, 359 Thrift (data format), 117-121 BinaryProtocol, 118 CompactProtocol, 119 field tags and schema evolution, 120 throughput, 13, 390 TIBCO, 137 Enterprise Message Service, 444 StreamBase (stream analytics), 466 time concurrency and, 187 cross-channel timing dependencies, 331 in distributed systems, 287-299 (see also clocks) clock synchronization and accuracy, 289 relying on synchronized clocks, 291-295 process pauses, 295-299 reasoning about, in stream processors, 468-472 event time versus processing time, 469, 477, 498 knowing when window is ready, 470 timestamp of events, 471 types of windows, 472 system models for distributed systems, 307 time-dependence in stream joins, 475 time-of-day clocks, 288 timeliness, 524 coordination-avoiding data systems, 528 correctness of dataflow systems, 525 timeouts, 279, 557 dynamic configuration of, 284 for failover, 158 length of, 281 timestamps, 343 assigning to events in stream processing, 471 for read-after-write consistency, 163 for transaction ordering, 295 insufficiency for enforcing constraints, 347 key range partitioning by, 203 Lamport, 345 logical, 494 ordering events, 291, 345 Titan (database), 50 tombstones, 74, 191, 456 topics (messaging), 137, 440 total order, 341, 557 limits of, 493 sequence numbers or timestamps, 344 total order broadcast, 348-352, 493, 522 consensus algorithms and, 366-368 Index | 587 implementation in ZooKeeper and etcd, 370 implementing with linearizable storage, 351 using, 349 using to implement linearizable storage, 350 tracking behavioral data, 536 (see also privacy) transaction coordinator (see coordinator) transaction manager (see coordinator) transaction processing, 28, 90-95 comparison to analytics, 91 comparison to data warehousing, 93 transactions, 221-267, 558 ACID properties of, 223 atomicity, 223 consistency, 224 durability, 226 isolation, 225 compensating (see compensating transac‐ tions) concept of, 222 distributed transactions, 352-364 avoiding, 492, 502, 521-528 failure amplification, 364, 495 in doubt/uncertain status, 358, 362 two-phase commit, 354-359 use of, 360-361 XA transactions, 361-364 OLTP versus analytics queries, 411 purpose of, 222 serializability, 251-266 actual serial execution, 252-256 pessimistic versus optimistic concur‐ rency control, 261 serializable snapshot isolation (SSI), 261-266 two-phase locking (2PL), 257-261 single-object and multi-object, 228-232 handling errors and aborts, 231 need for multi-object transactions, 231 single-object writes, 230 snapshot isolation (see snapshots) weak isolation levels, 233-251 preventing lost updates, 242-246 read committed, 234-238 transitive closure (graph algorithm), 424 trie (data structure), 88 triggers (databases), 161, 441 implementing change data capture, 455 implementing replication, 161 588 | Index triple-stores, 55-59 SPARQL query language, 59 tumbling windows (stream processing), 472 (see also windows) in microbatching, 477 tuple spaces (programming model), 507 Turtle (RDF data format), 56 Twitter constructing home timelines (example), 11, 462, 474, 511 DistributedLog (event log), 448 Finagle (RPC framework), 135 Snowflake (sequence number generator), 294 Summingbird (processing library), 497 two-phase commit (2PC), 353, 355-359, 558 confusion with two-phase locking, 356 coordinator failure, 358 coordinator recovery, 363 how it works, 357 issues in practice, 363 performance cost, 360 transactions holding locks, 362 two-phase locking (2PL), 257-261, 329, 558 confusion with two-phase commit, 356 index-range locks, 260 performance of, 258 type checking, dynamic versus static, 40 U UDP (User Datagram Protocol) comparison to TCP, 283 multicast, 442 unbounded datasets, 439, 558 (see also streams) unbounded delays, 558 in networks, 282 process pauses, 296 unbundling databases, 499-515 composing data storage technologies, 499-504 federation versus unbundling, 501 need for high-level language, 503 designing applications around dataflow, 504-509 observing derived state, 509-515 materialized views and caching, 510 multi-partition data processing, 514 pushing state changes to clients, 512 uncertain (transaction status) (see in doubt) uniform consensus, 365 (see also consensus) uniform interfaces, 395 union type (in Avro), 125 uniq (Unix tool), 392 uniqueness constraints asynchronously checked, 526 requiring consensus, 521 requiring linearizability, 330 uniqueness in log-based messaging, 522 Unix philosophy, 394-397 command-line batch processing, 391-394 Unix pipes versus dataflow engines, 423 comparison to Hadoop, 413-414 comparison to relational databases, 499, 501 comparison to stream processing, 464 composability and uniform interfaces, 395 loose coupling, 396 pipes, 394 relation to Hadoop, 499 UPDATE statement (SQL), 40 updates preventing lost updates, 242-246 atomic write operations, 243 automatically detecting lost updates, 245 compare-and-set operations, 245 conflict resolution and replication, 246 using explicit locking, 244 preventing write skew, 246-251 V validity (consensus), 365 vBuckets (partitioning), 199 vector clocks, 191 (see also version vectors) vectorized processing, 99, 428 verification, 528-533 avoiding blind trust, 530 culture of, 530 designing for auditability, 531 end-to-end integrity checks, 531 tools for auditable data systems, 532 version control systems, reliance on immutable data, 463 version vectors, 177, 191 capturing causal dependencies, 343 versus vector clocks, 191 Vertica (database), 93 handling writes, 101 replicas using different sort orders, 100 vertical scaling (see scaling up) vertices (in graphs), 49 property graph model, 50 Viewstamped Replication (consensus algo‐ rithm), 366 view number, 368 virtual machines, 146 (see also cloud computing) context switches, 297 network performance, 282 noisy neighbors, 284 reliability in cloud services, 8 virtualized clocks in, 290 virtual memory process pauses due to page faults, 14, 297 versus memory management by databases, 89 VisiCalc (spreadsheets), 504 vnodes (partitioning), 199 Voice over IP (VoIP), 283 Voldemort (database) building read-only stores in batch processes, 413 hash partitioning, 203-204, 211 leaderless replication, 177 multi-datacenter support, 184 rebalancing, 213 reliance on read repair, 179 sloppy quorums, 184 VoltDB (database) cross-partition serializability, 256 deterministic stored procedures, 255 in-memory storage, 89 output streams, 456 secondary indexes, 207 serial execution of transactions, 253 statement-based replication, 159, 479 transactions in stream processing, 477 W WAL (write-ahead log), 82 web services (see services) Web Services Description Language (WSDL), 133 webhooks, 443 webMethods (messaging), 137 WebSocket (protocol), 512 Index | 589 windows (stream processing), 466, 468-472 infinite windows for changelogs, 467, 474 knowing when all events have arrived, 470 stream joins within a window, 473 types of windows, 472 winners (conflict resolution), 173 WITH RECURSIVE syntax (SQL), 54 workflows (MapReduce), 402 outputs, 411-414 key-value stores, 412 search indexes, 411 with map-side joins, 410 working set, 393 write amplification, 84 write path (derived data), 509 write skew (transaction isolation), 246-251 characterizing, 246-251, 262 examples of, 247, 249 materializing conflicts, 251 occurrence in practice, 529 phantoms, 250 preventing in snapshot isolation, 262-265 in two-phase locking, 259-261 options for, 248 write-ahead log (WAL), 82, 159 writes (database) atomic write operations, 243 detecting writes affecting prior reads, 264 preventing dirty writes with read commit‐ ted, 235 WS-* framework, 133 (see also services) WS-AtomicTransaction (2PC), 355 590 | Index X XA transactions, 355, 361-364 heuristic decisions, 363 limitations of, 363 xargs (Unix tool), 392, 396 XML binary variants, 115 encoding RDF data, 57 for application data, issues with, 114 in relational databases, 30, 41 XSL/XPath, 45 Y Yahoo!

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann

active measures, Amazon Web Services, billion-dollar mistake, bitcoin, blockchain, business intelligence, business logic, business process, c2.com, cloud computing, collaborative editing, commoditize, conceptual framework, cryptocurrency, data science, database schema, deep learning, DevOps, distributed ledger, Donald Knuth, Edward Snowden, end-to-end encryption, Ethereum, ethereum blockchain, exponential backoff, fake news, fault tolerance, finite state, Flash crash, Free Software Foundation, full text search, functional programming, general-purpose programming language, Hacker News, informal economy, information retrieval, Infrastructure as a Service, Internet of things, iterative process, John von Neumann, Ken Thompson, Kubernetes, Large Hadron Collider, level 1 cache, loose coupling, machine readable, machine translation, Marc Andreessen, microservices, natural language processing, Network effects, no silver bullet, operational security, packet switching, peer-to-peer, performance metric, place-making, premature optimization, recommendation engine, Richard Feynman, self-driving car, semantic web, Shoshana Zuboff, social graph, social web, software as a service, software is eating the world, sorting algorithm, source of truth, SPARQL, speech recognition, SQL injection, statistical model, surveillance capitalism, systematic bias, systems thinking, Tragedy of the Commons, undersea cable, web application, WebSocket, wikimedia commons

Public cloud services such as EC2 are notorious for having frequent transient network glitches [14], and well-managed private datacenter networks can be stabler environments. Nevertheless, nobody is immune from network problems: for example, a problem during a software upgrade for a switch could trigger a network topology reconfiguration, during which network packets could be delayed for more than a minute [17]. Sharks might bite undersea cables and damage them [18]. Other surprising faults include a network interface that sometimes drops all inbound packets but sends outbound packets successfully [19]: just because a network link works in one direction doesn’t guarantee it’s also working in the opposite direction. Network partitions When one part of the network is cut off from the rest due to a network fault, that is sometimes called a network partition or netsplit.

…

imperative query APIs, Declarative Queries on the Web InfoSphere Streams (CEP engine), Complex event processing MQ (messaging), Message brokers compared to databasesdistributed transaction support, XA transactions System R (database), The Slippery Concept of a Transaction WebSphere (messaging), Message brokers idempotence, The problems with remote procedure calls (RPCs), Idempotence, Glossaryby giving operations unique IDs, Operation identifiers, Multi-partition request processing idempotent operations, Exactly-once execution of an operation immutabilityadvantages of, Advantages of immutable events, Designing for auditability deriving state from event log, State, Streams, and Immutability-Limitations of immutability for crash recovery, Hash Indexes in B-trees, B-tree optimizations, Indexes and snapshot isolation in event sourcing, Event Sourcing inputs to Unix commands, Transparency and experimentation limitations of, Limitations of immutability Impala (query engine)for data warehouses, The divergence between OLTP databases and data warehouses hash joins, Broadcast hash joins native code generation, The move toward declarative query languages use of HDFS, Diversity of processing models impedance mismatch, The Object-Relational Mismatch imperative languages, Query Languages for Datasetting element styles (example), Declarative Queries on the Web in doubt (transaction status), Coordinator failureholding locks, Holding locks while in doubt orphaned transactions, Recovering from coordinator failure in-memory databases, Keeping everything in memorydurability, Durability serial transaction execution, Actual Serial Execution incidentscascading failures, Software Errors crashes due to leap seconds, Clock Synchronization and Accuracy data corruption and financial losses due to concurrency bugs, Weak Isolation Levels data corruption on hard disks, Durability data loss due to last-write-wins, Converging toward a consistent state, Timestamps for ordering events data on disks unreadable, Mapping system models to the real world deleted items reappearing, Custom conflict resolution logic disclosure of sensitive data due to primary key reuse, Leader failure: Failover errors in transaction serializability, Maintaining integrity in the face of software bugs gigabit network interface with 1 Kb/s throughput, Summary network faults, Network Faults in Practice network interface dropping only inbound packets, Network Faults in Practice network partitions and whole-datacenter failures, Faults and Partial Failures poor handling of network faults, Network Faults in Practice sending message to ex-partner, Ordering events to capture causality sharks biting undersea cables, Network Faults in Practice split brain due to 1-minute packet delay, Leader failure: Failover, Network Faults in Practice vibrations in server rack, Describing Performance violation of uniqueness constraint, Maintaining integrity in the face of software bugs indexes, Data Structures That Power Your Database, Glossaryand snapshot isolation, Indexes and snapshot isolation as derived data, Derived Data, Composing Data Storage Technologies-What’s missing?

…

pessimistic versus optimistic concurrency control, Pessimistic versus optimistic concurrency control serial execution, Actual Serial Execution-Summary of serial executionpartitioning, Partitioning using stored procedures, Encapsulating transactions in stored procedures, Using total order broadcast serializable snapshot isolation (SSI), Serializable Snapshot Isolation (SSI)-Performance of serializable snapshot isolationdetecting stale MVCC reads, Detecting stale MVCC reads detecting writes that affect prior reads, Detecting writes that affect prior reads distributed execution, Performance of serializable snapshot isolation, Limitations of distributed transactions performance of SSI, Performance of serializable snapshot isolation preventing write skew, Decisions based on an outdated premise-Detecting writes that affect prior reads two-phase locking (2PL), Two-Phase Locking (2PL)-Index-range locksindex-range locks, Index-range locks performance, Performance of two-phase locking Serializable (Java), Language-Specific Formats serialization, Formats for Encoding Data(see also encoding) service discovery, Current directions for RPC, Request Routing, Service discoveryusing DNS, Request Routing, Service discovery service level agreements (SLAs), Describing Performance service-oriented architecture (SOA), Dataflow Through Services: REST and RPC(see also services) services, Dataflow Through Services: REST and RPC-Data encoding and evolution for RPCmicroservices, Dataflow Through Services: REST and RPCcausal dependencies across services, The limits of total ordering loose coupling, Making unbundling work relation to batch/stream processors, Batch Processing, Stream processors and services remote procedure calls (RPCs), The problems with remote procedure calls (RPCs)-Data encoding and evolution for RPCissues with, The problems with remote procedure calls (RPCs) similarity to databases, Dataflow Through Services: REST and RPC web services, Web services, Current directions for RPC session windows (stream processing), Types of windows(see also windows) sessionization, GROUP BY sharding (see partitioning) shared mode (locks), Implementation of two-phase locking shared-disk architecture, Distributed Data, MapReduce and Distributed Filesystems shared-memory architecture, Distributed Data shared-nothing architecture, Approaches for Coping with Load, Distributed Data-Distributed Data, Glossary(see also replication) distributed filesystems, MapReduce and Distributed Filesystems(see also distributed filesystems) partitioning, Partitioning use of network, Unreliable Networks sharksbiting undersea cables, Network Faults in Practice counting (example), MapReduce Querying-MapReduce Querying finding (example), Query Languages for Data website about (example), Declarative Queries on the Web shredding (in relational model), Which data model leads to simpler application code? siblings (concurrent values), Merging concurrently written values, Conflict resolution and replication(see also conflicts) similarity searchedit distance, Full-text search and fuzzy indexes genome data, Summary k-nearest neighbors, Specialization for different domains single-leader replication (see leader-based replication) single-threaded execution, Atomic write operations, Actual Serial Executionin batch processing, Bringing related data together in the same place, Dataflow engines, Parallel execution in stream processing, Logs compared to traditional messaging, Concurrency control, Uniqueness in log-based messaging size-tiered compaction, Performance optimizations skew, Glossaryclock skew, Relying on Synchronized Clocks-Clock readings have a confidence interval, Implementing Linearizable Systems in transaction isolationread skew, Snapshot Isolation and Repeatable Read, Summary write skew, Write Skew and Phantoms-Materializing conflicts, Decisions based on an outdated premise-Detecting writes that affect prior reads(see also write skew) meanings of, Snapshot Isolation and Repeatable Read unbalanced workload, Partitioning of Key-Value Datacompensating for, Skewed Workloads and Relieving Hot Spots due to celebrities, Skewed Workloads and Relieving Hot Spots for time-series data, Partitioning by Key Range in batch processing, Handling skew slaves (see leader-based replication) sliding windows (stream processing), Types of windows(see also windows) sloppy quorums, Sloppy Quorums and Hinted Handoff(see also quorums) lack of linearizability, Implementing Linearizable Systems slowly changing dimension (data warehouses), Time-dependence of joins smearing (leap seconds adjustments), Clock Synchronization and Accuracy snapshots (databases)causal consistency, Ordering and Causality computing derived data, Creating an index in change data capture, Initial snapshot serializable snapshot isolation (SSI), Serializable Snapshot Isolation (SSI)-Performance of serializable snapshot isolation, What Makes a System Linearizable?

Atlantic: Great Sea Battles, Heroic Discoveries, Titanic Storms & a Vast Ocean of a Million Stories by Simon Winchester

Beryl Markham, British Empire, cable laying ship, Charles Lindbergh, colonial rule, financial engineering, friendly fire, Intergovernmental Panel on Climate Change (IPCC), intermodal, Isaac Newton, Louis Blériot, Malcom McLean invented shipping containers, Nelson Mandela, North Sea oil, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, Piper Alpha, polynesian navigation, Suez canal 1869, supervolcano, three-masted sailing ship, trade route, transatlantic slave trade, transcontinental railway, undersea cable

In the middle of the century, a small group of entrepreneurs wondering how to speed up transatlantic messaging began focusing on this island—and it did so because Newfoundland offered the closest points in North America to Europe: it was a mere 1,600 miles from the harbor opening at St. John’s to the cliffs of Connemara. An undersea cable—for such had already been invented in Europe: a telegraph cable had been laid between Britain and France in August 1850, and soon afterward others were lowered into the seas between Britain and Holland, and between Scotland and Ireland—could perhaps be laid across the Cabot Strait, at the mouth of the St.

…

Wyville, 136, 139 Thoreau, Henry David, 305 Thorfinsson, Snorri, 83–84 Tides, 112, 114, 121 Tierra del Fuego, 125–26, 128, 208–11, 444 Times Atlas of the Oceans (book), 325–26 Tin, 65–66 Titanic (ship), 254, 263, 321–23 Tobacco, 84n, 90 Tomlinson, Charles, 205 Toothed whales, 285n Toothfish (Chilean sea bass), 362–63, 378, 384–86, 389 Topography, 121 Torrey Canyon (ship), 325–26 f Tor Rocks, 7–8 Toxic wastes, 357–58 Tracks, shipping, 348–49 Trade Hanseatic League and rules of, 275–80 Phoenician, in Mediterranean Sea area, 62–69 routes, 319 f Trade winds, 121, 424 Trafalgar Battle, 243–46, 251 Tramp ships, 295–96 Travailleur (ship), 141 Trawling, 368, 370 Treaty of Paris, 366 Treaty of Tordesillas, 93 Trial of Black Bart’s Men, 227–30 Triana, Rodrigo de, 86–87 Triangular slave trade, 232–39 Triassic period, 40 Triple Divide Peak, 147–48 Triremes (ships), 211 Tristan da Cunha, 112, 183n, 437–38 Tristan und Isolde (opera), 194 Tropical cyclones, 423. See also Hurricanes Tsunamis, 432–36. See also Seismic activity Tuna, 363 Turbidity currents, 434 Turner, J. M. W., 197–98 Tuvalu Islands, 412n Typhoon (book), 205 Typhoons, 423n U-boats, German, 261–66, 268–70 Uluburun archaeology site, 64–65 Undersea cables. See Cables, undersea Undersea oil exploration, 403–4 Undertow (painting), 198–99 Union Castle passenger liners, 186 United Nations, 176, 372n United States air transport to and from, 330–32 British Post Office service to, 291–93 Civil War of, 247–50 commercial whaling of, 287–90 literature of, 200–203 motivations of, for Atlantic Ocean exploration, 123–24 Naval Observatory, 130 rivers as Atlantic Ocean sources, 147–48 slave trade and, 221, 230–39 steel ships and, 257 transatlantic cable and, 305–10 War of 1812 of, 246–47 United States Exploring Expedition, 126–29 Ur supercontinent, 38 Uruguay, 257–60 van de Velde, Willem, 166 vaz de Torres, Luis, 112 Venezuela, 87–88 Verdi, Giuseppe, 194 Verne, Jules, 128, 445 Vernet, Claude, 167 Vespucci, Amerigo, 93–96 Viarsa (ship), 389 Victoria, Queen, 309 Victoria Falls, 47 Victory (ship), 243 Vikings, 70–71, 75–85, 89–90, 212–15, 282.

Neutrino Hunters: The Thrilling Chase for a Ghostly Particle to Unlock the Secrets of the Universe by Ray Jayawardhana

Albert Einstein, Alfred Russel Wallace, anti-communist, Arthur Eddington, cosmic microwave background, dark matter, Eddington experiment, Ernest Rutherford, Higgs boson, invention of the telescope, Isaac Newton, it's over 9,000, Johannes Kepler, Large Hadron Collider, Magellanic Cloud, New Journalism, race to the bottom, random walk, Richard Feynman, Schrödinger's Cat, seminal paper, Skype, South China Sea, Stephen Hawking, time dilation, undersea cable, uranium enrichment

One of the wackier ideas for putting neutrinos to practical use involves employing them for long-distance communication, since they travel unhindered through virtually anything. You could imagine sending an encoded beam of neutrinos from one side of the Earth to the other straight through the planet, whereas a radio signal needs to be bounced off multiple satellites in orbit or transmitted through undersea cables circling the globe. Along similar lines, one scientist has suggested using neutrino beams to send messages to submarines deep underwater. To have any chance of success, though, the beam would have to be about a million times more intense than those used in current experiments, and there would be some daunting challenges at the receiving end.

War of Shadows: Codebreakers, Spies, and the Secret Struggle to Drive the Nazis From the Middle East by Gershom Gorenberg

anti-communist, Bletchley Park, British Empire, Charles Lindbergh, colonial rule, computer age, defense in depth, European colonialism, illegal immigration, Internet Archive, plutocrats, Scientific racism, undersea cable

Troops poured in from England, Australia, New Zealand, and India. The Australians made the strongest impression: they were huge, loud, and happy and drank prodigiously.21 ANTHONY EDEN RETURNED to London in early November with word from Wavell. A messenger was the only means of communication that Wavell trusted. Italy had cut Britain’s undersea cables in the Mediterranean, and Wavell feared the treachery of wireless. Anything sent by it, even in the best code you had, could give you away.22 Churchill heard the general’s plans and “purred like six cats,” as he said himself.23 By the light of a half-moon, the thirty thousand men of the Western Desert Force marched forward on the night of December 7, passing between Graziani’s isolated fortified camps.

…

The Foreign Office asked Menzies, who forwarded the question to Travis, who would have most likely asked Dudley-Smith to get answers. A note initialed by Travis came back to the Foreign Office. It said American diplomatic telegrams to Washington were safe from interception—by the Axis. The unstated implication was that American traffic traveled on an intact British undersea cable. As for the wider question of US diplomatic codes and ciphers, this was delicate. The State Department had three code systems, the GCHQ note said. One system was used rarely, for the most secret messages. So far it was unbreakable. “The second-best book,” used more often, for material that was just slightly less secret, was “legible but patchy.”

How We Got Here: A Slightly Irreverent History of Technology and Markets by Andy Kessler

Albert Einstein, Andy Kessler, animal electricity, automated trading system, bank run, Big bang: deregulation of the City of London, Black Monday: stock market crash in 1987, Bletchley Park, Bob Noyce, Bretton Woods, British Empire, buttonwood tree, Charles Babbage, Claude Shannon: information theory, Corn Laws, cotton gin, Dennis Ritchie, Douglas Engelbart, Edward Lloyd's coffeehouse, Fairchild Semiconductor, fiat currency, fixed income, floating exchange rates, flying shuttle, Fractional reserve banking, full employment, GPS: selective availability, Grace Hopper, invention of the steam engine, invention of the telephone, invisible hand, Isaac Newton, Jacquard loom, James Hargreaves, James Watt: steam engine, John von Neumann, joint-stock company, joint-stock limited liability company, Joseph-Marie Jacquard, Ken Thompson, Kickstarter, Leonard Kleinrock, Marc Andreessen, Mary Meeker, Maui Hawaii, Menlo Park, Metcalfe's law, Metcalfe’s law, military-industrial complex, Mitch Kapor, Multics, packet switching, pneumatic tube, price mechanism, probability theory / Blaise Pascal / Pierre de Fermat, profit motive, proprietary trading, railway mania, RAND corporation, Robert Metcalfe, Silicon Valley, Small Order Execution System, South Sea Bubble, spice trade, spinning jenny, Steve Jobs, Suez canal 1869, supply-chain management, supply-chain management software, systems thinking, three-martini lunch, trade route, transatlantic slave trade, tulip mania, Turing machine, Turing test, undersea cable, UUNET, Wayback Machine, William Shockley: the traitorous eight

A Santa Barbara IMP went up in November and Utah in December 1969. One of the researchers at the Stanford site, Norm Abramson, was a surfer dude who spent a lot of time in Hawaii. The University of Hawaii had locations scattered across the Islands and was trying to figure out how to hook up a data network between them. It couldn’t afford to run undersea cable and modems were too slow, so they hit on the idea of using radio signals to transmit data. The problem was interference. Maui might transmit at the same time and step on the Big Island’s signal. It could use packet networks, but that didn’t solve the interference problem. In 1970, Abramson devised a system that checked for errors in the packets received.

The Longing for Less: Living With Minimalism by Kyle Chayka

Airbnb, Blue Bottle Coffee, Frank Gehry, gentrification, Guggenheim Bilbao, Jony Ive, Kickstarter, Lao Tzu, Mason jar, offshore financial centre, prosperity theology / prosperity gospel / gospel of success, public intellectual, Ralph Waldo Emerson, Richard Florida, Silicon Valley, Steve Jobs, Stewart Brand, technoutopianism, undersea cable, Whole Earth Catalog

In one way, Apple follows the rule: The devices have only a few visual qualities. But it’s also an illusion. The company has to manufacture its own batteries to be as flat as possible and removes ports—see headphone jacks—any chance it gets. The iPhone’s function depends on an enormous, complex, ugly superstructure of satellites and undersea cables that certainly aren’t designed in pristine whiteness. Minimalist design encourages us to forget everything a product relies on and imagine, in this example, that the internet consists of carefully shaped glass and steel alone. The contrast between simple form and complex consequences brings to mind what the British writer Daisy Hildyard called “the second body”22 in her 2017 book of the same name.

Thank You for Being Late: An Optimist's Guide to Thriving in the Age of Accelerations by Thomas L. Friedman

3D printing, additive manufacturing, affirmative action, Airbnb, AltaVista, Amazon Web Services, Anthropocene, Apple Newton, autonomous vehicles, Ayatollah Khomeini, barriers to entry, Berlin Wall, Bernie Sanders, Big Tech, biodiversity loss, bitcoin, blockchain, Bob Noyce, business cycle, business process, call centre, carbon tax, centre right, Chris Wanstrath, Clayton Christensen, clean tech, clean water, cloud computing, cognitive load, corporate social responsibility, creative destruction, CRISPR, crowdsourcing, data science, David Brooks, deep learning, demand response, demographic dividend, demographic transition, Deng Xiaoping, digital divide, disinformation, Donald Trump, dual-use technology, end-to-end encryption, Erik Brynjolfsson, fail fast, failed state, Fairchild Semiconductor, Fall of the Berlin Wall, Ferguson, Missouri, first square of the chessboard / second half of the chessboard, Flash crash, fulfillment center, game design, gig economy, global pandemic, global supply chain, Great Leap Forward, illegal immigration, immigration reform, income inequality, indoor plumbing, intangible asset, Intergovernmental Panel on Climate Change (IPCC), Internet of things, invention of the steam engine, inventory management, Irwin Jacobs: Qualcomm, Jeff Bezos, job automation, John Markoff, John von Neumann, Khan Academy, Kickstarter, knowledge economy, knowledge worker, land tenure, linear programming, Live Aid, low interest rates, low skilled workers, Lyft, Marc Andreessen, Mark Zuckerberg, mass immigration, Maui Hawaii, Menlo Park, Mikhail Gorbachev, mutually assured destruction, Neil Armstrong, Nelson Mandela, ocean acidification, PalmPilot, pattern recognition, planetary scale, power law, pull request, Ralph Waldo Emerson, ransomware, Ray Kurzweil, Richard Florida, ride hailing / ride sharing, Robert Gordon, Ronald Reagan, Salesforce, Second Machine Age, self-driving car, shareholder value, sharing economy, Silicon Valley, Skype, smart cities, Solyndra, South China Sea, Steve Jobs, subscription business, supercomputer in your pocket, synthetic biology, systems thinking, TaskRabbit, tech worker, TED Talk, The Rise and Fall of American Growth, Thomas L Friedman, Tony Fadell, transaction costs, Transnistria, uber lyft, undersea cable, urban decay, urban planning, Watson beat the top human players on Jeopardy!, WikiLeaks, women in the workforce, Y2K, Yogi Berra, zero-sum game

“In twenty years,” he added, “we went from maybe this is a good idea to there’s no turning back to hitting the physical limits … With ping I found out how close to the physical limits we were, and it was pretty startling. It is a way big revolution.” This revolution happened, Bucksbaum explained, thanks to a kind of Moore’s law that has been steadily quickening the transmission speeds of data and voice down fiber-optic cables. “The speed at which we can transmit data over undersea cables just keeps accelerating,” said Bucksbaum. The short version of the story, he explained, goes like this: We started out sending voice and data using a digital radio frequency over coaxial cable made primarily of copper wire. That is what your first cable/phone company sent into your house and into the box on your television set.

…

And it included wavelength division multiplexing, using different colors of light to carry different phone conversations at once—and then combinations of the two. They are not done accelerating. “The history of the last twenty years is that we just keep finding faster, better ways to divide the different properties of light to pack ever more information,” said Bucksbaum. “The rate of data transfer for an undersea cable today is now trillions of bits per second.” At some point, you end up “bumping up against the laws of physics,” he added, but we are not there yet. Companies are now experimenting not just with ways to change the pulse or the color of light to create more capacity, but also with new ways of shaping that light that can deliver more than one hundred trillion bits per second down their fiber lines.

The Rise and Fall of American Growth: The U.S. Standard of Living Since the Civil War (The Princeton Economic History of the Western World) by Robert J. Gordon

3D printing, Affordable Care Act / Obamacare, airline deregulation, airport security, Apple II, barriers to entry, big-box store, blue-collar work, business cycle, Capital in the Twenty-First Century by Thomas Piketty, carbon tax, Charles Lindbergh, classic study, clean water, collective bargaining, computer age, cotton gin, creative destruction, deindustrialization, Detroit bankruptcy, discovery of penicillin, Donner party, Downton Abbey, driverless car, Edward Glaeser, en.wikipedia.org, Erik Brynjolfsson, everywhere but in the productivity statistics, feminist movement, financial innovation, food desert, Ford Model T, full employment, general purpose technology, George Akerlof, germ theory of disease, glass ceiling, Glass-Steagall Act, Golden age of television, government statistician, Great Leap Forward, high net worth, housing crisis, Ida Tarbell, immigration reform, impulse control, income inequality, income per capita, indoor plumbing, industrial robot, inflight wifi, interchangeable parts, invention of agriculture, invention of air conditioning, invention of the sewing machine, invention of the telegraph, invention of the telephone, inventory management, James Watt: steam engine, Jeff Bezos, jitney, job automation, John Markoff, John Maynard Keynes: Economic Possibilities for our Grandchildren, labor-force participation, Les Trente Glorieuses, Lewis Mumford, Loma Prieta earthquake, Louis Daguerre, Louis Pasteur, low skilled workers, manufacturing employment, Mark Zuckerberg, market fragmentation, Mason jar, mass immigration, mass incarceration, McMansion, Menlo Park, minimum wage unemployment, mortgage debt, mortgage tax deduction, new economy, Norbert Wiener, obamacare, occupational segregation, oil shale / tar sands, oil shock, payday loans, Peter Thiel, Phillips curve, pink-collar, pneumatic tube, Productivity paradox, Ralph Nader, Ralph Waldo Emerson, refrigerator car, rent control, restrictive zoning, revenue passenger mile, Robert Solow, Robert X Cringely, Ronald Coase, school choice, Second Machine Age, secular stagnation, Skype, Southern State Parkway, stem cell, Steve Jobs, Steve Wozniak, Steven Pinker, streetcar suburb, The Market for Lemons, The Rise and Fall of American Growth, Thomas Malthus, total factor productivity, transaction costs, transcontinental railway, traveling salesman, Triangle Shirtwaist Factory, undersea cable, Unsafe at Any Speed, Upton Sinclair, upwardly mobile, urban decay, urban planning, urban sprawl, vertical integration, warehouse robotics, washing machines reduced drudgery, Washington Consensus, Watson beat the top human players on Jeopardy!, We wanted flying cars, instead we got 140 characters, working poor, working-age population, Works Progress Administration, yellow journalism, yield management

The Civil War itself showcased these inventions when northern trains sped Yankee troops to the front and steamships blockaded supplies to the south from Britain, hastening southern defeat. And no longer was news delayed by days or weeks. Half a century earlier, the Battle of New Orleans had been fought on January 8, 1815, three weeks after the Treaty of Ghent was signed to end the War of 1812. Before development of the telegraph and undersea cable, news traveled very slowly. But during the Civil War, the daily newspapers carried dispatches announcing the outcomes of battles mere hours after they occurred. The flood of inventions that followed the Civil War utterly transformed life, transferring human attention and energy from the mundane to soaring skyscrapers and airplanes.

…

Within only two years, there were nine telegraph companies, whose 2,000 miles of wire stretched all the way from Portland, Maine, to Chicago and New Orleans.14 The telegraph network was ubiquitous in the east by 1855, the transcontinental telegraph debuted in late 1861, and after an abortive start in the late 1850s, a working undersea cable linking Britain and America was laid in 1866. Contemporary observers recognized the importance of the telegraph almost immediately. As early as 1847, the telegraph was seen as “facilitating Human Intercourse and producing Harmony among Men and Nations.” In the late 1860s, a writer anticipated “when the missing links shall have been completed of the great chain that will bring all civilized nations into instantaneous communication with each other … breaking down the barriers of evil prejudice and custom …” At the same time another optimistically suggested that “whenever science achieves a victory, a rivet is loosened from the chains of the oppressed.”15 Though the cost of sending a telegram was initially too high for the telegraph to gain widespread usage by private households, it was immediately viewed as essential by three industries—finance, railroads, and newspapers.

…

The telegraph could transmit prices of commodity and financial asset data, thus reducing or eliminating the role of arbitrage for traders who had particularly good or bad access to information. By squeezing margins, news provided by telegraph began to eliminate wholesalers and middlemen who had depended on differential access to information. A domestic financial transaction could be ordered and confirmed by 1890 in less than two minutes. Before the undersea cable, the six-week delay for a round-trip crossing from New York to London could lead to inefficient purchase and sale decisions for commodities and other goods, implying that the welfare benefits of the telegraph and cable included not just financial markets, but also the real sector of the economy.16 The telegraph became an essential tool to railroads in controlling the flow of passenger and freight trains.

The Snowden Files: The Inside Story of the World's Most Wanted Man by Luke Harding

affirmative action, air gap, airport security, Anton Chekhov, Apple's 1984 Super Bowl advert, Berlin Wall, Big Tech, Bletchley Park, Chelsea Manning, disinformation, don't be evil, drone strike, Edward Snowden, Etonian, Firefox, Google Earth, information security, Jacob Appelbaum, job-hopping, Julian Assange, Khan Academy, kremlinology, Laura Poitras, Mark Zuckerberg, Maui Hawaii, MITM: man-in-the-middle, national security letter, operational security, Panopticon Jeremy Bentham, pre–internet, Ralph Waldo Emerson, rolodex, Rubik’s Cube, Silicon Valley, Skype, social graph, Steve Jobs, TechCrunch disrupt, undersea cable, web application, WikiLeaks

The shared goal among the US’s disgruntled allies was to make it harder for the NSA to get access to national data. For authoritarian countries such as Russia, there was an added bonus. Greater state control of the internet made it easier to snoop on their own citizens and keep a lid on dissent. The most vociferous reaction came from Brazil. In October, Rousseff announced plans to build a new undersea cable linking South America with Europe. This would, in theory, shut out the US and make it harder for the NSA to siphon off Brazilian information. The president also mulled over legislation that would force Google and other US tech giants to store the data for Brazilian users on local servers. Thousands of federal workers, meanwhile, were ordered to adopt a form of highly encrypted email.

The System: Who Owns the Internet, and How It Owns Us by James Ball

"World Economic Forum" Davos, behavioural economics, Big Tech, Bill Duvall, bitcoin, blockchain, Cambridge Analytica, Chelsea Manning, cryptocurrency, digital divide, don't be evil, Donald Trump, Douglas Engelbart, Edward Snowden, en.wikipedia.org, fake news, financial engineering, Firefox, Frank Gehry, Internet of things, invention of movable type, Jeff Bezos, jimmy wales, John Gilmore, John Perry Barlow, Julian Assange, Kickstarter, Laura Poitras, Leonard Kleinrock, lock screen, Marc Andreessen, Mark Zuckerberg, Menlo Park, military-industrial complex, Minecraft, Mother of all demos, move fast and break things, Network effects, Oculus Rift, packet switching, patent troll, Peter Thiel, pre–internet, ransomware, RFC: Request For Comment, risk tolerance, Ronald Reagan, Rubik’s Cube, self-driving car, Shoshana Zuboff, Silicon Valley, Silicon Valley startup, Skype, Snapchat, Steve Crocker, Stuxnet, surveillance capitalism, systems thinking, The Chicago School, the long tail, undersea cable, uranium enrichment, WikiLeaks, yield management, zero day

From there, it makes its final stop – to a server in a Twitter data centre in Kansas, from where I can get the website I would like to see. This connection is pretty simple – most of the stops along the way actually belong to the company I pay for my internet. But Twitter doesn’t actually own any of the undersea cables across the Atlantic, so is paying someone for its use, meaning that this route shows at least some money must change hands. That’s not an unreasonable proposition, as building and maintaining the transatlantic cables is something of an astonishing task. The cables are smaller than you might think – about the width of a hosepipe for much of their distance, and most of that is just armour for the tiny fibre-optic core – but need to be laid, maintained and monitored.

The Dead Hand: The Untold Story of the Cold War Arms Race and Its Dangerous Legacy by David Hoffman

Able Archer 83, active measures, anti-communist, banking crisis, Berlin Wall, Boeing 747, Chuck Templeton: OpenTable:, crony capitalism, cuban missile crisis, disinformation, failed state, guns versus butter model, It's morning again in America, joint-stock company, Kickstarter, Korean Air Lines Flight 007, launch on warning, Mikhail Gorbachev, military-industrial complex, mutually assured destruction, nuclear winter, Oklahoma City bombing, radical decentralization, Robert Hanssen: Double agent, rolodex, Ronald Reagan, Ronald Reagan: Tear down this wall, Seymour Hersh, Silicon Valley, standardized shipping container, Stanislav Petrov, Strategic Defense Initiative, Thomas L Friedman, undersea cable, uranium enrichment, Vladimir Vetrov: Farewell Dossier, warehouse robotics, zero-sum game

The KGB, he recalled, harvested a rich crop of secrets from a walk-in to the Soviet Embassy in 1980, an employee of the National Security Agency, which ran American global electronic eavesdropping. Yurchenko said he only knew of this agent as "Mr. Long," and gave his debriefers some details. He said Mr. Long sold to the Soviets the details of the U.S. operation to tap the Soviet undersea cables in the Sea of Okhotsk. This was the monitoring operation known as Ivy Bells, which had been discovered and removed by the Soviets in 1981. (A second undersea cable-tapping operation in the Barents Sea had not been compromised.) The FBI launched a manhunt for Mr. Long, and four months later arrested Ronald Pelton, a communications specialist with the NSA who sold the classified data to the Soviets for $35,000.

Pax Technica: How the Internet of Things May Set Us Free or Lock Us Up by Philip N. Howard

Aaron Swartz, Affordable Care Act / Obamacare, Berlin Wall, bitcoin, blood diamond, Bretton Woods, Brian Krebs, British Empire, butter production in bangladesh, call centre, Chelsea Manning, citizen journalism, Citizen Lab, clean water, cloud computing, corporate social responsibility, creative destruction, crowdsourcing, digital map, Edward Snowden, en.wikipedia.org, Evgeny Morozov, failed state, Fall of the Berlin Wall, feminist movement, Filter Bubble, Firefox, Francis Fukuyama: the end of history, Google Earth, Hacker News, Howard Rheingold, income inequality, informal economy, information security, Internet of things, John Perry Barlow, Julian Assange, Kibera, Kickstarter, land reform, M-Pesa, Marshall McLuhan, megacity, Mikhail Gorbachev, mobile money, Mohammed Bouazizi, national security letter, Nelson Mandela, Network effects, obamacare, Occupy movement, off-the-grid, packet switching, pension reform, prediction markets, sentiment analysis, Silicon Valley, Skype, spectrum auction, statistical model, Stuxnet, Tactical Technology Collective, technological determinism, trade route, Twitter Arab Spring, undersea cable, uranium enrichment, WikiLeaks, zero day

There is no consciousness-raising singularity here. But at the end of March 2013, more traffic flowed between China, Korea, and Japan than flowed between the United States, the United Kingdom, and Europe. A significant amount of digital traffic flows through the cables at the bottom of the Pacific Ocean. However, it’s not undersea cables that carry the most traffic, it’s overland cables. The fastest-growing region for internet traffic is within Asia—between China’s largest cities and between China’s cities and other cities in Asia. Indeed, one of the more pressing infrastructure problems in China is the need for faster internet connections from Shanghai, Hong Kong, and Beijing out to the country’s many large provincial capitals.

Who Gets What — and Why: The New Economics of Matchmaking and Market Design by Alvin E. Roth

Affordable Care Act / Obamacare, Airbnb, algorithmic trading, barriers to entry, behavioural economics, Berlin Wall, bitcoin, Build a better mousetrap, centralized clearinghouse, Chuck Templeton: OpenTable:, commoditize, computer age, computerized markets, crowdsourcing, deferred acceptance, desegregation, Dutch auction, experimental economics, first-price auction, Flash crash, High speed trading, income inequality, Internet of things, invention of agriculture, invisible hand, Jean Tirole, law of one price, Lyft, market clearing, market design, medical residency, obamacare, PalmPilot, proxy bid, road to serfdom, school choice, sealed-bid auction, second-price auction, second-price sealed-bid, Silicon Valley, spectrum auction, Spread Networks laid a new fibre optics cable between New York and Chicago, Steve Jobs, The Wealth of Nations by Adam Smith, two-sided market, uber lyft, undersea cable

Thus responding to prices on the English market (which reflected supply and demand there) essentially took the better part of a month. After the cable was completed, price information could cross the Atlantic in a day. As a result, cotton shipments began to more closely match the fluctuations in the market, and prices on the cotton market became less volatile. Before the undersea cable, it was harder to match supply with demand because the news that arrived in New York was already more than a week old—slower than the pace of events even in that era. Speeding up the news let traders react to the market better. Speed had made for a better market in cotton. But as we’ll see next, speeding up the arrival of news so that decisions can be made with better information is different from speeding up decisions so that they have to be made before news arrives.

The Twittering Machine by Richard Seymour

4chan, anti-communist, augmented reality, behavioural economics, Bernie Sanders, Big Tech, Black Lives Matter, Cal Newport, Californian Ideology, Cass Sunstein, Chelsea Manning, citizen journalism, classic study, colonial rule, Comet Ping Pong, correlation does not imply causation, credit crunch, crisis actor, crowdsourcing, dark triade / dark tetrad, disinformation, don't be evil, Donald Trump, Elon Musk, Erik Brynjolfsson, Evgeny Morozov, fake news, false flag, Filter Bubble, Gabriella Coleman, gamification, Google Chrome, Google Earth, hive mind, informal economy, Internet of things, invention of movable type, invention of writing, James Bridle, Jaron Lanier, Jeremy Corbyn, Jon Ronson, Jony Ive, Kevin Kelly, Kevin Roose, knowledge economy, late capitalism, Lewis Mumford, liberal capitalism, Mark Zuckerberg, Marshall McLuhan, meta-analysis, Mohammed Bouazizi, moral panic, move fast and break things, Network effects, new economy, packet switching, patent troll, Philip Mirowski, post scarcity, post-industrial society, post-truth, RAND corporation, Rat Park, rent-seeking, replication crisis, sentiment analysis, Shoshana Zuboff, Silicon Valley, Silicon Valley ideology, Skinner box, smart cities, Snapchat, Social Justice Warrior, Steve Bannon, Steve Jobs, Stewart Brand, Stuxnet, surveillance capitalism, TaskRabbit, technological determinism, technoutopianism, TED Talk, the scientific method, Tim Cook: Apple, Timothy McVeigh, Twitter Arab Spring, undersea cable, upwardly mobile, white flight, Whole Earth Catalog, WikiLeaks

It supposedly represents a unique person, living her best life, from the best angle, in the best light. But it does so using a technology that, as Adam Greenfield puts it, distributes and smudges the self across ‘a global mesh of nodes and links’.60 This hard infrastructure, from sensors in the smartphone to cellular base stations, undersea cables, microwave relays and networks of users, organizes from end to end a person’s experience of the world, her selfhood. In addition to breaking up the self into digitalized components, the technologies of the selfie also have the alarming effect of making everyone look the same. Some of the repetitive banality of selfies can be blamed on the conventions of selfie-taking.

There's a War Going on but No One Can See It by Huib Modderkolk

AltaVista, ASML, Berlin Wall, Big Tech, call centre, COVID-19, disinformation, Donald Trump, drone strike, Edward Snowden, end-to-end encryption, Evgeny Morozov, fake news, Fall of the Berlin Wall, Firefox, Google Chrome, information security, Jacob Appelbaum, John Markoff, Julian Assange, Laura Poitras, machine translation, millennium bug, NSO Group, ransomware, Skype, smart meter, speech recognition, Stuxnet, undersea cable, unit 8200, uranium enrichment, WikiLeaks, zero day

From these exchanges, data packets continue on their way to the data centre of the recipient’s ISP. That might be in the same country or it might be halfway around the world, in which case the packets speed across the oceans along thick, multi-layered, black cables on the seabed. Some of those undersea cables enter Europe through the Netherlands, at Beverwijk and Katwijk on the west coast and in the north at Eemshaven in the province of Groningen. The location of that last cable was enough to persuade Google to build a data centre in Groningen, conveniently close to the transatlantic superhighway.

The Girl in the Road by Monica Byrne

Kim Stanley Robinson, medical residency, quantum entanglement, stem cell, undersea cable, uranium enrichment

Well, because of breakthroughs in materials science, the TALG is anchored to the seafloor by means of Gossamoor, synthetic silk modeled on the draglines of Darwin’s bark spider, native to Madagascar, which is not only the strongest substance known but weighs about twelve milligrams per thousand meters. And thanks to HydraCorp’s partnership with China Telecom, the anchors parallel the SEA-ME-WE 3 undersea cable that carries data between Mumbai and Djibouti before veering up the Red Sea. And how does the TALG survive the many intrusions of maritime traffic? Well, gentle viewer, it turns out that the segments are programmed to sense oncoming ships and take on seawater, sinking up to thirty meters to let the ship pass, and then pumping the water back out to regain buoyancy.

Daring Raids of World War Two: Heroic Land, Sea and Air Attacks by Peter Jacobs

friendly fire, three-masted sailing ship, undersea cable

On arrival in Australia during April, the captain of the Bonaventure, Captain William Fell, actively sought opportunities for the midgets to be used. Initially, there were no possible targets identified, but the following month Fell was given the opportunity to use his midgets to cut some telegraph cables off Hong Kong. By cutting the undersea cables connecting Singapore, Saigon, Hong Kong and Tokyo, the Japanese would be forced to use radio communications and, therefore, open themselves to message interception. Missions to cut the Hong Kong to Saigon telephone cables were carried out in July 1945 by two midget subs under Operations Sabre and Foil.

Cyber War: The Next Threat to National Security and What to Do About It by Richard A. Clarke, Robert Knake

air gap, barriers to entry, complexity theory, data acquisition, Dr. Strangelove, escalation ladder, Golden arches theory, Herman Kahn, information security, Just-in-time delivery, launch on warning, military-industrial complex, MITM: man-in-the-middle, nuclear winter, off-the-grid, packet switching, RAND corporation, Robert Hanssen: Double agent, Ronald Reagan, Seymour Hersh, Silicon Valley, smart grid, South China Sea, Steve Jobs, systems thinking, Timothy McVeigh, trade route, undersea cable, Y2K, zero day

The notion contained in the “arsonist principle” is one that can be applied to cyber war. While we talk about cyberspace as an abstract fifth dimension, it is made up of physical components. These physical components, from the high-speed fiber-optic trunks, to every router, server, and “telecom hotel,” are all in sovereign nations, except perhaps for the undersea cables and the space-based relays. Even they are owned by countries or companies that have real-world physical addresses. Some people like to contend that there is a “sovereignty problem” on the Internet, that because no one owns cyberspace in its entirety, no one has any responsiblility for its integrity or security.

Everything for Everyone: The Radical Tradition That Is Shaping the Next Economy by Nathan Schneider

1960s counterculture, Aaron Swartz, Adam Curtis, Affordable Care Act / Obamacare, Airbnb, altcoin, Amazon Mechanical Turk, antiwork, back-to-the-land, basic income, Berlin Wall, Bernie Sanders, bitcoin, Black Lives Matter, blockchain, Brewster Kahle, Burning Man, Capital in the Twenty-First Century by Thomas Piketty, carbon footprint, carbon tax, Clayton Christensen, collaborative economy, collective bargaining, commons-based peer production, Community Supported Agriculture, corporate governance, creative destruction, crowdsourcing, cryptocurrency, Debian, degrowth, disruptive innovation, do-ocracy, Donald Knuth, Donald Trump, Edward Snowden, Elon Musk, emotional labour, Ethereum, ethereum blockchain, Evgeny Morozov, Fairphone, Food sovereignty, four colour theorem, future of work, Gabriella Coleman, gentrification, gig economy, Google bus, holacracy, hydraulic fracturing, initial coin offering, intentional community, Internet Archive, Jeff Bezos, Jeremy Corbyn, jimmy wales, John Perry Barlow, joint-stock company, Joseph Schumpeter, Julian Assange, Kevin Roose, Kickstarter, low interest rates, Lyft, M-Pesa, Marc Andreessen, Mark Zuckerberg, Marshall McLuhan, mass immigration, means of production, Money creation, multi-sided market, Murray Bookchin, new economy, offshore financial centre, old-boy network, Peter H. Diamandis: Planetary Resources, Pier Paolo Pasolini, post-work, precariat, premature optimization, pre–internet, profit motive, race to the bottom, Richard Florida, Richard Stallman, ride hailing / ride sharing, Rutger Bregman, Salesforce, Sam Altman, Satoshi Nakamoto, self-driving car, shareholder value, sharing economy, Silicon Valley, Slavoj Žižek, smart contracts, Steve Bannon, Steve Jobs, Steve Wozniak, Stewart Brand, surveillance capitalism, tech worker, TED Talk, transaction costs, Turing test, Uber and Lyft, uber lyft, underbanked, undersea cable, universal basic income, Upton Sinclair, Vanguard fund, Vitalik Buterin, W. E. B. Du Bois, white flight, Whole Earth Catalog, WikiLeaks, women in the workforce, working poor, workplace surveillance , Y Combinator, Y2K, Zipcar

AP’s origins lie in unrecorded conspiracies among New York City newspapermen between 1846 and 1848. Their papers were fierce competitors, but they were stuck. If they all used the new telegraph lines separately, there wouldn’t be enough lines. So the competitors had no choice but to cooperate—to import news from Europe via undersea cables, to co-sponsor horse-borne dispatches from the Mexican front, to circulate one another’s reporting. The New York papers shared ownership and control of the enterprise, and they sold their dispatches to other organizations beyond the docks of the Hudson.30 The temptations to hoard this news-gathering resource were immense, and their defeat came only gradually, over the course of a century.

Zeitgeist by Bruce Sterling

anti-communist, Ayatollah Khomeini, Berlin Wall, bread and circuses, cotton gin, Frank Gehry, Grace Hopper, informal economy, invisible hand, Iridium satellite, jitney, market bubble, Maui Hawaii, new economy, offshore financial centre, PalmPilot, rolodex, sexual politics, shareholder value, Silicon Valley, Ted Kaczynski, the scientific method, undersea cable, upwardly mobile, urban decay, Y2K

“ECHELON is the legendary capitalist global surveillance system. It’s the worldwide signals intelligence directorate! ECHELON is the crown jewel of the antiprogressive Dark Forces!” “Uh, yeah. That would be the alleged phenomenon.” “ECHELON is run by the UK, USA, Australia, and New Zealand. It uses undersea-cable taps, and surveillance satellites like ‘Aquacade,’ ‘Rhyolite,’ and ‘Magnum.’ It taps the Internet through its major routing centers and does comprehensive word searches on e-mail traffic.” “Hey, shut up,” Tim protested over the satellite phone. “That’s all totally classified.” Starlitz put the phone on his shoulder and squinted in the sunlight.

Ten Technologies to Save the Planet: Energy Options for a Low-Carbon Future by Chris Goodall

barriers to entry, carbon footprint, carbon tax, congestion charging, decarbonisation, electricity market, energy security, Indoor air pollution, Intergovernmental Panel on Climate Change (IPCC), Kickstarter, land tenure, load shedding, New Urbanism, oil shock, profit maximization, Silicon Valley, smart grid, smart meter, statistical model, undersea cable

One of the crucial reasons this location was chosen, apart from its tidal speeds, was the availability of a nearby connection to the electricity transmission system. One of the problems with tidal energy around the world is going to be connecting the turbines to the power grid. The best tidal locations tend to be far from high-voltage transmission lines and to use the full force of the Pentland Firth, the U.K. will need a new offshore undersea cable running down the east coast to London and on to the rest of Europe, using the same HVDC technology that will bring Saharan solar power to Germany and other countries, as described in Chapter 2. The third high-profile competitor for the wide-open market for tidal power generators is the Irish firm OpenHydro.

The Perfect Police State: An Undercover Odyssey Into China's Terrifying Surveillance Dystopia of the Future by Geoffrey Cain

airport security, Alan Greenspan, AlphaGo, anti-communist, Bellingcat, Berlin Wall, Black Lives Matter, Citizen Lab, cloud computing, commoditize, computer vision, coronavirus, COVID-19, deep learning, DeepMind, Deng Xiaoping, Edward Snowden, European colonialism, fake news, Geoffrey Hinton, George Floyd, ghettoisation, global supply chain, Kickstarter, land reform, lockdown, mass immigration, military-industrial complex, Nelson Mandela, Panopticon Jeremy Bentham, pattern recognition, phenotype, pirate software, post-truth, purchasing power parity, QR code, RAND corporation, Ray Kurzweil, ride hailing / ride sharing, Right to Buy, self-driving car, sharing economy, Silicon Valley, Skype, smart cities, South China Sea, speech recognition, TikTok, Tim Cook: Apple, trade liberalization, trade route, undersea cable, WikiLeaks

“We assess with high confidence that the increasing role of international companies and foreign individuals in US information technology supply chains and services will increase the potential for persistent, stealthy subversions,” the US National Security Agency (NSA), charged with intercepting communications around the world, wrote in a 2010 slide leaked by the whistleblower Edward Snowden.31 Edward Snowden’s leaked documents also revealed that the NSA was tracking twenty Chinese hacker teams attempting to break into US government networks and companies like Google. Huawei was also laying undersea cables that the NSA hoped to intercept, so it could read the communications of Huawei customers that happened to be “high-priority targets,” like Cuba, Iran, Afghanistan, and Pakistan.32 The NSA hacked Huawei’s headquarters, monitored the communications of its top executives, and ran an operation called Shotgiant aimed at finding a link between Huawei and the People’s Liberation Army.

Nomad Century: How Climate Migration Will Reshape Our World by Gaia Vince

3D printing, An Inconvenient Truth, Anthropocene, biodiversity loss, bitcoin, Boris Johnson, carbon tax, charter city, circular economy, clean water, colonial exploitation, coronavirus, COVID-19, decarbonisation, degrowth, Donald Trump, Dunbar number, European colonialism, failed state, gentrification, global pandemic, Global Witness, green new deal, Haber-Bosch Process, high-speed rail, housing crisis, ice-free Arctic, illegal immigration, Intergovernmental Panel on Climate Change (IPCC), invention of agriculture, invention of the printing press, job automation, joint-stock company, Kim Stanley Robinson, labour mobility, load shedding, lockdown, low skilled workers, Mahatma Gandhi, Malacca Straits, mass immigration, mass incarceration, mega-rich, megacity, negative emissions, new economy, ocean acidification, old age dependency ratio, open borders, Patri Friedman, Peace of Westphalia, Pearl River Delta, Peter Thiel, place-making, planetary scale, plyscraper, polynesian navigation, quantitative easing, randomized controlled trial, rewilding, Rishi Sunak, sharing economy, Shenzhen special economic zone , Silicon Valley, special economic zone, trade route, transatlantic slave trade, undersea cable, urban planning, urban sprawl, white flight, women in the workforce, working-age population, zero-sum game, Zipcar

High-voltage direct-current transmission lines will then relay this power to the cities in higher latitudes, as well as to regional cities. Models for such systems already exist – Australia, for instance, is currently building the world’s largest solar battery installation in its northern desert, which will send 24-hour renewable electricity to Singapore via a 4,500-kilometre undersea cable system, to be completed by 2027. Australia already produces far more solar power than it can use or store. North Africa is also sending high-voltage direct current to Europe from its solar arrays, and more regional transmission is promised from hydropower plants to cities. Africa’s deserts are prime for large-scale solar and wind power – Morocco already has the world’s biggest concentrated solar power project, in Ouarzazate, with others under construction.

GCHQ by Richard Aldrich

belly landing, Berlin Wall, Bletchley Park, British Empire, Charles Babbage, colonial exploitation, cuban missile crisis, disinformation, friendly fire, illegal immigration, index card, it's over 9,000, lateral thinking, machine translation, Menlo Park, Mikhail Gorbachev, Neil Kinnock, New Journalism, operational security, packet switching, private military company, Robert Hanssen: Double agent, Ronald Reagan, Seymour Hersh, social intelligence, South China Sea, Suez crisis 1956, undersea cable, unit 8200, University of East Anglia, Yom Kippur War, Zimmermann PGP

(‘Fred’) Stannard RN 1965–1969 Directors of Communications-Electronics Security Group 1969– Air Vice Marshal Arthur Foden 1969–1975 Brian Tovey 1975–1978 Dr John Johnson 1978–1980 Major Gen Alistair Anderson 1980–1985 Paul Foster 1985–1989 Air Vice Marshal John Porter 1989–1991 Andrew Saunders 1991–1998 Richard Walton 1999–2002 Huw Rees 2002–2005 Dr John Widdowson 2005– Appendix 2 GCHQ Timeline 1 Nov. 1919 GC&CS formed from a merger of the Army’s MI1b and the Navy’s Room 40 1921 Alastair Denniston becomes Director of GG&CS Aug. 1939 GC&CS moves to Bletchley Park to avoid wartime bombing 1942 Holden agreement on Anglo-American naval sigint 1943 BRUSA agreement on Anglo–American military sigint 1944 Revised Holden agreement on Anglo–American naval sigint 1944 Edward Travis becomes Director of GC&CS Feb. 1944 Cypher Policy Board created due to penetration of Allied cyphers 15 Sept. 1945 US Army code-breakers redesignated Army Security Agency (ASA) 22 Feb. 1946 Commonwealth sigint cooperation conference in London begins 5 Mar. 1946 Revised BRUSA agreement on Anglo–American sigint cooperation May 1946 Revised BRUSA technical appendices on Anglo–American sigint cooperation Jul. 1948 UK–USA Communications Intelligence Agreement 1 Aug. 1948 Captain Edmund Wilson and his team are shown over Oakley and Benhall sites 29 Oct. 1948 ‘Black Friday’ – a major change occurs in Russian cypher procedures 1 Nov. 1948 The term ‘London Signals Intelligence Centre’ is abolished in favour of GCHQ 20 May 1949 US Armed Forces Security Agency created, a weak forerunner of NSA Aug. 1949 Loss of the USS Cochino 1949 Vienna tunnel operations begin 10 Mar. 1950 US Communications Intelligence Board created Mar. 1952 Eric Jones becomes Director of GCHQ, taking over from Edward Travis Jul. 1952 Move from Eastcote to Cheltenham begins 24 Oct. 1952 President Truman signs order to create NSA, following failure to warn of Korean War 1953 LCSA created with Major General William Penney as Director 12 Mar. 1953 Loss of an RAF Lincoln over the Inner German Border Feb. 1954 Move from Eastcote to Cheltenham completed 2 Sept. 1954 Work on the Berlin tunnel begins 21 Apr. 1956 Eastern Bloc troops break into the Berlin tunnel 1956 NSA moves to new headquarters at Fort Meade 1957 RAF Hambuhren handed over to German communications units 1957 Move from HMS Anderson sigint site to Perkar on Ceylon 1957 Government White Paper suggests coming end of National Service 1957 Karamursel NSA station built in Turkey to the south of Istanbul 11 Oct. 1957 Jodrell Bank, the first radiotelescope, is completed and is also used for sigint 1 Nov. 1957 Captain Robert Stannard RN becomes Director of LCSA, taking over from William Penney 21 Aug. 1958 The RAF’s sigint unit, 192 Squadron, is renumbered as 51 Squadron 2 Sept. 1958 US Sigint C-130A Hercules shot down over Armenia with loss of seventeen crew 1958 Berlin tunnel translators move from SIS to GCHQ as London Processing Group 1958 Comet sigint aircraft enter service with 51 Squadron 1959 1 Special Wireless Regiment renamed 13 Signals Regiment May 1959 RAF Habbaniya in Iraq closed and sigint personnel moved to Cyprus 3 Jun. 1959 One of 51 Squadron’s new Comets destroyed by fire 1 Sept. 1959 2 Wireless Regiment at Ayios Nikolaos renamed 9 Signals Regiment 12 Nov. 1959 The first dedicated US sigint-gathering ship, USS Oxford, is authorised 1 May 1960 Shooting down of Gary Powers’ U-2 spy plane 1960 Clive Loehnis becomes Director of GCHQ, taking over from Eric Jones Sept. 1960 Joe Hooper introduces joint sigint equipment purchase with services Dec. 1960 Templer Report on integration of service interception 1961 George Blake arrested and early compromise of Berlin tunnel realised 1961 Collection begins from Teufelsberg in Berlin operating out of ASA vans Nov. 1961 Project Sandra receives Treasury approval 1962 Perkar sigint site on Ceylon closed Jul. 1962 Sir Stuart Hampshire’s review of GCHQ Dec. 1962 Hampshire visits NSA for three weeks Mar. 1963 LCSA becomes London Communications–Electronics Security Agency May 1963 Hampshire review of GCHQ completed Jun. 1963 Permission for a communications facility on Diego Garcia requested 1963 MC 74/1 – NATO Cryptographic Policy agreed 1963 Project Sandra begins operations on Cyprus 1 Jan. 1964 Little Sai Wan station in Hong Kong passes from RAF control to GCHQ Sept. 1964 Cypher clerks at the British Embassy in Moscow repel KGB ‘firemen’ 1965 Joe Hooper becomes Director of GCHQ, taking over from Clive Loehnis 1965 LCSA takes over SCDU and JSRU and is renamed CESD Nov. 1965 Diego Garcia agreement completed and 2,300 inhabitants removed 1966 Pine Gap station opened in Australia Oct. 1966 Decision taken to replace sigint Comets with Nimrod R1s 1967 Scharfoldendorf station near the Inner German Border closed 7 Jun. 1967 USS Liberty AGTR ship attacked off Israel 1967 Chapman Pincher and the ‘D-Notice affair’ exposes cable vetting 1967 Teufelsberg sigint site in Berlin begins operations Dec. 1967 First voyage of the USS Pueblo Jan. 1968 Geoffrey Prime offers his services to the KGB 23 Jan. 1968 USS Pueblo ALGER-2 ship captured by North Korea 24 Jan. 1968 HMS Totem, now renamed Dakar, sinks off Cyprus 20 Aug. 1968 Russian invasion of Czechoslovakia takes GCHQ and NSA by surprise Late 1968 Dick White review of DIS and JIC in the wake of Czech invasion Late 1968 Dick White inquiry into rising sigint costs 9 Sept. 1968 Geoffrey Prime begins work at GCHQ’s London Processing Group 1969 GCHQ’s nuclear-powered sigint ship project abandoned Jun. 1969 Decision to merge CESD with GCHQ and change its name to CESG 1969 Arthur Foden becomes Director of CESG, taking over from Robert Stannard 30 Mar. 1972 Kizildere incident: eight GCHQ staff taken hostage and three killed by TPLA terrorists Apr. 1972 JIC (A) delegation meets Groupe de Synthèse et Prévision in Paris 1972 Skynet III decision 6 Mar. 1973 Ryolite satellite launched to verify the SALT 1 arms control treaty 18 Jun. 1973 Closure of Cobra Mist facility at Orford Ness announced Aug. 1973 Arthur Bonsall becomes Director of GCHQ, taking over from Joe Hooper Aug. 1973 Nixon–Kissinger ‘cut-off’ of intelligence cooperation attempted 1973 Transfer of London Processing Group to Cheltenham begun 1973 James Ellis discovers the asymmetric algorithm, later the foundation of RSA 3 May 1974 First operational flight by a Nimrod R1 1974 Cyprus invasion by Turkey 1974 US sigint bases in Turkey shut down Feb. 1975 Government Secure Speech Network cancelled Jul. 1975 Ivy Bells undersea cable-tapping operation begins using USS Halibut Aug. 1975 Work on Diego Garcia expansion begins 1975 GCHQ Mauritius station closed 1975 NSA takes delivery of its first Cray computer Sept. 1975 Brian Tovey becomes Director of CESG, taking over from Arthur Foden 22 Mar. 1976 Geoffrey Prime moved from London to Cheltenham 1976 ‘ABC trial’ of Aubrey, Berry and Campbell begins 1976 Geoffrey Prime promoted to Section Head in J Division 1977 Transfer of London Processing Group to Cheltenham completed 1977 14 Signals Regiment (Electronic Warfare) formed 1977 GCHQ’s Wincombe station closed 1977 GCHQ’s Flowerdown station closed 28 Sept. 1977 Geoffrey Prime resigns from GCHQ 1978 GCHQ’s Gilnahirk station in Northern Ireland closed 1978 Special Collection Service, a joint NSA–CIA black-bag unit created 1978 GCHQ station at Two Boats on Ascension Island reactivated 1978 John Johnson becomes Director of CESG, taking over from Brian Tovey 1978 Brian Tovey becomes Director of GCHQ, taking over from Arthur Bonsall Jan. 1979 Iranian revolution – NSA and GCHQ listening posts in Iran are lost 23 Feb. 1979 One-day strike triggers Brian Tovey’s thinking on union removal at GCHQ 1980 Alastair Anderson becomes Director of CESG, taking over from John Johnson 1980 Ivy Bells submarine tapping operation blown by Ronald Pelton, KGB agent in NSA 9 Mar. 1981 One-day strike at GCHQ, then disruptive action to April 16 Nov. 1981 Geoffrey Prime makes his last contact with the KGB in East Berlin 1982 Gordon Welchman threatened with OSA over Hut Six Story 1982 GCHQ’s Little Sai Wan closed down and moved to Chum Hom Kok 26 Jun. 1982 Geoffrey Prime confesses 15 Jul. 1982 Prime remanded in custody on OSA charges 23 Sept. 1982 James Bamford’s Puzzle Palace published 10 Nov. 1982 Geoffrey Prime pleads guilty 1983 STU-III secure speech unit introduced by NSA Sept. 1983 Peter Marychurch becomes Director of GCHQ, taking over from Brian Tovey 1 Dec. 1983 Pilot polygraph machine arrives at R12 at Cheltenham Dec. 1983 Decision on de-unionisation taken by ad hoc Cabinet committee 25 Jan. 1984 GCHQ staff receive GN 100/84 letter on unions 17 Apr. 1984 WPC Yvonne Fletcher shot outside the Libyan People’s Bureau in London 1984 GCHQ’s Brora station in Sutherland closed 1985 Paul Foster becomes Director of CESG, taking over from Alastair Anderson 1985 Interception of Communications Act 1 Aug. 1985 KGB officer Vitaly Yurchenko defects and reveals NSA spy Ronald Pelton 5 Jun. 1986 Pelton convicted of espionage for the KGB 6 Jun. 1986 Jonathan Pollard pleads guilty to espionage for Mossad 1986 Joint Speech Research Unit at GCHQ amalgamated with Speech Research Group at RSRE to form Speech Research Unit 20 Jan. 1987 European Commission for Human Rights declares GCHQ trade union case inadmissible 31 Jul. 1988 Hawklaw station in Fife closed 1989 Hilltop station at Teufelberg in Berlin closed 1989 John Porter becomes Director of CESG, taking over from Paul Foster 1989 John Adye becomes Director of GCHQ, taking over from Peter Marychurch Mar. 1991 Andrew Saunders becomes Director of CESG, taking over from John Porter Late 1993 Sir Michael Quinlan asked to look at government spending on intelligence 22 Dec. 1993 T of R for Quinlan’s ‘Review of Intelligence Requirements and Resources’ agreed Jun. 1994 ‘Review of Intelligence Requirements and Resources’ completed 1994 GCHQ’s Earl’s Court station at Empress Building closed Oct. 1994 Operations at GCHQ’s Chum Hom Kok station end 2 Nov. 1994 Intelligence Services Act 12 Dec. 1994 Roger Hum Special Study of GCHQ commissioned Jan. 1995 Chum Hom Kok closed down and operations move to Geraldton in Australia Jan. 1995 Operations at Cheadle end and station closed in June 25 Mar. 1995 Roger Hum Special Study completed Mar. 1995 13 Signals Regiment in Germany disbanded, some move to JSSU at RAF Digby 16 May 1995 XW666, one of the three sigint Nimrod R1s, ditches in the Moray Firth 1995 RAF 51 Squadron moves from RAF Wyton to RAF Waddington 16 Oct. 1995 New high-level post created to represent GCHQ in London 6 Nov. 1995 J, K and V Divisions abolished.

…

AB Cryptoteknik 213 ‘ABC’ trial (1977–78) 8, 359–61, 423, 459 Abernethy, Barbara 70 Abu Hamza al-Masri 542 Abyssinia (Ethiopia) 19 Adams, Gerry 500 Aden (Yemen) 6, 156, 164, 345 Admiralty Signals Division 137 Admoni, Nahum 471 Adye, John 427, 431, 476, 483, 494, 495, 598, 608 aerial reconnaissance 31, 59 Afghanistan 387, 420, 421, 510, 511, 533–9 Africa 99, 148, 182, 268, 299, 336, 454–5, 479 Aid, Matthew 521 Aiken, John 326, 327, 328, 331 Airborne Rafter programme 267, 538 Aitken, Jonathan 493–5 al-Badr, Imam 163–4 al-Jazeera 513–14 al-Qaeda 9, 509–11, 513–14, 517, 532 Alanbrooke, Field Marshal Lord 45 Aldeburgh (Suffolk) 286 Aldridge, Peter 449 Alexander, A.V. 123–4 Alexander, Hugh 25, 27, 78, 96, 599 Allen, Lew 357 Allied Commission for Austria and London 170 Alp, Saffet 314 Alvear, Soledad 519 American Office of Naval Intelligence 143 American Type-777 satellite 348 Amery, Julian 162–3, 295 Ames, Aldrich 385, 444 Amory, Robert 158 Amsterdam 487 Anaya, Admiral Jorge 389, 393, 395 Anderson, Jack 344 Andrew, Christopher 2, 362 Anglo-American-Commonwealth sigint 64, 82, 152 Anglo-American relations 7–8; and Balkans 472–5; and Berlin tunnel operation 172–6; changing nature of 441–3, 449–50; and cost of cooperation 222–3; and Cuba 341–2; deterioration in 281–95, 333; and Far East 151–2; and global sigint 89–101; gradual improvement in 295–8; impact of politics on 278; and liberating of Axis sigint 47–56; and Nimrod programme 268–70, 273–4; and Project Sandra 322–3; and public disclosure of sigint material on 355–7, 358, 361–2; and sale of cypher machines 209–15; shared problems 333–4; and sigint 7–8, 38–46, 91–2; and sigint satellite and computer revolution 347–54, 437–8; successful Russian intercepts 279–81; and Suez crisis 157–9; and Third World bases 334–9; and trade unions at GCHQ 421–2, 429; in Turkey 302; and Venona Project 72–88; in wartime 38–46 Angola 357, 454–5 Ankara 58, 254, 302, 303, 305, 310–11, 313, 315, 318, 330 Annan, Kofi 523–4 Antalya (Turkey) 326 Anti-Christ Doom Squad 487 Anti-Smuggling Task Force (Hong Kong) 477 AQ Khan network 531 Aquarius (computer) 349 Arab-Israeli War (1973) see Yom Kippur War Arab states 109 Arafat, Yasser 277 Aral Sea 306 Arbuthnot, Mrs 70 Arctic Circle 133, 136–9, 144–6, 147, 265 Argentina 307–8, 388–415 Argentine Air Force 401, 408, 410, 415 Argentine Army 396 Argentine Navy 395, 405, 408 Argus satellite 377 Arlington Hall (US Army code-breaking centre, Washington) 45, 74, 77, 80, 83 arms control 203, 257, 287, 288, 290 Armstrong, Sir Robert 416, 425, 427, 428, 430–1 Army Intelligence Corps 230 Army Security Agency 80 as-Sallal, Abdullah 163–4 Ascension Island 162, 278, 321, 392, 414 Ash, Timothy Garton 465 Athens 324, 330 Atlantic, Battle of (WWII) 42, 60 Atlas (computer) 349–50 Atomic Demolition Munitions (ADMS) 249 Atomic Energy Authority (AEA) 261 Atomic Energy Intelligence 155 atomic and nuclear weapons 2, 5, 36, 45; and Anglo-US arms control 287–9; at Los Alamos 75, 76, 82; British 163, 249; Chinese 155; need for better intelligence on 253, 255, 321–3; Soviet 107–8, 112, 114, 116, 119, 131–2, 148, 157, 173, 301–2; spiralling of arms race 438–9; US 249; US put on alert (1973) 293–4 Atomic Weapons Establishment (Aldermaston, Berkshire) 418–19 Attlee, Clement 73, 86 ATV 432 Aubrey, Crispin 358–9, 360 Auckland (New Zealand) 487 Audiotel 480 Augsburg (Germany) 48 Austin, Harris M. 115 Australia 79, 80, 85–8, 89, 90, 92–4, 98, 154, 164, 165, 167, 168, 213, 467, 477, 487, 533; Australian Security Intelligence Organisation (ASIO) 86–7, 88; Central Bureau 92; Defence Signals Branch (DSB) 151, 153, 213; Defence Signals Department (DSD) 348; Royal Australian Air Force 150; Royal Australian Navy 266; Royal Australian Signals 166 Austria 96, 371, 384 Automatic Data Processing 353 Ayios Nikolaos (Cyprus) 156, 162, 163, 230, 234, 327, 328, 358–9, 360, 383, 384 Azerbaijan 109 Bad Aibling (Germany) 423 Bad Godesberg (Germany) 215 Badger’s Lair (SAS training exercise) 249–50 BAE Systems Ltd 544 Baghdad (Iraq) 161, 468, 469, 471, 524 Baghdad Pact 161 Bahia Paraios (Argentinean ship) 394, 395 Bahrain 347 Baillie, George 122 Bain, Helen 445 Baldwin, Stanley 18, 72, 400 Balgat (Turkey) 303 Bali bombing (2002) 511, 513 Balkanabteilung (German code-breaking HQ) 50 Balkans 51 Baltic 112, 114, 116, 125, 285 Baltic Sea 273 Bamford, James 263, 361–2, 521 Bandaranaike, Solomon 160 Bank of England 241, 487 Banner, Gordon 312–16, 318 Barbieri, Major 52, 53–4 Barents Sea 114, 115 Barker, Nick 391–2 Barkley, Howard 77 Barsby, Mrs 374–5 Basra (Iraq) 466, 525 Battle of Britain 29 Bay of Pigs crisis (1961) 8, 226 BBC 330, 348, 429, 501, 517, 523–4; BBC Scotland 459 Beach, Sir Hugh 380, 381 Bearman, Sid 595 Beasley, Tony 133–9 Beaumanor Hall (Leicestershire) 63 Beijing 476 Belbasi (Turkey) 302 Belfast 261, 500, 501 Belgium 442, 492 Bell, Rod 409, 410 Benitez, Rafael 114, 115 Benjamin, Ralph 216 Benn, Tony 227 Bennett, Ralph 59 Bentinck, Victor Cavendish 67 Bergen (Norway) 450 Bergold, Harry 295 Beria, Lavrentii 107 Berlin 112, 127–8, 130, 196–7, 227, 228, 253, 270, 369, 370, 372, 478; Berlin Blockade (1948) 71, 113; Berlin Cryptographic Centre 50; Berlin tunnel 169, 170, 172–6, 373, 477 Berlusconi, Silvio 532 Berry, John 358–9, 360 Betts, Richard 600 Beulmann, Major 50 Bevin, Ernest 70 BfV (German domestic security service) 452–3 bin Laden, Osama 511, 513, 514, 549 Binalshibh, Ramzi 514 Bingham, Lord 481 Birch, Frank 43 Bitburg (Germany) 131 Black, Jeremy 407, 408 black chambers 4, 14 ‘Black Friday’ (29 October 1948) 81, 108, 119, 169, 280 Black Sea 112, 131–2, 301, 302, 311, 313, 317, 319 Blair, Tony 4, 436, 497–8, 500, 504, 506, 509, 515, 517, 519, 530, 532 Blake, George 173, 174–5, 176, 178, 179, 238, 385 Bleckede 127 Bletchley Park 103, 186, 188, 221, 354, 549; Americans at 39, 43; and breaking ‘Red’ 25–6; closure of 67–71; and cypher security 54–9; expansion and reorganisation 27–8, 62–3; GCHQ as successor to 1, 5; Huts Three and Six 23, 25, 36, 48, 64–5, 119, 121, 356, 362–3, 364, 387; and machine-based espionage 5; military emphasis at 22–3; post-war role 60–1, 63–7; release of records on 355; SIS GC&CS moves to 22–8; takeover of Axis sigint effort 47–54; tight security measures at 69; unmasking of 362–3; wartime value and achievements 59–60, 61–3; wartime work of 1–2, 5, 25–9, 30, 31–46, 109–10 Blix, Hans 520 Block, Lieutenant 145 Bloomer-Reeve, Carlos 394, 412 ‘Blue Book’ 2–3, 395–6 Bluff Cove (Falklands) 400 Blum, Eberhard 450, 452 Blunkett, David 510, 522 Blunt, Anthony 36, 37, 83, 188, 224, 225, 364, 367 Board of Trade 241 Bodsworth, William 69, 80 Boizenburg (Germany) 127 Bolivia 52, 300 Bonnet, Georges 52 Bonsall, Arthur ‘Bill’ 110 Bontoft, Gerry 352 Borehamwood (aka ‘Department B’, London) 181, 182, 190 Borisenko, Alexsandr Ivanovich 236, 237 Borneo 6, 148, 164–8, 250 Bosnia 8, 9, 471–4, 512 BOSS (South African secret service) 357 Bourbon (Soviet radio intercepts) 75 Boyce, Ed 377 Bracknell (Berkshire) 113 Bradley, General Omar 116 Bradshaw, Mike 423 Brauntoltz, George 432 Bremerhaven (Germany) 116 Brezhnev, Leonid 245, 247 Brezhnev Doctrine 244 Bride (UK code-name for Venona) 80 Bridges, Sir Edward 42, 141–2, 181 Brinks Mat bullion robbery (1983) 505 British Army 14–15, 63, 451, 469, 472; 9 Signals Regiment 162, 327, 383–4; 13 Signals Regiment 166, 228, 493; 14 Signals Regiment 524, 525, 534; 40 Commando 525; British Army on the Rhine 248, 412; First Armoured Division 467, 525; Queen’s Dragoon Guards 525; ‘Rhino Force’ 467; Royal Armoured Corps 248; satellites 198, 208–9, 223, 243, 258, 262–3, 340, 342, 344, 345–8, 347–8, 376, 377, 401, 415, 421, 437–8, 441, 442–3, 445–7, 460–1, 478; sigint units 162, 166–7, 218–19, 228, 327, 383–4, 493, 524, 525, 534; Special Air Service (SAS) 164, 165, 168, 248–51, 359, 409, 410, 411, 468, 472, 475, 536, 583; Special Boat Service (SBS) 333, 409, 410, 411, 443, 468; Special Operations Executive (SOE) 36, 51; Special Reconnaissance Squadron 248 British European Airways (BEA) 129 British Indian Ocean Territories (BIOT) 334 ‘British Intelligence and Weapons of Mass Destruction’ (Butler Report) 2 British Leyland 367–8 British Military Mission (Brixmis) 123, 245–6, 247, 252 British Nuclear Fuels Ltd 500 British Psychological Society 433–4 British Tabulating Machine Company 349 British Telecom 545; Medium Wave Tower (Holyhead) 500 Britten, Douglas 230–8, 369, 382 Broadcasting Standards Authority 481 Broadside operation (US Embassy intercepts in Moscow) 280–1 Brockway, Ernst 383 Brook, Sir Norman 142, 219–20 Brooks, Richard 399 Brooks Field (Michigan) 120 Brown, Gordon 498 Brundrett, Sir Frederick 177, 178 Brunei, Sultan of 164, 586 BRUSA agreement (1943) 43, 44, 121, 151 Brussels 179, 253 Brzezinski, Zbigniew 448 Buffham, Benson 381, 424 Bufton, Air Vice Marshal 207 bugging operations 176–82, 193, 196–7, 473–4, 479, 482, 499 Bulganin, Nikolai 140 Bulgaria 518, 519 Bundesnachrichtendienst (BND, German foreign intelligence service) 214–15, 422–3, 438, 447–51, 452–4, 455, 456, 471, 472, 524 Bundy, William ‘Bill’ 50, 356, 363, 364 Bunyan, Tony 361 Burgess, Guy 8, 37, 73, 82, 83, 84–5, 224, 225, 238, 367 Burma 65 Burrough, John 598 Burton, Sir Edmund 527 Burton-Miller, T.R.W. 57, 191, 577 Bush, George H.W. 356–7 Bush, George W. 511, 517, 532 Butler, Lord 2, 428, 482, 529–31, 610 Byers, Stephen 506–7 Byrnes, Jimmy 53 ‘C’ (head of SIS) 16, 24, 67 Cabell, Charles 96 Cabinet Office Intelligence Coordinator 241, 245, 264, 353, 354, 360, 387, 399, 504 Cable, Danielle 505, 506 Cable, James 278 cable vetting 238–41 Cable & Wireless 240, 312, 392 Caccia, Harold 171 Cadogan, Sir Alexander 23–4, 26, 39, 45, 70 Cairncross, John 36–7, 73, 82, 364 Cairo (Egypt) 58, 159, 179, 185 Callaghan, James 324, 325, 330, 333, 361, 391 Calvi, Roberto 407 Calvocoressi, Peter 59, 61 Cameron, Stephen 505 Campaign for Nuclear Disarmament (CND) 368 Campbell, Alastair 517, 523, 530 Campbell, Duncan 8, 358, 360, 361, 362, 423, 458–9 Canada 38, 89, 93, 95, 97–8, 178, 447, 533; Canadian CBNRC 348; Canadian Communications Security Establishment (CSE) 381, 447; Canadian Joint Intelligence Committee 92 Canine, Ralph 101, 174, 336 Canyon satellite 376 Cape Canaveral (Florida) 322, 437, 461 Cape Matapan, Battle of (1941) 60 Capenhurst Tower (Cheshire) 500–1 Caraman, Mihai 253–4 Carey-Foster, William 185, 188, 228 Carlile of Berriew, Lord 543–4 Carpenter, Harry 598 Carrington, Lord 282–3, 284, 294, 389, 394, 396, 423–4 Carsamba (Turkey) 311–12 Carter, Jimmy 390, 448 Carter, Marshall 264, 272, 273–4, 350 Carter, Pat 448 Cartwright, Ian 328 Carver, Michael 334 Casey, Bill 457 Caspian Sea 112, 132, 161, 301, 302 Castle, Barbara 240 Castro, Fidel 226, 341 Catroux, Georges 52 Caucasus 131, 157 Caviar (Soviet encyphered traffic) 49, 69 Cayan, Mahir 309, 310–15, 314 Caygill, David 445 Celebes 167 Cellnet 481 Cemgil, Sinan 306 Central Gunnery School (Leconfield) 126 Central Signals Establishment 113–14 Ceylon 58, 69, 160, 259, 352 Chagos Islands 278, 335 Chamberlain, Neville 3, 18 Charles, Prince 482 Cheadle (Cheshire) 34, 63, 231 Chechnya 494–5 chemical weapons 470–1, 516 Chevaline project (upgrading of Polaris) 438–40, 459 Cheyne, Bill 166 Chicago Islands 278 Chicksands (Bedfordshire) 63, 360, 535 Chifley, Ben 86, 88, 94 Chile 357, 394, 396, 517 China 98, 129, 150–5, 193–5, 256, 273, 277, 282, 285, 376, 475–8; Chinese Communist Party 150; People’s Liberation Army 151 Chinese language teaching 598 Ching Peng 149, 150 Chippewa Falls (Minnesota) 350 Chirac, Jacques 520–1 Chitty, Brigadier 55 Chiverton, Roy 476 Chum Hom Kok (Hong Kong) 475–6 Church, Frank 356, 357 Church Committee 356, 357 Churchill, Winston 84; addiction to ‘Ultra’ intelligence 1; and airborne incidents 128, 129; and bugging operations 177; comments on the Belgians and Dutch 52; and cypher security 56; and diplomatic intelligence 41; and elint airborne operations 124; and Kuznetsov-Marshall affair 187, 189; meets Stalin 47; as recipient of sigint 3, 5, 40; supports work at Bletchley 26–7, 59, 65, 362 CIA (Central Intelligence Agency) 85, 87, 91; and Bay of Pigs 8, 226; and Berlin-Vienna tunnel operations 169–76; and counter-espionage coup 253–4; and Cyprus 324, 326–7; Foreign Broadcast Information Service 155; and invasion of Czechoslovakia 246; and invasion of the Falklands 399; Jewish sympathisers in 97; and Korean War 100; and Libyan terrorism 457; and Middle East 157, 158; and Nixon administration 4, 278; Office of National Estimates 151; revelations concerning 356–8; Russian spies in 444; security measures 101, 381; and shooting of Che Guevara 300; and Soviet weapons 108, 439; Special Activities Division 514–15; and Turkey 472; and U-2 spy planes 142, 226, 292, 296; and use of communication satellites 348; and Venona Project 77, 82; and Zinnia 322 Cianchi, Commander 54 CIB3 (Metropolitan Police anti-corruption squad) 506 City of London 487, 510 Civil Service 381, 421, 427; Civil and Public Servants Association (CPSA) 419–20; Civil Service Medical Officers Group 382; Civil Service Order in Council (1982) 431; Civil Service Union (CSU) 418, 420, 422–3; First Division Association 435; Society of Civil and Public Servants 426 Claret operations (1964–66) 165–8 Clarke, Kenneth 481, 482 Clarke, Liam 608 Clarke, Peter 266–7 Clarke, William F. 66 Cleveland, Paul 444–5 Clifford, Clark 333 Clinton, Bill 492 Clipper Chip (encryption bypass system) 492 Cobra (Cabinet emergency planning committee) 509–10, 527, 532, 610 Cobra Mist (Over the Horizon Radar) 285–7 code-breakers 2, 6, 198; and Anglo-American relationship 38–46; Australian 92; and Berlin tunnel operation 174; collaboration with Baltic states 31; combined operations 15–16; and computers 340, 348–9, 350–1; and decypherment of ‘Fish’ messages 28; diplomatic 27–8, 37, 43–4, 52–3; during WWII 25–9, 30–46, 58–9; Egyptian 164; and ending of WWII 61; expansion of 63–4; Far Eastern 40; Finnish 32, 35, 91; and French intercepts 52, 53, 209; and global sigint 96; importance of Hong Kong to 151; international work 16–17; and internet 493; Italian 52–4; military operations 14–15, 19–20; move from Bletchley to Cheltenham 5; naval 15; and Pelton affair 444; and personal computing and the internet 488; as Post Office department 14; pre-WWII 22–5; and Prime case 380; quadripartite meeting on computer hackers and encryption 489; revival on eve of WWI 14–15; Russian focus 17–19, 33–8, 169; and sale of cypher machines 209–15; and supply of intelligence reports 2–4, 6–7; and telephone tapping 170–1; and use of cypher machines 21–2; value of 60; and Venona Project 72, 74–88; wartime secrecy 354 COINS (Community On-line Intelligence System) 353 Colby, William ‘Bill’ 293, 327, 329, 330 Cold War 47, 377, 420; airborne incidents 125–33; benefits of intelligence in 175–6; Berlin blockade 71; calming of nerves in 257–8; computing in 349; end of 461, 465, 477, 478–9, 493; flashpoints 203; high profile espionage activity in 8; and planning for future war 247–53; seaborne incidents 133–47; secret service operations 484; sigint in 1, 2, 5, 107–24, 125–47, 402; Soviet nuclear weaponry 108; telephone and bugging operations 169–82; thawing of 195–7; and Venona Project 72–88 Cole, David 379, 380 Coleman, Don 131 Coleridge (Soviet teletype system) 78 Colombia 486, 538 Colonial Office 150 Colorob (computer) 349 Colossus (computer) 28, 48, 68, 70, 349 Combined Cypher Machine 98 Comet aircraft 121–2, 268, 273, 295 Cominstum (digest of hot material) 96 comint (communications intelligence) 96, 101, 110, 111, 122, 123, 228, 252–3, 266, 385, 402, 413 Comintern (Communist International) 19, 30, 37–8, 79 Commonwealth 85–8, 89, 92, 95, 97, 148, 266, 352, 447 Communications Branch of the National Research Council (CBNRC) 94 Communications Data Bill (2009) 543 Communications-Electronics Security Department 241–2 Communications Trials Ship (purpose-built sigint ship) 260–2 Communist Party of Great Britain 19, 188, 367–8, 417 Communist Party of India 87 Communist Party of USA 87 Comprehensive Comparative Radar Library 266 computers 198, 219, 220, 222, 340–1, 342–3, 348–54, 458, 486–93, 507, 507–8, 513, 527–8, 546–8 comsec (communications security) 90, 191–3, 195–7, 211, 218, 241–2, 377 Confederation of British Industry (CBI) 241 Conflict (Vienna tunnel code) 171 Congo crisis (1960) 336 Control Orders 543 Cook, Robin 498 Cooney Hill psychiatric hospital (Gloucester) 382 Cooper, Arthur 598 Cooper, Frank 423 Cooper, Josh 20, 34, 213 Coote, John 136, 137, 138, 143 Copenhagen (Denmark) 179 Corona (satellite) 208 Corporal (battlefield missile system) 249 Cosby, Bill 327 Costello, John 364 Cot, Pierre 53 counter-terrorism 456, 516, 528 Cox, Arthur 301 Crabb, Lionel ‘Buster’ 140–3, 207 Cradock, Percy 193–4, 294 Crankshaw, Edward 34, 35–6, 64 Cray super computers 350–1 Crete 157, 265, 292 Croatia 471–4 Croft, John 37–8 Cromer, Rowley 287–8, 294, 297, 337 Croslieve Mountain (Northern Ireland) 501 Crossman, Richard 227 Cruise missiles 511, 513 Crypto AG 212–15, 457 CSE (Communications Security Establishment) Watton 131 Cuba 226, 341–2; Cuban Missile Crisis (1962) 203, 253, 260 Cukr, Baclav 112 Cummings, Mansfield (aka ‘C’) 16 Cunningham, Andrew 91 Current Intelligence Groups 291, 396 Currie, Laughlin 87 Currier, Prescott 39 Curry, John 79 Curzon, George 15, 16, 18 cyber attacks 487–93 cypher machines; capture of 48–9, 264; Chinese capture of 194, 195; commercial origins of 20; development and use of 20–2, 57, 192; military 54; online 28, 210; and proposed Anglo-US collaboration 98–9; radiation or emanation from 215–18; supplying to NATO countries 209–15; see also Enigma Cypher Policy Board 42, 56–7 cypher security 54–9, 98–9, 191 Cyprus 7–8, 154, 155–6, 159, 161, 162–3, 219, 229, 234, 235, 259, 265, 277, 285, 292, 294, 295, 302, 319, 320–34, 337, 338, 345, 348, 356, 359, 369, 372, 382, 383–4, 419, 423, 466, 471, 504; Cyprus Eight 385 Czechoslovakia 203, 244–7, 253, 387; Czech Air Force Association 112 D-Notice affair (1967) 226, 238–41, 242, 362–3 Damascus (Syria) 159, 291, 300 Darwin (Falklands) 410 Data Encryption Standard (DES) 489 data-mining 486, 546–8 Daubney, Claude 103 Davies, Philip 27 de Gaulle, Charles 52, 195 de Grey, Nigel 35, 43 Deaf Aid (elint reception and analysis kit) 123 Dean, Patrick 142 Decabral, Alan 506 Defcon 3 (US nuclear alert) 293–4 Defence Intelligence and Security Centre (Chicksands) 63, 360, 535 Defence Intelligence Staff (DIS) 245, 246, 353, 369, 397, 402, 414, 527–8 Defence Reviews 329 Defence Signals Branch 93, 151, 153, 213, 578 Demos-1 (Hong Kong sigint station) 475–6 Demos-4 (Hong Kong sigint station) 476 Denham, Flight Sergeant 126 Denmark 442, 533 Denmark Hill (Metropolitan Police intercept station, London) 37 Denniston, Alastair 15, 18, 21, 24–5, 27, 28, 31, 34, 38, 40, 43–4, 70, 79 Denton Green, Robert 400, 402, 403, 407 Department of Economic Affairs 241 Department of Trade and Industry (DTI) 487 détente 247 Detica (security company) 544 Dexter, Harry 87 Dhekelia 163, 233, 327 Diana, Princess of Wales 479–83 Dictionary (keywords/predesignated phrases system) 343 Diego Garcia 321, 332, 335–9, 597 Dieppe raid (1942) 55 Diffie, Whitfield 490 Dimbleby, Jonathan 524 Dingli (Malta) 32, 156, 162 diplomatic intelligence 27–8, 37, 43–6, 52–3, 62, 69, 148, 159, 164, 176–82, 349, 355, 377 Diplomatic Protection Squad 506 Diplomatic Wireless Service (DWS) 58–9, 123, 181, 185–90, 192, 262, 417, 418, 585 Director General of Intelligence (DGI) 246 Directorate of Scientific Intelligence 123 Discovery (space shuttle) 437 Diyarbakir (Turkey) 300, 301–2, 306 Dobrynin, Anatoly 454 Domazet, Davor 472 domestic surveillance and intercepts 540–50 Donoughue, Bernard 3, 325 Doran, Frank 126 Doublecross system 229, 255 Douglas-Home, Sir Alec 163, 282, 312–13 Dozier, James 407, 452 Drake, Edward 94 Drew, John 229 drugs 486, 503, 514, 538 Drumheller, Tyler 529–30 Drummond (Argentine frigate) 395 DS.19 (MoD unit) 368 Dubček, Alexander 244 Dublin 501 Dudley-Smith, Russell 49, 55, 261 Duff, Antony 360, 399 Duffton, Nancy 435 Dulles, Allen 157, 158, 174, 176, 203 Dulles, John Foster 157 Dunderdale, ‘Biffy’ 21 Dunlap, Jack 355 Dunnell, Peter 127 Dwyer, Peter 82 East Africa 335–6 East Asia 120 East Germany 123, 131, 195–7, 345, 370, 379, 385, 453, 605 East-West summit (Paris, 1960) 204 Eastcote (London Signals Intelligence Centre) 62, 68–9, 79, 80, 103, 191, 349 Eastern Bloc 123, 175, 244, 245, 247, 256, 267, 282, 447, 465 Eastern Europe 33, 53, 78, 99, 119, 256, 284 Easton, James 82 Eavesdropper revelations (1976) 358, 359 Echelon (Anglo-US communications network) 7 economic intelligence 240–1 Ecuador 52 Eden, Anthony 46, 85, 129, 140, 141, 142–3, 155–6, 160, 178, 189 Eemnes (Netherlands) 415 Eger (Norwegian ship) 117 Egypt 58, 109, 155–9, 259, 263–4, 271, 277, 290–2, 295, 320, 467 Eichmann, Adolf 307–8 Eisenhower, Dwight D. 140, 157, 158, 202, 205, 219 Electrical Trades Union 368 Electronic Warfare Conferences 122, 123, 307–10 elint (electronic intelligence); air-based 111–14, 118–19, 122, 124, 250–2, 267–73; and Anglo-American relations 111–12; and European cooperation 591; in Germany 247, 250–1; and invasion of the Falklands 401, 413; and jamming of Whetstone monitoring station 190; land-based 117–20, 123; naval 114–17; near the Soviet Union 169; postwar expansion 110; rejection of 247; in Turkey 306; wartime use of 110 Elizabeth II, Queen 480, 482 Elkins, Robert 143 Elliott Brothers Ltd 598 Ellis, James 490, 492 Elmers School (GC&CS Diplomatic Sections) 23 email 488, 507, 513–14, 521–3, 541 embassies 151; Anglo-US-Canadian intercepts in Moscow 280–1; attacks and raids on 193–5; as forward listening stations 31; KGB in 82–3, 283–4; and MI5 watcher operations 183–90; security headache 195–7; sifting of waste-baskets in 56; spies in 84; tapping and bugging operations 171, 176–82, 193, 197–8, 281, 477; ultra-secret short-range sigint stations in 244–5; worldwide collection of intercepts from 45, 53, 79, 112, 159, 242–3, 385 Employment Select Committee of the House of Commons 424 Engulf (Egyptian Embassy cypher machine operation) 216 Enigma (German cypher machine) 1, 20–2, 23, 25–6, 27, 35, 38, 39, 42–3, 43, 51, 68, 78, 80, 354, 387 EOKA (Cypriot guerrilla force) 163 Episkopi (Cyprus) 234 Erim, Nihat 306, 312, 315 Eritrean Liberation Front 336 Escobar, Pablo 538, 549 Ethiopia 299, 334, 335–6 European Convention on Human Rights 433, 483 European Economic Community (EEC) 284 European Principals Meeting 450–1 European Union (EU) 540 Evatt, Dr H.V. 85, 86 Evere (NATO-GCHQ cell) 254, 255–7 Exocets (sea-skimming missiles) 390, 406–7, 414, 415 Faisal, King 160–1 Falkland Islands 6, 424, 429, 441, 442, 452, 467; Argentinean ambitions towards 389–92; Argentinean invasion of 392–3, 394–401; British troops on 408–14; comint and elint on 401, 413; diplomatic exchanges with 403–4; effect of war on British sigint 415; French help on 415; improvised communications with GCHQ 402–3; inadequate intelligence on 392–401; leaseback idea 392; naval action 404–8; near-miss air disaster 408; Norwegian help on 401, 442; scrap-metal incident 393–4; surprise attack on 388–9; Task Force sent to 398, 401, 403, 404–8; US denies pre-knowledge 601 Famagusta (Cyprus) 163, 235, 327, 328 Far East 39, 69, 78, 93, 129, 148–51, 164–8 Farrell, Terry 496 Faslane naval base (Scotland) 145, 146 Fatah (Palestinian organisation) 304, 308 FBI (Federal Bureau of Investigation) 76–7, 81 Federation of Malaysia 164–8 Ferranti 349, 598 ‘ferrets’ (flying intelligence stations) 111–14, 203–7 Fetherstone-Haugh, Timothy 383 Fetterlein, Ernst 17–18 Fieldhouse, Admiral 392 Fiji 446 Finland 31, 32, 76, 83, 91, 371, 489 First World War 14, 15, 16–17 Firyubin, Nikolay 279–80 Fischer, David 407 Fish (encyphered teleprinter) 28, 48, 49, 51 Fitz, Harold 127 Fleet Headquarters (Northwood) 400, 401, 402 Fletcher, WPC Yvonne 455–6 Florida 341 Flowers, Tommy 28, 349 Foden, Arthur 242 Foot, Michael 433 Ford, Gerald 297 Foreign and Commonwealth Office 16, 22, 45, 46, 56, 58, 66, 70, 83, 103, 128, 171, 172, 190–1, 192, 220–1, 239, 245, 273, 281, 287, 333, 335, 339, 353, 355, 360, 392–3, 398, 417, 420, 428; South-East European Department 317; Technical Maintenance Service 182 Forest Moor (wireless station near Harrogate) 96 Fort Bridgelands (Kent) 63 Fort Knox (Kentucky) 101 Fort Meade (NSA HQ, Washington) 102, 157, 174, 223, 271, 513, 528 Förvarets Radionstalt (FRA) 91 Foss, Hugh 21, 64 Fox, Katherine 598 FRA (Swedish sigint service) 421, 438, 456, 483–4 France 21, 32, 44, 52, 52–3, 109, 130, 268, 442, 445, 450, 467, 492 Franks, George 382 Free French 28, 52 Freedom of Information Act 482–3 Freeman, John 279 Freeman, Peter 531 French Guyana 415 Friedman, William 39, 44, 95, 213, 214 Friedrich, Lt Colonel 50 Fuchs, Klaus 72, 82, 83, 87, 104, 238 Fyjis-Walker, Richard 316 Fylingdales (Yorkshire) 287 Gaddafi, Muammar 455, 457, 531 Gagarin, Yuri 301 Gaitskell, Hugh 141 Galvin, John 474 Gambier-Parry, Richard 57, 181, 186, 188 Gamma-Guppy (Soviet intelligence intercepts) 244–5 Gardner, Meredith 75, 79, 80 Garner, Joe 249 Gates, Robert 457 GC&CS (Government Code and Cypher School) 361; and Anglo-American collaboration 40–1; civil achievements 28–9; and cypher security 56–7; diplomatic centre at Berkeley Street 27–8, 37, 43–4, 52–3; divided into civil and military sections 27–8; and European collaboration 20–2; military interests 19–20; post-war role 61, 63–7; relocation to Bletchley 22–7; Russian interests 17–19, 30–2; setting up of 16 GCHQ (Government Communications Headquarters) 1, 31, 104; and al Qaeda 511–12; Benhall 350, 360, 497, 526; and Bosnian-Croatian conflict 472–5; Bude (formerly CSO Morwenstow) 342, 343–4; budget figures 587; building of ‘Doughnut’ 9, 497, 526, 527–8; ceases exchanging intelligence with NSA 289–90; and changing nature of global threats 504–5; and closer relationship with MI5 and SIS 503–4; and Cobra Mist/Orford Ness problems 285–7; code-breaking and intelligence-gathering 6–8; combined NATO-GCHQ cell at Evere 254, 255–7; computers in 527–8; cooperation with NSA 222–3, 278, 282–3, 346, 347–54, 438, 448–58, 461; declining position of 422–3, 438, 441; development of new systems 342–54; diplomatic initiatives 108–9; Directors of 551–2; domestic surveillance and interceptions 540–50; DWS operations 186; E Division (Personnel) 425, 427; and economic intelligence 493; Empress Building (Earl’s Court) 382; encryption problems in banking and commerce 487–93; and end of Empire 148–55; expansion of 79, 169; F Division 476; file storage 598; funding of 219–23, 334, 458, 493, 494, 495; Free Trade Union 430; future purpose of 485; and global sigint 92, 94, 95–100; and Gulf War (1991) 466, 469; H Division (mathematicians and cryptographers) 432; and hoax letters prank 469–70; increased intelligence operations 120, 121–4; influence on foreign policy 321; installation of dedicated computer unit 507–8; and internal surveillance 9–10; and internet 100; and invasion of the Falklands 392–403, 405–6, 411; and IRA 498–503; and Iraqi dossiers 516–17, 530; J Division (Special sigint—Russian) 346, 374, 376, 419, 429, 434–5, 438, 495; J-Ops 429, 434–5, 438; K Division (non-Russian sigint) 218, 222, 402, 420–1, 478, 495; and KGB espionage 108, 189, 424–5; and Korean War 101; language problem 512–13, 516; legal identity of 484–5; and Libyan Embassy affair 455, 456; London office (Palmer Street) 192, 497; loss of Hong Kong listening station 475–8; merger with com sec 241–2; and Middle East 155–64; moles in 368–85; move to Cheltenham (1952) 102–3, 120–1, 122, 191; need for 8–9; ‘need to share’ problem 503–4; Nimrod programme 267–70, 271–4; and Noye affair 505–7; Oakley 360, 380, 427, 496, 497, 526; ocean-going activities 6, 136; organisation overviews 563–5; overhaul of operations 493–7; positive vetting at 227–8; post-war organisation and location 67–71; and Princess Diana 482, 483; and problems with ‘special relationship’ 441–3; promotion and career structures 576; purpose-built sigint ship 260–4; R Division (security) 425; reads HVA traffic 605; reinstatement of unions at 497–8; relationship with private companies 240; removal of trade unions at 416–36; and Russian problem 46, 71, 75, 78, 169, 299; S Division 261; secret pact with armed services 5–6, 117–18, 132–3; size of 227; ‘Station X’ 69; as successor to Bletchley Park 1, 5; suicides connected to 382–3; and supply of cypher machines to NATO 209–15; T Division 123; Tempest 216–18; Trade Union Campaign 498; Turkish operations 300–1, 311–19; unmasking of 355–64; use of deaf and dumb civilian personnel 153; use of name ‘GCHQ’ 61, 67; and use of polygraph at 425–6; and Venona Project 77–81; visibility of 1, 2, 8, 341, 436, 484; W Division 261; and War on Terror 533, 539–40; and West German defections 455–6; whistleblower in 521–3; X Division 6, 350, 353; Z Division (use of sigint) 388, 503; Zionist interests 109 GEC-Marconi 433 General Belgrano (Argentine cruiser) 404–6 General Strike (1926) 18 Geneva (Switzerland) 178; Peace Conference (1954) 178 George VI, King 59, 191 Georkadijis, Polycarpos 323–4 Geraldton (DSD site, Australia) 477 Germany 15, 29, 30, 31, 32–3, 44, 47–50, 55, 62, 78, 96, 127–9, 130, 142, 170–1, 219, 229, 247, 256, 270, 492; Army 26, 29, 43, 47; High Command (OKW) 49, 349; Navy 42; see also East Germany; West Germany Gezmiş, Deniz 310, 311 Giant Reach (SR-71 flights from US to Middle East) 292–3 Gibraltar 162, 398, 415 Gibson, Sir Peter 502 Gilbey, James 479–81 Glazebrook, George 92 Glidwell, Mr Justice 430–1 Glover, Sir James 413 Godfrey, Admiral 32 Golan Heights 297 Golden Valley Hotel (Cheltenham) 432–3 Goldsmith, Lord 522–3 Golombek, Harry 25 Goodpaster, Andrew 256 Goonhilly Downs (Cornwall) 342–3, 597 Goose Green (Falklands) 404, 410, 411 Gorbachev, Mikhail 456 Gordievsky, Oleg 385, 478 Gore Booth, Sir Paul 339 Gosport (Hampshire) 134 Gouzenko, Igor 85 Government Communications Staff Federation 428, 429 Government and Overseas Cable and Wireless Operators Association 418 Government Technical Assistance Centre (GTAC) 507, 547 Government Telecommunications Advisory Centre 504 Gow, Ian 482 Gowrie, Lord 428 Grab (Galactic Radiation and Background) satellite 208 Grant (MI5 computer scheme) 528 Grantham, Sir Alexander 152 Granville (Argentine frigate) 395 Grechko, Andrei 245 Gredjeva, Nina Michailovna 189 Greece 163, 259, 319, 320, 324, 334, 450, 472 Green Light (US special atomic demolition munitions programme) 249 Greenhill, Denis 239, 284 Greenock naval base (Scotland) 144 Greenpeace 446 Grey (US diplomatic code) 40 Grindley, Mike 430 Gromyko, Andrei 205, 206 Groupe de Synthèse et Prévision (France) 284 GRU (Soviet Military Intelligence) 88, 173, 230 Guardrail (US airborne tactical sigint systems) 251–2, 272 Guernica bombing (1937) 22 Guevara, (Ernesto) Che 300 Gulf War (1990–91) 452, 465–71, 529 Gulf War (2003) 479, 516–26 Gun, Katharine 521–3 Gurdon, Adam 396 Gurkhas 164–5 Gurney, Sir Henry 149 Habbaniya (Iraq) 20 Hagelin, Boris Jnr 212–13 Hagelin, Boris Snr 212 Hagelin (cypher machine) 56, 78 Haig, Alexander 403–4 Halifax, Lord 24 Hall, Theodore 73 Hallock, Richard 74–5 Hamilton, Alexander 432 Hampshire, Sir Stuart 221–5, 260 Handel, Michael 362 Hankey, Lord 36 Hanley, Michael 361, 587 Hanley, William J. 305, 306 Hanslope Park (Buckinghamshire) 57, 58, 182, 185, 186, 187, 192, 196, 418 Hanssen, Robert 444 Hardy, Tim 166 Harland & Wolff 261 Harman, Harriet 368 Harrier jets 403, 404, 407, 408, 441 Harris, Robert 59 Hart, Herbert 225 Harty, Russell 359 Harvest (computer) 350 Hashmi, Jabron 535 Hastings, Edward 95 Hawaii 92 Hawkes, John 216 Hay, Malcolm 15 Hayden, Michael 508 Hayter, William 97 Healey, Denis 168, 245, 256, 399, 426, 429, 433 Heath, Edward 4, 239, 279, 315, 337, 338, 439 Heliopolis (Egypt) 92, 155, 162 Hellenbroich, Heribert 452–3 Hellman, Martin 490 Helmand province (Afghanistan) 534, 535 Helms, Richard 292, 356 Hemblys-Scales, Roger 86 Henderson, Nicholas 2–3 Hennessy, Peter 90, 577 Herman, Michael 261, 287, 419, 421, 435 Heseltine, Michael 426 Hibberson, Anthony 189 Hill, Jim 86–7 Hill, Major 120 Hillenkoeter, Roscoe 85 Hillgruber, Andreas 59 Hinsley, Harry 59, 64 Hiroshima 2 Hiss, Alger 88 Hitler, Adolf 3, 5, 29, 30, 31, 32–3, 48, 221, 290 Hoad, Norman 132 Hokkaido (Japan) 112 Holden Agreement (1942 & 1944) 43 Hollis, Sir Roger 79, 86, 182, 367 Holmberg, Elena 389–90 Home Office 507, 538, 544 Honest John (battlefield missile system) 249 Honeywell (computers) 458 Hong Kong 19, 30, 32, 96, 100, 151–5, 219, 256, 277, 419, 475–8 Hooper, Joe 191, 222, 223, 228, 273–4, 285–6, 343, 353, 419, 448, 466, 585 Hoover Commission 219 Hosenball, Mark 358 House of Commons Public Accounts Committee 440, 459 House of Commons Select Committee on Employment (1985) 433–4 Howard, Edward Lee 384, 444, 447 Howard, Michael 364 Howarth, Jack 189 Howe, Geoffrey 426, 427–8, 429, 431–2, 436, 460 Howse, Philip 79, 84 Hughes, Chief Inspector 187 Hughes, Robert D. 111 Hungary 46, 158 Hunt, Sir John 329–30, 337–8, 356–7, 361 Hunters Stones Post Office Tower 346 Hurd, Douglas 495 Hurley, Michael 144–6 Hurn, Roger 495, 526 Hussein, King 161, 164, 290 Hussein, Saddam 467, 516, 524, 525 Husum-Milstedt (intercept station, Germany) 50 Hutton, Lord 529 Huxley, Aldous 549 HVA 605 Iacobescu, Ion 253 IBM 350, 352, 489 Igloo White (ground sensors) 252 Imre, Nahit 254 Incirlik (Turkey) 326 India 4, 18, 19, 30, 32, 85, 95, 178, 334 Indonesia 153, 164–8 Information Research Department 156 Ingebrigsten, Jan 450 Ingham, Bernard 428 Inman, Bobby Ray 399, 422, 601 Intelligence Assault Units 47–8 Intelligence and Security Committee 484–5, 529, 539 Intelligence Services Act (1994) 484–5, 488 Intelligence Support Activity 168 Intelsat 342 Intercept Control Centre 250 Intercept Modernisation Programme (IMP) 543–5, 547–8 International Control Commission on Vietnam 178 International Regulations on Sigint (IRSIG) 90 International Security Assistance Force (ISAF) 533 internet 8, 488, 507–8, 541, 544–5 IRA 345, 455, 479, 481–2, 494, 498–503, 593 Iran 109, 112, 155, 268, 295, 299, 302, 421, 467, 472, 605 Iran, Shah of 299 Iraq 155, 156, 160–2, 259, 295, 320, 465–71, 479, 516–17, 528–31; 124 Electronic Warfare Regiment 525 IRSIG (Instructions and Regulations concerning the Security of Signals Intelligence) 503–4 Iscot (wartime Comintern traffic) 37–8 ISI (Pakistani intelligence service) 514 Ismailia (Egypt) 32, 185 Ismay, General Hastings ‘Pug’ 27 Israel 97, 164, 263–4, 277, 290–1, 293, 307–8, 415, 471; Israeli Sea Corps 264–5 Istanbul 307, 309, 310, 316, 318 Italy 19, 44, 52, 96, 345, 452; Italian Cryptographic Bureau 54–5 ITT (telecoms company) 341, 342 Ivy Bells (undersea cable-tapping) 384 Jakarta (Indonesia) 167, 168 Japan 17, 29, 39, 40, 44, 65, 100, 110, 152, 445, 446 Jebb, Gladwyn 64 Jenkins, Roy 51 Johnson, Lyndon B. 50, 238, 353 Johnson, Robert 346 Johnstone, Sir Charles 164 Johnstone, Colonel Hugh 327–8, 359, 360 Joint Intelligence Committee (JIC) 67; and Arab states 109; Chevaline project 440; circulation of BJs 70; collection of elint on Soviet air-defence capabilities 132; considers Soviet threats a bluff 204; Crabb incident 141–2; D-Notice affair 239; deployment of equipment in Eastern Bloc 123; failures and inaccuracies of 108, 245–6, 387–8; focus on economic, technological and scientific subjects 241; French cooperation 284–5; intelligence failures 387–8; and invasion of the Falklands 391, 395–7; and Iraq 466; Joint Intelligence Committee Far East 167; and new technology 353; and Palestine 97; and possible Soviet move inside Eastern Europe 256; rethinking of European targets 345; review of aerial and submarine surveillance 207; review of GCHQ spending 221; and Soviet invasion of Czechoslovakia 245–6; and Soviet Union 46; and surprise Soviet nuclear attacks 321; and Turkish invasion of Cyprus 319, 325; and Vienna tunnel 171; and Yemen Civil War 164 Joint Technical Services Language School (Tangmere, Surrey) 370 Jones, Eric 46, 121, 132, 142, 159, 188, 191, 197, 418, 585 Jones, Colonel H. 411 Jones, R.V. 102–3, 110, 111, 579 Jordan 157, 164, 308 Joseph, Keith 286 Jowell, Tessa 498 July Bomb Plot (1944) 221 Kabul (Afghanistan) 533 Kagnew (Ethiopia) 335–6 Kaiser, Michael 240–1 Kapustin Yar (Soviet Union) 112, 131, 301 Karadſić, Radovan 473 Karamursel (Turkey) 301 Karlshorst (Germany) 371 Katanga (Congo) 336 Kazakhstan 107 Keepnet (recording equipment) 458 Keith, Bruce 69, 93, 94 Kelly, Gerry 500 Kennan, George 177 Kennedy, Jacqueline 180 Kennedy, Paul 59 Kenya 125, 334, 370, 511 Ker, Leonard Douglas 189 Kern, Dick 449–50 Kerr, Sir Archibald Clark 84 Key Recovery (or Key Escrow) 492, 506–7 KGB (Russian secret service) 137, 230, 285, 419, 538; agents working for 36–7, 185–90, 224–5, 231–7, 354–5, 369–85; and Airborne Rafter programme 267; defections from 478–9; Eight Directorate 377; expulsion from London embassy 283–4; intercepts on 53, 96; microwave intercepts 281; and miners’ strike (1982) 368; and release of material on GCHQ 355; Sixteenth Directorate 377; surveillance operations 4, 183–5, 190–1; tapping and bugging operations 170, 173, 175–82, 193; and Tempest 216–17; and Venona Project 72–88, 98, 104 Khalid Sheikh Mohammed 514, 515 Khrushchev, Nikita 4, 140, 142, 173, 179, 180, 202, 204 kidnapping and hostage-taking 452, 513–14; in Turkey 302–19; see also terrorists, terrorism Kiev (Soviet Union) 126 Killian, James R. 219 King, Tom 426, 427 Kingsdown (Kent) 34 Kinnock, Neil 433 Kipling, Rudyard 13–14 Kirknewton airbase (Scotland) 118 Kirkpatrick, Sir Ivone 130 Kissinger, Henry 277–81, 283–4, 287–90, 292, 293, 294–7, 319, 324, 326, 329, 330, 331, 337–8, 403, 441, 444 Kizildere (Turkey) 312–19 Klemme, General 50 Klugman, James 36, 188 Knockholt (Kent) 120 Knox, Dilly 21 Kohl, Helmut 453 Kola Peninsula (Soviet Union) 118, 136 Komer, Robert 303 Korea 6 Korean War (1950–53) 99–101, 116, 118–19, 120, 129, 152, 178 Kosovo 8, 512 Kosygin, Alexei 280 Koza, Frank 517–18, 521 Kuala Lumpur (Malaysia) 150–1 Kubat, Ferit 313, 314 Kuching (Malaysia) 166 Kurchatov, Igor 107 Kurku, Ertugrul 313–14, 315 Kursk (Soviet Union) 36–7 Kuwait 465–70, 524 Kuznetsov, Pavel 183–8 La Belle discothèque (West Berlin) 457 Labuan (Malaysia) 166 Lagos, Ricardo 519 Lamphere, Robert 76–7 Lander, Stephen 494 Lange, David 444–5 Langley (Virginia) 292, 441 Laos 446 Larnaca (Cyprus) 384 Latakia (Syria) 331 Law, John 312–16, 318 Lawson, Nigel 460 Le Bailly, Louis 246, 286–7, 291, 439 Leach, Henry 395 Lebanon 161, 308 Lee, Raymond 39, 40–1 Libya 109, 295, 334, 455–8, 531; Libyan People’s Bureau (London) 455–6 Liddell, Guy 367 Light Electronic Warfare Teams (LEWTs) 534–5 Limassol (Cyprus) 325 Lindsay, Michael 151 Little, Peter 420–1 Little, Rod 402 Little Sai Wan (Hong Kong) 153, 475 Livebait (comparison of different signals) 458 Llanos, Gonzales 408 Lobban, Ian 542 Lockerbie incident (1988) 457–8, 605 Lockhart, John Bruce 171 Lockheed 476 Loehnis, Clive 31, 197, 210–11, 223, 466 Lohan, Sammy 239 Lombardo, Juan 393, 395 London bombings (2005) 532–6 London Communications Security Agency (LCSA) 103, 191–3, 210, 211, 213, 217, 585 London Communications Security Board 585 London Processing Group (LPG) 372–3 London Signals Intelligence Board 51–2, 109, 142; London Signals Intelligence Centre 69; London Signals Intelligence Committee 267, 268 Longfellow (Soviet cypher system) 78 Lonsdale, Gordon 238 Lord (Vienna tunnel code) 171 Luanda (Angola) 59, 455 Lucas, George 144, 145 Luftwaffe 26, 33–4, 35, 36, 43, 50 Luga airport (Malta) 295 Luneburg Heath (Germany) 127 Lunn, Peter 171, 172, 174 Lyalin, Oleg 283 Lyttelton, Oliver 149 MacArthur, General Douglas 45 McCormack, Alfred 45 Macdonald, Ken 543, 548 McGuinness, Martin 500 Machon, Annie 456 Mackay of Clashfern, Lord 484–5 Mackenzie King, William Lyon 94 Maclean, Donald 8, 37, 72, 73, 76, 82, 84, 87, 104, 238, 367 McManners, Hugh 409, 410 Macmillan, Harold 3–4, 143, 179, 204–7, 226, 364 McNamara, Robert 168 McNeill, Hector 224 Magdeburg (Germany) 50 Magic (Japanese cypher) 29, 39, 41, 44, 69 Maguire, Harold 268, 269 Major, John 398, 484 Makarios, Archbishop Mihail 163, 296, 320, 323–5, 328, 330 Malatya (Turkey) 306 Malaya 6, 30, 38, 125; Malayan Communist Party (MCP) 149, 150; Malayan Emergency 149–51 Malaysia see Federation of Malaysia Malinovsky, Rodion 204 Malta 156, 162, 295 Maltby, Ted 58, 79, 187 Manchester University 349 Manchuria 19 Mandelson, Peter 498 Manhattan Project (Los Alamos atomic bomb project) 75, 76, 82, 219 Manningham-Buller, Dame Eliza 515 Mao Tse-tung 4, 151, 193, 195 Marchetti, Victor 356 Marconi (company) 311 Marconi, Guglielmo 13 Marenches, Alexandre de 442 Marr-Johnson, Patrick 77 Marshall, George 42 Marshall, William 184–90 Martin, William H. 176, 355, 423 Marychurch, Peter 428, 434, 445, 448, 449, 451, 458–60 Mask operation 19 Mason, Roy 332 Mathison, Alan 25 Mauborgne, Joseph 18 Mauritius 334, 335, 338 Mazzini, Giuseppe 14 Medical Research Council 434 Mediterranean 16, 44, 114, 273, 295, 319 Meir, Golda 290–1 Menendez, Mario 412–13 Mentyukov, Igor 201–2 Menwith Hill (Yorkshire) 345–6, 347, 421, 449 Menzies, Sir Stewart 23–4, 26, 27, 28, 38, 39, 42, 45, 51–2, 55, 67, 82, 142 Methods to Improve (MTI) 220 Mexico 15, 517, 519–20 Meyer, Cord 358, 595 Meyer, John C. 271, 272 MI5 see Security Service MI6 see Secret Intelligence Service Middle East 7, 15, 16, 19–20, 32, 34, 41, 51, 97, 148, 155–64, 181, 182, 271, 277, 282, 290–5, 299, 320, 333, 334, 336, 376, 385, 454, 467, 472, 479, 494; Middle East Technical University (Ankara) 303, 304; Middle East War (1973) see Yom Kippur War Middle Six countries 519 Millward, Bill 63, 121, 221–2, 253 Milne, Alasdair 459 Milner, Ian 86 Milner-Barry, Stuart 25, 27, 364, 387 Milošević, Slobodan 473 Ministry of Defence (MoD) 240, 272, 286–7, 291, 312, 368, 423, 456, 495, 510 missiles see rockets and missiles MIT (Turkish National Intelligence Agency) 304, 314 Mitchell, Bernon F. 176, 355, 423 Mitchell, Graham 367 Mitchell, J.R. 118 Mitterrand, François 414–15 Mladić, Ratko 473, 474 mobile phones 492–3, 505, 538, 541, 548 Modin, Yuri 82–3, 84 Moffit, Bill 296–7 Molotov, Vyacheslav 178–9 Monterey (California) 332 Montgomery, Field Marshal Bernard Law 58, 85 Moon-bounce project 262–3 Morgan, Gerry 78 Moriarty, D.M. 587 Morocco 334 Morris, Gareth 433 Morwenstow (Cornwall) 342 Moscow 4, 18, 19, 30, 33, 35, 36, 37, 45, 47, 53, 59, 74, 82, 84, 86, 104, 126, 148, 151, 175, 176–8, 179, 185, 186, 189, 201, 203, 205, 284, 465; Moscow Peace Treaty (1942) 32 Mossad (Israeli secret service) 291, 299–300, 307–8, 444, 472 Mottram, Richard 577 Mount Tumbledown (Falklands) 413 Mountbatten, Lord Louis 110, 143, 192 Mowlam, Mo 500 Mubarak, Hosni 467 Mullah Dadullah 535 Muller, Wilma 127 Mullet Creek (Falklands) 397 Munich Crisis (1938) 3, 22 Murmansk (Soviet Union) 133, 143 Murray, Len 416–17, 426, 427, 428 Muslims 472, 473, 474, 537, 538, 542 Mustard (Enigma key) 35 Mutual Balanced Force Reduction programme 287–9, 319 Mutual Weapons Development Programme 211 Nagasaki (Japan) 2 Nan-Szu-Pu (Taiwan) 152 Narvik raid (1940) 55 Narwal (Argentine trawler) 407–8 Nasser, Gamal Abdel 155–6, 161, 164 National Central Electronic Reconnaissance Agency (NSEI, Croatia) 473 National Council for Civil Liberties 361 National Criminal Intelligence Service 504 National Infrastructure Security Coordination Centre 609 National Security Agency (NSA) 7; and al Qaeda 510–12; and Berlin tunnel operation 174; and BND 422–3, 438, 447–51, 452–4, 455; and Bosnian-Croatian conflict 472–5; ceases intelligence exchange with GCHQ 289–90; and commercial encryption 488, 489; cooperation with GCHQ 222–3, 278, 282–3, 346, 347–54, 438, 441–3, 448–58, 461; creation of 101–2; data silo in Utah 546; development of new systems 345–6; development of Technical Research Ships 260; and documents on Princess Diana 483; and downing of Powers’ U-2 202–3; elint and comint responsibilities 122–3; funding of 334, 346; influence on foreign policy 321; and internet 8, 508; and invasion of the Falklands 398, 399, 415; and North Korea 100; and polygraph 434; and Prime case 376; public mention of 242–3, 355, 358, 361–2; relationship with US armed services 271–2; and Russian nuclear forces 119; Russian spies in 384–5, 444; and sale of cypher machines 209–15; and strategic elint 267; and Suez crisis 157, 158; and Tempest 216–18; and trade unions at GCHQ 421–2, 424, 429; Turkish operations 300–19; use of security measures 381; visibility of 341; and Yom Kippur War 291–2 National Service 117, 153, 219, 229, 369 NATO 126, 130, 209–15, 217–18, 242, 247, 248, 253–7, 268–9, 270, 272, 283, 300, 319, 328, 345, 429, 448, 452, 456, 467, 533, 534; Military Committee 256; Nuclear Planning Group 332; Special Committee 257 Nauticus Corporation 265 Naval Intelligence 32, 116, 143 Naval Mechanics School (Buenos Aires) 389 Nave, Eric 19 Nazi-Soviet Pact (1939) 31 Neff, Paul 50, 76 Netherlands 442, 443 ‘Never Again’ agreement (1982) 440–1 New People’s Army (Philippines) 452 New Zealand 89, 90, 92, 93, 98, 164, 165, 438, 442, 444–7, 448, 487; Government Communications Security Bureau (GCSB) 445–6 Newman, Dr 145 Newman, Max 28, 70 Nicoll, Douglas 387–8, 396, 421, 529 Nicoll Report (1981) 388 Nicosia (Cyprus) 326 Nimrods 268–70, 271–4, 326, 414, 415, 442, 474, 515, 536, 537 9/11 9, 509–14, 531 Nixon, Richard M. 4, 277–8, 279, 281, 283–4, 288, 290, 293, 297–8, 304–5, 325–6, 337, 338, 434 Noakes, John 348 Noise-Induced Hearing Loss 608 Noise Investigation Bureau 110 Nokia (telecoms company) 489 Norland, Selmer 48, 78 North Atlantic Council 254 North Cape, Battle of (1943) 60 North Korea 100, 120, 129, 175, 264 Northern Ireland 329, 498–503; Peace Process 501 Northwood Hills (communications security establishment) 192, 400, 401, 402, 469 Norway 55, 99, 116–17, 134, 139, 269 Nott, John 395, 396, 397–8, 414–15, 429, 601 Noye, Kenneth 505–6 Nuclear Planning Group 255 Oakleigh Park North (Soviet-radio monitoring station, Whetstone) 190–1 O’Connor, Morris J. 144, 145 Odette (Army intercept equipment) 474, 525, 534, 536–7 Odom, William ‘Bill’ 214, 385, 413, 434, 442–4, 446–52, 454–5, 457, 458, 471, 476 Oedipus (computer) 349 Oeljeschaeger, Major 50 Office of Strategic Services 87, 91 Official Secrets Act 8, 359, 360, 363, 383, 522 oil 298, 336, 356 Okinawa (Japan) 152 OKK-5 (Soviet codebook) 35 Oldfield, Maurice 82, 358 Omagh (Northern Ireland) 501–3 Oman 271, 345 Omand, Sir David 9, 272, 398, 495–7, 498 one-time pads (encryption system) 18–19, 20, 56, 74, 81, 83, 108 Operation Citadel (1943) 36 Operation Claret (1956) 140–2 Operation Damage (Comet sorties in the Mediterranean) 273 Operation Debenture (1954) 152–3 Operation Defiant (1955) 137 Operation Desert Storm (1991) 467–9 Operation Duster (sigint flight operations during Yom Kippur War) 295 Operation Gold (1948) 97 Operation Halfmoon (1948) 96 Operation Hem (sigint flight operations during Yom Kippur War) 295 Operation Musketeer (1956) 156–9 Operation Nigeria (against journalists and their sources) 506 Operation Overlord (1944) 59 Operation Pat (Comet sorties over the Baltic) 273 Operation Sanjak (1955) 137–9 Operation Storm (1995) 472–3 Operation Tartan (1955) 136–7 Operation Trail Hammer 536 Orford Ness (Suffolk) 285–7, 322 Organ, Helena see Prime, Helena Orion (sigint satellite) 437 Ormsby Gore, David 206 Orwell, George 549 Oshima, Baron 29 Ottawa (Canada) 57, 85, 92, 94, 97 Over the Horizon Radar 285–7, 322 Owen, Dr David 299–300, 332–3, 360, 391 Padeborn (Germany) 248 Pakistan 323, 334, 384, 513–14, 519, 537 Palestine 97, 109, 155, 156, 320 Paris 21, 25, 52, 53, 158, 194–5, 243, 284–5, 510 Parker-Bowles, Camilla 480, 482 Parliamentary Select Committee on Foreign Affairs 529 Patagonia 393 Patchett, Brian 228–30, 369 Paterson, Brian 507 Pearl, Daniel 514 Pearl Harbor attack (7 December 1941) 29, 290 Peking 150, 151, 194 Pelton, Ronald 384–5, 443–4, 447 Penkovsky, Oleg 322–3 Penney, William 192 Pepper, David 526, 527, 528, 532, 539 Perkar (Ceylon) 160 Perkins, Alice 495 Permanent Secretaries Committee on the Intelligence Services (PSIS) 219–20, 241, 260, 423 Perrin, Ken 267, 270, 363 Peshawar (Pakistan) 384 Petersfield (Hampshire) 133 PGP (Pretty Good Privacy, code-making programme) 490–1 Philby, Kim 37, 82–3, 84–5, 225, 226, 238, 242, 354–5, 367, 385 Philco (telecoms company) 350 Philippines 445 Phillips, Cecil 75, 79 Pilsey Island (Sussex) 140 Pincher, Chapman 17, 226, 238, 239, 240, 242, 386 Pine Gap (Australia) 345 Pinner (Middlesex) 68, 69 Pinochet, Augusto 519 Pirinclik Air Base (Turkey) 301–2, 306 Pither, Judith 383 Plessey (telecoms company) 212, 267–8, 311, 433 PLO (Palestine Liberation Organisation) 277 Poets Systems (Soviet cypher machines) 78, 81 Poland 21–2, 31, 46, 178, 387, 421, 515 Polaris missile system 266, 322, 335, 337, 438–40 Pollard, Jonathan 444 Polyarnoe (Soviet Union) 34, 36 polygraphs 381, 383, 424–6, 433–4, 444 Port Said (Egypt) 156 Port Stanley (Falklands) 390, 397, 413 Portsmouth (Hampshire) 134 Portugal 44 Poseidon (missile system) 439 positive vetting 88, 227–8, 229, 229–30 Posner, Gerald 483 Post Hostilities Planning (PHP) Committee 46, 85 Post Office 241, 286; develops fibre-optic cables 604; Research Department (Dollis Hill) 28, 68, 171, 172, 349; ‘Secret Department’ 14 Potts, Archie 154–5 Poulden, Teddy 93, 94, 352–3, 354 Powell, Jonathan 500 Powers, Gary 8, 201–7, 208, 226 Prague 244; Prague Spring (1968) 387 Prime, Geoffrey 8, 368–86, 423, 424, 425, 444, 447, 600 Prime, Helena 374–5 Prime, Rhona 375, 378–9 Princeton University 350 Prior, James 422 Profumo affair (1963) 8, 226, 228 Project Clipeus (British ADM programme) 249 Project Cobra Shoe (US intelligence station on Cyprus) 323, 348, 356 Project K (NSA HQ) 102 Project Minaret (US illegal monitoring of domestic radicals) 357 Project Sambo (tracking low-frequency submarine radio transmissions) 378 Project Sandra (intelligence facility on Cyprus) 321–3, 348, 356 Public Interest Immunity certificates 505 Public Key Cryptography 489–93, 508, 512 Public Record Office (Kew) 355 Puerto Belgrano (Argentina) 393 Punta Arenas (Chile) 414 Purple (Japanese cypher machine) 29, 38 Purves, Peter 348 Pym, Francis 423 Pyramider satellite 377 Quinlan, Sir Michael 493 Racal (telecoms company) 401, 433, 469, 524, 590 radar 110, 126, 132, 133, 136, 138, 145, 154, 202, 266, 285–7, 301–2, 306, 406, 408, 466 Radcliffe, Lord 239–40 Radcliffe Inquiry into Security in the Civil Service (1962) 381, 418 Radio Corporation of Cuba 341 Radio Operators (GCHQ) 15, 185, 186, 228, 261, 382, 418, 419, 420, 422, 432, 435, 458 Radio Reconnaissance Teams (Afghanistan) 535 Radio Security Service 37, 58, 79, 221 Radio Warfare Special Branch 133, 134 Rainbow Warrior (Greenpeace ship) 446 Rangoon (Burma) 110 Rattan (Soviet radio intercepts) 75 RC-135 ‘Looking Glass’ aircraft 273 Reagan, Ronald 398, 403, 457 ‘Real IRA’ 501, 502 Reenan, Cyril 479, 480 Rees-Mogg, Lord William 481 Reeve, James 196, 197 Regulation of Investigatory Powers Act (RIPA) (2000) 547 Reid, John 516 Reijn, Joop van der 308 Reilly, Patrick 205, 206 rendition programmes 539–40 Rendle, John 578 Rennie, John 156 Res (reserved cypher material) 44–5 Review of Intelligence Requirements and Resources (1994) 493 Review of Intercept as Evidence (2004) 541 Rhodes, Miriam 378 Rhodesia 3, 209 Ribbentrop, Joachim von 31 Richards, Brooks 387 Richards, Francis 504, 511, 526 Riddington, Tony 598 Ridley, Nicholas 392 Rimington, Stella 372, 494, 521 Ring of Five 442–3 Ritter, Scott 470–1 Riyadh (Saudi Arabia) 513 Roake, Alfie 147, 265 Robinson (computer) 349 Robinson, Robin 493 rockets and missiles 5, 108, 110, 131, 169, 201–2, 203, 207, 266, 271, 301, 306, 315, 322–3, 335, 337, 376, 390, 439–40, 510, 511, 515, 525 Rockex (UK cypher machines) 57–8, 194 Rolf, Vic 213 Romania 52, 256, 257, 284; foreign intelligence service (DGIE) 253–4 Rome 372 Rommel, Erwin 5 Roosevelt, Franklin D. 39, 41, 47, 73, 84, 87 Rose, Michael 472, 473, 474 Rosenberg, Ethel and Julius 83 Rosenheim (Germany) 48 Roussilhe, François 254–5 Rowe, Vivian 402 Rowlands, Ted 390, 399–400, 401 Rowlett, Frank 44, 174 Rowntree Foundation 361 Royal Air Force (RAF) 369, 537: 192 Squadron (monitoring aircraft unit) 113, 122, 125, 149, 159; 199 Squadron (radio wafare unit) 113, 125, 131; invited onto Blue Peter children’s programme 348; loss of Avro Lincoln (1953) 125–9, 580; and Malayan Emergency 149–50; near-miss incidents 131; negotiations on air corridors 129–30; 100 Countermeasure Group 112; as part of GC&CS 20; RAF Akrotiri 273, 295, 296, 323, 325; RAF Brampton 469; RAF Brawdy 377; RAF Brize Norton 204, 375; RAF Celle 128; RAF Crail 369; RAF Digby 230, 231, 237; RAF Gatow 112, 370, 383; RAF Habbaniya 155, 161–2, 230; RAF Hammersley Hayes 231; RAF Lakenheath 116, 142; RAF Leconfield 126; RAF Mildenhall 257, 273, 284, 297; RAF North Luffenham 153; RAF Northolt 537; RAF Oakhanger 262; RAF Pergamos 162, 323; RAF Scharfoldendorf 128; RAF Sharjah 271; RAF Strike Command 270; RAF Upper Heyford 457; RAF Watton 112–14, 122, 125, 131, 159; RAF Wythall 153; RAF Wyton 273, 295; ‘Rock Apes’ Regiment 154; sigint and elint operations 110–14, 116–18, 121–2, 124, 131–3, 153, 154, 206, 218–19, 269–70, 272–3, 285–7; Signals Units 153, 154, 161; surveillance operations 537–8; and U-2 overflight programme 202; Y stations in Kent and Cheshire 34, 63, 111 Royal Commission on Criminal Justice Procedure 425–6 Royal Electrical and Mechanical Engineers 499 Royal Marines 390, 391, 394, 402, 411, 413, 524, 535; Y Troop 524, 525, 535 Royal Navy 159, 441, 474, 477; and Buster Crabb incident 140–2; HMS Affray 133; HMS Albion 259; HMS Anderson (sigint station in Ceylon) 69, 92, 93, 94, 160; HMS Antrim 409, 410; HMS Ardent 410; HMS Conqueror 404; HMS Coventry 406; HMS Dolphin 265; HMS Endurance 391–2, 394, 396; HMS Glamorgan 410; HMS Glasgow 406; HMS Hermes 402–3, 411; HMS Invincible 407; HMS Maidstone 139; HMS Mariner 134; HMS Mercury (RN Signals School) 133, 134, 139; HMS Pickle 134; HMS Pucklechurch 133, 134; HMS Sheffield 406; HMS Sir Galahad 400; HMS Superb 259; HMS Taciturn 144–6; HMS Totem (later Dakar) 137, 264, 265; HMS Truculent 133; HMS Truelove 134; HMS Truncheon 264, 265; HMS Turpin 135–9, 140, 143, 146–7, 264; influence on GC&CS 16; intercept sites at Scarborough and Winchester 63; Kipling story involving 13; Provost Branch 139; and radio station at Polyarnoe 34; ‘Room 40’ code-breakers 15; sigint and elint operations 6, 114, 133–9, 143–7, 218–19, 264–7 Royal Radar Establishment (Malvern) 501 Royal Ulster Constabulary (RUC) 501, 502–3; RUC Special Branch 499, 502, 503 Rumsfeld, Donald 511 Rushworth, Edward 48, 78 Russian Mafia 504 Ryolite satellites 345, 346, 376, 421 Sabah (Borneo) 165 Sabri, Naji 530 Sadat, Anwar 296 Sadi, Sener 314 Safford, Laurance 39 Samford, General 101 Samsun (Turkey) 301, 311 San Carlos Water (Falklands) 408–11 San Francisco 92 Sangin province (Afghanistan) 535 Santa Fe (Argentine submarine) 397 Sarafand (Palestine) 20, 32, 155, 162 Sarajevo (Bosnia and Herzegovina) 473, 474 Sarawak 165 Sarell, Sir Roderick 308, 312, 313, 315, 319 Saudi Arabia 164, 467, 468 Saunders, Andrew 458 Savimbi, Joseph 454 Scan Odd (airborne radar) 121 Scarborough (Yorkshire) 63 Scargill, Arthur 416 Scarman, Lord 431 Scarus (portable interception kit) 534 Schlesinger, James 294–5, 326, 329, 330, 332 Schmidt, Hans 21 Schröder, Gerhard 520 Schulze, Reinhard and Sonja 455–6 Scott-Farnie, G.R. 34 SCUD missiles 251, 468, 469 SDECE (French secret intelligence service) 442 Second World War 135, 335; end of 47–50, 59–60; events leading up to 22; and improved communications 5; and release of sigint material 355–6; Russian interceptions in 30, 31–2; sigint and code-breaking in 2, 5, 28–9, 32–46, 250 Secret Intelligence Service (SIS; MI6) 1, 57, 82, 91, 97; agents sent into the Eastern Bloc 123; at Bletchley 23, 27; and Automatic Data Processing 353; and Berlin-Vienna tunnel operations 169–76; and bungled surveillance operations 140–2; buys missiles on the open market 407; closer relationship with MI5 and GCHQ 503–4; considered organisational basket-case 24; and Cyprus radio station 156; and end of influence over sigint 142; and Enigma decypherment 21; funding of 494; move to Broadway Buildings 17; move to Century House (1966) 195; move to Vauxhall Cross 496–7; no trade unions in 417; obsession with Russia 17, 31; origins of 14; and Penkovsky 322–3; and post-war re-absorption of GC&CS 67; recruitment to 61; Section V 37; Section VIII 57–9; Section Y (Carlton Gardens) 171, 178, 372; Technical Collection Service 132; and Yemen 164 Secret Service Committee 15 Security Commission 368–9, 381–2, 384 Security Service (MI5) 1, 57, 182, 361; and Automatic Data Processing 353; Britten affair 237; bugging operations 177; closer relationship with SIS and GCHQ 503–4; and CND 368; fears concerning Communists in trade unions 418–19; funding of 494; interception of telegrams and telexes 239; and John Cairncross 37; and Libyan Embassy affair 456; no trade unions in 417; origins of 14–17; and Peter Wright 223, 224; and polygraph 433–4; and possible sub-agents 382; problems with Orford Ness 287; and sigint security 70–1; ‘Squidgygate’ affair 482; and Venona 79, 84, 85, 86; ‘watchers’ operations 183–90; and West German defections 455–6; Whetstone radio monitoring station 190–1 Selby-Bennett, Harry 134, 139 Selwyn Lloyd, John 128, 158, 159 Semipalatinsk (Soviet Union) 107, 302 Services Liaison Department 70 Sexton, Jamie 305 Seychelles 334, 335 Shanghai 19 Sharq el-Adna (radio station) 156 Shayler, David 521 Shedden, Sir Frederick 86, 92 Sheldon, Robert 459 Shergold, Harry ‘Shergy’ 142 Short, Clare 523–4 Sidewinder air-to-air missiles 441 Sigaba (US cypher machine) 98–9 Sigdasys project (improved flow of sigint to front-line units) 451, 456 Sigint Conference (1946) 95 Sigint Electronic Warfare Operation Centre (Afghanistan) 535 sigint satellites 437–8 sigint ships 259–67 Sigint/EW Operations Centre in Regional Command (South) 534 SIGMod 539 signals intelligence (sigint); in Afghanistan 534–9; airborne 121–2, 125–33, 144, 202–7, 223, 251–2, 257, 267–74, 295, 377; and allied cut-offs 444–7; American 39–40, 143–4, 271–2, 273–4; benefits of 401–2; blue jackets (BJs) 17, 70–1; bugging 176–82; cost of 218–23; Croatian success 473–5; and Cuba 341–2; cypher security 57, 191–3; in Cyprus 320–39; D-Notice affair 238–41; defections from 228–9; dependence on SAS-type activities 250–1; and ‘Dodgies’ or Mystery Trips 6; and domestic/ international blurring 344; during Cold War 1, 5, 8, 108–24, 125–47, 257–8; during WWII 2, 5, 32–46, 57–8; expansion in India 30; failures of 253–5; and Far East 148–51, 164–8; GCHQ at heart of 5–10; German 35, 49–50, 62; global alliances 89–101; ground-based 117–20, 252–3; and Gulf War (1991) 468; in Hong Kong 151–5; importance of bases 151–5, 277–8; and Indonesia-Malaya confrontation 164–8; and invasion of Czechoslovakia 245–6; and invasion of Falklands 397–415; and Iraq War (2003) 525–6; and Kipling 13–14; legalities of 344; and making/influencing of foreign policy 321; in medieval times 4; and Middle East 155–64; modern formation of 58; and political leaders 2–4, 7; and private companies 17; problem of language 512–13; public disclosure of 356–64; rethinking of targets 345; relations between Western allies 444–7; release of wartime material on 355; and rescuing of enemy matériel 47–56; risks 203; and Russians 33–8, 280–1; seaborne 114–16, 133–47, 208, 259–67, 377–8; security disasters 228–38; size of 227–8; and speed of communication 4–5; support for front-line units 449; tactical units 250–2; and Third World 169; in Turkey 300–19, 330–1; value and importance of 60, 62–71 Signals Intelligence Centre 93 Signals Intelligence Service (USA) 74 Sillitoe, Sir Percy 86, 190 Silvey, Reg 412 Simakov, Alexander 478–9 Simkin, Anthony 187 Sinclair, Hugh ‘Quex’ 16, 22, 23 Sinclair, Sir John 142 Sinews (Sigint NEW Systems) 496 Singapore 19, 40, 96, 164, 166, 167 Singleton, Valerie 348 Sinkov Mission (1941) 39 Sinn Féin 500 Sinop (Turkey) 301 Six-Day War (1967) 253, 263–4, 271, 284 Skardon, William 87, 188 Skynet (communications satellite) 347–8, 403, 408, 438 Slessor, John 190–1 Slim, Field Marshal William 65 Sly, Ken 153–4, 375 Smallwood, Sir Denis 271 Smith, F.M. 302 Smith, Ian 3 Smith, Jack 151 Smith, Jacqui 543 Smith, Rupert 474–5 Smiths Industries 495 Snow, Leading Seaman ‘Snowy’ 135 social intelligence 178–9 Solidarity (Polish trade union) 432, 465 Somerville, John 121, 317–18, 419, 420, 428 Sony 480–1 Soothsayer (Army intercept equipment) 537 SOSUS (undersea microphones) 377 South Africa 209, 446, 454 South Georgia 390, 391, 393–6, 409 South-East Asia Command 110 Southern Thule 389, 390–1 Soviet Air Force 118, 119, 360; Air Defence Command 201; Strategic Air Command 256; Strategic Rocket Force 256 Soviet Army 175, 371; General Staff 255–6 Soviet Communist Party 19, 368 Soviet Navy 114–16, 133, 256, 301; Naval Intelligence 73; Northern Fleet 136 Soviet Union 33–8, 336, 349; apparent war preparations 255–7; and Berlin-Vienna tunnel operations 169–76; and biological weapons 611; capture of German cryptographic assets 50–1; Cold War espionage 8; elint on invasion of the Falklands 401; end of Cold War 493; enters WWII 28; ‘ferret’ programmes 112; Hitler’s invasion of 29, 32–3, 290; invasion of Czechoslovakia 244–7; Italian code-breaking concerning 52, 54; nuclear weapons 107–8, 112, 114, 116; and post-war confrontation 47; raid on Arcos building 18; Red Army 46, 78, 245–6, 249, 319; release of UK intelligence material 355; secret submarine missions against 6; and ‘strayed’ aircraft 126–33; successful sigint operations against 279–81, 344, 361; Turkish (Anglo-US) operations against 301–2; UK-US obsession with 17–19, 30–2, 45–6, 321–3; and Venona Project 72–88; and Yom Kippur War 291 Spain 44, 505, 533 Special Branch 141, 150, 151, 166, 187, 453, 456, 459, 509 Special Liaison Units 57–8 Special Radio Installation Flight (SRIF) 114 Spedding, David 494 spy planes 257, 267–74, 292–3, 296–7, 538 ‘Squidgygate’ affair (1990) 479–82 SR-71 Blackbird Mach 3 reconnaissance aircraft 284, 292–3, 296–7 Stalin, Joseph 31, 33, 46, 47, 173 Standard Cable & Wireless Ltd 17 Standard Telegraph & Cable Ltd 342 Stankovic, Milos 473 Stanmore (Middlesex) 62, 68 Stannard, Robert ‘Fred’ 210–11, 213, 218, 242 Starmer, Keir 543 State Research Association 361 stay-behind patrols 247–50 Stella Polaris (sale of Russian codebooks) 91 Stephanie operation (Canadian embassy intercepts in Moscow) 280–1 Stephenson, Sir Hugh 207, 221 Stevens, Geoffrey 40, 45, 50, 94 Stewart, Brian 353 Stewart, Michael 228 Stockholm 192 Straw, Jack 505, 519 Stripp, Alan 109 submarines 6, 114–17, 125, 133–9, 142–7, 197, 259, 264–7, 337, 377–8, 384, 397, 404 Suez Crisis (1956) 156–9, 160, 181, 182, 213 Sugar (Vienna tunnel code) 171 Sugar Grove (West Virginia) 262 Suharto, President 168 Sukarno, President 164, 165, 167–8 Sunay, Cevdet 311 Super Antelope programme (modernising/upgrading Polaris submarines) 337 Supreme Allied Commander Europe (SACEUR) 256 surveillance operations 4, 183–90, 197, 201–7, 208–9, 247–53, 292–3, 295, 296–7, 322, 368, 406, 421, 472, 480–1, 500, 506, 537–8, 540–50 SUSLO (Special United States Liaison Officer based in UK) 381 Sutton Common (Cheshire) 500 Sweden 31, 212, 214, 269 Switzerland 52, 212, 214, 215, 457, 492 Sykes, Richard 297, 298, 335, 593 Syria 156, 157, 271, 291, 300, 301–2, 304, 308, 336, 344 Tai Mo Shan (New Territories) 154 Taipei 195 Taiwan 152, 195, 323 Taliban 535, 537 Tanzania 511 Taper (Soviet cypher traffic) 54, 108 Tartus (Syria) 331 TASS News Agency 190 Taylor, Telford 43 Tebbit, Kevin 498, 504 Technical Committee of London Signals Intelligence Committee 267 Technical Radio Interception Committee 131 Tedder, Lord 5 Tel Aviv 157–8, 180 telephone tapping and intercepts 170–6, 180, 244–5, 299, 340, 341–6, 376, 377, 474–5, 479–83, 486, 499, 500–1, 523, 541–5 Tempest (radiation/emanation phenomenon) 209, 215–18 Templer, Sir Gerald 150, 219 terrorists, terrorism 9, 168, 277, 307, 320–1, 452, 531; and 9/11 509–15; domestic 539; and Heathrow plot (2003) 515–16; and IRA 498–503; Libyan 455–8; see also kidnapping and hostage-taking Teufelsberg (Germany) 478 Thatcher, Margaret 8; and Falklands conflict 298, 396, 400, 403–4; and mole-mania 363, 367; 1987 general election 433; obsession with secrecy 492; and polygraph 434; and removal of trade unions from GCHQ 415, 416–17, 423, 425, 426, 427, 428, 429, 430–1, 435; and tightening of the Official Secrets Act 363; and Zircon project 442, 460 Third World 203, 259, 334 Thistlethwaite, Dick 255 Thomas, Richard 544 Thomas, Teddy 234 Thompson, Julian 402, 410–12 Thompson, Ralph 94 Thompson-CSF (arms company) 489 Thomson, Mike 131 Thorneycroft, Peter 163 Thorpe, Peter 535 Tiananmen Square massacre (1989) 476 Tickell, Crispin 290 TICOM (Target Intelligence Committee) teams 48–56, 76, 78 Tiltman, John 19, 31–2, 42, 44, 67, 78, 79, 96, 213 Tirpitz (German battleship) 35 Titchner, Lambert 186 Tito, Josip 4 Tomlinson, Richard 521 Tonkin, Derek 330, 333 Tornado Multi-Role Combat Aircraft 345 Toumlin, George 432 Tovey, Brian 167, 414, 415, 421–2, 423, 424, 428, 433, 442, 448, 461, 490 trade unions 317, 368, 389, 416–36, 497–8 Trades Union Congress (TUC) 416, 417, 419, 426, 427, 509 Travis, Edward 27, 28, 36, 43, 48, 49, 53, 56, 60, 67–9, 69, 92, 94, 101, 121 Trawlerman (DIS computer scheme) 527–8 Trend, Burke 221, 240, 241, 242, 269–70, 288, 322–3, 354, 364 Trevor-Roper, Hugh 221 Tromsø 134 Truman, Harry S. 73, 85, 91, 101, 108, 109, 116 TRW Inc (telecoms company) 377 Tryst operation (British Embassy intercepts in Moscow) 280–1 ‘Tunny’ (German cypher machine) 28, 349 Turing, Alan 2, 25, 27, 349, 492 Turkey 52, 109, 131–2, 157, 268, 269, 299–319, 325–8, 330–1, 334, 338, 357, 423, 472; Air Force 312–13, 313; Army 315; Foreign Ministry 312–13 Turkish People’s Liberation Army (TPLA) 300, 303–19 Turkish People’s Liberation Front (TPLF) 300, 308–10, 312 Turnbull, Andrew 524 Turner, Charles 312–16, 318 TUSLOG (US Logistics Organisation in Turkey) 302 Tuxedo (British nuclear weapons stockpile in Cyprus) 163 25 Mayo (Argentine aircraft carrier) 404 Typex (UK cypher machine) 56, 98 Tyuratam (Soviet Union) 306 U-2 spy planes 142, 226, 292, 296, 332, 471 UK-USA Technical Conference (1946) 95 Ukraine 472, 533 UKUSA (UK-USA signals intelligence agreements) 7, 213, 241, 273, 376, 577; combined comsec/sigint agencies 242; and deterioration of Anglo-US relationship 285; development of 89–95; and elint 111; existence of 1948 aagreement 577; and GCHQ 95–9, 222; and Hong Kong 152; and Israel 471; and Korea 99–101; and satellite collection 437–8; second-party members 444, 447; standardisation of equipment in 424; third-party members 209, 447, 452; and tightening of security 381; value of British Empire to 149 Ultra (WWII decrypts) 1, 24, 26, 32–8, 41, 42, 43, 57, 59, 60, 62, 72–3, 113, 354, 362, 363 Underwater Development Establishment 145 Unit 8200 (Israel sigint agency) 470 United Nations 66, 295, 336, 445–6, 472, 523–4; Monitoring, Inspection and Verification Commission 520; Protection Force 472; Security Council 517, 522; Security Council Resolution 1373 511–12; Special Commission (UNSCOM) 470–1 United States; and Cold War espionage 8; Communications Intelligence Board 97, 152; cyphers worked on by GC&CS 17, 29; Department of Defense 295; Division of Scientific Intelligence 322; Europe Command 180; Foreign Intelligence Advisory Board 263; Information Service 305; National Photographic Intelligence Center 296; National Reconnaissance Office 449, 458; National Security Council 334; and proposed German-Mexican alliance 15; and UKUSA intelligence treaty 89 University of Pennsylvania 426 Unye (Turkey) 313, 316 US Air Force 96, 101, 129, 152; 47th Radio Squadron 118; Griffiss Air Force Base (New York) 292; Johnson Air Force Base (North Carolina) 293; Security Service 118, 120, 301; Strategic Air Command 272 US Army 39, 40, 42, 43, 44, 52, 74–5, 77, 91, 94, 99, 102, 336, 413, 474; Intelligence Support Activity 452; Marines 525; Security Agency 99, 152, 302, 345–6; Seventh Army Corps 467; Special Forces 249; US Army Air Force 110–11 US Navy 39, 40, 42, 43, 75, 91, 97, 99, 143, 207; US Naval Intelligence 157; Radio Research Station Program 262; Security Group 115, 301; Sixth Fleet 323–4; USS Belmont 260; USS Cochino 112–17, 135; USS Georgetown 260; USS Jamestown 260; USS Liberty 260, 263–4; USS Muller 260; USS Oxford 260; USS Pueblo 260, 264; USS Stickleback 143; USS Tusk 114–15; USS Valdez 260, 605; USS Vincennes 408 USM-49 (US sigint base in Turkey) 303 Vampire (UK intercept unit) 474, 534 ‘Vasiley’ (KGB officer) 232–4 Vass, Sir Douglas 423 Vassall, John 226, 238 Vatican 52 Venona project (intercepted Russian messages) 445–6; and Anglo-American collaboration 72, 78–80; and British Commonwealth 85–8; exploitation of reprinted pages 74–5; and exposure of agents 80–8; extreme secrecy of 73, 77–8; first code-breaks 75–6; and global sigint 90, 94, 98; and Manhattan Project 76; size and importance of 72–3; Soviets alerted to work on 73, 80–1 Vernon, Mike 422, 423 Viehoff (Germany) 50 Vienna 169, 170–1, 172, 372, 373, 375, 376 Vienna Summit (1961) 180 Vietnam 153, 167, 168, 178, 203, 376, 446 Vietnam War 123, 243, 252, 269, 271–2, 277, 279–80, 298, 356, 387 Virgin 545 Vladivostok 129 Voice of Egypt (radio station) 156 Wait, Dave 385–6 Wal Bin Chang 153 Waldegrave, William 429 Walker, John 264, 377, 384, 447 Walker, Walter 165, 166 War Office 22 War on Terror 533, 539–40 Warsaw Pact 114, 244, 245, 247, 248, 251, 253, 257–8, 319, 321, 369, 402, 465 Washington 7, 38, 39, 40, 41, 42, 43, 44, 45, 47, 57, 64, 74, 77, 82, 83, 90, 92, 96, 97, 99, 101, 102, 111, 119, 121, 143, 151, 157, 203, 205, 212, 243, 253, 292, 325, 329, 335, 355–6, 381, 413 Watchkeeper 450 drone 536 Watergate House (London) 15–16 Watergate scandal (1973) 4, 279, 288, 290, 293, 297, 298, 325–6, 356 Wavendon Manor (radio station, Buckinghamshire) 49 weapons of mass destruction (WMD) 516, 520, 523, 526, 528–31 Weatherill, Bernard 459 Weinberger, Casper 441 Weisband, William 80–1, 169 Welchman, Gordon 25, 26, 27, 43, 57, 61, 64, 65, 362–3, 364, 387 Wenger, Joseph 42, 53, 79, 95, 243 West Germany 345, 442, 447–9, 455–6 West Irian 167 West, Lord 544 West, Nigel 88 Western Union 240 Whaddon Hall (Buckinghamshire) 23, 57, 181 Wharfe, Ken 483 White, Ray 501 White, Sir Dick 142, 176, 181, 187, 225, 241–2, 243, 245, 246, 264, 285–6, 353, 354, 364, 453 Whitelaw, Willie 404, 426, 427 Wieck, George 214 Wiesbaden (Germany) 158 Wigg, George 227–8, 240 Wilkes, Detective Sergeant 378 Wilkinson, Peter 206 Williams, Sir Anthony 392–3 Wilson, Edmund 42, 54, 57, 68, 576 Wilson, Harold 225; and Anglo-US relations 356–7; antagonism towards the press 239, 242; and Chevaline project 440; and Cyprus problem 325, 329; and Diego Garcia 338; fascinated and terrified by intelligence and espionage 3, 168, 226–7, 357; and Radcliffe Committee 239–40; and Skynet 438; and U-2 flights from Cyprus 295–6; and Vietnam War 277 Wilson, Jim 292 Wilson, Richard 527 Winnifrith, John 418 Winterbotham, Frederick 35, 354 Wolfenden, Jack 383 Woodward, Admiral Sandy 402–3, 407, 408, 410–11 Wormwood Scrubs prison 238, 385–6 Wreford-Brown, Christopher 404–5 Wright, Georgina 317 Wright, Peter 216, 223, 224, 267, 363, 492, 538, 587–8 Wyllie, Sean 490 Y services (armed forces listening units) 26–7, 33–5, 63, 68, 103, 111, 117, 411 Yarallakos (Cyprus) 320, 328 Yardley, Herbert O. 38 Yemen 148, 163–4 Yom Kippur War (1973) 277, 290–5, 320, 337, 387 York, Duchess of 482 Young, Courtney 86 Yugoslavia 256, 257, 284, 471–5, 503, 512, 534 Yunnan (China) 151 ‘Yuri’ (KGB officer) 231–3 ‘Zhora’ (Weisband’s code name) 80–1 Zimmermann, Phil 490–1 Zimmermann Telegram (1917) 15 Zinnia (missile-detection system) 322–3 Zinser, Aguilar 519–20 Zionist movement 97, 109 Zircon project (GCHQ sigint satellite) 415, 438, 442–3, 449–50, 458–61 Acknowledgements On 9 December 1993 the Lord Chancellor, Lord Mackay of Clashfern, introduced the Intelligence Services Act in the House of Lords.

The New Harvest: Agricultural Innovation in Africa by Calestous Juma

agricultural Revolution, Albert Einstein, barriers to entry, bioinformatics, business climate, carbon footprint, clean water, colonial rule, conceptual framework, creative destruction, CRISPR, double helix, electricity market, energy security, energy transition, export processing zone, global value chain, high-speed rail, impact investing, income per capita, industrial cluster, informal economy, Intergovernmental Panel on Climate Change (IPCC), Joseph Schumpeter, knowledge economy, land tenure, M-Pesa, microcredit, mobile money, non-tariff barriers, off grid, out of africa, precautionary principle, precision agriculture, Recombinant DNA, rolling blackouts, search costs, Second Machine Age, self-driving car, Silicon Valley, sovereign wealth fund, structural adjustment programs, supply-chain management, synthetic biology, systems thinking, total factor productivity, undersea cable

A decade ago it cost more than US$5,000 to install one kilometer of standard fiber-optic cable. The price has dropped to less than US$300. However, for Africa to take advantage of the infrastructure, the cost of bandwidth must decline. Already, Internet service providers are offering more bandwidth for the same cost. By 2011, the four undersea cables operating in Africa had resulted in a quadrupling of data transfer speeds and a 90% price reduction. Access to broadband is challenging Africa’s youth to demonstrate their creativity and African leaders to provide a vision of the role of infrastructure in economic transformation. The emergence of Safaricom’s M-PESA service—a revolutionary 52 THE NEW HARVEST way to transmit money by mobile phones—is an indicator of great prospects for using new technologies for economic improvement.26 In fact, these technologies are creating radically new industries, such as branchless banks, that are revolutionizing the service sector.27 Geographic Information Systems (GIS) The diffusion of geographic information systems is creating new opportunities for development in general and Africa in particular, with regard to agriculture.28 Several countries in Africa have invested in the development of space-based technologies through satellite programs in order to monitor land use and natural resources.

Heart of the Machine: Our Future in a World of Artificial Emotional Intelligence by Richard Yonck

3D printing, AI winter, AlphaGo, Apollo 11, artificial general intelligence, Asperger Syndrome, augmented reality, autism spectrum disorder, backpropagation, Berlin Wall, Bletchley Park, brain emulation, Buckminster Fuller, call centre, cognitive bias, cognitive dissonance, computer age, computer vision, Computing Machinery and Intelligence, crowdsourcing, deep learning, DeepMind, Dunning–Kruger effect, Elon Musk, en.wikipedia.org, epigenetics, Fairchild Semiconductor, friendly AI, Geoffrey Hinton, ghettoisation, industrial robot, Internet of things, invention of writing, Jacques de Vaucanson, job automation, John von Neumann, Kevin Kelly, Law of Accelerating Returns, Loebner Prize, Menlo Park, meta-analysis, Metcalfe’s law, mirror neurons, Neil Armstrong, neurotypical, Nick Bostrom, Oculus Rift, old age dependency ratio, pattern recognition, planned obsolescence, pneumatic tube, RAND corporation, Ray Kurzweil, Rodney Brooks, self-driving car, Skype, social intelligence, SoftBank, software as a service, SQL injection, Stephen Hawking, Steven Pinker, superintelligent machines, technological singularity, TED Talk, telepresence, telepresence robot, The future is already here, The Future of Employment, the scientific method, theory of mind, Turing test, twin studies, Two Sigma, undersea cable, Vernor Vinge, Watson beat the top human players on Jeopardy!, Whole Earth Review, working-age population, zero day

Shouldn’t it take great sophistication and understanding for one person to read and manipulate another person using such advanced technology? Well, yes and no. It certainly does take a highly sophisticated technology to accomplish all of this, just as it takes a very sophisticated technology to call a person instantaneously on the other side of the planet using an elaborate network of satellites, undersea cables, cell towers, and supercomputers, sometimes known as a smartphone. Most of us don’t give this latter process a moment’s thought because it has been removed from our awareness by layer upon layer of abstraction until we take no more notice of it than we do flicking a light switch. (Which by the way, is also abstracting numerous complex processes most of us never want to be bothered with.)

Rocket Billionaires: Elon Musk, Jeff Bezos, and the New Space Race by Tim Fernholz

Amazon Web Services, Apollo 13, autonomous vehicles, business climate, Charles Lindbergh, Clayton Christensen, cloud computing, Colonization of Mars, corporate governance, corporate social responsibility, deep learning, disruptive innovation, Donald Trump, Elon Musk, fail fast, fulfillment center, Gene Kranz, high net worth, high-speed rail, Iridium satellite, Jeff Bezos, Kickstarter, Kim Stanley Robinson, Kwajalein Atoll, low earth orbit, Marc Andreessen, Mark Zuckerberg, Mars Society, Masayoshi Son, megaproject, military-industrial complex, minimum viable product, multiplanetary species, mutually assured destruction, Neal Stephenson, Neil Armstrong, new economy, no-fly zone, nuclear paranoia, paypal mafia, Peter H. Diamandis: Planetary Resources, Peter Thiel, pets.com, planetary scale, private spaceflight, profit maximization, RAND corporation, Richard Feynman, Richard Feynman: Challenger O-ring, Ronald Reagan, satellite internet, Scaled Composites, shareholder value, Silicon Valley, skunkworks, SoftBank, sovereign wealth fund, space junk, SpaceShipOne, Stephen Hawking, Steve Jobs, Strategic Defense Initiative, trade route, undersea cable, vertical integration, Virgin Galactic, VTOL, We wanted flying cars, instead we got 140 characters, X Prize, Y2K

The internet was not as controversial as resource-extraction concessions or low-wage manufacturing, if only because its power to shape political events was only slowly becoming clear. His second key lesson was that putting wires underground was costly and time-consuming. It was no way to bring the information age to sparsely populated areas or those, like Rwanda, that are far from the undersea cables that link the continents together. Why not just use satellites to do the job of linking the local network to the outside world? In 2007, Wyler cofounded a company called O3b, which stood for the “other three billion”—a shorthand for the underserved chunk of the global population it targeted.

Moonshot: The Inside Story of Mankind's Greatest Adventure by Dan Parry

Apollo 11, Apollo 13, Apollo Guidance Computer, Charles Lindbergh, Dr. Strangelove, en.wikipedia.org, Gene Kranz, low earth orbit, Neil Armstrong, Norman Mailer, orbital mechanics / astrodynamics, undersea cable, white flight

As the Earth spun on its axis, at least one of the three powerful antennas would be in sight of the Moon throughout the mission, with each station handing control to the next to maintain a continuous connection. Voice signals, tracking data and telemetry were passed to Houston via NASA's Communications Network, which linked the ground stations and other facilities through two million miles of landlines and undersea cables. Relying on six intermediate switching centres around the world, and two communications satellites, the network included redundant signal routes to increase reliability. Some remote switching points were triggered automatically whenever the CapCom began to speak. Each time he pressed his microphone button, it emitted a short beep known as a Quindar tone.

The God Species: Saving the Planet in the Age of Humans by Mark Lynas

Airbus A320, Anthropocene, back-to-the-land, Berlin Wall, biodiversity loss, carbon credits, carbon footprint, clean water, Climategate, Climatic Research Unit, data science, David Ricardo: comparative advantage, decarbonisation, degrowth, dematerialisation, demographic transition, Easter island, Eyjafjallajökull, Great Leap Forward, Haber-Bosch Process, ice-free Arctic, Intergovernmental Panel on Climate Change (IPCC), invention of the steam engine, James Watt: steam engine, megacity, meta-analysis, moral hazard, Negawatt, New Urbanism, ocean acidification, oil shale / tar sands, out of africa, peak oil, planetary scale, precautionary principle, quantitative easing, race to the bottom, rewilding, Ronald Reagan, special drawing rights, Stewart Brand, synthetic biology, Tragedy of the Commons, two and twenty, undersea cable, University of East Anglia, We are as Gods

., new wind installations fell by half in 2010 compared to the previous year, thanks to the recession and political wrangling over energy policy (or lack of it), even while the industry went from strength to strength in Europe and Asia.71 One of the big remaining hurdles in America is a straightforward lack of wires: New wind installations often lack grid connections to take their clean electricity to markets. In October 2010 Google Foundation became a partner in the Atlantic Wind Connection, a plan for a high-voltage undersea cable to bring large-scale offshore wind to the U.S. East Coast.72 In the U.K. onshore wind is increasingly being stymied by local opposition: According to one report, at least 230 local anti-wind farm campaigns now operate across the country, and approvals for new wind farms have fallen to an all-time low.73 From a climate change perspective, opposing wind is just as reprehensible as opposing nuclear, and as I discuss later in this book, I do not think any of the planetary boundaries offer compelling environmental reasons for completely ruling out wind—even though problems with bird and bat kills are real and severe in some cases.

The Myth of Capitalism: Monopolies and the Death of Competition by Jonathan Tepper

"Friedman doctrine" OR "shareholder theory", Affordable Care Act / Obamacare, air freight, Airbnb, airline deregulation, Alan Greenspan, bank run, barriers to entry, Berlin Wall, Bernie Sanders, Big Tech, big-box store, Bob Noyce, Boston Dynamics, business cycle, Capital in the Twenty-First Century by Thomas Piketty, citizen journalism, Clayton Christensen, collapse of Lehman Brothers, collective bargaining, compensation consultant, computer age, Cornelius Vanderbilt, corporate raider, creative destruction, Credit Default Swap, crony capitalism, diversification, don't be evil, Donald Trump, Double Irish / Dutch Sandwich, Dunbar number, Edward Snowden, Elon Musk, en.wikipedia.org, eurozone crisis, Fairchild Semiconductor, Fall of the Berlin Wall, family office, financial innovation, full employment, gentrification, German hyperinflation, gig economy, Gini coefficient, Goldman Sachs: Vampire Squid, Google bus, Google Chrome, Gordon Gekko, Herbert Marcuse, income inequality, independent contractor, index fund, Innovator's Dilemma, intangible asset, invisible hand, Jeff Bezos, Jeremy Corbyn, Jevons paradox, John Nash: game theory, John von Neumann, Joseph Schumpeter, junk bonds, Kenneth Rogoff, late capitalism, London Interbank Offered Rate, low skilled workers, Mark Zuckerberg, Martin Wolf, Maslow's hierarchy, means of production, merger arbitrage, Metcalfe's law, multi-sided market, mutually assured destruction, Nash equilibrium, Network effects, new economy, Northern Rock, offshore financial centre, opioid epidemic / opioid crisis, passive investing, patent troll, Peter Thiel, plutocrats, prediction markets, prisoner's dilemma, proprietary trading, race to the bottom, rent-seeking, road to serfdom, Robert Bork, Ronald Reagan, Sam Peltzman, secular stagnation, shareholder value, Sheryl Sandberg, Silicon Valley, Silicon Valley billionaire, Skype, Snapchat, Social Responsibility of Business Is to Increase Its Profits, SoftBank, Steve Jobs, stock buybacks, tech billionaire, The Chicago School, The Wealth of Nations by Adam Smith, Thomas Kuhn: the structure of scientific revolutions, too big to fail, undersea cable, Vanguard fund, vertical integration, very high income, wikimedia commons, William Shockley: the traitorous eight, you are the product, zero-sum game

Kindle Edition, locations 322–323. 63. https://www.seattletimes.com/business/tech-giants-put-the-squeeze-on-startups-squelching-their-chances-of-success/. 64. https://www.theguardian.com/technology/2017/oct/20/tech-startups-facebook-amazon-google-apple. 65. https://www.recode.net/2017/5/10/15602814/amazon-invested-startup-nucleus-cloned-alexa-echo-show-voice-control-touchscreen-video. 66. https://en.wikipedia.org/wiki/Google_data_centers. 67. https://peering.google.com/#/infrastructure. 68. http://www.dailymail.co.uk/sciencetech/article-5275893/Google-reveals-plan-build-THREE-new-undersea-cables.html. 69. http://www.wired.co.uk/article/google-facebook-plcn-internet-cable/. 70. https://www.ben-evans.com/benedictevans/2017/10/12/scale-wetxp. 71. https://www.theringer.com/tech/2018/5/18/17362452/microsoft-antitrust-lawsuit-netscape-internet-explorer-20-years. 72. https://www.sfgate.com/politics/article/Microsoft-Asked-Apple-to-Knife-the-Baby-Court-2980345.php. 73. https://promarket.org/google-facebooks-kill-zone-weve-taken-focus-off-rewarding-genius-innovation-rewarding-capital-scale/.

Augmented: Life in the Smart Lane by Brett King

23andMe, 3D printing, additive manufacturing, Affordable Care Act / Obamacare, agricultural Revolution, Airbnb, Albert Einstein, Amazon Web Services, Any sufficiently advanced technology is indistinguishable from magic, Apollo 11, Apollo Guidance Computer, Apple II, artificial general intelligence, asset allocation, augmented reality, autonomous vehicles, barriers to entry, bitcoin, Bletchley Park, blockchain, Boston Dynamics, business intelligence, business process, call centre, chief data officer, Chris Urmson, Clayton Christensen, clean water, Computing Machinery and Intelligence, congestion charging, CRISPR, crowdsourcing, cryptocurrency, data science, deep learning, DeepMind, deskilling, different worldview, disruptive innovation, distributed generation, distributed ledger, double helix, drone strike, electricity market, Elon Musk, Erik Brynjolfsson, Fellow of the Royal Society, fiat currency, financial exclusion, Flash crash, Flynn Effect, Ford Model T, future of work, gamification, Geoffrey Hinton, gig economy, gigafactory, Google Glasses, Google X / Alphabet X, Hans Lippershey, high-speed rail, Hyperloop, income inequality, industrial robot, information asymmetry, Internet of things, invention of movable type, invention of the printing press, invention of the telephone, invention of the wheel, James Dyson, Jeff Bezos, job automation, job-hopping, John Markoff, John von Neumann, Kevin Kelly, Kickstarter, Kim Stanley Robinson, Kiva Systems, Kodak vs Instagram, Leonard Kleinrock, lifelogging, low earth orbit, low skilled workers, Lyft, M-Pesa, Mark Zuckerberg, Marshall McLuhan, megacity, Metcalfe’s law, Minecraft, mobile money, money market fund, more computing power than Apollo, Neal Stephenson, Neil Armstrong, Network effects, new economy, Nick Bostrom, obamacare, Occupy movement, Oculus Rift, off grid, off-the-grid, packet switching, pattern recognition, peer-to-peer, Ray Kurzweil, retail therapy, RFID, ride hailing / ride sharing, Robert Metcalfe, Salesforce, Satoshi Nakamoto, Second Machine Age, selective serotonin reuptake inhibitor (SSRI), self-driving car, sharing economy, Shoshana Zuboff, Silicon Valley, Silicon Valley startup, Skype, smart cities, smart grid, smart transportation, Snapchat, Snow Crash, social graph, software as a service, speech recognition, statistical model, stem cell, Stephen Hawking, Steve Jobs, Steve Wozniak, strong AI, synthetic biology, systems thinking, TaskRabbit, technological singularity, TED Talk, telemarketer, telepresence, telepresence robot, Tesla Model S, The future is already here, The Future of Employment, Tim Cook: Apple, trade route, Travis Kalanick, TSMC, Turing complete, Turing test, Twitter Arab Spring, uber lyft, undersea cable, urban sprawl, V2 rocket, warehouse automation, warehouse robotics, Watson beat the top human players on Jeopardy!, white picket fence, WikiLeaks, yottabyte

Today, we know Western Union as a money transmitter but, back in 1856, Western Union was the largest provider of telegraph services across the United States, and by 1890 its reach extended even across the Atlantic. Inflation adjusted, Western Union was capitalised at around US$850 million dollars (US$41 million actual) in 1876. Western Union was operating more than a million miles of telegraph lines and two international undersea cables at the time. When Alexander Graham Bell and his partners5 patented the telephone in 1876, initially referring to it as a “talking telegraph”, they offered Western Union the patent for the sum of US$100,000.6 The established communications monopoly declined their offer, deciding that higher dividends were more important than investment in new technology.

The Scientist as Rebel by Freeman Dyson

"World Economic Forum" Davos, Albert Einstein, Asilomar, Boeing 747, British Empire, Claude Shannon: information theory, dark matter, double helix, Edmond Halley, Ernest Rutherford, experimental subject, Fellow of the Royal Society, From Mathematics to the Technologies of Life and Death, Gregor Mendel, Henri Poincaré, Isaac Newton, Johannes Kepler, John von Neumann, kremlinology, Mikhail Gorbachev, military-industrial complex, Norbert Wiener, Paul Erdős, Plato's cave, precautionary principle, quantum entanglement, Recombinant DNA, Richard Feynman, Ronald Reagan, seminal paper, Silicon Valley, Stephen Hawking, Thomas Kuhn: the structure of scientific revolutions, traveling salesman, undersea cable

To determine longitude at a remote place such as Dakar or Haiphong, it was necessary to compare the local time, obtained from local astronomical measurements, with Paris time, obtained by receiving an accurate time signal from Paris. The accuracy of maps therefore depended on the accuracy of long-distance transmission of time signals. The transmission of time signals, first by overland telegraph lines and undersea cables, and later by radio, was a difficult technical problem. Signals were attenuated by transmission losses and corrupted by ambient noise. The transmission introduced delays which had to be accurately calculated, so that Paris time could be correctly deduced from the observed time of reception of signals.

Sandworm: A New Era of Cyberwar and the Hunt for the Kremlin's Most Dangerous Hackers by Andy Greenberg

"World Economic Forum" Davos, air freight, air gap, Airbnb, Bellingcat, Bernie Sanders, bitcoin, blockchain, call centre, Citizen Lab, clean water, data acquisition, disinformation, Donald Trump, Edward Snowden, false flag, global supply chain, Hacker News, hive mind, information security, Julian Assange, Just-in-time delivery, Kickstarter, machine readable, Mikhail Gorbachev, no-fly zone, open borders, pirate software, pre–internet, profit motive, ransomware, RFID, speech recognition, Steven Levy, Stuxnet, supply-chain attack, tech worker, undersea cable, unit 8200, uranium enrichment, Valery Gerasimov, WikiLeaks, zero day

Lee, of course, says little about the details of where that intelligence came from. But thanks in part to Edward Snowden, we know that it included a broad array of secret data-gathering tools, labeled broadly as “signals intelligence,” or “sigint,” that ranged from the ability to siphon vast quantities of raw internet data from undersea cables to hacking enemy systems administrators and looking over their shoulders at private networks. “When you’re given access to essentially the entirety of the U.S. sigint system and then surrounded with the smartest people doing this on the planet, you get spun up pretty quickly,” Lee says. For the next four years, he and a small team of around six analysts spent every working hour tracking the burgeoning, post-Stuxnet world of industrial control system hackers.

The Crux by Richard Rumelt

activist fund / activist shareholder / activist investor, air gap, Airbnb, AltaVista, AOL-Time Warner, Bayesian statistics, behavioural economics, biodiversity loss, Blue Ocean Strategy, Boeing 737 MAX, Boeing 747, Charles Lindbergh, Clayton Christensen, cloud computing, cognitive bias, commoditize, coronavirus, corporate raider, COVID-19, creative destruction, crossover SUV, Crossrail, deep learning, Deng Xiaoping, diversified portfolio, double entry bookkeeping, drop ship, Elon Musk, en.wikipedia.org, financial engineering, Ford Model T, Herman Kahn, income inequality, index card, Internet of things, Jeff Bezos, Just-in-time delivery, Larry Ellison, linear programming, lockdown, low cost airline, low earth orbit, Lyft, Marc Benioff, Mark Zuckerberg, Masayoshi Son, meta-analysis, Myron Scholes, natural language processing, Neil Armstrong, Network effects, packet switching, PageRank, performance metric, precision agriculture, RAND corporation, ride hailing / ride sharing, Salesforce, San Francisco homelessness, search costs, selection bias, self-driving car, shareholder value, sharing economy, Silicon Valley, Skype, Snapchat, social distancing, SoftBank, software as a service, statistical model, Steve Ballmer, Steve Jobs, stochastic process, Teledyne, telemarketer, TSMC, uber lyft, undersea cable, union organizing, vertical integration, WeWork

For now, we think our reputation and customer relationships will carry us, but in the longer term … ? Our development efforts are currently concentrated in Ohio, where we have seventy-five people working on three different efforts. One is drone-based sensors. A second is working on sensors to be integrated into undersea cables, and the third is looking at integrating our sensors into Wi-Fi and Internet environments. Right now, they are wired rather than wireless. The owners brought new blood into the board last year, and … I would like to say it was a breath of fresh air. One of the new board members, ‘John Cherold,’ is from a finance background, and he says XRS is underperforming.

Age of Discovery: Navigating the Risks and Rewards of Our New Renaissance by Ian Goldin, Chris Kutarna

"World Economic Forum" Davos, 2013 Report for America's Infrastructure - American Society of Civil Engineers - 19 March 2013, 3D printing, Airbnb, Albert Einstein, AltaVista, Asian financial crisis, asset-backed security, autonomous vehicles, banking crisis, barriers to entry, battle of ideas, Bear Stearns, Berlin Wall, bioinformatics, bitcoin, Boeing 747, Bonfire of the Vanities, bread and circuses, carbon tax, clean water, collective bargaining, Colonization of Mars, Credit Default Swap, CRISPR, crowdsourcing, cryptocurrency, Dava Sobel, demographic dividend, Deng Xiaoping, digital divide, Doha Development Round, double helix, driverless car, Edward Snowden, Elon Musk, en.wikipedia.org, epigenetics, experimental economics, Eyjafjallajökull, failed state, Fall of the Berlin Wall, financial innovation, full employment, Galaxy Zoo, general purpose technology, Glass-Steagall Act, global pandemic, global supply chain, Higgs boson, Hyperloop, immigration reform, income inequality, indoor plumbing, industrial cluster, industrial robot, information retrieval, information security, Intergovernmental Panel on Climate Change (IPCC), intermodal, Internet of things, invention of the printing press, Isaac Newton, Islamic Golden Age, Johannes Kepler, Khan Academy, Kickstarter, Large Hadron Collider, low cost airline, low skilled workers, Lyft, Mahbub ul Haq, Malacca Straits, mass immigration, Max Levchin, megacity, Mikhail Gorbachev, moral hazard, Nelson Mandela, Network effects, New Urbanism, non-tariff barriers, Occupy movement, On the Revolutions of the Heavenly Spheres, open economy, Panamax, Paris climate accords, Pearl River Delta, personalized medicine, Peter Thiel, post-Panamax, profit motive, public intellectual, quantum cryptography, rent-seeking, reshoring, Robert Gordon, Robert Metcalfe, Search for Extraterrestrial Intelligence, Second Machine Age, self-driving car, Shenzhen was a fishing village, Silicon Valley, Silicon Valley startup, Skype, smart grid, Snapchat, special economic zone, spice trade, statistical model, Stephen Hawking, Steve Jobs, Stuxnet, synthetic biology, TED Talk, The Future of Employment, too big to fail, trade liberalization, trade route, transaction costs, transatlantic slave trade, uber lyft, undersea cable, uranium enrichment, We are the 99%, We wanted flying cars, instead we got 140 characters, working poor, working-age population, zero day

From a user perspective, the Internet is an invisible field that connects us everywhere. But this user experience has a physical dimension—data centers and fiber optic cables that concentrate our connectivity in dangerous ways. The same strait and canal chokepoints through which the world’s ships must pass are also the best routes along which to lay undersea cables. Every year in Africa, the networks of whole countries go dark when one of these threads is cut by a passing ship’s anchor—or by an axe. In 2013, the Egyptian coast guard caught three men in a fishing boat who had pulled up, and were trying to hack through, a Suez cable deliberately. Their target, the SEA-ME-WE 4 cable, is one of the principal data links between Europe, Africa and Asia.

Seasteading: How Floating Nations Will Restore the Environment, Enrich the Poor, Cure the Sick, and Liberate Humanity From Politicians by Joe Quirk, Patri Friedman

3D printing, access to a mobile phone, addicted to oil, Affordable Care Act / Obamacare, agricultural Revolution, Albert Einstein, barriers to entry, biodiversity loss, Biosphere 2, Branko Milanovic, British Empire, Buckminster Fuller, Burning Man, business climate, business cycle, business process, California gold rush, Celtic Tiger, Charles Lindbergh, clean water, Colonization of Mars, Dean Kamen, Deng Xiaoping, drone strike, Elon Musk, en.wikipedia.org, export processing zone, failed state, financial intermediation, Garrett Hardin, Gini coefficient, Great Leap Forward, happiness index / gross national happiness, income inequality, intentional community, Intergovernmental Panel on Climate Change (IPCC), joint-stock company, joint-stock limited liability company, Kickstarter, low skilled workers, Machinery of Freedom by David Friedman, Mark Zuckerberg, megacity, megaproject, minimum wage unemployment, Neil Armstrong, Network effects, new economy, obamacare, ocean acidification, off-the-grid, offshore financial centre, One Laptop per Child (OLPC), open borders, Patri Friedman, paypal mafia, peak oil, Peter H. Diamandis: Planetary Resources, Peter Thiel, price stability, profit motive, radical decentralization, Ronald Coase, Ronald Reagan, Shenzhen special economic zone , Shenzhen was a fishing village, Silicon Valley, special economic zone, standardized shipping container, stem cell, TechCrunch disrupt, TED Talk, trade route, Tragedy of the Commons, UNCLOS, UNCLOS, undersea cable, young professional

If seaweed can clean the tiniest proportion of these zones, producing food or fuel at a profit, it can be scaled up to colossal proportions. What works in microcosm can work better in macrocosm. That’s the magic of profits. It’s how one oil well becomes a million oil wells, one lightbulb becomes a trillion lightbulbs, and one voice carried along a wire to the next room becomes a planet crisscrossed with telegraph lines and undersea cables. Ricardo cites an oft-repeated fact in his academic paper: “Up to 90 per cent of wastewater flows untreated into densely populated coastal zones contributing to growing marine dead zones.” But he offers a unique solution: “[I]t is logical . . . to take advantage of these nutrient-rich water sources for production, as well as cleaning them through ‘bioremediation.’ ” Bioremediation is sciencespeak for the process of culturing microorganisms to metabolize the resources that humans consider pollution.

The Land Grabbers: The New Fight Over Who Owns the Earth by Fred Pearce

activist lawyer, Asian financial crisis, banking crisis, big-box store, Black Monday: stock market crash in 1987, blood diamond, British Empire, Buy land – they’re not making it any more, Cape to Cairo, carbon credits, carbon footprint, clean water, company town, corporate raider, credit crunch, Deng Xiaoping, Elliott wave, en.wikipedia.org, energy security, farmers can use mobile phones to check market prices, Garrett Hardin, Global Witness, index fund, Jeff Bezos, Kickstarter, Kondratiev cycle, land reform, land tenure, Mahatma Gandhi, market fundamentalism, megacity, megaproject, Mohammed Bouazizi, Nelson Mandela, Nikolai Kondratiev, offshore financial centre, out of africa, quantitative easing, race to the bottom, Ronald Reagan, smart cities, structural adjustment programs, too big to fail, Tragedy of the Commons, undersea cable, urban planning, urban sprawl, vertical integration, WikiLeaks

At least the Purari Development Association had a formal existence when it was granted a special lease for 1.5 million acres. But the purpose raised eyebrows. It plans to construct a giant 1,800-megawatt hydroelectric plant that would flood much of the valley of the Purari River and send power by undersea cable 300 miles across the Coral Sea to Queensland. Some locals in the flood zone said they knew about the project, but others claimed they had “never been consulted or given permission” for the takeover. There seemed to be more local support for a scheme from a South Korean company known as Changhae Tapioka.

Wireless by Charles Stross

air gap, anthropic principle, back-to-the-land, Benoit Mandelbrot, Buckminster Fuller, Cepheid variable, cognitive dissonance, colonial exploitation, cosmic microwave background, Easter island, epigenetics, finite state, Georg Cantor, gravity well, hive mind, hydroponic farming, jitney, Khyber Pass, Late Heavy Bombardment, launch on warning, lifelogging, Magellanic Cloud, mandelbrot fractal, MITM: man-in-the-middle, Neil Armstrong, peak oil, phenotype, Pluto: dwarf planet, security theater, sensible shoes, Turing machine, undersea cable

Radio can’t punch through more than a few thousand miles of air directly, however powerful the transmitters; on Earth they used to bounce signals off the ionosphere or the moon, but that doesn’t work here—the other disks are too far away to use as relays. There’s a chain of transceiver buoys marching out across the ocean at two-thousand-kilometer intervals, but the equipment is a pig to maintain, very expensive to build, and nobody is even joking about stringing undersea cables across a million kilometers of seafloor. Misha’s problem is that the expedition, himself included, is effectively stranded back in the eighteenth century, without even the telegraph to tie civilization together—which is a pretty pickle to find yourself in when you’re the bearer of news that will make the Politburo shit a brick.

Cuckoo's Egg by Clifford Stoll

affirmative action, call centre, Golden Gate Park, hiring and firing, information security, John Markoff, Menlo Park, old-boy network, Paul Graham, Richard Stallman, Silicon Valley, Strategic Defense Initiative, undersea cable

Steve dealt in worldwide communications like a taxi driver in midtown traffic. “Why isn’t he on a satellite link?” “Probably because it’s a Sunday—the cable channels are less crowded.” “You mean that people prefer cable to satellite links?” “Sure. Every time you connect through a satellite, there’s a quarter second delay. The undersea cables don’t slow down your messages so much.” “Who would care?” “People on the telephone, mostly,” Steve said. “Those delays make for jittery conversations. You know, where each person tries to speak at the same time, then they both back off.” “So if the phone companies try to route over the cables, who wants the satellites?”

The Hacker and the State: Cyber Attacks and the New Normal of Geopolitics by Ben Buchanan

active measures, air gap, Bernie Sanders, bitcoin, blockchain, borderless world, Brian Krebs, British Empire, Cass Sunstein, citizen journalism, Citizen Lab, credit crunch, cryptocurrency, cuban missile crisis, data acquisition, disinformation, Donald Trump, drone strike, Edward Snowden, fake news, family office, Hacker News, hive mind, information security, Internet Archive, Jacob Appelbaum, John Markoff, John von Neumann, Julian Assange, Kevin Roose, Kickstarter, kremlinology, Laura Poitras, MITM: man-in-the-middle, Nate Silver, operational security, post-truth, profit motive, RAND corporation, ransomware, risk tolerance, Robert Hanssen: Double agent, rolodex, Ronald Reagan, Russian election interference, seminal paper, Silicon Valley, South China Sea, Steve Jobs, Stuxnet, subscription business, technoutopianism, undersea cable, uranium enrichment, Vladimir Vetrov: Farewell Dossier, Wargames Reagan, WikiLeaks, zero day

For espionage purposes, the Chinese have hacked military targets, valuable but obscure offices in the United States, key parts of the computing supply chain, major cloud service providers with gigantic amounts of data, and many other targets.8 The success of companies like Huawei in manufacturing, building, and repairing undersea cables and new 5G telecommunications infrastructure raises fears that the Chinese government may try to match the advantages in passive collection that the Five Eyes enjoy, according to statements of former government officials and leaked documents.9 Russian agencies have likewise continued to expand their cyber espionage repertoire.

The Ultimate Engineer: The Remarkable Life of NASA's Visionary Leader George M. Low by Richard Jurek

additive manufacturing, affirmative action, Apollo 11, Apollo 13, Charles Lindbergh, cognitive dissonance, en.wikipedia.org, Ford Model T, fudge factor, Gene Kranz, human-factors engineering, it's over 9,000, John Conway, low earth orbit, Mars Rover, Neil Armstrong, operation paperclip, orbital mechanics / astrodynamics, private spaceflight, Ronald Reagan, Silicon Valley, Silicon Valley ideology, Stewart Brand, undersea cable, uranium enrichment, Whole Earth Catalog, Winter of Discontent, women in the workforce

He named some of the recent practical applications in science and medicine that were currently providing major benefits, before he centered on the field of communications. “Breakthroughs are already being made. For example, fifteen years ago, no one would have predicted color television broadcasts from China would have been possible.” He discussed the use of satellites for voice communication, over the antiquated use of undersea cable systems. He discussed solar-energy applications, pointing out that NASA was now exploring “ways of collecting the sun’s energy” for practical use on Earth. Satellites, he said, while showing the audience a picture from space of sewage spills off Long Island, can also spot acid slides and sewage dumping and thus can provide officials with the tools to respond and combat these environmental problems.

The Great Firewall of China by James Griffiths;

A Declaration of the Independence of Cyberspace, activist fund / activist shareholder / activist investor, Albert Einstein, anti-communist, bike sharing, bitcoin, Black Lives Matter, borderless world, call centre, Cambridge Analytica, Chelsea Manning, Citizen Lab, Deng Xiaoping, digital divide, digital rights, disinformation, don't be evil, Donald Trump, Edward Snowden, end-to-end encryption, Evgeny Morozov, fake news, gig economy, Great Leap Forward, high-speed rail, jimmy wales, John Gilmore, John Perry Barlow, Mark Zuckerberg, megacity, megaproject, microaggression, Mikhail Gorbachev, Mitch Kapor, mobile money, Occupy movement, pets.com, profit motive, QR code, race to the bottom, RAND corporation, ride hailing / ride sharing, Ronald Reagan, Silicon Valley, Silicon Valley startup, Skype, Snapchat, South China Sea, Steve Jobs, Stewart Brand, Stuxnet, technoutopianism, The future is already here, undersea cable, WikiLeaks, zero day

The ones most people will be familiar with is when a personal modem isn’t working and the request never leaves the house, or (less often today than years ago) the ISP’s servers are down and the request can’t be converted into an IP address. But disruption can also happen at the other end of the equation. In 2008, millions of users across the Middle East were cut off from servers in the US when several undersea cables were damaged by unknown causes.53 While that outage was accidental, the Xinjiang internet blackout the following year showed that the same effect could be achieved intentionally. China is particularly vulnerable in this regard, as all internet backbones and major exchanges are owned or controlled by government entities.54 China Unicom, the colossal state-owned telecoms company that administers much of the country’s internet infrastructure, has three major ‘national-level’ exchanges in Beijing, Guangzhou and Shanghai,55 from where traffic is routed to the rest of the world.

Dangerous Ideas: A Brief History of Censorship in the West, From the Ancients to Fake News by Eric Berkowitz

Albert Einstein, algorithmic management, anti-communist, Ayatollah Khomeini, Big Tech, Black Lives Matter, Bonfire of the Vanities, borderless world, Brexit referendum, British Empire, Charlie Hebdo massacre, Chelsea Manning, colonial rule, coronavirus, COVID-19, deplatforming, disinformation, Donald Trump, Edward Snowden, Evgeny Morozov, fake news, Filter Bubble, high-speed rail, Index librorum prohibitorum, Jeff Bezos, Julian Assange, lockdown, Mark Zuckerberg, microaggression, Mikhail Gorbachev, Minecraft, New Urbanism, post-truth, pre–internet, QAnon, Ralph Nader, Saturday Night Live, Silicon Valley, source of truth, Steve Bannon, surveillance capitalism, undersea cable, W. E. B. Du Bois, WikiLeaks

Add to that new, wide-ranging prohibitions on sedition—typical were New Zealand’s criminalization of words that “excite disloyalty” and the US law against speech that could bring the government into “disrepute”—and the foundation for what the conservative and later Nazi jurist Carl Schmitt called state intervention in “all spheres of human existence” was laid.39 Winston Churchill, then first lord of the Admiralty, expressed the British government’s view about war reportage when he told editors, “The best place for correspondence about this war will be in London,” and that it should be “fought in a fog.”40 As soon as the fighting began, in August, all the undersea cables surrounding Britain except those that were British-owned were cut, and all communications entering the country were intercepted. Britain’s secretary of state for war, Lord Horatio Kitchener, banned correspondents from the area surrounding the British Expeditionary Force. The hasty passage of the Defence of the Realm Act greatly magnified the state’s power to control information and punish its misuse.

Radical Technologies: The Design of Everyday Life by Adam Greenfield

3D printing, Airbnb, algorithmic bias, algorithmic management, AlphaGo, augmented reality, autonomous vehicles, bank run, barriers to entry, basic income, bitcoin, Black Lives Matter, blockchain, Boston Dynamics, business intelligence, business process, Californian Ideology, call centre, cellular automata, centralized clearinghouse, centre right, Chuck Templeton: OpenTable:, circular economy, cloud computing, Cody Wilson, collective bargaining, combinatorial explosion, Computer Numeric Control, computer vision, Conway's Game of Life, CRISPR, cryptocurrency, David Graeber, deep learning, DeepMind, dematerialisation, digital map, disruptive innovation, distributed ledger, driverless car, drone strike, Elon Musk, Ethereum, ethereum blockchain, facts on the ground, fiat currency, fulfillment center, gentrification, global supply chain, global village, Goodhart's law, Google Glasses, Herman Kahn, Ian Bogost, IBM and the Holocaust, industrial robot, informal economy, information retrieval, Internet of things, Jacob Silverman, James Watt: steam engine, Jane Jacobs, Jeff Bezos, Jeff Hawkins, job automation, jobs below the API, John Conway, John Markoff, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John Perry Barlow, John von Neumann, joint-stock company, Kevin Kelly, Kickstarter, Kiva Systems, late capitalism, Leo Hollis, license plate recognition, lifelogging, M-Pesa, Mark Zuckerberg, means of production, megacity, megastructure, minimum viable product, money: store of value / unit of account / medium of exchange, natural language processing, Network effects, New Urbanism, Nick Bostrom, Occupy movement, Oculus Rift, off-the-grid, PalmPilot, Pareto efficiency, pattern recognition, Pearl River Delta, performance metric, Peter Eisenman, Peter Thiel, planetary scale, Ponzi scheme, post scarcity, post-work, printed gun, proprietary trading, RAND corporation, recommendation engine, RFID, rolodex, Rutger Bregman, Satoshi Nakamoto, self-driving car, sentiment analysis, shareholder value, sharing economy, Shenzhen special economic zone , Sidewalk Labs, Silicon Valley, smart cities, smart contracts, social intelligence, sorting algorithm, special economic zone, speech recognition, stakhanovite, statistical model, stem cell, technoutopianism, Tesla Model S, the built environment, The Death and Life of Great American Cities, The Future of Employment, Tony Fadell, transaction costs, Uber for X, undersea cable, universal basic income, urban planning, urban sprawl, vertical integration, Vitalik Buterin, warehouse robotics, When a measure becomes a target, Whole Earth Review, WikiLeaks, women in the workforce

Beyond the satellites, camera cars and servers we’ve already identified, the moment-to-moment flow of our experience rests vitally on the smooth interfunctioning of all the many parts of this infrastructure—an extraordinarily heterogeneous and unstable meshwork, in which cellular base stations, undersea cables, and microwave relays are all invoked in what seems like the simplest and most straightforward tasks we perform with the device. The very first lesson of mapping on the smartphone, then, is that the handset is primarily a tangible way of engaging something much subtler and harder to discern, on which we have suddenly become reliant and over which we have virtually no meaningful control.

This Machine Kills Secrets: Julian Assange, the Cypherpunks, and Their Fight to Empower Whistleblowers by Andy Greenberg

air gap, Apple II, Ayatollah Khomeini, Berlin Wall, Bill Gates: Altair 8800, Bletchley Park, Burning Man, Chelsea Manning, computerized markets, crowdsourcing, cryptocurrency, disinformation, domain-specific language, driverless car, drone strike, en.wikipedia.org, Evgeny Morozov, Fairchild Semiconductor, fault tolerance, hive mind, information security, Jacob Appelbaum, John Gilmore, John Perry Barlow, Julian Assange, Lewis Mumford, Mahatma Gandhi, military-industrial complex, Mitch Kapor, MITM: man-in-the-middle, Mohammed Bouazizi, Mondo 2000, Neal Stephenson, nuclear winter, offshore financial centre, operational security, PalmPilot, pattern recognition, profit motive, Ralph Nader, real-name policy, reality distortion field, Richard Stallman, Robert Hanssen: Double agent, Silicon Valley, Silicon Valley ideology, Skype, social graph, SQL injection, statistical model, stem cell, Steve Jobs, Steve Wozniak, Steven Levy, Teledyne, three-masted sailing ship, undersea cable, Vernor Vinge, We are Anonymous. We are Legion, We are the 99%, WikiLeaks, X Prize, Zimmermann PGP

Assange, then using the handle Proff, took note of Mudge’s cleverer hacks and often distributed L0pht’s advisories to the Best of Security group that Assange ran on Suburbia’s servers. But the two hackers had a mutual regard for the other’s alpha geek status that went beyond reading each other’s research, and they exchanged ideas over the undersea cables that connected Melbourne and Boston. More than once, they met in the physical world, including at least one meal together at the Chaos Communication Congress in Berlin. “Everyone knew everyone on IRC,” says one ex-member of the L0pht. “But when Mudge and Assange met at conferences or had dinner, that seemed like a different sort of connection.”

The Dawn of Innovation: The First American Industrial Revolution by Charles R. Morris

air freight, American ideology, British Empire, business process, California gold rush, Charles Babbage, clean water, colonial exploitation, computer age, Cornelius Vanderbilt, cotton gin, Dava Sobel, en.wikipedia.org, flying shuttle, Ford Model T, glass ceiling, high-speed rail, hiring and firing, Ida Tarbell, if you build it, they will come, interchangeable parts, Isaac Newton, Jacquard loom, James Hargreaves, James Watt: steam engine, John Harrison: Longitude, joint-stock company, lone genius, manufacturing employment, megaproject, new economy, New Urbanism, old age dependency ratio, On the Economy of Machinery and Manufactures, purchasing power parity, QWERTY keyboard, refrigerator car, Robert Gordon, scientific management, spinning jenny, Stephen Hawking, The Wealth of Nations by Adam Smith, trade route, transcontinental railway, traveling salesman, undersea cable

The ensuing battle may be the only naval engagement of the war that turned almost entirely on a commander’s thoughtful battle preparation. The Plattsburgh harbor was on a bay sheltered from northerly winds. Macdonough anchored his vessels in a line broadsides out, in a narrow portion of the bay. The details of his positioning gave him two crucial advantages. The first is that he set kedges, or auxiliary anchors, with undersea cabling so crews could winch their ships around to change broadsides without losing anchorage. The second is that the narrowness of his anchorage site helped to neutralize the Confiance’s great advantage in heavy long guns. (Macdonough implicitly counted on the British penchant for attacking without regard to tactical details.)

The Ascent of Money: A Financial History of the World by Niall Ferguson

Admiral Zheng, Alan Greenspan, An Inconvenient Truth, Andrei Shleifer, Asian financial crisis, asset allocation, asset-backed security, Atahualpa, bank run, banking crisis, banks create money, Bear Stearns, Black Monday: stock market crash in 1987, Black Swan, Black-Scholes formula, Bonfire of the Vanities, Bretton Woods, BRICs, British Empire, business cycle, capital asset pricing model, capital controls, Carmen Reinhart, Cass Sunstein, central bank independence, classic study, collateralized debt obligation, colonial exploitation, commoditize, Corn Laws, corporate governance, creative destruction, credit crunch, Credit Default Swap, credit default swaps / collateralized debt obligations, currency manipulation / currency intervention, currency peg, Daniel Kahneman / Amos Tversky, deglobalization, diversification, diversified portfolio, double entry bookkeeping, Edmond Halley, Edward Glaeser, Edward Lloyd's coffeehouse, equity risk premium, financial engineering, financial innovation, financial intermediation, fixed income, floating exchange rates, Fractional reserve banking, Francisco Pizarro, full employment, Future Shock, German hyperinflation, Greenspan put, Herman Kahn, Hernando de Soto, high net worth, hindsight bias, Home mortgage interest deduction, Hyman Minsky, income inequality, information asymmetry, interest rate swap, Intergovernmental Panel on Climate Change (IPCC), Isaac Newton, iterative process, James Carville said: "I would like to be reincarnated as the bond market. You can intimidate everybody.", John Meriwether, joint-stock company, joint-stock limited liability company, Joseph Schumpeter, junk bonds, Kenneth Arrow, Kenneth Rogoff, knowledge economy, labour mobility, Landlord’s Game, liberal capitalism, London Interbank Offered Rate, Long Term Capital Management, low interest rates, market bubble, market fundamentalism, means of production, Mikhail Gorbachev, Modern Monetary Theory, Money creation, money market fund, money: store of value / unit of account / medium of exchange, moral hazard, mortgage debt, mortgage tax deduction, Myron Scholes, Naomi Klein, National Debt Clock, negative equity, Nelson Mandela, Nick Bostrom, Nick Leeson, Northern Rock, Parag Khanna, pension reform, price anchoring, price stability, principal–agent problem, probability theory / Blaise Pascal / Pierre de Fermat, profit motive, quantitative hedge fund, RAND corporation, random walk, rent control, rent-seeking, reserve currency, Richard Thaler, risk free rate, Robert Shiller, rolling blackouts, Ronald Reagan, Savings and loan crisis, savings glut, seigniorage, short selling, Silicon Valley, South Sea Bubble, sovereign wealth fund, spice trade, stocks for the long run, structural adjustment programs, subprime mortgage crisis, tail risk, technology bubble, The Wealth of Nations by Adam Smith, The Wisdom of Crowds, Thomas Bayes, Thomas Malthus, Thorstein Veblen, tontine, too big to fail, transaction costs, two and twenty, undersea cable, value at risk, W. E. B. Du Bois, Washington Consensus, Yom Kippur War

In the three decades before 1914, trade in goods reached almost as large a proportion of global output as in the past thirty years.10 In a world of less regulated borders, international migration was almost certainly larger relative to world population; more than 14 per cent of the US population was foreign born in 1910 compared with less than 12 per cent in 2003.11 Although, in gross terms, stocks of international capital were larger in relation to global GDP during the 1990s than they were a century ago, in net terms the amounts invested abroad - particularly by rich countries in poor countries - were much larger in the earlier period.12 Over a century ago, enterprising businessmen in Europe and North America could see that there were enticing opportunities throughout Asia. By the middle of the nineteenth century, the key technologies of the industrial revolution could be transferred anywhere. Communication lags had been dramatically reduced thanks to the laying of an international undersea cable network. Capital was abundantly available and, as we shall see, British investors were more than ready to risk their money in remote countries. Equipment was affordable, energy available and labour so abundant that manufacturing textiles in China or India ought to have been a hugely profitable line of business.13 Yet, despite the investment of over a billion pounds of Western funds, the promise of Victorian globalization went largely unfulfilled in most of Asia, leaving a legacy of bitterness towards what is still remembered to this day as colonial exploitation.

The Code Book: The Science of Secrecy From Ancient Egypt to Quantum Cryptography by Simon Singh

Bletchley Park, Charles Babbage, Donald Davies, friendly fire, information security, Leo Hollis, Mikhail Gorbachev, old-boy network, operational security, quantum cryptography, Ronald Reagan, Schrödinger's Cat, Simon Singh, Turing machine, unbiased observer, undersea cable, Zimmermann PGP

Zimmermann Zimmermann had to encrypt his telegram because Germany was aware that the Allies were intercepting all its transatlantic communications, a consequence of Britain’s first offensive action of the war. Before dawn on the first day of the First World War, the British ship Telconia approached the German coast under cover of darkness, dropped anchor, and hauled up a clutch of undersea cables. These were Germany’s transatlantic cables—its communication links to the rest of the world. By the time the sun had risen, they had been severed. This act of sabotage was aimed at destroying Germany’s most secure means of communication, thereby forcing German messages to be sent via insecure radio links or via cables owned by other countries.

The Empire Project: The Rise and Fall of the British World-System, 1830–1970 by John Darwin

anti-communist, banking crisis, Bretton Woods, British Empire, capital controls, classic study, cognitive bias, colonial rule, Corn Laws, disinformation, European colonialism, floating exchange rates, full employment, imperial preference, Joseph Schumpeter, Khartoum Gordon, Kickstarter, labour mobility, land tenure, liberal capitalism, liquidationism / Banker’s doctrine / the Treasury view, Mahatma Gandhi, Monroe Doctrine, new economy, New Urbanism, open economy, railway mania, reserve currency, Right to Buy, rising living standards, scientific management, Scientific racism, South China Sea, Suez canal 1869, Suez crisis 1956, tacit knowledge, the market place, The Wealth of Nations by Adam Smith, trade route, transaction costs, transcontinental railway, undersea cable

In the tropical empire (which British migrants avoided) they supplied much of the labour and business expertise to promote commercial expansion. Lastly, the spheres of British expansion were progressively linked by a complex system of communication. From the 1840s onwards, this was provided by subsidised mail services, telegraph wires, undersea cables, an expanding rail network, fast passenger steamers and (in the twentieth century) imperial air routes. They catered for, and stimulated, the growing volume and frequency of the traffic in news, information, private correspondence, personnel and ideas that flowed between Britain and other parts of the system, as well as between those constituent parts.

…

It secured Britain's claims on a huge range of assets, most of them safely remote from the great powers in Europe. It helped to sustain the flow of migration, Britain's demographic imperialism. Last, but not least, it preserved Britain's lead in communications technology, especially the telegraph and undersea cables that made London (and Britain) the information hub of the world. But Britain's position in the new world economy could not be taken for granted. It required adaptations of practice and outlook that were just as far-reaching as those that tested political nerves in Whitehall and Westminster. The scale of commercial activity as ‘new regions’ were drawn into the ‘high-pressure zone’ of the Atlantic economy; the intricate network of multilateral payments on which trade growth depended; the demand for finance, both credit and capital; and the competitive threat posed by new industrial producers: all posed new challenges to the commercial apparatus that the mid-Victorians had fashioned.

Jihad vs. McWorld: Terrorism's Challenge to Democracy by Benjamin Barber

airport security, Alvin Toffler, anti-communist, Apple's 1984 Super Bowl advert, Ayatollah Khomeini, Berlin Wall, borderless world, Bretton Woods, British Empire, classic study, computer age, Corn Laws, Corrections Corporation of America, David Brooks, deindustrialization, Deng Xiaoping, digital map, export processing zone, Fall of the Berlin Wall, Francis Fukuyama: the end of history, full employment, George Gilder, global village, invisible hand, It's morning again in America, Joan Didion, Kevin Kelly, laissez-faire capitalism, late capitalism, Live Aid, market fundamentalism, Marshall McLuhan, minimum wage unemployment, new economy, Norbert Wiener, North Sea oil, off-the-grid, pirate software, Plato's cave, postnationalism / post nation state, profit motive, race to the bottom, Right to Buy, road to serfdom, Ronald Reagan, The Wealth of Nations by Adam Smith, Thomas L Friedman, undersea cable, vertical integration, young professional, zero-sum game

Ignacio Ramonet argues that in the global economy neither capital nor work nor material is the determining factor, but rather the “optimal relationship between these three,” which pushes us into the world of information, communication, and administration where traditional nation-states can exert little control and are bound to feel more and more uncomfortable.13 Robert Kuttner reports that the state-of-the-art handle for the postindustrial company—which clearly is also the post—nation-state company—is “the virtual corporation” where “the company is no longer a physical entity with a stable mission or location, but a shifting set of temporary relationships connected by computer network, phone and fax.”14 McWorld is a kind of virtual reality, created by invisible but omnipotent high-tech information networks and fluid transnational economic markets, so the virtual corporation is not just a provocative turn of phrase.15 Without even trying, reporter Julie Edelson Halpert gives it concrete meaning in the portrait she draws of Ford Motor Company’s Mondeo project: Seeking to shave months and millions of dollars from car design, Ford has consolidated management of its European, North American and Asian design operations into a single international network using powerful work stations based on Silicon Graphics Inc. technology linked by Ethernet networking software. …The Ford system … was brought under a single “electronic roof” … based in Dearborn, Michigan. The other main sites on the network are in Dunton, England; Cologne, Germany; Turin, Italy; Valencia, California; Hiroshima, Japan; and Melbourne, Australia. The circuits—satellite links, undersea cables and land lines—are purchased from telecommunications carriers.16 The virtual corporation also exists in the labor market as an employer of “virtual” rather than actual laborers. The ideal virtual laborer is a robot: an interactive, “intelligent,” fully programmed worker capable of working twenty-four hours a day with no sustenance and minimal upkeep.

The Death of Money: The Coming Collapse of the International Monetary System by James Rickards

"World Economic Forum" Davos, Affordable Care Act / Obamacare, Alan Greenspan, Asian financial crisis, asset allocation, Ayatollah Khomeini, bank run, banking crisis, Bear Stearns, Ben Bernanke: helicopter money, bitcoin, Black Monday: stock market crash in 1987, Black Swan, Boeing 747, Bretton Woods, BRICs, business climate, business cycle, buy and hold, capital controls, Carmen Reinhart, central bank independence, centre right, collateralized debt obligation, collective bargaining, complexity theory, computer age, credit crunch, currency peg, David Graeber, debt deflation, Deng Xiaoping, diversification, Dr. Strangelove, Edward Snowden, eurozone crisis, fiat currency, financial engineering, financial innovation, financial intermediation, financial repression, fixed income, Flash crash, floating exchange rates, forward guidance, G4S, George Akerlof, global macro, global reserve currency, global supply chain, Goodhart's law, Growth in a Time of Debt, guns versus butter model, Herman Kahn, high-speed rail, income inequality, inflation targeting, information asymmetry, invisible hand, jitney, John Meriwether, junk bonds, Kenneth Rogoff, labor-force participation, Lao Tzu, liquidationism / Banker’s doctrine / the Treasury view, liquidity trap, Long Term Capital Management, low interest rates, mandelbrot fractal, margin call, market bubble, market clearing, market design, megaproject, Modern Monetary Theory, Money creation, money market fund, money: store of value / unit of account / medium of exchange, mutually assured destruction, Nixon triggered the end of the Bretton Woods system, obamacare, offshore financial centre, oil shale / tar sands, open economy, operational security, plutocrats, Ponzi scheme, power law, price stability, public intellectual, quantitative easing, RAND corporation, reserve currency, risk-adjusted returns, Rod Stewart played at Stephen Schwarzman birthday party, Ronald Reagan, Satoshi Nakamoto, Silicon Valley, Silicon Valley startup, Skype, Solyndra, sovereign wealth fund, special drawing rights, Stuxnet, The Market for Lemons, Thomas Kuhn: the structure of scientific revolutions, Thomas L Friedman, too big to fail, trade route, undersea cable, uranium enrichment, Washington Consensus, working-age population, yield curve

In a surprising coda to the revelations of U.S. spying on allies emerging from defector Edward Snowden, Brazil announced plans in September 2013 to build a twenty-thousand-mile undersea fiber optic cable network from Fortaleza, Brazil, to Vladivostok, Russia, with links in Cape Town, South Africa, Chennai, India, and Shantou, China, to be completed by 2015. This system is tantamount to a BRICS Internet intended to be free from U.S. surveillance. The United States has long had excellent capability in tapping into undersea cables, so the actual security of the new system may be problematic. Nevertheless, the proprietary nature of this system could easily be adapted to include a BRICS interbank payments system, which would facilitate the use of any BRICS-sponsored alternatives to dollar payments. In addition to the regular meetings of BRICS leaders, a large number of ancillary and shadow institutions have sprung up around the BRICS, including the BRICS Think Tanks Council, the BRICS Business Council, and a BRICS virtual secretariat, among others.

Empire: How Britain Made the Modern World by Niall Ferguson

British Empire, Cape to Cairo, colonial rule, Corn Laws, death from overwork, European colonialism, imperial preference, income per capita, information security, John Harrison: Longitude, joint-stock company, Khartoum Gordon, Khyber Pass, land reform, land tenure, liberal capitalism, Livingstone, I presume, low interest rates, Mahatma Gandhi, mass immigration, military-industrial complex, night-watchman state, Panopticon Jeremy Bentham, profit motive, Scramble for Africa, spice trade, Suez canal 1869, Suez crisis 1956, The inhabitant of London could order by telephone, sipping his morning tea in bed, the various products of the whole earth, the new new thing, The Wealth of Nations by Adam Smith, Thomas Malthus, three-masted sailing ship, trade route, transatlantic slave trade, undersea cable, union organizing, zero-sum game

By the late 1840s it was clear that the telegraph would revolutionize overland communications; by the 1850s construction in India was sufficiently advanced for the telegraph to play a decisive part in suppressing the Mutiny.* However, the crucial development from the point of view of imperial rule was the construction of durable undersea cables. Significantly, it was an imperial product – a rubber-like substance from Malaya called gutta-percha – that solved the problem, allowing the first cross-Channel cable to be laid in 1851 and the first transatlantic cable to follow fifteen years later. When the Anglo-American Telegraph Company’s cable finally reached the American coast on 27 July 1866, having been successfully unrolled and dropped along the ocean floor by Isambard Kingdom Brunel’s mighty Great Eastern, it was plainly the dawn of a new era.

Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia by Anthony M. Townsend

1960s counterculture, 4chan, A Pattern Language, Adam Curtis, air gap, Airbnb, Amazon Web Services, anti-communist, Apple II, Bay Area Rapid Transit, Big Tech, bike sharing, Boeing 747, Burning Man, business process, call centre, carbon footprint, charter city, chief data officer, clean tech, clean water, cloud computing, company town, computer age, congestion charging, congestion pricing, connected car, crack epidemic, crowdsourcing, DARPA: Urban Challenge, data acquisition, Deng Xiaoping, digital divide, digital map, Donald Davies, East Village, Edward Glaeser, Evgeny Morozov, food desert, game design, garden city movement, General Motors Futurama, gentrification, Geoffrey West, Santa Fe Institute, George Gilder, ghettoisation, global supply chain, Grace Hopper, Haight Ashbury, Hedy Lamarr / George Antheil, Herman Kahn, hive mind, Howard Rheingold, interchangeable parts, Internet Archive, Internet of things, Jacquard loom, Jane Jacobs, Jevons paradox, jitney, John Snow's cholera map, Joi Ito, Khan Academy, Kibera, Kickstarter, knowledge worker, Lewis Mumford, load shedding, lolcat, M-Pesa, machine readable, Mark Zuckerberg, megacity, megaproject, messenger bag, mobile money, mutually assured destruction, new economy, New Urbanism, Norbert Wiener, Occupy movement, off grid, One Laptop per Child (OLPC), openstreetmap, packet switching, PalmPilot, Panopticon Jeremy Bentham, Parag Khanna, patent troll, Pearl River Delta, place-making, planetary scale, popular electronics, power law, RFC: Request For Comment, RFID, ride hailing / ride sharing, Robert Gordon, scientific management, self-driving car, sharing economy, Shenzhen special economic zone , Silicon Valley, SimCity, Skype, smart cities, smart grid, smart meter, social graph, social software, social web, SpaceShipOne, special economic zone, Steve Jobs, Steve Wozniak, Stuxnet, supply-chain management, technoutopianism, Ted Kaczynski, telepresence, The Death and Life of Great American Cities, too big to fail, trade route, Twitter Arab Spring, Tyler Cowen, Tyler Cowen: Great Stagnation, undersea cable, Upton Sinclair, uranium enrichment, urban decay, urban planning, urban renewal, Vannevar Bush, working poor, working-age population, X Prize, Y2K, zero day, Zipcar

Founded in 1984 by husband and wife Len Bosack and Sandy Lerner, who built Stanford University’s campus network in the early 1980s, Cisco has grown into the world’s leading supplier for the sophisticated switches and routers that power the Internet. Cisco’s products not only push bits around offices, schools, and homes, but also sling them back and forth across undersea cables that link continents. It’s one of Silicon Valley’s largest and most-watched bellwethers. For a brief period in March 2000, at the height of the telecom bubble, it was the most valuable company in the world. But with size comes stagnation. Finding growth opportunities has become a constant struggle for Cisco, and to make a dent on the bottom line it needs to have billion-dollar payouts.

The Turing Option by Harry Harrison, Marvin Minsky

industrial robot, pattern recognition, Silicon Valley, telepresence, telerobotics, theory of mind, Turing test, undersea cable, VTOL

He tapped the telephone on his belt. "I have my own number for this. About most of the time it rings wherever I am in the continental United States." "Is it a satellite link?" "No, satellite connections are too slow for most uses— particularly telepresence. Everything is fiber optics now— even the undersea cables. Cheap and fast. With plenty of room for communication with eight thousand megahertz band-width capacity available everywhere—and all of it two-way." Brian nodded. "I get your drift, Ben. What you're saying is that there is very little chance that I had a local mechanical backup. It was undoubtedly an electronic one.

Mapping Mars: Science, Imagination and the Birth of a World by Oliver Morton

Apollo 11, Charles Babbage, classic study, Colonization of Mars, computer age, double entry bookkeeping, Dr. Strangelove, Kim Stanley Robinson, Mars Rover, Mars Society, Menlo Park, Mercator projection, Neil Armstrong, nuclear winter, planetary scale, RAND corporation, Richard Feynman, sexual politics, the scientific method, trade route, undersea cable, V2 rocket, Works Progress Administration

The once imperial cityscape is woven from threads that stretch throughout the world. Across the river to the east sits the squat black-glass bulk of Reuters, information from around the globe splashing into its rooftop dishes. Upstream and on the near side sit the long, low workshops where for more than a century men have made undersea cables to tie the continents together. New skyscrapers devoted to global businesses sit in the redeveloped heart of the docks that used to handle the lion’s share of the world’s sea trade. Within the park itself there are plants from every continent except Antarctica. At its foot sits the old naval college, where generations of Britannia’s officers, my late father included, learned to rule the waves.

Ways of Being: Beyond Human Intelligence by James Bridle

Ada Lovelace, Airbnb, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Anthropocene, Any sufficiently advanced technology is indistinguishable from magic, autonomous vehicles, behavioural economics, Benoit Mandelbrot, Berlin Wall, Big Tech, Black Lives Matter, blockchain, Californian Ideology, Cambridge Analytica, carbon tax, Charles Babbage, cloud computing, coastline paradox / Richardson effect, Computing Machinery and Intelligence, corporate personhood, COVID-19, cryptocurrency, DeepMind, Donald Trump, Douglas Hofstadter, Elon Musk, experimental subject, factory automation, fake news, friendly AI, gig economy, global pandemic, Gödel, Escher, Bach, impulse control, James Bridle, James Webb Space Telescope, John von Neumann, Kickstarter, Kim Stanley Robinson, language acquisition, life extension, mandelbrot fractal, Marshall McLuhan, microbiome, music of the spheres, negative emissions, Nick Bostrom, Norbert Wiener, paperclip maximiser, pattern recognition, peer-to-peer, planetary scale, RAND corporation, random walk, recommendation engine, self-driving car, SETI@home, shareholder value, Silicon Valley, Silicon Valley ideology, speech recognition, statistical model, surveillance capitalism, techno-determinism, technological determinism, technoutopianism, the long tail, the scientific method, The Soul of a New Machine, theory of mind, traveling salesman, trolley problem, Turing complete, Turing machine, Turing test, UNCLOS, undersea cable, urban planning, Von Neumann architecture, wikimedia commons, zero-sum game

It was the same sensation I experienced when beginning to understand, and bring into view, the infrastructure of the internet: the vast, planet-spanning network of cables, wires, machines and electromagnetic signals which sustains and regulates humanity today. These microprocessors and data centres, undersea cables and wireless transmissions are our own mycorrhizal network, interpenetrating everyday life, managing our supplies and demands, disturbing the climate and touching upon our own skin – even, in the case of cochlear implants, pacemakers, and perhaps soon neural links, breaching it. It speaks volumes about our lives today that when we hear about networks and communication it is these artificial networks which we think of first.

Debt of Honor by Tom Clancy

airport security, banking crisis, Berlin Wall, Boeing 747, book value, buttonwood tree, classic study, complexity theory, cuban missile crisis, defense in depth, disinformation, Easter island, job satisfaction, Kwajalein Atoll, low earth orbit, margin call, New Journalism, oil shock, Silicon Valley, tulip mania, undersea cable

I want the COs to drill their people up. You're never ready enough, Ron. Never." "True," the civilian conceded. He'd come over with SOSUS printouts to demonstrate that all known submarine contacts were off the screen. Two hydrophone arrays that were operated from the island of Guam were no longer available. Though connected by undersea cable to the rest of the network, they'd evidently been turned off by the monitoring facility on Guam, and nobody at Pearl had yet been able to trick them back on. The good news was that a backup array off Samar in the Philippines was still operating, but it could not detect the Japanese SSKs shown by satellite to be replenishing off Agana.

…

"I hope it's not about Wall Street," Jack replied in as unguarded a voice as he could muster. "No, it's about Guam. Why are the phone lines out?" "Bob, did you ask the phone company that?" Ryan tried. "Yeah. They say there was a storm that took a lot of lines down. Except for a couple of things. One, there wasn't any storm. Two, there's an undersea cable and a satellite link. Three, a week is a long time. What's going on?" the reporter asked. "How many people are asking?" "Right now, just me and a TV station in Little Rock that put a request up on the AP wire. Another thirty minutes and it's going to be a lot more. What gives? Some sort of—" "Bob, why don't you come on down here," Ryan suggested.

Pacific: Silicon Chips and Surfboards, Coral Reefs and Atom Bombs, Brutal Dictators, Fading Empires, and the Coming Collision of the World's Superpowers by Simon Winchester

9 dash line, Albert Einstein, Boeing 747, BRICs, British Empire, California gold rush, classic study, colonial rule, company town, Deng Xiaoping, desegregation, Easter island, Frank Gehry, Intergovernmental Panel on Climate Change (IPCC), Korean Air Lines Flight 007, Kwajalein Atoll, land tenure, Larry Ellison, Loma Prieta earthquake, Maui Hawaii, Monroe Doctrine, ocean acidification, oil shock, polynesian navigation, Ralph Waldo Emerson, RAND corporation, Ronald Reagan, Seymour Hersh, Silicon Valley, South China Sea, The Day the Music Died, three-masted sailing ship, trade route, transcontinental railway, UNCLOS, UNCLOS, undersea cable, uranium enrichment

The system has numberless branches and offshoots, and were its waters to be drained from the ocean and the planet dried out, its ridges would look like a web of fibers somehow helping to hold the earth together, like the stitches on a baseball or the sutures on a skull. Its existence was confirmed only recently, although there were suggestions as early as Victorian times that lines of unexpected shallows were to be found out in the mid-ocean deeps. HMS Challenger, surveying the Atlantic in 1872 to find the optimal route for an undersea cable, found that depths in the middle of the ocean were many thousands of feet less than expected, and a century later, German oceanographers noticed that this same upwelling continued around the African coast right up past Madagascar. Yet Albert Bumstead’s classic 1936 National Geographic map of the Pacific (the same one used, as mentioned, by Colonel Charles Bonesteel III to divide postwar Korea) still gives no clue that anything similarly significant had yet been noticed on the Pacific side of the world.

Confessions of a Wall Street Analyst: A True Story of Inside Information and Corruption in the Stock Market by Daniel Reingold, Jennifer Reingold

Alan Greenspan, AOL-Time Warner, barriers to entry, Bear Stearns, Berlin Wall, corporate governance, deal flow, estate planning, Fall of the Berlin Wall, fixed income, George Gilder, high net worth, informal economy, junk bonds, margin call, Mary Meeker, mass immigration, Michael Milken, new economy, pets.com, Robert Metcalfe, rolodex, Saturday Night Live, shareholder value, short selling, Silicon Valley, stem cell, Telecommunications Act of 1996, thinkpad, traveling salesman, undersea cable, UUNET

Gary alone cashed in over $700 million over the course of four years, second only to Phil Anschutz—who pocketed $1.9 billion on Qwest—as the industry’s champion roulette player.1 But the bankruptcy filing was particularly tragic in that the promise of this new world had been exposed as nothing more than hype. And even those of us who had questioned the hype had not questioned it enough. It was this century’s equivalent of the railroad industry boom and bust in the late 1880s. Customers might use all of those undersea cables one day. But not anytime soon. Global’s collapse came in the shadow of Enron’s sudden meltdown the previous December, which had taken the entire financial world by surprise—the auditors, the regulators, and the Street. Now everyone was scrambling to figure out who had left the barn door open and how.

Civilization: The West and the Rest by Niall Ferguson

Admiral Zheng, agricultural Revolution, Albert Einstein, Andrei Shleifer, Atahualpa, Ayatollah Khomeini, Berlin Wall, BRICs, British Empire, business cycle, clean water, collective bargaining, colonial rule, conceptual framework, Copley Medal, corporate governance, creative destruction, credit crunch, David Ricardo: comparative advantage, Dean Kamen, delayed gratification, Deng Xiaoping, discovery of the americas, Dissolution of the Soviet Union, Easter island, European colonialism, Fall of the Berlin Wall, financial engineering, Francisco Pizarro, full employment, Great Leap Forward, Gregor Mendel, guns versus butter model, Hans Lippershey, haute couture, Hernando de Soto, income inequality, invention of movable type, invisible hand, Isaac Newton, James Hargreaves, James Watt: steam engine, John Harrison: Longitude, joint-stock company, Joseph Schumpeter, Kickstarter, Kitchen Debate, land reform, land tenure, liberal capitalism, Louis Pasteur, Mahatma Gandhi, market bubble, Martin Wolf, mass immigration, means of production, megacity, Mikhail Gorbachev, new economy, Pearl River Delta, Pierre-Simon Laplace, power law, probability theory / Blaise Pascal / Pierre de Fermat, profit maximization, purchasing power parity, quantitative easing, rent-seeking, reserve currency, retail therapy, road to serfdom, Ronald Reagan, savings glut, Scramble for Africa, Silicon Valley, South China Sea, sovereign wealth fund, special economic zone, spice trade, spinning jenny, Steve Jobs, Steven Pinker, subprime mortgage crisis, Suez canal 1869, Suez crisis 1956, The Great Moderation, the market place, the scientific method, The Wealth of Nations by Adam Smith, Thomas Kuhn: the structure of scientific revolutions, Thomas Malthus, Thorstein Veblen, total factor productivity, trade route, transaction costs, transatlantic slave trade, undersea cable, upwardly mobile, uranium enrichment, wage slave, Washington Consensus, women in the workforce, work culture , World Values Survey

The opening of the Suez Canal (1869) and the Panama Canal (1914) shrank the world still further, the former reducing the distance of the London–Bombay route by more than two-fifths, the latter cutting the cost of shipping from the East to the West Coast of the United States by a third.48 By the late 1860s, thanks to the introduction of gutta-percha coating, undersea cables could be laid and telegrams sent from London to Bombay or to Halifax.49 News of the Indian Mutiny had taken forty-six days to reach London in 1857, travelling at an effective speed of 3.8 miles an hour. News of the huge Nobi earthquake in Japan in 1891 took a single day, travelling at 246 miles an hour, sixty-five times faster.50 Labour flowed across borders as never before.

Exploding the Phone: The Untold Story of the Teenagers and Outlaws Who Hacked Ma Bell by Phil Lapsley

air freight, Apple II, Bill Gates: Altair 8800, Bob Noyce, card file, classic study, cuban missile crisis, dumpster diving, Garrett Hardin, Hush-A-Phone, index card, Jason Scott: textfiles.com, John Markoff, Menlo Park, military-industrial complex, Neal Stephenson, popular electronics, Richard Feynman, Saturday Night Live, Silicon Valley, Steve Jobs, Steve Wozniak, Steven Levy, The Home Computer Revolution, the new new thing, the scientific method, Tragedy of the Commons, undersea cable, urban renewal, wikimedia commons

.: See Fettgather, Jim Whistle/whistling 2111 conference 163–164, 202, 206 Cap’n Crunch 154–155, 166, 202, 209, 326 Draper, John 153, 165–166, 209 Engressia, Joe 122–126, 130, 132, 173, 326, 335 Pyne, Charlie 77 Trips 154 Wozniak, Steve 218 See also: Davy Crockett Cat and Canary Bird Call Flute, Tonette Flute White House 800 number 229, 268, 280, 385–386, 393–394 Hotline 254 Secret telephone system 127–128 See also: Ford, Gerald; Nixon, Richard White Plains Tandem 2 175–176, 203, 207 WHRB (Harvard student radio station) 68–70, 72–73, 77 Wiretapping Ashley-Gravitt scandal 303–304, 306 Bookies 101 Engressia 127–128, 131 FBI 249–261, 272, 278 Legality 109–115, 288 Soviet undersea cables 182 Verification/blue box 249–261 White House 127–128, 229 See also: Greenstar; laws, 47 USC 605 Wink 54, 175, 371 Woolworth’s 29–30 Wozniak, Steve 218–225, 229, 243–244, 281, 309, 312–313, 333, 335 XBT: See crossbar tandem YIPL: See Youth International Party Line Yippies: See Youth International Party Young Engineering 66 Young, Shane 326 Youth International Party 185–186, 197, 214, 257 Youth International Party Line 185–186, 192–193, 197–200, 213–214, 230, 238, 240, 330–331 See also: Technological American Party Zzzzyzzerrific Funline 328 The Playground Table of Contents Cover Halftitle Title Copyright Dedication Contents Epigraph Foreword A Note on Names and Tenses Chapter One Chapter Two Chapter Three Chapter Four Chapter Five Chapter Six Chapter Seven Chapter Eight Chapter Nine Chapter Ten Chapter Eleven Photo Insert Chapter Twelve Chapter Thirteen Chapter Fourteen Chapter Fifteen Chapter Sixteen Chapter Seventeen Chapter Eighteen Chapter Nineteen Chapter Twenty Chapter Twenty-one Epilogue Sources and Notes Acknowledgments Index The Playground

Red November: Inside the Secret U.S.-Soviet Submarine War by W. Craig Reed

Albert Einstein, Any sufficiently advanced technology is indistinguishable from magic, cable laying ship, centre right, cuban missile crisis, en.wikipedia.org, fixed-gear, nuclear winter, operation paperclip, Ronald Reagan, Saturday Night Live, Silicon Valley, undersea cable, upwardly mobile

In October 1971, the upgraded Halibut, carrying a stern-mounted decompression hyperbaric chamber and loaded with deep-sea saturation divers, cruised toward the Soviet Union with Commander “Smiling Jack” McNish in charge. The Soviet communications cable they sought ran thousands of miles under the ocean deep in the Sea of Okhotsk, a large body of water, surrounded by land, that from the air looked like a massive Russian lake. The undersea cable ran from the Petropavlovsk submarine base and missile-testing facilities on the Kamchatka Peninsula, down to Vladivostok, headquarters of the Soviet “Far East” Pacific Fleet. Commander McNish successfully located the cable’s approximate position via Halibut’s periscope by finding a small sign posted on a beach warning passersby to avoid running into a buried cable.

Chaos Monkeys: Obscene Fortune and Random Failure in Silicon Valley by Antonio Garcia Martinez

Airbnb, airport security, always be closing, Amazon Web Services, Big Tech, Burning Man, business logic, Celtic Tiger, centralized clearinghouse, cognitive dissonance, collective bargaining, content marketing, corporate governance, Credit Default Swap, crowdsourcing, data science, deal flow, death of newspapers, disruptive innovation, Dr. Strangelove, drone strike, drop ship, El Camino Real, Elon Musk, Emanuel Derman, Fairchild Semiconductor, fake it until you make it, financial engineering, financial independence, Gary Kildall, global supply chain, Goldman Sachs: Vampire Squid, Hacker News, hive mind, How many piano tuners are there in Chicago?, income inequality, industrial research laboratory, information asymmetry, information security, interest rate swap, intermodal, Jeff Bezos, Kickstarter, Malcom McLean invented shipping containers, Marc Andreessen, Mark Zuckerberg, Maui Hawaii, means of production, Menlo Park, messenger bag, minimum viable product, MITM: man-in-the-middle, move fast and break things, Neal Stephenson, Network effects, orbital mechanics / astrodynamics, Paul Graham, performance metric, Peter Thiel, Ponzi scheme, pre–internet, public intellectual, Ralph Waldo Emerson, random walk, Reminiscences of a Stock Operator, Ruby on Rails, Salesforce, Sam Altman, Sand Hill Road, Scientific racism, second-price auction, self-driving car, Sheryl Sandberg, Silicon Valley, Silicon Valley startup, Skype, Snapchat, social graph, Social Justice Warrior, social web, Socratic dialogue, source of truth, Steve Jobs, tech worker, telemarketer, the long tail, undersea cable, urban renewal, Y Combinator, zero-sum game, éminence grise

Every time you load a new page in Facebook, not to mention most of the Internet, optical signals crisscross the globe to hundreds of waiting machines, all announcing, like some phalanx of royal trumpeters, your impending arrival. So what happens to those globe-spanning bid requests, assuming they don’t drown in their trans-Atlantic undersea cable? The people listening for your presence in billions of such requests per day are known as “demand-side platforms” (DSPs), and they’re the stockbrokers of this real-time media world, working in the employ of the advertiser or agency who wants to sell you something.* The DSP quickly unpacks the bid request, and queries its data for anything it knows about you: sites you’ve browsed, shopping carts you’ve abandoned, that airfare quote you got and never acted upon.

Countdown to Zero Day: Stuxnet and the Launch of the World's First Digital Weapon by Kim Zetter

air gap, Ayatollah Khomeini, Brian Krebs, crowdsourcing, data acquisition, Doomsday Clock, drone strike, Edward Snowden, facts on the ground, false flag, Firefox, friendly fire, Google Earth, information retrieval, information security, John Markoff, Julian Assange, Kickstarter, Loma Prieta earthquake, machine readable, Maui Hawaii, military-industrial complex, MITM: man-in-the-middle, Morris worm, pre–internet, RAND corporation, rolling blackouts, Silicon Valley, skunkworks, smart grid, smart meter, South China Sea, Stuxnet, Timothy McVeigh, two and twenty, undersea cable, unit 8200, uranium enrichment, Vladimir Vetrov: Farewell Dossier, WikiLeaks, Y2K, zero day

To track Hassan Ghul, an associate of Osama bin Laden who was killed in a drone strike in 2012, the NSA deployed “an arsenal of cyber-espionage tools” to seize control of laptops, siphon audio files, and track radio transmissions—all to determine where Ghul might “bed down” at night, according to Snowden documents obtained by the Washington Post.30 And since 2001, the NSA has also penetrated a vast array of systems used by al-Qaeda associates in Yemen, Africa, and elsewhere to collect intelligence it can’t otherwise obtain through bulk-data collection programs from internet companies like Google and Yahoo or from taps of undersea cables and internet nodes. Terrorism suspects aren’t the NSA’s only targets, however. Operations against nation-state adversaries have exploded in recent years as well. In 2011, the NSA mounted 231 offensive cyber operations against other countries, according to the documents, three-fourths of which focused on “top-priority” targets like Iran, Russia, China, and North Korea.

Dark Mirror: Edward Snowden and the Surveillance State by Barton Gellman

4chan, A Declaration of the Independence of Cyberspace, Aaron Swartz, active measures, air gap, Anton Chekhov, Big Tech, bitcoin, Cass Sunstein, Citizen Lab, cloud computing, corporate governance, crowdsourcing, data acquisition, data science, Debian, desegregation, Donald Trump, Edward Snowden, end-to-end encryption, evil maid attack, financial independence, Firefox, GnuPG, Google Hangouts, housing justice, informal economy, information security, Jacob Appelbaum, job automation, John Perry Barlow, Julian Assange, Ken Thompson, Laura Poitras, MITM: man-in-the-middle, national security letter, off-the-grid, operational security, planetary scale, private military company, ransomware, Reflections on Trusting Trust, Robert Gordon, Robert Hanssen: Double agent, rolodex, Ronald Reagan, Saturday Night Live, seminal paper, Seymour Hersh, Silicon Valley, Skype, social graph, standardized shipping container, Steven Levy, TED Talk, telepresence, the long tail, undersea cable, Wayback Machine, web of trust, WikiLeaks, zero day, Zimmermann PGP

Internet Companies in Broad Secret Program,” Washington Post, June 6, 2013, http://wapo.st/1LcAw6p, archived at https://archive.is/cYyFe. piece of the backbone: What is known as the backbone of the global communications network is a complex infrastructure with major components including trunk lines of fiber optic cable, high-capacity switches known as core routers, cable landing stations where undersea cables join terrestrial networks, and Internet Exchange Points. The majority of worldwide telephone and internet traffic passes to, from, or through U.S. infrastructure. $394 million: FY 2013 Congressional Budget Justification, Volume 1, National Intelligence Program Summary, on file with author.

Lonely Planet Iceland (Travel Guide) by Lonely Planet, Carolyn Bain, Alexis Averbuck

Airbnb, banking crisis, car-free, carbon footprint, cashless society, centre right, DeepMind, European colonialism, Eyjafjallajökull, food miles, Kickstarter, low cost airline, megaproject, Mikhail Gorbachev, New Urbanism, post-work, presumed consent, ride hailing / ride sharing, Ronald Reagan, undersea cable

An Energetic Future Tourism now exceeds fishing as Iceland's dominant industry, and has helped Iceland bounce back from its banking crisis. But to ensure that prosperity continues, even if the tourism boom comes to a crashing halt, Iceland is shoring up its position as a green-energy superpower, looking to export its know-how (and quite possibly its actual energy, transmitted via undersea cables) to foreign shores. It's wooing more big-business energy users, to try to convince them to set up shop (large aluminium smelters are already here for the cheap, abundant power). Public and private enterprises are exploring the options for producing 'green fuels' such as biomethane and biodiesel.

In the Plex: How Google Thinks, Works, and Shapes Our Lives by Steven Levy

"World Economic Forum" Davos, 23andMe, AltaVista, Andy Rubin, Anne Wojcicki, Apple's 1984 Super Bowl advert, autonomous vehicles, Bill Atkinson, book scanning, Brewster Kahle, Burning Man, business process, clean water, cloud computing, crowdsourcing, Dean Kamen, discounted cash flows, don't be evil, Donald Knuth, Douglas Engelbart, Douglas Engelbart, Dutch auction, El Camino Real, Evgeny Morozov, fault tolerance, Firefox, General Magic , Gerard Salton, Gerard Salton, Google bus, Google Chrome, Google Earth, Googley, high-speed rail, HyperCard, hypertext link, IBM and the Holocaust, informal economy, information retrieval, Internet Archive, Jeff Bezos, John Markoff, Ken Thompson, Kevin Kelly, Kickstarter, large language model, machine translation, Mark Zuckerberg, Menlo Park, one-China policy, optical character recognition, PageRank, PalmPilot, Paul Buchheit, Potemkin village, prediction markets, Project Xanadu, recommendation engine, risk tolerance, Rubik’s Cube, Sand Hill Road, Saturday Night Live, search inside the book, second-price auction, selection bias, Sheryl Sandberg, Silicon Valley, SimCity, skunkworks, Skype, slashdot, social graph, social software, social web, spectrum auction, speech recognition, statistical model, Steve Ballmer, Steve Jobs, Steven Levy, subscription business, Susan Wojcicki, Ted Nelson, telemarketer, The future is already here, the long tail, trade route, traveling salesman, turn-by-turn navigation, undersea cable, Vannevar Bush, web application, WikiLeaks, Y Combinator

But Baidu’s biggest boost came from the Chinese government. The government would often slow or block Google’s service and at one point even redirected Google traffic to Baidu. An apparent whispering campaign attributed the problem to Google’s alleged ineptness in serving China. The most charitable fiction was that an undersea cable had broken, cutting off service from the United States. Google had hoped that its decision to create a search engine in the .cn domain—one that followed government rules of censorship—would lead to a level playing field. But even as Google had rolled out its .cn web address, there were indications that its compromise would not satisfy the Chinese government.

The Long Game: China's Grand Strategy to Displace American Order by Rush Doshi

"World Economic Forum" Davos, American ideology, anti-communist, Asian financial crisis, autonomous vehicles, Black Lives Matter, Bretton Woods, capital controls, coronavirus, COVID-19, crony capitalism, cross-border payments, cryptocurrency, defense in depth, deindustrialization, Deng Xiaoping, deplatforming, disinformation, Dissolution of the Soviet Union, Donald Trump, drone strike, energy security, European colonialism, eurozone crisis, financial innovation, George Floyd, global pandemic, global reserve currency, global supply chain, global value chain, Great Leap Forward, high-speed rail, Internet Archive, Internet of things, Kickstarter, kremlinology, Malacca Straits, middle-income trap, Mikhail Gorbachev, MITM: man-in-the-middle, Monroe Doctrine, Network effects, Nixon triggered the end of the Bretton Woods system, offshore financial centre, positional goods, post-truth, purchasing power parity, RAND corporation, reserve currency, rolodex, Ronald Reagan, South China Sea, special drawing rights, special economic zone, TikTok, trade liberalization, transaction costs, UNCLOS, UNCLOS, undersea cable, zero-sum game

•Provide Select Alternative Financing with Allies and Partners: The United States cannot and should not provide a counter to every project Beijing chooses to support. Working with allies and partners, the United States should fund alternatives to those projects that have the greatest strategic potential (e.g., dual-use port projects, undersea cables, airfields) or work to multilateralize Chinese funding to ensure the United States has a seat at the table. •Counter Chinese Technology Acquisition and Theft: China has sought to “catch up and surpass” the West in technology, and while some of this effort is driven by domestic investment and research, a significant portion also seeks to exploit the openness of US financial markets, universities, and companies to accelerate China’s technological programs.

This Is How They Tell Me the World Ends: The Cyberweapons Arms Race by Nicole Perlroth

4chan, active measures, activist lawyer, air gap, Airbnb, Albert Einstein, Apollo 11, barriers to entry, Benchmark Capital, Bernie Sanders, Big Tech, bitcoin, Black Lives Matter, blood diamond, Boeing 737 MAX, Brexit referendum, Brian Krebs, Citizen Lab, cloud computing, commoditize, company town, coronavirus, COVID-19, crony capitalism, crowdsourcing, cryptocurrency, dark matter, David Vincenzetti, defense in depth, digital rights, disinformation, don't be evil, Donald Trump, driverless car, drone strike, dual-use technology, Edward Snowden, end-to-end encryption, failed state, fake news, false flag, Ferguson, Missouri, Firefox, gender pay gap, George Floyd, global pandemic, global supply chain, Hacker News, index card, information security, Internet of things, invisible hand, Jacob Appelbaum, Jeff Bezos, John Markoff, Ken Thompson, Kevin Roose, Laura Poitras, lockdown, Marc Andreessen, Mark Zuckerberg, mass immigration, Menlo Park, MITM: man-in-the-middle, moral hazard, Morris worm, move fast and break things, mutually assured destruction, natural language processing, NSO Group, off-the-grid, offshore financial centre, open borders, operational security, Parler "social media", pirate software, purchasing power parity, race to the bottom, RAND corporation, ransomware, Reflections on Trusting Trust, rolodex, Rubik’s Cube, Russian election interference, Sand Hill Road, Seymour Hersh, Sheryl Sandberg, side project, Silicon Valley, Skype, smart cities, smart grid, South China Sea, Steve Ballmer, Steve Bannon, Steve Jobs, Steven Levy, Stuxnet, supply-chain attack, TED Talk, the long tail, the scientific method, TikTok, Tim Cook: Apple, undersea cable, unit 8200, uranium enrichment, web application, WikiLeaks, zero day, Zimmermann PGP

CHAPTER 7: THE GODFATHER Gosler’s favorite Price Pritchett quote can be found in his 1994 book, The Employee Handbook of New Work Habits for a Radically Changing World: 13 Ground Rules for Job Success in the Information Age. I relied on Matthew Carle’s 2013 retrospective on Operation Ivy Bells to describe the undersea cable tap: “40 Years Ago, The Navy’s ‘Operation Ivy Bells’ Ended With a 70s Version of Edward Snowden,” published by Business Insider. A history of Los Alamos is available via the U.S. Department of Energy’s Office of History and Heritage Resources in “The Manhattan Project: An Interactive History.”

A Fiery Peace in a Cold War: Bernard Schriever and the Ultimate Weapon by Neil Sheehan

Albert Einstein, anti-communist, Berlin Wall, Boeing 747, Bretton Woods, British Empire, Charles Lindbergh, cuban missile crisis, disinformation, double helix, Dr. Strangelove, European colonialism, it's over 9,000, John von Neumann, Menlo Park, Mikhail Gorbachev, military-industrial complex, mutually assured destruction, Neil Armstrong, Norman Macrae, nuclear winter, operation paperclip, RAND corporation, Ronald Reagan, social contagion, undersea cable, uranium enrichment

Helena, where Napoleon was exiled to die. Ascension was to be the preliminary terminal point for Atlas and Titan. Twelve modified Second World War cargo ships, fitted out with appropriate gear, put to sea whenever there was to be a launch, filling in the gaps, particularly between Antigua and Ascension in the South Atlantic. Undersea cables tied many of the stations back to Cape Canaveral, providing instantaneous and secure communications. The radar at the stations, while sophisticated and precise, fulfilled the relatively simple task of tracking the missile. The radio monitors had a more complicated job. They had to pick up and record the telemetry from the sensors fitted into the missiles.

America in the World by Robert B. Zoellick

Albert Einstein, anti-communist, banking crisis, battle of ideas, Berlin Wall, Bretton Woods, British Empire, classic study, Corn Laws, coronavirus, cuban missile crisis, defense in depth, Deng Xiaoping, Donald Trump, Douglas Engelbart, Douglas Engelbart, energy security, European colonialism, facts on the ground, Fall of the Berlin Wall, foreign exchange controls, Great Leap Forward, guns versus butter model, hypertext link, Ida Tarbell, illegal immigration, immigration reform, imperial preference, Isaac Newton, Joseph Schumpeter, land reform, linear model of innovation, Mikhail Gorbachev, MITM: man-in-the-middle, Monroe Doctrine, mutually assured destruction, Nixon triggered the end of the Bretton Woods system, Norbert Wiener, Paul Samuelson, public intellectual, RAND corporation, reserve currency, Ronald Reagan, Ronald Reagan: Tear down this wall, scientific management, Scramble for Africa, Silicon Valley, Strategic Defense Initiative, The Wealth of Nations by Adam Smith, trade liberalization, transcontinental railway, undersea cable, Vannevar Bush, War on Poverty

The agreements and accompanying declarations covered a diverse range of topics, including rules for submarines and a ban on poison gases, Pacific security consultations, China’s independence and territorial integrity (including the Open Door principles), Chinese control of its customs charges, undersea cable rights across the Pacific, and a Sino-Japanese accord to restore the Shantung Peninsula to China. The four-power accord among the United States, Britain, Japan, and France to consult on aggressive actions and controversies in the Pacific sparked opposition from senators fearful of commitments to “guarantee” security.

The Stack: On Software and Sovereignty by Benjamin H. Bratton

1960s counterculture, 3D printing, 4chan, Ada Lovelace, Adam Curtis, additive manufacturing, airport security, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, algorithmic trading, Amazon Mechanical Turk, Amazon Robotics, Amazon Web Services, Andy Rubin, Anthropocene, augmented reality, autonomous vehicles, basic income, Benevolent Dictator For Life (BDFL), Berlin Wall, bioinformatics, Biosphere 2, bitcoin, blockchain, Buckminster Fuller, Burning Man, call centre, capitalist realism, carbon credits, carbon footprint, carbon tax, carbon-based life, Cass Sunstein, Celebration, Florida, Charles Babbage, charter city, clean water, cloud computing, company town, congestion pricing, connected car, Conway's law, corporate governance, crowdsourcing, cryptocurrency, dark matter, David Graeber, deglobalization, dematerialisation, digital capitalism, digital divide, disintermediation, distributed generation, don't be evil, Douglas Engelbart, Douglas Engelbart, driverless car, Edward Snowden, Elon Musk, en.wikipedia.org, Eratosthenes, Ethereum, ethereum blockchain, Evgeny Morozov, facts on the ground, Flash crash, Frank Gehry, Frederick Winslow Taylor, fulfillment center, functional programming, future of work, Georg Cantor, gig economy, global supply chain, Google Earth, Google Glasses, Guggenheim Bilbao, High speed trading, high-speed rail, Hyperloop, Ian Bogost, illegal immigration, industrial robot, information retrieval, Intergovernmental Panel on Climate Change (IPCC), intermodal, Internet of things, invisible hand, Jacob Appelbaum, James Bridle, Jaron Lanier, Joan Didion, John Markoff, John Perry Barlow, Joi Ito, Jony Ive, Julian Assange, Khan Academy, Kim Stanley Robinson, Kiva Systems, Laura Poitras, liberal capitalism, lifelogging, linked data, lolcat, Mark Zuckerberg, market fundamentalism, Marshall McLuhan, Masdar, McMansion, means of production, megacity, megaproject, megastructure, Menlo Park, Minecraft, MITM: man-in-the-middle, Monroe Doctrine, Neal Stephenson, Network effects, new economy, Nick Bostrom, ocean acidification, off-the-grid, offshore financial centre, oil shale / tar sands, Oklahoma City bombing, OSI model, packet switching, PageRank, pattern recognition, peak oil, peer-to-peer, performance metric, personalized medicine, Peter Eisenman, Peter Thiel, phenotype, Philip Mirowski, Pierre-Simon Laplace, place-making, planetary scale, pneumatic tube, post-Fordism, precautionary principle, RAND corporation, recommendation engine, reserve currency, rewilding, RFID, Robert Bork, Sand Hill Road, scientific management, self-driving car, semantic web, sharing economy, Silicon Valley, Silicon Valley ideology, skeuomorphism, Slavoj Žižek, smart cities, smart grid, smart meter, Snow Crash, social graph, software studies, South China Sea, sovereign wealth fund, special economic zone, spectrum auction, Startup school, statistical arbitrage, Steve Jobs, Steven Levy, Stewart Brand, Stuxnet, Superbowl ad, supply-chain management, supply-chain management software, synthetic biology, TaskRabbit, technological determinism, TED Talk, the built environment, The Chicago School, the long tail, the scientific method, Torches of Freedom, transaction costs, Turing complete, Turing machine, Turing test, undersea cable, universal basic income, urban planning, Vernor Vinge, vertical integration, warehouse automation, warehouse robotics, Washington Consensus, web application, Westphalian system, WikiLeaks, working poor, Y Combinator, yottabyte

“The rainfall from Typhoon Morakot caused rivers to flood in Taiwan flushing large volumes of sediment into the ocean. This led to several submarine landslides which broke at least nine communications cables 4000m down. It disrupted the Internet and telecommunications between Taiwan, China, Hong Kong and other parts of Southeast Asia”60 Flooded data centers and compromised undersea cables can knock out whole networks, which is especially concerning when you realize that “over 95% of global communications traffic is handled by just 1 million km of undersea fiber-optic cable. Rising sea levels increase the risk of flooding of coastal cable facilities and may also affect the stability of the seabed, making cables even more vulnerable.”

The House of Morgan: An American Banking Dynasty and the Rise of Modern Finance by Ron Chernow

Alan Greenspan, always be closing, bank run, banking crisis, Bear Stearns, Big bang: deregulation of the City of London, Black Monday: stock market crash in 1987, Bolshevik threat, book value, Boycotts of Israel, Bretton Woods, British Empire, buy and hold, California gold rush, capital controls, Carl Icahn, Charles Lindbergh, collective bargaining, Cornelius Vanderbilt, corporate raider, death from overwork, Dutch auction, Etonian, financial deregulation, financial engineering, fixed income, German hyperinflation, Glass-Steagall Act, index arbitrage, interest rate swap, junk bonds, low interest rates, margin call, Michael Milken, military-industrial complex, money market fund, Monroe Doctrine, North Sea oil, oil shale / tar sands, old-boy network, paper trading, plutocrats, Robert Gordon, Ronald Reagan, short selling, stock buybacks, strikebreaker, Suez canal 1869, Suez crisis 1956, the market place, the payments system, too big to fail, transcontinental railway, undersea cable, Yom Kippur War, young professional

It specialized in deals of strategic import, requiring sensitive contacts abroad or covert government support. Many of its deals were directed against British interests; first and foremost, 23 Wall operated as an arm of Washington. Consider telecommunications. After the war, the United States feared the British military monopoly of undersea cable communications, which had yielded invaluable wartime intelligence. The U.S. Navy favored the use of a new private corporation, supported by Washington, to battle Britain in the emerging field of radio technology. Privately, President Wilson notified General Electric that he wanted to counter Britain’s cable monopoly with an American radio monopoly.

Great North Road by Peter F. Hamilton

airport security, business process, company town, corporate governance, data acquisition, dematerialisation, disinformation, family office, illegal immigration, invention of the telescope, inventory management, plutocrats, stem cell, tech baron, the map is not the territory, undersea cable, VTOL

Libra day, and outside the camp personnel were gathering for the Sunday-night barbecue, which was fast becoming a tradition for the expedition camps. He told his e-i to establish a secure link to Vermekia. A secure connection through a six-thousand-kilometer e-Ray relay above the jungle, then an undersea cable, followed by another four-thousand-kilometer landline with dozens of civilian relays and cells was something of a joke, but the call was audio only and AIA encryption was still the best. “Two deaths?” Vermekia asked. “One death, one missing,” Vance said, wishing he didn’t sound so defensive.

The Best of 2600: A Hacker Odyssey by Emmanuel Goldstein

affirmative action, Apple II, benefit corporation, call centre, disinformation, don't be evil, Firefox, game design, Hacker Ethic, hiring and firing, information retrieval, information security, John Markoff, John Perry Barlow, late fees, license plate recognition, Mitch Kapor, MITM: man-in-the-middle, Oklahoma City bombing, optical character recognition, OSI model, packet switching, pirate software, place-making, profit motive, QWERTY keyboard, RFID, Robert Hanssen: Double agent, rolodex, Ronald Reagan, satellite internet, Silicon Valley, Skype, spectrum auction, statistical model, Steve Jobs, Steve Wozniak, Steven Levy, Telecommunications Act of 1996, telemarketer, undersea cable, UUNET, Y2K

With the exception of bilateral agreements between neighboring countries to make each other’s national systems compatible, especially in border regions, all international systems in use are: C5, C6, C7, and R2. R2 is limited to a single numbering region by policy and must use one of the three remaining systems for overseas working. There are few technical limitations to prevent R2 from working with satellites, TASI, or other analog/digital underseas cables. The spec is flexible enough to allow overseas working, but is not done at the present time. R2 is likely to displace C5 on the remaining analog trunks in the near future. National Signaling Systems CCITT 1, 2, and 3 are early international standards for signaling the distant end. C1 is just a 500 Hz line signaling tone, and was used to alert the operator at a distant switchboard that there was traffic and no DC path, due to amplifiers or repeaters on a relatively long circuit.

Industry 4.0: The Industrial Internet of Things by Alasdair Gilchrist

3D printing, additive manufacturing, air gap, AlphaGo, Amazon Web Services, augmented reality, autonomous vehicles, barriers to entry, business intelligence, business logic, business process, chief data officer, cloud computing, connected car, cyber-physical system, data science, deep learning, DeepMind, deindustrialization, DevOps, digital twin, fault tolerance, fulfillment center, global value chain, Google Glasses, hiring and firing, industrial robot, inflight wifi, Infrastructure as a Service, Internet of things, inventory management, job automation, low cost airline, low skilled workers, microservices, millennium bug, OSI model, pattern recognition, peer-to-peer, platform as a service, pre–internet, race to the bottom, RFID, Salesforce, Skype, smart cities, smart grid, smart meter, smart transportation, software as a service, stealth mode startup, supply-chain management, The future is already here, trade route, undersea cable, vertical integration, warehouse robotics, web application, WebRTC, Y2K