microservices

24 results back to index

Monolith to Microservices: Evolutionary Patterns to Transform Your Monolith by Sam Newman

Airbnb, business logic, business process, continuous integration, Conway's law, database schema, DevOps, fail fast, fault tolerance, ghettoisation, inventory management, Jeff Bezos, Kubernetes, loose coupling, microservices, MVC pattern, price anchoring, pull request, single page application, single source of truth, software as a service, source of truth, sunk-cost fallacy, systems thinking, telepresence, two-pizza team, work culture

What You Will Learn This book is designed as a deep dive into how you think about, and execute, breaking apart existing systems into a microservice architecture. We will touch on many topics related to microservice architecture, but the focus is on the decomposition side of things. For a more general guide to microservice architectures, my previous book Building Microservices would be a good place to start. In fact, I strongly recommend that you consider that book to be a companion to this one. Chapter 1 contains an overview of what microservices are, and explores further the ideas that led us to these sorts of architectures. It should work well for people who are new to microservices, but I also strongly urge those of you with more experience to not skip this chapter.

…

As such, this chapter will provide an explanation of microservice architectures, look briefly at how microservices developed (which means, naturally, taking a look at monoliths), and examine some of the advantages and challenges of working with microservices. What Are Microservices? Microservices are independently deployable services modeled around a business domain. They communicate with each other via networks, and as an architecture choice offer many options for solving the problems you may face. It follows that a microservice architecture is based on multiple collaborating microservices. They are a type of service-oriented architecture (SOA), albeit one that is opinionated about how service boundaries should be drawn, and that independent deployability is key. Microservices also have the advantage of being technology agnostic.

…

This makes it easier to maintain customer-focused teams who build domain expertise, often with embedded product owners guiding their work. Breaking Changes A microservice exists as part of a wider system. It either consumes functionality provided by other microservices, exposes its own functionality to other microservice consumers, or possibly does both. With a microservice architecture, we are striving for independent deployability, but for that to happen, we need to make sure that when we make a change to a microservice we don’t break our consumers. We can think of the functionality we expose to other microservices in terms of a contract. It’s not just about saying, “This is the data I’ll return.”

Hands-On RESTful API Design Patterns and Best Practices by Harihara Subramanian

blockchain, business logic, business process, cloud computing, continuous integration, create, read, update, delete, cyber-physical system, data science, database schema, DevOps, disruptive innovation, domain-specific language, fault tolerance, information security, Infrastructure as a Service, Internet of things, inventory management, job automation, Kickstarter, knowledge worker, Kubernetes, loose coupling, Lyft, machine readable, microservices, MITM: man-in-the-middle, MVC pattern, Salesforce, self-driving car, semantic web, single page application, smart cities, smart contracts, software as a service, SQL injection, supply-chain management, web application, WebSocket

API versions can differ, the data formats of messages getting exchanged between microservices APIs can create mismatch problems, network congestions can occur, and the varying loads on microservices may contribute for the slowdown and even failure of microservices. Testing microservices brings its own problems. Monolithic applications can be easily tested because they have a combined, single code base. That isn't the case with microservices. In the case of microservices, every microservice has to be individually tested. Further on, microservices also have to be tested collectively. There are newer approaches that simplify testing microservices. Microservices acquire special consideration because of the widespread adoption of DevOps tools.

…

A workaround is to bring forth a new route to the API: The Client can then fetch this single timeline resource to get all the data it requires to render the timeline view. That is, it creates an additional timeline microservice that lives on top of the three data microservices. This new microservice treats each underlying microservice as a resource. This top-level microservice joins the data from the underlying microservices and exposes the joined result to the client. In the microservice era, the localized database operations are being accomplished through a composite microservice: The much-discussed performance issue does not creep in here as the timeline service is predominantly hosted in containers along with the other three services.

…

Every domain is looked at as a business functionality and hence implemented as a microservice. There can be multiple methods within a microservice. There can be intra- as well as inter-microservice communication and collaboration. The microservices within a domain are supposed to interact frequently. There are situations where microservices in a domain have to connect and correspond with microservices in other domains in order to fulfill different business processes and activities. Thus, an e-business and e-commerce application is bound to have several domains and hundreds of microservices. With containers emerging as the best fit for hosting and running microservices in a fault-tolerant manner, there can be thousands of containers to run microservices-centric applications.

Building Microservices by Sam Newman

airport security, Amazon Web Services, anti-pattern, business logic, business process, call centre, continuous integration, Conway's law, create, read, update, delete, defense in depth, don't repeat yourself, Edward Snowden, fail fast, fallacies of distributed computing, fault tolerance, index card, information retrieval, Infrastructure as a Service, inventory management, job automation, Kubernetes, load shedding, loose coupling, microservices, MITM: man-in-the-middle, platform as a service, premature optimization, pull request, recommendation engine, Salesforce, SimCity, social graph, software as a service, source of truth, sunk-cost fallacy, systems thinking, the built environment, the long tail, two-pizza team, web application, WebSocket

message brokers, Technology Choices metricslibraries for, Service Metrics service metrics, Service Metrics tracking across multiple services, Metric Tracking Across Multiple Services Metrics library, Tailored Service Template microservicesappropriate application of, When Shouldn’t You Use Microservices? autonomy and, Autonomous, Building a Team benefits of, Microservices composability of, Composability definition of term, What Are Microservices? deployment ease of, Ease of Deployment drawbacks of, No Silver Bullet, When Shouldn’t You Use Microservices? key principles of, Bringing It All Together organizational alignment and, Organizational Alignment origins of, Microservices replaceability and, Optimizing for Replaceability resilience of, Resilience scaling and, Scaling size and, Small, and Focused on Doing One Thing Well technology heterogeneity of, Technology Heterogeneity vs. modules, Modules vs. service-oriented architecture, What About Service-Oriented Architecture?

…

If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-491-95035-7 [LSI] Preface Microservices are an approach to distributed systems that promote the use of finely grained services with their own lifecycles, which collaborate together. Because microservices are primarily modeled around business domains, they avoid the problems of traditional tiered architectures. Microservices also integrate new technologies and techniques that have emerged over the last decade, which helps them avoid the pitfalls of many service-oriented architecture implementations. This book is full of concrete examples of microservice use around the world, including in organizations like Netflix, Amazon, Gilt, and the REA group, who have all found that the increased autonomy this architecture gives their teams is a huge advantage.

…

For people new to the subject, I’ve structured the chapters in a way that I hope will make sense to read from beginning to end. Here is an overview of what we cover: Chapter 1, Microservices We’ll begin with an introduction to microservices, including the key benefits as well as some of the downsides. Chapter 2, The Evolutionary Architect This chapter discusses the difficulties we face in terms of making trade-offs as architects, and covers specifically just how many things we need to think about with microservices. Chapter 3, How to Model Services Here we’ll start to define the boundary of microservices, using techniques from domain-driven design to help focus our thinking. Chapter 4, Integration This is where we start getting a bit deeper into specific technology implications, as we discuss what sorts of service collaboration techniques will help us most.

Principles of Web API Design: Delivering Value with APIs and Microservices by James Higginbotham

Amazon Web Services, anti-pattern, business intelligence, business logic, business process, Clayton Christensen, cognitive dissonance, cognitive load, collaborative editing, continuous integration, create, read, update, delete, database schema, DevOps, fallacies of distributed computing, fault tolerance, index card, Internet of things, inventory management, Kubernetes, linked data, loose coupling, machine readable, Metcalfe’s law, microservices, recommendation engine, semantic web, side project, single page application, Snapchat, software as a service, SQL injection, web application, WebSocket

Async APIs for Eventing and Streaming The Problem with API Polling Async APIs Create New Possibilities A Review of Messaging Fundamentals Async API Styles Designing Async APIs Documenting Async APIs Summary V: Refining the API Design 10. From APIs to Microservices What Are Microservices? Microservices Reduce Coordination Costs The Difference Between APIs and Microservices Weighing the Complexity of Microservices Synchronous and Asynchronous Microservices Microservice Architecture Styles Right-Sizing Microservices Decomposing APIs Into Microservices Considerations When Transitioning to Microservices Summary 11. Improving the Developer Experience Creating a Mock API Implementation Providing Helper Libraries and SDKs Offering CLIs for APIs Summary 12.

…

For example, the complexity of payment processing integration may merit shifting this behavior to a separate microservice. Considerations When Transitioning to Microservices While there are many benefits to moving to microservices, the transition shouldn't be taken lightly. After some time and reflection, some organizations have chosen to simplify their microservice journey, others have decided to abandon their journey in favor of thinking smaller but without microservices, and the rest continue move forward with microservices. First, verify that a microservices-based approach is being applied to the correct context. Some microservice initiatives are dictated from the executive team without proper context.

…

They discovered that complexity increases far outweighed the value that microservices provided. Uber engineering refers to this approach as Domain-Oriented Microservice Architecture (DOMA) and have written a nice article that summarizes the approach. It resembles many of the elements of a cell-based architecture by reducing the complexity of a large-scale microservice architecture while maintaining the flexibility and benefits that it provides. Right-Sizing Microservices Organizations on the path to microservices often struggle with finding the right size for microservices. Teams will often ask, "What is the maximum allowable size for a microservice?" A better question would be, "What is the right size for this microservice based on what is needed today?"

API Marketplace Engineering: Design, Build, and Run a Platform for External Developers by Rennay Dorasamy

Airbnb, Amazon Web Services, barriers to entry, business logic, business process, butterfly effect, continuous integration, DevOps, digital divide, disintermediation, fault tolerance, if you build it, they will come, information security, Infrastructure as a Service, Internet of things, Jeff Bezos, Kanban, Kubernetes, Lyft, market fragmentation, microservices, minimum viable product, MITM: man-in-the-middle, mobile money, optical character recognition, platform as a service, pull request, ride hailing / ride sharing, speech recognition, the payments system, transaction costs, two-pizza team, Uber and Lyft, uber lyft, underbanked, web application

Kubernetes as a Service offerings abstract complexities of infrastructure management, network fabric configuration, and help teams to easily achieve complex objectives such as setting up persistent volumes. Microservices I may be heavily biased – but I consider our microservices layer to be the magic ingredient of our Platform. To be fair, microservices are underpinned by Containers. My view is that the Container is a Best Supporting Actor, while a microservice is the Best Actor. I did not always feel this way. Having spent most of my career in established integration environments, components such as an Enterprise Service Bus (ESB), Application and Process servers, all deployed across complex cluster configurations, were deeply ingrained when I joined the project.

…

I was especially pleased with my efforts as the shared library would allow in-process communication between microservice and backend integration logic. Not only would this solution be super performant, but it would also eliminate the horror of code duplication. I was so enamored with the shared library concept, I dubbed it a “framework.” The home for this prized asset was a git submodule, which every microservice referenced. Figure 5-4Shared integration library Unfortunately, as the number of microservice developers on the team grew, the framework grew more and more fragmented. The submodule concept also resulted in each microservice referencing a specific variant of the framework.

…

The basic premise is that the integration component is extracted, externalized, and only accessible via a well-defined interface. Figure 5-5Dedicated integration adapter This has led to the creation of a new category of microservice in our platform, which we have labelled as “middleware components.” The various taxonomies of microservices in our platform are discussed later in this chapter. The inherent benefit is that this reduces code duplication and results in a leaner, more streamlined microservice deployment. That is, instead of packaging a version of the framework inside the microservice, the framework functionality is externalized and accessed via an interface. Unfortunately, this opens the door to a potential performance drawback of out-of-process execution.

Architecting For Scale by Lee Atchison

Amazon Web Services, business logic, business process, cloud computing, continuous integration, DevOps, Internet of things, microservices, platform as a service, risk tolerance, software as a service, web application

Chapter 10, Building Systems with Reduced Risk In this chapter, I give suggestions on how to reduce risk within your applications and build applications with lower risk. Part III, “Services and Microservices” Services and microservices are an architecture strategy for building larger and more complicated applications that need to operate at higher scale. Chapter 11, Why Use Services? This chapter explores why services are important to building highly scalable applications. Chapter 12, Using Microservices Here, I provide an introduction on creating microservice-based architectures, focusing on sizing of services and determining where service boundaries should be created in order to improve scaling and availability.

…

One common place where modern software construction principles tend to increase complexity more than perhaps is necessary is in microservice-based architectures. Microservice-based architectures reduce the complexity of individual components substantially, making it possible for the individual services to be easily understood and built using simpler techniques and designs. However, although they reduce the complexity of the individual microservice, they increase the number of independent modules (microservices) necessary to build a large-scale application. By having a larger number of independent modules working together, you increase the interdependence on the modules, and increase the overall complexity of the application.

…

This is a good question, and one that does not have a single correct answer. Some companies that “service-ize” split their application into many (hundreds or thousands) of very tiny microservices. Others split their application into only a handful of larger services. There is no right answer to this problem. However, the industry is trending toward smaller microservices, and more of them. Technologies such as Docker have made these larger number of microservices a viable system topology. We use the term services and microservices interchangeably in this book. Dividing into Services So, how do you decide where service boundaries should be? Company organization, culture, and the type of application can play a major role in determining service boundaries.

Seeking SRE: Conversations About Running Production Systems at Scale by David N. Blank-Edelman

Affordable Care Act / Obamacare, algorithmic trading, AlphaGo, Amazon Web Services, backpropagation, Black Lives Matter, Bletchley Park, bounce rate, business continuity plan, business logic, business process, cloud computing, cognitive bias, cognitive dissonance, cognitive load, commoditize, continuous integration, Conway's law, crowdsourcing, dark matter, data science, database schema, Debian, deep learning, DeepMind, defense in depth, DevOps, digital rights, domain-specific language, emotional labour, en.wikipedia.org, exponential backoff, fail fast, fallacies of distributed computing, fault tolerance, fear of failure, friendly fire, game design, Grace Hopper, imposter syndrome, information retrieval, Infrastructure as a Service, Internet of things, invisible hand, iterative process, Kaizen: continuous improvement, Kanban, Kubernetes, loose coupling, Lyft, machine readable, Marc Andreessen, Maslow's hierarchy, microaggression, microservices, minimum viable product, MVC pattern, performance metric, platform as a service, pull request, RAND corporation, remote working, Richard Feynman, risk tolerance, Ruby on Rails, Salesforce, scientific management, search engine result page, self-driving car, sentiment analysis, Silicon Valley, single page application, Snapchat, software as a service, software is eating the world, source of truth, systems thinking, the long tail, the scientific method, Toyota Production System, traumatic brain injury, value engineering, vertical integration, web application, WebSocket, zero day

We aggregate this information for all our critical microservices and compare the trend to the previous 30 days. It isn’t meant to be near real time, it’s meant to be more operational — are you getting better or worse. Let’s say a team owns three microservices, and the goal is for the microservices to have four-nines availability for calls from clients. If these microservices stay above four-nines, the team will never receive an email (availability report). If the last 7 days compared to the previous 21 have a deviation in a negative manner, or if the microservices are failing to achieve the four-nines, a report will be sent to the team.

…

Clicking on a bar in the graph will deliver a detailed report that shows for the entire window call rates of upstream (client) services talking to the microservice and the availability to those calls. On the right panel is a view of downstream dependency call and success rates, which is helpful as the microservice availability might be reduced by downstream dependency. We call this internal implementation the “microservice availability scorecard framework.” At this point, the SRE team (working with Data Analytics) has provided information to a team about the microservices they own, and in particular their microservice availability to dependent client services, independent of Netflix service availability.

…

The Problem with Systems You can think of the engineering organization at Netflix as about a hundred small engineering teams, of around five to seven people each. The service — that product that the customers use — is architected as many hundreds of microservices working together. Every microservice is owned by just one team, and that team is responsible for everything: features, roadmap, operations, and uptime for that microservice. An example of a microservice might be a customer microservice, which for a given customer ID serves up the metadata stored with that person. Another might be a personalization service that stores information about preferences related to a customer so that when a customer looks for something to watch on Netflix, we decorate their experience with context based on what they previously watched, and so on.

Building Secure and Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems by Heather Adkins, Betsy Beyer, Paul Blankinship, Ana Oprea, Piotr Lewandowski, Adam Stubblefield

air gap, anti-pattern, barriers to entry, bash_history, behavioural economics, business continuity plan, business logic, business process, Cass Sunstein, cloud computing, cognitive load, continuous integration, correlation does not imply causation, create, read, update, delete, cryptocurrency, cyber-physical system, database schema, Debian, defense in depth, DevOps, Edward Snowden, end-to-end encryption, exponential backoff, fault tolerance, fear of failure, general-purpose programming language, Google Chrome, if you see hoof prints, think horses—not zebras, information security, Internet of things, Kubernetes, load shedding, margin call, microservices, MITM: man-in-the-middle, NSO Group, nudge theory, operational security, performance metric, pull request, ransomware, reproducible builds, revision control, Richard Thaler, risk tolerance, self-driving car, single source of truth, Skype, slashdot, software as a service, source of truth, SQL injection, Stuxnet, the long tail, Turing test, undersea cable, uranium enrichment, Valgrind, web application, Y2K, zero day

Risks include a SQL injection vulnerability in the catalog search code allowing an attacker to obtain sensitive user data, like names or shipping addresses, or a remote code execution vulnerability in the web application server, such as CVE-2010-1870, permitting an attacker to read or modify any part of the application’s database. Small TCBs and strong security boundaries We can improve the security of our design by splitting the application into microservices. In this architecture, each microservice handles a self-contained part of the application’s functionality and stores data in its own separate database. These microservices communicate with each other via RPCs and treat all incoming requests as not necessarily trustworthy, even if the caller is another internal microservice. Using microservices, we might restructure the application as shown in Figure 6-3. Instead of a monolithic server, we now have a web application frontend and separate backends for the product catalog and purchasing-related functionality.

…

Isolate Your Rate-Limiting Mechanism to Increase Reliability If your system can deploy changes extremely quickly, how can you protect against unwanted rapid execution? One strategy is to build the rate-limiting mechanism as an independent, standalone, single-purpose microservice that constrains the rate of change to a particular system or systems. Make this microservice as simple as possible and amenable to rigorous testing. For example, a rate-limiting microservice might provide a short-lived cryptographic token that asserts that the microservice reviewed and approved a particular change at a certain time. The rate-limiting service is also an excellent point at which to collect change logs for auditing purposes.

…

To limit the blast radius of such a compromise, different jobs should typically be run as different roles. For example, if you have two microservices that need access to two different classes of data (say, photos and text chats), running these two microservices as different roles can increase the resilience of your system even if the two microservices are developed and run by the same team. Location Separation Location separation helps to limit an attacker’s impact along an additional dimension: the location where the microservice is running. For example, you might want to prevent an adversary who has physically compromised a single datacenter from being able to read data in all your other datacenters.

Ask Your Developer: How to Harness the Power of Software Developers and Win in the 21st Century by Jeff Lawson

Airbnb, AltaVista, Amazon Web Services, barriers to entry, big data - Walmart - Pop Tarts, Big Tech, big-box store, bitcoin, business process, call centre, Chuck Templeton: OpenTable:, cloud computing, coronavirus, COVID-19, create, read, update, delete, cryptocurrency, data science, David Heinemeier Hansson, deep learning, DevOps, Elon Musk, financial independence, global pandemic, global supply chain, Hacker News, Internet of things, Jeff Bezos, Kanban, Lean Startup, loose coupling, Lyft, Marc Andreessen, Marc Benioff, Mark Zuckerberg, microservices, minimum viable product, Mitch Kapor, move fast and break things, Paul Graham, peer-to-peer, ride hailing / ride sharing, risk tolerance, Ruby on Rails, Salesforce, side project, Silicon Valley, Silicon Valley startup, Skype, social distancing, software as a service, software is eating the world, sorting algorithm, Startup school, Steve Ballmer, Steve Jobs, Telecommunications Act of 1996, Toyota Production System, transaction costs, transfer pricing, two-pizza team, Uber and Lyft, uber lyft, ubercab, web application, Y Combinator

Once you’ve divided and subdivided the business into small teams that specialize in particular areas, offering microservices for each other to use, with well-documented interfaces and pricing that represents the true costs of delivering those services—well, why develop all of those microservices internally? Why devote your own developers to microservices that you could instead buy from other companies? Why write your own microservice for calculating currency translation on international sales when you can just buy that microservice from a vendor that specializes in currency translation software? So your developers start plugging in pieces from specialist providers, and boom—you have a software supply chain.

…

Cloud platforms are replacing legacy infrastructure in nearly every category. These commercial microservices are the raw ingredients of almost every app you use. Behind that user interface, apps are actually a patchwork comprising hundreds or even thousands of microservices, some created by a company’s own developers and some from commercial providers. Today, there are still many places where developers have no choice but to build microservices from scratch because off-the-shelf alternatives don’t exist yet. But the beauty of the microservice model is that it’s possible to swap out a microservice without interfering with the rest of the code in your app.

…

“But if one team has a request for something that would be cool to do, and another team has a project that will unlock $90 million in recurring revenue for the company, we’re going to solve that one first and put the other request on our backlog.” Composable over Monolithic Our software is based on a microservices architecture composed of hundreds of microservices. Each microservice performs a single function or capability. The advantage of microservices is that we can route around or absorb a failure. If one service fails, it won’t bring down the entire Twilio voice system, for example. The services are all loosely coupled. They’re all built by different teams, who can work independently. One microservice might be version one or two, and another might be on version five. But as long as they all “speak” to the API that connects them, that’s fine.

Team Topologies: Organizing Business and Technology Teams for Fast Flow by Matthew Skelton, Manuel Pais

anti-pattern, business logic, business process, call centre, cognitive load, continuous integration, Conway's law, database schema, DevOps, different worldview, Dunbar number, holacracy, information security, Infrastructure as a Service, Internet of things, Jeff Bezos, Kanban, Kickstarter, knowledge worker, Kubernetes, Lean Startup, loose coupling, meta-analysis, microservices, Norbert Wiener, operational security, platform as a service, pull request, remote working, systems thinking, two-pizza team, web application

The homomorphic force identified by Conway’s law is exerting a strong pull on the “natural” software architecture to emerge from the current organization design and communication paths. For example, let’s say that we want to use a microservices architecture for some new cloud-based software systems, where each separate service is independent and has its own data store (Figure 2.3, see page 21). Figure 2.3: Microservices Architecture with Independent Services and Data Stores A microservices-based architecture with four separate services, each with its own data store, API layer, and front-end client. By applying the reverse Conway maneuver, we can design our teams to “match” the required software architecture by having separate developers for the client applications and the API, and a database developer within the team rather than separate from it (Figure 2.4, see page 22).

…

Accelerate Learning and Adoption of New Practices Constant Evolution of Team Topologies Combining Teams Topologies for Greater Effectiveness Triggers for Evolution of Team Topologies Self Steer Design and Development Summary: Evolving Team Topologies Conclusion: The Next-Generation Digital Operating Model Four Team Types and Three Interaction Modes Team-First Thinking: Cognitive Load, Team API, Team-Sized Architecture Strategic Application of Conway’s Law Evolve Organization Design for Adaptability and Sensing Team Topologies Alone Are Not Sufficient for IT Effectiveness Next Steps: How to Get Started with Team Topologies Glossary Recommended Reading References Notes Index Acknowledgments About the Authors FIGURES & TABLES FIGURES 0.1: The Four Team Types and Three Interaction Modes 1.1: Org Chart with Actual Lines of Communication 1.2: Obstacles to Fast Flow 2.1: Four Teams Working on a Software System 2.2: Software Architecture from Four-Team Organization 2.3: Microservices Architecture with Independent Services and Data Stores 2.4: Team Design for Microservices Architecture with Independent Services and Data Stores 2.5: Inter-Team Communication 3.1: Scaling Teams Using Dunbar’s Number 3.2: No More than One Complicated or Complex Domain per Team 3.3: Typical vs. Team-First Software Subsystem Boundaries 3.4: Office Layout at CDL 4.1: Organization not Optimized for Flow of Change 4.2: Organization Optimized for Flow of Change 4.3: Relationship between SRE Team and Application Team 4.4: Influence of Size and Engineering Discipline on Team Interaction Patterns 5.1: The Four Fundamental Team Topologies 5.2: Platform Composed of Several Fundamental Team Topologies 5.3: Traditional Infrastructure Team Organization 5.4: Support Teams Aligned to Stream of Change 6.1: Mobile, Cloud, and IoT Technology Fracture Plane Scenario 7.1: Collaboration vs.

…

By applying the reverse Conway maneuver, we can design our teams to “match” the required software architecture by having separate developers for the client applications and the API, and a database developer within the team rather than separate from it (Figure 2.4, see page 22). Figure 2.4: Team Design for Microservices Architecture with Independent Services and Data Stores An organization design that anticipates the homomorphic force behind Conway’s law to help produce a software architecture with four independent microservices. (Again, this is basically the diagram in Figure 2.3 rotated ninety degrees.) According to Conway’s law, this team design will most “naturally” produce the desired software architecture.

Whistleblower: My Journey to Silicon Valley and Fight for Justice at Uber by Susan Fowler

"Susan Fowler" uber, Airbnb, Albert Einstein, Big Tech, Burning Man, cloud computing, data science, deep learning, DevOps, Donald Trump, Elon Musk, end-to-end encryption, fault tolerance, Grace Hopper, Higgs boson, Large Hadron Collider, Lyft, Maui Hawaii, messenger bag, microservices, Mitch Kapor, Richard Feynman, ride hailing / ride sharing, self-driving car, Silicon Valley, TechCrunch disrupt, Travis Kalanick, Uber for X, uber lyft, work culture

If we could, then the teams wouldn’t need a SWAT team to come in and fix everything; instead, they’d be able to fix their own problems. If we could pull this off, it would make a huge difference: the teams that ran the microservices didn’t always know that the issues they were facing were problems that other teams had solved before. Furthermore, there was no standardization across the company for how our software systems should be built. Over a thousand independent microservices, spread out across countless engineering teams, all had to work together for the Uber app to function correctly; these microservices didn’t always work together the way they needed to, and the lack of standardization was largely to blame.

…

I wanted to keep working with him, but the teams he directly managed didn’t have any room for me. I petitioned Charles to let me work with one of the other cloud-related teams but was instead pushed into the newly formed “consulting SRE team.” Out of over a thousand microservices, only the most business-critical ones had site reliability engineers on their engineering teams, and those site reliability engineers made sure that the software was always up and running. The other microservice teams that didn’t have site reliability engineers desperately wanted them, but there weren’t enough to go around (it was always much easier to hire general software engineers than specialized reliability engineers).

…

After working on the SRE consulting team for several months, I noticed that all the different engineering teams I’d been helping fix their software suffered from similar problems. I made a checklist of these common problems—significant problems that caused the microservices to break, which would bring the Uber app down or make it unbearably slow—and wrote down the ways we could fix them. From that point on, whenever I met with a new team, I sat down with them, looked through their code, evaluated the architecture of their microservices, and then together we’d go through the items on my checklist one by one. By the end of the meeting, we had a pretty good idea of what was broken and a clear, step-by-step plan for how to fix it.

Docker Deep Dive by Nigel Poulton

Debian, DevOps, Kubernetes, microservices, platform as a service, source of truth, web application

We’ll split this apter into the usual three parts: • e TLDR • e deep dive • e commands Deploying apps with Compose - The TLDR Modern cloud-native apps are made of multiple smaller services that interact to form a useful app. We call this paern “microservices”. A simple example might be an app with the following seven services: • • • • • • • Web front-end Ordering Catalog Ba-end database Logging Authentication Authorization Get all of these working together, and you have a useful application. Deploying and managing lots of small microservices like these can be hard. is is where Doer Compose comes in to play. Instead of gluing ea microservice together with scripts and long docker commands, Doer Compose lets you describe an entire app in a single declarative configuration file, and deploy it with a single command.

…

An engine for orestrating microservices apps On the clustering front, Swarm groups one or more Doer nodes and lets you manage them as a cluster. Outof-the-box, you get an encrypted distributed cluster store, encrypted networks, mutual TLS, secure cluster join tokens, and a PKI that makes managing and rotating certificates a breeze. You can even non-disruptively add and remove nodes. It’s a beautiful thing. While we cover some aspects of Swarm security in this apter, we go a lot deeper in Chapter 15. On the orestration front, Swarm exposes a ri API that allows you to deploy and manage complex microservices apps with ease.

…

At its core, Swarm has a secure clustering component, and an orestration component. 149 e secure clustering component is enterprise-grade and offers a wealth of security and HA features that are automatically configured and extremely simple to modify. e orestration component allows you to deploy and manage cloud-native microservices applications in a simple declarative manner. We’ll dig deeper into deploying cloud-native microservices apps in a declarative manner in Chapter 14. 11: Docker Networking It’s always the network! Any time there’s a an infrastructure problem, we always blame the network. Part of the reason is that networks are at the center of everything — no network, no app!

Chaos Engineering: System Resiliency in Practice by Casey Rosenthal, Nora Jones

Amazon Web Services, Asilomar, autonomous vehicles, barriers to entry, blockchain, business continuity plan, business intelligence, business logic, business process, cloud computing, cognitive load, complexity theory, continuous integration, cyber-physical system, database schema, DevOps, fail fast, fault tolerance, hindsight bias, human-factors engineering, information security, Kanban, Kubernetes, leftpad, linear programming, loose coupling, microservices, MITM: man-in-the-middle, no silver bullet, node package manager, operational security, OSI model, pull request, ransomware, risk tolerance, scientific management, Silicon Valley, six sigma, Skype, software as a service, statistical model, systems thinking, the scientific method, value engineering, WebSocket

The Chaos Automation Platform (ChAP)5 was developed at Netflix while Nora was an engineer and Casey was manager of the Chaos Engineering Team. It exemplifies the advanced principles highlighted throughout this book as well as the feedback loop we expect in CV. ChAP is fully automated. It introspects the microservice architecture, selects microservices for inspection, constructs experiments for that microservice, and runs the experiment during business hours. The experiment hypotheses take the following form: under conditions X for microservice Y, customers still watch a normal amount of video streams. X is usually an amount of downstream latency, and normal is defined by a control group. ChAP: Selecting Experiments As part of ChAP’s rollout, a product called Monocle was created.

…

Specifically this means whether or not the system owner has things to fix before they can safely run a chaos experiment. If it is safe to fail, then that microservice may be added to the list and prioritized for experimentation. ChAP: Running Experiments Through integration with Netflix’s CD tool Spinnaker, ChAP is able to introspect the code deployed for the chosen microservice in the prioritization process provided by Monocle. For this microservice, there are a number of instances running in a cluster. ChAP will spin up two additional instances. One acts as the control, the other as the variable.

…

Experiments typically run for 45 minutes (with a minimum being 20 minutes). If, on the other hand, the KPIs deviate, then the experiment is immediately shut down. No more requests are routed to the experiment. The control and variable instances are torn down. Most importantly, the team responsible for that microservice is notified that under conditions X for their microservice, customers have a bad time. This discovery informs the team about their own safety margin and it is up to them to figure out why customers have a bad time under those conditions and what to do about it. The Advanced Principles in ChAP Notice how ChAP applies the advanced principles from Chapter 3: Build a hypothesis around steady-state behavior At Netflix the steady state behavior is modeled using the KPI of how many video streams start per second.

Industry 4.0: The Industrial Internet of Things by Alasdair Gilchrist

3D printing, additive manufacturing, air gap, AlphaGo, Amazon Web Services, augmented reality, autonomous vehicles, barriers to entry, business intelligence, business logic, business process, chief data officer, cloud computing, connected car, cyber-physical system, data science, deep learning, DeepMind, deindustrialization, DevOps, digital twin, fault tolerance, fulfillment center, global value chain, Google Glasses, hiring and firing, industrial robot, inflight wifi, Infrastructure as a Service, Internet of things, inventory management, job automation, low cost airline, low skilled workers, microservices, millennium bug, OSI model, pattern recognition, peer-to-peer, platform as a service, pre–internet, race to the bottom, RFID, Salesforce, Skype, smart cities, smart grid, smart meter, smart transportation, software as a service, stealth mode startup, supply-chain management, The future is already here, trade route, undersea cable, vertical integration, warehouse robotics, web application, WebRTC, Y2K

Security goes beyond simple authentication and confidentiality, and also includes authorization and integrity. When it comes to ease of implementation, SOAP is the web service at the forefront. Microservices The use of microservices and web APIs has become very popular in web and cloud-based applications and they are both ideally suited for the IIoT. The term microservice has no standard, formal definition; however, there are certain characteristics that identify them. Essentially, there is a distinguishable microservice architecture, which provides a method of developing software applications as a suite of small, modular services. These independent services run a unique process and communicate through a well-defined, lightweight web service or other mechanism in order to deliver a specific result. 151 152 Chapter 9 | Software Design Concepts What makes microservices valuable are that applications can be constructed that decouple the underlying complexities of the host systems from the application’s purpose.

…

With microservices, we see the introduction of a different approach, one that is ideal for the IoT, and by extension the Industrial Internet. The microservices approach is to build your own or consume an open microservice, which is specific to a function. Programmers then construct applications using these independent modules called microservices, which the programmer can scale and modify in isolation, leaving the rest of the application untouched. By not compromising the overall integrity of the application, programmers can tweak an individual service before redeploying it. Similarly, they can add or customize a service for a new device type of model without affecting any other service in the application, for example, any existing device type’s user experience.

…

Similarly, they can add or customize a service for a new device type of model without affecting any other service in the application, for example, any existing device type’s user experience. Microservices are scalable, adaptable, and modular, making them ideal for cloud-based applications that have to provide dynamic, yet consistent user experience across a wide array, and ever changing, list of devices. To clarify the benefits of microservices, and their scalability we can look to many large web-scaled companies, such as Netflix, Amazon, and eBay. All of these web giants have migrated over the last decade to microservice-based applications. Netflix, for example, receives around one billion API calls per day from over 800 different types and models of devices.

Terraform: Up and Running: Writing Infrastructure as Code by Yevgeniy Brikman

Amazon Web Services, cloud computing, DevOps, en.wikipedia.org, full stack developer, functional programming, general-purpose programming language, microservices, Ruby on Rails

For example, you could create a canonical module that defines how to deploy a single microservice—including how to run a cluster, how to scale the cluster in response to load, and how to distribute traffic requests across the cluster— and each team could use this module to manage their own microservices with just a few lines of code. To make such a module work for multiple teams, the Terraform code in that module must be flexible and configurable. For example, one team may want to use your mod‐ ule to deploy a single instance of their microservice with no load balancer while another may want a dozen instances of their microservice with a load balancer to dis‐ tribute traffic between those instances.

…

It’s an easy lan‐ guage to learn, so in the span of about 25 pages, you’ll go from running your first Terraform commands all the way up to using Terraform to deploy a cluster of servers with a load balancer that distributes traffic across them. This infrastructure is a good starting point for running scalable, highly-available web services and microservices. In subsequent chapters, you’ll evolve this example even further. Terraform can provision infrastructure across many different types of cloud provid‐ ers, including Amazon Web Services (AWS), Azure, Google Cloud, DigitalOcean, and many others. For just about all of the code examples in this chapter and the rest of the book, you are going to use AWS.

…

File layout | 83 • global: A place to put resources that are used across all environments, such as user management (e.g. S3, IAM). Within each environment, there are separate folders for each “component.” The com‐ ponents differ for every project, but the typical ones are: • vpc: The network topology for this environment. • services: The apps or microservices to run in this environment, such as a Ruby on Rails frontend or a Scala backend. Each app could even live in its own folder to isolate it from all the other apps. • data-storage: The data stores to run in this environment, such as MySQL or Redis. Each data store could even live in its own folder to isolate it from all other data stores.

The Nature of Software Development: Keep It Simple, Make It Valuable, Build It Piece by Piece by Ron Jeffries

Amazon Web Services, anti-pattern, bitcoin, business cycle, business intelligence, business logic, business process, c2.com, call centre, cloud computing, continuous integration, Conway's law, creative destruction, dark matter, data science, database schema, deep learning, DevOps, disinformation, duck typing, en.wikipedia.org, fail fast, fault tolerance, Firefox, Hacker News, industrial robot, information security, Infrastructure as a Service, Internet of things, Jeff Bezos, Kanban, Kubernetes, load shedding, loose coupling, machine readable, Mars Rover, microservices, Minecraft, minimum viable product, MITM: man-in-the-middle, Morris worm, move fast and break things, OSI model, peer-to-peer lending, platform as a service, power law, ransomware, revision control, Ruby on Rails, Schrödinger's Cat, Silicon Valley, six sigma, software is eating the world, source of truth, SQL injection, systems thinking, text mining, time value of money, transaction costs, Turing machine, two-pizza team, web application, zero day

A startup in the hypergrowth stage probably values scaling the tech team much more than it values long-term evolution of the business requirements. An established enterprise that needs to depreciate its capital expenditure over five years needs to evolve along business requirements and also the technology platform. A Note on Microservices Microservices are a technological solution to an organizational problem. As an organization grows, the number of communication pathways grows exponentially. Similarly, as a piece of software grows, the number of possible dependencies within the software grows exponentially. Classes tend toward a power-law distribution.

…

Developers need a longer ramp-up period before they can work safely in the codebase. (At some point, that ramp-up time exceeds your average developer tenure!) Microservices promise to break the paralysis by curtailing the size of any piece of software. Ideally it should be no bigger than what fits in one developer’s head. I don’t mean that metaphorically. When shown on screen, the length of the code should be smaller than the coder’s melon. That forces you to either write very small services or hire a very oddly proportioned development staff. Another subtle issue about microservices that gets lost in the excitement is that they’re great when you are scaling up your organization.

…

But what happens when you need to downsize? Services can get orphaned easily. Even if they get adopted into a good home, it’s easy to get overloaded when you have twice as many services as developers. Don’t pursue microservices just because the Silicon Valley unicorns are doing it. Make sure they address a real problem you’re likely to suffer. Otherwise, the operational overhead and debugging difficulty of microservices will outweigh your benefits. Loose Clustering Systems should exhibit loose clustering. In a loose cluster, the loss of an individual instance is no more significant than the fall of a single tree in a forest.

Designing Reactive Systems: The Role of Actors in Distributed Architecture by Hugh McKee

Amazon Web Services, business logic, fault tolerance, Internet of things, microservices

Summary Technology adoption is rarely cyclical; however, in case of the Actor model (created in the early 1970s) the spotlight is swinging back to this unique approach to distributed, concurrent computation. As Forrester Research points out in “How To Capture The Benefits Of Microservice Design” (2016), the Actor model is receiving “renewed interest as cloud concurrency challenges grow” in enterprises building microservices architectures. This report is targeted toward decision makers in the enterprise and provides some high-level insight into how actors and actor systems can be used to create lightweight business systems that evolve quickly, that can scale, and that can run without stopping.

…

Actors can be implemented to react to nodes leaving and joining a cluster. Work can be distributed across a cluster. Actors and actor systems provide an abstraction layer that allows for higher levels of concurrency. Chapter 6. Conclusion Today, it is now possible to create distributed, microservices-based systems that were impossible to even dream of just a few short years ago. Enterprises across all industries now desire the ability to create systems that can evolve at the speed of the business and cater to the whims of users. We can now elastically scale systems that support massive numbers of users and process huge volumes of data.

…

We can now elastically scale systems that support massive numbers of users and process huge volumes of data. It is now possible to harden systems with a level of resilience that enables them to run with such low downtime that it’s measured in seconds, rather than hours. One of the foundational technologies that enables us to create microservices architectures that evolve quickly, that can scale, and that can run without stopping, is systems based on the Actor model. It’s the Actor model that provides the core functionality of Reactive systems, defined in the Reactive Manifesto as responsive, resilient, elastic, and message driven (see Figure 6-1). In this report, we have reviewed some of the features and characteristics of how actors are used in actor systems, but we have only scratched the surface of how actor systems are being used today.

Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps by Valliappa Lakshmanan, Sara Robinson, Michael Munn

A Pattern Language, Airbnb, algorithmic trading, automated trading system, business intelligence, business logic, business process, combinatorial explosion, computer vision, continuous integration, COVID-19, data science, deep learning, DevOps, discrete time, en.wikipedia.org, Hacker News, industrial research laboratory, iterative process, Kubernetes, machine translation, microservices, mobile money, natural language processing, Netflix Prize, optical character recognition, pattern recognition, performance metric, recommendation engine, ride hailing / ride sharing, selection bias, self-driving car, sentiment analysis, speech recognition, statistical model, the payments system, web application

When the entire codebase is inextricably linked, it becomes difficult for individual developers to debug errors and work independently on different parts of the application. In recent years, monolithic apps have been replaced in favor of a microservices architecture where individual pieces of business logic are built and deployed as isolated (micro) packages of code. With microservices, a large application is split into smaller, more manageable parts so that developers can build, debug, and deploy pieces of an application independently. This monolith-versus-microservice discussion provides a good analogy for scaling ML workflows, enabling collaboration, and ensuring ML steps are reproducible and reusable across different workflows.

…

Even if this ability to export the model as a SavedModel did not exist, we could have extracted the weights, written a mathematical model to carry out the linear model, containerized it, and deployed the container image into a serving platform. Prediction library Instead of deploying the serving function as a microservice that can be invoked via a REST API, it is possible to implement the prediction code as a library function. The library function would load the exported model the first time it is called, invoke model.predict() with the provided input, and return the result. Application developers who need to predict with the library can then include the library with their applications. A library function is a better alternative than a microservice if the model cannot be called over a network either because of physical reasons (there is no network connectivity) or because of performance constraints.

…

You might run training and evaluation multiple times, performing additional feature engineering and tweaking your model architecture. Once you are happy with your model’s performance during evaluation, you’ll likely want to serve your model so that others can access it to make predictions. We use the term serving to refer to accepting incoming requests and sending back predictions by deploying the model as a microservice. The serving infrastructure could be in the cloud, on-premises, or on-device. The process of sending new data to your model and making use of its output is called prediction. This can refer both to generating predictions from local models that have not yet been deployed as well as getting predictions from deployed models.

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann

active measures, Amazon Web Services, billion-dollar mistake, bitcoin, blockchain, business intelligence, business logic, business process, c2.com, cloud computing, collaborative editing, commoditize, conceptual framework, cryptocurrency, data science, database schema, deep learning, DevOps, distributed ledger, Donald Knuth, Edward Snowden, end-to-end encryption, Ethereum, ethereum blockchain, exponential backoff, fake news, fault tolerance, finite state, Flash crash, Free Software Foundation, full text search, functional programming, general-purpose programming language, Hacker News, informal economy, information retrieval, Internet of things, iterative process, John von Neumann, Ken Thompson, Kubernetes, Large Hadron Collider, level 1 cache, loose coupling, machine readable, machine translation, Marc Andreessen, microservices, natural language processing, Network effects, no silver bullet, operational security, packet switching, peer-to-peer, performance metric, place-making, premature optimization, recommendation engine, Richard Feynman, self-driving car, semantic web, Shoshana Zuboff, social graph, social web, software as a service, software is eating the world, sorting algorithm, source of truth, SPARQL, speech recognition, SQL injection, statistical model, surveillance capitalism, systematic bias, systems thinking, Tragedy of the Commons, undersea cable, web application, WebSocket, wikimedia commons

ISBN: 978-0-122-33435-1 Summary | 141 [28] Russell Housley, Warwick Ford, Tim Polk, and David Solo: “RFC 2459: Internet X.509 Public Key Infrastructure: Certificate and CRL Profile,” IETF Network Work‐ ing Group, Standards Track, January 1999. [29] Lev Walkin: “Question: Extensibility and Dropping Fields,” lionet.info, Septem‐ ber 21, 2010. [30] Jesse James Garrett: “Ajax: A New Approach to Web Applications,” adaptive‐ path.com, February 18, 2005. [31] Sam Newman: Building Microservices. O’Reilly Media, 2015. ISBN: 978-1-491-95035-7 [32] Chris Richardson: “Microservices: Decomposing Applications for Deployability and Scalability,” infoq.com, May 25, 2014. [33] Pat Helland: “Data on the Outside Versus Data on the Inside,” at 2nd Biennial Conference on Innovative Data Systems Research (CIDR), January 2005. [34] Roy Thomas Fielding: “Architectural Styles and the Design of Network-Based Software Architectures,” PhD Thesis, University of California, Irvine, 2000. [35] Roy Thomas Fielding: “REST APIs Must Be Hypertext-Driven,” roy.gbiv.com, October 20 2008. [36] “REST in Peace, SOAP,” royal.pingdom.com, October 15, 2010. [37] “Web Services Standards as of Q1 2007,” innoq.com, February 2007. [38] Pete Lacey: “The S Stands for Simple,” harmful.cat-v.org, November 15, 2006. [39] Stefan Tilkov: “Interview: Pete Lacey Criticizes Web Services,” infoq.com, December 12, 2006. [40] “OpenAPI Specification (fka Swagger RESTful API Documentation Specifica‐ tion) Version 2.0,” swagger.io, September 8, 2014. [41] Michi Henning: “The Rise and Fall of CORBA,” ACM Queue, volume 4, number 5, pages 28–34, June 2006. doi:10.1145/1142031.1142044 [42] Andrew D.

…

.: “Feral Concurrency Control: An Empirical Investigation of Modern Application Integrity,” at ACM International Conference on Management of Data (SIGMOD), June 2015. doi: 10.1145/2723372.2737784 [37] Guy Steele: “Re: Need for Macros (Was Re: Icon),” email to ll1-discuss mailing list, people.csail.mit.edu, December 24, 2001. [38] David Gelernter: “Generative Communication in Linda,” ACM Transactions on Programming Languages and Systems (TOPLAS), volume 7, number 1, pages 80–112, January 1985. doi:10.1145/2363.2433 [39] Patrick Th. Eugster, Pascal A. Felber, Rachid Guerraoui, and Anne-Marie Ker‐ marrec: “The Many Faces of Publish/Subscribe,” ACM Computing Surveys, volume 35, number 2, pages 114–131, June 2003. doi:10.1145/857076.857078 [40] Ben Stopford: “Microservices in a Streaming World,” at QCon London, March 2016. [41] Christian Posta: “Why Microservices Should Be Event Driven: Autonomy vs Authority,” blog.christianposta.com, May 27, 2016. [42] Alex Feyerke: “Say Hello to Offline First,” hood.ie, November 5, 2013. [43] Sebastian Burckhardt, Daan Leijen, Jonathan Protzenko, and Manuel Fähndrich: “Global Sequence Protocol: A Robust Abstraction for Replicated Shared State,” at Summary | 547 29th European Conference on Object-Oriented Programming (ECOOP), July 2015. doi:10.4230/LIPIcs.ECOOP.2015.568 [44] Mark Soper: “Clearing Up React Data Management Confusion with Flux, Redux, and Relay,” medium.com, December 3, 2015. [45] Eno Thereska, Damian Guy, Michael Noll, and Neha Narkhede: “Unifying Stream Processing and Interactive Queries in Apache Kafka,” confluent.io, October 26, 2016. [46] Frank McSherry: “Dataflow as Database,” github.com, July 17, 2016. [47] Peter Alvaro: “I See What You Mean,” at Strange Loop, September 2015. [48] Nathan Marz: “Trident: A High-Level Abstraction for Realtime Computation,” blog.twitter.com, August 2, 2012. [49] Edi Bice: “Low Latency Web Scale Fraud Prevention with Apache Samza, Kafka and Friends,” at Merchant Risk Council MRC Vegas Conference, March 2016. [50] Charity Majors: “The Accidental DBA,” charity.wtf, October 2, 2016. [51] Arthur J.

…

This approach is often used to decompose a large application into smaller services by area of functionality, such that one service makes a request to another when it requires some functionality or data from that other service. This way of building applications has traditionally been called a serviceoriented architecture (SOA), more recently refined and rebranded as microservices architecture [31, 32]. In some ways, services are similar to databases: they typically allow clients to submit and query data. However, while databases allow arbitrary queries using the query lan‐ guages we discussed in Chapter 2, services expose an application-specific API that only allows inputs and outputs that are predetermined by the business logic (applica‐ tion code) of the service [33].

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann

active measures, Amazon Web Services, billion-dollar mistake, bitcoin, blockchain, business intelligence, business logic, business process, c2.com, cloud computing, collaborative editing, commoditize, conceptual framework, cryptocurrency, data science, database schema, deep learning, DevOps, distributed ledger, Donald Knuth, Edward Snowden, end-to-end encryption, Ethereum, ethereum blockchain, exponential backoff, fake news, fault tolerance, finite state, Flash crash, Free Software Foundation, full text search, functional programming, general-purpose programming language, Hacker News, informal economy, information retrieval, Infrastructure as a Service, Internet of things, iterative process, John von Neumann, Ken Thompson, Kubernetes, Large Hadron Collider, level 1 cache, loose coupling, machine readable, machine translation, Marc Andreessen, microservices, natural language processing, Network effects, no silver bullet, operational security, packet switching, peer-to-peer, performance metric, place-making, premature optimization, recommendation engine, Richard Feynman, self-driving car, semantic web, Shoshana Zuboff, social graph, social web, software as a service, software is eating the world, sorting algorithm, source of truth, SPARQL, speech recognition, SQL injection, statistical model, surveillance capitalism, systematic bias, systems thinking, Tragedy of the Commons, undersea cable, web application, WebSocket, wikimedia commons

ISBN: 978-0-122-33435-1 [28] Russell Housley, Warwick Ford, Tim Polk, and David Solo: “RFC 2459: Internet X.509 Public Key Infrastructure: Certificate and CRL Profile,” IETF Network Working Group, Standards Track, January 1999. [29] Lev Walkin: “Question: Extensibility and Dropping Fields,” lionet.info, September 21, 2010. [30] Jesse James Garrett: “Ajax: A New Approach to Web Applications,” adaptivepath.com, February 18, 2005. [31] Sam Newman: Building Microservices. O’Reilly Media, 2015. ISBN: 978-1-491-95035-7 [32] Chris Richardson: “Microservices: Decomposing Applications for Deployability and Scalability,” infoq.com, May 25, 2014. [33] Pat Helland: “Data on the Outside Versus Data on the Inside,” at 2nd Biennial Conference on Innovative Data Systems Research (CIDR), January 2005. [34] Roy Thomas Fielding: “Architectural Styles and the Design of Network-Based Software Architectures,” PhD Thesis, University of California, Irvine, 2000

…

[38] David Gelernter: “Generative Communication in Linda,” ACM Transactions on Programming Languages and Systems (TOPLAS), volume 7, number 1, pages 80–112, January 1985. doi:10.1145/2363.2433 [39] Patrick Th. Eugster, Pascal A. Felber, Rachid Guerraoui, and Anne-Marie Kermarrec: “The Many Faces of Publish/Subscribe,” ACM Computing Surveys, volume 35, number 2, pages 114–131, June 2003. doi:10.1145/857076.857078 [40] Ben Stopford: “Microservices in a Streaming World,” at QCon London, March 2016. [41] Christian Posta: “Why Microservices Should Be Event Driven: Autonomy vs Authority,” blog.christianposta.com, May 27, 2016. [42] Alex Feyerke: “Say Hello to Offline First,” hood.ie, November 5, 2013. [43] Sebastian Burckhardt, Daan Leijen, Jonathan Protzenko, and Manuel Fähndrich: “Global Sequence Protocol: A Robust Abstraction for Replicated Shared State,” at 29th European Conference on Object-Oriented Programming (ECOOP), July 2015. doi:10.4230/LIPIcs.ECOOP.2015.568 [44] Mark Soper: “Clearing Up React Data Management Confusion with Flux, Redux, and Relay,” medium.com, December 3, 2015

…

This approach is often used to decompose a large application into smaller services by area of functionality, such that one service makes a request to another when it requires some functionality or data from that other service. This way of building applications has traditionally been called a service-oriented architecture (SOA), more recently refined and rebranded as microservices architecture [31, 32]. In some ways, services are similar to databases: they typically allow clients to submit and query data. However, while databases allow arbitrary queries using the query languages we discussed in Chapter 2, services expose an application-specific API that only allows inputs and outputs that are predetermined by the business logic (application code) of the service [33].

The Docker Book by James Turnbull

Airbnb, continuous integration, Debian, DevOps, domain-specific language, false flag, fault tolerance, job automation, Kickstarter, Kubernetes, microservices, MVC pattern, platform as a service, pull request, Ruby on Rails, software as a service, standardized shipping container, web application

Fast, efficient development life cycle Docker aims to reduce the cycle time between code being written and code being tested, deployed, and used. It aims to make your applications portable, easy to build, and easy to collaborate on. Encourages service orientated architecture Docker also encourages service-oriented and microservices architectures. Docker recommends that each container run a single application or process. This promotes a distributed application model where an application or service is represented by a series of inter-connected containers. This makes it very easy to distribute, scale, debug and introspect your applications.

…

Containers can be built in development and promoted to testing environments and, in turn, to production. Running stand-alone services and applications consistently across multiple environments, a concept especially useful in service-oriented architectures and deployments that rely heavily on micro-services. Using Docker to create isolated instances to run tests like, for example, those launched by a Continuous Integration (CI) suite like Jenkins CI. Building and testing complex applications and architectures on a local host prior to deployment into a production environment. Building a multi-user Platform-as-a-Service (PAAS) infrastructure.

…

Due to the distributed nature of these applications, service discovery mechanisms also need to be distributed. As they are usually the "glue" between components of distributed applications they also need to be dynamic, reliable, resilient and able to quickly and consistently share data about these services. Docker, with its focus on distributed applications and service-oriented and microservices architectures, is an ideal candidate for integration with a service discovery tool. Each Docker container can register its running service or services with the tool. This provides the information needed, for example an IP address or port or both, to allow interaction between services. Our example service discovery tool, Consul, is a specialized datastore that uses consensus algorithms.

Docker: Up & Running: Shipping Reliable Containers in Production by Sean P. Kane, Karl Matthias

Airbnb, Amazon Web Services, business logic, business process, cloud computing, Colossal Cave Adventure, continuous integration, Debian, DevOps, don't repeat yourself, false flag, interchangeable parts, Kubernetes, loose coupling, Lyft, microservices, revision control, software as a service, source of truth, web application

Achieving this resiliency and scalability requires relying on containerization and eventually container orchestration technologies such as Kubernetes. In addition to these container tools, cloud-native applications are generally built with serviceoriented or microservice architectures. I’m often asked if Docker is replacing virtual machines (VMs), if microservices are a requirement, or if companies should forget about Docker now that serverless patterns are becoming more popular. The answer is always no! Tools in the cloud-native eco‐ xi system are additive, not exclusive. Docker and VMs are not competing with one another, and instead should be used together to achieve maximum benefit.

…

A secondary benefit of shipping applications in Docker containers is that in places where there are multiple applications that talk to each other remotely via something like an API, developers of one application can easily develop against a version of the other service that is currently tagged for the environment they require, like produc‐ tion or staging. Developers on each team don’t have to be experts in how the other service works or is deployed just to do development on their own application. If you expand this to a service-oriented architecture with innumerable microservices, Docker containers can be a real lifeline to developers or QA engineers who need to wade into the swamp of inter-microservice API calls. A common practice in organizations that run Docker containers in production is for automated integration tests to pull down a versioned set of Docker containers for dif‐ ferent services, matching the current deployed versions.

…

Docker and VMs are not competing with one another, and instead should be used together to achieve maximum benefit. Serverless patterns are most powerful when they are used with containers, and I would argue that serverless wouldn’t have pushed to the forefront if ephemeral, lightweight con‐ tainers didn’t enable them to do so. Microservices aren’t a prerequisite for containers either, though you may find that you’ll be able to reap more benefits when your archi‐ tecture allows your services to be smaller, too. Using Docker allows developers to expand their operational responsibilities and take more ownership of what they build. It can remove silos in your organization by mak‐ ing details like dependencies a responsibility of the development team, not solely the operations team.

Clean Agile: Back to Basics by Robert C. Martin

Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Boeing 737 MAX, c2.com, cognitive load, continuous integration, DevOps, disinformation, double entry bookkeeping, en.wikipedia.org, failed state, Frederick Winslow Taylor, index card, iterative process, Kanban, Kubernetes, loose coupling, microservices, remote working, revision control, scientific management, Turing machine

Teams struggle with this even when they don’t need to coordinate with other teams. When coordinating the work of multiple teams, monolithic, centralized, preplanned architectures create massive dependencies between teams that tend to force them to work in lock step, thus defeating much of the promise of Agile. Simple Design, especially when used with practices such as a microservices architecture, enables Agility in the large. The Future of Agile Coaching In the last few years, professional coaching and professional facilitation have started to make their way into the Agile curriculum. The Scrum Alliance’s Advanced Certified Scrum Master (ACSM) course has a few learning objectives related to coaching and facilitation, and their Certified Team Coach (CTC) and Certified Enterprise Coach (CEC) programs require that you acquire even more facilitation and coaching skills than that.

…

If developers are discussing practices that would enable them to build systems in a better way, there is no reason to involve businesspeople. Business should only be involved when there is a significant impact in the cost or duration of the project. There is a difference between re-architecting a whole monolith system into microservices and doing TDD. The former has a very significant impact on the cost and duration of the project; the latter doesn’t, as long as developers are comfortable with the technique. Whether or not developers automate their tests should not be relevant to the business. It should be even less relevant whether the automated tests are written before or after the production code.

…

They have been key in promoting XP, and many other technical practices to many developers and companies around the world. It is through Software Craftsmanship communities that many developers are learning TDD, Continuous Integration, Pair Programming, Simple Design, SOLID principles, Clean Code, and Refactoring. They are also learning how to architect systems using microservices, how to automate their deployment pipelines, and how to migrate their systems to the cloud. They are learning different programming languages and paradigms. They are learning new technologies and different ways to test and maintain their applications. Developers in the Craftsmanship community are creating safe and friendly spaces where they can meet like-minded people and talk about their profession.

Working Backwards: Insights, Stories, and Secrets From Inside Amazon by Colin Bryar, Bill Carr

Amazon Web Services, barriers to entry, Big Tech, Black Lives Matter, business logic, business process, cloud computing, coronavirus, COVID-19, data science, delayed gratification, en.wikipedia.org, fulfillment center, iterative process, Jeff Bezos, late fees, loose coupling, microservices, Minecraft, performance metric, search inside the book, shareholder value, Silicon Valley, six sigma, Steve Jobs, subscription business, Toyota Production System, two-pizza team, web application, why are manhole covers round?

The rolling regeneration of the way we built and deployed technology was a bold move, an expensive investment that stretched over several years of intensive and delicate work. Today the advantages of a microservices-based architecture are well understood, and the approach has been adopted by many tech companies. The benefits include improved agility, developer productivity, scalability, and a better ability to resolve and recover from outages and failures. In addition, with microservices, it becomes possible to establish small, autonomous teams that can assume a level of ownership of their code that isn’t possible with a monolithic approach. The switch to microservices removed the shackles that had prevented the Amazon software teams from moving fast, and enabled the transition to small, autonomous teams.

…

See also Amazon Prime Mechanisms Amazon Leadership Principles and annual planning process compensation plan S-Team goals metrics Amazon flywheel Amazon history and origins of anatomy of metrics chart anecdote and correct, controllable input metrics the deck (data package) DMAIC (Define-Measure-Analyze-Improve-Control) DMAIC analyze stage DMAIC control stage DMAIC define stage DMAIC improve stage DMAIC measure stage Fast Track In Stock and life cycle of output and input metrics pitfall of disaster meetings pitfall of noise obscuring signal weekly and monthly metrics on single graph at Weekly Business Review meetings year-over-year (YOY) trends microservices-based architecture Microsoft Excel MIT Media Lab Mobipocket mock-ups Music 2.0 (digital music industry conference) Napster Narrative Information Multiplier narratives sample narrative tenets and FAQs See also Press Release/Frequently Asked Questions process (PR/FAQ); six-pager NBC NBC Universal Netflix House of Cards (original series) Watch Now (video streaming service) New Project Initiatives (NPI) choosing our priorities with force-ranking our options with News Corp Nichols, Dorothy Nintendo Wii noise obscuring signal Obidos Offer Through Onboarding (Bar Raiser step) 1-Click button operating plan annual planning process OP1 process OP2 O’Reilly, Tim O’Reilly Emerging Technology conference O’Reilly Group Ownership leadership principle Palm Computing Peacock (NBC’s streaming service) performance metrics personal bias Phone Screen (Bar Raiser step) Piacentini, Diego PlayStation PowerPoint (PP) drawbacks of six-pager compared with in S-Team meetings pre-authorization, credit card Press Release/Frequently Asked Questions process (PR/FAQ) Amazon Leadership Principles and Amazon Web Services and dependencies and example of Melinda (Smart Mailbox) FAQ components features and benefits Fire Phone and history and origins of Kindle and narrative forms and PowerPoint compared with press release components price and Prime and Prime Video and process and product Working Backwards and Price, Roy Prime.

Kill It With Fire: Manage Aging Computer Systems by Marianne Bellotti

anti-pattern, barriers to entry, business logic, cloud computing, cognitive bias, computer age, continuous integration, create, read, update, delete, Daniel Kahneman / Amos Tversky, data science, database schema, Dennis Ritchie, DevOps, fault tolerance, fear of failure, Google Chrome, Hans Moravec, iterative process, Ken Thompson, loose coupling, microservices, minimum viable product, Multics, no silver bullet, off-by-one error, platform as a service, pull request, QWERTY keyboard, Richard Stallman, risk tolerance, Schrödinger's Cat, side project, software as a service, Steven Levy, systems thinking, web application, Y Combinator, Y2K

One of the benefits of microservices, for example, is that it allows many teams to contribute to the same system independently from one another. Whereas a monolith would require coordination in the form of code reviews—a personal, direct interaction between colleagues—service-oriented architecture scales the same guarantees with process. Engineers document contracts and protocols; automation is applied to ensure that those contracts are not violated, and it prescribes a course of action if they are. For that reason, engineers who want to “jump ahead” and build something with microservices from the beginning often struggle.

…

See baudot code internet service providers (ISPs), 13 internet, the home vs. work access, 12 pricing, 12–13 iPhones, 10 iteration in place, 55–56 J just culture, 166–168 K Kafka, 7 keyboards, 19 Kohn, Alfie, 164 Kruchten, Philippe, 173 L leap second, 203–205 Legacy Code Rocks, 199 Linux, 22–25, 32, 65 Lipmanowicz, Henri, 135 logical view, 173 looms, weaving, 20 Loopy, 212 Lotus 1-2-3, 61 M magnetic tape, 23 mainframes, 1, 12, 40, 66, 157, 198 comparison to cloud computing 2, 9–11, 17 punch cards, 18 Unisys ClearPath Dorados, 2 maintenance mode, 54 McCandless, Keith, 135 mean time to recovery (MTTR), 113, 220 memory leaks, 196 mere-exposure effect, 22, 34 message queues, 208–209 microservices, 101, 148 Microsoft, 33 Excel, 61 Exchange Server, 67 Internet Explorer, 67 microswitch, 26 middleware, 143 migrations, 65–69, 87, 104 minimum viable product (MVP), 32, 39, 76, 79 mobile phones. See cellphones momentum, 75–90, 117, 122, 130 monoliths, 50–51, 56, 85–87, 101–108, 148 Moravec, Hans, 63 morse code, 20 Mozilla, 204 MTTR (mean time to recovery), 113, 220 Multics, 21 murder boards, 125–127 MVP (minimum viable product), 32, 39, 76, 79 Mythical Man-Month, The, 140, 213 N NASA, 198 NASA’s Ames Research Center, 125 National Science Foundation Network, 10 Netflix, 204 networks, 13 nationalization, 11 nines of availability, 113 normal accidents, 46 O Obama administration, the, 79 objectives and key results (OKRs), 182 object-oriented, 70 object relational mapping (ORM), 105 observability, 52 on-call rotations, 109, 208 Operation Aurora, 119 opportunity costs, 90–94 optimizing, 83, 105 ORM (object relational mapping), 105 overgrowth, 64 P performance, 42–44, 52, 92, 113, 144 Perrow, Charles, 46 personal computer (PC), 10 Pew Research, 5 physical view, 173 Pinterest, 204 platform as a service (PaaS), 69 POSIX, 27 postmortem, 100, 167–168, 187–190 probabilistic outcome-based decision-making, 138 problem setting, 129–130, 159 processing power, 13 process view, 173 programming languages, 36 ALGOL60, 28–31 Assembly, 29, 40, 66 bash, 65 BASIC, 30 BCPL, 28 C, 28, 31 COBOL, 28–31, 39–41, 61, 70 CoffeeScript, 70 CPL, 31 CSS, 150 FORTRAN, 30 HTML, 150 Java, 30, 68, 70 JavaScript, 36, 67, 70, 150 JCL, 65 Lisps, 31 Python, 30, 69 SQL, 65, 105 Typescript, 70 protocols, 67 FTP, 67 HTTP, 209 NTP, 197 SMTP, 67 TCP/IP, 67 TLS/SSL, 206 pull requests, 23 Q Qantas Airways, 204 QWERTY, 27 R railroad tickets, 18 Reddit, 204 refactoring, 51–52, 71, 103 reorgs, 141, 151–152, 156 research institutions, 11 resilience, 112, 169 resilience engineering, 172 responsibility gaps, 99, 207 resulting, 60 retrospectives, 188 reverse engineering, 71 rewrites, 34, 54–55, 145–147 risk, 34, 88, 146, 162–171 Ritchie, Dennis, 23 Robert’s Rules of Order, 193 Rumelt, Richard, 184 S SaaS (software as a service), 95 Salus, Peter, 23 sandbox, 174–175 scaling, 62–63, 78, 110, 149, 195 Schrödinger’s cat, 124 SDK (software development kits), 67 second system syndrome, 33 security, 89 Selectric, 27 Service Dominate Logic (S-D Logic), 8 service level agreements (SLAs), 94 service level objectives (SLOs), 94, 106, 113, 144, 149, 169, 220 service-oriented architecture (SOA), 101, 148 service recovery paradox, 170 shell scripts, 65 Sholes, Christopher Latham, 19 site reliability engineering (SRE), 99, 113, 150, 157, 218 Slack, 102 SLAs (service level agreements), 94 SLOs (service level objectives), 94, 106, 113, 144, 149, 169, 220 SOA (service-oriented architecture), 101, 148 software as a service (SaaS), 95 software development kits (SDK), 67 software renovation, 71 Soule, Samuel W., 19 source code, 23 split in place, 56 Spolsky, Joel, 33, 145 SRE (site reliability engineering), 218 Stack Overflow, 33 Stallman, Richard, 25–26 standards, 11, 66, 77, 103, 107 static analysis, 69, 71–72 stocks, 210 storage capacity, 13 stored procedures, 65 Stricker, Gabriel, 119 success criteria, 83, 182–185 supercomputers, 10 Surprising Power of Liberating Structures, 135 system stability, 89, 169 T tabulating machines, 18–20 technical debt, 38–40, 55, 79, 106, 210–215 telegraphs, 19–21 testing, 51, 55, 57, 70, 85, 109, 124 Texas Instruments, 198 Thompson, Ken, 23 TOPS-10, 197 Torvalds, Linus, 25 trade-offs, 42, 83 transpilers, 69–71 true but irrelevant, 82 trust, 54, 100, 108, 123, 168–170, 219 Twitter, 62, 204 typewriters, 19 U United Nations (UN), 181 United Parcel Service (UPS), 203 University of Cambridge, 31 University of North Carolina at Chapel Hill, 140 Unix, 21–27, 197 UNIX-HATERS Handbook, The, 26 US Army/Marine Corps Counterinsurgency Field Manual, The, 129 US Digital Service (USDS), 68, 144, 164–165 V virtual machines (VM), 49–50, 85–87, 111, 176 W Working Effectively with Legacy Code, 55 working groups, 191–193 World Computer Corporation, 197 Y Y2K, 196, 200 yak shaving, 153 YouTube, 102 Z Zajonc, Robert, 22, 34

An Elegant Puzzle: Systems of Engineering Management by Will Larson

Ben Horowitz, Cass Sunstein, Clayton Christensen, data science, DevOps, en.wikipedia.org, fault tolerance, functional programming, Google Earth, hive mind, Innovator's Dilemma, iterative process, Kanban, Kickstarter, Kubernetes, loose coupling, microservices, MITM: man-in-the-middle, no silver bullet, pull request, Richard Thaler, seminal paper, Sheryl Sandberg, Silicon Valley, statistical model, systems thinking, the long tail, web application

“Design Patterns for Container-Based Distributed Systems” The move to container-based deployment and orchestration has introduced a whole new set of vocabulary, including “sidecars” and “adapters.” This paper provides a survey of the patterns that have evolved over the past decade, as microservices and containers have become increasingly prominent infrastructure components: In the late 1980s and early 1990s, object-oriented programming revolutionized software development, popularizing the approach of building of applications as collections of modular components. Today we are seeing a similar revolution in distributed system development, with the increasing popularity of microservice architectures built from containerized software components. Containers are particularly well-suited as the fundamental “object” in distributed systems by virtue of the walls they erect at the container boundary.

…

Much like the Google File System paper was an inspiration for the Hadoop File System, this paper was itself a major inspiration for Hadoop. “Dapper, a Large-Scale Distributed Systems Tracing Infrastructure” The Dapper paper introduces a performant approach to tracing requests across many services, which has become increasingly relevant as more companies refactor core monolithic applications into dozens or hundreds of micro-services. From the abstract: Here we introduce the design of Dapper, Google’s production distributed systems tracing infrastructure, and describe how our design goals of low overhead, application-level transparency, and ubiquitous deployment on a very large-scale system were met. Dapper shares conceptual similarities with other tracing systems, particularly Magpie and X-Trace, but certain design choices were made that have been key to its success in our environment, such as the use of sampling and restricting the instrumentation to a rather small number of common libraries.

Postcapitalism: A Guide to Our Future by Paul Mason

air traffic controllers' union, Alan Greenspan, Alfred Russel Wallace, bank run, banking crisis, banks create money, Basel III, basic income, Bernie Madoff, Bill Gates: Altair 8800, bitcoin, Bletchley Park, Branko Milanovic, Bretton Woods, BRICs, British Empire, business cycle, business process, butterfly effect, call centre, capital controls, carbon tax, Cesare Marchetti: Marchetti’s constant, Claude Shannon: information theory, collaborative economy, collective bargaining, commons-based peer production, Corn Laws, corporate social responsibility, creative destruction, credit crunch, currency manipulation / currency intervention, currency peg, David Graeber, deglobalization, deindustrialization, deskilling, discovery of the americas, disinformation, Downton Abbey, drone strike, en.wikipedia.org, energy security, eurozone crisis, factory automation, false flag, financial engineering, financial repression, Firefox, Fractional reserve banking, Frederick Winslow Taylor, fulfillment center, full employment, future of work, game design, Glass-Steagall Act, green new deal, guns versus butter model, Herbert Marcuse, income inequality, inflation targeting, informal economy, information asymmetry, intangible asset, Intergovernmental Panel on Climate Change (IPCC), Internet of things, job automation, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Perry Barlow, Joseph Schumpeter, Kenneth Arrow, Kevin Kelly, Kickstarter, knowledge economy, knowledge worker, late capitalism, low interest rates, low skilled workers, market clearing, means of production, Metcalfe's law, microservices, middle-income trap, Money creation, money: store of value / unit of account / medium of exchange, mortgage debt, Network effects, new economy, Nixon triggered the end of the Bretton Woods system, Norbert Wiener, Occupy movement, oil shale / tar sands, oil shock, Paul Samuelson, payday loans, Pearl River Delta, post-industrial society, power law, precariat, precautionary principle, price mechanism, profit motive, quantitative easing, race to the bottom, RAND corporation, rent-seeking, reserve currency, RFID, Richard Stallman, Robert Gordon, Robert Metcalfe, scientific management, secular stagnation, sharing economy, Stewart Brand, structural adjustment programs, supply-chain management, technological determinism, The Future of Employment, the scientific method, The Wealth of Nations by Adam Smith, Transnistria, Twitter Arab Spring, union organizing, universal basic income, urban decay, urban planning, vertical integration, Vilfredo Pareto, wages for housework, WikiLeaks, women in the workforce, Yochai Benkler

Crucially, it would have to create new markets beyond production, in the field of services. The 250-year history of capitalism has been about pushing market forces into sectors where they did not exist before. Info-capitalism would have to take this to its extremes, creating new forms of person-to-person micro-services, paid for using micro-payments, and mainly in the private sector. And finally, for info-capitalism to succeed it would have to find work for the millions of people whose jobs are automated. These could not be in the majority low-paid jobs because the traditional escape mechanism needs labour costs to rise: human life has to become more complex, needing more labour inputs, not fewer, as in the four cyclical upswings described by long-cycle theory.

…

The elements of such a solution are there in modern economies: Apple is the classic price monopolist, Amazon’s business model the classic strategy for capturing externalities; commodity speculation the classic driver of energy and raw material costs above their value; while the rise of personal micro-services – dog minding, nail salons, personal concierges and the like – shows capitalism commercializing activities we used to provide through friendship or informality. But there are clear structural obstacles to making this work. First, the normal escape route – innovation creates expensive new technologies that replace info-tech – is blocked.

…

Rich people are already surrounded by such post-modern servants, but to replace 47 per cent of all jobs this way would require the mass commercialization of ordinary human life. And here’s where you hit the third obstacle – what philosopher André Gorz called the ‘limits of economic rationality’.37 At a certain level, human life and interaction resist commercialization. An economy in which large numbers of people perform micro-services for each other can exist, but as a form of capitalism it would be highly inefficient and intrinsically low-value. You could pay wages for housework, turn all sexual relationships into paid work, mums with their toddlers in the park could charge each other a penny each time they took turns to push the swings.

Site Reliability Engineering: How Google Runs Production Systems by Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy

"Margaret Hamilton" Apollo, Abraham Maslow, Air France Flight 447, anti-pattern, barriers to entry, business intelligence, business logic, business process, Checklist Manifesto, cloud computing, cognitive load, combinatorial explosion, continuous integration, correlation does not imply causation, crowdsourcing, database schema, defense in depth, DevOps, en.wikipedia.org, exponential backoff, fail fast, fault tolerance, Flash crash, George Santayana, Google Chrome, Google Earth, if you see hoof prints, think horses—not zebras, information asymmetry, job automation, job satisfaction, Kubernetes, linear programming, load shedding, loose coupling, machine readable, meta-analysis, microservices, minimum viable product, MVC pattern, no silver bullet, OSI model, performance metric, platform as a service, proprietary trading, reproducible builds, revision control, risk tolerance, side project, six sigma, the long tail, the scientific method, Toyota Production System, trickle-down economics, warehouse automation, web application, zero day

External Factors Affecting SRE External factors have traditionally pressured the SRE organization and its resources in several ways. Google is increasingly following the industry trend of moving toward microservices.1 As a result, both the number of requests for SRE support and the cardinality of services to support have increased. Because each service has a base fixed operational cost, even simple services demand more staffing. Microservices also imply an expectation of lower lead time for deployment, which was not possible with the previous PRR model (which had a lead time of months). Hiring experienced, qualified SREs is difficult and costly.

…

However, the adoption of frameworks is becoming a prominent influence on building production-ready services at Google as well as profoundly expanding the SRE contribution, lowering service management overhead, and improving baseline service quality across the organization. 1 See the Wikipedia page on microservices at http://en.wikipedia.org/wiki/Microservices. 2 Occasionally, there were consulting engagements by SRE teams with some non-onboarded services, but consultations were a best-effort approach and limited in number and scope. 3 The new model of service management changes the SRE staffing model in two ways: (1) because a lot of service technology is common, it reduces the number of required SREs per service; (2) it enables the creation of production platforms with separation of concerns between production platform support (done by SREs) and service-specific business-logic support, which remains with the development team.

…

Significantly lower operational overhead A production platform built on top of frameworks with stronger conventions significantly reduced operational overhead, for the following reasons: It supports strong conformance tests for coding structure, dependencies, tests, coding style guides, and so on. This functionality also improves user data privacy, testing, and security conformance. It features built-in service deployment, monitoring, and automation for all services. It facilitates easier management of large numbers of services, especially micro-services, which are growing in number. It enables much faster deployment: an idea can graduate to fully deployed SRE-level production quality in a matter of days! Universal support by design The constant growth in the number of services at Google means that most of these services can neither warrant SRE engagement nor be maintained by SREs.

Essential Sqlalchemy by Jason Myers, Rick Copeland

create, read, update, delete, database schema, microservices, Ruby on Rails, side project, SQL injection, web application

This encapsulation can make it easy to make database interactions feel more like normal Python code. Most common applications lend themselves to being modeled in this way. It can also be a highly effective way to inject domain-driven design into a legacy application or one with raw SQL statements sprinkled throughout. Microservices also benefit from the abstraction of the underlying database, allowing the developer to focus on just the process being implemented. However, because the ORM is built on top of SQLAlchemy Core, you can use its ability to work with services like Oracle Data Warehousing and Amazon Redshift in the same manner that it interoperates with MySQL.

…

Index A add( ) method (Session), Inserting Data add_column( ) method (Alembic), Building a Migration Manually Alembic, Getting Started with Alembic-Configuring the Migration Environmentdocumentation for, Autogenerating a Migration, Generating SQL empty migration, creating, Generating a Base Empty Migration-Generating a Base Empty Migration installing, Getting Started with Alembic migration environment, configuring, Configuring the Migration Environment-Configuring the Migration Environment migration environment, creating, Creating the Migration Environment-Creating the Migration Environment migration level, determining, Determining a Database’s Migration Level-Determining a Database’s Migration Level migration level, setting, Marking the Database Migration Level migration, autogenerating, Autogenerating a Migration-Autogenerating a Migration migration, building manually, Building a Migration Manually-Building a Migration Manually migration, downgrading, Downgrading Migrations-Downgrading Migrations operations performed by, list of, Building a Migration Manually SQL, generating, Generating SQL-Generating SQL alembic current command, Determining a Database’s Migration Level alembic downgrade command, Downgrading Migrations alembic history command, Determining a Database’s Migration Level alembic init alembic command, Creating the Migration Environment alembic revision command, Generating a Base Empty Migration alembic stamp command, Marking the Database Migration Level alembic.ini file, Creating the Migration Environment, Configuring the Migration Environment alias( ) function, Aliases alias( ) method (Table), Aliases aliases for tables, Aliases-Aliases all( ) method (Session), Querying Data-Conjunctions alter_column( ) method (Alembic), Building a Migration Manually and_( ) function, Conjunctions-Conjunctions, Boolean Operators app factory pattern, Integrating SQLAlchemy with Flask The Architecture of Open Source Applications (Bayer), Where to Go from Here arithmetic operators, in queries, Operators, Operators array slice notation, Limiting association proxies, Association Proxy-Association Proxy association_proxy( ) method, Association Proxy AttributeError exception, AttributeError-AttributeError attributeshybrid, Hybrid Attributes-Hybrid Attributes non-existent, error for, AttributeError-AttributeError pointer to, across relationships, Association Proxy-Association Proxy attrs collection (inspector), Session States autoload argument (Table), Reflecting Individual Tables autoload_with argument (Table), Reflecting Individual Tables automap_base objects, Reflecting a Database with Automap-Reflected Relationships B backref( ) method, Relationships Base objects, Reflecting a Database with Automap-Reflected Relationships Bayer, Mike (developer, SQLAlchemy), Introduction to SQLAlchemy Beaulieu, Alan (Learning SQL), Who This Book Is For begin( ) method (connection), Transactions-Transactions between( ) method (ClauseElement), ClauseElements BIGINT type, Types BigInteger type, Types bitwise logical operators, in queries, Boolean Operators BLOB type, Types bool type, Types Boolean operators, in queries, Boolean Operators, Boolean Operators Boolean type, Types BOOLEAN type, Types bulk_save_objects( ) method (Session), Inserting Data business objects, Choosing Between SQLAlchemy Core and ORM byte type, Types BYTEA type, Types C cast( ) function, Operators chaining queries, Chaining-Chaining, Chaining-Chaining CheckConstraint, Keys and Constraints Chinook database, Reflection ClauseElement objects, ClauseElements, Filtering CLOB type, Types code examplesChinook database, Reflection downloading, Using Code Examples in IPython notebooks, How to Use the Examples permission to use, Using Code Examples Column objects, ResultProxy columnscontrolling for a query, Controlling the Columns in the Query, Controlling the Columns in the Query default value for, Defining Tables via ORM Classes grouping, Grouping, Grouping labeling, in query results, Built-In SQL Functions and Labels, Built-In SQL Functions and Labels required, Defining Tables via ORM Classes resetting on update, Defining Tables via ORM Classes in Table objects, Columns-Columns commit( ) method (Session), Inserting Data-Inserting Data commit( ) method (transaction), Transactions-Transactions comparison operators, in queries, Operators, Operators compile( ) method, Inserting Data concat( ) method (ClauseElement), ClauseElements conditional chaining, Chaining, Chaining conjunctions, in queries, Conjunctions-Conjunctions, Conjunctions connect() method (engine), Connecting to a Database connection string, Connecting to a Database-Connecting to a Database connection timeouts, Connecting to a Database constraintsin ORM classes, Keys, Constraints, and Indexes in Table objects, Keys and Constraints-Keys and Constraints, Relationships and ForeignKeyConstraints-Relationships and ForeignKeyConstraints contact information for this book, How to Contact Us contains( ) method (ClauseElement), ClauseElements conventions used in this book, Conventions Used in This Book Core (see SQLAlchemy Core) count( ) function, Built-In SQL Functions and Labels create_all( ) method (MetaData), Persisting the Tables-Persisting the Tables, Persisting the Schema create_check_constraint( ) method (Alembic), Building a Migration Manually create_engine( ) function, Connecting to a Database-Connecting to a Database create_foreign_key( ) method (Alembic), Building a Migration Manually create_index( ) method (Alembic), Autogenerating a Migration, Building a Migration Manually create_primary_key( ) method (Alembic), Building a Migration Manually create_table( ) method (Alembic), Autogenerating a Migration, Building a Migration Manually create_unique_constraint( ) method (Alembic), Building a Migration Manually custom types, Types D data warehouse, Choosing Between SQLAlchemy Core and ORM DataAccessLayer class, Testing with a Test Database-Testing with a Test Database databasesconnecting to, Connecting to a Database-Connecting to a Database deleting data, Deleting Data, Deleting Data drivers required for, Installing Database Drivers inserting data, Inserting Data-Inserting Data, Deleting Data-Deleting Data, Inserting Data-Inserting Data, Deleting Data-Deleting Data migrations (see Alembic) querying data, Querying Data-Conjunctions, Joins-Raw Queries, Querying Data-Querying Data, Joins-Raw Queries reflecting (see reflection) supported, Introduction to SQLAlchemy, Installing Database Drivers tables in (see tables) updating data, Updating Data, Updating Data Date type, Types DATE type, Types DateTime type, Types DATETIME type, Types datetime.date type, Types datetime.datetime type, Types datetime.time type, Types datetime.timedelta type, Types db.py file, Configuring the Migration Environment Decimal type, Types DECIMAL type, Types decimal.Decimal type, Types declarative classes (see ORM classes) declarative_base objects, Defining Schema with SQLAlchemy ORM-Defining Tables via ORM Classes decorators, Using Mocks delete( ) function, Deleting Data, Deleting Data delete( ) method (Table), Deleting Data, Deleting Data deleting dataSQLAlchemy Core, Deleting Data SQLAlchemy ORM, Deleting Data desc( ) function, Ordering, Ordering detached session state, Session States DetachedInstanceError exception, DetachedInstanceError-DetachedInstanceError distinct( ) method (ClauseElement), ClauseElements domain-driven design, Choosing Between SQLAlchemy Core and ORM downgrade( ) method (Alembic), Generating a Base Empty Migration, Autogenerating a Migration, Building a Migration Manually drop_column( ) method (Alembic), Building a Migration Manually drop_constraint( ) method (Alembic), Building a Migration Manually drop_index( ) method (Alembic), Autogenerating a Migration, Building a Migration Manually drop_table( ) method (Alembic), Autogenerating a Migration, Building a Migration Manually E echo argument (create_engine), Connecting to a Database encoding argument (create_engine), Connecting to a Database endswith( ) method (ClauseElement), ClauseElements engine, creating, Connecting to a Database-Connecting to a Database Enum type, Types ENUM type, Types env.py file, Creating the Migration Environment, Configuring the Migration Environment error handlingAttributeError exception, AttributeError-AttributeError DetachedInstanceError exception, DetachedInstanceError-DetachedInstanceError IntegrityError exception, IntegrityError-IntegrityError, Transactions InvalidRequestError exception, Transactions MultipleResultsFound exception, MultipleResultsFound Exception-MultipleResultsFound Exception SQLAlchemy Core, Exceptions-Handling Errors try/except block, Handling Errors-Handling Errors, MultipleResultsFound Exception, DetachedInstanceError execute( ) method (Alembic), Building a Migration Manually execute( ) method (connection), Inserting Data, Inserting Data expunge( ) method (Session), Session States F fetchall( ) method (ResultProxy), Querying Data fetchone( ) method (ResultProxy), ResultProxy filter( ) function, Filtering filtering query results, Filtering-Conjunctions, Filtering-Filtering filter_by( ) function, Filtering first( ) method (ResultProxy), ResultProxy first( ) method (Session), Querying Data Flask, Integrating SQLAlchemy with Flask-Integrating SQLAlchemy with Flask, Where to Go from Here Flask Web Development (Grinberg), Where to Go from Here Flask-SQLalchemy package, Integrating SQLAlchemy with Flask Float type, Types float type, Types FLOAT type, Types flush( ) method (Session), Inserting Data fonts used in this book, Conventions Used in This Book foreign keysdefining, Relationships and ForeignKeyConstraints-Relationships and ForeignKeyConstraints, Relationships reflecting, Reflecting Individual Tables ForeignKeyConstraint, Relationships and ForeignKeyConstraints-Relationships and ForeignKeyConstraints functional testing, Testing with a Test Database-Testing with a Test Database, Testing with a Test Database-Testing with a Test Database G generative statement building, Inserting Data, Inserting Data, Ordering generic types, Types-Types Grinberg, Miguel (Flask Web Development), Where to Go from Here grouping in queries, Grouping, Grouping group_by( ) method (Table), Grouping, Grouping H hybrid attributes, Hybrid Attributes-Hybrid Attributes I icons used in this book, Conventions Used in This Book ilike( ) method (ClauseElement), ClauseElements indexesin Table objects, Indexes __init__( ) method, Association Proxy __init__.py file, Configuring the Migration Environment, Integrating SQLAlchemy with Flask insert( ) function, Inserting Data, Deleting Data insert( ) method (Table), Inserting Data-Inserting Data, Deleting Data inserted_primary_key( ) method (ResultProxy), Inserting Data inserting dataSQLAlchemy Core, Inserting Data-Inserting Data, Deleting Data-Deleting Data SQLAlchemy ORM, Inserting Data-Inserting Data, Deleting Data-Deleting Data inspect( ) method, Session States Int type, Types int type, Types Integer type, Types INTEGER type, Types IntegrityError exception, IntegrityError-IntegrityError, Transactions Interval type, Types INTERVAL type, Types Introducting Python (Lubanovic), Who This Book Is For Introduction to Python (videos, McKellar), Who This Book Is For InvalidRequestError exception, Transactions in_( ) method (ClauseElement), ClauseElements IPython, How to Use the Examples isolation level, setting, Connecting to a Database isolation_level argument (create_engine), Connecting to a Database is_( ) method (ClauseElement), ClauseElements J join( ) method (Table), Joins, Joins joins in queries, Joins-Joins, Joins-Joins K keys (see foreign keys; primary keys) keys( ) method (ResultProxy), ResultProxy L label( ) function, Built-In SQL Functions and Labels, Built-In SQL Functions and Labels LargeBinary type, Types Learning SQL (Beaulieu), Who This Book Is For like( ) method (ClauseElement), ClauseElements, Filtering limit( ) function, Limiting, Limiting limiting query results, Limiting, Limiting logging database actions, Connecting to a Database Lubanovic, Bill (Introducing Python), Who This Book Is For M McKellar, Jessica (Introduction to Python, videos), Who This Book Is For MetaData objects, Metadata MetaData.tables dictionary, Metadata microservices, Choosing Between SQLAlchemy Core and ORM migrations (see Alembic) mock library, Using Mocks, Using Mocks mocks, for testing, Using Mocks-Using Mocks, Using Mocks-Using Mocks modified property (inspector), Session States MultipleResultsFound exception, MultipleResultsFound Exception-MultipleResultsFound Exception MySQLconnecting to, Connecting to a Database connection timeouts, Connecting to a Database drivers for, Installing Database Drivers versions supported, Installing Database Drivers N not_( ) function, Conjunctions, Conjunctions not…( ) methods (ClauseElement), ClauseElements Numeric type, Types NUMERIC type, Types O one( ) method (Session), Querying Data, MultipleResultsFound Exception operators, in queries, Operators-Boolean Operators, Operators-Boolean Operators ordering query results, Ordering-Ordering, Ordering order_by( ) function, Ordering-Ordering, Ordering ORM (see SQLAlchemy ORM) ORM classesconstraints in, Keys, Constraints, and Indexes defining tables using, Defining Tables via ORM Classes-Keys, Constraints, and Indexes deleting data, Deleting Data inserting data, Inserting Data-Inserting Data, Deleting Data-Deleting Data keys in, Keys, Constraints, and Indexes persisting, Persisting the Schema querying data, Querying Data-Querying Data, Joins-Raw Queries relationships between, Relationships-Relationships, Deleting Data-Deleting Data, Reflected Relationships updating data, Updating Data or_( ) function, Conjunctions-Conjunctions, Boolean Operators outerjoin( ) method (Table), Joins, Joins P patch( ) method (mock), Using Mocks pending session state, Session States persistent session state, Session States persistingORM classes, Persisting the Schema Table objects, Persisting the Tables-Persisting the Tables pip install command, Installing SQLAlchemy and Connecting to a Database, Getting Started with Alembic pool_recycle argument (create_engine), Connecting to a Database PostgreSQLconnecting to, Connecting to a Database drivers for, Installing Database Drivers primary keysdefining, Keys and Constraints, Keys, Constraints, and Indexes determining, for inserted record, Inserting Data PrimaryKeyConstraint, Keys and Constraints Psycopg2 driver, Installing Database Drivers PyMySQL driver, Installing Database Drivers pytest, resources for, Where to Go from Here PythonREPL for, How to Use the Examples resources for, Who This Book Is For versions supported, Installing SQLAlchemy and Connecting to a Database Python DBAPI (PEP-249), Installing Database Drivers PyVideo website, Where to Go from Here Q query( ) method (Session), Querying Data-Conjunctions querying datareflected objects, Query Building with Reflected Objects-Query Building with Reflected Objects SQLAlchemy Core, Querying Data-Conjunctions, Joins-Raw Queries SQLAlchemy ORM, Querying Data-Conjunctions, Joins-Raw Queries querying reflected objects, Reflecting a Database with Automap R raw queries, Raw Queries, Raw Queries read, eval, print loop (REPL), How to Use the Examples REAL type, Types reflect( ) method (Metadata), Reflecting a Whole Database reflectionAutomap, Reflecting a Database with Automap-Reflected Relationships SQLAcodegen, SQLAcodegen-SQLAcodegen SQLAlchemy Core, Reflection-Reflecting a Whole Database SQLAlchemy ORM, Reflection with SQLAlchemy ORM and Automap relationship( ) method, Relationships, Joins remote_side option, Joins rename_table( ) method (Alembic), Building a Migration Manually, Building a Migration Manually REPL (read, eval, print loop), How to Use the Examples __repr__( ) method, The Session-The Session resources, Who This Book Is For, How to Contact Us, Where to Go from Here(see also website resources) Flask Web Development (Grinberg), Where to Go from Here Introducing Python (Lubanovic), Who This Book Is For Introduction to Python (videos, McKellar), Who This Book Is For IPython, How to Use the Examples Learning SQL (Beaulieu), Who This Book Is For ResultProxy objects, ResultProxy-ResultProxy rollback( ) method (Session), Transactions-Transactions rollback( ) method (transaction), Transactions-Transactions S scalar( ) function, Built-In SQL Functions and Labels scalar( ) method (ResultProxy), ResultProxy scalar( ) method (Session), Querying Data schema, Choosing Between SQLAlchemy Core and ORM, Defining Schema with SQLAlchemy ORM script.py.mako file, Creating the Migration Environment select( ) function, Querying Data, Controlling the Columns in the Query select( ) method (Table), Querying Data select_from( ) method (Table), Joins Session class, The Session-The Session sessionmaker class, The Session-The Session sessions, The Session-The Sessionstates of, Session States-Session States as transactions, The Session, Inserting Data-Inserting Data, Transactions-Transactions SMALLINT type, Types SQLgenerating for migration, Generating SQL-Generating SQL raw queries using, Raw Queries, Raw Queries resources for, Who This Book Is For SQL Expression Language, SQLAlchemy Core and the SQL Expression Language(see also SQLAlchemy Core) SQL functions, using in queries, Built-In SQL Functions and Labels, Built-In SQL Functions and Labels-Built-In SQL Functions and Labels SQLAcodegen, SQLAcodegen-SQLAcodegen sqlacodegen command, SQLAcodegen, SQLAcodegen SQLAlchemy, Introduction to SQLAlchemy-Choosing Between SQLAlchemy Core and ORMC extensions for, disabling, Installing SQLAlchemy and Connecting to a Database documentation for, Where to Go from Here engine, creating, Connecting to a Database-Connecting to a Database installing, Installing SQLAlchemy and Connecting to a Database-Installing Database Drivers plug-ins and extensions, Where to Go from Here SQLAlchemy Corecompared to ORM, Choosing Between SQLAlchemy Core and ORM-Choosing Between SQLAlchemy Core and ORM deleting data, Deleting Data error handling, Exceptions-Handling Errors inserting data, Inserting Data-Inserting Data, Deleting Data-Deleting Data querying data, Querying Data-Conjunctions, Joins-Raw Queries tables represented in (see Table objects) testing, Testing-Using Mocks transactions, Transactions-Transactions updating data, Updating Data using in combination with ORM, Choosing Between SQLAlchemy Core and ORM SQLAlchemy ORM, ORMcompared to Core, Choosing Between SQLAlchemy Core and ORM-Choosing Between SQLAlchemy Core and ORM deleting data, Deleting Data error handling, MultipleResultsFound Exception-DetachedInstanceError inserting data, Inserting Data-Inserting Data, Deleting Data-Deleting Data querying data, Querying Data-Conjunctions, Joins-Raw Queries reflection, Reflection with SQLAlchemy ORM and Automap sessions, The Session-The Session, The SQLAlchemy Session-Session States tables represented in (see ORM classes) testing, Testing with SQLAlchemy ORM-Using Mocks transactions, The Session, Inserting Data-Inserting Data, Transactions-Transactions updating data, Updating Data using in combination with Core, Choosing Between SQLAlchemy Core and ORM sqlalchemy.dialects module, Types sqlalchemy.func module, Built-In SQL Functions and Labels sqlalchemy.sql.func module, Built-In SQL Functions and Labels sqlalchemy.types module, Types SQLCompiler object, Inserting Data SQLiteconnecting to, Connecting to a Database version supported, Installing Database Drivers standard SQL types, Types startswith( ) method (ClauseElement), ClauseElements str type, Types, Types STRING type, Types strings, concatenating, Operators, Operators sum( ) function, Built-In SQL Functions and Labels, Built-In SQL Functions and Labels T __table__ attribute, Defining Tables via ORM Classes Table objects, Schema and Types, Tables-Relationships and ForeignKeyConstraintsconstraints in, Keys and Constraints-Keys and Constraints, Relationships and ForeignKeyConstraints-Relationships and ForeignKeyConstraints constructing, Tables deleting data, Deleting Data indexes in, Indexes inserting data, Inserting Data-Inserting Data, Deleting Data-Deleting Data keys in, Keys and Constraints persisting, Persisting the Tables-Persisting the Tables querying data, Querying Data-Conjunctions, Joins-Raw Queries relationships between, Relationships and ForeignKeyConstraints-Relationships and ForeignKeyConstraints updating data, Updating Data __tablename__ attribute, Defining Tables via ORM Classes-Defining Tables via ORM Classes tablescolumns in, Columns-Columns declarative classes representing (see ORM classes) reflecting all in a database, Reflecting a Whole Database reflecting individually, Reflecting Individual Tables-Reflecting Individual Tables representations of, Schema and Types user-defined (see Table objects) __table_args__ attribute, Keys, Constraints, and Indexes testingSQLAlchemy Core, Testing-Using Mocks SQLAlchemy ORM, Testing with SQLAlchemy ORM-Using Mocks Text type, Types TEXT type, Types, Types text( ) function, Raw Queries Time type, Types transaction objects, Transactions-Transactions transactionsSQLAlchemy Core, Transactions-Transactions SQLAlchemy ORM, The Session, Inserting Data-Inserting Data, Transactions-Transactions transient session state, Session States try/except block, Handling Errors-Handling Errors, MultipleResultsFound Exception, DetachedInstanceError types, Types-Typescustom types, Types generic, Types-Types standard SQL types, Types vendor-specific types, Types typographical conventions used in this book, Conventions Used in This Book U Unicode type, Types unicode type, Types UNICODE type, Types UniqueConstraint, Keys and Constraints Unit of Work pattern, Inserting Data unit testing, Testing with a Test Database-Testing with a Test Database, Testing with a Test Database-Testing with a Test Database unittest module, Testing with a Test Database-Testing with a Test Database, Testing with a Test Database-Testing with a Test Database update( ) function, Updating Data, Updating Data update( ) method (Table), Updating Data, Updating Data updating dataSQLAlchemy Core, Updating Data SQLAlchemy ORM, Updating Data upgrade( ) method (Alembic), Generating a Base Empty Migration, Autogenerating a Migration, Building a Migration Manually V values( ) function, Inserting Data VARCHAR type, Types vendor-specific types, Types versions directory, Creating the Migration Environment W website resources, How to Contact Us, Where to Go from Here(see also resources) Alembic documentation, Autogenerating a Migration, Generating SQL The Architecture of Open Source Applications (Bayer), Where to Go from Here association proxies, Association Proxy bulk operations, Inserting Data Chinook database, Reflection code examples, Using Code Examples hybrid attributes, Hybrid Attributes IPython, How to Use the Examples plug-ins and extensions, Where to Go from Here Psycopg2 driver, Installing Database Drivers pytest, Where to Go from Here PyVideo, Where to Go from Here reflected relationships, Reflected Relationships SQLAcodegen, SQLAcodegen SQLAlchemy documentation, Where to Go from Here Zen of Python, Connecting to a Database where( ) function, Filtering-Conjunctions Z Zen of Python (website), Connecting to a Database About the Authors Jason Myers works at Cisco as a Software Engineer working on OpenStack.

WTF?: What's the Future and Why It's Up to Us by Tim O'Reilly

"Friedman doctrine" OR "shareholder theory", 4chan, Affordable Care Act / Obamacare, Airbnb, AlphaGo, Alvin Roth, Amazon Mechanical Turk, Amazon Robotics, Amazon Web Services, AOL-Time Warner, artificial general intelligence, augmented reality, autonomous vehicles, barriers to entry, basic income, behavioural economics, benefit corporation, Bernie Madoff, Bernie Sanders, Bill Joy: nanobots, bitcoin, Blitzscaling, blockchain, book value, Bretton Woods, Brewster Kahle, British Empire, business process, call centre, Capital in the Twenty-First Century by Thomas Piketty, Captain Sullenberger Hudson, carbon tax, Carl Icahn, Chuck Templeton: OpenTable:, Clayton Christensen, clean water, cloud computing, cognitive dissonance, collateralized debt obligation, commoditize, computer vision, congestion pricing, corporate governance, corporate raider, creative destruction, CRISPR, crowdsourcing, Danny Hillis, data acquisition, data science, deep learning, DeepMind, Demis Hassabis, Dennis Ritchie, deskilling, DevOps, Didi Chuxing, digital capitalism, disinformation, do well by doing good, Donald Davies, Donald Trump, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, fake news, Filter Bubble, Firefox, Flash crash, Free Software Foundation, fulfillment center, full employment, future of work, George Akerlof, gig economy, glass ceiling, Glass-Steagall Act, Goodhart's law, Google Glasses, Gordon Gekko, gravity well, greed is good, Greyball, Guido van Rossum, High speed trading, hiring and firing, Home mortgage interest deduction, Hyperloop, income inequality, independent contractor, index fund, informal economy, information asymmetry, Internet Archive, Internet of things, invention of movable type, invisible hand, iterative process, Jaron Lanier, Jeff Bezos, jitney, job automation, job satisfaction, John Bogle, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John Zimmer (Lyft cofounder), Kaizen: continuous improvement, Ken Thompson, Kevin Kelly, Khan Academy, Kickstarter, Kim Stanley Robinson, knowledge worker, Kodak vs Instagram, Lao Tzu, Larry Ellison, Larry Wall, Lean Startup, Leonard Kleinrock, Lyft, machine readable, machine translation, Marc Andreessen, Mark Zuckerberg, market fundamentalism, Marshall McLuhan, McMansion, microbiome, microservices, minimum viable product, mortgage tax deduction, move fast and break things, Network effects, new economy, Nicholas Carr, Nick Bostrom, obamacare, Oculus Rift, OpenAI, OSI model, Overton Window, packet switching, PageRank, pattern recognition, Paul Buchheit, peer-to-peer, peer-to-peer model, Ponzi scheme, post-truth, race to the bottom, Ralph Nader, randomized controlled trial, RFC: Request For Comment, Richard Feynman, Richard Stallman, ride hailing / ride sharing, Robert Gordon, Robert Metcalfe, Ronald Coase, Rutger Bregman, Salesforce, Sam Altman, school choice, Second Machine Age, secular stagnation, self-driving car, SETI@home, shareholder value, Silicon Valley, Silicon Valley startup, skunkworks, Skype, smart contracts, Snapchat, Social Responsibility of Business Is to Increase Its Profits, social web, software as a service, software patent, spectrum auction, speech recognition, Stephen Hawking, Steve Ballmer, Steve Jobs, Steven Levy, Stewart Brand, stock buybacks, strong AI, synthetic biology, TaskRabbit, telepresence, the built environment, the Cathedral and the Bazaar, The future is already here, The Future of Employment, the map is not the territory, The Nature of the Firm, The Rise and Fall of American Growth, The Wealth of Nations by Adam Smith, Thomas Davenport, Tony Fadell, Tragedy of the Commons, transaction costs, transcontinental railway, transportation-network company, Travis Kalanick, trickle-down economics, two-pizza team, Uber and Lyft, Uber for X, uber lyft, ubercab, universal basic income, US Airways Flight 1549, VA Linux, warehouse automation, warehouse robotics, Watson beat the top human players on Jeopardy!, We are the 99%, web application, Whole Earth Catalog, winner-take-all economy, women in the workforce, Y Combinator, yellow journalism, zero-sum game, Zipcar

But modern web applications may well be servicing millions of simultaneous users, and their functions have been decomposed into what are now called “microservices”—collections of individual functional building blocks that each do one thing, and do it very well. If a traditional monolithic application like Microsoft Word were reimplemented as a set of microservices, you could easily swap out the spell-checker for a better one, or add a new service that would turn web links into footnotes, or the reverse. Microservices are an evolution of the communications-oriented design pattern that we saw in the design of Unix and the Internet, and in Jeff Bezos’s platform memo. Microservices are defined by their inputs and outputs—how they communicate with other services—not by their internal implementation.

Ansible for DevOps: Server and Configuration Management for Humans by Jeff Geerling

Abraham Maslow, AGPL, Amazon Web Services, cloud computing, continuous integration, database schema, Debian, defense in depth, DevOps, fault tolerance, Firefox, full text search, Google Chrome, inventory management, loose coupling, microservices, Minecraft, MITM: man-in-the-middle, punch-card reader, Ruby on Rails, web application

Zero-downtime multi-server deployments A single server deployment strategy is all that’s needed for many applications, but if you need to run an application on multiple servers for horizontal scalability or redundancy, deployments can be cumbersome—but not when you use Ansible! Server Check.in is a simple server and website monitoring service that has a microservices-based architecture; there is a website, an API application, and a server checking application. The server checking application needs to run on a variety of servers hosted around the world by different providers to provide redundancy and reliability. Server Check.in uses Ansible to manage rolling deployments for this application, so new code can be deployed across all the servers in minutes while maintaining 100% uptime!

…

The basic concept involves bringing up a parallel production infrastructure, then switching over to it. The cutover may take only a few milliseconds and no active production infrastructure is ever offline during the deployment process. A few different technologies and concepts, like container-based infrasturcture and microservices (which are faster to deploy), and better cloud autoscaling and load balancing options, have made blue-green deployments much easier than in the past. This book won’t go through a detailed example of this style of deployment, as the process is similar to other examples provided, the only difference being an additional task of switching a load balancer from the old to the new infrastructure once it’s up and running.

Architecting Modern Data Platforms: A Guide to Enterprise Hadoop at Scale by Jan Kunigk, Ian Buss, Paul Wilkinson, Lars George

Amazon Web Services, barriers to entry, bitcoin, business intelligence, business logic, business process, cloud computing, commoditize, computer vision, continuous integration, create, read, update, delete, data science, database schema, Debian, deep learning, DevOps, domain-specific language, fault tolerance, Firefox, FOSDEM, functional programming, Google Chrome, Induced demand, information security, Infrastructure as a Service, Internet of things, job automation, Kickstarter, Kubernetes, level 1 cache, loose coupling, microservices, natural language processing, Network effects, platform as a service, single source of truth, source of truth, statistical model, vertical integration, web application

Pods are guaranteed to run on the same Linux host, and they provide a means to group several microservices into a single colocated service unit. Kubernetes allows you to easily deploy and manage many pods as large-scale distributed applications across multiple physical servers, which it refers to as nodes. The plurality of all nodes and pods form a Kubernetes cluster. OpenShift can significantly simplify the process of building applications and platform-as-a-service environments. For many enterprises, OpenShift and Kubernetes pave the way toward scalable microservices. They are actually used to build public clouds as well, like Deutsche Telekom’s AppAgile service.

…

Summary The ease of containerized application pods combined with horizontal scalability makes OpenShift intriguing for rapid development/deployment cycles and DevOps-driven environments. The current focus of OpenShift is to excel in scalability and provide orchestration for microservices. Hadoop is typically not a good fit for microservices, but that being said, OpenShift comprises all the required mechanisms to cloudify Hadoop. What is essentially missing is a cohesive component that automates Hadoop deployments and uses the existing affinity/anti-affinity mechanisms for Hadoop compute and storage. In addition, Hadoop distributor support for running frameworks such as Spark on Docker and Kubernetes has not materialized yet, though it is likely to emerge soon.

Software Engineering at Google: Lessons Learned From Programming Over Time by Titus Winters, Tom Manshreck, Hyrum Wright

anti-pattern, computer vision, continuous integration, defense in depth, en.wikipedia.org, functional programming, Jevons paradox, job automation, loss aversion, microservices, reproducible builds, supply-chain attack, transaction costs, Turing complete

This question can arise at many levels of the software stack, because it is regularly the case that a bespoke solution customized for your narrow problem space may outperform the general utility solution that needs to handle all possibilities. By forking or reimplementing utility code and customizing it for your narrow domain you can add new features with greater ease, or optimize with greater certainty, regardless of whether we are talking about a microservice, an in-memory cache, a compression routine, or anything else in our software ecosystem. Perhaps more importantly, the control you gain from such a fork isolates you from changes in your underlying dependencies: those changes aren’t dictated by another team or third-party provider. You are in control of how and when to react to the passage of time and necessity to change.

Docker in Action by Jeff Nickoloff, Stephen Kuenzli

air gap, Amazon Web Services, cloud computing, computer vision, continuous integration, database schema, Debian, end-to-end encryption, exponential backoff, fail fast, failed state, information security, Kubernetes, microservices, MITM: man-in-the-middle, peer-to-peer, software as a service, web application

Finally, containers and networking have had a complicated history. In the last few years, that story became just a little bit more complicated with the emergence of service mesh platforms and other complementary technologies. A service mesh is a platform of application-aware smart pipes that provide microservice networking best practices out of the box. They use proxies to provide point-to-point encryption, authentication, authorization, circuit-breakers, and advanced request routing. The container networking fundamentals presented in this book should prove useful in understanding and evaluating service mesh technologies.

…

That means it’s a bad idea to blindly run customer-provided containers in a co-located environment. 1.5. Docker in the larger ecosystem Today the greater container ecosystem is rich with tooling that solves new or higher-level problems. Those problems include container orchestration, high-availability clustering, microservice life cycle management, and visibility. It can be tricky to navigate that market without depending on keyword association. It is even trickier to understand how Docker and those products work together. Those products work with Docker in the form of plugins or provide a certain higher-level functionality and depend on Docker.

The Art of Monitoring by James Turnbull

Amazon Web Services, anti-pattern, business logic, cloud computing, continuous integration, correlation does not imply causation, Debian, DevOps, domain-specific language, failed state, functional programming, Kickstarter, Kubernetes, microservices, performance metric, pull request, Ruby on Rails, single source of truth, software as a service, source of truth, web application, WebSocket

Tracing Before we move on from this chapter it would be remiss not to mention another diagnostic tool available for application monitoring: tracing, and more specifically, distributed tracing. This is a technique most useful for identifying performance problems and latency in distributed systems or microservices architectures. We're not going to cover it in the book but you should be aware of it if you go down the microservices path. A good example of a distributed tracing tool is Twitter's Zipkin project. Zipkin is in turn based on Google's Dapper paper on tracing. Summary In this chapter we explored several ways to monitor and instrument our applications and their workflow, including understanding where to place our application monitoring.

Modern Vim: Craft Your Development Environment With Vim 8 and Neovim by Drew Neil

bash_history, Bram Moolenaar, data science, Debian, DevOps, en.wikipedia.org, functional programming, microservices, pull request, remote working, text mining

Second Edition A single dramatic software failure can cost a company millions of dollars—but can be avoided with simple changes to design and architecture. This new edition of the best-selling industry standard shows you how to create systems that run longer, with fewer failures, and recover better when bad things happen. New coverage includes DevOps, microservices, and cloud-native architecture. Stability antipatterns have grown to include systemic problems in large-scale systems. This is a must-have pragmatic guide to engineering for production systems. Michael Nygard (376 pages) ISBN: 9781680502398 $47.95 Adopting Elixir Adoption is more than programming.

Mastering Ansible by Jesse Keating

cloud computing, Debian, DevOps, don't repeat yourself, fail fast, microservices, remote working

The final two plays will just have skipped tasks: [ 150 ] Chapter 6 Now, let's execute the playbook again, and this time, we'll define the upgrade as true at runtime: [ 151 ] Minimizing Downtime with Rolling Deployments This time, we can see that our handlers are skipped but the final two plays have tasks that execute. In a real world scenario, where many more things are happening in the microA and microB roles, and potentially other micro-service roles on other hosts, the difference could be of many minutes or more. Clustering the restarts at the end can reduce the interruption period significantly. Running destructive tasks only once Destructive tasks come in many flavors. They can be one-way tasks that are extremely difficult to roll back, one-time tasks that cannot easily be rerun, or they can be race condition tasks that, if performed in parallel, would result in catastrophic failure.

Making Work Visible: Exposing Time Theft to Optimize Workflow by Dominica Degrandis, Tonianne Demaria

cloud computing, cognitive bias, cognitive load, DevOps, Elon Musk, en.wikipedia.org, informal economy, Jeff Bezos, Kanban, loose coupling, microservices, Parkinson's law, Sheryl Sandberg, sunk-cost fallacy, systems thinking, TED Talk, transaction costs, two-pizza team

., test teams within a project team are measured by the number of software bugs), whereas product teams are measured by the business value derived. Senior Director of Technology at Pivotal Cornelia Davis noted in conversation at the 2017 DOES Forum, “Architecture is so tied to how we do our work. The preferred architecture is loosely coupled components, individual microservices, built by individual teams—autonomous product teams, not project teams.”1 EXERCISE The “Oh, By the Way” Dependency Matrix PURPOSE: To bring visibility to dependencies across teams, to help people anticipate what’s headed their way, and to prevent delays from unknown or invisible dependencies.

Demystifying Smart Cities by Anders Lisdorf

3D printing, artificial general intelligence, autonomous vehicles, backpropagation, behavioural economics, Big Tech, bike sharing, bitcoin, business intelligence, business logic, business process, chief data officer, circular economy, clean tech, clean water, cloud computing, computer vision, Computing Machinery and Intelligence, congestion pricing, continuous integration, crowdsourcing, data is the new oil, data science, deep learning, digital rights, digital twin, distributed ledger, don't be evil, Elon Musk, en.wikipedia.org, facts on the ground, Google Glasses, hydroponic farming, income inequality, information security, Infrastructure as a Service, Internet of things, Large Hadron Collider, Masdar, microservices, Minecraft, OSI model, platform as a service, pneumatic tube, ransomware, RFID, ride hailing / ride sharing, risk tolerance, Salesforce, self-driving car, smart cities, smart meter, software as a service, speech recognition, Stephen Hawking, Steve Jobs, Steve Wozniak, Stuxnet, Thomas Bayes, Turing test, urban sprawl, zero-sum game

These properties cannot be seen directly from the data but supplement the data and describes how to interpret and use the data. Both spheres are equally important in order to have a holistic concept of data value. Figure 10-1The seven dimensions of data value The data sphere Consumption – “How can the data be accessed?” For many years, the default answer has been microservices or another type of API as the way to access data. These are ways to programmatically interact with application functionality from other applications in a Request-Response type of interaction. The consuming application calls the API and requests a piece of data, and the API returns the data. The problem is if you are building a machine learning solution and need to loop through millions of entities as part of the process, this may not be the optimal solution.

API Design Patterns by Jj Geewax

Amazon Web Services, anti-pattern, bitcoin, blockchain, business logic, cognitive load, continuous integration, COVID-19, database schema, en.wikipedia.org, exponential backoff, imposter syndrome, Internet of things, Kubernetes, lateral thinking, loose coupling, machine readable, microservices, natural language processing, Paradox of Choice, ride hailing / ride sharing, social graph, sorting algorithm

: any; } interface GetOperationRequest { id: string; } interface ListOperationsRequest { filter: string; } interface ListOperationsResponse<ResultT, MetadataT> { results: Operation<ResultT, MetadataT>[]; } interface WaitOperationRequest { id: string; } interface CancelOperationRequest { id: string; } interface PauseOperationRequest { id: string; } interface ResumeOperationRequest { id: string; } interface AnalyzeMessagesRequest { parent: string; } interface MessageAnalysis { chatRoom: string; messageCount: number; participantCount: number; userGradeLevels: map<string, number>; } interface AnalyzeMessagesMetadata { chatRoom: string; paused: boolean; messagesProcessed: number; messagesCounted: number; } 10.4 Trade-offs Clearly there are many ways to handle API calls that might take a while. For example, one of the simplest also happens to be the easiest to manage for clients: just make the request take as long as it needs. The trade-off with these more simple options is that they don’t lend themselves very well to distributed architecture (e.g., one microservice to initiate the work, another to monitor its progress). Another trade-off is that it can become a bit more complicated to actually monitor the progress of an API call that hangs until completion. While the API can push data over time, if the connection is broken for any reason (perhaps a process crashes, a network connection is interrupted, etc.), we lose our ability to resume monitoring that progress.

…

Luckily, since versions are almost completely frozen after they are made available to users we can often freeze and continue to run a binary, but the overhead can still be quite daunting. That said, modern infrastructure orchestration and management systems (e.g., Kubernetes; https://kubernetes.io/) and modern development paradigms (e.g., microservices) can help to alleviate much of the pain brought about by this problem, so relying on these tools can help quite a bit. That said, it’s not necessarily required that versions always live forever. On the contrary, there’s nothing wrong with defining a deprecation policy and stating from the time a version is made available how long it’s expected to be maintained and continue running.

Forge Your Future with Open Source by VM (Vicky) Brasseur

AGPL, anti-pattern, Benevolent Dictator For Life (BDFL), call centre, continuous integration, Contributor License Agreement, Debian, DevOps, don't repeat yourself, en.wikipedia.org, Firefox, FOSDEM, Free Software Foundation, Guido van Rossum, information security, Internet Archive, Larry Wall, microservices, Perl 6, premature optimization, pull request, Richard Stallman, risk tolerance, Turing machine

Second Edition A single dramatic software failure can cost a company millions of dollars—but can be avoided with simple changes to design and architecture. This new edition of the best-selling industry standard shows you how to create systems that run longer, with fewer failures, and recover better when bad things happen. New coverage includes DevOps, microservices, and cloud-native architecture. Stability antipatterns have grown to include systemic problems in large-scale systems. This is a must-have pragmatic guide to engineering for production systems. Michael Nygard (376 pages) ISBN: 9781680502398 $47.95 Your Code as a Crime Scene Jack the Ripper and legacy codebases have more in common than you’d think.

The Manager’s Path by Camille Fournier

Big Tech, emotional labour, failed state, fear of failure, hiring and firing, hive mind, interchangeable parts, job automation, Kanban, Larry Wall, microservices, pull request, risk tolerance, Schrödinger's Cat, side project, Steve Jobs, WebSocket, work culture

When I joined, the engineering team was divided into roughly two groups: storefront, which did all development for the customer-facing website, and warehouse, which supported the software that ran the warehouse operations. We quickly evolved storefront to be frontend and backend because we were rewriting BOOTSTRAPPING CULTURE | 209 the code from a PHP monolith to a Java- and Ruby-based microservices architecture. Toward the end of my first year, we ran an experiment. We had a new product we wanted to build for the customer, a feature based on our customer photo reviews. Because finding a dress that would fit well was a challenge for our customers, we wanted to enable shoppers to see photos that other customers had uploaded showing themselves in the dresses, along with customer-provided information about their normal size, height, weight, and “shape” (athletic, pear, curvy, etc.).

Brotopia: Breaking Up the Boys' Club of Silicon Valley by Emily Chang

"Margaret Hamilton" Apollo, "Susan Fowler" uber, "World Economic Forum" Davos, 23andMe, 4chan, Ada Lovelace, affirmative action, Airbnb, Alan Greenspan, Andy Rubin, Apollo 11, Apple II, augmented reality, autism spectrum disorder, autonomous vehicles, barriers to entry, Benchmark Capital, Bernie Sanders, Big Tech, Burning Man, California gold rush, Chuck Templeton: OpenTable:, clean tech, company town, data science, David Brooks, deal flow, Donald Trump, Dr. Strangelove, driverless car, Elon Musk, emotional labour, equal pay for equal work, fail fast, Fairchild Semiconductor, fake news, Ferguson, Missouri, game design, gender pay gap, Google Glasses, Google X / Alphabet X, Grace Hopper, Hacker News, high net worth, Hyperloop, imposter syndrome, Jeff Bezos, job satisfaction, Khan Academy, Lyft, Marc Andreessen, Mark Zuckerberg, Mary Meeker, Maui Hawaii, Max Levchin, Menlo Park, meritocracy, meta-analysis, microservices, Parker Conrad, paypal mafia, Peter Thiel, post-work, pull request, reality distortion field, Richard Hendricks, ride hailing / ride sharing, rolodex, Salesforce, Saturday Night Live, shareholder value, Sheryl Sandberg, side project, Silicon Valley, Silicon Valley startup, Skype, Snapchat, Steve Jobs, Steve Jurvetson, Steve Wozniak, Steven Levy, subscription business, Susan Wojcicki, tech billionaire, tech bro, tech worker, TED Talk, Tim Cook: Apple, Travis Kalanick, uber lyft, women in the workforce, Zenefits

At that point, she was told she was still not eligible, because her performance score had been retroactively recalibrated. Fowler says HR explained that she didn’t appear to have an “upward career trajectory” and needed to prove herself as an engineer. This explanation was particularly suspect, given that Fowler was publishing an engineering book, Production-Ready Microservices: Building Standardized Systems Across an Engineering Organization, and had spoken at major tech conferences—quite impressive for a rank-and-file employee. Fowler says she went home and cried. The newly negative review would not only hurt her take-home pay but also disqualify her from continuing in an Uber-sponsored graduate computer science program at Stanford that she had been attending.

Coders: The Making of a New Tribe and the Remaking of the World by Clive Thompson

"Margaret Hamilton" Apollo, "Susan Fowler" uber, 2013 Report for America's Infrastructure - American Society of Civil Engineers - 19 March 2013, 4chan, 8-hour work day, Aaron Swartz, Ada Lovelace, AI winter, air gap, Airbnb, algorithmic bias, AlphaGo, Amazon Web Services, Andy Rubin, Asperger Syndrome, augmented reality, Ayatollah Khomeini, backpropagation, barriers to entry, basic income, behavioural economics, Bernie Sanders, Big Tech, bitcoin, Bletchley Park, blockchain, blue-collar work, Brewster Kahle, Brian Krebs, Broken windows theory, call centre, Cambridge Analytica, cellular automata, Charles Babbage, Chelsea Manning, Citizen Lab, clean water, cloud computing, cognitive dissonance, computer vision, Conway's Game of Life, crisis actor, crowdsourcing, cryptocurrency, Danny Hillis, data science, David Heinemeier Hansson, deep learning, DeepMind, Demis Hassabis, disinformation, don't be evil, don't repeat yourself, Donald Trump, driverless car, dumpster diving, Edward Snowden, Elon Musk, Erik Brynjolfsson, Ernest Rutherford, Ethereum, ethereum blockchain, fake news, false flag, Firefox, Frederick Winslow Taylor, Free Software Foundation, Gabriella Coleman, game design, Geoffrey Hinton, glass ceiling, Golden Gate Park, Google Hangouts, Google X / Alphabet X, Grace Hopper, growth hacking, Guido van Rossum, Hacker Ethic, hockey-stick growth, HyperCard, Ian Bogost, illegal immigration, ImageNet competition, information security, Internet Archive, Internet of things, Jane Jacobs, John Markoff, Jony Ive, Julian Assange, Ken Thompson, Kickstarter, Larry Wall, lone genius, Lyft, Marc Andreessen, Mark Shuttleworth, Mark Zuckerberg, Max Levchin, Menlo Park, meritocracy, microdosing, microservices, Minecraft, move 37, move fast and break things, Nate Silver, Network effects, neurotypical, Nicholas Carr, Nick Bostrom, no silver bullet, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, Oculus Rift, off-the-grid, OpenAI, operational security, opioid epidemic / opioid crisis, PageRank, PalmPilot, paperclip maximiser, pattern recognition, Paul Graham, paypal mafia, Peter Thiel, pink-collar, planetary scale, profit motive, ransomware, recommendation engine, Richard Stallman, ride hailing / ride sharing, Rubik’s Cube, Ruby on Rails, Sam Altman, Satoshi Nakamoto, Saturday Night Live, scientific management, self-driving car, side project, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, single-payer health, Skype, smart contracts, Snapchat, social software, software is eating the world, sorting algorithm, South of Market, San Francisco, speech recognition, Steve Wozniak, Steven Levy, systems thinking, TaskRabbit, tech worker, techlash, TED Talk, the High Line, Travis Kalanick, Uber and Lyft, Uber for X, uber lyft, universal basic income, urban planning, Wall-E, Watson beat the top human players on Jeopardy!, WeWork, WikiLeaks, women in the workforce, Y Combinator, Zimmermann PGP, éminence grise

Consider Uber: In February 2017, the programmer Susan Fowler single-handedly shone a light into its culture, with a blog post describing “one very, very strange year” of working there. The company had a famously hard-driving, macho culture. Fowler enjoyed the insane pace of work and her talented colleagues; in her few spare hours, she authored a best-selling book on programming “microservices.” But she quickly discovered that harassment ran wild at Uber. Not long after Fowler had arrived, her manager propositioned her, via chat, for sex. When she told HR and top management, they said her manager was a “high performer” and this was his first offense, so “they wouldn’t feel comfortable giving him anything other than a warning and a stern talking-to.”