microservices

24 results back to index


pages: 355 words: 81,788

Monolith to Microservices: Evolutionary Patterns to Transform Your Monolith by Sam Newman

Airbnb, business process, continuous integration, database schema, DevOps, fault tolerance, ghettoisation, inventory management, Jeff Bezos, Kubernetes, loose coupling, microservices, MVC pattern, price anchoring, pull request, single page application, software as a service, source of truth, telepresence

What You Will Learn This book is designed as a deep dive into how you think about, and execute, breaking apart existing systems into a microservice architecture. We will touch on many topics related to microservice architecture, but the focus is on the decomposition side of things. For a more general guide to microservice architectures, my previous book Building Microservices would be a good place to start. In fact, I strongly recommend that you consider that book to be a companion to this one. Chapter 1 contains an overview of what microservices are, and explores further the ideas that led us to these sorts of architectures. It should work well for people who are new to microservices, but I also strongly urge those of you with more experience to not skip this chapter.

As such, this chapter will provide an explanation of microservice architectures, look briefly at how microservices developed (which means, naturally, taking a look at monoliths), and examine some of the advantages and challenges of working with microservices. What Are Microservices? Microservices are independently deployable services modeled around a business domain. They communicate with each other via networks, and as an architecture choice offer many options for solving the problems you may face. It follows that a microservice architecture is based on multiple collaborating microservices. They are a type of service-oriented architecture (SOA), albeit one that is opinionated about how service boundaries should be drawn, and that independent deployability is key. Microservices also have the advantage of being technology agnostic.

This makes it easier to maintain customer-focused teams who build domain expertise, often with embedded product owners guiding their work. Breaking Changes A microservice exists as part of a wider system. It either consumes functionality provided by other microservices, exposes its own functionality to other microservice consumers, or possibly does both. With a microservice architecture, we are striving for independent deployability, but for that to happen, we need to make sure that when we make a change to a microservice we don’t break our consumers. We can think of the functionality we expose to other microservices in terms of a contract. It’s not just about saying, “This is the data I’ll return.”


pages: 540 words: 103,101

Building Microservices by Sam Newman

airport security, Amazon Web Services, anti-pattern, business process, call centre, continuous integration, create, read, update, delete, defense in depth, don't repeat yourself, Edward Snowden, fault tolerance, index card, information retrieval, Infrastructure as a Service, inventory management, job automation, Kubernetes, load shedding, loose coupling, microservices, MITM: man-in-the-middle, platform as a service, premature optimization, pull request, recommendation engine, social graph, software as a service, source of truth, the built environment, web application, WebSocket

message brokers, Technology Choices metricslibraries for, Service Metrics service metrics, Service Metrics tracking across multiple services, Metric Tracking Across Multiple Services Metrics library, Tailored Service Template microservicesappropriate application of, When Shouldn’t You Use Microservices? autonomy and, Autonomous, Building a Team benefits of, Microservices composability of, Composability definition of term, What Are Microservices? deployment ease of, Ease of Deployment drawbacks of, No Silver Bullet, When Shouldn’t You Use Microservices? key principles of, Bringing It All Together organizational alignment and, Organizational Alignment origins of, Microservices replaceability and, Optimizing for Replaceability resilience of, Resilience scaling and, Scaling size and, Small, and Focused on Doing One Thing Well technology heterogeneity of, Technology Heterogeneity vs. modules, Modules vs. service-oriented architecture, What About Service-Oriented Architecture?

If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-491-95035-7 [LSI] Preface Microservices are an approach to distributed systems that promote the use of finely grained services with their own lifecycles, which collaborate together. Because microservices are primarily modeled around business domains, they avoid the problems of traditional tiered architectures. Microservices also integrate new technologies and techniques that have emerged over the last decade, which helps them avoid the pitfalls of many service-oriented architecture implementations. This book is full of concrete examples of microservice use around the world, including in organizations like Netflix, Amazon, Gilt, and the REA group, who have all found that the increased autonomy this architecture gives their teams is a huge advantage.

For people new to the subject, I’ve structured the chapters in a way that I hope will make sense to read from beginning to end. Here is an overview of what we cover: Chapter 1, Microservices We’ll begin with an introduction to microservices, including the key benefits as well as some of the downsides. Chapter 2, The Evolutionary Architect This chapter discusses the difficulties we face in terms of making trade-offs as architects, and covers specifically just how many things we need to think about with microservices. Chapter 3, How to Model Services Here we’ll start to define the boundary of microservices, using techniques from domain-driven design to help focus our thinking. Chapter 4, Integration This is where we start getting a bit deeper into specific technology implications, as we discuss what sorts of service collaboration techniques will help us most.


pages: 255 words: 55,018

Architecting For Scale by Lee Atchison

Amazon Web Services, business process, cloud computing, continuous integration, DevOps, Internet of things, microservices, platform as a service, risk tolerance, software as a service, web application

Chapter 10, Building Systems with Reduced Risk In this chapter, I give suggestions on how to reduce risk within your applications and build applications with lower risk. Part III, “Services and Microservices” Services and microservices are an architecture strategy for building larger and more complicated applications that need to operate at higher scale. Chapter 11, Why Use Services? This chapter explores why services are important to building highly scalable applications. Chapter 12, Using Microservices Here, I provide an introduction on creating microservice-based architectures, focusing on sizing of services and determining where service boundaries should be created in order to improve scaling and availability.

One common place where modern software construction principles tend to increase complexity more than perhaps is necessary is in microservice-based architectures. Microservice-based architectures reduce the complexity of individual components substantially, making it possible for the individual services to be easily understood and built using simpler techniques and designs. However, although they reduce the complexity of the individual microservice, they increase the number of independent modules (microservices) necessary to build a large-scale application. By having a larger number of independent modules working together, you increase the interdependence on the modules, and increase the overall complexity of the application.

This is a good question, and one that does not have a single correct answer. Some companies that “service-ize” split their application into many (hundreds or thousands) of very tiny microservices. Others split their application into only a handful of larger services. There is no right answer to this problem. However, the industry is trending toward smaller microservices, and more of them. Technologies such as Docker have made these larger number of microservices a viable system topology. We use the term services and microservices interchangeably in this book. Dividing into Services So, how do you decide where service boundaries should be? Company organization, culture, and the type of application can play a major role in determining service boundaries.


Seeking SRE: Conversations About Running Production Systems at Scale by David N. Blank-Edelman

Affordable Care Act / Obamacare, algorithmic trading, Amazon Web Services, bounce rate, business continuity plan, business process, cloud computing, cognitive bias, cognitive dissonance, commoditize, continuous integration, crowdsourcing, dark matter, database schema, Debian, defense in depth, DevOps, domain-specific language, en.wikipedia.org, fault tolerance, fear of failure, friendly fire, game design, Grace Hopper, information retrieval, Infrastructure as a Service, Internet of things, invisible hand, iterative process, Kubernetes, loose coupling, Lyft, Marc Andreessen, microservices, minimum viable product, MVC pattern, performance metric, platform as a service, pull request, RAND corporation, remote working, Richard Feynman, risk tolerance, Ruby on Rails, search engine result page, self-driving car, sentiment analysis, Silicon Valley, single page application, Snapchat, software as a service, software is eating the world, source of truth, the scientific method, Toyota Production System, web application, WebSocket, zero day

We aggregate this information for all our critical microservices and compare the trend to the previous 30 days. It isn’t meant to be near real time, it’s meant to be more operational — are you getting better or worse. Let’s say a team owns three microservices, and the goal is for the microservices to have four-nines availability for calls from clients. If these microservices stay above four-nines, the team will never receive an email (availability report). If the last 7 days compared to the previous 21 have a deviation in a negative manner, or if the microservices are failing to achieve the four-nines, a report will be sent to the team.

Clicking on a bar in the graph will deliver a detailed report that shows for the entire window call rates of upstream (client) services talking to the microservice and the availability to those calls. On the right panel is a view of downstream dependency call and success rates, which is helpful as the microservice availability might be reduced by downstream dependency. We call this internal implementation the “microservice availability scorecard framework.” At this point, the SRE team (working with Data Analytics) has provided information to a team about the microservices they own, and in particular their microservice availability to dependent client services, independent of Netflix service availability.

The Problem with Systems You can think of the engineering organization at Netflix as about a hundred small engineering teams, of around five to seven people each. The service — that product that the customers use — is architected as many hundreds of microservices working together. Every microservice is owned by just one team, and that team is responsible for everything: features, roadmap, operations, and uptime for that microservice. An example of a microservice might be a customer microservice, which for a given customer ID serves up the metadata stored with that person. Another might be a personalization service that stores information about preferences related to a customer so that when a customer looks for something to watch on Netflix, we decorate their experience with context based on what they previously watched, and so on.


pages: 1,380 words: 190,710

Building Secure and Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems by Heather Adkins, Betsy Beyer, Paul Blankinship, Ana Oprea, Piotr Lewandowski, Adam Stubblefield

anti-pattern, barriers to entry, bash_history, business continuity plan, business process, Cass Sunstein, cloud computing, continuous integration, correlation does not imply causation, create, read, update, delete, cryptocurrency, cyber-physical system, database schema, Debian, defense in depth, DevOps, Edward Snowden, fault tolerance, fear of failure, general-purpose programming language, Google Chrome, Internet of things, Kubernetes, load shedding, margin call, microservices, MITM: man-in-the-middle, performance metric, pull request, ransomware, revision control, Richard Thaler, risk tolerance, self-driving car, Skype, slashdot, software as a service, source of truth, Stuxnet, Turing test, undersea cable, uranium enrichment, Valgrind, web application, Y2K, zero day

Risks include a SQL injection vulnerability in the catalog search code allowing an attacker to obtain sensitive user data, like names or shipping addresses, or a remote code execution vulnerability in the web application server, such as CVE-2010-1870, permitting an attacker to read or modify any part of the application’s database. Small TCBs and strong security boundaries We can improve the security of our design by splitting the application into microservices. In this architecture, each microservice handles a self-contained part of the application’s functionality and stores data in its own separate database. These microservices communicate with each other via RPCs and treat all incoming requests as not necessarily trustworthy, even if the caller is another internal microservice. Using microservices, we might restructure the application as shown in Figure 6-3. Instead of a monolithic server, we now have a web application frontend and separate backends for the product catalog and purchasing-related functionality.

Isolate Your Rate-Limiting Mechanism to Increase Reliability If your system can deploy changes extremely quickly, how can you protect against unwanted rapid execution? One strategy is to build the rate-limiting mechanism as an independent, standalone, single-purpose microservice that constrains the rate of change to a particular system or systems. Make this microservice as simple as possible and amenable to rigorous testing. For example, a rate-limiting microservice might provide a short-lived cryptographic token that asserts that the microservice reviewed and approved a particular change at a certain time. The rate-limiting service is also an excellent point at which to collect change logs for auditing purposes.

To limit the blast radius of such a compromise, different jobs should typically be run as different roles. For example, if you have two microservices that need access to two different classes of data (say, photos and text chats), running these two microservices as different roles can increase the resilience of your system even if the two microservices are developed and run by the same team. Location Separation Location separation helps to limit an attacker’s impact along an additional dimension: the location where the microservice is running. For example, you might want to prevent an adversary who has physically compromised a single datacenter from being able to read data in all your other datacenters.


Industry 4.0: The Industrial Internet of Things by Alasdair Gilchrist

3D printing, additive manufacturing, Amazon Web Services, augmented reality, autonomous vehicles, barriers to entry, business intelligence, business process, chief data officer, cloud computing, connected car, cyber-physical system, deindustrialization, DevOps, digital twin, fault tolerance, global value chain, Google Glasses, hiring and firing, industrial robot, inflight wifi, Infrastructure as a Service, Internet of things, inventory management, job automation, low cost airline, low skilled workers, microservices, millennium bug, pattern recognition, peer-to-peer, platform as a service, pre–internet, race to the bottom, RFID, Skype, smart cities, smart grid, smart meter, smart transportation, software as a service, stealth mode startup, supply-chain management, trade route, undersea cable, web application, WebRTC, Y2K

Security goes beyond simple authentication and confidentiality, and also includes authorization and integrity. When it comes to ease of implementation, SOAP is the web service at the forefront. Microservices The use of microservices and web APIs has become very popular in web and cloud-based applications and they are both ideally suited for the IIoT. The term microservice has no standard, formal definition; however, there are certain characteristics that identify them. Essentially, there is a distinguishable microservice architecture, which provides a method of developing software applications as a suite of small, modular services. These independent services run a unique process and communicate through a well-defined, lightweight web service or other mechanism in order to deliver a specific result. 151 152 Chapter 9 | Software Design Concepts What makes microservices valuable are that applications can be constructed that decouple the underlying complexities of the host systems from the application’s purpose.

With microservices, we see the introduction of a different approach, one that is ideal for the IoT, and by extension the Industrial Internet. The microservices approach is to build your own or consume an open microservice, which is specific to a function. Programmers then construct applications using these independent modules called microservices, which the programmer can scale and modify in isolation, leaving the rest of the application untouched. By not compromising the overall integrity of the application, programmers can tweak an individual service before redeploying it. Similarly, they can add or customize a service for a new device type of model without affecting any other service in the application, for example, any existing device type’s user experience.

Similarly, they can add or customize a service for a new device type of model without affecting any other service in the application, for example, any existing device type’s user experience. Microservices are scalable, adaptable, and modular, making them ideal for cloud-based applications that have to provide dynamic, yet consistent user experience across a wide array, and ever changing, list of devices. To clarify the benefits of microservices, and their scalability we can look to many large web-scaled companies, such as Netflix, Amazon, and eBay. All of these web giants have migrated over the last decade to microservice-based applications. Netflix, for example, receives around one billion API calls per day from over 800 different types and models of devices.


Terraform: Up and Running: Writing Infrastructure as Code by Yevgeniy Brikman

Amazon Web Services, cloud computing, DevOps, en.wikipedia.org, full stack developer, general-purpose programming language, microservices, Ruby on Rails

For example, you could create a canonical module that defines how to deploy a single microservice—including how to run a cluster, how to scale the cluster in response to load, and how to distribute traffic requests across the cluster— and each team could use this module to manage their own microservices with just a few lines of code. To make such a module work for multiple teams, the Terraform code in that module must be flexible and configurable. For example, one team may want to use your mod‐ ule to deploy a single instance of their microservice with no load balancer while another may want a dozen instances of their microservice with a load balancer to dis‐ tribute traffic between those instances.

It’s an easy lan‐ guage to learn, so in the span of about 25 pages, you’ll go from running your first Terraform commands all the way up to using Terraform to deploy a cluster of servers with a load balancer that distributes traffic across them. This infrastructure is a good starting point for running scalable, highly-available web services and microservices. In subsequent chapters, you’ll evolve this example even further. Terraform can provision infrastructure across many different types of cloud provid‐ ers, including Amazon Web Services (AWS), Azure, Google Cloud, DigitalOcean, and many others. For just about all of the code examples in this chapter and the rest of the book, you are going to use AWS.

File layout | 83 • global: A place to put resources that are used across all environments, such as user management (e.g. S3, IAM). Within each environment, there are separate folders for each “component.” The com‐ ponents differ for every project, but the typical ones are: • vpc: The network topology for this environment. • services: The apps or microservices to run in this environment, such as a Ruby on Rails frontend or a Scala backend. Each app could even live in its own folder to isolate it from all the other apps. • data-storage: The data stores to run in this environment, such as MySQL or Redis. Each data store could even live in its own folder to isolate it from all other data stores.


pages: 31 words: 9,168

Designing Reactive Systems: The Role of Actors in Distributed Architecture by Hugh McKee

Amazon Web Services, fault tolerance, Internet of things, microservices

Summary Technology adoption is rarely cyclical; however, in case of the Actor model (created in the early 1970s) the spotlight is swinging back to this unique approach to distributed, concurrent computation. As Forrester Research points out in “How To Capture The Benefits Of Microservice Design” (2016), the Actor model is receiving “renewed interest as cloud concurrency challenges grow” in enterprises building microservices architectures. This report is targeted toward decision makers in the enterprise and provides some high-level insight into how actors and actor systems can be used to create lightweight business systems that evolve quickly, that can scale, and that can run without stopping.

Actors can be implemented to react to nodes leaving and joining a cluster. Work can be distributed across a cluster. Actors and actor systems provide an abstraction layer that allows for higher levels of concurrency. Chapter 6. Conclusion Today, it is now possible to create distributed, microservices-based systems that were impossible to even dream of just a few short years ago. Enterprises across all industries now desire the ability to create systems that can evolve at the speed of the business and cater to the whims of users. We can now elastically scale systems that support massive numbers of users and process huge volumes of data.

We can now elastically scale systems that support massive numbers of users and process huge volumes of data. It is now possible to harden systems with a level of resilience that enables them to run with such low downtime that it’s measured in seconds, rather than hours. One of the foundational technologies that enables us to create microservices architectures that evolve quickly, that can scale, and that can run without stopping, is systems based on the Actor model. It’s the Actor model that provides the core functionality of Reactive systems, defined in the Reactive Manifesto as responsive, resilient, elastic, and message driven (see Figure 6-1). In this report, we have reviewed some of the features and characteristics of how actors are used in actor systems, but we have only scratched the surface of how actor systems are being used today.


Martin Kleppmann-Designing Data-Intensive Applications. The Big Ideas Behind Reliable, Scalable and Maintainable Systems-O’Reilly (2017) by Unknown

active measures, Amazon Web Services, bitcoin, blockchain, business intelligence, business process, c2.com, cloud computing, collaborative editing, commoditize, conceptual framework, cryptocurrency, database schema, DevOps, distributed ledger, Donald Knuth, Edward Snowden, Ethereum, ethereum blockchain, fault tolerance, finite state, Flash crash, full text search, general-purpose programming language, informal economy, information retrieval, Internet of things, iterative process, John von Neumann, Kubernetes, loose coupling, Marc Andreessen, microservices, natural language processing, Network effects, packet switching, peer-to-peer, performance metric, place-making, premature optimization, recommendation engine, Richard Feynman, self-driving car, semantic web, Shoshana Zuboff, social graph, social web, software as a service, software is eating the world, sorting algorithm, source of truth, SPARQL, speech recognition, statistical model, undersea cable, web application, WebSocket, wikimedia commons

ISBN: 978-0-122-33435-1 Summary | 141 [28] Russell Housley, Warwick Ford, Tim Polk, and David Solo: “RFC 2459: Internet X.509 Public Key Infrastructure: Certificate and CRL Profile,” IETF Network Work‐ ing Group, Standards Track, January 1999. [29] Lev Walkin: “Question: Extensibility and Dropping Fields,” lionet.info, Septem‐ ber 21, 2010. [30] Jesse James Garrett: “Ajax: A New Approach to Web Applications,” adaptive‐ path.com, February 18, 2005. [31] Sam Newman: Building Microservices. O’Reilly Media, 2015. ISBN: 978-1-491-95035-7 [32] Chris Richardson: “Microservices: Decomposing Applications for Deployability and Scalability,” infoq.com, May 25, 2014. [33] Pat Helland: “Data on the Outside Versus Data on the Inside,” at 2nd Biennial Conference on Innovative Data Systems Research (CIDR), January 2005. [34] Roy Thomas Fielding: “Architectural Styles and the Design of Network-Based Software Architectures,” PhD Thesis, University of California, Irvine, 2000. [35] Roy Thomas Fielding: “REST APIs Must Be Hypertext-Driven,” roy.gbiv.com, October 20 2008. [36] “REST in Peace, SOAP,” royal.pingdom.com, October 15, 2010. [37] “Web Services Standards as of Q1 2007,” innoq.com, February 2007. [38] Pete Lacey: “The S Stands for Simple,” harmful.cat-v.org, November 15, 2006. [39] Stefan Tilkov: “Interview: Pete Lacey Criticizes Web Services,” infoq.com, December 12, 2006. [40] “OpenAPI Specification (fka Swagger RESTful API Documentation Specifica‐ tion) Version 2.0,” swagger.io, September 8, 2014. [41] Michi Henning: “The Rise and Fall of CORBA,” ACM Queue, volume 4, number 5, pages 28–34, June 2006. doi:10.1145/1142031.1142044 [42] Andrew D.

.: “Feral Concurrency Control: An Empirical Investigation of Modern Application Integrity,” at ACM International Conference on Management of Data (SIGMOD), June 2015. doi: 10.1145/2723372.2737784 [37] Guy Steele: “Re: Need for Macros (Was Re: Icon),” email to ll1-discuss mailing list, people.csail.mit.edu, December 24, 2001. [38] David Gelernter: “Generative Communication in Linda,” ACM Transactions on Programming Languages and Systems (TOPLAS), volume 7, number 1, pages 80–112, January 1985. doi:10.1145/2363.2433 [39] Patrick Th. Eugster, Pascal A. Felber, Rachid Guerraoui, and Anne-Marie Ker‐ marrec: “The Many Faces of Publish/Subscribe,” ACM Computing Surveys, volume 35, number 2, pages 114–131, June 2003. doi:10.1145/857076.857078 [40] Ben Stopford: “Microservices in a Streaming World,” at QCon London, March 2016. [41] Christian Posta: “Why Microservices Should Be Event Driven: Autonomy vs Authority,” blog.christianposta.com, May 27, 2016. [42] Alex Feyerke: “Say Hello to Offline First,” hood.ie, November 5, 2013. [43] Sebastian Burckhardt, Daan Leijen, Jonathan Protzenko, and Manuel Fähndrich: “Global Sequence Protocol: A Robust Abstraction for Replicated Shared State,” at Summary | 547 29th European Conference on Object-Oriented Programming (ECOOP), July 2015. doi:10.4230/LIPIcs.ECOOP.2015.568 [44] Mark Soper: “Clearing Up React Data Management Confusion with Flux, Redux, and Relay,” medium.com, December 3, 2015. [45] Eno Thereska, Damian Guy, Michael Noll, and Neha Narkhede: “Unifying Stream Processing and Interactive Queries in Apache Kafka,” confluent.io, October 26, 2016. [46] Frank McSherry: “Dataflow as Database,” github.com, July 17, 2016. [47] Peter Alvaro: “I See What You Mean,” at Strange Loop, September 2015. [48] Nathan Marz: “Trident: A High-Level Abstraction for Realtime Computation,” blog.twitter.com, August 2, 2012. [49] Edi Bice: “Low Latency Web Scale Fraud Prevention with Apache Samza, Kafka and Friends,” at Merchant Risk Council MRC Vegas Conference, March 2016. [50] Charity Majors: “The Accidental DBA,” charity.wtf, October 2, 2016. [51] Arthur J.

This approach is often used to decompose a large application into smaller services by area of functionality, such that one service makes a request to another when it requires some functionality or data from that other service. This way of building applications has traditionally been called a serviceoriented architecture (SOA), more recently refined and rebranded as microservices architecture [31, 32]. In some ways, services are similar to databases: they typically allow clients to submit and query data. However, while databases allow arbitrary queries using the query lan‐ guages we discussed in Chapter 2, services expose an application-specific API that only allows inputs and outputs that are predetermined by the business logic (applica‐ tion code) of the service [33].


pages: 1,237 words: 227,370

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann

active measures, Amazon Web Services, bitcoin, blockchain, business intelligence, business process, c2.com, cloud computing, collaborative editing, commoditize, conceptual framework, cryptocurrency, database schema, DevOps, distributed ledger, Donald Knuth, Edward Snowden, Ethereum, ethereum blockchain, fault tolerance, finite state, Flash crash, full text search, general-purpose programming language, informal economy, information retrieval, Infrastructure as a Service, Internet of things, iterative process, John von Neumann, Kubernetes, loose coupling, Marc Andreessen, microservices, natural language processing, Network effects, packet switching, peer-to-peer, performance metric, place-making, premature optimization, recommendation engine, Richard Feynman, self-driving car, semantic web, Shoshana Zuboff, social graph, social web, software as a service, software is eating the world, sorting algorithm, source of truth, SPARQL, speech recognition, statistical model, undersea cable, web application, WebSocket, wikimedia commons

ISBN: 978-0-122-33435-1 [28] Russell Housley, Warwick Ford, Tim Polk, and David Solo: “RFC 2459: Internet X.509 Public Key Infrastructure: Certificate and CRL Profile,” IETF Network Working Group, Standards Track, January 1999. [29] Lev Walkin: “Question: Extensibility and Dropping Fields,” lionet.info, September 21, 2010. [30] Jesse James Garrett: “Ajax: A New Approach to Web Applications,” adaptivepath.com, February 18, 2005. [31] Sam Newman: Building Microservices. O’Reilly Media, 2015. ISBN: 978-1-491-95035-7 [32] Chris Richardson: “Microservices: Decomposing Applications for Deployability and Scalability,” infoq.com, May 25, 2014. [33] Pat Helland: “Data on the Outside Versus Data on the Inside,” at 2nd Biennial Conference on Innovative Data Systems Research (CIDR), January 2005. [34] Roy Thomas Fielding: “Architectural Styles and the Design of Network-Based Software Architectures,” PhD Thesis, University of California, Irvine, 2000

[38] David Gelernter: “Generative Communication in Linda,” ACM Transactions on Programming Languages and Systems (TOPLAS), volume 7, number 1, pages 80–112, January 1985. doi:10.1145/2363.2433 [39] Patrick Th. Eugster, Pascal A. Felber, Rachid Guerraoui, and Anne-Marie Kermarrec: “The Many Faces of Publish/Subscribe,” ACM Computing Surveys, volume 35, number 2, pages 114–131, June 2003. doi:10.1145/857076.857078 [40] Ben Stopford: “Microservices in a Streaming World,” at QCon London, March 2016. [41] Christian Posta: “Why Microservices Should Be Event Driven: Autonomy vs Authority,” blog.christianposta.com, May 27, 2016. [42] Alex Feyerke: “Say Hello to Offline First,” hood.ie, November 5, 2013. [43] Sebastian Burckhardt, Daan Leijen, Jonathan Protzenko, and Manuel Fähndrich: “Global Sequence Protocol: A Robust Abstraction for Replicated Shared State,” at 29th European Conference on Object-Oriented Programming (ECOOP), July 2015. doi:10.4230/LIPIcs.ECOOP.2015.568 [44] Mark Soper: “Clearing Up React Data Management Confusion with Flux, Redux, and Relay,” medium.com, December 3, 2015

This approach is often used to decompose a large application into smaller services by area of functionality, such that one service makes a request to another when it requires some functionality or data from that other service. This way of building applications has traditionally been called a service-oriented architecture (SOA), more recently refined and rebranded as microservices architecture [31, 32]. In some ways, services are similar to databases: they typically allow clients to submit and query data. However, while databases allow arbitrary queries using the query languages we discussed in Chapter 2, services expose an application-specific API that only allows inputs and outputs that are predetermined by the business logic (application code) of the service [33].


pages: 333 words: 64,581

Clean Agile: Back to Basics by Robert C. Martin

Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, c2.com, continuous integration, DevOps, double entry bookkeeping, en.wikipedia.org, failed state, Frederick Winslow Taylor, index card, iterative process, Kubernetes, loose coupling, microservices, remote working, revision control, Turing machine

Teams struggle with this even when they don’t need to coordinate with other teams. When coordinating the work of multiple teams, monolithic, centralized, preplanned architectures create massive dependencies between teams that tend to force them to work in lock step, thus defeating much of the promise of Agile. Simple Design, especially when used with practices such as a microservices architecture, enables Agility in the large. The Future of Agile Coaching In the last few years, professional coaching and professional facilitation have started to make their way into the Agile curriculum. The Scrum Alliance’s Advanced Certified Scrum Master (ACSM) course has a few learning objectives related to coaching and facilitation, and their Certified Team Coach (CTC) and Certified Enterprise Coach (CEC) programs require that you acquire even more facilitation and coaching skills than that.

If developers are discussing practices that would enable them to build systems in a better way, there is no reason to involve businesspeople. Business should only be involved when there is a significant impact in the cost or duration of the project. There is a difference between re-architecting a whole monolith system into microservices and doing TDD. The former has a very significant impact on the cost and duration of the project; the latter doesn’t, as long as developers are comfortable with the technique. Whether or not developers automate their tests should not be relevant to the business. It should be even less relevant whether the automated tests are written before or after the production code.

They have been key in promoting XP, and many other technical practices to many developers and companies around the world. It is through Software Craftsmanship communities that many developers are learning TDD, Continuous Integration, Pair Programming, Simple Design, SOLID principles, Clean Code, and Refactoring. They are also learning how to architect systems using microservices, how to automate their deployment pipelines, and how to migrate their systems to the cloud. They are learning different programming languages and paradigms. They are learning new technologies and different ways to test and maintain their applications. Developers in the Craftsmanship community are creating safe and friendly spaces where they can meet like-minded people and talk about their profession.


pages: 378 words: 110,518

Postcapitalism: A Guide to Our Future by Paul Mason

Alfred Russel Wallace, bank run, banking crisis, banks create money, Basel III, basic income, Bernie Madoff, Bill Gates: Altair 8800, bitcoin, Branko Milanovic, Bretton Woods, BRICs, British Empire, business cycle, business process, butterfly effect, call centre, capital controls, Cesare Marchetti: Marchetti’s constant, Claude Shannon: information theory, collaborative economy, collective bargaining, Corn Laws, corporate social responsibility, creative destruction, credit crunch, currency manipulation / currency intervention, currency peg, David Graeber, deglobalization, deindustrialization, deskilling, discovery of the americas, Downton Abbey, drone strike, en.wikipedia.org, energy security, eurozone crisis, factory automation, financial repression, Firefox, Fractional reserve banking, Frederick Winslow Taylor, full employment, future of work, game design, income inequality, inflation targeting, informal economy, information asymmetry, intangible asset, Intergovernmental Panel on Climate Change (IPCC), Internet of things, job automation, John Maynard Keynes: Economic Possibilities for our Grandchildren, Joseph Schumpeter, Kenneth Arrow, Kevin Kelly, Kickstarter, knowledge economy, knowledge worker, late capitalism, low skilled workers, market clearing, means of production, Metcalfe's law, microservices, money: store of value / unit of account / medium of exchange, mortgage debt, Network effects, new economy, Norbert Wiener, Occupy movement, oil shale / tar sands, oil shock, Paul Samuelson, payday loans, Pearl River Delta, post-industrial society, precariat, price mechanism, profit motive, quantitative easing, race to the bottom, RAND corporation, rent-seeking, reserve currency, RFID, Richard Stallman, Robert Gordon, Robert Metcalfe, secular stagnation, sharing economy, Stewart Brand, structural adjustment programs, supply-chain management, The Future of Employment, the scientific method, The Wealth of Nations by Adam Smith, Transnistria, union organizing, universal basic income, urban decay, urban planning, Vilfredo Pareto, wages for housework, WikiLeaks, women in the workforce

Crucially, it would have to create new markets beyond production, in the field of services. The 250-year history of capitalism has been about pushing market forces into sectors where they did not exist before. Info-capitalism would have to take this to its extremes, creating new forms of person-to-person micro-services, paid for using micro-payments, and mainly in the private sector. And finally, for info-capitalism to succeed it would have to find work for the millions of people whose jobs are automated. These could not be in the majority low-paid jobs because the traditional escape mechanism needs labour costs to rise: human life has to become more complex, needing more labour inputs, not fewer, as in the four cyclical upswings described by long-cycle theory.

The elements of such a solution are there in modern economies: Apple is the classic price monopolist, Amazon’s business model the classic strategy for capturing externalities; commodity speculation the classic driver of energy and raw material costs above their value; while the rise of personal micro-services – dog minding, nail salons, personal concierges and the like – shows capitalism commercializing activities we used to provide through friendship or informality. But there are clear structural obstacles to making this work. First, the normal escape route – innovation creates expensive new technologies that replace info-tech – is blocked.

Rich people are already surrounded by such post-modern servants, but to replace 47 per cent of all jobs this way would require the mass commercialization of ordinary human life. And here’s where you hit the third obstacle – what philosopher André Gorz called the ‘limits of economic rationality’.37 At a certain level, human life and interaction resist commercialization. An economy in which large numbers of people perform micro-services for each other can exist, but as a form of capitalism it would be highly inefficient and intrinsically low-value. You could pay wages for housework, turn all sexual relationships into paid work, mums with their toddlers in the park could charge each other a penny each time they took turns to push the swings.


pages: 719 words: 181,090

Site Reliability Engineering: How Google Runs Production Systems by Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy

Air France Flight 447, anti-pattern, barriers to entry, business intelligence, business process, Checklist Manifesto, cloud computing, combinatorial explosion, continuous integration, correlation does not imply causation, crowdsourcing, database schema, defense in depth, DevOps, en.wikipedia.org, fault tolerance, Flash crash, George Santayana, Google Chrome, Google Earth, information asymmetry, job automation, job satisfaction, Kubernetes, linear programming, load shedding, loose coupling, meta analysis, meta-analysis, microservices, minimum viable product, MVC pattern, performance metric, platform as a service, revision control, risk tolerance, side project, six sigma, the scientific method, Toyota Production System, trickle-down economics, web application, zero day

External Factors Affecting SRE External factors have traditionally pressured the SRE organization and its resources in several ways. Google is increasingly following the industry trend of moving toward microservices.1 As a result, both the number of requests for SRE support and the cardinality of services to support have increased. Because each service has a base fixed operational cost, even simple services demand more staffing. Microservices also imply an expectation of lower lead time for deployment, which was not possible with the previous PRR model (which had a lead time of months). Hiring experienced, qualified SREs is difficult and costly.

However, the adoption of frameworks is becoming a prominent influence on building production-ready services at Google as well as profoundly expanding the SRE contribution, lowering service management overhead, and improving baseline service quality across the organization. 1 See the Wikipedia page on microservices at http://en.wikipedia.org/wiki/Microservices. 2 Occasionally, there were consulting engagements by SRE teams with some non-onboarded services, but consultations were a best-effort approach and limited in number and scope. 3 The new model of service management changes the SRE staffing model in two ways: (1) because a lot of service technology is common, it reduces the number of required SREs per service; (2) it enables the creation of production platforms with separation of concerns between production platform support (done by SREs) and service-specific business-logic support, which remains with the development team.

Significantly lower operational overhead A production platform built on top of frameworks with stronger conventions significantly reduced operational overhead, for the following reasons: It supports strong conformance tests for coding structure, dependencies, tests, coding style guides, and so on. This functionality also improves user data privacy, testing, and security conformance. It features built-in service deployment, monitoring, and automation for all services. It facilitates easier management of large numbers of services, especially micro-services, which are growing in number. It enables much faster deployment: an idea can graduate to fully deployed SRE-level production quality in a matter of days! Universal support by design The constant growth in the number of services at Google means that most of these services can neither warrant SRE engagement nor be maintained by SREs.


pages: 458 words: 46,761

Essential Sqlalchemy by Jason Myers, Rick Copeland

create, read, update, delete, database schema, microservices, Ruby on Rails, side project, web application

This encapsulation can make it easy to make database interactions feel more like normal Python code. Most common applications lend themselves to being modeled in this way. It can also be a highly effective way to inject domain-driven design into a legacy application or one with raw SQL statements sprinkled throughout. Microservices also benefit from the abstraction of the underlying database, allowing the developer to focus on just the process being implemented. However, because the ORM is built on top of SQLAlchemy Core, you can use its ability to work with services like Oracle Data Warehousing and Amazon Redshift in the same manner that it interoperates with MySQL.

Index A add( ) method (Session), Inserting Data add_column( ) method (Alembic), Building a Migration Manually Alembic, Getting Started with Alembic-Configuring the Migration Environmentdocumentation for, Autogenerating a Migration, Generating SQL empty migration, creating, Generating a Base Empty Migration-Generating a Base Empty Migration installing, Getting Started with Alembic migration environment, configuring, Configuring the Migration Environment-Configuring the Migration Environment migration environment, creating, Creating the Migration Environment-Creating the Migration Environment migration level, determining, Determining a Database’s Migration Level-Determining a Database’s Migration Level migration level, setting, Marking the Database Migration Level migration, autogenerating, Autogenerating a Migration-Autogenerating a Migration migration, building manually, Building a Migration Manually-Building a Migration Manually migration, downgrading, Downgrading Migrations-Downgrading Migrations operations performed by, list of, Building a Migration Manually SQL, generating, Generating SQL-Generating SQL alembic current command, Determining a Database’s Migration Level alembic downgrade command, Downgrading Migrations alembic history command, Determining a Database’s Migration Level alembic init alembic command, Creating the Migration Environment alembic revision command, Generating a Base Empty Migration alembic stamp command, Marking the Database Migration Level alembic.ini file, Creating the Migration Environment, Configuring the Migration Environment alias( ) function, Aliases alias( ) method (Table), Aliases aliases for tables, Aliases-Aliases all( ) method (Session), Querying Data-Conjunctions alter_column( ) method (Alembic), Building a Migration Manually and_( ) function, Conjunctions-Conjunctions, Boolean Operators app factory pattern, Integrating SQLAlchemy with Flask The Architecture of Open Source Applications (Bayer), Where to Go from Here arithmetic operators, in queries, Operators, Operators array slice notation, Limiting association proxies, Association Proxy-Association Proxy association_proxy( ) method, Association Proxy AttributeError exception, AttributeError-AttributeError attributeshybrid, Hybrid Attributes-Hybrid Attributes non-existent, error for, AttributeError-AttributeError pointer to, across relationships, Association Proxy-Association Proxy attrs collection (inspector), Session States autoload argument (Table), Reflecting Individual Tables autoload_with argument (Table), Reflecting Individual Tables automap_base objects, Reflecting a Database with Automap-Reflected Relationships B backref( ) method, Relationships Base objects, Reflecting a Database with Automap-Reflected Relationships Bayer, Mike (developer, SQLAlchemy), Introduction to SQLAlchemy Beaulieu, Alan (Learning SQL), Who This Book Is For begin( ) method (connection), Transactions-Transactions between( ) method (ClauseElement), ClauseElements BIGINT type, Types BigInteger type, Types bitwise logical operators, in queries, Boolean Operators BLOB type, Types bool type, Types Boolean operators, in queries, Boolean Operators, Boolean Operators Boolean type, Types BOOLEAN type, Types bulk_save_objects( ) method (Session), Inserting Data business objects, Choosing Between SQLAlchemy Core and ORM byte type, Types BYTEA type, Types C cast( ) function, Operators chaining queries, Chaining-Chaining, Chaining-Chaining CheckConstraint, Keys and Constraints Chinook database, Reflection ClauseElement objects, ClauseElements, Filtering CLOB type, Types code examplesChinook database, Reflection downloading, Using Code Examples in IPython notebooks, How to Use the Examples permission to use, Using Code Examples Column objects, ResultProxy columnscontrolling for a query, Controlling the Columns in the Query, Controlling the Columns in the Query default value for, Defining Tables via ORM Classes grouping, Grouping, Grouping labeling, in query results, Built-In SQL Functions and Labels, Built-In SQL Functions and Labels required, Defining Tables via ORM Classes resetting on update, Defining Tables via ORM Classes in Table objects, Columns-Columns commit( ) method (Session), Inserting Data-Inserting Data commit( ) method (transaction), Transactions-Transactions comparison operators, in queries, Operators, Operators compile( ) method, Inserting Data concat( ) method (ClauseElement), ClauseElements conditional chaining, Chaining, Chaining conjunctions, in queries, Conjunctions-Conjunctions, Conjunctions connect() method (engine), Connecting to a Database connection string, Connecting to a Database-Connecting to a Database connection timeouts, Connecting to a Database constraintsin ORM classes, Keys, Constraints, and Indexes in Table objects, Keys and Constraints-Keys and Constraints, Relationships and ForeignKeyConstraints-Relationships and ForeignKeyConstraints contact information for this book, How to Contact Us contains( ) method (ClauseElement), ClauseElements conventions used in this book, Conventions Used in This Book Core (see SQLAlchemy Core) count( ) function, Built-In SQL Functions and Labels create_all( ) method (MetaData), Persisting the Tables-Persisting the Tables, Persisting the Schema create_check_constraint( ) method (Alembic), Building a Migration Manually create_engine( ) function, Connecting to a Database-Connecting to a Database create_foreign_key( ) method (Alembic), Building a Migration Manually create_index( ) method (Alembic), Autogenerating a Migration, Building a Migration Manually create_primary_key( ) method (Alembic), Building a Migration Manually create_table( ) method (Alembic), Autogenerating a Migration, Building a Migration Manually create_unique_constraint( ) method (Alembic), Building a Migration Manually custom types, Types D data warehouse, Choosing Between SQLAlchemy Core and ORM DataAccessLayer class, Testing with a Test Database-Testing with a Test Database databasesconnecting to, Connecting to a Database-Connecting to a Database deleting data, Deleting Data, Deleting Data drivers required for, Installing Database Drivers inserting data, Inserting Data-Inserting Data, Deleting Data-Deleting Data, Inserting Data-Inserting Data, Deleting Data-Deleting Data migrations (see Alembic) querying data, Querying Data-Conjunctions, Joins-Raw Queries, Querying Data-Querying Data, Joins-Raw Queries reflecting (see reflection) supported, Introduction to SQLAlchemy, Installing Database Drivers tables in (see tables) updating data, Updating Data, Updating Data Date type, Types DATE type, Types DateTime type, Types DATETIME type, Types datetime.date type, Types datetime.datetime type, Types datetime.time type, Types datetime.timedelta type, Types db.py file, Configuring the Migration Environment Decimal type, Types DECIMAL type, Types decimal.Decimal type, Types declarative classes (see ORM classes) declarative_base objects, Defining Schema with SQLAlchemy ORM-Defining Tables via ORM Classes decorators, Using Mocks delete( ) function, Deleting Data, Deleting Data delete( ) method (Table), Deleting Data, Deleting Data deleting dataSQLAlchemy Core, Deleting Data SQLAlchemy ORM, Deleting Data desc( ) function, Ordering, Ordering detached session state, Session States DetachedInstanceError exception, DetachedInstanceError-DetachedInstanceError distinct( ) method (ClauseElement), ClauseElements domain-driven design, Choosing Between SQLAlchemy Core and ORM downgrade( ) method (Alembic), Generating a Base Empty Migration, Autogenerating a Migration, Building a Migration Manually drop_column( ) method (Alembic), Building a Migration Manually drop_constraint( ) method (Alembic), Building a Migration Manually drop_index( ) method (Alembic), Autogenerating a Migration, Building a Migration Manually drop_table( ) method (Alembic), Autogenerating a Migration, Building a Migration Manually E echo argument (create_engine), Connecting to a Database encoding argument (create_engine), Connecting to a Database endswith( ) method (ClauseElement), ClauseElements engine, creating, Connecting to a Database-Connecting to a Database Enum type, Types ENUM type, Types env.py file, Creating the Migration Environment, Configuring the Migration Environment error handlingAttributeError exception, AttributeError-AttributeError DetachedInstanceError exception, DetachedInstanceError-DetachedInstanceError IntegrityError exception, IntegrityError-IntegrityError, Transactions InvalidRequestError exception, Transactions MultipleResultsFound exception, MultipleResultsFound Exception-MultipleResultsFound Exception SQLAlchemy Core, Exceptions-Handling Errors try/except block, Handling Errors-Handling Errors, MultipleResultsFound Exception, DetachedInstanceError execute( ) method (Alembic), Building a Migration Manually execute( ) method (connection), Inserting Data, Inserting Data expunge( ) method (Session), Session States F fetchall( ) method (ResultProxy), Querying Data fetchone( ) method (ResultProxy), ResultProxy filter( ) function, Filtering filtering query results, Filtering-Conjunctions, Filtering-Filtering filter_by( ) function, Filtering first( ) method (ResultProxy), ResultProxy first( ) method (Session), Querying Data Flask, Integrating SQLAlchemy with Flask-Integrating SQLAlchemy with Flask, Where to Go from Here Flask Web Development (Grinberg), Where to Go from Here Flask-SQLalchemy package, Integrating SQLAlchemy with Flask Float type, Types float type, Types FLOAT type, Types flush( ) method (Session), Inserting Data fonts used in this book, Conventions Used in This Book foreign keysdefining, Relationships and ForeignKeyConstraints-Relationships and ForeignKeyConstraints, Relationships reflecting, Reflecting Individual Tables ForeignKeyConstraint, Relationships and ForeignKeyConstraints-Relationships and ForeignKeyConstraints functional testing, Testing with a Test Database-Testing with a Test Database, Testing with a Test Database-Testing with a Test Database G generative statement building, Inserting Data, Inserting Data, Ordering generic types, Types-Types Grinberg, Miguel (Flask Web Development), Where to Go from Here grouping in queries, Grouping, Grouping group_by( ) method (Table), Grouping, Grouping H hybrid attributes, Hybrid Attributes-Hybrid Attributes I icons used in this book, Conventions Used in This Book ilike( ) method (ClauseElement), ClauseElements indexesin Table objects, Indexes __init__( ) method, Association Proxy __init__.py file, Configuring the Migration Environment, Integrating SQLAlchemy with Flask insert( ) function, Inserting Data, Deleting Data insert( ) method (Table), Inserting Data-Inserting Data, Deleting Data inserted_primary_key( ) method (ResultProxy), Inserting Data inserting dataSQLAlchemy Core, Inserting Data-Inserting Data, Deleting Data-Deleting Data SQLAlchemy ORM, Inserting Data-Inserting Data, Deleting Data-Deleting Data inspect( ) method, Session States Int type, Types int type, Types Integer type, Types INTEGER type, Types IntegrityError exception, IntegrityError-IntegrityError, Transactions Interval type, Types INTERVAL type, Types Introducting Python (Lubanovic), Who This Book Is For Introduction to Python (videos, McKellar), Who This Book Is For InvalidRequestError exception, Transactions in_( ) method (ClauseElement), ClauseElements IPython, How to Use the Examples isolation level, setting, Connecting to a Database isolation_level argument (create_engine), Connecting to a Database is_( ) method (ClauseElement), ClauseElements J join( ) method (Table), Joins, Joins joins in queries, Joins-Joins, Joins-Joins K keys (see foreign keys; primary keys) keys( ) method (ResultProxy), ResultProxy L label( ) function, Built-In SQL Functions and Labels, Built-In SQL Functions and Labels LargeBinary type, Types Learning SQL (Beaulieu), Who This Book Is For like( ) method (ClauseElement), ClauseElements, Filtering limit( ) function, Limiting, Limiting limiting query results, Limiting, Limiting logging database actions, Connecting to a Database Lubanovic, Bill (Introducing Python), Who This Book Is For M McKellar, Jessica (Introduction to Python, videos), Who This Book Is For MetaData objects, Metadata MetaData.tables dictionary, Metadata microservices, Choosing Between SQLAlchemy Core and ORM migrations (see Alembic) mock library, Using Mocks, Using Mocks mocks, for testing, Using Mocks-Using Mocks, Using Mocks-Using Mocks modified property (inspector), Session States MultipleResultsFound exception, MultipleResultsFound Exception-MultipleResultsFound Exception MySQLconnecting to, Connecting to a Database connection timeouts, Connecting to a Database drivers for, Installing Database Drivers versions supported, Installing Database Drivers N not_( ) function, Conjunctions, Conjunctions not…( ) methods (ClauseElement), ClauseElements Numeric type, Types NUMERIC type, Types O one( ) method (Session), Querying Data, MultipleResultsFound Exception operators, in queries, Operators-Boolean Operators, Operators-Boolean Operators ordering query results, Ordering-Ordering, Ordering order_by( ) function, Ordering-Ordering, Ordering ORM (see SQLAlchemy ORM) ORM classesconstraints in, Keys, Constraints, and Indexes defining tables using, Defining Tables via ORM Classes-Keys, Constraints, and Indexes deleting data, Deleting Data inserting data, Inserting Data-Inserting Data, Deleting Data-Deleting Data keys in, Keys, Constraints, and Indexes persisting, Persisting the Schema querying data, Querying Data-Querying Data, Joins-Raw Queries relationships between, Relationships-Relationships, Deleting Data-Deleting Data, Reflected Relationships updating data, Updating Data or_( ) function, Conjunctions-Conjunctions, Boolean Operators outerjoin( ) method (Table), Joins, Joins P patch( ) method (mock), Using Mocks pending session state, Session States persistent session state, Session States persistingORM classes, Persisting the Schema Table objects, Persisting the Tables-Persisting the Tables pip install command, Installing SQLAlchemy and Connecting to a Database, Getting Started with Alembic pool_recycle argument (create_engine), Connecting to a Database PostgreSQLconnecting to, Connecting to a Database drivers for, Installing Database Drivers primary keysdefining, Keys and Constraints, Keys, Constraints, and Indexes determining, for inserted record, Inserting Data PrimaryKeyConstraint, Keys and Constraints Psycopg2 driver, Installing Database Drivers PyMySQL driver, Installing Database Drivers pytest, resources for, Where to Go from Here PythonREPL for, How to Use the Examples resources for, Who This Book Is For versions supported, Installing SQLAlchemy and Connecting to a Database Python DBAPI (PEP-249), Installing Database Drivers PyVideo website, Where to Go from Here Q query( ) method (Session), Querying Data-Conjunctions querying datareflected objects, Query Building with Reflected Objects-Query Building with Reflected Objects SQLAlchemy Core, Querying Data-Conjunctions, Joins-Raw Queries SQLAlchemy ORM, Querying Data-Conjunctions, Joins-Raw Queries querying reflected objects, Reflecting a Database with Automap R raw queries, Raw Queries, Raw Queries read, eval, print loop (REPL), How to Use the Examples REAL type, Types reflect( ) method (Metadata), Reflecting a Whole Database reflectionAutomap, Reflecting a Database with Automap-Reflected Relationships SQLAcodegen, SQLAcodegen-SQLAcodegen SQLAlchemy Core, Reflection-Reflecting a Whole Database SQLAlchemy ORM, Reflection with SQLAlchemy ORM and Automap relationship( ) method, Relationships, Joins remote_side option, Joins rename_table( ) method (Alembic), Building a Migration Manually, Building a Migration Manually REPL (read, eval, print loop), How to Use the Examples __repr__( ) method, The Session-The Session resources, Who This Book Is For, How to Contact Us, Where to Go from Here(see also website resources) Flask Web Development (Grinberg), Where to Go from Here Introducing Python (Lubanovic), Who This Book Is For Introduction to Python (videos, McKellar), Who This Book Is For IPython, How to Use the Examples Learning SQL (Beaulieu), Who This Book Is For ResultProxy objects, ResultProxy-ResultProxy rollback( ) method (Session), Transactions-Transactions rollback( ) method (transaction), Transactions-Transactions S scalar( ) function, Built-In SQL Functions and Labels scalar( ) method (ResultProxy), ResultProxy scalar( ) method (Session), Querying Data schema, Choosing Between SQLAlchemy Core and ORM, Defining Schema with SQLAlchemy ORM script.py.mako file, Creating the Migration Environment select( ) function, Querying Data, Controlling the Columns in the Query select( ) method (Table), Querying Data select_from( ) method (Table), Joins Session class, The Session-The Session sessionmaker class, The Session-The Session sessions, The Session-The Sessionstates of, Session States-Session States as transactions, The Session, Inserting Data-Inserting Data, Transactions-Transactions SMALLINT type, Types SQLgenerating for migration, Generating SQL-Generating SQL raw queries using, Raw Queries, Raw Queries resources for, Who This Book Is For SQL Expression Language, SQLAlchemy Core and the SQL Expression Language(see also SQLAlchemy Core) SQL functions, using in queries, Built-In SQL Functions and Labels, Built-In SQL Functions and Labels-Built-In SQL Functions and Labels SQLAcodegen, SQLAcodegen-SQLAcodegen sqlacodegen command, SQLAcodegen, SQLAcodegen SQLAlchemy, Introduction to SQLAlchemy-Choosing Between SQLAlchemy Core and ORMC extensions for, disabling, Installing SQLAlchemy and Connecting to a Database documentation for, Where to Go from Here engine, creating, Connecting to a Database-Connecting to a Database installing, Installing SQLAlchemy and Connecting to a Database-Installing Database Drivers plug-ins and extensions, Where to Go from Here SQLAlchemy Corecompared to ORM, Choosing Between SQLAlchemy Core and ORM-Choosing Between SQLAlchemy Core and ORM deleting data, Deleting Data error handling, Exceptions-Handling Errors inserting data, Inserting Data-Inserting Data, Deleting Data-Deleting Data querying data, Querying Data-Conjunctions, Joins-Raw Queries tables represented in (see Table objects) testing, Testing-Using Mocks transactions, Transactions-Transactions updating data, Updating Data using in combination with ORM, Choosing Between SQLAlchemy Core and ORM SQLAlchemy ORM, ORMcompared to Core, Choosing Between SQLAlchemy Core and ORM-Choosing Between SQLAlchemy Core and ORM deleting data, Deleting Data error handling, MultipleResultsFound Exception-DetachedInstanceError inserting data, Inserting Data-Inserting Data, Deleting Data-Deleting Data querying data, Querying Data-Conjunctions, Joins-Raw Queries reflection, Reflection with SQLAlchemy ORM and Automap sessions, The Session-The Session, The SQLAlchemy Session-Session States tables represented in (see ORM classes) testing, Testing with SQLAlchemy ORM-Using Mocks transactions, The Session, Inserting Data-Inserting Data, Transactions-Transactions updating data, Updating Data using in combination with Core, Choosing Between SQLAlchemy Core and ORM sqlalchemy.dialects module, Types sqlalchemy.func module, Built-In SQL Functions and Labels sqlalchemy.sql.func module, Built-In SQL Functions and Labels sqlalchemy.types module, Types SQLCompiler object, Inserting Data SQLiteconnecting to, Connecting to a Database version supported, Installing Database Drivers standard SQL types, Types startswith( ) method (ClauseElement), ClauseElements str type, Types, Types STRING type, Types strings, concatenating, Operators, Operators sum( ) function, Built-In SQL Functions and Labels, Built-In SQL Functions and Labels T __table__ attribute, Defining Tables via ORM Classes Table objects, Schema and Types, Tables-Relationships and ForeignKeyConstraintsconstraints in, Keys and Constraints-Keys and Constraints, Relationships and ForeignKeyConstraints-Relationships and ForeignKeyConstraints constructing, Tables deleting data, Deleting Data indexes in, Indexes inserting data, Inserting Data-Inserting Data, Deleting Data-Deleting Data keys in, Keys and Constraints persisting, Persisting the Tables-Persisting the Tables querying data, Querying Data-Conjunctions, Joins-Raw Queries relationships between, Relationships and ForeignKeyConstraints-Relationships and ForeignKeyConstraints updating data, Updating Data __tablename__ attribute, Defining Tables via ORM Classes-Defining Tables via ORM Classes tablescolumns in, Columns-Columns declarative classes representing (see ORM classes) reflecting all in a database, Reflecting a Whole Database reflecting individually, Reflecting Individual Tables-Reflecting Individual Tables representations of, Schema and Types user-defined (see Table objects) __table_args__ attribute, Keys, Constraints, and Indexes testingSQLAlchemy Core, Testing-Using Mocks SQLAlchemy ORM, Testing with SQLAlchemy ORM-Using Mocks Text type, Types TEXT type, Types, Types text( ) function, Raw Queries Time type, Types transaction objects, Transactions-Transactions transactionsSQLAlchemy Core, Transactions-Transactions SQLAlchemy ORM, The Session, Inserting Data-Inserting Data, Transactions-Transactions transient session state, Session States try/except block, Handling Errors-Handling Errors, MultipleResultsFound Exception, DetachedInstanceError types, Types-Typescustom types, Types generic, Types-Types standard SQL types, Types vendor-specific types, Types typographical conventions used in this book, Conventions Used in This Book U Unicode type, Types unicode type, Types UNICODE type, Types UniqueConstraint, Keys and Constraints Unit of Work pattern, Inserting Data unit testing, Testing with a Test Database-Testing with a Test Database, Testing with a Test Database-Testing with a Test Database unittest module, Testing with a Test Database-Testing with a Test Database, Testing with a Test Database-Testing with a Test Database update( ) function, Updating Data, Updating Data update( ) method (Table), Updating Data, Updating Data updating dataSQLAlchemy Core, Updating Data SQLAlchemy ORM, Updating Data upgrade( ) method (Alembic), Generating a Base Empty Migration, Autogenerating a Migration, Building a Migration Manually V values( ) function, Inserting Data VARCHAR type, Types vendor-specific types, Types versions directory, Creating the Migration Environment W website resources, How to Contact Us, Where to Go from Here(see also resources) Alembic documentation, Autogenerating a Migration, Generating SQL The Architecture of Open Source Applications (Bayer), Where to Go from Here association proxies, Association Proxy bulk operations, Inserting Data Chinook database, Reflection code examples, Using Code Examples hybrid attributes, Hybrid Attributes IPython, How to Use the Examples plug-ins and extensions, Where to Go from Here Psycopg2 driver, Installing Database Drivers pytest, Where to Go from Here PyVideo, Where to Go from Here reflected relationships, Reflected Relationships SQLAcodegen, SQLAcodegen SQLAlchemy documentation, Where to Go from Here Zen of Python, Connecting to a Database where( ) function, Filtering-Conjunctions Z Zen of Python (website), Connecting to a Database About the Authors Jason Myers works at Cisco as a Software Engineer working on OpenStack.


pages: 561 words: 157,589

WTF?: What's the Future and Why It's Up to Us by Tim O'Reilly

4chan, Affordable Care Act / Obamacare, Airbnb, Alvin Roth, Amazon Mechanical Turk, Amazon Web Services, artificial general intelligence, augmented reality, autonomous vehicles, barriers to entry, basic income, Bernie Madoff, Bernie Sanders, Bill Joy: nanobots, bitcoin, blockchain, Bretton Woods, Brewster Kahle, British Empire, business process, call centre, Capital in the Twenty-First Century by Thomas Piketty, Captain Sullenberger Hudson, Chuck Templeton: OpenTable:, Clayton Christensen, clean water, cloud computing, cognitive dissonance, collateralized debt obligation, commoditize, computer vision, corporate governance, corporate raider, creative destruction, crowdsourcing, Danny Hillis, data acquisition, deskilling, DevOps, Donald Davies, Donald Trump, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, Filter Bubble, Firefox, Flash crash, full employment, future of work, George Akerlof, gig economy, glass ceiling, Google Glasses, Gordon Gekko, gravity well, greed is good, Guido van Rossum, High speed trading, hiring and firing, Home mortgage interest deduction, Hyperloop, income inequality, index fund, informal economy, information asymmetry, Internet Archive, Internet of things, invention of movable type, invisible hand, iterative process, Jaron Lanier, Jeff Bezos, jitney, job automation, job satisfaction, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, Kevin Kelly, Khan Academy, Kickstarter, knowledge worker, Kodak vs Instagram, Lao Tzu, Larry Wall, Lean Startup, Leonard Kleinrock, Lyft, Marc Andreessen, Mark Zuckerberg, market fundamentalism, Marshall McLuhan, McMansion, microbiome, microservices, minimum viable product, mortgage tax deduction, move fast and break things, move fast and break things, Network effects, new economy, Nicholas Carr, obamacare, Oculus Rift, packet switching, PageRank, pattern recognition, Paul Buchheit, peer-to-peer, peer-to-peer model, Ponzi scheme, race to the bottom, Ralph Nader, randomized controlled trial, RFC: Request For Comment, Richard Feynman, Richard Stallman, ride hailing / ride sharing, Robert Gordon, Robert Metcalfe, Ronald Coase, Sam Altman, school choice, Second Machine Age, secular stagnation, self-driving car, SETI@home, shareholder value, Silicon Valley, Silicon Valley startup, skunkworks, Skype, smart contracts, Snapchat, Social Responsibility of Business Is to Increase Its Profits, social web, software as a service, software patent, spectrum auction, speech recognition, Stephen Hawking, Steve Ballmer, Steve Jobs, Steven Levy, Stewart Brand, strong AI, TaskRabbit, telepresence, the built environment, The Future of Employment, the map is not the territory, The Nature of the Firm, The Rise and Fall of American Growth, The Wealth of Nations by Adam Smith, Thomas Davenport, transaction costs, transcontinental railway, transportation-network company, Travis Kalanick, trickle-down economics, Uber and Lyft, Uber for X, uber lyft, ubercab, universal basic income, US Airways Flight 1549, VA Linux, Watson beat the top human players on Jeopardy!, We are the 99%, web application, Whole Earth Catalog, winner-take-all economy, women in the workforce, Y Combinator, yellow journalism, zero-sum game, Zipcar

But modern web applications may well be servicing millions of simultaneous users, and their functions have been decomposed into what are now called “microservices”—collections of individual functional building blocks that each do one thing, and do it very well. If a traditional monolithic application like Microsoft Word were reimplemented as a set of microservices, you could easily swap out the spell-checker for a better one, or add a new service that would turn web links into footnotes, or the reverse. Microservices are an evolution of the communications-oriented design pattern that we saw in the design of Unix and the Internet, and in Jeff Bezos’s platform memo. Microservices are defined by their inputs and outputs—how they communicate with other services—not by their internal implementation.


pages: 313 words: 75,583

Ansible for DevOps: Server and Configuration Management for Humans by Jeff Geerling

AGPL, Amazon Web Services, cloud computing, continuous integration, database schema, Debian, defense in depth, DevOps, fault tolerance, Firefox, full text search, Google Chrome, inventory management, loose coupling, microservices, Minecraft, MITM: man-in-the-middle, Ruby on Rails, web application

Zero-downtime multi-server deployments A single server deployment strategy is all that’s needed for many applications, but if you need to run an application on multiple servers for horizontal scalability or redundancy, deployments can be cumbersome—but not when you use Ansible! Server Check.in is a simple server and website monitoring service that has a microservices-based architecture; there is a website, an API application, and a server checking application. The server checking application needs to run on a variety of servers hosted around the world by different providers to provide redundancy and reliability. Server Check.in uses Ansible to manage rolling deployments for this application, so new code can be deployed across all the servers in minutes while maintaining 100% uptime!

The basic concept involves bringing up a parallel production infrastructure, then switching over to it. The cutover may take only a few milliseconds and no active production infrastructure is ever offline during the deployment process. A few different technologies and concepts, like container-based infrasturcture and microservices (which are faster to deploy), and better cloud autoscaling and load balancing options, have made blue-green deployments much easier than in the past. This book won’t go through a detailed example of this style of deployment, as the process is similar to other examples provided, the only difference being an additional task of switching a load balancer from the old to the new infrastructure once it’s up and running.


pages: 214 words: 31,751

Software Engineering at Google: Lessons Learned From Programming Over Time by Titus Winters, Tom Manshreck, Hyrum Wright

anti-pattern, computer vision, continuous integration, defense in depth, en.wikipedia.org, job automation, loss aversion, microservices, transaction costs, Turing complete

This question can arise at many levels of the software stack, because it is regularly the case that a bespoke solution customized for your narrow problem space may outperform the general utility solution that needs to handle all possibilities. By forking or reimplementing utility code and customizing it for your narrow domain you can add new features with greater ease, or optimize with greater certainty, regardless of whether we are talking about a microservice, an in-memory cache, a compression routine, or anything else in our software ecosystem. Perhaps more importantly, the control you gain from such a fork isolates you from changes in your underlying dependencies: those changes aren’t dictated by another team or third-party provider. You are in control of how and when to react to the passage of time and necessity to change.


pages: 514 words: 111,012

The Art of Monitoring by James Turnbull

Amazon Web Services, anti-pattern, cloud computing, continuous integration, correlation does not imply causation, Debian, DevOps, domain-specific language, failed state, Kickstarter, Kubernetes, microservices, performance metric, pull request, Ruby on Rails, software as a service, source of truth, web application, WebSocket

Tracing Before we move on from this chapter it would be remiss not to mention another diagnostic tool available for application monitoring: tracing, and more specifically, distributed tracing. This is a technique most useful for identifying performance problems and latency in distributed systems or microservices architectures. We're not going to cover it in the book but you should be aware of it if you go down the microservices path. A good example of a distributed tracing tool is Twitter's Zipkin project. Zipkin is in turn based on Google's Dapper paper on tracing. Summary In this chapter we explored several ways to monitor and instrument our applications and their workflow, including understanding where to place our application monitoring.


Mastering Ansible by Jesse Keating

cloud computing, Debian, DevOps, don't repeat yourself, microservices, remote working

The final two plays will just have skipped tasks: [ 150 ] Chapter 6 Now, let's execute the playbook again, and this time, we'll define the upgrade as true at runtime: [ 151 ] Minimizing Downtime with Rolling Deployments This time, we can see that our handlers are skipped but the final two plays have tasks that execute. In a real world scenario, where many more things are happening in the microA and microB roles, and potentially other micro-service roles on other hosts, the difference could be of many minutes or more. Clustering the restarts at the end can reduce the interruption period significantly. Running destructive tasks only once Destructive tasks come in many flavors. They can be one-way tasks that are extremely difficult to roll back, one-time tasks that cannot easily be rerun, or they can be race condition tasks that, if performed in parallel, would result in catastrophic failure.


Demystifying Smart Cities by Anders Lisdorf

3D printing, artificial general intelligence, autonomous vehicles, bitcoin, business intelligence, business process, chief data officer, clean water, cloud computing, computer vision, continuous integration, crowdsourcing, data is the new oil, digital twin, distributed ledger, don't be evil, Elon Musk, en.wikipedia.org, facts on the ground, Google Glasses, income inequality, Infrastructure as a Service, Internet of things, Masdar, microservices, Minecraft, platform as a service, ransomware, RFID, ride hailing / ride sharing, risk tolerance, self-driving car, smart cities, smart meter, software as a service, speech recognition, Stephen Hawking, Steve Jobs, Steve Wozniak, Stuxnet, Thomas Bayes, Turing test, urban sprawl, zero-sum game

These properties cannot be seen directly from the data but supplement the data and describes how to interpret and use the data. Both spheres are equally important in order to have a holistic concept of data value. Figure 10-1The seven dimensions of data value The data sphere Consumption – “How can the data be accessed?” For many years, the default answer has been microservices or another type of API as the way to access data. These are ways to programmatically interact with application functionality from other applications in a Request-Response type of interaction. The consuming application calls the API and requests a piece of data, and the API returns the data. The problem is if you are building a machine learning solution and need to loop through millions of entities as part of the process, this may not be the optimal solution.


pages: 266 words: 79,297

Forge Your Future with Open Source by VM (Vicky) Brasseur

AGPL, anti-pattern, Benevolent Dictator For Life (BDFL), call centre, continuous integration, Debian, DevOps, don't repeat yourself, en.wikipedia.org, Firefox, Guido van Rossum, Internet Archive, Larry Wall, microservices, Perl 6, premature optimization, pull request, Richard Stallman, risk tolerance, Turing machine

Second Edition A single dramatic software failure can cost a company millions of dollars—but can be avoided with simple changes to design and architecture. This new edition of the best-selling industry standard shows you how to create systems that run longer, with fewer failures, and recover better when bad things happen. New coverage includes DevOps, microservices, and cloud-native architecture. Stability antipatterns have grown to include systemic problems in large-scale systems. This is a must-have pragmatic guide to engineering for production systems. Michael Nygard (376 pages) ISBN: 9781680502398 $47.95 Your Code as a Crime Scene Jack the Ripper and legacy codebases have more in common than you’d think.


The Manager’s Path by Camille Fournier

failed state, fear of failure, hiring and firing, hive mind, interchangeable parts, job automation, Larry Wall, microservices, pull request, risk tolerance, Schrödinger's Cat, side project, Steve Jobs, WebSocket

When I joined, the engineering team was divided into roughly two groups: storefront, which did all development for the customer-facing website, and warehouse, which supported the software that ran the warehouse operations. We quickly evolved storefront to be frontend and backend because we were rewriting BOOTSTRAPPING CULTURE | 209 the code from a PHP monolith to a Java- and Ruby-based microservices architecture. Toward the end of my first year, we ran an experiment. We had a new product we wanted to build for the customer, a feature based on our customer photo reviews. Because finding a dress that would fit well was a challenge for our customers, we wanted to enable shoppers to see photos that other customers had uploaded showing themselves in the dresses, along with customer-provided information about their normal size, height, weight, and “shape” (athletic, pear, curvy, etc.).


pages: 334 words: 104,382

Brotopia: Breaking Up the Boys' Club of Silicon Valley by Emily Chang

23andMe, 4chan, Ada Lovelace, affirmative action, Airbnb, Apple II, augmented reality, autonomous vehicles, barriers to entry, Bernie Sanders, Burning Man, California gold rush, Chuck Templeton: OpenTable:, David Brooks, Donald Trump, Elon Musk, equal pay for equal work, Ferguson, Missouri, game design, gender pay gap, Google Glasses, Google X / Alphabet X, Grace Hopper, high net worth, Hyperloop, Jeff Bezos, job satisfaction, Khan Academy, Lyft, Marc Andreessen, Mark Zuckerberg, Maui Hawaii, Menlo Park, meta analysis, meta-analysis, microservices, paypal mafia, Peter Thiel, post-work, pull request, ride hailing / ride sharing, rolodex, Saturday Night Live, shareholder value, side project, Silicon Valley, Silicon Valley startup, Skype, Snapchat, Steve Jobs, Steve Wozniak, Steven Levy, subscription business, Tim Cook: Apple, Travis Kalanick, uber lyft, women in the workforce

At that point, she was told she was still not eligible, because her performance score had been retroactively recalibrated. Fowler says HR explained that she didn’t appear to have an “upward career trajectory” and needed to prove herself as an engineer. This explanation was particularly suspect, given that Fowler was publishing an engineering book, Production-Ready Microservices: Building Standardized Systems Across an Engineering Organization, and had spoken at major tech conferences—quite impressive for a rank-and-file employee. Fowler says she went home and cried. The newly negative review would not only hurt her take-home pay but also disqualify her from continuing in an Uber-sponsored graduate computer science program at Stanford that she had been attending.


pages: 499 words: 144,278

Coders: The Making of a New Tribe and the Remaking of the World by Clive Thompson

2013 Report for America's Infrastructure - American Society of Civil Engineers - 19 March 2013, 4chan, 8-hour work day, Ada Lovelace, AI winter, Airbnb, Amazon Web Services, Asperger Syndrome, augmented reality, Ayatollah Khomeini, barriers to entry, basic income, Bernie Sanders, bitcoin, blockchain, blue-collar work, Brewster Kahle, Brian Krebs, Broken windows theory, call centre, cellular automata, Chelsea Manning, clean water, cloud computing, cognitive dissonance, computer vision, Conway's Game of Life, crowdsourcing, cryptocurrency, Danny Hillis, David Heinemeier Hansson, don't be evil, don't repeat yourself, Donald Trump, dumpster diving, Edward Snowden, Elon Musk, Erik Brynjolfsson, Ernest Rutherford, Ethereum, ethereum blockchain, Firefox, Frederick Winslow Taylor, game design, glass ceiling, Golden Gate Park, Google Hangouts, Google X / Alphabet X, Grace Hopper, Guido van Rossum, Hacker Ethic, HyperCard, illegal immigration, ImageNet competition, Internet Archive, Internet of things, Jane Jacobs, John Markoff, Jony Ive, Julian Assange, Kickstarter, Larry Wall, lone genius, Lyft, Marc Andreessen, Mark Shuttleworth, Mark Zuckerberg, Menlo Park, microservices, Minecraft, move fast and break things, move fast and break things, Nate Silver, Network effects, neurotypical, Nicholas Carr, Oculus Rift, PageRank, pattern recognition, Paul Graham, paypal mafia, Peter Thiel, pink-collar, planetary scale, profit motive, ransomware, recommendation engine, Richard Stallman, ride hailing / ride sharing, Rubik’s Cube, Ruby on Rails, Sam Altman, Satoshi Nakamoto, Saturday Night Live, self-driving car, side project, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, single-payer health, Skype, smart contracts, Snapchat, social software, software is eating the world, sorting algorithm, South of Market, San Francisco, speech recognition, Steve Wozniak, Steven Levy, TaskRabbit, the High Line, Travis Kalanick, Uber and Lyft, Uber for X, uber lyft, universal basic income, urban planning, Wall-E, Watson beat the top human players on Jeopardy!, WikiLeaks, women in the workforce, Y Combinator, Zimmermann PGP, éminence grise

Consider Uber: In February 2017, the programmer Susan Fowler single-handedly shone a light into its culture, with a blog post describing “one very, very strange year” of working there. The company had a famously hard-driving, macho culture. Fowler enjoyed the insane pace of work and her talented colleagues; in her few spare hours, she authored a best-selling book on programming “microservices.” But she quickly discovered that harassment ran wild at Uber. Not long after Fowler had arrived, her manager propositioned her, via chat, for sex. When she told HR and top management, they said her manager was a “high performer” and this was his first offense, so “they wouldn’t feel comfortable giving him anything other than a warning and a stern talking-to.”