16 results back to index
Building Microservices by Sam Newman
airport security, Amazon Web Services, anti-pattern, business process, call centre, continuous integration, create, read, update, delete, defense in depth, don't repeat yourself, Edward Snowden, fault tolerance, index card, information retrieval, Infrastructure as a Service, inventory management, job automation, load shedding, loose coupling, platform as a service, premature optimization, pull request, recommendation engine, social graph, software as a service, source of truth, the built environment, web application, WebSocket, x509 certificate
Having an increased number of hosts has potential downsides, though. We have more servers to manage, and there might also be a cost implication of running more distinct hosts. Despite these problems, this is still the model I prefer for microservice architectures. And we’ll talk about a few things we can do to reduce the overhead of handling large numbers of hosts shortly. Platform as a Service When using a platform as a service (PaaS), you are working at a higher-level abstraction than at a single host. Most of these platforms rely on taking a technology-specific artifact, such as a Java WAR file or Ruby gem, and automatically provisioning and running it for you. Some of these platforms will transparently attempt to handle scaling the system up and down for you, although a more common (and in my experience less error-prone) way will allow you some control over how many nodes your service might run on, but it handles the rest.
N nested bounded contexts, Turtles All the Way Down network segregation, Network Segregation nonfunctional requirements, Cross-Functional Testing normalization of deviance, Flaky and Brittle Tests O on-demand provisioning systems, Scaling on-demand virtualization, Microservices onion architecture, The Technical Boundary Open Web Application Security Project (OWASP), Baking Security In OpenID Connect, Common Single Sign-On Implementations, Use SAML or OpenID Connect operating system artifacts, Operating System Artifacts operating systems security, Operating System orchestration architecture, Orchestration Versus Choreography organizational alignment, Organizational Alignment organizational structureConway's law and, Conway’s Law and System Design effect on systems design, Evidence loose vs. tightly coupled, Loose and Tightly Coupled Organizations orphaned services, The Orphaned Service? OSGI (Open Source Gateway Initiative), Modules ownershipshared, Drivers for Shared Services system design and, Service Ownership P Packer, Custom Images Pact, Pact Pacto, Pact partition tolerancein CAP theorem, CAP Theorem sacrificing, Sacrificing Partition Tolerance? passwords, Go with the Well Known performance tests, Performance Tests platform as a service (PaaS), Platform as a Service platform-specific artifacts, Platform-Specific Artifacts Polly for .NET, Bulkheads Postel's Law, Defer It for as Long as Possible predictive scaling, Autoscaling principal party, Authentication and Authorization privacy issues, Be Frugal property testing, Types of Tests proxy caching, Client-Side, Proxy, and Server-Side Caching R RabbitMQ, Technology Choices RDBMS (relational database management systems), Scaling for Reads reactive extensions (Rx), Reactive Extensions reactive scaling, Autoscaling read replicas, Scaling for Reads redesign, Starting Again refactoring databases, Staging the Break remote procedure calls, Remote Procedure Calls-Is RPC Terrible?
, Synthetic Monitoring semantic versioning, Use Semantic Versioning server-side caching, Client-Side, Proxy, and Server-Side Caching service accounts, Use SAML or OpenID Connect service boundaries, Zoning(see also modeling services) service calls, data retrieval via, Data Retrieval via Service Calls service configuration, Service Configuration service discovery, Service Discovery service ownershipcomprehensive approach, Service Ownership shared, Drivers for Shared Services service provider, Common Single Sign-On Implementations service separation, staging, Staging the Break(see also database decomposition) service templates, Tailored Service Template service testsCohn's Test Pyramid, Test Scope implementation of, Implementing Service Tests mocking vs. stubbing, Mocking or Stubbing Mountebank server for, A Smarter Stub Service scope of, Service Tests service-oriented architectures (SOA)concept of, What About Service-Oriented Architecture? drawbacks of, What About Service-Oriented Architecture? reuse of functionality in, Composability vs. microservices, What About Service-Oriented Architecture? service-to-host mapping, Service-to-Host Mapping-Platform as a Serviceapplication containers, Application Containers multiple services per host, Multiple Services Per Host platform as a service (PaaS), Platform as a Service single service per host, Single Service Per Host terminology, Service-to-Host Mapping service-to-service authentication/authorization, Service-to-Service Authentication and Authorization-The Deputy ProblemAPI keys, API Keys client certificates, Client Certificates confused deputy problem, The Deputy Problem HMAC over HTTP, HMAC Over HTTP HTTP(S) basic, HTTP(S) Basic Authentication man-in-the-middle attacks, Allow Everything Inside the Perimeter SAML/OpenID Connect, Use SAML or OpenID Connect sharding, Scaling for Writes shared code, DRY and the Perils of Code Reuse in a Microservice World shared data, Example: Shared Data shared libraries, Shared Libraries shared models, Shared and Hidden Models shared static data, Example: Shared Static Data shared tables, Example: Shared Tables sharing behavior, The Shared Database single sign-on (SSO), Common Single Sign-On Implementations-Single Sign-On Gateway smoke test suites, Separating Deployment from Release spies, Mocking or Stubbing SSH-multiplexing, Single Service, Multiple Servers SSL certificates, HTTP(S) Basic Authentication SSL termination, Load Balancing standards enforcementexemplars, Exemplars tailored service templates, Tailored Service Template standards establishment, The Required Standard-Architectural Safetyarchitectural safety, Architectural Safety importance of, The Required Standard interfaces, Interfaces monitoring, Monitoring static data, Example: Shared Static Data Strangler Application Pattern, The Strangler Pattern, Architectural Safety Measures strategic goalsreal-world example, A Real-World Example understanding, Strategic Goals stubbing vs. mocking, Mocking or Stubbing Suro, The Future Swagger, Swagger synchronous communication, Synchronous Versus Asynchronous synthetic monitoring, Synthetic Monitoring system designaccountability and, People adapting to communication pathways, Adapting to Communication Pathways bounded contexts, Bounded Contexts and Team Structures case study, Case Study: RealEstate.com.au Conway's law of, Conway’s Law and System Design delivery bottlenecks, Delivery Bottlenecks effect on organizational structure, Conway’s Law in Reverse feature teams, Feature Teams internal open source model, Internal Open Source organizational structure and, Evidence orphaned services, The Orphaned Service?
Albert Einstein, Apple's 1984 Super Bowl advert, barriers to entry, Bay Area Rapid Transit, business continuity plan, call centre, carbon footprint, Clayton Christensen, cloud computing, corporate social responsibility, crowdsourcing, iterative process, Maui Hawaii, Nicholas Carr, platform as a service, Silicon Valley, software as a service, Steve Ballmer, Steve Jobs
See also V2MOM Platform-as-a-Service. See Paas Playboy, 217 Positioning: developing strategies, 37–38; importance of, 23–25 Positioning (Ries and Trout), 37 Post-event responses, 56 Powell, General Colin, 136, 137, 138–139, 144, 146 Power: personal, 17 Power of Us, 146–147, 160 PowerUP, 144–145, 148 PR Week, 37 President’s Summit for America’s Future, 136–137 Pricing products, 78–79 Product development: communities of collaboration for, 131–132, 145; delivering products quickly, 107–108; ease of customer adoption, 119; encouraging customer feedback on, 13–14; ﬁnding talent for, 7–9, 15–17; harnessing customers’ ideas, 127–130; including global capabilities in, 169–170; innovative, 103–106; intelligent reaction in, 132–133; letting customers drive, 115–118; Platform-as-a-Service, 120–125; providing marketplace for solutions, 125–126; reusing infrastructure technology, 109–110; speed and simplicity in, 106–107; testing product usability, 13–14; transparency in, 110–113, 114 Product launch: choosing partners for, 179–182; cultural sensitivity for global, 192, 197; expansion without overspending, 176–177; global strategies for, 174–175; handling global disputes, 188–192; instilling corporate spirit in global leaders, 170–172; preparing for global capabilities, 169–170; projecting success for, 175–176; selecting global leadership, 186–188; sequential growth strategies, 177–178; telesales teams for, 172–173 Professional services.
Before long, there were more users, more activity, and more transactions moving through the Web services API than there were through the application. 119 BEHIND THE CLOUD Play #59: Transcend Technical Paradigms One of the most pivotal decisions we made as a company was to make our code available to let other companies build their own complementary online services. This idea to become a platform, or an operating system for the Internet (similar to how Windows is an operating system for PCs), offered a way to allow everyone to create applications online and gave us an opportunity to attract and retain more customers. This was the way to grow our company. Despite my bullish belief in Platform-as-a-Service (PaaS), the decision of whether or not to go ahead with the idea was a daunting one. Could we build an Internet operating system? There were potentially gigantic compatibility risks involved in allowing someone else’s code to operate on our system. We didn’t even know if customers would trust it. It wasn’t surprising that there wasn’t much internal support for such an untested idea. However, creating a platform offered a way to resolve our biggest problem: customers were clamoring for more applications, and we didn’t have the resources to build everything ourselves.
., 143 Merrill Lynch, 95 Metaphors for marketing, 44 Methods, 226, 228 Metrics: basing marketing on, 44–45; criticisms of, 227, 228; market segmenting and, 80–81; measuring employee success, 251; monitoring success with, 101; revenue vs. proﬁtability, 208–209 MGM, 203 Microloan Foundation, 157 Microsoft, 42, 43, 66, 172, 176, 253 Microsoft Hotmail, 20, 105 Millham, Brian, 83 Miner, Allen, 180–181 Minor, Halsey, 205 Mizuho Information Research, 185 Moellenhoff, Dave, 9–10, 11, 21, 106–107, 117–118, 139–140 Monster.com, 53 Moore, Charlie, 217–218 Morale, 245 Morgan Stanley, 123 Multiforce, 122 Murase, Haruo, 184–185 MyStarbucksIdea.com, 120, 129 N Nakada, Paul, 12, 237–238 National Geographic Society, 167 NativeEnergy, 155, 162 Negotiating sales terms, 94 Nestlé, 167 Netscape, 89 NetSuite, 161 Networking, 56 New York Stock Exchange, 153, 210–211, 212–213, 217, 258 New York Times, 41, 215–217, 220 Newsom, Gavin, 152 Nikkei, 185 No Software logo, 28–29, 32 Nucor, 255 O OASIS (Oracle Automatic Sales and Automatic Systems), 5 Obstacles: correcting, 233–236; recognizing strategic, 226–227, 228 1–1–1 Model: corporate structure for, 164–165; developing globally, 178; evolution of, 141–144; in-kind product donations, 156–158, 166; involving others in, 159–161; sharing publically, 146–147; success of, 256 Operation Smile, 141 Oprah Winfrey Show, 138 Oracle, 5, 14–16, 30–31, 63, 64, 104, 109, 137, 170–172, 179, 225–226, 253 Oracle’s Promise, 137, 140, 145 OutCast Communications, 24, 33 P PaaS (Platform-as-a-Service): AppExchange and, 125–126; developing applications with, 123–125; extending SaaS with, 122–123 Page, Larry, 160–161 PalmOne, 161 Panozza, Kevin, 190–191 Paperwork for sales, 94 Partnerships: collaborating with, 131–132; leveraging business, 59 Passion, 16 PayPal, 160, 172 275 INDEX Peek, Jeffrey, 93 PeopleFinder project, 163–164 PeopleSoft, 218 Performance incentives, 245–246 Philanthropic models: choosing cause for, 144–146; employee-inspired foundations, 161–165; Google’s philanthropic commitment, 160–161; in-kind product donations, 156–158, 166; including foundation in business model, 140–144; incorporating in existing company, 135–139; integrating with corporate organization, 139–140; involving business partners and networks in, 159–161; listening to constituents, 148–152; self-sustaining, 153–156, 166–168; sharing publically, 146–147 Pinkham, Elizabeth, 37 Planning: events, 59–60; global expansion, 182–186; headquarters and territories, 172–173; long-term revenue, 223; tax, 220; V2MOM for, 225–230 Planning.
The Practice of Cloud System Administration: DevOps and SRE Practices for Web Services, Volume 2 by Thomas A. Limoncelli, Strata R. Chalup, Christina J. Hogan
active measures, Amazon Web Services, anti-pattern, barriers to entry, business process, cloud computing, commoditize, continuous integration, correlation coefficient, database schema, Debian, defense in depth, delayed gratification, DevOps, domain-specific language, en.wikipedia.org, fault tolerance, finite state, Firefox, Google Glasses, information asymmetry, Infrastructure as a Service, intermodal, Internet of things, job automation, job satisfaction, load shedding, loose coupling, Malcom McLean invented shipping containers, Marc Andreessen, place-making, platform as a service, premature optimization, recommendation engine, revision control, risk tolerance, side project, Silicon Valley, software as a service, sorting algorithm, statistical model, Steven Levy, supply-chain management, Toyota Production System, web application, Yogi Berra
First printing, September 2014 Contents at a Glance Contents Preface About the Authors Introduction Part I Design: Building It Chapter 1 Designing in a Distributed World Chapter 2 Designing for Operations Chapter 3 Selecting a Service Platform Chapter 4 Application Architectures Chapter 5 Design Patterns for Scaling Chapter 6 Design Patterns for Resiliency Part II Operations: Running It Chapter 7 Operations in a Distributed World Chapter 8 DevOps Culture Chapter 9 Service Delivery: The Build Phase Chapter 10 Service Delivery: The Deployment Phase Chapter 11 Upgrading Live Services Chapter 12 Automation Chapter 13 Design Documents Chapter 14 Oncall Chapter 15 Disaster Preparedness Chapter 16 Monitoring Fundamentals Chapter 17 Monitoring Architecture and Practice Chapter 18 Capacity Planning Chapter 19 Creating KPIs Chapter 20 Operational Excellence Epilogue Part III Appendices Appendix A Assessments Appendix B The Origins and Future of Distributed Computing and Clouds Appendix C Scaling Terminology and Concepts Appendix D Templates and Examples Appendix E Recommended Reading Bibliography Index Contents Preface About the Authors Introduction Part I Design: Building It 1 Designing in a Distributed World 1.1 Visibility at Scale 1.2 The Importance of Simplicity 1.3 Composition 1.3.1 Load Balancer with Multiple Backend Replicas 1.3.2 Server with Multiple Backends 1.3.3 Server Tree 1.4 Distributed State 1.5 The CAP Principle 1.5.1 Consistency 1.5.2 Availability 1.5.3 Partition Tolerance 1.6 Loosely Coupled Systems 1.7 Speed 1.8 Summary Exercises 2 Designing for Operations 2.1 Operational Requirements 2.1.1 Configuration 2.1.2 Startup and Shutdown 2.1.3 Queue Draining 2.1.4 Software Upgrades 2.1.5 Backups and Restores 2.1.6 Redundancy 2.1.7 Replicated Databases 2.1.8 Hot Swaps 2.1.9 Toggles for Individual Features 2.1.10 Graceful Degradation 2.1.11 Access Controls and Rate Limits 2.1.12 Data Import Controls 2.1.13 Monitoring 2.1.14 Auditing 2.1.15 Debug Instrumentation 2.1.16 Exception Collection 2.1.17 Documentation for Operations 2.2 Implementing Design for Operations 2.2.1 Build Features in from the Beginning 2.2.2 Request Features as They Are Identified 2.2.3 Write the Features Yourself 2.2.4 Work with a Third-Party Vendor 2.3 Improving the Model 2.4 Summary Exercises 3 Selecting a Service Platform 3.1 Level of Service Abstraction 3.1.1 Infrastructure as a Service 3.1.2 Platform as a Service 3.1.3 Software as a Service 3.2 Type of Machine 3.2.1 Physical Machines 3.2.2 Virtual Machines 3.2.3 Containers 3.3 Level of Resource Sharing 3.3.1 Compliance 3.3.2 Privacy 3.3.3 Cost 3.3.4 Control 3.4 Colocation 3.5 Selection Strategies 3.6 Summary Exercises 4 Application Architectures 4.1 Single-Machine Web Server 4.2 Three-Tier Web Service 4.2.1 Load Balancer Types 4.2.2 Load Balancing Methods 4.2.3 Load Balancing with Shared State 4.2.4 User Identity 4.2.5 Scaling 4.3 Four-Tier Web Service 4.3.1 Frontends 4.3.2 Application Servers 4.3.3 Configuration Options 4.4 Reverse Proxy Service 4.5 Cloud-Scale Service 4.5.1 Global Load Balancer 4.5.2 Global Load Balancing Methods 4.5.3 Global Load Balancing with User-Specific Data 4.5.4 Internal Backbone 4.6 Message Bus Architectures 4.6.1 Message Bus Designs 4.6.2 Message Bus Reliability 4.6.3 Example 1: Link-Shortening Site 4.6.4 Example 2: Employee Human Resources Data Updates 4.7 Service-Oriented Architecture 4.7.1 Flexibility 4.7.2 Support 4.7.3 Best Practices 4.8 Summary Exercises 5 Design Patterns for Scaling 5.1 General Strategy 5.1.1 Identify Bottlenecks 5.1.2 Reengineer Components 5.1.3 Measure Results 5.1.4 Be Proactive 5.2 Scaling Up 5.3 The AKF Scaling Cube 5.3.1 x: Horizontal Duplication 5.3.2 y: Functional or Service Splits 5.3.3 z: Lookup-Oriented Split 5.3.4 Combinations 5.4 Caching 5.4.1 Cache Effectiveness 5.4.2 Cache Placement 5.4.3 Cache Persistence 5.4.4 Cache Replacement Algorithms 5.4.5 Cache Entry Invalidation 5.4.6 Cache Size 5.5 Data Sharding 5.6 Threading 5.7 Queueing 5.7.1 Benefits 5.7.2 Variations 5.8 Content Delivery Networks 5.9 Summary Exercises 6 Design Patterns for Resiliency 6.1 Software Resiliency Beats Hardware Reliability 6.2 Everything Malfunctions Eventually 6.2.1 MTBF in Distributed Systems 6.2.2 The Traditional Approach 6.2.3 The Distributed Computing Approach 6.3 Resiliency through Spare Capacity 6.3.1 How Much Spare Capacity 6.3.2 Load Sharing versus Hot Spares 6.4 Failure Domains 6.5 Software Failures 6.5.1 Software Crashes 6.5.2 Software Hangs 6.5.3 Query of Death 6.6 Physical Failures 6.6.1 Parts and Components 6.6.2 Machines 6.6.3 Load Balancers 6.6.4 Racks 6.6.5 Datacenters 6.7 Overload Failures 6.7.1 Traffic Surges 6.7.2 DoS and DDoS Attacks 6.7.3 Scraping Attacks 6.8 Human Error 6.9 Summary Exercises Part II Operations: Running It 7 Operations in a Distributed World 7.1 Distributed Systems Operations 7.1.1 SRE versus Traditional Enterprise IT 7.1.2 Change versus Stability 7.1.3 Defining SRE 7.1.4 Operations at Scale 7.2 Service Life Cycle 7.2.1 Service Launches 7.2.2 Service Decommissioning 7.3 Organizing Strategy for Operational Teams 7.3.1 Team Member Day Types 7.3.2 Other Strategies 7.4 Virtual Office 7.4.1 Communication Mechanisms 7.4.2 Communication Policies 7.5 Summary Exercises 8 DevOps Culture 8.1 What Is DevOps?
Strategies for choosing between these different services are summarized at the end of the chapter. The term “cloud” is ambiguous; it means different things to different people and has been made meaningless by marketing hype. Instead, we use the following terms to be specific: • Infrastructure as a Service (IaaS): Computer and network hardware, real or virtual, ready for you to use. • Platform as a Service (PaaS): Your software running in a vendor-provided framework or stack. • Software as a Service (SaaS): An application provided as a web site. Figure 3.1 depicts the typical consumer of each service. SaaS applications are for end users and fulfill a particular market niche. PaaS provides platforms for developers. IaaS is for operators looking to build their own platforms on which applications will be built, thus providing the most customizability.
Many provide both local load balancing and global load balancing, as will be described in Chapter 4. Some provide elastic scaling services, which automatically allocate and configure additional machines on demand as capacity is needed. Providers that offer both IaaS and PaaS often blur the line between the two by providing high-level managed services that are available to both. 3.1.2 Platform as a Service PaaS enables you to run your applications from a vendor-provided framework. These services offer you a high level of value, as they manage all aspects of the infrastructure, even much of the application stack. They offer very elastic scaling services, handling additional load without any input required from you. Generally you are not even aware of the specific resources dedicated to your application.
Puppet Essentials by Felix Frank
Craig has put up a comprehensive description on his blog, and the design has since been adopted by many users. Taking Puppet to the cloud It's time to finally talk about the cloud, which I managed to avoid when describing the different use cases. We will focus on the Infrastructure as a Service (IaaS) paradigm. These IaaS clouds consist of a network of virtual machines connected to the Internet. Each machine runs a basic operating system, which is chosen by the administrator. If you need a Platform as a Service (PaaS) implementation, read on to learn how you can practically implement your own PaaS system on top of an IaaS cloud using Puppet. From Puppet's point of view, an IaaS cloud is not much different from a data center. After all, this kind of cloud was conceived to serve as a stand-in for physical data centers. It just replaces rack-mounted servers with virtual machines, along with virtualized network connections.
[ 206 ] Index A agents initializing, in cloud 185 resources, exporting to 141 anchor pattern about 90 URL 91 antipatterns avoiding 154, 155 apt-get command 8 arrays 15 autorequire feature 125 autoscaling feature about 198 certificates, managing 198-200 round trip times, limiting 200-202 autosigning URL 200 autosigning script 198 B backends selecting 165 URL, for online documentation 165 beaker about 105 URL 105 before metaparameter 19, 21, 24 C classes about 66 component classes, writing 73, 74 comprehensive classes, writing 71, 72 creating, with parameters 92 declaring 66, 67 defining 66, 67 definitions, nesting 82 differentiating, with defined types 69, 70 include keyword, preferring 93 parameterized classes, consequences 92, 93 class inheritance 149 cloud agents, initializing in 185 manifests, building for 187 cloud-provisioner module using 186 collectors used, for realizing resources 140, 141 component classes writing 73, 74 composite design 71 comprehensive classes writing 71, 72 configuration data structuring, in hierarchy 161, 162 containers events, passing between classes and defined types 83-85 limitations 86-89 limitations, mitigating 90 ordering 86 relationships, establishing among 83 containers, limitations anchor pattern 90 contain function 91 control structures adding, in manifest 13, 14 creates parameter 28 cron resource type 29 custom attribute 191 custom facts about 53 Facter, extending with 53-55 custom functions about 96 used, for refining custom module interface 126-128 custom module building 105 enhancing, through facts 125 implementing 106-109 interface, refining through custom functions 126-128 making, portable across platforms 128, 129 naming 106 using 106 utilities, creating for derived manifests 110 custom types 117 D data resources, converting to 172-174 data, defining in manifest consequences 159, 160 defined types about 66 creating 67-69 differentiating, with classes 69, 70 used, for exploiting array values 78-81 using 67-69 using, as macros 77, 78 using, as resource multiplexers 76 using, as resource wrappers 74, 75 dependency 20 documentation, modules 98, 99 domain-specific language (DSL) 8 dynamic configuration files templating 134 dynamic scoping 154 E enabled property 10 ensure property 10 environment.conf file 100 environment locations configuring 100, 101 environments maintaining 99, 100 modules, installing 101, 102 modules, obtaining 101, 102 used, for testing modules 104, 105 evaluation order circular dependencies, avoiding 21, 22 controlling 16 dependencies, declaring 17-20 error propagation 20 events about 23 passing, between classes and defined types 83-85 exec resource type 27 external facts using 55, 56 External Node Classifiers (ENCs) 174 F Faces 186 Facter example 62 extending, with custom facts 53-55 goals 57 systems, summarizing with 50, 51 facts URL, for documentation 125 used, for enhancing custom module 125 fact values accessing 52, 53 using 52, 53 flexibility, providing to classes about 148 class inheritance 149 inheriting class, naming 151 parameters, making safer through inheritance 151 [ 208 ] Forge modules' characteristics, identifying 130 URL 130 used, for searching modules 130 fqdn_rand function 41 fully qualified domain name (FQDN) 52 G group resource type 26 H hashes 14 Hiera arrays, handling 170-172 class parameter values, binding 167-169 configuring 163 data, storing 164 hashes, handling 170-172 lookups, defining 179 practical example 177, 178 using, in different contexts 175, 176 values, retrieving 165 values, using in manifest 165 working with simple values 166, 167 hiera_array function 170 hiera_hash function 171 hierarchy configuration data, structuring in 161, 162 I immutability, variables 14 include keyword preferring 93 Infrastructure as a Service (IaaS) 184 Infrastructure as Code paradigm 105 inheriting class naming 151 installation, modules 101, 102 instances method 123 M manifest about 182 control structures, adding in 13, 14 dry-testing 12 structure 9 manifest, and Hiera designs selecting between 175 manifest, building for cloud about 187 arbitrary configuration files, composing 194-196 certificate names, selecting 190, 191 distributed catalog, creating 191-194 functionality, mapping to nodes 187-189 instance deletions, handling 197, 198 metaparameters 18 model substantiating, with providers 59, 60 modules about 96 agent, enhancing through plugins 116, 117 best practices 102 content structure 97, 98 documentation 98, 99 generalization, avoiding 103 identifying, in Forge 130 important parts 96 installing 101, 102 manifest files, gathering 102, 103 obtaining 101, 102 searching, in Forge 130 testing 104 testing, with environments 104, 105 URL, for publishing 98 monolithic implementation 71 mount resource type 29, 30 N Nginx about 45 Phusion Passenger, using with 45, 46 nodes file 100 Notice keyword 20 [ 209 ] O operatingsystemrelease fact 53 output interpreting, of puppet apply command 11, 12 P Proudly sourced and uploaded by [StormRG] Kickass Torrents | TPB | ExtraTorrent | h33t parameterized classes consequences 92, 93 parameters versus properties 10 parser functions 96 performance bottlenecks avoiding, from templates 136 performance considerations about 42 basic tuning 46 Passenger, using with Nginx 45 switching, to Phusion Passenger 43, 44 Phusion Passenger switching to 43, 44 URL, for installation instructions 45 using, with Nginx 45, 46 Platform as a Service (PaaS) 184 plugins about 116 custom types, creating 118 custom types, naming 118 management commands, declaring 121 provider, adding 121 provider, allowing to prefetch existing resources 123, 124 provider functionality, implementing 122, 123 resource names, using 120 resource type interface, creating 119 sensible parameter hooks, designing 120 types, making robust 125 used, for enhancing modules agent 116, 117 plugins, types custom facts 116 parser functions 116 providers 116 types 116 processorcount fact 52 properties about 10 versus parameters 10 providerless resource types 61 provider parameter 10 providers model, substantiating with 59, 60 summarizing 61 Puppet about 182 installing 8 modules 96 typical scopes 182 URL 182 Puppet agent certificate, renewing 40 life cycle 38, 39 running, from cron 41 setting up 35-37 puppet apply command about 9, 31 output, interpreting of 11, 12 PuppetBoard 186 Puppet Dashboard 186 Puppet Explorer 186 Puppet Labs URL 8 URL, for advanced approaches 43 URL, for core resource types 61 URL, for style guide 52 URL, for system installation information 32 URL, for Troubleshooting section 47 puppetlabs-strings module URL 99 Puppet master about 31 configuration settings, inspecting 35 master machine, setting up 32 master manifest, creating 33, 34 tasks 32 puppetmaster system service 33 puppet module install command 101 Puppet support, for SSL CSR attributes URL 199 [ 210 ] Puppet, taking to cloud about 184 agents, initializing 185 cloud-provisioner module, using 186 Puppet toolchain 46 rspec-puppet module about 105 URL 105 R separate data storage need for 158 singletons 135 site manifest 33 SSL troubleshooting 47, 48 stdlib module 101 strings 15 subscribe metaparameter 23 successful provisioning, ensuring about 202 manifests, testing 204, 205 necessary relationships, adding 203 systems summarizing, with Facter 50, 51 S realize function 138, 139 redundancy saving, resource defaults used 152, 153 relationships, containers performance implications 89 require metaparameter 19 resource chaining 17 resource defaults used, for saving redundancy 152, 153 resource interaction implementing 22-24 resource parameters overriding 147, 148 resources about 10 converting, to data 172-174 exporting 142 exporting, to agents 141 importing 142 realizing, collectors used 140, 141 resources, exporting about 141 central firewall, maintaining 146 custom configuration, automating 144 hosts files, managing 144 master configuration, for storing exported resources 142 Nagios configuration, simplifying 145, 146 SSH host keys, exporting 143 resource type life cycle, agent side 58, 59 resource types cron 29 examining 25, 26 exec 27, 28 group 26 mount 29, 30 user 26 revocation 39 Roles and Profiles pattern 183 T templates performance bottlenecks, avoiding from 136 using 135, 136 template syntax learning 134, 135 transaction 57 Trusted Facts 189 types about 117 summarizing 61 type system 57 typical scopes, Puppet about 182 profiles 183, 184 roles 183, 184 U user resource type 26 utilities, custom module complexity, dealing 115, 116 configuration items, adding 111, 112 creating, for derived manifests 110 [ 211 ] customization, allowing 113 unwanted configuration items, removing 114, 115 W Warning keyword 20 V Y Vagrant 182 variables using 14 variable types about 14 arrays 15 hashes 14 strings 15 virtual resources creating 137, 138 yum command 8 [ 212 ] Thank you for buying Puppet Essentials About Packt Publishing Packt, pronounced 'packed', published its first book "Mastering phpMyAdmin for Effective MySQL Management" in April 2004 and subsequently continued to specialize in publishing highly focused books on specific technologies and solutions.
The Art of Scalability: Scalable Web Architecture, Processes, and Organizations for the Modern Enterprise by Martin L. Abbott, Michael T. Fisher
always be closing, anti-pattern, barriers to entry, Bernie Madoff, business climate, business continuity plan, business intelligence, business process, call centre, cloud computing, combinatorial explosion, commoditize, Computer Numeric Control, conceptual framework, database schema, discounted cash flows, en.wikipedia.org, fault tolerance, finite state, friendly fire, hiring and firing, Infrastructure as a Service, inventory management, new economy, packet switching, performance metric, platform as a service, Ponzi scheme, RFC: Request For Comment, risk tolerance, Rubik’s Cube, Search for Extraterrestrial Intelligence, SETI@home, shareholder value, Silicon Valley, six sigma, software as a service, the scientific method, transaction costs, Vilfredo Pareto, web application, Y2K
Nov 13, 2006. http://www.businessweek.com/magazine/content/ 06_46/b4009001.htm. 427 428 C HAPTER 28 C LOUDS AND G RIDS • Software as a Service (SaaS). This was the original Blah as a Service term and started with software such as customer relationship management (CRM) software as some of the earliest offerings. Almost any form of software can be offered in this manner and it can be done either over the Web or via download. • Platform as a Service (PaaS). This model provides all the required components for developing and deploying Web applications and services. These components include workflow management, integrated development environments, testing, deployment, and hosting. • Infrastructure as a Service (IaaS). This is the concept of offering computing infrastructure such as servers, storage, network, and bandwidth for use as necessary by clients.
. • Developing alongside the idea of cloud computing was the concept of Software as a Service, Infrastructure as a Service, and many more “as a Service” concepts. • Software as a Service refers to almost any form of software that is offered in a pay as you use model. • Infrastructure as a Service is the idea of offering infrastructure such as storage, servers, network, and bandwidth in a pay as you use model. • Platform as a Service provides all the required components for developing and deploying Web applications and services. • Everything as a Service is the idea of being able to have small components that can be pieced together to provide a new service. • Grid computing as a concept has been around for almost two decades. It is used to describe the use of two or more computers processing individual parts of an overall task. • There are three types of cloud vendors: service providers, backbones, and virtualization software providers. 437 This page intentionally left blank Chapter 29 Soaring in the Clouds This is called, using the conquered foe to augment one’s own strength.
.”, 447 control, 444 IP address limitations, 444–445 limitations, 444–445 load balancing, 444–445 performance, 445–446 portability, 443 security, 443 summary of, 446–447 third-party software certification, 444–445 top ten obstacles, 447 Cloud computing, history of Artificial Intelligence, 427–428 dot com bubble, 427 IaaS (Infrastructure as a Service), 427–428 IBM, Autonomic Computing Manifesto, 427–428 PaaS (Platform-as-a-Service), 427–428 SaaS (Software as a Service), 427–428 XaaS (Everything as a Service), 427–428 Cloud computing, pros cost, 440–441 flexibility, 442 speed, 441–442 summary of, 442 Clouds backbones, 435–436 definition, 426 vs. grids, 434–436 hypervisors, 433 541 542 I NDEX Clouds (continued) public vs. private, 426 service providers, 435–436 types of, 435–436 virtualization software, 435–436 Clouds, characteristics of multiple tenants, 432–433, 434 pay by usage, 431, 434 in private clouds, 434 scale on demand, 431–432, 434 summary of, 433 virtualization, 433, 434 CMM (Capability Maturity Model), 124–125 CMMI (Capability Maturity Model Integrated), 124–125 COBIT (Control Objectives and related Technology), 133–134 Code complexity, AKF Scale Cube for applications, 346 Code reviews, as barrier conditions, 274 Codebase increases, scaling for, 343–344 Collins, Jim, 71 Commoditization over time, 301–302 Commodity hardware, architectural principles, 203 Communication breakdown from the business end, 106–107 destructive interference, 105–106, 212 educational mismatch, 106 the experiential chasm, 105–108 between management and technical teams, 105–108 matrix organization teams, 59 team size, warning sign, 50–51 from the technical end, 107–108 Communications improvement check lists and guidelines, 107–108 organizational design, influences on, 44 team size, 48 Communications and control, crisis management, 157–158 Comparing pros and cons, making tradeoff decisions, 291–292 real vs. ideal situations, 504 Compassion, leadership, 68 Competitive components, identifying, 239 Complexity business processes, managing, 128–130 grid drawback, 459–460 Concepts vs. rules, AKF Scale Cube, 325–326 Conditional approval by the ARB, 226, 229 Configuration management AKF Scale Cube for applications, 342 x-axis splits, databases, 361 Conflicts, organizational design, 17 Consensus, JAD (Joint Architecture Design), 218 Consulted persons, RASCI, 38 Content delivery networks (CDNs), 389–390 Control, cloud computing drawback, 444 Control Objectives and related Technology (COBIT), 133–134 Corporate mindset.
Flask Web Development: Developing Web Applications With Python by Miguel Grinberg
Messages logged on lesser levels can be logged to a file, syslog, or any other supported method by adding the proper logging handlers. The logging method to use for these messages largely depends on the hosting platform. Tip If you have cloned the application’s Git repository on GitHub, you can run git checkout 17b to check out this version of the application. Cloud Deployment The latest trend in application hosting is to host in the “cloud.” This technology, which is formally known as Platform as a Service (PaaS), frees the application developer from the mundane tasks of installing and maintaining the hardware and software platforms on which the application runs. In the PaaS model, a service provider offers a fully managed platform in which applications can run. The application developer uses tools and libraries from the provider to integrate the application with the platform. The application is then uploaded to the servers maintained by the provider and usually is deployed within seconds.
Industry 4.0: The Industrial Internet of Things by Alasdair Gilchrist
3D printing, additive manufacturing, Amazon Web Services, augmented reality, autonomous vehicles, barriers to entry, business intelligence, business process, chief data officer, cloud computing, connected car, cyber-physical system, deindustrialization, fault tolerance, global value chain, Google Glasses, hiring and firing, industrial robot, inflight wifi, Infrastructure as a Service, Internet of things, inventory management, job automation, low skilled workers, millennium bug, pattern recognition, peer-to-peer, platform as a service, pre–internet, race to the bottom, RFID, Skype, smart cities, smart grid, smart meter, smart transportation, software as a service, stealth mode startup, supply-chain management, trade route, web application, WebRTC, WebSocket, Y2K
There are three categories of service—IaaS (Infrastructure as a Service), PaaS (Platform as a Service), and SaaS (Software as a Service). Each category defines a set of services available to the customer, and this is key to the cloud— everything is offered as a service. This is based on the earlier SOA (service orientated architecture), where web services were used to access application functions. Similarly, the cloud operators use web services to expose their features and products as services. • IaaS (Infrastructure as a Service)—AWS’s basic product back in 2005 and it offered their excess infrastructure for lease to companies. Instead of buying hardware and establishing a server room or data center a SME could rent compute, storage, and network from Amazon, the beauty being they would only pay for what they used. • PaaS (Platform as a Service)—Came about as Microsoft and others realized that developers required not just infrastructure but access to software development languages, libraries, APIs, and microservices in order to build Windows-based applications.
Architecting For Scale by Lee Atchison
The server must be managed and maintained by the application owner, including software and security upgrades. Optimized Use Cases Cloud-based servers are good for a wide variety of general-purpose use cases and can be utilized for most scaling needs. Compute Slices Compute slices are an alternative execution model that involves executing applications without knowledge of which server they are running on. They involve taking an application’s software and deploying it to a Platform-as-a-Service (PaaS) infrastructure that will execute the stack in a managed way. This is done without exposing the specifics of the server on which software is running. There are several examples of compute slice–based compute engines, but Heroku Dynos is a classic example. Advantages Easy to vary allocated capacity at a relatively granular scale. Providers of compute slices have a strategy of over-provisioning of slices.
Site Reliability Engineering by Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy
Air France Flight 447, anti-pattern, barriers to entry, business intelligence, business process, Checklist Manifesto, cloud computing, combinatorial explosion, continuous integration, correlation does not imply causation, crowdsourcing, database schema, defense in depth, DevOps, en.wikipedia.org, fault tolerance, Flash crash, George Santayana, Google Chrome, Google Earth, job automation, job satisfaction, linear programming, load shedding, loose coupling, meta analysis, meta-analysis, minimum viable product, MVC pattern, performance metric, platform as a service, revision control, risk tolerance, side project, six sigma, the scientific method, Toyota Production System, trickle-down economics, web application, zero day
Once you’ve found the factors that caused the problem, it’s time to write up notes on what went wrong with the system, how you tracked down the problem, how you fixed the problem, and how to prevent it from happening again. In other words, you need to write a postmortem (although ideally, the system is alive at this point!). Case Study App Engine,16 part of Google’s Cloud Platform, is a platform-as-a-service product that allows developers to build services atop Google’s infrastructure. One of our internal customers filed a problem report indicating that they’d recently seen a dramatic increase in latency, CPU usage, and number of running processes needed to serve traffic for their app, a content-management system used to build documentation for developers.17 The customer couldn’t find any recent changes to their code that correlated with the increase in resources, and there hadn’t been an increase in traffic to their app (see Figure 12-3), so they were wondering if a change in the App Engine service was responsible.
She has previously written documentation for Google’s Data Center and Hardware Operations Teams in Mountain View and across its globally distributed datacenters. Before moving to New York, Betsy was a lecturer on technical writing at Stanford University. En route to her current career, Betsy studied International Relations and English Literature, and holds degrees from Stanford and Tulane. Chris Jones is a Site Reliability Engineer for Google App Engine, a cloud platform-as-a-service product serving over 28 billion requests per day. Based in San Francisco, he has previously been responsible for the care and feeding of Google’s advertising statistics, data warehousing, and customer support systems. In other lives, Chris has worked in academic IT, analyzed data for political campaigns, and engaged in some light BSD kernel hacking, picking up degrees in Computer Engineering, Economics, and Technology Policy along the way.
Digital Bank: Strategies for Launching or Becoming a Digital Bank by Chris Skinner
algorithmic trading, Amazon Web Services, Any sufficiently advanced technology is indistinguishable from magic, augmented reality, bank run, Basel III, bitcoin, business intelligence, business process, business process outsourcing, call centre, cashless society, clean water, cloud computing, corporate social responsibility, credit crunch, crowdsourcing, cryptocurrency, demand response, disintermediation, don't be evil, en.wikipedia.org, fault tolerance, fiat currency, financial innovation, Google Glasses, high net worth, informal economy, Infrastructure as a Service, Internet of things, Jeff Bezos, Kevin Kelly, Kickstarter, M-Pesa, margin call, mass affluent, mobile money, Mohammed Bouazizi, new economy, Northern Rock, Occupy movement, Pingit, platform as a service, Ponzi scheme, prediction markets, pre–internet, QR code, quantitative easing, ransomware, reserve currency, RFID, Satoshi Nakamoto, Silicon Valley, smart cities, software as a service, Steve Jobs, strong AI, Stuxnet, trade route, unbanked and underbanked, underbanked, upwardly mobile, We are the 99%, web application, Y2K
A slightly confusing and technical discussion, so let’s start with the idea of cloud computing in banking. Cloud Computing is a wide and diverse operation that has gained a panacea status of being all things to all people. It’s Salesforce.com, Azure, Exalogic, Amazon and more. Put in “Cloud Computing” to Google, who also provide clouds, and you get sponsored adverts from HP, Intel, Siemens and more all talking about clouds. It’s Software as a Service, Platform as a Service, and Infrastructure as a Service. It’s public clouds, private clouds, hybrid clouds. It’s every and any darned thing you want and, as a result, it’s lost its meaning. As a result, bank CIO’s have heard about Cloud Computing, but have no idea how to articulate what it is to their Board and CEO, how to justify it, how to present it as meaningful and how to get a decision. The Board and CEO have heard of cloud, but hear it’s dangerous.
Bayesian statistics, business intelligence, business process, cellular automata, Celtic Tiger, cloud computing, collateralized debt obligation, conceptual framework, congestion charging, corporate governance, correlation does not imply causation, crowdsourcing, discrete time, George Gilder, Google Earth, Infrastructure as a Service, Internet Archive, Internet of things, invisible hand, knowledge economy, late capitalism, lifelogging, linked data, Masdar, means of production, Nate Silver, natural language processing, openstreetmap, pattern recognition, platform as a service, recommendation engine, RFID, semantic web, sentiment analysis, slashdot, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, smart grid, smart meter, software as a service, statistical model, supply-chain management, the scientific method, The Signal and the Noise by Nate Silver, transaction costs
Since then, the relative share of digital data has continued to grow, especially with the development of distributed storage and services through cloud computing and data centres. Cloud computing takes two forms that often work cooperatively: utility clouds and data clouds (Farber et al. 2011). Utility clouds provide IT capabilities as locationindependent, on-demand services accessible via the Internet, including ‘infrastructure as a service’ (IaaS) such as storage, servers and networks, ‘platform as a service’ (PaaS) comprising an execution environment for the development of custom applications and databases, and ‘software as a service’ (SaaS) that enables users to access their applications and to process data remotely (Farber et al. 2011; Hancke et al. 2012). Data clouds enable massive volumes of data, that might be generated across an enterprise, to be linked, stored and processed remotely, drawing on the computational power of hundreds of machines, and analysed via utility services (Farber et al. 2011).
Apache Solr 3 Enterprise Search Server by Unknown
bioinformatics, continuous integration, database schema, en.wikipedia.org, fault tolerance, Firefox, full text search, information retrieval, Internet Archive, natural language processing, performance metric, platform as a service, Ruby on Rails, web application
Multi-site search is a strength of Drupal and provides the support of running multiple sites on a single codebase, such as drupal.org, groups.drupal.org, and api.drupal.org. Currently, part of the Apache Solr module is the ability to track where a document came from when indexed, and as a result, add the various sites as new filters in the search interface. Acquia's hosted search product is a great example of Platform as a Service (PaaS), and hosted Solr search is a very common integration approach for many organizations that don't wish to manage their own Java infrastructure or need to customize the behavior of Solr drastically. For a list of all the companies offering hosted Solr search please visit http://wiki.apache.org/solr/SolrHostingProviders. Ruby on Rails integrations There has been a lot of churn in the Ruby on Rails world for adding Solr support, with a number of competing libraries attempting to support Solr in the most Rails-native way.
3D printing, additive manufacturing, Airbus A320, Albert Einstein, Amazon Web Services, Any sufficiently advanced technology is indistinguishable from magic, asset-backed security, augmented reality, barriers to entry, bitcoin, bounce rate, business intelligence, business process, business process outsourcing, call centre, capital controls, citizen journalism, Clayton Christensen, cloud computing, credit crunch, crowdsourcing, disintermediation, en.wikipedia.org, fixed income, George Gilder, Google Glasses, high net worth, I think there is a world market for maybe five computers, Infrastructure as a Service, invention of the printing press, Jeff Bezos, jimmy wales, London Interbank Offered Rate, M-Pesa, Mark Zuckerberg, mass affluent, Metcalfe’s law, microcredit, mobile money, more computing power than Apollo, Northern Rock, Occupy movement, optical character recognition, peer-to-peer, performance metric, Pingit, platform as a service, QR code, QWERTY keyboard, Ray Kurzweil, recommendation engine, RFID, risk tolerance, Robert Metcalfe, self-driving car, Skype, speech recognition, stem cell, telepresence, Tim Cook: Apple, transaction costs, underbanked, US Airways Flight 1549, web application
BofA has set the target to create scaling and capability similar to that of Amazon Web Services environment too, according to Brad Spiers, Head of Compute Innovation at BofA. Interesting to note is that BofA is heavily investing in graphics processing capability, solid-state storage and in-memory databases, with large, fast processing and decision-making capability as the objective. NAB (previously National Australia Bank) of Australia has also committed extensively to private cloud infrastructure as it has moved to a platform-as-a-service concept as part of its programmes built around what it calls NextGen. NAB’s Next Generation Platform programme started as a core systems replacement, but has quickly morphed to encompass cloud capability. UBank, NAB’s online direct banking brand that launched in 2010, was the first to be deployed on this platform in the market. As mentioned earlier, UBank has been an outstanding success.
Clojure Programming by Chas Emerick, Brian Carper, Christophe Grand
Amazon Web Services, Benoit Mandelbrot, cloud computing, continuous integration, database schema, domain-specific language, don't repeat yourself, en.wikipedia.org, failed state, finite state, Firefox, game design, general-purpose programming language, Guido van Rossum, Larry Wall, mandelbrot fractal, Paul Graham, platform as a service, premature optimization, random walk, Ruby on Rails, Schrödinger's Cat, semantic web, software as a service, sorting algorithm, Turing complete, type inference, web application
However, if you’re game, there are some specific Clojure-friendly toolchains that can make application deployment a lot simpler, easier, and more automated than most other options. We’ll take a look at one, Amazon’s Elastic Beanstalk service, that is broadly applicable to Clojure web applications, automating the provisioning and configuration of servers and deployment of applications to those servers. Deploying Clojure Apps to Amazon’s Elastic Beanstalk Amazon’s Elastic Beanstalk (EB) is a platform as a service that provides a thin layer of automation and deployment management tools on top of Amazon Web Services’s (AWS) lower-level EC2 compute and load balancer services. EB allows you to programmatically provision and control environments (collections of one or more application servers fronted by a load balancer), to which you can deploy different versions of your application. The load balancers used by EB are integrated with this provisioning mechanism, so that when your application experiences higher load (based on metrics you define, such as number of requests or aggregate bandwidth utilized per minute), the corresponding EB environment is expanded to contain more app servers to service that load.
The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling by Ralph Kimball, Margy Ross
active measures, Albert Einstein, business intelligence, business process, call centre, cloud computing, data acquisition, discrete time, inventory management, iterative process, job automation, knowledge worker, performance metric, platform as a service, side project, supply-chain management, zero-sum game
Rather, plan for disruptive changes coming from every direction: new data types, competitive challenges, programming approaches, hardware, networking technology, and services offered by literally hundreds of new big data providers. For the foreseeable future, maintain a balance among several implementation approaches including Hadoop, traditional grid computing, pushdown optimization in an RDBMS, on-premise computing, cloud computing, and even the mainframe. None of these approaches will be the single winner in the long run. Platform as a service (PaaS) providers offer an attractive option that can help assemble a compatible set of tools. Think of Hadoop as a ﬂexible, general purpose environment for many forms of ETL processing, where the goal is to add sufficient structure and context to big data so that it can be loaded into an RDBMS. The same data in Hadoop can be accessed and transformed with Hive, Pig, HBase, and MapReduce code written in a variety of languages, even simultaneously.