You may be wondering, who we are, and why we wrote this book.
At the end of Harry’s last book, Test-Driven Development with Python, he found himself asking a bunch of questions about architecture — what’s the best way of structuring your application so that it’s easy to test? More specifically, so that your core business logic is covered by unit tests, and so that we minimise the number of integration and end-to-end tests we need? He made vague references to "Hexagonal Architecture" and "Ports and Adapters" and "Functional Core, Imperative Shell," but if he was honest, he’d have to admit that these weren’t things he really understood or had done in practice.
And then he was lucky enough to run into Bob, who has the answers to all these questions.
Bob ended up a software architect because nobody else on his team was doing it. He turned out to be pretty bad at it, but he was lucky enough to run into Ian Cooper, who taught him new ways of writing and thinking about code.
We both work for MADE.com - a European e-commerce company who sell furniture online - where we apply the techniques in this book to build distributed systems that model real world business problems. Our example domain is the first system Bob built for MADE, and this book is an attempt to write down all the stuff we have to teach new programmers when they join one of our teams.
MADE.com operate a global supply chain of freight partners and manufacturers. To try and keep costs low, we try to optimise the delivery of stock to our warehouses so that we don’t have unsold goods lying around the place.
Ideally, the sofa that you want to buy will arrive in port on the very day that you decide to buy it, and we’ll ship it straight to your house without ever storing it. Getting the timing right is a tricky balancing act when goods take 3 months to arrive by container ship. Along the way things get broken, or water damaged; storms cause unexpected delays, logistics partners mishandle goods, paperwork goes missing, customers change their minds and amend their orders, and so on.
We solve those problems by building intelligent software that represents the kind of operations taking place in the real world so that we can automate as much of the business as possible.
If you’re reading this book, we probably don’t need to convince you that Python is great, so the real question is "Why does the Python community need a book like this?"
The answer is about Python’s popularity and maturity - although Python is probably the world’s fastest-growing programming language, and nearing the top of the absolute popularity tables, it’s only just starting to take on the kinds of problems that the C# and Java world have been working on for years. Startups become real businesses, web apps and scripted automations are becoming (whisper it) enterprise software.
In the Python world, we often quote the Zen of Python:[1] "there should be one—and preferably only one—obvious way to do it." Unfortunately, as project size grows, the most obvious way of doing things isn’t always the way that helps you manage complexity and evolving requirements.
None of the techniques and patterns we’re going to discuss in this book are new, but they are mostly new to the Python world. And this book won’t be a replacement for the classics in the field like Eric Evans' Domain-Driven Design or Martin Fowler’s Patterns of Enterprise Application Architecture (both of which we often refer to and encourage you to go and read).
But all the classic code examples in the literature do tend to be written in Java or C++/#, and if you’re a Python person and haven’t used either of those languages in a long time (or indeed ever), it can make them quite trying. There’s a reason the latest edition of that other classic text, Refactoring is in JavaScript.
So we hope this book will make for a lightweight introduction to some of the key architectural patterns that support domain-driven design (DDD) and event-driven microservices, that it will serve as a reference for implementing them in a Pythonic way, and that it will serve as a first step for those who want to do further research in this field.
Here are a few things we assume about you, dear reader.
We assume you’ve been close to some reasonably complex Python applications.
We assume you’ve seen some of the pain that comes with trying to manage that complexity.
We do not assume that you already know anything about DDD, or any of the classic application architecture patterns.
We structure our explorations of architectural patterns around an example app, building it up chapter by chapter. We use test-driven development (TDD) at work, so we tend to show listings of tests first, followed by implementation. If you’re not used to working test-first, it may feel a little strange at the beginning, but we hope you’ll soon get used to seeing code "being used," i.e. from the outside, before you see how it’s built on the inside.
We use some specific Python (version 3) frameworks and technologies, like Flask, SQLAlchemy, and Pytest, as well as Docker and Redis. If you’re already familiar with them, that won’t hurt, but we don’t think it’s required. One of our main aims with this book is to build an architecture where specific technology choices become minor implementation details.
- Chapter 1: Domain Modelling and DDD
-
At some level, everyone has learned the lesson that complex business problems need to be reflected in code, in the form of a model of the domain. But why does it always seem to be so hard to do it, without getting tangled up with infrastructure concerns, with our web frameworks, or whatever else? In this chapter we give a broad overview of domain modelling and DDD, and show how to get started with a model that has no external dependencies, and fast unit tests.
- Chapter 2, 4 and 5: Repository, Service Layer and Unit of Work Patterns
-
In these three chapters we present three closely related and mutually reinforcing patterns that support our ambition to keep the model free of extraneous dependencies. We build a layer of abstraction around persistent storage, and we build a Service Layer to define the entrypoints to our system, and capture the primary use cases. We show how this layer makes it easy to build very thin entrypoints to our system, be it a Flask API or a CLI.
- Chapter 3: An Aside on Coupling and Abstractions
-
After presenting the first abstraction (Repository pattern), we take the opportunity for a general discussion of how to choose abstractions, and what their role is in choosing how our software is coupled together.
- Chapter 6: Aggregate Pattern
-
A brief return to the world of DDD, where we discuss how to choose the right Aggregate, and how this choice relates to questions of data integrity
- Chapters 7, 8 and 9: Event-Driven Architecture
-
We introduce three more mutually-reinforcing patterns, starting with the concept of Domain Events, a vehicle for capturing the idea that some interactions with a system are triggers for others. We use a Message Bus to allow actions to trigger events, and call appropriate Handlers. We move on to discuss how events can be used as a pattern for integration between services, in a microservices architecture. Finally we add the distinction between Commands and Events. Our application is now fundamentally a message-processing system.
- Chapter 10: CQRS
-
An example of command-query responsibility segregation, with and without events.
- Chapter 11 Dependency Injection
-
We tidy up our explicit and implicit dependencies, and implement a very simple dependency injection framework.
Implementing architectural patterns always looks easy when you show a simple example, starting from scratch, but many of you will probably be wondering how to apply these principles to existing software. We’ll attempt to provide a few pointers in this last chapter and some links to further reading.
You’re reading a book, but you’ll probably agree with us when we say that the best way to learn about code is to code. We learned most of what we know from pairing with people, writing code with them, and learning by doing, and we’d like to recreate that experience as much as possible for you in this book.
As a result, we’ve structured the book around a single example project (although we do sometimes throw in other examples), which we build up as we go, and the narrative of the book is as if you’re pairing with us as we go, and we’re explaining what we’re doing and why at each step.
But to really get to grips with these patterns, you need to mess about with the code and actually get a feel for how it works. You’ll find all the code on GitHub; each chapter has its own branch. You can find a list of them here: https://github.com/python-leap/code/branches/all
Here’s three different ways you might code along with the book:
-
Start your own repo and try and build up the app as we do, following the examples from listings in the book, and occasionally looking to our repo for hints. A word of warning however, if you’ve read Harry’s previous book and coded along with that, you’ll find there is much more to figure out on your own; you may need to lean pretty heavily on the working versions on GitHub.
-
Try to apply these each pattern, chapter-by-chapter, to your own (preferably small/toy) project, and see if you can make it work for your use case. This is high-risk / high-reward (and high effort besides!). It may take quite some work to get things working for the specifics of your project, but on the other hand you’re likely to learn the most
-
For lower effort, in each chapter we’ll outline an "exercise for the reader," and point you to a Github location where you can download some partially-finished code for the chapter with a few missing parts to write yourself.
If you want to go all the way to town, why not try and build up the code as you read along? Particularly if you’re intending to apply some of these patterns in your own projects, then working through a simple example can really help you to get some safe practice.
The code (and the online version of the book) is licensed under a Creative Commons CC-By-ND license. If you want to re-use any of the content from this book and you have any worries about the license terms you can contact O’Reilly at [email protected].
The following typographical conventions are used in this book:
- Italic
-
Indicates new terms, URLs, email addresses, filenames, and file extensions.
- Constant width
-
Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.
Constant width bold-
Shows commands or other text that should be typed literally by the user.
- Constant width italic
-
Shows text that should be replaced with user-supplied values or by values determined by context.
|
Tip
|
This element signifies a tip or suggestion. |
|
Note
|
This element signifies a general note. |
|
Warning
|
This element indicates a warning or caution. |
|
Note
|
Safari (formerly Safari Books Online) is a membership-based training and reference platform for enterprise, government, educators, and individuals. |
Members have access to thousands of books, training videos, Learning Paths, interactive tutorials, and curated playlists from over 250 publishers, including O’Reilly Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and Course Technology, among others.
For more information, please visit http://oreilly.com/safari.
Please address comments and questions concerning this book to the publisher:
- O’Reilly Media, Inc.
- 1005 Gravenstein Highway North
- Sebastopol, CA 95472
- 800-998-9938 (in the United States or Canada)
- 707-829-0515 (international or local)
- 707-829-0104 (fax)
We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://www.oreilly.com/catalog/<catalog page>.
To comment or ask technical questions about this book, send email to [email protected].
For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia
python -c "import this"