preface I was introduced to orchestrators when I started at Google in 2007. And my introduction was not to Borg but rather Ganeti. Ganeti was an internally developed cluster management system that operated on virtual machines. At the time, it was a basic wrapper around the open source version of Xen, and it provided a clustered solution that allowed us to provide virtual (instead of physical) machines to engineers.
We didn’t refer to Ganeti as an orchestrator, nor did we talk about it in the same vein as Borg. In hindsight, I don’t think it’s too much of a stretch to consider Ganeti a kind of orchestrator. Instead of operating on tasks (in the form of containers), it operated on virtual machines. Internally at Google, Ganeti served as a bridge from a world where some engineers could run their applications on physical machines to a world where every engineer ran their applications on Borg.
Several years later, I got a proper introduction to Borg when we rewrote the life cycle management system we built to manage Ganeti clusters and virtual machines. We ran it on Borg. Fast-forward to 2020. The COVID pandemic hit, and like everyone else, I found myself working from home. Suddenly, I had three-plus extra hours per day as a result of not having to commute to a Manhattan office. What to do?
Of course, the obvious thing to do was to start a personal project of some kind. But what? After 13 years of working with orchestrators, I thought it might be fun to try to write one from scratch. How hard could it be?
I spent most of the summer of 2020 working on my orchestrator. I called it Cube in an effort to continue the Star Trek theme. Surprisingly, I got it working in less than 3,000 lines of code.
Around the same time, I read Thorsten Ball’s Writing an Interpreter in Go. While I wasn’t necessarily interested in interpreters or writing programming languages, I was interested in learning how they work. And then it hit me! I could do a book about writing an orchestrator in Go. It would be the book that I wish I’d had back in 2007! Thus was born the book you have in front of you now.
I realized early on in the writing process that orchestration is a big topic. It’s easy to get distracted by secondary concerns when talking about orchestration systems. How do you handle service discovery? How do you handle DNS? What about consensus? I wanted to strip away all the stuff that gets piled on top of orchestration systems and present just the core, the foundation on which all that other stuff sits. Not that things like service discovery, DNS, and load balancing are unimportant. But in the context of an orchestration system, we talk about those things because they are tools in service to the core function of an orchestrator: scheduling applications to run on a pool of nodes and managing their life cycle.
So in a nutshell, that’s what this book is about: taking a request from a user to run an application, identifying a machine that can run the application, and then sending a request to the chosen machine to start the application. It seems simple when you put it that way, doesn’t it?
In addition to presenting the foundational concepts of an orchestration system, another goal in writing this book is to make the content approachable to a broad audience. So while I’ve chosen to write the Cube orchestrator in the Go programming language, my hope is that anyone can work through the book and get the code working, even if you’ve never written a line of Go in your life. All of the code uses basic Go features. While we do use goroutines to do some basic concurrency, we don’t use channels (there are many great resources to learn about concurrency if you’re interested). And we don’t use generics. (Shortly after Go 1.18 was released, I did attempt to refactor the code and manuscript to use generics. While I got the code working, I realized it introduced unnecessary complexity to the book. It became one more thing to explain in an already long list of things to explain.)
I hope you have fun while you read this book. And in the process of having fun, I hope you learn as much from this book as I did writing it.
acknowledgments With most things in life, we get a lot of help from others. And this book is no different. I’d like to start by acknowledging all the folks at Manning who helped make this book a reality. I’d like to thank Andy Waldron for taking on my book and believing in it throughout its many shapes. Katie Sposato Johnson was instrumental in helping me navigate the Manning process. Without her help, this book would not exist. I’d also like to thank the many other folks at Manning who have worked on the production and marketing of the book. I want to thank the reviewers who read the manuscript at various stages and provided thoughtful feedback: Alain Lompo, Alessandro Campeis, Andres Sacco, Becky Huett, Bobby Lin, Christopher Villanueva, Clifford Thurber, David Paccoud, Emanuele Piccinelli, Ernesto Bossi, Fernando Bernardino, Fernando Rodrigues, Geert Van Laethem, Gregory Reshetniak, Katia Patkin, Kosmas Chatzimichalis, Larry Cai, Lucian Enache, Madiha Khalid, Matthias Busch, Michael Bright, Muneeb Shaikh, Nathan B. Crocker, Nghia To, Richard Vaughan, Sanket Naik, Simone Sguazza, Thomas Dybdahl, Timothy R. J. Langford, Tim van Deurzen, and Vamsi Krishna. Special thanks also go out to Mike Haller, technical proofreader, for his thorough review of the code shortly before the book went into production. It’s quite challenging to keep the code presented in the book in sync with the source code, but Mike was invaluable in helping me clean up the many discrepancies.
about this book Build an Orchestrator in Go (From Scratch) was written to help you better understand the fundamental components of orchestration systems. Whether you work as a DevOps engineer, site reliability engineer (SRE), or software engineer, much of today’s technology can seem like a black box. You just deploy it to the cloud, and then magical “stuff” happens. As we all know, magical technology is great when it works! When it fails—and it will fail!—that magical aspect can be a barrier to quickly identifying problems and fixing them. As more developers move their applications to the cloud, they are running them (or will do so) on an orchestration system. Unless they work at a larger company that has a dedicated DevOps or SRE staff, they will likely need to deploy and manage their applications themselves. This includes handling problems when they arise. My hope is that this book will remove some of the magic from how applications run on an orchestrator. Who should read this book Build an Orchestrator in Go (From Scratch) is for anyone responsible for deploying and operating an orchestration system (i.e., DevOps engineers and SREs) and for anyone responsible for deploying and managing applications that run on an orchestration system (i.e., software engineers). If you want to learn how orchestrators work, you could read the source code for either Kubernetes or Nomad, both open source projects available on GitHub. Kubernetes has 5 million lines of Go code. Nomad has a considerably smaller codebase, but it’s still 500,000-plus lines of Go. I don’t know about you, but I would struggle to get much value from trying to make sense of half a million lines of code, let alone 5 million lines! How this book is organized: A road map The book has five parts that cover 13 chapters. Part 1 introduces the mental model for the Cube orchestrator and sets up the skeleton codebase that will be implemented throughout the rest of the book: Chapter 1 briefly explains the purpose of orchestration systems and then describes the mental model for Cube, the orchestrator implemented throughout the rest of the book. Chapter 2 uses the mental model from chapter 1 to create a skeleton codebase for the core concepts of the Cube orchestrator. Chapter 3 illustrates how we’ll implement the codebase by taking the skeleton for the Task object and fleshing it out in detail. Part 2 implements the concepts necessary for the worker component: Chapter 4 fleshes out the implementation details of the Worker object. Chapter 5 builds an API for the Worker. Chapter 6 creates a framework for the worker to expose metrics about its state and the state of the tasks it’s running. Part 3 implements the concepts necessary for the manager component: Chapter 7 fleshes out the implementation details of the Manager object. Chapter 8 builds an API for the Manager. Chapter 9 explores failure scenarios and implements solutions to handle them. Part 4 walks the reader through refactoring two components from the initial implementation: Chapter 10 describes a scheduler interface and implements a more sophisticated scheduling algorithm. Chapter 11 designs and builds a storage interface that allows the manager and worker components to store their tasks in-memory or persistently in a database. Part 5 implements a command-line interface (CLI) that allows the reader to operate the orchestrator: Chapter 12 builds a CLI that implements commands for starting the manager and worker, starting and stopping tasks, and getting the status of tasks in the system. Chapter 13 offers a summary of what we’ve accomplished and provides some suggestions for where to go from here.
About the code This book contains many examples of source code both in numbered listings and in line with normal text. In both cases, source code is formatted in a fixed-width font like this to separate it from ordinary text. Sometimes code is also in bold to highlight code that has changed from previous steps in the chapter, such as when a new feature is added to an existing line of code.
In many cases, the original source code has been reformatted; we’ve added line breaks and reworked indentation to accommodate the available page space in the book. In rare cases, even this was not enough, and listings include line-continuation markers (➥). Additionally, comments in the source code have often been removed from the listings when the code is described in the text. Code annotations accompany many of the listings, highlighting important concepts.
You can get executable snippets of code from the liveBook (online) version of this book at https://livebook.manning.com/book/build-an-orchestrator-in-go-from-scratch. The complete code for the examples in the book is available for download from the Manning website at https://www.manning.com/books/build-an-orchestrator-in-go -from-scratch and from GitHub at https://github.com/buildorchestratoringo/code.
liveBook discussion forum Purchase of Build an Orchestrator in Go (From Scratch) includes free access to a private web forum run by Manning Publications where you can make comments about the book, ask technical questions, and receive help from the author and from other users. To access the forum and subscribe to it, point your web browser to https://www.manning .com/books/build-an-orchestrator-in-go-from-scratch. This page provides information on how to get on the forum once you’re registered, what kind of help is available, and the rules of conduct on the forum.
Manning’s commitment to our readers is to provide a venue where a meaningful dialog between individual readers and between readers and the author can take place. It is not a commitment to any specific amount of participation on the part of the author, whose contributions to the forum remain voluntary (and unpaid). We suggest you ask the author challenging questions lest his interest stray!
about the author TIM BORING is a software engineer with 20+ years of industry experience. For most of those years, he has been a user of orchestration systems, including Borg, Kubernetes, and Nomad.
about the cover illustration The figure on the cover of Build an Orchestrator in Go (From Scratch) is “Femme Baschkirienne,” or “Baschkirian woman,” taken from a collection by Jacques Grasset de Saint-Sauveur, published in 1788. The illustration is finely drawn and colored by hand. In those days, it was easy to identify where people lived and what their trade or station in life was just by their dress. Manning celebrates the inventiveness and initiative of the computer business with book covers based on the rich diversity of regional culture centuries ago, brought back to life by pictures from collections such as this one.
Part 1 Introduction The first part of this book lays the groundwork for your journey to writing an orchestration system—from scratch! In chapter 1, you will learn the core components that make up every orchestration system. From these core components, you will build a mental model for the Cube orchestrator, which we will implement together through the rest of the book.
Chapter 2 guides you through creating code skeletons from the mental model you learned in chapter 1.
In chapter 3, you will take the skeleton for the Task object and flesh it out in detail. This exercise will illustrate the process we’ll use to implement the rest of Cube’s codebase.