Hanging some flesh on the task skeleton
This chapter covers Reviewing how to start and stop Docker containers via the command line Introducing the Docker API calls for starting and stopping containers Implementing the Task concept to start and stop a container
Think about cooking your favorite meal. Let’s say you like making homemade pizza. To end up pulling a delicious, hot pizza out of your oven, you have to perform a number of tasks. If you like onions, green peppers, or any other veggies on your pizza, you have to cut them up. You must knead the dough and spread it on a baking sheet. Next, you spread tomato sauce across the dough and sprinkle cheese over it. Finally, on top of the cheese, you layer your veggies and other ingredients. A task in an orchestration system is similar to one of the individual steps in making a pizza. Like most companies these days, yours most likely has a website. That company’s website runs on a web server, perhaps the ubiquitous Apache web server. That’s a task. The website may use a database, like MySQL or PostgreSQL, to store dynamic content. That’s a task.
In our pizza-making analogy, the pizza wasn’t made in a vacuum. It was created in a specific context, which is a kitchen. The kitchen provides the necessary resources to make the pizza: a refrigerator where the cheese is stored, cabinets where the pizza sauce is kept, an oven in which to cook the pizza, and a knife to cut the pizza into slices. Similarly, a task operates in a specific context. In our case, that context will be a Docker container. Like the kitchen, the container will provide the resources necessary for the task to run: it will provide CPU cycles, memory, and networking according to the needs of the task. As a reminder, the task is the foundation of an orchestration system. Figure 1.1 shows a modified version of our mental model from chapter 1.
Figure 3.1 The main purpose of an orchestration system is to accept tasks from users and run them on the system’s worker nodes. Here, we see a user submitting a task to the Manager node, which then selects Worker2 to run the task. The dotted lines to Worker1 and Worker3 represent that these nodes were considered but ultimately not selected to run the task.
In the rest of this chapter, we’ll flesh out the Task skeleton we wrote in the previous chapter. But first, let’s quickly review some Docker basics.
3.1 Docker: Starting, stopping, and inspecting containers from the command line If you are a developer, you have probably used Docker containers to run your application and its backend database on your laptop while working on your code. If you are a DevOps engineer, you may have deployed Docker containers to your company’s production environment. Containers allow the developer to package their code, along with all its dependencies, and then ship the container to production. If a DevOps team is responsible for deployments to production, then they only have to worry about deploying the container. They don’t have to worry about whether the machine where the container will run has the correct version of the PostgreSQL library that the application uses to connect to its database. TIP If you need a more detailed review of Docker containers and how to control them, check out chapter 2 of Docker in Action (http://mng.bz/PRq8).
To run a Docker container, we can use the docker run command, an example of which can be seen in the next listing. Here, the docker run command is starting up a PostgreSQL database in a container, which might be used as a backend datastore while developing a new application.
Listing 3.1 Running the Postgres database server as a Docker container
This command runs the container in the foreground, meaning we can see its log output (-it), gives the container the name of postgres, and sets the POSTGRES_USER and POSTGRES_PASSWORD environment variables. Once a container is running, it performs the same functions it would if you were running it as a regular process on your laptop or desktop. In the case of the Postgres database from listing 3.1, I can now log into the database server using the psql command-line client and create a table like that in the following listing.
Listing 3.2 Logging in to the Postgres server and creating a table
Because we specified -p 5432:5432 in the docker run command in the previous listing, we can tell the psql client to connect to that port on the local machine.
Once a container is up and running, we can get information about it using the docker inspect command. The output from this command is extensive, so I will only list the State info.
Listing 3.3 Using the docker inspect command
Finally, we can stop a Docker container using the docker stop cube-book command. There isn’t any output from the command, but if we run the docker inspect cube-book command now, we’ll see that the state has changed from running to exited.
Listing 3.4 Running docker inspect cube-book after docker stop cube-book
3.2 Docker: Starting, stopping, and inspecting containers from the API In our orchestration system, the worker will be responsible for starting, stopping, and providing information about the tasks it’s running. To perform these functions, the worker will use Docker’s API. The API is accessible via the HTTP protocol using a client like curl or the HTTP library of a programming language. The following listing shows an example of using curl to get the same information we got from the docker inspect command previously.
Listing 3.5 Querying the Docker API with the curl HTTP client
Notice we’re passing the –unix-socket flag to the curl command. By default, Docker listens on a unix socket, but it can be configured to listen on a tcp socket. The URL, http://docker/containers/6970e8469684/json, contains the ID of the container to inspect, which I got from the docker ps command on my machine. Finally, the output from curl is piped to the jq command, which prints the output in a more readable format than curl’s. We could use Go’s HTTP library in our orchestration system, but that would force us to deal with many low-level details like HTTP methods, status codes, and serializing requests and deserializing responses. Instead, we’re going to use Docker’s SDK, which handles all the low-level HTTP details for us and allows us to focus on our primary task: creating, running, and stopping containers. The SDK provides the following six methods that will meet our needs: NewClientWithOpts—A helper method that instantiates an instance of the client and returns it to the caller ImagePull—Pulls the image down to the local machine where it will be run ContainerCreate—Creates a new container with a given configuration ContainerStart—Sends a request to Docker Engine to start the newly created container ContainerStop—Sends a request to Docker Engine to stop a running container ContainerRemove—Removes the container from the host NOTE Docker’s Golang SDK has extensive documentation (https://pkg.go .dev/github.com/docker/docker) that’s worth reading. In particular, the docs about the Go client (https://pkg.go.dev/github.com/docker/docker/ client) are relevant to our work throughout the rest of this book. The docker command-line examples we reviewed in the previous section use the Go SDK under the hood. Later in this chapter, we’ll implement a Run() method that uses the ImagePull, ContainerCreate, and Container- Start methods to create and start a container. Figure 3.2 provides a graphic representation of our custom code and the docker command using the SDK. By using the Go SDK for controlling the Docker containers in our orchestration system, we don’t have to reinvent the wheel. We can simply reuse the same code used by the docker command every day.
Figure 3.2 Regardless of the starting point, all paths to creating and running a container go through the Docker SDK.
Running container
3.3 Task configuration To run our tasks as containers, they need a configuration. What is a configuration? Think back to our pizza analogy from the beginning of the chapter. One of the tasks in making our pizza was cutting the onions (if you don’t like onions, insert your veggie of choice). To perform that task, we would use a knife and a cutting board, and we would cut the onions in a particular way. Perhaps we cut them into thin, even slices or dice them into small cubes. This is all part of the “configuration” of the task of cutting onions. (Okay, I’m probably stretching the pizza analogy a bit far, but I think you get the point.) For a task in our orchestration system, we’ll describe its configuration using the Config struct in listing 3.6. This struct encapsulates all the necessary bits of information about a task’s configuration. The comments should make the intent of each field obvious, but there are a couple of fields worth highlighting. The Name field will be used to identify a task in our orchestration system, and it will perform double duty as the name of the running container. Throughout the rest of the book, we’ll use this field to name our containers like test-container-1. The Image field, as you probably guessed, holds the name of the image the container will run. Remember, an image can be thought of as a package: it contains the collection of files and instructions necessary to run a program. This field can be set to a value as simple as postgres, or it can be set to a more specific value that includes a version, like postgres:13. The Memory and Disk fields will serve two purposes. The scheduler will use them to find a node in the cluster capable of running a task. They will also be used to tell the Docker daemon the number of resources a task requires. The Env field allows a user to specify environment variables that will get passed into the container. In our command to run a Postgres container, we set two environment variables: -e POSTGRES_USER=cube to specify the database user and -e POSTGRES_ PASSWORD=secret to specify that user’s password. Finally, the RestartPolicy field tells the Docker daemon what to do if a container dies unexpectedly. This field is one of the mechanisms that provides resilience in our orchestration system. As you can see from the comment, the acceptable values are an empty string, always, unless-stopped, or on-failure. Setting this field to always will, as its name implies, restart a container if it stops. Setting it to unless-stopped will restart a container unless it has been stopped (e.g., by docker stop). Setting it to on-failure will restart the container if it exits due to an error (i.e., a nonzero exit code). There are a few details that are spelled out in the documentation (http://mng.bz/1JdQ). We’re going to add the Config struct in the next listing to the task.go file from chapter 2.
Listing 3.6 The Config struct that will hold the configuration for orchestration tasks
3.4 Starting and stopping tasks Now that we’ve talked about a task’s configuration, let’s move on to starting and stopping a task. Remember, the worker in our orchestration system will be responsible for running tasks for us. That responsibility will mostly involve starting and stopping tasks. Let’s start by adding the code for the Docker struct you see in listing 3.7 to the task.go file. This struct will encapsulate everything we need to run our task as a Docker container. The Client field will hold a Docker client object that we’ll use to interact with the Docker API. The Config field will hold the task’s configuration. And once a task is running, it will also contain the ContainerId. This ID will allow us to interact with the running task.
Listing 3.7 The Docker struct
For the sake of convenience, let’s create a struct called DockerResult. We can use this struct as the return value in methods that start and stop containers, providing a wrapper around common information that is useful for callers. The struct contains an Error field to hold any error messages. It has an Action field that can be used to identify the action being taken, for example, start or stop. It has a ContainerId field to identify the container to which the result pertains. And, finally, there is a Result field that can hold arbitrary text that provides more information about the result of the operation.
Listing 3.8 The DockerResult struct
Now we’re ready for the exciting part: actually writing the code to create and run a task as a container. To do this, let’s start by adding a method to the Docker struct we created earlier. Let’s call that method Run.
The first part of our Run method will pull the Docker image our task will use from a container registry such as Docker Hub. A container registry is simply a repository of images and allows for the easy distribution of the images it hosts. To pull the image, the Run method first creates a context, which is a type that holds values that can be passed across boundaries such as APIs and processes. It’s common to use a context to pass along deadlines or cancellation signals in requests to an API. In our case, we’ll use an empty context returned from the Background function. Next, Run calls the ImagePull method on the Docker client object, passing the context object, the image name, and any options necessary to pull the image. The ImagePull method returns two values: an object that fulfills the io.ReadCloser interface and an error object. It stores these values in the reader and err variables. The next step in the method checks the error value returned from ImagePull. If the value is not nil, the method prints the error message and returns as a DockerResult. Finally, the method copies the value of the reader variable to os.Stdout via the io.Copy function. io.Copy is a function from the io package in Golang’s standard library, and it simply copies data to a destination (os.Stdout) from a source (reader). Because we’ll be working from the command line whenever we’re running the components of our orchestration system, it’s useful to write the reader variable to Stdout as a way to communicate what happened in the ImagePull method.
Listing 3.9 The start of our Run() method
Similar to running a container from the command line, the method begins by pulling the container’s image.
Once the Run method has pulled the image and checked for errors (and found none, we hope), the next bit of business on the agenda is to prepare the configuration to be sent to Docker. Before we do that, however, let’s take a look at the signature of the ContainerCreate method from the Docker client. This is the method we’ll use to create the container. As you can see in listing 3.10, ContainerCreate takes several arguments. Similar to the ImagePull method used earlier, it takes a context.Context as its first argument. The next argument is the actual container configuration, which is a pointer to a container.Config type. We’ll copy the values from our own Config type into this one. The third argument is a pointer to a container.HostConfig type. This type will hold the configuration a task requires of the host on which the container will run, for example, a Linux machine. The fourth argument is also a pointer and points to a network.NetworkingConfig type. This type can be used to specify networking details, such as the network ID container, any links to other containers that are needed, and IP addresses. For our purposes, we won’t use the network configuration, instead allowing Docker to handle those details for us. The fifth argument is another pointer, and it points to a specs.Platform type. This type can be used to specify details about the platform on which the image runs. It allows you to specify things like the CPU architecture and the operating system. We won’t be making use of this argument either. The sixth and final argument to ContainerCreate is the container name, passed as a string.
Listing 3.10 The Docker client’s ContainerCreate method
Now we know what information we need to pass along in the ContainerCreate method, so let’s gather it from our Config type and massage it into the appropriate types that ContainerCreate will accept. What we’ll end up with is what you see in listing 3.11.
First, we’ll create a variable called rp. This variable will hold a container.Restart- Policy type, and it will contain the RestartPolicy we defined earlier in our Config struct in listing 3.6. Following the rp variable, let’s declare a variable called r. This variable will hold the resources required by the container in a container.Resources type. The most common resources we’ll use for our orchestration system will be memory. Next, let’s create a variable called cc to hold our container configuration. This variable will be of the type container.Config, and into it, we’ll copy two values from our Config type. The first is the Image our container will use. The second is any environment variables, which go into the Env field. Finally, we take the rp and r variables we defined and add them to a third variable called hc. This variable is a container.HostConfig type. In addition to specifying the RestartPolicy and Resources in the hc variable, we’ll also set the PublishAllPorts field to true. What does this field do? Remember our example docker run command in listing 3.2, where we start up a PostgreSQL container? In that command, we used -p 5432:5432 to tell Docker that we wanted to map port 5432 on the host running our container to port 5432 inside the container. Well, that’s not the best way to expose a container’s ports on a host. There is an easier way. Instead, we can set PublishAllPorts to true, and Docker will expose those ports automatically by randomly choosing available ports on the host.
The following listing creates four variables to hold configuration information, which gets passed to the ContainerCreate method.
Listing 3.11 The next phase of running a container
We’ve done all the necessary prep work, and now we can create the container and start it. We’ve already touched on the ContainerCreate method in listing 3.10, so all that’s left to do is to call it like in listing 3.12. One thing to notice, however, is that we pass nil values as the fourth and fifth arguments, which, as you’ll recall from listing 3.10, are the networking and platform arguments. We won’t be making use of these features in our orchestration system, so we can ignore them for now. As with the ImagePull method earlier, ContainerCreate returns two values: a response, which is a pointer to a container.ContainerCreateCreatedBody type, and an error type. The ContainerCreateCreatedBody type gets stored in the resp variable, and we put the error in the err variable. Next, we check the err variable for any errors and, if we find any, print them and return them in a DockerResult type. Great! We’ve got all our ingredients together, and we’ve formed them into a container. All that’s left to do is start it. To perform this final step, we call the Container- Start method. Besides a context argument, ContainerStart takes the ID of an existing container, which we get from the resp variable returned from ContainerCreate, and any options necessary to start the container. In our case, we don’t need any options, so we simply pass an empty types.ContainerStartOptions. ContainerStart only returns one type, an error, so we check it in the same way we have with the other method calls we’ve made. If there is an error, we print it and then return it in a DockerResult.
Listing 3.12 The penultimate phase
At this point, if all was successful, we have a container running the task. All that’s left to do now is to take care of some bookkeeping, which you can see in listing 3.13. We start by adding the container ID to the configuration object (which will ultimately be stored, but let’s not get ahead of ourselves!). Similar to printing the results of the ImagePull operation to stdout, we do the same with the result of starting the container. This is accomplished by calling the ContainerLogs method and then writing the return value to stdout using the stdcopy.StdCopy(os.Stdout, os.Stderr, out) call.
Listing 3.13 The final phase of creating and running a container
As a reminder, the Run method we’ve written in listings 3.9, 3.11, 3.12, and 3.13 perform the same operations as the docker run command. When you type docker run on the command line, under the hood, the docker binary is using the same SDK methods we’re using in our code to create and run the container. Now that we can create a container and start it, let’s write the code to stop a container. Compared to our Run method, the Stop method will be much simpler, as you can see in listing 3.14. Because there isn’t the necessary prep work to do for stopping a container, the process simply involves calling the ContainerStop method with the ContainerID and then calling the ContainerRemove method with the ContainerID and the requisite options. Again, in both operations, the code checks the value of the err returned from the method. As with the Run method, our Stop method performs the same operations carried out by the docker stop and docker rm commands.
Listing 3.14 Stopping a container
Now, let’s update our main.go program that we created in chapter 2 to create and stop a container. First, add the createContainer function in listing 3.15 to the bottom of the main.go file. Inside it, we’ll set up the configuration for the task and store it in a variable called c, and then we’ll create a new Docker client and store it in dc. Next, let’s create the d object, which is of type task.Docker. From this object, we call the Run() method to create the container.
Listing 3.15 The createContainer function
Second, add the stopContainer function below createContainer. This function accepts a single argument, d, which is the same d object created in createContainer in listing 3.15. All that’s left to do is call d.Stop().
Listing 3.16 The stopContainer function
Finally, we call the createContainer and stopContainer functions we created from our main() function in main.go. To do that, add the code from listing 3.17 to the bottom of your main function.
As you can see, the code is fairly simple. It starts by printing a useful message that it’s going to create a container; it then calls the createContainer() function and stores the results in two variables, dockerTask and createResult. Then it checks for errors by comparing the value of createResult.Error to nil. If it finds an error, it prints it and exits by calling os.Exit(1). To stop the container, the main function simply calls stopContainer and passes it the dockerTask object returned by the earlier call to createContainer.
Listing 3.17 Calling the createContainer and stopContainer functions
Time for another moment of truth. Let’s run the code!
Listing 3.18 Running the code to create and stop a container
At this point, our container has been created and is running. You can check it with docker ps and docker logs test-container-1.
Finally, we stop the container and remove it. Again, you can verify the operation with docker ps to see that it’s not running and docker ps -a to see that it has been removed.
Here we can see our code pulling the image for the container.
At this point, we have the foundation of our orchestration system in place. We can create, run, stop, and remove containers, which provide the technical implementation of our Task concept. The other components in our system—namely, the Worker and Manager—will use this Task implementation to perform their necessary roles.
Summary The task concept, and its technical implementation, is the fundamental unit of our orchestration system. All the other components—worker, manager, and scheduler—exist for the purpose of starting, stopping, and inspecting tasks. The Docker API provides the ability to manipulate containers programmatically. The three most important methods are ContainerCreate, Container- Start, and ContainerStop. These methods allow a developer to perform the same operations from their code that they can do from the command line (i.e., docker run, docker start, and docker stop). A container has a configuration. The configuration can be broken down into the following categories: identification (i.e., how to identify containers), resource allocation, networking, and error handling. A task is the smallest unit of work performed by our orchestration system and can be thought of as similar to running a program on your laptop or desktop. We use Docker in this book because it abstracts away many of the concerns of the underlying operating system. We could implement our orchestration system to run tasks as regular operating system processes. Doing so, however, means our system would need to be intimately familiar with the details of how processes run across OSs (e.g., Linux, Mac, Windows). An orchestration system consists of multiple machines, called a cluster.