Useful Docker tips and tricks

This article lists some experiences from learning and working with the Docker platform.

Image by Kinsey on Unsplash

1. Resolving error: Cannot connect to the Docker daemon at unix:/var/run/docker.sock. Is the docker daemon running?

Due to the fact that an app I was working on had a large number of containers running, and also due to the fact that these Docker containers were starting on boot up, this was significantly slowing my boot times and my machine’s general performance. Since the machine’s RAM was already maxed out, the only thing I could do was use systemctl to prevent the auto-bootup of the said Docker containers. Which worked great, except when I wanted to boot them up manually, I got the error referenced in the above title:

Cannot connect to the Docker daemon at unix:/var/run/docker.sock. Is the docker daemon running?

The solution was to simply run the following command:

systemctl start docker

Once I did that, my Docker containers obeyed all other commands, which makes perfect sense, because the above command is the one that actually starts the docker daemon.

2. What does docker run do?

The docker run command starts a container:

• we give it an image name out of which to make a container instance
• we give it a process to run (aka “the main process”)

When we run a container we can name it, or if we choose not to, Docker itself will assign a random name to a container.

Once the main process exits, the container is finished. Even if we have other processes started in the said container, when the main process finishes, the container finishes.

Here’s an example of a docker run with the ubuntu image and the echo process:

docker run ubuntu echo 'Hello'

The above command will output Hello, than the main process will finish and so the container will finish.

3. What does docker run --rm do?

When we want to just run something in a container, then delete it when the process is finished, we run the docker run --rm command.

It’s a one-liner for the following workflow:

• docker run <container-name>
• *the container does its work and the main process finishes`
• docker rm <container-name>

The docker run --rm command is handy because it’s a shortcut for the above process.

4. What does docker ps do?

The docker ps is one of the basic commands in Docker. It checks for running containers. The ps command is short for “Process Status”. This command was borrowed in Docker from Linux operating system with the same meaning: “Process Status”.

5. Install Docker on Ubuntu 20.04

To install Docker on Ubuntu 20.04, you need to run the following commands:

sudo apt update
sudo apt upgrade
sudo apt install docker.io
sudo usermod -aG docker awv
docker --version
sudo docker run hello world

If everything works as expected, you’ll get a nice Hello World message in the bash, as well as some explanations of all the steps that Docker took to run the Hello World image on your machine.

Here’s a sample output:

sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0a04fdcd24d3: Pull complete 
Digest: sha256:a451b...d
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64)
 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Now we can run docker ps command to list running containers.

With docker ps -a we can list all the containers.

6. Restart multiple containers on the command line easily using the $() syntax

Here’s a quick use case. We have three docker containers running. We check their ids using:

docker ps -q

We get the following sample container ids:

abc123123123
def789789789
cde258258258

Now we’d like to restart all three. We could copy paste their id’s and type out a command like this:

docker restart abc123123123 def789789789 cde258258258

But we can do it better, using the $() syntax:

docker restart $(docker ps -q)

The end result is still the same, but the second approach is much better because it saves time and is more generalized (we’re skipping hard-coded values).

7. What’s a docker image?

A Docker image is a file which contains a “recipe” for building a container (a fully functional, self-contained application, which utilizes the host OS kernel).

When we run Docker image, it can become one or multiple instances of that container.

A docker image is:

• immutable (can never change)
• shareable (with other people, e.g on Docker hub)

As an analogy using JavaScript, an image is a constructor function, and the container is an object instance of that constructor function.

To list docker images, use:

docker images

We get the output similar to this:

REPOSITORY  TAG  IMAGE ID   CREATED  SIZE
...         ...  ...        ...      ...

The dots above represent actual image data. Here’s what each column means:

• REPOSITORY: where the image came from
• TAG: version number
• IMAGE ID: internal docker id for this specific image
• CREATED:
• SIZE:

To refer to a specific image, use the combination:

<repository>:<tag>

Or use the image id.

The second approach is better because images don’t have to be named (but they must have the image id).

8. How to run the bash app from a Docker image of containerized Ubuntu?

You can do it if the image is of an OS, like Ubuntu.

The docker run command uses a specific image to build a container instance from that image:

docker run -ti <image> bash

The -ti is an abbreviation for “terminal interactive”.

To exit this image’s bash, just type:

exit

9. Format the output of the docker ps command

This section is inspired by the Learning Docker course by Arthur Ulfeldt, available from Linkedin Learning.

To format the output of the docker ps command, we use the --format flag:

docker ps --format=<how-to-format>

For example, if we can replace the <how-to-format> section with a variable. We can, for example, name that variable like this: $FORMAT.

But how do we get the variable into bash?

We just write a simple shell script, like this:

export FORMAT="\nID\t{{ .ID }}\nIMAGE\t{{ .Image }}\nCOMMAND\t{{ .Command }}\nCREATED\t{{ .RunningFor }}\nSTATUS\t{{ .Status }}\nPORTS\t{{ .Ports }}\nNAMES\t{{ .Names }}\n"

# this shell will only work on bash

We can name the shell script, for example, reformat.sh.

Now we need to make this new script executable. Like this:

chmod +x reformat.sh

Finally, we also need to run this script, like this:

sh reformat.sh

Alternatively, we could have just ran the code from the reformat.sh script directly in the terminal, like this:

export FORMAT="\nID\t{{ .ID}} \nIMAGE\t{{ .Image}} \nCOMMAND\t{{ .Command}} \nCREATED\t{{ .RunningFor}} \nSTATUS\t{{ .Status}} \nPORTS\t{{ .Ports}} \nNAMES\t{{ .Names}} \n"

Now we have the $FORMAT script available on our command line:

$FORMAT

The above command returns:

\nID\t{{ .ID}} \nIMAGE\t{{ .Image}} \nCOMMAND\t{{ .Command}} \nCREATED\t{{ .RunningFor}} \nSTATUS\t{{ .Status}} \nPORTS\t{{ .Ports}} \nNAMES\t{{ .Names}} \n

Now we can go back to the first command in this section:

docker ps --format=$FORMAT

10. What happens in a container instance, stays in a container instance

If we spin up two container instances from the same image of, say an Ubuntu OS, and then add a new file to one of these Ubuntu containers, that file is going to only be present in that one Ubuntu OS instance. The file won’t be magically added to the image, nor will it be available in the other container instance.

11. The basic anatomy of docker commands

The way that we can write the docker commands in bash follows a simple pattern:

docker <option> <command> <arguments>

For a list of all commands we can use, just type docker:

docker

To see the detailed description of each command, type:

docker <command> --help

For example:

docker run --help

The above command will give detailed info on the Docker’s run command.

To get an overview of what’s happening with Docker on our machine, we can run the info command:

docker info

12. Get a list of the most recently stopped containers

The last exited container can be found with:

docker ps -l

The output from the above command returns the following columns:

• ID
• IMAGE
• COMMAND
• CREATED
• STATUS
• PORTS
• NAMES

The STATUS column gives us the reason for exit in round brankets, for example Exited (0), or Exited(127).

13. Make an image out of a container

We can do this using the docker commit command.

So, with docker run we make a new container from a docker image. Then we might make some changes inside the container instance. We can then commit those changes to a new image, using the docker commit command.

This way, the immutability of docker images is preserved.

To make an image from the most recently stopped container, we need to get that container’s id, like this:

docker ps -l

Then, we’ll copy the ID key’s value, for example: 1234567890ab.

Then we’ll run the commit command with the above ID provided:

docker commit 1234567890ab

The above command will produce a huge SHA256 hash, which is a unique ID of our new image. To make this more human-readable, we can use tags, like this:

docker tag <the-entire-copy-pasted-sha256-string> <tag-name>

For example:

docker tag 123abc234bcd... my-own-image

Docker will also give each of our containers a custom name, listed in the NAMES column, such as:

fervent_sammet
determined_ishizaka
romantic_mendel
amazing_zhukovsky
etc...

Because committing and tagging docker images is commonplace, there’s a handy shortcut command that automates this process:

docker commit <name> <tag>

For example, let’s say we got this back from running docker ps -l:

CONTAINER ID    12cd7701db78
IMAGE           my-own-image
COMMAND         "/hello"
CREATED         26 minutes ago
STATUS          Exited (0) 26 minutes ago
PORTS
NAMES           fervent_sammet

Now we can commit the above container as a new image, like this:

docker commit fervent_sammet my-second-image

Now if we inspected the available images with docker image, we’ll get the listing of the available images, with the first listed image being the most recently-added one; in our case, the my-second-image image.

14. Find a Docker image on the Docker hub

You don’t even need to visit the website; everything can be done from the terminal, like this:

docker search ubuntu

The above command will return a list of images we can download. There are plenty of images there. We’ll just go with the first result, the ubuntu image.

To actually download an image, we can do:

docker pull <image-name>

Thus:

docker pull ubuntu

Here’s the output:

Using default tag: latest
latest: Pulling from library/ubuntu
83ee3a23efb7: Pull complete
db98fc6f11f0: Pull complete
f611acd52c6c: Pull complete
Digest: sha256:123abc123abc...cba321
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest

15. What is a detached Docker container?

Detached mode is when a Docker container runs in the background of our terminal. It does not receive input or display output.

For example, we can run our newly downloaded ubuntu image in detached mode, as follows.

First, we’ll locate it using the images command:

docker images

Then, we can use the ID to run it, for example, our ubuntu image id is: a6318a119b2a, and thus we’ll run:

docker run -d a6318a119b2a

Alternatively, we could use the human-friendly NAME parameter:

docker run -d ubuntu

Either way, we’ll get back that huge SHA string so that we can identify the running container.

Now we can see the most recent running container with docker ps -l:

docker ps -l

As expected, we get back all the data about that most recently ran container, which, indeed, is the one we’ve just ran with the -d flag, including the container name.

Now we can attach the currently detached container:

docker attach <container-name>

This way, I’ve gotten into the container that was previously detached.

Additionally, we can convert an attached container into a detached one, using a sequence of two keyboard shortcuts:

1. CTRL + p
2. CTRL + q

This way, we can exit the container *but the container is still running, detached**. This means that later on, we can attach one more time.

16. Set the maximum memory a container can use

To prevent from too much resources being used, we can run the --memory flag, with options, for example, like this:

docker run --memory <max-allowed-memory> <image-name> <command>

A great feature is to assign a portion of the cpu-shares, like this:

docker run --cpu-shares

…the above command is a value that takes into account all the running containers, then assigns the appropriate amount based on the cpu shares. This is more flexible, because if we have two containers and one doesn’t take up any resources, the other will then be able to take more resources because that’s how we specified it should work.

Otherwise, we can set the --cpu-quota flag to, say 20%, no more, no less, and this will always be the hard-coded value that we’ll be using.

I like to think of it as seats on a bus. In the above scenarios, the former is like saying “spread to as many empty seats available”, and the latter is like saying, “whatever happens, only use your seat”.

17. How to run a process in a Docker container, then stop the running container

Here’s an example of running bash in our ubuntu container.

docker run -d -ti ubuntu bash

The -ti is an abbreviation for terminal interactive.

We can then run the docker ps command to get the info on the running container(s), then pick the container that was built using the ubuntu image, and copy that container’s id. Once we have the id in the clipboard, we can run the following command to stop the currently running container:

docker stop a1b2c3d4e5a6b7

With the a1b2c3d4e5a6b7 SHA stub being the previously copied ID of the then-running container.

18. How to attach to a detached Docker container?

To see all the containers running (including detached ones), we run:

docker ps

Then, to attach a specific container, we run:

docker attach <container-name>

19. How to add another process to a running container in Docker?

We use this command:

docker exec

It’s useful for debugging and DB administration.

20. Giving a name to a running container, on-the-fly

If we want to specify a name to a container, at the time we run it, we can use the following command:

docker run --name <our-custom-name> -d <container-image>

For example, we can run our ubuntu image in a container instance we named testname:

docker run --name testname -d ubuntu

21. Inspecting Docker logs to find data about errors in a container

Sometimes we can start a container but it doesn’t do what we expect. We can use the docker logs command to inspect the logs of that container by passing the container NAME, like this:

docker logs <container-name>

An example of passing a custom name and a wrong command (so that we can inspect the logs):

docker run --name testname -d ubuntu bash -c "ech osomething"

In the above command, we’re running the ubuntu image:

• with the custom name of testname
• detached (specified with -d) so that it just runs in the background

Additionally, we’ve passed it the process to run, bash, and we’re running this command: ech osomething. The command was “announced” using the -c flag.

Obviously, the above command is wrong: we typed ech osomething but we should have typed echo something. The above command won’t run, because the ech command, contrary to the echo command, doesn’t exist.

Let’s inspect what happend to make sure:

docker logs testname

Indeed, the ech command was the problem; it’s confirmed by the output:

bash: ech: command not found

Here’s an important note: we need to keep our docker logs slim. If there’s too much stuff being output to the logs, we can slow down docker, even so much that it’s simply unusable.

22. Stop a container with the kill command

Using the kill command moves the container to a stopped state.

Any stopped container can be removed with the rm command.

Thus:

docker kill <container-name>
docker rm <container-name>

For example:

docker run ubuntu

We get the container’s NAME, in our case it was lucid_newton, so:

docker kill lucid_newton

That stopped our container.

Now, if we inspect the most recent container that was stopped, using:

docker ps -l

… we’d get back that indeed, it was lucid_newton, plus the STATUS is: Exited (137) 26 seconds ago.

Now we can remove it:

docker rm lucid_newton

This will just output the affected container’s name:

lucid_newton

That’s it, the container is removed, and the luci_newton name is now again freely available. Otherwise, if we wanted to use the same name sometimes in the future, we’d get a notification saying “that name is already in use”>

23. Make sure your container includes your dependencies (not downloads them at run time)

For example, if we’re using containerized Node.js, and we start running it, and it’s set up so that it downloads its dependecies on start, this can work fine, until…

For example, a library gets removed from the Node.js repositories, and then our container is broken, and can’t run.

24. The importance of naming Docker containers

We can have containers without NAME value specified.

We can also keep some important stuff in such containers.

Don’t do it!

Because, during container clean-up, it’s easier to erase a container that is “unlabelled”, and thus accidentally erase some work that you should’ve kept.

25. Using container ports

Container ports have to do with container networking.

Containers, by default, are set up initially so that they can’t access the internet.

We can group multiple containers into separate local networks, and we can specify exactly how they get connected to each other.

To connect to the internet, we can also explicitly expose a port (aka “publish” a port).

We can expose a port by setting up:

• the internal port that a containerized software is listening on
• the “outside” port that a container is listening on
• the protocol to use (optional)

Here’s an example of exposing the exact same port on the inside and the outside of a container:

-p 34567:34567

We can expose more than one port, by simply specifying another one, like this:

-p 34567:34567 -p 34566:34566 -p 34565:34565

Above, we’ve just specified three inner ports matching three outter ports.

Here’s a command sending data between three different bash instances on the same machine. The first one is the “server”:

docker run --rm -ti -p 34567:34567 -p 34568:L34568 --name the-server ubuntu bash

If we ran docker run --help, we’d find in the output instructions that:

• the --rm flag automatically removes the container when it exits,
• the -ti:
  • -t : Allocate a pseudo-tty
  • -i : Keep STDIN open even if not attached
• the -p, as just described:
  • ` -p, –publish list : Publish a container’s port(s) to the host`
• the --name:
  • --name string : Assign a name to the container

Now we can run in the same bash instance, the netcat program, with the nc command:

nc -lp 34567 | nc -lp 34568

So we’re listening with one nc instance on port 34567, and we’re piping that to another nc instance on port 34568.

In the second bash instance, we’ll run nc localhost 34567, and in the third, we’ll run nc localhost 34568, so that, when we type a “message” in the second bash instance, we’ll get that “message” in the third bash instance. The “message” is just some text that we type and that gets passed through by the netcat utility.

To reiterate, we’ll have 3 separate bash instances open. The first one is a server, so we type:

docker run --rm -ti -p 34567:34567 -p 34568:34568 --name the-server ubuntu

Then the the-server container’s bash will open:

nc -lp 34567 | nc -lp 34568

But, the nc is not installed on the system, so:

bash: nc: command not found
bash: nc: command not found

This is an opportunity to containerize an app (or a utility in our case).

To do that with our netcat, we’ll just run a docker container that has it installed.

26. How to expose ports dynamically?

The port inside the container is fixed.

The port on the host is chosen from the unused ports.

This allows many containers running programs with fixed ports.

This is often used with a service discovery program.

For example, I can specify only the port inside the container, and let docker choose the port on the outside:

docker run --rm -ti -p 34567 -p 34568 --name the-server ubuntu:14.04 bash

Now we’re inside the the-server container, and we can run nc:

nc -lp 34567 | nc -lp 34568

Now, in another bash instance, we can check for the the-server container’s external port:

docker port the-server

Here’s a sample output:

34567/tcp -> 0.0.0.0:32777
34568/tcp -> 0.0.0.0:32776

Now in the other two bash instances, I can run the exernal ports that Docker picked dynamically:

nc localhost 32777

…and in the other instance:

nc localhost 32776

27. It’s enough to run the docker rm command with the first 4 characters of the id

It’s as simple as, for example, this:

docker rm 76f9

This is great for a quick cleanup.

28. Containers shouldn’t directly refer to a container by an IP address

So how can containers “address” the container that is hosting them?

By using the container’s host name: host.docker.internal.

29. In Docker, can we use ports with udp protocol instead of the tcp protocol?

Yes.

We specify them with a slash and the port on the end of the command:

docker run -p <outside-port>:<inside-port>/<protocol-tcp-or-udp)>

For example:

docker run -p 23456:23456/udp

Practically, in one bash instance, we’ll run this:

docker run --rm -ti 34567/udp --name the-server ubuntu:14.04 bash

And then in another container we check the port:

docker port the-server # returns, e.g: 34567/udp -> 0.0.0.0:32774

Back in the first one, once inside the container, we’ll run:

nc -ulp 34567

Back in the second:

Hello from second

The message from the second is now passed in to first:

nc -ulp 34567
Hello from second

30. Networking between containers and inspecting Docker networks defaults

There are a number of ways and options of how our Docker containers connect to one another.

For example, once we exposed a port of a container, this opens a network path from the outside of that computer to that container.

Other containers are able to connect to it by going out to the host and then back in along the said network path.

To inspect Docker networks, run:

docker network ls

This is a sample output:

NETWORK ID          NAME                DRIVER              SCOPE
abdcabc5fd59        bridge              bridge              local
1dca4d941c2f        host                host                local
f450fbf5a6c4        none                null                local

Bridge is the network used by containers that don’t specify a preference to put into any other network.

Host is when you want a container to have no isolation at all. This does have some security concerns.

And none is for when a container should have no networking.

31. Build a new Docker network

Here’s the command: docker network create <network-name>, e.g:

docker network create my-network

To run a machine on our Docker network, we’ll run:

docker run --rm -ti --net my-network

We can also give our machine a name, by appending the --name <name-we-choose> to the above command, like this:

docker run --rm -ti --net my-network --name my-server

Names are very useful when using private networks in Docker because different containers inside the network can refer to each other by those names, so it makes it very easy for them to find each other.

Now that we have the name, we can run the rest, namely ubuntu:14.04 bash, so that the entire command now looks like this:

docker run --rm -ti --net my-network --name my-server ubuntu:14.04 bash

Now we can verify that we can send traffic to ourselves:

ping my-server

Indeed, we can, which is proven by the output we get:

PING my-server (172.18.0.2) 56(84) bytes of data.
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=1 ttl=64 time=0.064 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=2 ttl=64 time=0.039 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=3 ttl=64 time=0.064 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=4 ttl=64 time=0.063 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=5 ttl=64 time=0.062 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=6 ttl=64 time=0.063 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=7 ttl=64 time=0.064 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=8 ttl=64 time=0.063 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=9 ttl=64 time=0.036 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=10 ttl=64 time=0.064 ms
^C
--- my-server ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9224ms
rtt min/avg/max/mdev = 0.036/0.058/0.064/0.011 ms

If we try to ping a non-existing server, nothing will happen. So let’ open another bash instance and build another custom server:

docker run --rm -ti --net my-network --name my-second-server ubuntu:14.04 bash

I’ve put my-second-server on the same network so that my-server and my-second-server can find each other.

Now in my second server, I can write:

nc -lp 1234

And now I can send a message to my-server:

hello from my-second-server

Back in my-server, I can received the messages from my-second-server, and I can send a message to my-second-server:

my-second-server 1234
hi from my-server

Our two containers communicate both ways, and are on the same network.

32. Communicating between Docker networks

I’m going to start up a new terminal here.

Let’s make another network.

docker network create catsonly

This is a network only for cats.

Okay, now let’s go ahead and put the cat machine onto the cat network.

docker network connect catsonly catserver

Okay, now, over here, let’s make things a little more interesting. I’ll start up another terminal, and I’m going to start the new member of our network.

docker run --rm -ti --net catsonly --name bobcatserver bash

It’s like a catserver, only moreso, running bash.

Ok, so from here, I can do pink catserver, and get data. If I do ping dogeserver, traffic is not allowed.

From the catserver in the middle, I can ping dogserver, hit CTRL + c, and it works.

And bobcatserver.

Now over here, on the dogserver, we can still only make connections to catserver, but we cannot connect to babcatserver, because the dogserver is not on the catsonly network. There are many, many more options to this that give us fine-grain control over how things connect here. But what was covered here is about 90% of the user cases. The rest will be left up to experience.

33. What are Dockerfiles?

It’s just a file used to build a Docker image.

We run it with docker build and the -t tag for “tag”. The below command builds a new Docker image and tags it with the <name-tag>, in the current folder.

docker build -t <name-tag> .