Fundamentals of statistics for Data Scientists and Analysts with Python Code
2021-10-08
Common mistakes and best practices on creating a Dockerfile
2021-10-19
Show all

Guide to different types of Docker Volumes

The data generated and used by containers are not persisted after we restart or remove containers. So, we can use Docker volumes and bind mounts to manage data in Docker containers to solve this issue. We can use it to persist data in a container or share data between containers. From this post, you will learn how to use Docker volumes and bind mounts in your project.

What Is a Volume?

The Docker File System

A docker container runs the software stack defined in an image. Images are made of a set of read-only layers that work on a file system called the Union File System. When we start a new container, Docker adds a read-write layer on the top of the image layers allowing the container to run as though on a standard Linux file system.

So, any file change inside the container creates a working copy in the read-write layer. However, when the container is stopped or deleted, that read-write layer is lost.

docker layers system

We can try this out by running a command that writes and then reads a file:

$ docker run bash:latest \ 
  bash -c "echo hello > file.txt && cat file.txt"

The result is:

hello

But if we run the same image with just the command to output the file’s contents:

$ docker run bash:latest bash -c "cat file.txt" 
cat: can't open 'file.txt': No such file or directory

The second run of the container runs on a clean file system, so the file is not found.

Setup

Docker uses the following types of volumes and bind mounts to persist data. For this setup, I’m using macOS.

  1. Anonymous volumes
  2. Named volumes
    1. Bind mounts

For this post, we will run a MySQL server and execute some commands. By default, MySQL will store its data files inside /var/lib/mysql directory in the container, and Docker volumes will help us to persist that data.

We have three docker-compose.yml files to demonstrate volumes and bind mounts. To start these files, you will need to use the following command.

docker compose up

Once our container is running, we can use the following commands to create a table inside our container for testing purposes.

# Access the container
docker exec -it mysql_db_1 bash# Connect to MySQL server
mysql -uroot -proot# Run MySQL commands
USE test_db;
SHOW TABLES;
CREATE TABLE users (
user_id int NOT NULL AUTO_INCREMENT,
name VARCHAR(20),
PRIMARY KEY (user_id)
);
SHOW TABLES;

1. Anonymous volumes

If we run the following docker-compose.yml file, an anonymous volume will be created. If we restart our container, the data will be visible, but not after we remove the container. Also, it’s not accessible by other containers. It is helpful if we want to persist data temporarily. These volumes are created inside /var/lib/docker/volume local host directory.

version: '3.8'
services:
db:
image: mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: test_db
ports:
- "3306:3306"
volumes:
- /var/lib/mysql

As we can see, we don’t have to specify the host directory. We just need to specify the directory inside the container.

If we remove the volume instruction from the docker-compose.yml file, the container will create an anonymous volume by default because it’s specified inside the MySQL Dockerfile. So, the MySQL image ensures that we can still access the data if we don’t provide any volume information.

VOLUME /var/lib/mysql

Now, we have an anonymous volume with a random identifier.

docker volume ls
DRIVER VOLUME NAME
local 4e679725b7179e63e8658bc157a1980f320948ab819f271fd5a44fe94c16bf23

Let’s inspect our Docker conatiner.

docker inspect mysql_db_1
.
.
."Mounts": [
{
"Type": "volume",
"Name": "4e679725b7179e63e8658bc157a1980f320948ab819f271fd5a44fe94c16bf23",
"Source": "/var/lib/docker/volumes/4e679725b7179e63e8658bc157a1980f320948ab819f271fd5a44fe94c16bf23/_data",
"Destination": "/var/lib/mysql",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
.
.
.

We can use the following command to remove the container and its associated anonymous volume.

docker rm -v mysql_db_1

If we don’t remove the anonymous volume and the container together, it becomes a dangling volume.

docker rm mysql_db_1

We can list and remove all the dangling volumes using the following commands.

docker volume ls -qf dangling=true
docker volume rm $(docker volume ls -qf dangling=true)

2. Named volumes

Named volumes can persist data after we restart or remove a container. Also, it’s accessible by other containers. These volumes are created inside /var/lib/docker/volume local host directory.

version: '3.8'
services:
db:
image: mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: test_db
ports:
- "3306:3306"
volumes:
- db_data:/var/lib/mysqlvolumes:
db_data:

Here, the first field is a unique name of the volume on a host machine. The second part is the path in the container.

Moreover, if we remove the container using the following command, we will still have the volume, unlike anonymous volumes.

docker rm -v mysql_db_1

3. Bind mounts

Bind mounts can persist data after we restart or remove a container. As we can see, named volumes and bind mounts are the same, except the named volumes can be found under a specific host directory, and bind mounts can be any host directory.

version: '3.8'
services:
db:
image: mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: test_db
ports:
- "3306:3306"
volumes:
- $PWD/data:/var/lib/mysql

Here, we are mounting a host folder. The first part is the path in the host machine. The second part is the path in the container.

Commands

Now, let’s list all the available commands for the volume instruction.

docker volume --help

Commands:
create Create a volume
inspect Display detailed information on one or more volumes
ls List volumes
prune Remove all unused local volumes
rm Remove one or more volumes

We can use these commands to manage anonymous volumes and named volumes.

# Creat a volume
docker volume create test-vol
# test-vol# Inspect a volume
docker inspect test-vol
# [
# {
# "CreatedAt": "2021-07-17T07:23:25Z",
# "Driver": "local",
# "Labels": {},
# "Mountpoint": "/var/lib/docker/volumes/test-vol/_data",
# "Name": "test-vol",
# "Options": {},
# "Scope": "local"
# }
# ]# List all volumes
docker volume create test-vol-2
docker volume ls
# DRIVER VOLUME NAME
# local test-vol
# local test-vol-2# Remove all volumes
docker volume prune
# WARNING! This will remove all local volumes not used by at least one container.
# Are you sure you want to continue? [y/N] y
# Deleted Volumes:
# test-vol
# test-vol-2# Remove volumes
docker volume create test-vol-3
docker volume rm test-vol-3
# test-vol-3docker volume create test-vol-4
docker volume create test-vol-5
docker volume rm test-vol-4 test-vol-5
# test-vol-4
# test-vol-5

I hope you have a clear understanding of Docker volumes and bind mounts. It will help you to persist data for your Docker projects.

References:

https://www.baeldung.com/ops/docker-volumes

https://towardsdatascience.com/the-complete-guide-to-docker-volumes-1a06051d2cce

Amir Masoud Sefidian
Amir Masoud Sefidian
Data Scientist, Machine Learning Engineer, Researcher, Software Developer

Comments are closed.