Backing Up Docker Volumes and Upload to S3

Docker volumes are an essential part of the containerized architecture, often used to persist data across container lifecycles. To safeguard against data loss, it’s vital to back up these volumes regularly. This article provides a shell script to automate the process of daily backups of Docker volumes, uploading them to AWS S3, and cleaning up old backups.

Prerequisites

1. Docker installed and running.
2. AWS Command Line Interface (CLI) installed and configured with appropriate permissions.
3. The `jq` tool installed to process JSON content (often used to parse Docker command outputs).

Script Overview

The script will do the following:

1. Loop over each Docker volume.
2. Create a backup of the volume using docker cp.
3. Compress the backup.
4. Upload the compressed backup to an S3 bucket.
5. Remove local backups older than 30 days.
6. Remove S3 backups older than 30 days.

The Script

#!/bin/bash

# Set variables
BACKUP_DIR="/path/to/backup/dir" # Local backup directory
S3_BUCKET="s3://your-bucket-name" # Your S3 bucket name
DAYS_TO_KEEP=30

# 1. Loop over each Docker volume
for VOLUME in $(docker volume ls -q); do
    TIMESTAMP=$(date +%Y%m%d%H%M%S)
    BACKUP_NAME="${VOLUME}_${TIMESTAMP}.tar.gz"

    # 2. Create a backup of the volume
    CONTAINER_ID=$(docker run -d -v ${VOLUME}:/volume busybox true)
    docker cp ${CONTAINER_ID}:/volume ${BACKUP_DIR}/${VOLUME}
    docker rm -v ${CONTAINER_ID}

    # 3. Compress the backup
    tar -czf ${BACKUP_DIR}/${BACKUP_NAME} -C ${BACKUP_DIR} ${VOLUME}
    rm -rf ${BACKUP_DIR}/${VOLUME}

    # 4. Upload the compressed backup to an S3 bucket
    aws s3 cp ${BACKUP_DIR}/${BACKUP_NAME} ${S3_BUCKET}/${BACKUP_NAME}

done

# 5. Remove local backups older than 30 days
find ${BACKUP_DIR} -name "*.tar.gz" -mtime +${DAYS_TO_KEEP} -exec rm {} \;

# 6. Remove S3 backups older than 30 days
OLDER_THAN_DATE=$(date -d "-${DAYS_TO_KEEP} days" +%Y%m%d)
aws s3 ls ${S3_BUCKET}/ | awk '{print $4}' | while read BACKUP; do
    BACKUP_DATE=$(echo ${BACKUP} | awk -F_ '{print $2}' | awk -F. '{print $1}')
    if [[ ${BACKUP_DATE} -lt ${OLDER_THAN_DATE} ]]; then
        aws s3 rm ${S3_BUCKET}/${BACKUP}
    fi
done

Execution

Save the script to a file, for example `docker_vol_backup.sh`.

1. Make the script executable:

   chmod +x docker_vol_backup.sh 
   

2. Schedule the script to run daily using cron:

   
   0 2 * * * /path/to/docker_vol_backup.sh >> /path/to/logfile.log 2>&1
   
   

   Copy

This cron configuration will run the script every day at 2 AM and log the output to a specified logfile.

Conclusion

Automating the backup process of Docker volumes ensures data safety and minimizes human intervention. By leveraging AWS S3, data can be stored in a scalable, secure, and accessible environment. Periodically cleaning up old backups both locally and in S3 helps manage storage costs and prevent unnecessary clutter. Always ensure you test backup and restore procedures in a safe environment before relying on them for production data.