Automating Backups with rsync and cron

Table of Contents

1. [Introduction](#introduction) 2. [Understanding rsync](#understanding-rsync) 3. [Understanding cron](#understanding-cron) 4. [Basic rsync Usage](#basic-rsync-usage) 5. [Advanced rsync Options](#advanced-rsync-options) 6. [Setting Up Automated Backups](#setting-up-automated-backups) 7. [Backup Strategies](#backup-strategies) 8. [Remote Backups](#remote-backups) 9. [Monitoring and Logging](#monitoring-and-logging) 10. [Troubleshooting](#troubleshooting) 11. [Best Practices](#best-practices)

Introduction

Automated backups are essential for data protection and disaster recovery. The combination of rsync and cron provides a powerful, flexible, and reliable solution for creating automated backup systems on Unix-like operating systems. This guide covers everything from basic concepts to advanced implementation strategies.

rsync is a fast, versatile file copying tool that can work locally or with remote systems. It uses a delta-transfer algorithm that sends only the differences between source and destination files, making it extremely efficient for incremental backups.

cron is a time-based job scheduler that allows you to run commands or scripts at specified intervals automatically. Together, these tools create a robust backup automation system.

Understanding rsync

What is rsync

rsync stands for "remote sync" and is a utility for efficiently transferring and synchronizing files between directories, disks, and networked computers. It was written by Andrew Tridgell and Paul Mackerras and first released in 1996.

Key Features of rsync

| Feature | Description | |---------|-------------| | Delta Transfer | Only transfers changed portions of files | | Preservation | Maintains file permissions, ownership, timestamps | | Compression | Can compress data during transfer | | Remote Sync | Works over SSH, RSH, or direct socket connections | | Incremental | Supports incremental backups efficiently | | Verification | Built-in data verification mechanisms | | Bandwidth Control | Can limit bandwidth usage | | Dry Run | Test mode to preview operations |

rsync Algorithm

The rsync algorithm works by:

1. Dividing files into fixed-size blocks 2. Computing checksums for each block 3. Comparing checksums between source and destination 4. Transferring only blocks that differ 5. Reconstructing files at the destination

This approach makes rsync extremely efficient for large files with small changes.

Understanding cron

What is cron

cron is a daemon that runs continuously in the background and executes scheduled tasks at specified times and dates. It reads configuration files called "crontabs" that contain lists of commands and their schedules.

Cron Components

| Component | Description | |-----------|-------------| | crond | The cron daemon that runs scheduled tasks | | crontab | Command to edit cron schedules | | /etc/crontab | System-wide cron configuration | | /var/spool/cron | User-specific cron files | | /etc/cron.d/ | Additional system cron files | | /etc/cron.hourly/ | Scripts run hourly | | /etc/cron.daily/ | Scripts run daily | | /etc/cron.weekly/ | Scripts run weekly | | /etc/cron.monthly/ | Scripts run monthly |

Crontab Syntax

The crontab format consists of six fields:

` * command │ │ │ │ │ │ │ │ │ └─── Day of week (0-7, Sunday = 0 or 7) │ │ │ └───── Month (1-12) │ │ └─────── Day of month (1-31) │ └───────── Hour (0-23) └─────────── Minute (0-59) `

Crontab Special Characters

| Character | Meaning | Example | |-----------|---------|---------| | | Any value | (every minute) | | , | List separator | 1,15 (1st and 15th minute) | | - | Range | 1-5 (minutes 1 through 5) | | / | Step values | /15 * (every 15 minutes) | | @reboot | At startup | @reboot /path/script | | @yearly | Once per year | @yearly /path/script | | @monthly | Once per month | @monthly /path/script | | @weekly | Once per week | @weekly /path/script | | @daily | Once per day | @daily /path/script | | @hourly | Once per hour | @hourly /path/script |

Basic rsync Usage

Installation

Most Linux distributions include rsync by default. If not installed:

`bash

Ubuntu/Debian

CentOS/RHEL/Fedora

or

macOS (using Homebrew)

Basic Syntax

`bash rsync [OPTIONS] SOURCE DESTINATION `

Essential rsync Options

| Option | Description | Example | |--------|-------------|---------| | -a | Archive mode (recursive, preserve attributes) | rsync -a /source/ /dest/ | | -v | Verbose output | rsync -av /source/ /dest/ | | -r | Recursive | rsync -r /source/ /dest/ | | -u | Update (skip newer files at destination) | rsync -au /source/ /dest/ | | -z | Compress during transfer | rsync -avz /source/ /dest/ | | -h | Human-readable output | rsync -avh /source/ /dest/ | | -P | Show progress and keep partial files | rsync -avP /source/ /dest/ | | --delete | Delete files in destination not in source | rsync -av --delete /source/ /dest/ | | --dry-run | Preview without making changes | rsync -av --dry-run /source/ /dest/ | | --exclude | Exclude files/directories | rsync -av --exclude '*.log' /source/ /dest/ |

Basic Examples

#### Local Directory Sync

`bash

Sync contents of /home/user/documents to /backup/documents

Note: Trailing slash on source matters

/source/ means "contents of source"

/source means "source directory itself"

#### Sync with Progress Display

`bash rsync -avP /home/user/documents/ /backup/documents/ `

#### Dry Run to Preview Changes

`bash rsync -av --dry-run /home/user/documents/ /backup/documents/ `

#### Exclude Specific Files

`bash

Exclude log files and temporary directories

Advanced rsync Options

File Selection and Filtering

| Option | Description | Example | |--------|-------------|---------| | --include | Include files matching pattern | --include '*.txt' | | --exclude | Exclude files matching pattern | --exclude '*.tmp' | | --exclude-from | Read exclude patterns from file | --exclude-from=excludes.txt | | --include-from | Read include patterns from file | --include-from=includes.txt | | --files-from | Transfer only files listed in file | --files-from=filelist.txt | | --max-size | Don't transfer files larger than SIZE | --max-size=100M | | --min-size | Don't transfer files smaller than SIZE | --min-size=1K |

Performance and Network Options

| Option | Description | Example | |--------|-------------|---------| | --bwlimit | Limit bandwidth usage (KB/s) | --bwlimit=1000 | | --compress-level | Set compression level (0-9) | --compress-level=6 | | --partial | Keep partially transferred files | --partial | | --partial-dir | Store partial files in directory | --partial-dir=/tmp/rsync-partial | | --timeout | Set I/O timeout in seconds | --timeout=300 | | --contimeout | Set connection timeout | --contimeout=60 |

Advanced Examples

#### Complex Exclude Pattern

`bash rsync -av \ --exclude '.git/' \ --exclude '*.log' \ --exclude '*.tmp' \ --exclude 'node_modules/' \ --exclude '__pycache__/' \ /home/user/projects/ /backup/projects/ `

#### Using Exclude File

Create an exclude file rsync-excludes.txt:

` *.log *.tmp .git/ node_modules/ __pycache__/ .DS_Store Thumbs.db `

Then use it:

`bash rsync -av --exclude-from=rsync-excludes.txt /home/user/projects/ /backup/projects/ `

#### Bandwidth Limited Transfer

`bash rsync -avz --bwlimit=500 --progress /large/dataset/ user@remote:/backup/dataset/ `

Setting Up Automated Backups

Creating Backup Scripts

#### Basic Local Backup Script

Create a file /usr/local/bin/backup-local.sh:

`bash #!/bin/bash

Local backup script using rsync

Author: System Administrator

Date: $(date +%Y-%m-%d)

Configuration

Create log entry

Perform backup

Check exit status

Make it executable:

`bash sudo chmod +x /usr/local/bin/backup-local.sh `

#### Advanced Backup Script with Rotation

Create /usr/local/bin/backup-incremental.sh:

`bash #!/bin/bash

Incremental backup script with rotation

Creates timestamped backups and maintains specified number of copies

Configuration

Function to log messages

Function to send notification (optional)

Start backup

Create backup directory

Perform incremental backup using hard links

Check backup status

Configuring Cron Jobs

#### View Current Cron Jobs

`bash

View current user's cron jobs

View specific user's cron jobs (as root)

View system cron jobs

#### Edit Cron Jobs

`bash

Edit current user's cron jobs

Edit specific user's cron jobs (as root)

#### Example Cron Schedules

| Schedule | Cron Expression | Description | |----------|----------------|-------------| | Every hour | 0 | At minute 0 of every hour | | Daily at 2 AM | 0 2 * | Every day at 2:00 AM | | Weekly on Sunday | 0 2 0 | Every Sunday at 2:00 AM | | Monthly on 1st | 0 2 1 | 1st day of every month at 2:00 AM | | Every 15 minutes | /15 * | Every 15 minutes | | Workdays at 6 PM | 0 18 1-5 | Monday through Friday at 6:00 PM | | Twice daily | 0 6,18 * | Every day at 6:00 AM and 6:00 PM |

#### Setting Up Automated Backups

Add to crontab (crontab -e):

`bash

Daily backup at 2 AM

Incremental backup every 6 hours

Weekly full backup on Sunday at 1 AM

Hourly sync of critical data

Backup Strategies

Types of Backups

| Backup Type | Description | Advantages | Disadvantages | |-------------|-------------|------------|---------------| | Full Backup | Complete copy of all data | Simple to restore, complete | Time-consuming, storage-intensive | | Incremental | Only changes since last backup | Fast, efficient storage | Complex restore process | | Differential | Changes since last full backup | Faster restore than incremental | Larger than incremental | | Mirror | Exact replica of source | Real-time sync possible | No versioning |

Implementing Different Strategies

#### Full Backup Strategy

`bash #!/bin/bash

full-backup.sh

SOURCE="/home/user/data" BACKUP_DIR="/backup/full/$(date +%Y-%m-%d)"

mkdir -p "$BACKUP_DIR" rsync -av "$SOURCE/" "$BACKUP_DIR/" `

#### Incremental Backup with Hard Links

`bash #!/bin/bash

incremental-backup.sh

SOURCE="/home/user/data" BACKUP_BASE="/backup/incremental" DATE=$(date +%Y-%m-%d_%H-%M) CURRENT="$BACKUP_BASE/$DATE" LATEST="$BACKUP_BASE/latest"

mkdir -p "$CURRENT"

if [ -d "$LATEST" ]; then rsync -av --link-dest="$LATEST" "$SOURCE/" "$CURRENT/" else rsync -av "$SOURCE/" "$CURRENT/" fi

rm -f "$LATEST" ln -s "$CURRENT" "$LATEST" `

#### Grandfather-Father-Son Rotation

`bash #!/bin/bash

gfs-backup.sh

SOURCE="/home/user/data" BACKUP_BASE="/backup/gfs" DOW=$(date +%u) # Day of week (1-7) DOM=$(date +%d) # Day of month (01-31) MOY=$(date +%m) # Month of year (01-12)

Determine backup type

mkdir -p "$BACKUP_DIR" rsync -av --delete "$SOURCE/" "$BACKUP_DIR/" `

Remote Backups

SSH Key Setup

For automated remote backups, set up SSH key authentication:

`bash

Generate SSH key pair

Copy public key to remote server

Test connection

Remote Backup Examples

#### Basic Remote Backup

`bash

Backup to remote server

Backup from remote server

#### Remote Backup Script

`bash #!/bin/bash

remote-backup.sh

Configuration

Function to log messages

Test remote connection

Create remote directory

Perform backup

if [ $? -eq 0 ]; then log_message "Backup completed successfully" else log_message "Backup failed with errors" exit 1 fi `

Using rsync Daemon

#### Server Configuration

Create /etc/rsyncd.conf:

`

Global settings

Backup module

Create secrets file /etc/rsyncd.secrets:

` backupuser:secretpassword `

Set permissions:

`bash chmod 600 /etc/rsyncd.secrets `

Start rsync daemon:

`bash rsync --daemon `

#### Client Usage

`bash

Set password in environment variable

Backup to rsync daemon

Monitoring and Logging

Comprehensive Logging

#### Enhanced Logging Script

`bash #!/bin/bash

backup-with-logging.sh

Configuration

Ensure log directory exists

Function to log with timestamp

Function to calculate directory size

Start backup process

Perform backup with detailed statistics

RSYNC_EXIT_CODE=$? END_TIME=$(date +%s) DURATION=$((END_TIME - START_TIME))

Log results

log "Duration: ${DURATION} seconds" log "Destination size: $(get_size "$BACKUP_DIR")" log "=== BACKUP FINISHED ==="

Extract and log statistics

Send email notification if configured

exit $RSYNC_EXIT_CODE `

Log Rotation

Create /etc/logrotate.d/backup:

` /var/log/backup/*.log { monthly rotate 12 compress delaycompress missingok notifempty create 644 root root } `

Monitoring Script

`bash #!/bin/bash

backup-monitor.sh

LOG_FILE="/var/log/backup/backup-$(date +%Y-%m).log" ALERT_EMAIL="admin@example.com" MAX_AGE_HOURS=25 # Alert if last backup is older than 25 hours

Check if log file exists

Get last backup timestamp

if [ -z "$LAST_BACKUP" ]; then echo "ERROR: No backup entries found in log" exit 1 fi

Convert to epoch time

Check backup age

Troubleshooting

Common rsync Error Codes

| Exit Code | Description | Common Causes | |-----------|-------------|---------------| | 0 | Success | No errors | | 1 | Syntax or usage error | Invalid command line options | | 2 | Protocol incompatibility | Version mismatch | | 3 | Errors selecting input/output files | Permission issues, missing files | | 4 | Requested action not supported | Unsupported feature | | 5 | Error starting client-server protocol | Connection issues | | 10 | Error in socket I/O | Network problems | | 11 | Error in file I/O | Disk issues, permission problems | | 12 | Error in rsync protocol data stream | Corruption, interruption | | 20 | Received SIGUSR1 or SIGINT | Interrupted by signal | | 23 | Partial transfer due to error | Some files failed to transfer | | 24 | Partial transfer due to vanished source files | Files deleted during transfer | | 25 | The --max-delete limit stopped deletions | Too many deletions prevented |

Debugging rsync Issues

#### Verbose Output

`bash

Increase verbosity

Show progress

Dry run to test

#### Network Debugging

`bash

Test SSH connection

Use specific SSH options

Check bandwidth

Common Issues and Solutions

#### Permission Denied Errors

`bash

Check file permissions

Run with appropriate user

Preserve permissions

#### Disk Space Issues

`bash

Check available space

Monitor space during transfer

Use partial transfer

#### Network Timeouts

`bash

Increase timeout values

Use compression for slow links

Cron Troubleshooting

#### Common Cron Issues

| Issue | Cause | Solution | |-------|-------|----------| | Script doesn't run | Incorrect path | Use full paths in scripts | | Permission denied | Wrong ownership/permissions | Check script permissions | | Command not found | Limited PATH in cron | Set PATH in script or use full paths | | No output/logging | Output not redirected | Redirect output to log files | | Environment differences | Different environment in cron | Set required environment variables |

#### Debugging Cron Jobs

`bash

Check cron service status

Check cron logs

Test script manually

Run with same environment as cron

Best Practices

Security Best Practices

#### File Permissions

`bash

Secure backup scripts

Secure SSH keys

Secure log files

#### SSH Configuration

Create ~/.ssh/config:

` Host backup-server HostName backup-server.example.com User backup IdentityFile ~/.ssh/backup_key IdentitiesOnly yes ServerAliveInterval 60 ServerAliveCountMax 3 ConnectTimeout 10 `

Performance Optimization

#### Rsync Performance Tips

| Optimization | Command Example | Benefit | |--------------|----------------|---------| | Use compression | rsync -avz | Reduces network usage | | Limit bandwidth | rsync -av --bwlimit=1000 | Prevents network saturation | | Use partial transfers | rsync -av --partial | Resume interrupted transfers | | Exclude unnecessary files | rsync -av --exclude '*.tmp' | Reduces transfer time | | Use appropriate block size | rsync -av --block-size=8192 | Optimizes for network/disk |

#### System Optimization

`bash

Increase file descriptor limits

Optimize network buffers for large transfers

Backup Verification

#### Checksum Verification Script

`bash #!/bin/bash

verify-backup.sh

SOURCE_DIR="/home/user/documents" BACKUP_DIR="/backup/documents" TEMP_DIR="/tmp/backup-verify" LOG_FILE="/var/log/backup-verify.log"

mkdir -p "$TEMP_DIR"

Generate checksums for source

Generate checksums for backup

Compare checksums

Maintenance Tasks

#### Cleanup Old Backups

`bash #!/bin/bash

cleanup-backups.sh

BACKUP_DIR="/backup" KEEP_DAYS=30

Remove backups older than KEEP_DAYS

Log cleanup activity

#### Health Check Script

`bash #!/bin/bash

backup-health-check.sh

Check disk space

if [ $USAGE -gt 90 ]; then echo "WARNING: Backup partition is ${USAGE}% full" exit 1 fi

Check last backup age

Check backup size consistency

if [ -n "$TODAY_SIZE" ] && [ -n "$YESTERDAY_SIZE" ]; then DIFF_PERCENT=$(( (TODAY_SIZE - YESTERDAY_SIZE) * 100 / YESTERDAY_SIZE )) if [ $DIFF_PERCENT -gt 50 ] || [ $DIFF_PERCENT -lt -50 ]; then echo "WARNING: Backup size changed by ${DIFF_PERCENT}%" fi fi

echo "Backup health check passed" exit 0 `

This comprehensive guide covers the essential aspects of automating backups with rsync and cron, from basic concepts to advanced implementation strategies. The combination of these tools provides a robust, efficient, and reliable backup solution suitable for various environments and requirements.