Automating Complex Workflows with Shell Scripts Guide

Master shell scripting for workflow automation. Learn advanced techniques, best practices, and real-world examples to streamline your processes.

Author:

Published:

Category: System Administration

Reading Time: 18

Automating Complex Workflows with Shell Scripts

Table of Contents

1. [Introduction](#introduction) 2. [Shell Script Fundamentals](#shell-script-fundamentals) 3. [Advanced Scripting Techniques](#advanced-scripting-techniques) 4. [Workflow Automation Patterns](#workflow-automation-patterns) 5. [Real-World Examples](#real-world-examples) 6. [Best Practices](#best-practices) 7. [Error Handling and Debugging](#error-handling-and-debugging) 8. [Performance Optimization](#performance-optimization)

Introduction

Shell scripting is a powerful tool for automating complex workflows, system administration tasks, and repetitive processes. By combining shell commands, control structures, and system utilities, you can create sophisticated automation solutions that save time and reduce errors.

Benefits of Shell Script Automation

- Consistency: Eliminates human error in repetitive tasks - Efficiency: Processes multiple operations simultaneously - Scalability: Handles large-scale operations effortlessly - Integration: Connects different systems and tools - Scheduling: Works with cron for automated execution

Shell Script Fundamentals

Basic Script Structure

`bash #!/bin/bash

Script: workflow_automation.sh

Purpose: Demonstrate complex workflow automation

Author: Your Name

Date: $(date)

Set strict mode

set -euo pipefail

Global variables

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" LOG_FILE="${SCRIPT_DIR}/automation.log" CONFIG_FILE="${SCRIPT_DIR}/config.conf"

Main execution

main() { log "INFO" "Starting workflow automation" # Your workflow logic here log "INFO" "Workflow completed successfully" }

Execute main function

main "$@" `

Essential Shell Script Components

| Component | Purpose | Example | |-----------|---------|---------| | Shebang | Specifies interpreter | #!/bin/bash | | Variables | Store data | USER_HOME="/home/user" | | Functions | Reusable code blocks | function backup_files() { ... } | | Conditionals | Decision making | if [ -f "$file" ]; then ... fi | | Loops | Iteration | for file in *.txt; do ... done | | Arrays | Multiple values | files=("file1.txt" "file2.txt") |

Variable Types and Usage

`bash #!/bin/bash

String variables

PROJECT_NAME="MyProject" VERSION="1.0.0"

Numeric variables

MAX_RETRIES=3 TIMEOUT=30

Arrays

ENVIRONMENTS=("dev" "staging" "production") FILES_TO_BACKUP=("/etc/nginx" "/var/www" "/home/user/data")

Environment variables

export PATH="/usr/local/bin:$PATH" export LOG_LEVEL="INFO"

Command substitution

CURRENT_DATE=$(date +"%Y-%m-%d") DISK_USAGE=$(df -h / | awk 'NR==2 {print $5}')

Parameter expansion

CONFIG_DIR="${HOME}/.config" BACKUP_FILE="${PROJECT_NAME}_backup_${CURRENT_DATE}.tar.gz" `

Advanced Scripting Techniques

Function Library System

`bash #!/bin/bash

lib/common.sh - Common functions library

Logging function with levels

log() { local level="$1" local message="$2" local timestamp=$(date '+%Y-%m-%d %H:%M:%S') echo "[$timestamp] [$level] $message" | tee -a "$LOG_FILE" # Send critical errors to syslog if [[ "$level" == "ERROR" ]]; then logger -p user.err "$message" fi }

Configuration file parser

parse_config() { local config_file="$1" if [[ ! -f "$config_file" ]]; then log "ERROR" "Configuration file not found: $config_file" return 1 fi # Source configuration with validation while IFS='=' read -r key value; do # Skip comments and empty lines [[ $key =~ ^[[:space:]]*# ]] && continue [[ -z "$key" ]] && continue # Export configuration variables export "$key"="$value" log "DEBUG" "Config loaded: $key=$value" done < "$config_file" }

Retry mechanism with exponential backoff

retry_with_backoff() { local max_attempts="$1" local delay="$2" local command="$3" local attempt=1 while [[ $attempt -le $max_attempts ]]; do log "INFO" "Attempt $attempt/$max_attempts: $command" if eval "$command"; then log "INFO" "Command succeeded on attempt $attempt" return 0 fi if [[ $attempt -eq $max_attempts ]]; then log "ERROR" "Command failed after $max_attempts attempts" return 1 fi log "WARN" "Command failed, retrying in ${delay}s..." sleep "$delay" # Exponential backoff delay=$((delay * 2)) attempt=$((attempt + 1)) done }

Progress indicator

show_progress() { local current="$1" local total="$2" local width=50 local percentage=$((current * 100 / total)) local filled=$((width * current / total)) printf "\rProgress: [" printf "%*s" "$filled" | tr ' ' '=' printf "%*s" $((width - filled)) | tr ' ' '-' printf "] %d%% (%d/%d)" "$percentage" "$current" "$total" if [[ $current -eq $total ]]; then echo fi } `

Advanced Control Structures

`bash #!/bin/bash

Complex conditional logic

check_system_requirements() { local requirements_met=true # Check disk space local disk_usage=$(df / | awk 'NR==2 {print $5}' | sed 's/%//') if [[ $disk_usage -gt 90 ]]; then log "ERROR" "Disk usage too high: ${disk_usage}%" requirements_met=false fi # Check memory local mem_available=$(free -m | awk 'NR==2{printf "%.0f", $7*100/$2}') if [[ $mem_available -lt 10 ]]; then log "ERROR" "Available memory too low: ${mem_available}%" requirements_met=false fi # Check required commands local required_commands=("git" "docker" "curl" "jq") for cmd in "${required_commands[@]}"; do if ! command -v "$cmd" &> /dev/null; then log "ERROR" "Required command not found: $cmd" requirements_met=false fi done $requirements_met }

Advanced loop patterns

process_files_parallel() { local source_dir="$1" local max_jobs="$2" local job_count=0 for file in "$source_dir"/*.txt; do [[ ! -f "$file" ]] && continue # Process file in background { log "INFO" "Processing file: $file" process_single_file "$file" log "INFO" "Completed processing: $file" } & job_count=$((job_count + 1)) # Limit concurrent jobs if [[ $job_count -ge $max_jobs ]]; then wait job_count=0 fi done # Wait for remaining jobs wait }

Case statement for complex decision making

handle_deployment_environment() { local environment="$1" local action="$2" case "$environment" in "development"|"dev") case "$action" in "deploy") deploy_to_development ;; "rollback") rollback_development ;; *) log "ERROR" "Unknown action for development: $action" return 1 ;; esac ;; "staging"|"stage") # Staging requires approval if ! request_approval "staging deployment"; then log "ERROR" "Staging deployment not approved" return 1 fi deploy_to_staging ;; "production"|"prod") # Production requires multiple approvals and checks perform_production_checks deploy_to_production ;; *) log "ERROR" "Unknown environment: $environment" return 1 ;; esac } `

Workflow Automation Patterns

Database Backup and Maintenance Workflow

`bash #!/bin/bash

database_maintenance.sh - Automated database maintenance workflow

source "$(dirname "$0")/lib/common.sh"

Configuration

DB_HOST="${DB_HOST:-localhost}" DB_USER="${DB_USER:-admin}" DB_PASSWORD="${DB_PASSWORD:-}" BACKUP_DIR="${BACKUP_DIR:-/var/backups/mysql}" RETENTION_DAYS="${RETENTION_DAYS:-30}" DATABASES=("app_db" "user_db" "analytics_db")

Database maintenance workflow

database_maintenance_workflow() { log "INFO" "Starting database maintenance workflow" # Step 1: Pre-maintenance checks if ! perform_pre_checks; then log "ERROR" "Pre-maintenance checks failed" return 1 fi # Step 2: Create backups if ! create_database_backups; then log "ERROR" "Database backup failed" return 1 fi # Step 3: Optimize databases if ! optimize_databases; then log "ERROR" "Database optimization failed" return 1 fi # Step 4: Clean old backups cleanup_old_backups # Step 5: Generate maintenance report generate_maintenance_report log "INFO" "Database maintenance workflow completed" }

perform_pre_checks() { log "INFO" "Performing pre-maintenance checks" # Check database connectivity for db in "${DATABASES[@]}"; do if ! mysql -h "$DB_HOST" -u "$DB_USER" -p"$DB_PASSWORD" -e "USE $db;" 2>/dev/null; then log "ERROR" "Cannot connect to database: $db" return 1 fi done # Check disk space local backup_disk_usage=$(df "$BACKUP_DIR" | awk 'NR==2 {print $5}' | sed 's/%//') if [[ $backup_disk_usage -gt 80 ]]; then log "WARN" "Backup directory disk usage high: ${backup_disk_usage}%" fi # Ensure backup directory exists mkdir -p "$BACKUP_DIR" return 0 }

create_database_backups() { log "INFO" "Creating database backups" local backup_date=$(date +"%Y%m%d_%H%M%S") local total_dbs=${#DATABASES[@]} local current_db=0 for db in "${DATABASES[@]}"; do current_db=$((current_db + 1)) show_progress "$current_db" "$total_dbs" local backup_file="${BACKUP_DIR}/${db}_${backup_date}.sql.gz" log "INFO" "Backing up database: $db" if ! mysqldump -h "$DB_HOST" -u "$DB_USER" -p"$DB_PASSWORD" \ --single-transaction --routines --triggers "$db" | \ gzip > "$backup_file"; then log "ERROR" "Failed to backup database: $db" return 1 fi # Verify backup integrity if ! gzip -t "$backup_file"; then log "ERROR" "Backup file corrupted: $backup_file" return 1 fi log "INFO" "Backup completed: $backup_file" done return 0 }

optimize_databases() { log "INFO" "Optimizing databases" for db in "${DATABASES[@]}"; do log "INFO" "Optimizing database: $db" # Get list of tables local tables=$(mysql -h "$DB_HOST" -u "$DB_USER" -p"$DB_PASSWORD" \ -e "SHOW TABLES;" "$db" 2>/dev/null | tail -n +2) # Optimize each table while read -r table; do [[ -z "$table" ]] && continue log "INFO" "Optimizing table: ${db}.${table}" mysql -h "$DB_HOST" -u "$DB_USER" -p"$DB_PASSWORD" \ -e "OPTIMIZE TABLE $table;" "$db" 2>/dev/null done <<< "$tables" done return 0 }

cleanup_old_backups() { log "INFO" "Cleaning up old backups (older than $RETENTION_DAYS days)" local deleted_count=0 while IFS= read -r -d '' backup_file; do log "INFO" "Deleting old backup: $backup_file" rm "$backup_file" deleted_count=$((deleted_count + 1)) done < <(find "$BACKUP_DIR" -name "*.sql.gz" -mtime +$RETENTION_DAYS -print0) log "INFO" "Deleted $deleted_count old backup files" }

generate_maintenance_report() { local report_file="${BACKUP_DIR}/maintenance_report_$(date +%Y%m%d).txt" { echo "Database Maintenance Report" echo "==========================" echo "Date: $(date)" echo "Host: $DB_HOST" echo "" echo "Database Sizes:" for db in "${DATABASES[@]}"; do local size=$(mysql -h "$DB_HOST" -u "$DB_USER" -p"$DB_PASSWORD" \ -e "SELECT ROUND(SUM(data_length + index_length) / 1024 / 1024, 1) AS 'DB Size in MB' FROM information_schema.tables WHERE table_schema='$db';" 2>/dev/null | tail -n +2) echo " $db: ${size} MB" done echo "" echo "Backup Files:" ls -lh "$BACKUP_DIR"/*.sql.gz 2>/dev/null | tail -5 echo "" echo "Disk Usage:" df -h "$BACKUP_DIR" } > "$report_file" log "INFO" "Maintenance report generated: $report_file" }

Execute workflow

database_maintenance_workflow "$@" `

CI/CD Pipeline Automation

`bash #!/bin/bash

cicd_pipeline.sh - Automated CI/CD pipeline

source "$(dirname "$0")/lib/common.sh"

Pipeline configuration

PROJECT_NAME="${PROJECT_NAME:-myapp}" GIT_REPO="${GIT_REPO:-https://github.com/user/myapp.git}" BUILD_DIR="${BUILD_DIR:-/tmp/builds}" DOCKER_REGISTRY="${DOCKER_REGISTRY:-registry.example.com}" ENVIRONMENTS=("development" "staging" "production")

Pipeline stages

PIPELINE_STAGES=( "checkout_code" "run_tests" "build_application" "security_scan" "build_docker_image" "deploy_to_environment" )

Main pipeline execution

execute_pipeline() { local target_env="$1" local git_branch="${2:-main}" log "INFO" "Starting CI/CD pipeline for $PROJECT_NAME" log "INFO" "Target environment: $target_env" log "INFO" "Git branch: $git_branch" # Validate environment if ! validate_environment "$target_env"; then log "ERROR" "Invalid target environment: $target_env" return 1 fi # Execute pipeline stages local total_stages=${#PIPELINE_STAGES[@]} local current_stage=0 for stage in "${PIPELINE_STAGES[@]}"; do current_stage=$((current_stage + 1)) log "INFO" "Executing stage $current_stage/$total_stages: $stage" show_progress "$current_stage" "$total_stages" if ! execute_stage "$stage" "$target_env" "$git_branch"; then log "ERROR" "Pipeline failed at stage: $stage" cleanup_pipeline return 1 fi log "INFO" "Stage completed: $stage" done log "INFO" "Pipeline completed successfully" cleanup_pipeline }

execute_stage() { local stage="$1" local environment="$2" local branch="$3" case "$stage" in "checkout_code") checkout_code "$branch" ;; "run_tests") run_tests ;; "build_application") build_application ;; "security_scan") security_scan ;; "build_docker_image") build_docker_image ;; "deploy_to_environment") deploy_to_environment "$environment" ;; *) log "ERROR" "Unknown pipeline stage: $stage" return 1 ;; esac }

checkout_code() { local branch="$1" log "INFO" "Checking out code from branch: $branch" # Clean build directory rm -rf "$BUILD_DIR" mkdir -p "$BUILD_DIR" # Clone repository if ! git clone --branch "$branch" --depth 1 "$GIT_REPO" "$BUILD_DIR"; then log "ERROR" "Failed to clone repository" return 1 fi # Get commit information cd "$BUILD_DIR" local commit_hash=$(git rev-parse HEAD) local commit_message=$(git log -1 --pretty=%B) log "INFO" "Checked out commit: $commit_hash" log "INFO" "Commit message: $commit_message" # Export build information export BUILD_COMMIT="$commit_hash" export BUILD_BRANCH="$branch" export BUILD_TIMESTAMP="$(date -u +%Y%m%d%H%M%S)" }

run_tests() { log "INFO" "Running test suite" cd "$BUILD_DIR" # Unit tests if [[ -f "package.json" ]]; then log "INFO" "Running npm tests" if ! npm test; then log "ERROR" "Unit tests failed" return 1 fi fi # Integration tests if [[ -f "docker-compose.test.yml" ]]; then log "INFO" "Running integration tests" if ! docker-compose -f docker-compose.test.yml up --build --abort-on-container-exit; then log "ERROR" "Integration tests failed" docker-compose -f docker-compose.test.yml down return 1 fi docker-compose -f docker-compose.test.yml down fi log "INFO" "All tests passed" }

build_application() { log "INFO" "Building application" cd "$BUILD_DIR" # Build based on project type if [[ -f "package.json" ]]; then log "INFO" "Building Node.js application" npm ci npm run build elif [[ -f "pom.xml" ]]; then log "INFO" "Building Maven project" mvn clean package -DskipTests elif [[ -f "Makefile" ]]; then log "INFO" "Building with Make" make build else log "WARN" "No build configuration found, skipping build step" fi }

security_scan() { log "INFO" "Performing security scan" cd "$BUILD_DIR" # Dependency vulnerability scan if command -v npm &> /dev/null && [[ -f "package.json" ]]; then log "INFO" "Scanning npm dependencies" if ! npm audit --audit-level moderate; then log "ERROR" "Security vulnerabilities found in dependencies" return 1 fi fi # Static code analysis if command -v sonar-scanner &> /dev/null; then log "INFO" "Running SonarQube analysis" sonar-scanner fi log "INFO" "Security scan completed" }

build_docker_image() { log "INFO" "Building Docker image" cd "$BUILD_DIR" local image_tag="${DOCKER_REGISTRY}/${PROJECT_NAME}:${BUILD_TIMESTAMP}" local latest_tag="${DOCKER_REGISTRY}/${PROJECT_NAME}:latest" # Build image if ! docker build -t "$image_tag" -t "$latest_tag" .; then log "ERROR" "Docker build failed" return 1 fi # Push to registry log "INFO" "Pushing image to registry" docker push "$image_tag" docker push "$latest_tag" # Export image information export DOCKER_IMAGE="$image_tag" log "INFO" "Docker image built and pushed: $image_tag" }

deploy_to_environment() { local environment="$1" log "INFO" "Deploying to environment: $environment" case "$environment" in "development") deploy_development ;; "staging") deploy_staging ;; "production") deploy_production ;; *) log "ERROR" "Unknown environment: $environment" return 1 ;; esac }

deploy_development() { log "INFO" "Deploying to development environment" # Simple docker-compose deployment cd "$BUILD_DIR" # Update docker-compose with new image sed -i "s|image:.*|image: $DOCKER_IMAGE|g" docker-compose.yml # Deploy docker-compose down docker-compose up -d # Health check sleep 10 if ! curl -f http://localhost:3000/health; then log "ERROR" "Development deployment health check failed" return 1 fi log "INFO" "Development deployment successful" }

cleanup_pipeline() { log "INFO" "Cleaning up pipeline resources" # Remove build directory rm -rf "$BUILD_DIR" # Clean up Docker images docker image prune -f log "INFO" "Pipeline cleanup completed" }

Pipeline execution with command line arguments

main() { local environment="${1:-development}" local branch="${2:-main}" # Load configuration if [[ -f "pipeline.conf" ]]; then source "pipeline.conf" fi # Execute pipeline execute_pipeline "$environment" "$branch" }

main "$@" `

Error Handling and Debugging

Comprehensive Error Handling Framework

`bash #!/bin/bash

error_handling.sh - Advanced error handling framework

Error handling configuration

set -euo pipefail # Exit on error, undefined vars, pipe failures IFS=

Automating Complex Workflows with Shell Scripts Guide

\n\t' # Secure Internal Field Separator

Global error handling variables

declare -g ERROR_LOG="${ERROR_LOG:-/var/log/script_errors.log}" declare -g DEBUG_MODE="${DEBUG_MODE:-false}" declare -g NOTIFICATION_EMAIL="${NOTIFICATION_EMAIL:-admin@example.com}"

Error codes

declare -rA ERROR_CODES=( [SUCCESS]=0 [GENERAL_ERROR]=1 [INVALID_ARGUMENT]=2 [FILE_NOT_FOUND]=3 [PERMISSION_DENIED]=4 [NETWORK_ERROR]=5 [DATABASE_ERROR]=6 [TIMEOUT_ERROR]=7 )

Trap functions for error handling

trap 'handle_error $? $LINENO $BASH_LINENO "$BASH_COMMAND" $(printf "%s " "${FUNCNAME[@]}")' ERR trap 'handle_exit $?' EXIT trap 'handle_interrupt' INT TERM

handle_error() { local exit_code="$1" local line_number="$2" local bash_line_number="$3" local command="$4" local function_stack="$5" local error_message="Error occurred in script: $0" error_message+="\n Exit Code: $exit_code" error_message+="\n Line: $line_number" error_message+="\n Command: $command" error_message+="\n Function Stack: $function_stack" error_message+="\n Timestamp: $(date)" # Log error echo -e "$error_message" | tee -a "$ERROR_LOG" # Generate stack trace generate_stack_trace # Send notification for critical errors if [[ $exit_code -ne 0 ]]; then send_error_notification "$error_message" fi # Cleanup on error cleanup_on_error }

handle_exit() { local exit_code="$1" if [[ $exit_code -eq 0 ]]; then log "INFO" "Script completed successfully" else log "ERROR" "Script exited with code: $exit_code" fi }

handle_interrupt() { log "WARN" "Script interrupted by user" cleanup_on_error exit 130 }

generate_stack_trace() { log "DEBUG" "Generating stack trace" local frame=0 while caller $frame; do ((frame++)) done | while read line func file; do echo " at $func ($file:$line)" | tee -a "$ERROR_LOG" done }

Safe command execution with retry and timeout

safe_execute() { local command="$1" local max_retries="${2:-3}" local timeout="${3:-30}" local retry_delay="${4:-5}" local attempt=1 while [[ $attempt -le $max_retries ]]; do log "DEBUG" "Executing (attempt $attempt/$max_retries): $command" if timeout "$timeout" bash -c "$command"; then log "DEBUG" "Command executed successfully" return 0 fi local exit_code=$? if [[ $exit_code -eq 124 ]]; then log "WARN" "Command timed out after ${timeout}s" else log "WARN" "Command failed with exit code: $exit_code" fi if [[ $attempt -eq $max_retries ]]; then log "ERROR" "Command failed after $max_retries attempts" return $exit_code fi log "INFO" "Retrying in ${retry_delay}s..." sleep "$retry_delay" attempt=$((attempt + 1)) done }

Input validation framework

validate_input() { local input="$1" local validation_type="$2" local field_name="${3:-input}" case "$validation_type" in "email") if [[ ! $input =~ ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$ ]]; then log "ERROR" "Invalid email format: $field_name" return ${ERROR_CODES[INVALID_ARGUMENT]} fi ;; "url") if [[ ! $input =~ ^https?://[A-Za-z0-9.-]+\.[A-Za-z]{2,}(/.*)?$ ]]; then log "ERROR" "Invalid URL format: $field_name" return ${ERROR_CODES[INVALID_ARGUMENT]} fi ;; "file_path") if [[ ! -f "$input" ]]; then log "ERROR" "File not found: $field_name ($input)" return ${ERROR_CODES[FILE_NOT_FOUND]} fi ;; "directory_path") if [[ ! -d "$input" ]]; then log "ERROR" "Directory not found: $field_name ($input)" return ${ERROR_CODES[FILE_NOT_FOUND]} fi ;; "numeric") if [[ ! $input =~ ^[0-9]+$ ]]; then log "ERROR" "Invalid numeric value: $field_name" return ${ERROR_CODES[INVALID_ARGUMENT]} fi ;; "non_empty") if [[ -z "$input" ]]; then log "ERROR" "Empty value not allowed: $field_name" return ${ERROR_CODES[INVALID_ARGUMENT]} fi ;; *) log "ERROR" "Unknown validation type: $validation_type" return ${ERROR_CODES[GENERAL_ERROR]} ;; esac return ${ERROR_CODES[SUCCESS]} }

Resource monitoring and limits

monitor_resources() { local max_memory_mb="${1:-1024}" local max_disk_usage_percent="${2:-80}" # Check memory usage local memory_usage=$(free -m | awk 'NR==2{printf "%.0f", $3}') if [[ $memory_usage -gt $max_memory_mb ]]; then log "WARN" "High memory usage: ${memory_usage}MB (limit: ${max_memory_mb}MB)" fi # Check disk usage local disk_usage=$(df / | awk 'NR==2 {print $5}' | sed 's/%//') if [[ $disk_usage -gt $max_disk_usage_percent ]]; then log "WARN" "High disk usage: ${disk_usage}% (limit: ${max_disk_usage_percent}%)" fi # Check load average local load_avg=$(uptime | awk -F'load average:' '{print $2}' | cut -d, -f1 | xargs) local cpu_cores=$(nproc) if (( $(echo "$load_avg > $cpu_cores" | bc -l) )); then log "WARN" "High system load: $load_avg (cores: $cpu_cores)" fi }

Debug mode functions

debug_print() { if [[ "$DEBUG_MODE" == "true" ]]; then echo "[DEBUG] $*" >&2 fi }

debug_vars() { if [[ "$DEBUG_MODE" == "true" ]]; then echo "[DEBUG] Current variables:" >&2 set | grep -E '^[A-Z_]+=' >&2 fi } `

Best Practices

Script Organization and Standards

| Practice | Description | Example | |----------|-------------|---------| | Modular Design | Break scripts into reusable functions | source lib/common.sh | | Configuration Files | External configuration management | config.conf with key=value pairs | | Logging Standards | Consistent logging with levels | log "INFO" "Process started" | | Error Handling | Comprehensive error management | set -euo pipefail | | Documentation | Inline comments and usage help | # Purpose: Backup system files | | Version Control | Track script changes | Git with semantic versioning |

Security Best Practices

`bash #!/bin/bash

security_practices.sh - Security-focused scripting practices

Secure script permissions

umask 077 # Restrictive file permissions

Validate all inputs

validate_and_sanitize_input() { local input="$1" local max_length="${2:-255}" # Remove potentially dangerous characters input=$(echo "$input" | tr -d ';<>&|`$(){}[]') # Limit input length if [[ ${#input} -gt $max_length ]]; then log "ERROR" "Input too long (max: $max_length characters)" return 1 fi echo "$input" }

Secure temporary file handling

create_secure_tempfile() { local template="${1:-secure_temp.XXXXXX}" local temp_file # Create secure temporary file temp_file=$(mktemp "/tmp/$template") # Set restrictive permissions chmod 600 "$temp_file" # Register for cleanup TEMP_FILES+=("$temp_file") echo "$temp_file" }

Cleanup temporary files

cleanup_temp_files() { for temp_file in "${TEMP_FILES[@]}"; do if [[ -f "$temp_file" ]]; then shred -u "$temp_file" 2>/dev/null || rm -f "$temp_file" fi done }

Secure password handling

read_password_securely() { local prompt="$1" local password echo -n "$prompt: " >&2 read -s password echo >&2 # Validate password strength if [[ ${#password} -lt 8 ]]; then log "ERROR" "Password too short (minimum 8 characters)" return 1 fi echo "$password" }

Network security checks

validate_remote_host() { local hostname="$1" # DNS lookup validation if ! nslookup "$hostname" >/dev/null 2>&1; then log "ERROR" "Cannot resolve hostname: $hostname" return 1 fi # SSL certificate validation for HTTPS if [[ $hostname =~ ^https:// ]]; then if ! curl -s --connect-timeout 10 "$hostname" >/dev/null; then log "ERROR" "SSL certificate validation failed: $hostname" return 1 fi fi return 0 } `

Performance Optimization Techniques

`bash #!/bin/bash

performance_optimization.sh - Performance optimization techniques

Parallel processing with job control

process_files_efficiently() { local source_dir="$1" local max_parallel="${2:-$(nproc)}" # Use GNU parallel if available if command -v parallel &> /dev/null; then find "$source_dir" -type f -name "*.txt" | \ parallel -j "$max_parallel" process_single_file {} else # Manual parallel processing local job_count=0 for file in "$source_dir"/*.txt; do [[ ! -f "$file" ]] && continue process_single_file "$file" & job_count=$((job_count + 1)) # Control job count if [[ $job_count -ge $max_parallel ]]; then wait job_count=0 fi done wait # Wait for remaining jobs fi }

Memory-efficient file processing

process_large_file() { local file_path="$1" local chunk_size="${2:-1000}" # Process file in chunks to avoid memory issues local line_count=0 local chunk_array=() while IFS= read -r line; do chunk_array+=("$line") line_count=$((line_count + 1)) # Process chunk when size limit reached if [[ $line_count -eq $chunk_size ]]; then process_chunk "${chunk_array[@]}" chunk_array=() line_count=0 fi done < "$file_path" # Process remaining lines if [[ ${#chunk_array[@]} -gt 0 ]]; then process_chunk "${chunk_array[@]}" fi }

Efficient string operations

optimize_string_operations() { local input_string="$1" # Use parameter expansion instead of external commands # Remove prefix local without_prefix="${input_string#prefix_}" # Remove suffix local without_suffix="${input_string%_suffix}" # Replace substring local replaced="${input_string/old/new}" # Convert to lowercase (Bash 4+) local lowercase="${input_string,,}" # Get string length local length="${#input_string}" echo "Optimized string operations completed" }

Caching mechanism

declare -A CACHE

cached_operation() { local key="$1" local operation="$2" # Check if result is cached if [[ -n "${CACHE[$key]:-}" ]]; then echo "${CACHE[$key]}" return 0 fi # Execute operation and cache result local result result=$(eval "$operation") CACHE[$key]="$result" echo "$result" }

Resource usage monitoring

monitor_script_performance() { local start_time=$(date +%s) local start_memory=$(free -m | awk 'NR==2{print $3}') # Your script operations here "$@" local end_time=$(date +%s) local end_memory=$(free -m | awk 'NR==2{print $3}') local duration=$((end_time - start_time)) local memory_diff=$((end_memory - start_memory)) log "INFO" "Performance metrics:" log "INFO" " Duration: ${duration}s" log "INFO" " Memory change: ${memory_diff}MB" } `

Summary

Shell script automation provides powerful capabilities for managing complex workflows. Key takeaways include:

1. Structure and Organization: Use modular design with libraries and configuration files 2. Error Handling: Implement comprehensive error handling and logging 3. Security: Validate inputs and handle sensitive data securely 4. Performance: Optimize for parallel processing and resource efficiency 5. Monitoring: Include progress indicators and performance metrics 6. Maintenance: Design for debugging, testing, and long-term maintenance

By following these patterns and practices, you can create robust, maintainable automation scripts that handle complex workflows reliably and efficiently.

Tags

  • Automation
  • DevOps
  • bash
  • shell-scripting
  • system-admin

Related Articles

Popular Technical Articles & Tutorials

Explore our comprehensive collection of technical articles, programming tutorials, and IT guides written by industry experts:

Browse all 8+ technical articles | Read our IT blog

Automating Complex Workflows with Shell Scripts Guide