Complete Guide to Setting Up SSH Key Authentication on Linux Servers

Password-based SSH authentication is one of the biggest security risks for Linux servers. Every minute, automated bots scan the internet for SSH servers and attempt brute force password attacks. SSH key authentication eliminates this risk entirely by replacing passwords with cryptographic key pairs that are virtually impossible to brute force.

How SSH Key Authentication Works

SSH keys work in pairs:

• Private key — Stays on your local machine, never shared (like a physical key)
• Public key — Placed on the server you want to access (like a lock)

When you connect, the server challenges your client to prove it holds the private key matching the authorized public key. No password is ever transmitted over the network.

Step 1: Generate Your SSH Key Pair

On your local machine (Windows, macOS, or Linux), generate an Ed25519 key pair:

# Generate Ed25519 key (recommended - most secure and fastest)
ssh-keygen -t ed25519 -C "your-name@your-machine"

# If you need RSA compatibility (older systems)
ssh-keygen -t rsa -b 4096 -C "your-name@your-machine"

When prompted:

• File location — Press Enter for default (~/.ssh/id_ed25519)
• Passphrase — Set a strong passphrase (adds another layer of security)

# Verify your keys were created
ls -la ~/.ssh/
# You should see:
# id_ed25519       (private key - NEVER share this)
# id_ed25519.pub   (public key - safe to share)

Step 2: Copy Your Public Key to the Server

Method A: ssh-copy-id (Easiest)

# Copy key to remote server
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server-ip

# If using non-standard SSH port
ssh-copy-id -i ~/.ssh/id_ed25519.pub -p 2222 user@server-ip

Method B: Manual Copy

# Display your public key
cat ~/.ssh/id_ed25519.pub

# On the server, add it to authorized_keys
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "ssh-ed25519 AAAA... your-name@machine" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Step 3: Test Key Authentication

# Connect using your key
ssh user@server-ip

# Verbose output to verify key is used
ssh -v user@server-ip 2>&1 | grep "Offering public key"

Step 4: Disable Password Authentication

Important: Only do this AFTER confirming key authentication works!

# Edit SSH daemon configuration
sudo nano /etc/ssh/sshd_config

# Set these directives:
PasswordAuthentication no
PubkeyAuthentication yes
ChallengeResponseAuthentication no
UsePAM no
PermitRootLogin prohibit-password

# Restart SSH daemon
sudo systemctl restart sshd

Step 5: Additional SSH Hardening

# Recommended sshd_config settings
Port 2222                    # Change default port
MaxAuthTries 3               # Limit authentication attempts
LoginGraceTime 30            # 30 seconds to authenticate
AllowUsers admin deployer    # Whitelist specific users
ClientAliveInterval 300      # 5 minute keepalive
ClientAliveCountMax 2        # Disconnect after 2 missed keepalives
X11Forwarding no             # Disable X11 unless needed
AllowTcpForwarding no        # Disable port forwarding unless needed

Managing Multiple Keys

Create an SSH config file for managing multiple servers:

# ~/.ssh/config
Host webserver
    HostName 192.168.1.10
    User admin
    Port 2222
    IdentityFile ~/.ssh/id_ed25519

Host dbserver
    HostName 192.168.1.20
    User dbadmin
    Port 22
    IdentityFile ~/.ssh/id_ed25519_db

Host production-*
    User deployer
    IdentityFile ~/.ssh/id_ed25519_prod
    StrictHostKeyChecking yes

# Now connect with just:
ssh webserver
ssh dbserver

SSH Agent for Passphrase Management

# Start SSH agent
eval "$(ssh-agent -s)"

# Add your key (enter passphrase once)
ssh-add ~/.ssh/id_ed25519

# List loaded keys
ssh-add -l

# On macOS, add to Keychain
ssh-add --apple-use-keychain ~/.ssh/id_ed25519

Troubleshooting Common Issues

• Permission denied — Check that ~/.ssh is 700 and authorized_keys is 600
• Key not accepted — Verify the correct public key is in authorized_keys
• Agent not forwarding — Use ssh -A or add ForwardAgent yes to config
• Still asking for password — Check sshd_config and restart the SSH service

Further Reading

• SSH Mastery: Secure Remote Administration
• OpenSSH Configuration & Tunneling Guide
• Linux Security Hardening