Skip to main content

​
SSH Authentication Debugging

​
When to Use This

You have an SSH key installed on a server, but connections fail or hang. This skill provides a systematic debugging methodology.

​
Step 1: Client-Side Verbose Output

Start with triple-verbose to see the full handshake: 
timeout 15 ssh -vvv -o ConnectTimeout=10 -o BatchMode=yes \
  root@target-host echo "OK" 2>&1 | tail -30

​
What to Look For

Client OutputMeaningNext Step
Connection refusedSSH service not running or wrong portCheck systemctl status ssh on server
Connection timed outHost unreachable or firewallCheck DNS, ping, firewall rules
No more authentication methodsAll auth methods exhaustedCheck key is in authorized_keys
Server accepts key then hangsServer found key, auth packet lostSee “Postponed publickey” section below
Permission denied (publickey)Key not recognized by serverFingerprint mismatch — see Step 3

​
Key Lines to Find

# 1. Which keys the client will try:
debug1: Will attempt key: /home/user/.ssh/id_ed25519 ED25519 SHA256:xxx agent

# 2. Whether the server recognizes the key:
debug1: Server accepts key: ... ED25519 SHA256:xxx agent

# 3. What auth method is used after PK_OK:
debug3: sign_and_send_pubkey: using publickey-hostbound-v00@openssh.com

​
Step 2: Server-Side Auth Log

On the target server, check the auth log for your connection attempts: 
tail -20 /var/log/auth.log

​
What to Look For

Server LogMeaning
Accepted publickey for root from ... ssh2Auth succeeded
Connection closed by authenticating user ... [preauth]Auth never completed — connection dropped during handshake
Failed publickey for root from ... ssh2Key signature verification failed
Invalid user ... from ...Username doesn’t exist on server

​
Step 3: Fingerprint Comparison

Verify the key on the client matches the key on the server: 
# Client side -- key file fingerprint:
ssh-keygen -lf ~/.ssh/id_ed25519.pub

# Client side -- agent fingerprint:
ssh-add -l

# Server side -- authorized_keys fingerprint:
ssh-keygen -lf /root/.ssh/authorized_keys
All three should show the same SHA256:xxx fingerprint.

​
Step 4: Permission Check

SSH silently rejects keys if file permissions are wrong: 
# On the server:
ls -la /root/.ssh/
ls -la /root/.ssh/authorized_keys
Required permissions:
PathPermissionsOwner
/root/.ssh/700 (drwx------)root:root
/root/.ssh/authorized_keys600 (-rw-------)root:root

​
Step 5: Server Debug Mode

The most powerful tool. Run sshd in debug mode on an alternate port: 
# On the server (via web terminal or existing session):
/usr/sbin/sshd -d -p 2222
Then connect from the client: 
timeout 15 ssh -v -o ConnectTimeout=10 -o BatchMode=yes \
  -p 2222 root@target-host echo "OK" 2>&1

​
Critical Server Debug Lines

# Key found in authorized_keys:
debug1: /root/.ssh/authorized_keys:2: matching key found: ED25519 SHA256:xxx
Accepted key ED25519 SHA256:xxx found at /root/.ssh/authorized_keys:2

# Auth postponed (normal -- waiting for signed challenge):
Postponed publickey for root from ... [preauth]

# Connection closed before auth completed (BAD -- packet lost):
Connection closed by authenticating user root ... [preauth]

​
The “Postponed publickey” Pattern

​
What It Means

In the SSH protocol, public key auth is a two-phase process:
  1. Probe: Client sends key blob (no signature). Server checks authorized_keys. If found, sends PK_OK.
  2. Sign: Client signs a challenge with the private key. Server verifies signature.
“Postponed publickey” means phase 1 succeeded (key found) and the server is waiting for phase 2 (signed response). If the connection then drops, the signed packet never arrived.

​
Common Causes

CauseDiagnosisFix
MTU/packet dropLarge signed packet silently dropped by networkLower MTU: ip link set eth0 mtu 1400
publickey-hostbound-v00 incompatibilityOpenSSH 10.x client vs older network infrastructureUse paramiko or password auth
Agent hangSSH agent can’t sign (locked, crashed)ssh-add -l to verify, restart agent

​
Network Diagnosis

# Check current MTU:
ip link show eth0 | grep mtu

# Check PMTU discovery:
cat /proc/sys/net/ipv4/ip_no_pmtu_disc
# 0 = enabled (default), 1 = disabled

# Lower MTU temporarily:
ip link set eth0 mtu 1400

​
SSH Config Files to Check

​
Override Files (Ubuntu 24.04)

ls /etc/ssh/sshd_config.d/
Files load in alphabetical order. Later files override earlier ones:
  • 50-cloud-init.conf — may set PasswordAuthentication yes
  • 60-cloudimg-settings.conf — may override to PasswordAuthentication no

​
Effective Configuration

sshd -T 2>/dev/null | grep -iE "usedns|maxauth|logingrace|pubkeyauth|passwordauth"

​
Quick Reference: Common Fixes

# Enable password auth (when pubkey is broken):
echo "PasswordAuthentication yes" > /etc/ssh/sshd_config.d/60-cloudimg-settings.conf
systemctl restart ssh

# Fix permissions:
chmod 700 /root/.ssh
chmod 600 /root/.ssh/authorized_keys
chown -R root:root /root/.ssh

# Restart SSH (Ubuntu 24.04):
systemctl restart ssh    # NOT sshd

# Test locally (rules out network issues):
ssh -o ConnectTimeout=5 root@localhost echo "LOCAL_OK"