8.3.2 — SSH, From the Host Key Warning to Certificates

The authenticity of host 'prod-01 (203.0.113.40)' can't be established.
ED25519 key fingerprint is SHA256:qX9…
Are you sure you want to continue connecting (yes/no)?

Almost everyone types yes without looking. That prompt is the only moment in the entire SSH relationship where you are asked to verify who you are talking to, and skipping it means the first connection could be to an attacker in the middle, whose key you then trust permanently.

Understanding why that prompt exists, and what replaces it at scale, is most of what there is to know about SSH's trust model.

1. Why SSH exists

Before 1995, remote administration used telnet, rlogin and rsh, which sent passwords in plain text across the network. Tatu Ylönen wrote SSH at Helsinki University of Technology after a password-sniffing attack on the university network — the tool was a direct response to an incident.

SSH-1 had protocol flaws, including a weak integrity check that allowed content injection. SSH-2 is a full redesign (RFCs 4251–4254, 2006) and is what everyone means today. OpenSSH is the dominant implementation.

The protocol has three layers, and knowing them makes the error messages readable:

• Transport layer — negotiates algorithms, performs key exchange, authenticates the server, and provides encryption and integrity from that point on.
• User authentication layer — proves who the client is, over the already-encrypted channel.
• Connection layer — multiplexes logical channels over the one connection: your shell, a file transfer, three port forwards, all at once.

Note the ordering: the server is authenticated first, then the user. That is why a wrong password is never sent in the clear — but it is also why the host key check is the step everything else rests on, since everything after it assumes you are talking to the right machine.

2. The handshake

1. Version exchange in plain text (SSH-2.0-OpenSSH_9.6). This is why banner-grabbing scanners can identify SSH versions.
2. Algorithm negotiation — key exchange, host key type, ciphers, MACs, compression.
3. Key exchange — ECDH (usually Curve25519), producing a shared secret with forward secrecy, exactly as in Chapter 8.2.3.
4. Server authentication — the server signs the exchange with its host key. The client checks that signature against what it knows about this host. This is the step the prompt is about.
5. Session keys derived; everything after this is encrypted.
6. User authentication — public key, password, or certificate.
7. Channels opened.

Rekeying happens periodically (by default after about an hour or a gigabyte), limiting how much data any one key protects.

3. Host keys and known_hosts

A server has a long-lived key pair per algorithm, generated at installation, living in /etc/ssh/ssh_host_ed25519_key. The client stores the public part in ~/.ssh/known_hosts on first connection.

This is trust on first use: you accept the identity the first time and detect changes afterwards.

When the stored key does not match:

@@@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @@@

This is a real warning with three possible causes, and the habit of deleting the line and reconnecting is exactly what an attacker relies on:

• The server was rebuilt and generated new host keys — common, benign, and it should have been communicated.
• You are connecting to a different machine behind the same name — a load balancer with unsynchronised host keys, or a recycled IP address.
• Someone is intercepting the connection.

The right response is to verify the fingerprint out of band — from the build pipeline output, the cloud console, or a colleague — not to remove the line.

Three ways to escape trust on first use:

Publish the fingerprint in your provisioning output so the first connection can be checked.

SSHFP records in DNS — publish the host key fingerprint in DNS, and with DNSSEC the client can verify automatically (VerifyHostKeyDNS yes).

Host certificates — section 8, and the answer that actually scales.

A detail worth knowing: modern OpenSSH hashes hostnames in known_hosts by default (HashKnownHosts), so a compromised laptop does not hand an attacker a list of every server you administer. ssh-keygen -F hostname searches it; ssh-keygen -R hostname removes an entry.

4. Public key authentication

How it works: you place your public key in ~/.ssh/authorized_keys on the server. At login, the server sends a challenge; your client signs it with the private key; the server verifies with the public key. The private key never leaves your machine and no secret crosses the wire — which is the whole advantage over passwords.

ssh-keygen -t ed25519 -C "ana@laptop"        # (1)
ssh-copy-id -i ~/.ssh/id_ed25519.pub prod-01 # (2)

(1) Ed25519 is the default choice — small, fast, and immune to the nonce failure of Chapter 8.2.3. RSA is the compatibility fallback and needs at least 3072 bits. ECDSA works and has no advantage over Ed25519. DSA is removed from modern OpenSSH. (2) Appends the public key to the remote authorized_keys with correct permissions.

Permissions are enforced strictly and the error is confusing.

Permissions 0644 for '/home/ana/.ssh/id_ed25519' are too open.

SSH refuses to use a private key that others can read. chmod 600 the key, 700 the .ssh directory. On the server side, authorized_keys must not be group-writable and neither must the home directory — a world-writable home silently breaks key authentication, and the reason only appears in the server log.

authorized_keys options restrict what a key may do, and they are underused:

command="/usr/local/bin/backup-only",no-pty,no-port-forwarding,from="10.0.0.0/8" ssh-ed25519 AAAA…

That key can run exactly one program, cannot get an interactive shell, cannot forward ports, and only works from one network. This is how you give a backup system access without giving it a shell, and it is far better than a second user account with a password.

5. The agent, and why forwarding is dangerous

A passphrase-protected key must be decrypted for every use. ssh-agent holds the decrypted key in memory and signs on request, so you type the passphrase once.

eval "$(ssh-agent -s)"
ssh-add -t 8h ~/.ssh/id_ed25519     # expires after eight hours

Agent forwarding (-A) is the feature to understand and mostly avoid. It exposes your agent's socket on the remote machine so you can hop onward. But anyone who is root on that machine — or who compromises it — can use that socket to sign with your keys for as long as you are connected. They cannot copy the key, and they do not need to: they can authenticate as you to every machine your key opens. This is a documented step in real intrusions.

ProxyJump is the correct replacement, and it is strictly better:

Host prod-*
  ProxyJump bastion.example.com    # (1)
  User deploy
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes               # (2)

(1) The connection is tunnelled through the bastion, and the bastion never sees your key or your traffic — the end-to-end SSH session is established with the final host. (2) Offer only this key. Without it, the client offers every key in the agent one at a time, which can hit MaxAuthTries and fail with a confusing "too many authentication failures".

If you genuinely must forward, use ssh-add -c so every signature requires a confirmation prompt on your laptop.

6. Tunnels

Three forms, and the direction is what people confuse.

Local forwarding (-L) brings a remote service to you.

ssh -L 5432:db.internal:5432 bastion
# now localhost:5432 on your laptop reaches db.internal:5432 through the bastion

The database is not exposed to the internet; your connection travels inside the SSH session. This is how you reach a private database without a VPN, and note that db.internal is resolved on the bastion, not on your laptop.

Remote forwarding (-R) exposes something of yours on the remote machine.

ssh -R 8080:localhost:3000 jump-host
# port 8080 on jump-host now reaches your laptop's port 3000

By default it binds to the remote loopback only; GatewayPorts would expose it more widely, and that is a decision to make deliberately, because it can punch a hole from the internet into a developer's machine.

Dynamic forwarding (-D) is a SOCKS proxy.

ssh -D 1080 bastion
# point a browser at socks5://localhost:1080 and it browses from the bastion

All three are why a bastion host with SSH access is a significant privilege. A user who can SSH to a jump host can generally reach anything that host can reach. no-port-forwarding in authorized_keys, or AllowTcpForwarding no in the server configuration, is how you grant shell access without granting network reach.

7. Configuration and hardening

Client ~/.ssh/config removes almost all of the flags you would otherwise type:

Host *
  ServerAliveInterval 60        # (1)
  AddKeysToAgent yes
  HashKnownHosts yes

Host bastion
  HostName bastion.example.com
  User ana
  ControlMaster auto            # (2)
  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlPersist 10m

(1) Sends a keepalive every 60 seconds so an idle session is not silently dropped by a NAT device or firewall (Chapter 5.3.3). (2) Connection multiplexing: the first connection opens a master, and later ones reuse it over the same TCP session. New sessions become instant because there is no handshake, which makes tools that open many SSH connections dramatically faster.

Server /etc/ssh/sshd_config, the settings that matter:

PermitRootLogin no              # (1)
PasswordAuthentication no       # (2)
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowGroups ssh-users           # (3)
MaxAuthTries 3
LoginGraceTime 20               # (4)
X11Forwarding no

(1) Root logs in through a named user plus sudo, so actions are attributable. (2) This is the single highest-value line — it removes brute-force and credential-stuffing entirely as a category. (3) An explicit allow list beats a deny list. (4) Unauthenticated connections are dropped quickly, limiting resource exhaustion.

On changing the port from 22: it reduces log noise from automated scanners and provides no security against anyone who scans all ports, which any real attacker does. It is a legitimate noise-reduction measure and not a control. Say that plainly rather than defending it as security.

fail2ban and equivalents ban IP addresses after repeated failures. With password authentication disabled, its value is mostly log volume — which is still worth having.

Always test a new sshd_config in a second session before closing the first one. sshd -t validates syntax; a mistake with the only session open means a trip to a console.

8. SSH certificates: the answer at scale

authorized_keys breaks down at scale in three ways: every joiner and leaver means editing files on every host, there is no expiry, and every first connection is a trust-on-first-use prompt.

SSH certificates fix all three. You run a CA — a key pair used only for signing — and it signs both user keys and host keys.

# Sign a user key: valid 8 hours, may log in as deploy or ana
ssh-keygen -s user_ca -I "ana@example.com" -n deploy,ana -V +8h id_ed25519.pub  # (1)

# Sign a host key
ssh-keygen -s host_ca -I prod-01 -h -n prod-01.example.com -V +52w \
  /etc/ssh/ssh_host_ed25519_key.pub                                            # (2)

(1) -n lists the principals — the usernames this certificate may log in as. -V +8h is the validity, and short validity is what removes the offboarding problem: revocation becomes expiry. (2) -h marks it a host certificate.

Then each server trusts the user CA (TrustedUserCAKeys) and presents its host certificate (HostCertificate), and each client trusts the host CA via one @cert-authority line in known_hosts.

The result: no per-host key distribution, no trust-on-first-use prompts ever again, and access that expires on its own. This is what every large infrastructure ends up doing, usually with a service that issues short-lived certificates after checking your single sign-on session — which ties SSH access to the identity system in Chapter 8.4 rather than to files scattered across machines.

9. Transferring files, and reading errors

scp is deprecated. It used a protocol with known weaknesses, including the server being able to influence which files the client writes. Modern OpenSSH implements the scp command over SFTP, and the guidance is to use sftp or rsync -e ssh directly. rsync is the right tool for anything repeated — it transfers only differences, resumes, and preserves attributes.

Errors worth being able to read:

Message	Cause
Permission denied (publickey)	No accepted key — wrong key, wrong user, or bad permissions on the server's home or authorized_keys
Too many authentication failures	The agent offered too many keys; set IdentitiesOnly yes
Permissions 0644 … are too open	chmod 600 the private key
REMOTE HOST IDENTIFICATION HAS CHANGED	Verify out of band before removing anything
Connection closed by remote host	Often a server-side policy: AllowGroups, MaxStartups, or fail2ban

ssh -vvv host prints the whole negotiation, including which keys were offered and why each was rejected. It answers "permission denied" faster than any amount of guessing.

What the interviewer will push on

"What does the host key warning actually protect against?" An attacker in the middle. SSH authenticates the server before the user, so the host key check is the only step that establishes who you are talking to — and trust on first use means the first connection is the vulnerable one. The tell is saying that the correct response to a changed key is out-of-band verification, not deleting the line.

"How does public key authentication work?" The server sends a challenge, the client signs it with the private key, the server verifies with the stored public key. No secret crosses the wire, which is the substantive difference from passwords. Then mention that the permissions on authorized_keys and the home directory are enforced, and that a group-writable home silently breaks it.

"Why is agent forwarding dangerous, and what do you use instead?" Anyone with root on the intermediate machine can use your agent socket to sign as you, for every host your key opens, while you are connected. ProxyJump tunnels through the bastion without exposing the key, and it is strictly better. Adding ssh-add -c for the rare case where forwarding is genuinely required is the complete answer.

"Explain -L versus -R." -L brings a remote service to your local port; -R exposes a local service on the remote machine. Then draw the security consequence: any user who can SSH to a bastion can generally reach whatever it can reach, so AllowTcpForwarding no or no-port-forwarding in authorized_keys is how you grant shell without network reach.

"How would you manage SSH access for 200 engineers and 2,000 servers?" SSH certificates. A CA signs short-lived user certificates with principals, servers trust the CA, and host certificates remove trust-on-first-use. Offboarding becomes expiry rather than editing files on every host. authorized_keys distribution is the wrong answer at that size.

"Is changing the SSH port a security measure?" No. It reduces automated scanner noise in your logs and provides nothing against anyone who scans all ports. Being willing to say that clearly, and then naming what actually helps — disabling password authentication — is what the question is testing.

One thing to volunteer: mention ControlMaster multiplexing. Reusing one TCP connection makes subsequent sessions instant, which transforms the speed of any tool that opens many SSH connections. It is a two-line configuration change that most engineers have never enabled.

Recall

• SSH has three layers — transport (key exchange, server authentication, encryption), user authentication, and connection (multiplexed channels). The server is authenticated before the user, which makes the host key check the step everything else rests on.
• Trust on first use: the host key is stored in known_hosts on first connection. A changed key has three causes and must be verified out of band, not deleted. Escape it with published fingerprints, SSHFP DNS records, or host certificates.
• Public key auth signs a server challenge — no secret crosses the wire. Use Ed25519. Permissions are enforced: 600 on the key, and a group-writable home silently breaks server-side authentication.
• authorized_keys options (command=, no-pty, no-port-forwarding, from=) grant narrow access without a shell.
• Agent forwarding lets anyone with root on the intermediate host sign as you for every machine your key opens. Use ProxyJump; if you must forward, use ssh-add -c. IdentitiesOnly yes prevents "too many authentication failures".
• -L brings a remote service to your local port, -R exposes yours remotely, -D is a SOCKS proxy. Bastion SSH access implies network reach unless forwarding is disabled.
• Hardening, in value order: PasswordAuthentication no, PermitRootLogin no, AllowGroups, MaxAuthTries, LoginGraceTime. Changing the port reduces log noise, not risk. Test config in a second session.
• SSH certificates solve scale: a CA signs user certificates with principals and short validity, and host certificates end trust-on-first-use. Offboarding becomes expiry. ControlMaster multiplexing makes repeat connections instant.

Self-test: Why is the host key check the step everything else depends on? · What does the server actually verify during public key authentication? · What can an attacker with root on a jump host do with a forwarded agent? · Which forwarding direction exposes your laptop? · What does an SSH certificate's -n flag control? · What does changing port 22 actually buy?

Next: 8.4.1 opens the identity folder — the ten pages covering how users prove who they are, starting with passwords and the account flows that leak accounts even when the hashing is right.