Skip to main content

​
The Packet That Never Arrived

Design Journal — 28 March 2026

​
The Setup

It was supposed to take five minutes. SSH into a VPS on Hostinger, confirm Claude Code CLI was installed, clone a repo, move on with my day. I had a runbook open — a verification checklist for our autonomous code execution pipeline — and Section 0.2 was meant to be the boring one. The “just SSH in and check” step. I typed the command I’ve typed a thousand times: 
ssh -o ConnectTimeout=5 root@vps.devarno.cloud echo "SSH OK"
Nothing. Not a timeout. Not a refusal. Not a password prompt. Just… silence, then disconnect. Connection closed by remote host. And that was the start of a four-hour debugging rabbit hole that took me through DNS records, packet captures, OpenSSH changelogs, MTU tuning, and eventually a workaround involving a Python library from 2003. The kind of session where you emerge blinking into daylight, simultaneously exhausted and buzzing, thinking: I’ve never felt more like a network nerd.

​
The First Misdirection: DNS

The obvious thing. vps.devarno.cloud — does it even resolve? 
dig vps.devarno.cloud +short
# (nothing)
No A record. Classic. I’d set up this VPS months ago, pointed the DNS at it, and at some point the record had vanished. Easy fix — create an A record pointing to 187.124.113.147, the Hostinger Docker/Traefik box. Waited for propagation. Tried again. 
ssh root@vps.devarno.cloud
Connection established… key exchange completed… then: Connection closed by remote host [preauth]. That [preauth] is the tell. The server accepted my key, started the authentication handshake, and then dropped me before completing it. This isn’t a DNS problem. This isn’t a firewall problem. Something weirder is going on.

​
Going Deeper: The Server’s Perspective

When SSH fails silently, the client gives you almost nothing useful. The server is where the truth lives. I needed sshd in debug mode — but I couldn’t SSH in to start it. Fortunately, I had another way in: the Hostinger web console. Clunky, laggy, but it works. I opened a root shell through the browser panel and started a debug SSH daemon on a separate port: 
/usr/sbin/sshd -d -p 2222
Then from my local machine: 
ssh -v -p 2222 root@vps.devarno.cloud
The server-side debug output told the real story: 
debug1: userauth-request for user root service ssh-connection method publickey-hostbound-v00@openssh.com
debug1: attempt 1 failures 0 [preauth]
Accepted key RSA SHA256:xxx... [preauth]
Postponed publickey-hostbound-v00@openssh.com for root [preauth]
...
Connection closed by 143.58.177.146 [preauth]
Read that carefully. The server accepted the key. It recognised my public key, validated it, and said “yes, this key is fine.” Then it postponed the auth (standard for multi-step authentication) and waited for the signed response packet. That packet never arrived.

​
The Ghost Packet

This is where it gets interesting, and where most debugging guides would fail you. Because from the client’s perspective, it did send the response. The verbose client log showed it constructing and transmitting the signed authentication packet. No errors, no warnings. Just… sent into the void. The key exchange — the initial Diffie-Hellman handshake — worked perfectly. Those are small packets. The signed authentication response in publickey-hostbound-v00@openssh.com is significantly larger. And somewhere between my machine (143.58.177.146) and the Hostinger VPS (187.124.113.147), that larger packet was being silently dropped. Plain English: Every small packet made it through fine. The initial handshake, the key offer, the server’s acceptance — all good. But the one large packet — the signed proof that I hold the private key — vanished in transit. The server waited, got nothing, and hung up.

​
The MTU Red Herring

My first instinct: MTU (Maximum Transmission Unit). If a packet exceeds the path MTU and the “Don’t Fragment” bit is set, intermediate routers will drop it. Classic cause of “some packets work, large ones don’t.” 
sudo ip link set dev wlan0 mtu 1400
Dropped from the default 1500 to 1400, forcing smaller fragments. Tried SSH again. Same failure. Tried 1200. Same. Tried 1000. Same. This wasn’t MTU. Whatever was dropping that packet didn’t care about size thresholds — it cared about something in the packet’s content or protocol negotiation. Or it was some middlebox doing deep packet inspection and choking on the publickey-hostbound-v00 extension.

​
The Culprit: OpenSSH 10.2 and publickey-hostbound-v00

Here’s what I eventually pieced together. OpenSSH 10.2 (which ships with recent Ubuntu/Arch releases) changed the default authentication method. Instead of plain publickey, it prefers publickey-hostbound-v00@openssh.com — an extension that binds the authentication proof to the specific host key, preventing certain MITM attacks. The mechanism is sound. The security improvement is real. But the signed response packet it produces is structurally different from traditional publickey auth, and some network paths silently drop it. This isn’t documented in any FAQ I could find. No “known issues” page. No Stack Overflow answer. Just a subtle protocol change that interacts badly with whatever middlebox or firewall sits between certain ISPs and certain hosting providers. The proof: Python’s paramiko library, which implements SSH from scratch and uses traditional publickey auth, connected instantly with the same key to the same server on the same network path. The key was fine. The server was fine. The protocol negotiation was the problem.

​
The Workaround (and Why I Chose It)

I had three options:
  1. Force OpenSSH to use legacy publickey auth — possible via config, but fragile and affects all connections
  2. Use paramiko as the SSH transport — works, but means wrapping every command in Python
  3. Enable password auth as a fallback — crude, but gets me unblocked in 30 seconds
I chose option 3. Through the Hostinger web console: 
echo "PasswordAuthentication yes" > /etc/ssh/sshd_config.d/60-cloudimg-settings.conf
systemctl restart ssh  # Ubuntu 24.04: 'ssh', not 'sshd'
Two lines. Problem solved. Not elegant, not permanent, but I had a runbook to finish and an autonomous execution pipeline to verify. (Side note: Ubuntu 24.04 renamed the SSH service from sshd to ssh. If you systemctl restart sshd you’ll get a “unit not found” error that makes you question your entire career for about 15 seconds.)

​
The Lesson: Layer-by-Layer, or You’ll Chase Ghosts

Here’s the thing about debugging network issues: every layer looks fine from its own perspective.
  • DNS: Resolved correctly (after the fix)
  • TCP: Connection established, three-way handshake completed
  • SSH transport: Key exchange succeeded
  • SSH auth: Key accepted by server
  • Application: Client sent the signed response
No errors at any individual layer. The failure existed between layers — in the interaction between a new SSH protocol extension and the network path’s handling of it. If I’d only looked at client logs, I’d have blamed the server. If I’d only looked at the server, I’d have blamed the client. The truth was neither. Plain English: I drew the standard network layer stack and worked through each one. Layers 1-4 were clean. Layer 7 was clean. The failure was at Layer 5 — the authentication protocol — where a new SSH extension produced a packet that the network path refused to carry. Every layer above and below it had no idea anything was wrong.

​
Why This Matters Beyond SSH

I’m building an autonomous code execution pipeline. FORGE reads a GitHub project board, resolves dependencies, selects the next unblocked ticket, and assembles an implementation prompt. TOMMY takes that prompt, SSHes into a VPS, and executes it via Claude Code CLI. When it works, it closes the GitHub issues automatically. The entire system depends on SSH. Not “SSH would be nice to have.” SSH is the literal transport layer between the orchestration brain (n8n on one VPS) and the execution muscle (Claude on another). If SSH breaks, the whole pipeline is dead. And this is the lesson for anyone building distributed systems, autonomous agents, or infrastructure that spans multiple providers: the connections between your boxes are as important as what’s inside them. Your architecture diagram has boxes and arrows. Everyone obsesses over what’s in the boxes. The arrows are where the ghosts live.

​
The Founder Moment

There’s a particular feeling you get at 2am when you’ve been debugging a network issue for hours and you finally see it. Not solve it — see it. The moment the mental model clicks and you understand why the packet disappeared. You’re staring at sshd -d output and your brain goes: “Oh. It’s the size of the auth response combined with the type of the auth method. The middlebox is inspecting the packet, doesn’t recognise the extension, and drops it.” It’s the same feeling as product-market fit, weirdly. That moment of clarity after fumbling in the dark. Except instead of customers, it’s packets. And instead of revenue, it’s a TTY prompt. I sat there for a second, grinning like an idiot at a terminal full of debug logs, thinking: I’ve never felt more like a network nerd. And honestly? That’s the best part of building things. Not the launches or the metrics. The understanding.

​
Your Debugging Checklist (Steal This)

If you ever hit a “connection closed [preauth]” and the key is definitely right:
  1. Server-side debug first: sshd -d -p 2222 — the client lies, the server knows
  2. Check the auth method: Look for publickey-hostbound-v00 — if you see it, your OpenSSH is using the new extension
  3. Test with paramiko: pip install paramiko and try connecting via Python — if it works, the protocol extension is the issue, not your key
  4. Check your OpenSSH version: ssh -V — if it’s 9.8+ you have the new default
  5. Don’t trust MTU alone: If MTU changes don’t help, it’s not a fragmentation issue — it’s content-based filtering
  6. Ubuntu 24.04 trap: The service is ssh, not sshd. You will forget this.

​
What’s Next

The SSH issue was blocking Section 0.2 of our FORGE x TOMMY verification runbook. With it resolved, we’ve moved through Sections 0.2 and 0.3, verified the GitHub epic, and are about to start Phase 1: triggering FORGE in manual mode to see if it correctly reads the project board, resolves block dependencies, and assembles a coherent implementation prompt. That’s a story for the next entry. But the shape of it is already visible: boxes and arrows, all the way down.

​
Campaign: Cross-Platform Strategy

​
Twitter/X Thread (5 tweets)

Tweet 1 (Hook):
SSH’d into a VPS today. Key exchange worked. Auth accepted. Then… silence. The signed auth packet literally vanished in transit. A thread on debugging ghost packets, OpenSSH 10.2, and why the arrows in your architecture diagram matter more than the boxes. :thread:
Tweet 2:
OpenSSH 10.2 changed the default auth method to publickey-hostbound-v00@openssh.com Better security. Bigger auth packet. And some network paths silently DROP it. No error. No timeout on the packet itself. Just… gone. Server waits, gives up, hangs up.
Tweet 3:
The proof: Python’s paramiko (traditional publickey auth) connected instantly. Same key. Same server. Same network. The protocol extension was the problem. The network path between my ISP and Hostinger couldn’t handle it.
Tweet 4:
Debugging method that saved me: sshd -d -p 2222 on the server ssh -v -p 2222 from client Server showed: “Accepted key… Postponed… Connection closed [preauth]” The key was FINE. The signed response never arrived.
Tweet 5 (CTA):
Full teardown with mermaid diagrams, the layer-by-layer debugging methodology, and an architecture cheatsheet companion piece: [link] If you’ve ever hit “connection closed [preauth]” and lost an afternoon — this one’s for you.

​
LinkedIn Post

The packet that never arrived. I spent four hours debugging an SSH connection that should have taken five minutes. Key exchange worked. Server accepted my public key. Then silence — the signed authentication response vanished somewhere between my machine and the hosting provider. The culprit: OpenSSH 10.2’s new publickey-hostbound-v00 extension produces a structurally different auth packet that some network paths silently drop. No error. No log entry. Just a ghost. Three things I learned:
  1. Always debug from the server side first (sshd -d). The client will lie to you.
  2. When “everything looks fine” at every layer, the failure is between layers — in the interaction.
  3. The arrows in your architecture diagram are where the ghosts live. Everyone obsesses over what’s inside the boxes.
I’m building an autonomous code execution pipeline where SSH is the transport between the AI brain and the execution server. If SSH breaks, everything breaks. This debugging session reminded me that infrastructure isn’t just provisioning — it’s understanding every packet on the wire. Full write-up with diagrams and a debugging checklist: [link] #SSH #Debugging #DevOps #InfraEngineering #BuildInPublic #FounderLife
  1. Cover: “THE PACKET THAT NEVER ARRIVED” — dark terminal aesthetic, green-on-black text
  2. The Setup: “SSH’d into a VPS. Key accepted. Then… silence.” — show the connection flow diagram
  3. The Ghost: Mermaid sequence diagram showing the dropped packet (simplified, large text)
  4. The Method: “Debug from the server, not the client” — sshd -d command highlighted
  5. The Culprit: “OpenSSH 10.2 changed the default auth method. Some networks can’t handle it.”
  6. The Lesson: “The arrows matter more than the boxes.” — CTA to full write-up

​
Short-Form Video (60-90s — TikTok/Reels/Shorts)

Hook (0-5s): Screen recording of terminal. “This SSH connection should have taken 5 minutes. It took 4 hours. Here’s why.” Body (5-60s): Fast-paced walkthrough:
  • Show the failed connection (dramatic pause on “Connection closed”)
  • Show sshd -d output (“The server ACCEPTED the key…”)
  • Whiteboard animation of the packet flow (small packets pass, big packet drops)
  • Reveal: “OpenSSH 10.2 changed the auth protocol. The network ate the new packet.”
  • Paramiko connecting instantly as proof
Close (60-75s): “The lesson: when everything looks fine at every layer, the bug lives BETWEEN the layers. Full debugging teardown linked in bio.” Music: Something tense/electronic — Mr. Robot soundtrack vibes

​
Newsletter Excerpt

This week’s deep-dive: The Packet That Never Arrived I hit one of those debugging sessions that reminds you why understanding infrastructure at the protocol level still matters in 2026. An SSH connection that silently dropped — no error, no timeout, just a vanishing packet. The root cause was a subtle interaction between OpenSSH 10.2’s new authentication extension and the network path between my ISP and a Hostinger VPS. The kind of bug that doesn’t show up in any single log, at any single layer. This one’s a genuine detective story with mermaid diagrams, a debugging checklist you can steal, and a companion “Architecture Layering Cheatsheet” that maps the full debugging methodology. [Read the full teardown →]

​
Virality Boosters

​
Emojis (use sparingly, context-appropriate)

  • :ghost: for “ghost packet” references
  • :detective: for the debugging narrative
  • :electric_plug: for connection/networking moments
  • :brain: for the “aha” moment
  • :thread: for Twitter thread opener

​
GIF Ideas

  • “This is fine” dog in burning room — for the moment every layer looks fine but nothing works
  • Mr. Robot Elliot staring at terminal — for the 2am debugging moment
  • “Enhance… enhance…” zoom meme — for going deeper through each network layer
  • Leonardo DiCaprio pointing at TV — for the moment you spot publickey-hostbound-v00 in the logs

​
Content Cadence Placement

Position: Technical deep-dive week This post alternates with the previous broad-audience campaign (“From GitHub Epic to Closed Issues — Zero Human Intervention”). The cadence:
WeekTypePost
PreviousBroadFORGE x TOMMY autonomous execution
This weekDeep-diveThe Packet That Never Arrived (this post)
This weekReferenceArchitecture Layering Cheatsheet (companion)
NextBroadPhase 1 verification — watching FORGE read a project board
The cheatsheet is an evergreen companion that outlives the blog post’s news cycle.

​
Extra Opportunities

​
Spin-Off Topics

  1. “MTU: The Most Misunderstood Network Setting” — deep-dive on when MTU matters and when it’s a red herring
  2. “SSH Config Tricks That Would’ve Saved Me 4 Hours” — practical ~/.ssh/config for multi-host setups
  3. “The Middlebox Problem: Why Your Packets Aren’t Your Own” — broader piece on DPI, carrier-grade NAT, and protocol ossification
  4. “paramiko vs OpenSSH: When the Reference Implementation Is the Problem” — technical comparison

​
Interactive Audience Prompts

  • “What’s your worst ‘it was DNS’ story? Except this time, it wasn’t DNS.”
  • “Drop your most cursed SSH debug session below. I’ll read the best ones on stream.”
  • Poll: “When SSH fails silently, what do you check first?” (DNS / Firewall / Server logs / Cry)

​
Series Connection

This is Design Journal Entry #7 in the FORGE x TOMMY build series. It connects backward to the autonomous execution announcement and forward to the verification phases. The debugging narrative humanises the infrastructure work that the marketing-focused posts gloss over.

​
Collaboration Angles

  • Hostinger DevRel — tag them on the hosting-specific network path issue. They may want to investigate the middlebox.
  • OpenSSH mailing list — submit a concise bug report about publickey-hostbound-v00 packet drops on certain transit paths. Could get cited.
  • ThePrimeagen / Fireship — “cursed SSH debugging” is exactly their content niche. A collab or reaction video is plausible.