Your home server, explained.

SSH is how you sit at another computer’s keyboard from far away: you type on your laptop, the commands run on the other machine, and nobody on the network in between can read or fake what you send. Tailscale is the app that lets your laptop find that machine at all, from anywhere, even though your home router normally turns away anyone knocking from outside.

You lean on both every day. They’re how you get to the home server and the DGX Spark, and soon how you’ll reach the model server and its dashboards on the Spark without opening them to the whole internet. Right now it all works by magic, which is fine until it doesn’t: a frightening warning the day you reinstall the server, a connection that’s suddenly slow, a door left open that bots find within hours. Knowing what’s really happening is what lets you fix those, and answer an interviewer who asks how a packet gets into a house that’s built to keep it out.

It reuses two ideas from earlier lessons, how machines are addressed in Packets and IP and how two strangers agree on a secret in public in How does HTTPS keep secrets?, and it unlocks the rest of your Spark work: Serving engines and Observability both reach the Spark the way you set up here.

Today

• Say what is actually running on your home server when you “have SSH” (a background program waiting by the door), and walk through one connection from the first packet to the prompt: how the two machines agree on a secret, how the server proves it’s really your server, how you prove you’re you, and how one connection carries many separate streams.
• Read the scary REMOTE HOST IDENTIFICATION HAS CHANGED warning and know whether to worry.
• Use a short settings file and a few tunnelling flags (~/.ssh/config, -L, -R, -D, ProxyJump) so that opening a Jupyter notebook running on the Spark in your laptop’s browser is one short command.
• Explain why your laptop can’t reach your home server from a cafe without help (the router shares one public address among every device in the house and turns away strangers), and why poking a hole in the router for it is the risky fix.
• Explain what Tailscale really does: encrypted tunnels straight between your devices (WireGuard), a central server that only swaps public keys and never sees your traffic, a trick for slipping through two routers at once, and relay servers for when that trick fails. Then prove it with Tailscale’s own diagnostics and by watching the packets go by.

Packets and IP and How does HTTPS keep secrets? aren’t strictly required first; they just make a few sections click faster. TCP vs UDP also helps: it explains why WireGuard sends its packets the quick, no-guarantees way. Budget one evening on the home server, plus half an hour with the Spark.

The Problem

You have a box at home. You installed Tailscale on it and on your laptop, typed ssh gaurav@homeserver, and it worked. From your apartment, from the office, from a cafe. Great.

Now some questions you cannot answer yet:

• The first time you connected, ssh asked Are you sure you want to continue connecting (yes/no/[fingerprint])?. You typed yes. What did you just agree to?
• One day it will say WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Is someone doing something nasty?
• Is password login still turned on? If your router ever forwards port 22, how long before a bot finds it?
• You want Jupyter running on the Spark to show up in your laptop’s browser. How?
• Before Tailscale, your laptop at the cafe could not reach the box at all. After installing an app, it can, and you never opened a port on your router. How does a packet get in? Is it going through Tailscale’s servers? Can Tailscale read it?

None of this is magic. It is three old ideas stacked up: a program that listens for connections (sshd), a careful handshake that makes two strangers share a secret (the same trick as TLS), and a way to sneak a conversation through two routers that were built to stop exactly that.

The Idea

A program that waits by the door

When people say “my server has SSH”, they mean a program called sshd is running on it. The trailing d is for daemondaemonA program that runs in the background with no terminal attached, usually started at boot, waiting to do a job. Unix names them with a trailing “d”: sshd, tailscaled, systemd..

sshd’s job is to sit on portportA number from 1 to 65535 that picks which program on a machine gets an incoming connection. An IP address finds the machine; the port finds the program on it. SSH’s standard port is 22. 22 and wait. In operating system terms it has asked the kernelkernelThe core of the operating system, the one program that talks to the hardware directly, including the network card. to listenlistenTo tell the kernel “if a TCP connection arrives for this port, hand it to me”. A listening program is like a phone that is plugged in and switched on. on TCP port 22. When a connection arrives, the kernel hands it over, sshd starts a new child process just for that one connection, and goes back to waiting. The child does the handshake, checks who you are, then becomes a shell running as your user.

You can see the listening part right now on the home server:

sudo ss -tlnp | grep ':22'
# LISTEN 0  128  0.0.0.0:22  0.0.0.0:*  users:(("sshd",pid=812,fd=3))

That 0.0.0.0:22 means “port 22 on every network interface this machine has”: the home Wi-Fi or Ethernet, and the Tailscale one. Keep that in mind; it matters when we harden things. (On recent Ubuntu, you may see systemd instead of sshd holding the port. That is socket activationsocket activationsystemd listens on the port itself and only starts sshd when a connection actually arrives. Same effect for you; it changes how you restart things after editing the port.. Same idea, different owner of the phone.)

The SSH connection, step by step

Your laptop and the server are strangers on a network where anyone in the middle (the cafe Wi-Fi, your ISP) can read and change packets. In about half a dozen round trips, they need to:

1. agree on a secret key that nobody in the middle knows,
2. make sure the laptop is talking to the real server, not an impostor,
3. make sure the server is talking to you,
4. then carry your keystrokes, and maybe other traffic, inside the encryption.

Step 1: TCP. SSH rides on a normal TCP connection to port 22. Nothing special yet: just a reliable pipe of bytes.

Step 2: hello. Each side sends one line of plain text naming its version, like SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13. Then each sends a menu: “for key exchange I can do these, for encryption these, for signatures these”. They pick the first option on the laptop’s list that the server also supports. This is all readable by anyone on the path, which is why a scanner knows exactly which OpenSSH version you run the moment it connects.

Step 3: key exchange. This is the same trick as in the TLS lesson, and it is worth having in your hands. Think of mixing paint. You both start with the same public base colour. You each mix in a secret colour of your own and send the mixture across. Each of you then adds your own secret to the other’s mixture, and you both end up with the same final colour. Someone watching saw the base and both mixtures, but un-mixing paint is hard, so they can’t get the final colour. The real version is Diffie-HellmanDiffie-HellmanA method (1976) for two parties to agree on a shared secret over a public channel. Each sends a public value derived from a private random number; combining your private number with their public value gives both sides the same secret. Modern SSH uses an elliptic-curve version, often on Curve25519., with numbers instead of paint, done fresh for every connection. That freshness gives forward secrecyforward secrecyBecause the shared secret comes from random numbers thrown away after the handshake, recording today’s traffic and stealing the server’s keys next year still doesn’t let you decrypt today’s session..

From the shared secret both sides derive the actual encryption keys (for example for AES-GCM or ChaCha20-Poly1305), and from the next message on, everything is encrypted. The connection also gets a unique session IDsession IDA hash of everything both sides said during the key exchange. Unique to this connection, known to both ends, and used later as the thing your key signs..

But who did you just share a secret with? Diffie-Hellman by itself only proves that you share a secret with whoever answered. If someone sat in the middle, they could do one exchange with you and another with the server and pass messages along, reading everything. So in the same message as its paint mixture, the server sends its host keyhost keyThe server’s own long-term key pair, created when OpenSSH was installed. The private half lives in /etc/ssh/ssh_host_ed25519_key; it identifies the machine, not any user. and a signaturesignatureA short blob made with a private key over some data. Anyone with the matching public key can check it, but only the private key holder could have made it. Think of a wax seal that only one ring can make but anyone can recognise. over the session ID. An impostor can send a host key, but can’t produce a valid signature for the real server’s key without stealing its private key.

That moves the question to: how does your laptop know which host key is the real one?

Trust on first use, and the scary warning

It doesn’t, the first time. That’s the yes/no prompt:

The authenticity of host 'homeserver (100.101.102.103)' can't be established.
ED25519 key fingerprint is SHA256:wnZW5dld27zSuDcPtaTBgnZX34K/FMABJEVwO5Y3Ktg.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

The fingerprintfingerprintA short hash of a public key, easy to compare by eye. Two keys with the same fingerprint are, for all practical purposes, the same key. is the host key’s ID. Typing yes means “I trust that this is my server”, and ssh writes the host key into ~/.ssh/known_hosts. Every later connection checks the key the server presents against that line. This model is called TOFUTOFUTrust On First Use. You accept a key the first time without proof, then insist it never changes. Weak once, strong forever after..

The careful way to say yes is to compare the fingerprint with the one the server itself reports. At the server’s own keyboard (or over a connection you already trust):

ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
# 256 SHA256:wnZW5dld27zSuDcPtaTBgnZX34K/FMABJEVwO5Y3Ktg root@homeserver (ED25519)

Now the warning. I made it happen on my Mac by running a throwaway sshd, connecting once, then giving that sshd a new host key and connecting again:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
...
Offending ED25519 key in /Users/you/.ssh/known_hosts:1
Host key verification failed.

Proving it’s you: keys, not passwords

Once the server has proved who it is, you prove who you are. Two common ways.

Password. You send your password, encrypted, to the server, which checks it. Two problems. A password is guessable: bots on the internet try millions of common ones. And the server sees your actual password. If you ever said yes to an impostor’s host key, the impostor now has it.

Public key. You make a key pairkey pairTwo linked keys. The private key stays secret on your laptop and can make signatures. The public key can be handed to anyone and can only check signatures. Knowing the public key doesn’t help you compute the private one. once, on the laptop:

ssh-keygen -t ed25519 -C "gaurav@laptop"
# writes ~/.ssh/id_ed25519      (private: never leaves this machine)
#    and ~/.ssh/id_ed25519.pub  (public: one line, safe to share)

Ed25519Ed25519A modern signature scheme (Bernstein and colleagues, 2011) on an elliptic curve. Keys are tiny (32 bytes) and fast, and it avoids several ways older schemes could be implemented badly. The default choice for new SSH keys. keys are small. The public key file I generated on my Mac is 86 bytes, one line: ssh-ed25519 AAAAC3Nza...SWcgV gaurav@laptop. You append that line to ~/.ssh/authorized_keys on the server (ssh-copy-id home does this for you).

At login, the laptop says “I’d like to log in as gaurav with this public key”. The server looks in gaurav’s authorized_keys. If the key is there, the laptop signs a message containing the session ID with the private key, and the server checks the signature with the public key. Here is that exchange from my local test, using ssh -v:

debug1: Offering public key: .../k ED25519 SHA256:TfaJk5JM... explicit
debug1: Server accepts key: .../k ED25519 SHA256:TfaJk5JM... explicit
Authenticated to 127.0.0.1 ([127.0.0.1]:2222) using "publickey".

Why this beats a password:

• Nothing guessable. An Ed25519 private key has roughly 128 bits of security. A bot trying passwords is trying the wrong kind of thing entirely.
• Nothing reusable. The server never sees your private key, only a signature. And the signature covers this session’s ID, so even an impostor who captures it can’t replay it to log in somewhere else.
• Revocable per device. One line in authorized_keys per laptop. Lose a laptop, delete its line.

Passphrases and the agent. A private key file is just a file; anyone who copies it is you. So ssh-keygen offers to encrypt it with a passphrase. Typing that passphrase on every connection gets old, so there’s ssh-agentssh-agentA small background program that holds your decrypted private keys in memory. ssh asks the agent “please sign this” instead of reading the key file, so you unlock the key once per login session, not once per connection.. On macOS, AddKeysToAgent yes and UseKeychain yes in your ssh config make the Mac’s keychain remember the passphrase. One thing not to do casually: ssh -A (agent forwarding) lets the remote machine use your agent while you’re connected, so anyone who is root on that machine can log in as you elsewhere for as long as you stay connected. Use ProxyJump instead (below).

Channels: one connection, many pipes

After login, the encrypted connection is a container for channelchannelAn independent stream of data inside one SSH connection, with its own ID and its own flow control. Your shell is one channel; each forwarded port connection is another.s. Your interactive shell is channel 0: a “session” channel, with a request for a ptyptyPseudo-terminal. A fake terminal device the server creates so programs like vim and htop think a real keyboard and screen are attached. and a shell. ssh home uptime opens a session channel that runs one command instead. And the ones that make SSH a Swiss army knife: port forwards.

• -L (local forward). ssh -L 8888:localhost:8888 spark makes ssh listen on your laptop’s port 8888. Anything that connects there gets carried through the tunnel, and the Spark’s sshd opens a connection to localhost:8888 as seen from the Spark. The service on the far end can stay bound to the Spark’s loopback address, invisible to everything else, and you still reach it.
• -R (remote forward). The mirror image: ssh -R 9000:localhost:3000 home makes the server listen on its port 9000 and carries those connections back to port 3000 on your laptop. Useful for showing a dev server to a machine that can’t reach you.
• -D (dynamic forward). ssh -D 1080 home turns ssh into a SOCKS proxySOCKS proxyA simple proxy protocol: the client says “connect me to host X port Y”, and the proxy makes that connection and relays bytes. Browsers and curl can be told to send everything through one.. Point your browser at localhost:1080 and every site you open is fetched by the home server. You can open your router’s admin page from a cafe.

Files ride channels too. scp and sftp copy files over an SSH channel (since OpenSSH 9.0, scp actually speaks the SFTP protocol underneath). rsync -avz dir/ home:dir/ runs rsync over ssh and only sends the parts of files that changed, which is what you want for repeated deploys.

~/.ssh/config: stop typing flags

Everything above can live in a config file so ssh spark just works. A host block is a nickname plus settings:

Host home
  HostName homeserver          # the MagicDNS name (or 100.x address)
  User gaurav

Two facts about this file that bite people. First, for each setting, the first value found wins, reading top to bottom. So specific Host blocks go first and a catch-all Host * goes last. Second, ProxyJump (or -J) lets you hop through one machine to reach another: ProxyJump home on a spark block means “SSH to home first, then from there open a connection to the Spark’s port 22 and run a second, separate SSH session through it”. The second session is encrypted end to end from your laptop to the Spark, so the middle machine only relays encrypted bytes. That is why it’s safer than agent forwarding.

You can always ask ssh what it will actually do for a host: ssh -G spark prints the final, merged settings.

Locking the door

An SSH server is a program running as root that talks to anyone who connects. Two recent reminders of why that matters. In July 2024, “regreSSHion” (CVE-2024-6387) was a bug in OpenSSH’s own sshd on Linux that let an unauthenticated attacker run code as root, with no login needed, on versions from 8.5p1 to 9.7p1. And in March 2024, a backdoor was found planted in the xz compression library, designed to give its author a way into sshd on some Linux distributions; it was caught by luck shortly before it reached stable releases. Both were patched fast. Both were dangerous mainly to servers whose port 22 was open to the whole internet.

So hardening has two halves: shrink what sshd will accept, and shrink who can reach sshd at all.

What sshd accepts lives in /etc/ssh/sshd_config, and on Ubuntu, in drop-in files under /etc/ssh/sshd_config.d/ that are read first. Same rule as the client: first value wins. Some cloud and installer images ship a 50-cloud-init.conf that says PasswordAuthentication yes, which silently beats anything you write lower down in the main file. The only trustworthy view is the one sshd prints itself: sudo sshd -T.

The settings that matter:

Who can reach sshd is three tools:

• ufwufw“Uncomplicated Firewall”, Ubuntu’s friendly front end to the Linux packet filter. You write rules like “allow anything that arrives on interface tailscale0” and it drops the rest. lets you say “accept SSH only from the tailnet and the home LAN”.
• fail2banfail2banA daemon that watches login logs and, when one address fails too many times, adds a firewall rule banning it for a while. stops a single address from hammering. Recent OpenSSH (9.8+) has a smaller built-in version of this, PerSourcePenalties. On a server only reachable over Tailscale, fail2ban matters much less, because the bots can’t reach it.
• unattended-upgradesunattended-upgradesAn Ubuntu/Debian package that installs security updates automatically every day. Cheap insurance against the next regreSSHion. means the next sshd bug is patched before you read about it.

Why the cafe can’t see your house

Now the networking half. Why couldn’t you reach the server from outside before Tailscale?

Your ISP gives your house one public IPv4 addressIPv4 addressA 32-bit number like 203.0.113.9 that routers use to deliver packets. There are only about 4.3 billion, far fewer than devices, which is the root of everything in this section.. Your laptop, phone, TV and server all share it. Inside the house, the router hands out private addressprivate addressAddresses from ranges reserved for local networks (10.x, 172.16-31.x, 192.168.x). Every home reuses the same ones, so they mean nothing on the public internet and routers there won’t deliver to them.es like 192.168.1.50. The router does NATNATNetwork Address Translation. The router rewrites outgoing packets to come from its one public address and a port it picks, remembers that mapping, and rewrites replies back. It lets many devices share one public IP..

When the server opens a connection outward, the router rewrites it and keeps a note, so replies find their way back. When your laptop at the cafe sends a packet to 203.0.113.9:22, the router has no note for it. Which of the five devices in the house is it for? It doesn’t know, so it drops the packet. That’s not a firewall rule you set; it’s simply how NAT works, and it’s accidentally one of the best security features a home network has.

It can be worse. Many ISPs, especially mobile and some fiber providers, put your router behind another NAT of their own, called CGNATCGNATCarrier-Grade NAT. The ISP shares one public address between many customers, so your router’s “public” WAN address is itself a private one, usually in 100.64.0.0/10. You can’t port-forward through a NAT you don’t control.. If the WAN address on your router’s status page differs from what curl ifconfig.me reports, you’re behind one.

The old fix: port forwarding, and why it’s risky

The classic fix is a port forwardport forwardA rule you add on the router: “any packet arriving at my public address on port X, send to this LAN device on port Y”. It turns the router’s accidental protection off for that port.: “anything arriving at 203.0.113.9:22 goes to 192.168.1.50:22”. Now the cafe laptop can connect.

So can everyone else. Bots continuously scan every IPv4 address on common ports; a fresh server with port 22 open typically logs its first password guesses within minutes to hours (measure it yourself if you dare: journalctl -u ssh | grep -c 'Invalid user'). They read your exact OpenSSH version from the hello line in step 2 and know which bugs it has. Every hardening step in the last section suddenly matters a great deal more, and it does nothing against an sshd bug that strikes before login, like regreSSHion. Port forwarding also doesn’t work at all behind CGNAT, and breaks when your ISP changes your public address.

WireGuard: a tunnel made of public keys

Tailscale is built on WireGuardWireGuardA VPN protocol (Donenfeld, 2017) that sends IP packets, encrypted, inside UDP packets. Each device is identified by a Curve25519 public key. It’s small enough to audit and in the Linux kernel since 5.6 (2020).. The idea fits in a paragraph. Each device has a key pair. Each device has a list of peers: “this public key, reachable at this IP:port, allowed to use these inside addresses”. When your laptop wants to send a packet to the server’s inside address, WireGuard encrypts the whole IP packet with keys derived from both sides’ key pairs, puts it inside a UDPUDPThe bare-bones transport: send a packet to an address and port, with no connection, no retries and no ordering. WireGuard wants exactly that and does its own reliability-free encryption on top. packet, and sends it to the peer’s real address. The peer decrypts it and hands the inner packet to its kernel as if it had arrived on a network card.

That network card is a virtual one: tailscale0 on Linux, a utun device on macOS. To every program on the machine, including sshd, it’s just another interface with an address. This is why sshd needed no changes to work over Tailscale. It’s listening on 0.0.0.0:22, which includes the Tailscale interface.

Three properties worth knowing:

• Silent to strangers. A WireGuard device doesn’t respond to anything that isn’t correctly encrypted for it by a known public key. A scanner gets no reply at all, not even “go away”.
• Small overhead. Each packet gains a 16-byte header and a 16-byte authentication tag, and the inner packet is padded to a multiple of 16 bytes. Plus the outer IP and UDP headers (28 bytes on IPv4). You’ll measure this.
• Cheap handshakes. Keys are re-agreed every couple of minutes with a one-round-trip handshake, so a stolen session key is useful only briefly.

What WireGuard doesn’t do is the hard part of your setup: it needs to know the peer’s public key and real address in advance, and it needs packets to actually arrive. Both devices are behind NAT. Neither knows the other’s key, and neither router will let the other in.

Tailscale: a phone book plus a lock-picking kit

Tailscale splits into two planes.

The control planecontrol planeThe part of a system that decides and configures (who exists, who may talk to whom, which keys) as opposed to the data plane, which carries the actual traffic. is Tailscale’s coordination servercoordination serverTailscale’s central service. Devices log in to it, upload their public keys and their candidate addresses, and download the list of their peers. It never sees private keys and never carries your traffic.. When tailscaled (the daemon on each device) starts, it makes a WireGuard key pair, keeps the private half on the device, logs in with your Google/GitHub identity, and uploads its public key. The control server assigns it a stable address like 100.101.102.103 and sends back a map: every peer you’re allowed to talk to, with their public keys, addresses and candidate endpoints. Every device keeps a connection open to the control server, so when something changes, the new map is pushed out.

The data planedata planeThe part that carries the actual packets. In Tailscale, that’s WireGuard between your devices, directly or via a relay. is WireGuard, device to device. Your SSH traffic goes from laptop to server, not through the control server. That’s the answer to “can Tailscale read it?”: the control server only ever had public keys, and the traffic is encrypted with private keys it never saw. (What you are trusting Tailscale with is the phone book. A malicious control server could hand your laptop a fake peer key. Tailscale has a feature called Tailnet Lock for people who want to remove that trust; check its docs.)

That leaves getting through two NATs. Here’s the trick.

Hole punching

Remember the NAT rule: a packet gets in only if it matches a note the router wrote when something inside sent a packet out. So: get both sides to send a packet out to each other at about the same time. Each router writes a note “I sent to that address, replies are welcome”. Each side’s packet then arrives at the other router looking like a reply.

For that, each side needs to know its own public address and port, which it can’t see from inside. It asks a STUNSTUNSession Traversal Utilities for NAT. A server that answers one question: “from where I sit, what address and port did your packet come from?” That tells a device what its NAT turned it into. server. Tailscale’s relay servers double as STUN servers. Step through it:

The “hard” case is a NAT that picks a new public port for every destination (sometimes called a symmetric NAT, common on corporate and some carrier networks). What STUN reported is the port it uses towards STUN, which is useless for anyone else. Tailscale has extra tricks for when only one side is hard (David Anderson’s “How NAT traversal works”, in Go Deeper, is the classic explanation), but when both sides are hard, a direct path may be impossible.

DERP: the relay when punching fails

When no direct path works, Tailscale falls back to a DERPDERPDesignated Encrypted Relay for Packets. Tailscale-run servers around the world that forward already-encrypted WireGuard packets between devices, over HTTPS (TCP 443) so they work even on networks that block UDP. They route by public key and can’t decrypt anything. server. Each device keeps a connection to its nearest DERP region. The laptop wraps its WireGuard packet with “deliver to public key X” and sends it to DERP; DERP passes it down the server’s open connection. Both connections are outgoing from each house’s point of view, so no NAT ever has to accept anything unexpected.

DERP can see which device keys talk to which, when, and how much. It can’t see what’s inside: the WireGuard encryption is end to end, and DERP never had the private keys. The cost is a detour. If you and the server are in the same city and the relay is a few hundred miles away, every round trip now travels to the relay and back. And relays have limited bandwidth per connection, so big rsyncs are slower.

Tailscale actually starts every new connection via DERP, because it works immediately, while it tries hole punching in the background. As soon as a direct path answers, it quietly switches. That’s why the first tailscale ping to a peer often says via DERP and the next one says via 203.0.113.9:41641.

Now put it all together. Toggle between the three ways of reaching the server and look at what each hop on the path can see:

The rest of the Tailscale toolkit

• 100.x addresses. Tailscale hands out addresses from 100.64.0.0/10, the same range ISPs use for CGNAT. That sounds odd, but it’s chosen precisely because it’s unlikely to collide with your home LAN (192.168.x) or office network (10.x). Each device also gets an IPv6 address in fd7a:115c:a1e0::/48.
• MagicDNSMagicDNSTailscale’s built-in name server. Each device on your tailnet gets a name (homeserver, and a full name like homeserver.tail1234.ts.net), and tailscaled answers lookups for them at 100.100.100.100.. It’s why ssh homeserver works without you writing any IP down.
• ACLs. By default every device on your tailnet can reach every other one on every port. The policy file in the admin console (newer docs call the rules “grants”) lets you say things like “my phone can reach the home server on 443 only”. Worth setting once you share the tailnet with anyone, or put an untrusted device on it.
• Tailscale SSH. sudo tailscale set --ssh on the server makes tailscaled itself answer port 22 on the Tailscale address and log you in based on your tailnet identity and an ssh rule in the policy file. No authorized_keys to manage. The trade: your login now depends on your Tailscale (and Google/GitHub) account security, instead of a key file on your laptop. Your normal sshd keeps working on the LAN. Check the docs for exact flags; this area changes.
• Key expiry. Device keys expire after 180 days by default, and the device drops off the tailnet until you log in again. For a headless server that’s a nasty surprise; disable key expiry for it in the admin console.

Build It

You’ll do this on the home server (Ubuntu or Debian assumed; adjust package commands otherwise), your Mac, and the Spark. Before touching sshd or the firewall, open a second SSH session to the server and leave it open. An existing session survives a bad config; a new one might not. If you can, also have a keyboard and screen on the server, or at least be on the home LAN.

ssh -V; dpkg -s openssh-server | grep '^Version'   # OpenSSH version
systemctl status ssh --no-pager | head -5   # service (or ssh.socket on newer Ubuntu)
sudo ss -tlnp | grep ':22'                  # who listens, on which addresses
ls /etc/ssh/sshd_config.d/                  # drop-in files read before the main config
sudo sshd -T | grep -Ei '^(passwordauthentication|kbdinteractiveauthentication|permitrootlogin|pubkeyauthentication|maxauthtries|allowusers|x11forwarding)'
ip -br addr                                 # interfaces: find the physical one and tailscale0
tailscale ip -4

ls -l ~/.ssh/
ssh -G home 2>/dev/null | grep -E '^(hostname|user|port|identityfile)'

ssh -o PasswordAuthentication=no -o KbdInteractiveAuthentication=no \
    -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 gaurav@homeserver true && echo KEY OK

# /etc/ssh/sshd_config.d/10-hardening.conf
# (your settings here, one per line, each with a comment)

sudo sshd -t && echo SYNTAX OK               # test before restarting
sudo systemctl reload ssh                    # existing sessions stay up
sudo sshd -T | grep -Ei 'passwordauth|kbdinteractive|permitroot|allowusers|maxauthtries'

ssh -o PubkeyAuthentication=no home true     # should fail: Permission denied (publickey)
ssh home true && echo STILL IN               # should succeed

sudo apt install fail2ban unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades     # answer Yes
cat /etc/apt/apt.conf.d/20auto-upgrades             # both lines should be "1"
sudo fail2ban-client status sshd

Host home
  HostName <tailscale name of the home server>
  User <you>

Host spark
  HostName <tailscale name of the Spark, or its LAN IP>
  User <you>
  # if the Spark is NOT on your tailnet but is on the home LAN:
  # ProxyJump home

Host *
  IgnoreUnknown UseKeychain
  UseKeychain yes
  AddKeysToAgent yes
  IdentityFile ~/.ssh/id_ed25519
  ServerAliveInterval 30
  ControlMaster auto
  ControlPath ~/.ssh/cm-%C
  ControlPersist 10m

jupyter lab --no-browser --ip 127.0.0.1 --port 8888

lsof -nP -iTCP:8888 -sTCP:LISTEN     # on the Mac: who is listening? (it should be ssh)
curl -sI http://<spark-100.x-address>:8888 | head -1   # direct: should fail, Jupyter is loopback-only

# Mac
tailscale netcheck
tailscale status
tailscale ping homeserver        # repeat a few times

# server, window 1: inside the tunnel
sudo tcpdump -ni tailscale0 -c 20 tcp port 22

# server, window 2: the physical wire (use your interface name from t0)
sudo tcpdump -ni <eth0> -c 20 udp and not port 53

npm run build                                       # in the workbench repo, writes dist/
rsync -avz --delete dist/ home:~/workbench-site/

python3 -m http.server 4321 --directory ~/workbench-site --bind "$(tailscale ip -4)"

Measure

Reference points from my machines, and what yours should look like:

netcheck. On my Mac on home Wi-Fi: UDP: true, MappingVariesByDestIP: false, nearest DERP Seattle at 8.3 ms, San Francisco about 32 ms. Your list will be different; what matters is the nearest one and whether mapping varies. Try it again on a phone hotspot: mobile carriers are where you’re most likely to see CGNAT and true.

tailscale ping. Expect the first pong via DERP(<region>) with a latency of roughly (you → relay) + (relay → server), then within a few pings a pong via <ip>:<port> at roughly your real network round trip. On the same LAN that’s a millisecond or two; across a city, often 5-20 ms. If it never upgrades, look at netcheck on both ends.

SSH connect time. On my Mac against a local sshd over loopback (zero network delay), a fresh ssh ... true took a median of 42 ms over 10 runs, and a multiplexed one 10 ms. That 42 ms is process startup plus the crypto. Over a real network, a fresh connection adds several round trips on top: TCP, the hello and menus, the key exchange, the auth request and the signature, opening the channel. Count about 6-8 round trips (approximate; some messages overlap). So over a 20 ms path, expect a fresh connection around 42 + 7 × 20 ≈ 180 ms, while the multiplexed one stays near one round trip plus the 10 ms. Through DERP at, say, 40 ms round trip, fresh connections cross 300 ms. This is exactly the latency ladder: a handshake is a stack of round trips, and round trips are set by distance.

Packet length. WireGuard’s data packet is a 16-byte header, the inner IP packet padded up to a multiple of 16, and a 16-byte tag. tcpdump’s UDP, length N is that whole thing. So for an inner packet of length L on tailscale0, predict N = 32 + (L rounded up to a multiple of 16) on the physical interface. Check it on three packets. The outer IPv4 and UDP headers add another 28 bytes on the actual wire, which is why Tailscale uses a smaller MTU (1280) on its interface: so the bigger outer packet still fits on normal networks.

Break It

1. Lock yourself out (safely). In your second, still-open session, add AllowUsers nobody to your drop-in, sshd -t, reload, and try a new connection. Denied, even with the right key. Your existing session is still alive, because each connection has its own sshd child that already finished auth. Remove the line and reload. This is the reason for “always keep a second session open”, and the reason sshd -t exists.

2. Be overridden. Create /etc/ssh/sshd_config.d/05-oops.conf containing PasswordAuthentication yes, reload, and check sudo sshd -T | grep passwordauth. Your 10-hardening.conf says no, but 05 sorts first and wins. Now try a password login from the Mac (ssh -o PubkeyAuthentication=no home): it offers a password prompt. Delete the file and reload. This is how a single cloud-init file quietly undoes hardening on real servers.

3. Force the relay. On the server, temporarily block outgoing UDP except DNS, so hole punching can’t work: sudo iptables -I OUTPUT -p udp ! --dport 53 -j DROP. (Do this from the LAN or with a console handy; your SSH session over Tailscale will stall for a few seconds while it fails over.) Watch tailscale ping homeserver from the Mac: direct pongs stop, and it falls back to via DERP. Time rsync of a 500 MB file both ways: direct, then relayed. Then remove the rule with sudo iptables -D OUTPUT -p udp ! --dport 53 -j DROP. You’ve just seen why DERP is a fallback, not the plan.

4. Change the host key. On the server, move the Ed25519 host key aside and generate a new one (sudo mv /etc/ssh/ssh_host_ed25519_key{,.bak}, same for .pub, then sudo ssh-keygen -A and reload). Connect from the Mac and read the full warning. Now do the right thing: compare the new fingerprint on the server, remove the old line with ssh-keygen -R homeserver (and the 100.x address), reconnect, accept. Or restore the backup and watch the warning disappear. Either way, you’ve now seen the scary message for an innocent reason, and know exactly how to tell.

Explain It Back

Answer out loud, in plain words, as if in an interview. Then check.

Go Deeper

• Tailscale, “How Tailscale works” by Avery Pennarun (tailscale.com/blog/how-tailscale-works). The whole architecture, control plane vs data plane, in one long, friendly post.
• Tailscale, “How NAT traversal works” by David Anderson (tailscale.com/blog/how-nat-traversal-works). The best explanation of hole punching anywhere, including hard NATs, birthday-paradox port guessing and why DERP exists.
• Jason A. Donenfeld, “WireGuard: Next Generation Kernel Network Tunnel”, NDSS 2017 (the whitepaper is at wireguard.com/papers/wireguard.pdf). Short, readable, and explains every packet format you saw in tcpdump.
• RFC 4253, “The Secure Shell (SSH) Transport Layer Protocol” and RFC 4252, “Authentication Protocol” (Ylönen and Lonvick, 2006). Read the key exchange section of 4253 and the public key method in 4252; you’ll recognise every step of Figure 001.
• OpenSSH manual pages: ssh(1), ssh_config(5), sshd_config(5) (also at man.openbsd.org). The authoritative list of every option, including the “first obtained value” rule.
• Michael W. Lucas, SSH Mastery (2nd ed., 2018). A short book covering keys, agents, forwarding, jump hosts and certificates with practical sysadmin taste.
• Qualys Security Advisory, “regreSSHion: RCE in OpenSSH’s server” (CVE-2024-6387), July 2024. A real, readable account of how a pre-login sshd bug works, and why exposure matters more than config.
• Tailscale docs: “Tailscale SSH”, “Tailscale Serve”, “Tailscale Funnel”, and “Using UFW to lock down an Ubuntu server” on tailscale.com/kb. Check these for current command syntax.

Connections

This lesson leans on How slow is slow? (a handshake is a stack of round trips) and pairs with TLS, which uses the same Diffie-Hellman-plus-signature idea with certificates instead of known_hosts. Packets and IP goes deeper on addresses, routing and NAT, and TCP vs UDP explains why a tunnel protocol like WireGuard wants UDP underneath (tunnelling TCP inside TCP makes two layers of retries fight each other).

It also sets up everything you’ll do on the Spark from here on: the -L forward is how you’ll reach the inference server in Serving engines and the dashboards in Observability without opening anything to the network.