The 61-Byte File That Took My Server Off the Network

If you got here from a search

You can confirm this is your bug in about thirty seconds. All of these will be true:

• The machine boots normally and you can log in at the console.
• It has no network at all — no DHCP lease, no ping, no SSH, and it never appears in your router's client list.
• ip -br link shows eth0 where you expected enp3s0 (or any enp* / ens* name).
• The interface is DOWN, but the MAC address is correct — it's the right hardware under the wrong name.
• Your netplan or networkd config matches the interface by name.
• There is a file ending in .link in /etc/systemd/network/.
• /proc/cmdline does not contain net.ifnames=0 — that's the other cause of the same symptom, and it's not this one.

If that's your machine, the .link file is the culprit and the fix is three lines. The rest of this post is why.

The symptom that lies

The machine is called amber. It's the always-on box in my basement — a repurposed all-in-one that holds a ZFS pool, runs a few containers, and generally has the decency not to need attention.

I had just been physically at it, in the BIOS, hunting for the setting that makes it power itself back on after an outage. I found the setting, saved, and let it boot. It came up to a login prompt exactly the way it should.

And it was completely unreachable. Not slow, not flaky — absent. No SSH. No ping. An ARP sweep of the whole subnet from another machine came back without its MAC anywhere in the table. From the network's point of view, the machine did not exist.

This is a genuinely nasty failure shape, because every signal you get from the console is reassuring. The kernel booted. The filesystems mounted. Nothing failed. systemctl --failed was empty. The machine was alive, healthy, and mute.

The red herring

I had changed exactly one thing in the previous ten minutes, and it was a BIOS setting on the machine that broke. You do not need to be careless to follow that thread — you'd be careless not to. Recency is the most persuasive evidence there is, and one of the weakest.

The BIOS was innocent. The change I'd made was on the power-on menu and touched nothing about the network controller. But that's hindsight; at the time it was the obvious suspect, and it cost a real detour.

There was a second dead end worth mentioning, because it's the kind of thing that eats twenty minutes: the machine has a WiFi adapter, so "just bring up WiFi" looks like an easy fallback. It wasn't. I had deliberately disabled that radio months earlier — driver blacklisted, rfkill blocked, its netplan config renamed out of the way — precisely so it would stop flapping and confusing the routing table. The interface didn't exist to fall back to. A fallback you disabled on purpose is not a fallback.

The thing that actually cracked it was mundane. Log in at the console and look:

$ ip -br link
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0             DOWN           aa:bb:cc:dd:ee:ff <BROADCAST,MULTICAST>

eth0. Not enp3s0, which is what that NIC had been called for the entire life of the machine. Same MAC — the right card, under a name nothing in my configuration had ever heard of.

And that is the whole bug, because this is what netplan had been told to configure:

# /etc/netplan/60-ethernet.yaml
network:
  version: 2
  ethernets:
    enp3s0:
      dhcp4: true
      dhcp4-overrides:
        route-metric: 100

Netplan matches by name. There was no enp3s0, so it matched nothing, configured nothing, and requested no DHCP lease. It did not error. There is no failed unit for "the device you named never showed up." The interface simply sat there, down, forever.

What udev is actually doing

Predictable interface names — enp3s0 instead of eth0 — don't come from the kernel. The kernel hands out eth0, eth1 and so on in whatever order drivers happen to probe, which is exactly the non-determinism the scheme exists to fix. The rename happens in userspace, in udev, driven by .link files.

Here is the part that bit me, and it's the one sentence worth taking away:

udev applies only the first matching .link file, sorted by filename — and it does not merge the rest.

Predictable naming is not a built-in behaviour. It's a policy, and it's written in an ordinary config file that ships with systemd:

# /usr/lib/systemd/network/99-default.link
[Match]
OriginalName=*

[Link]
NamePolicy=keep kernel database onboard slot path
AlternativeNamesPolicy=database onboard slot path
MACAddressPolicy=persistent

It matches every interface, and it sorts to 99 so that anything you write wins over it. That's a deliberate, sensible design — it's how you're meant to override things.

Back in July I had armed this machine for Wake-on-LAN, so that if it ever went dark I could wake it from another box instead of walking downstairs. The documented way to persist that is a .link file, and I wrote the obvious one:

# /etc/systemd/network/50-wol.link   — 61 bytes, and wrong
[Match]
MACAddress=aa:bb:cc:dd:ee:ff

[Link]
WakeOnLan=magic

It does what it says. WoL was armed, and it stayed armed. But look at what else it does, given the rule above:

1. 50-wol.link sorts before 99-default.link.
2. It matches the NIC, by MAC.
3. First match wins, so 99-default.link never runs at all.
4. My file specifies no NamePolicy, so no naming policy is applied — not a different one, none.
5. With no policy, the interface keeps the name the kernel gave it: eth0.
6. Netplan is looking for enp3s0. Nothing matches. No address.

Nothing about that is a bug in systemd. Every step is documented behaviour, and "the highest-priority matching file wins outright" is a perfectly defensible rule. It's just that the failure is silent and the consequence lands somewhere that looks unrelated.

Five weeks of nothing

The detail I keep turning over is the delay.

I wrote that file on 21 July. Nothing broke. Wake-on-LAN worked, which was the thing I was testing, so I marked it done and moved on. The interface kept its name because the machine kept running — udev names interfaces at boot, and there was no boot.

The bomb sat there for five weeks. It went off on 28 August, three minutes after I changed something else entirely, on a completely different subsystem, standing in a basement thinking about power supplies.

If you change anything that only takes effect at boot — udev rules, .link files, netplan, fstab, initramfs, kernel parameters — on a machine that is awkward to reach physically, then reboot it while you are still standing next to it. Not later. Not when it's convenient. The reboot is not a formality after the change; the reboot is the test, and everything before it is a guess.

Diagnosing it without rebooting

Once you suspect naming, you don't have to reboot to confirm it — which matters, because at this point a reboot is exactly what you're afraid of. udev will tell you what it would do:

$ sudo udevadm test-builtin net_setup_link /sys/class/net/eth0
Using default interface naming scheme 'v255'.
eth0: MAC address on the device already matches policy "persistent".
eth0: Policy *path* yields "enp3s0".
ID_NET_NAME=enp3s0

That's a dry run. It reads your current .link files, applies the same logic udev applies at boot, and prints the name it would assign — without touching the running system. If it says enp3s0 and your interface is currently called eth0, you've confirmed both the diagnosis and the fix in one command.

It's also worth explicitly ruling out the impostor. The exact same symptom — eth0 where you expected enp3s0 — is produced by disabling predictable naming on the kernel command line, which is a much more commonly documented cause:

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.0-138-generic root=UUID=… ro

No net.ifnames=0, no biosdevname=0. Different cause, same symptom, and the fix below would have done nothing for it. Rule it out before you start editing files.

The fix

Carry the naming keys into your own file. You're overriding 99-default.link whether you meant to or not, so override it completely:

# /etc/systemd/network/50-wol.link
[Match]
MACAddress=aa:bb:cc:dd:ee:ff

[Link]
WakeOnLan=magic
NamePolicy=keep kernel database onboard slot path
AlternativeNamesPolicy=database onboard slot path
MACAddressPolicy=persistent

Three added lines, copied verbatim from the file you're shadowing. Re-run the dry run to confirm it now yields enp3s0, then reboot to prove it. Mine came back in sixty-two seconds with the right name, a DHCP lease on the address it had always had, and — worth checking, since it was the entire point of the file — Wake-on-LAN still armed:

$ ip -br a
enp3s0           UP             192.168.70.65/22 metric 100

$ sudo ethtool enp3s0 | grep Wake-on
	Wake-on: g

If you'd rather not have a naming policy scattered across files at all, the alternative is to stop matching on names: give netplan a match: block keyed on the MAC address, and the interface can be called anything. That's arguably the more robust configuration, and if you're building a machine from scratch it's what I'd choose.

I deliberately didn't do it here, and the reasoning is worth stating because it's a trap in its own right. You cannot have both a name match and a MAC match for one device — they conflict — so switching means replacing a configuration that had worked untouched for months with one that had never been tested, immediately before the reboot that was supposed to verify the actual fix. If that reboot had then failed, I'd have had two candidate causes and no way to tell them apart, on a machine I could only reach by walking down a flight of stairs. Fix the root cause; change one thing; harden later, deliberately, when a failure won't be ambiguous.

The lesson I didn't expect

The naming bug is the interesting mechanism, but it isn't the thing that actually changed how I think.

I had built what I believed was a remote-recovery story for this machine. Wake-on-LAN armed on the NIC, a script on another box that fires magic packets, an uptime monitor that calls that script automatically whenever the server goes dark. It's real, it works, and I'd tested it.

It was completely useless here. Wake-on-LAN is not a recovery path for a network configuration fault. It can turn a machine on. It cannot help a machine that is already on, booted cleanly, and has no address. Those are different failures and I had quietly filed them under one heading, which meant my recovery plan had a hole in it shaped exactly like the thing that happened.

That's the generalisable bit, and it isn't really about systemd. Every remote-recovery mechanism covers a specific failure class, and it's worth being precise about which one — because the comfort it provides is not similarly limited. Mine covered "the machine is off." The failure was "the machine is on and deaf." No amount of the first fixes the second, and I'd never had cause to notice the difference until I was standing in the basement at half past four on a Friday.