Follow-up audit — six things we learned after migrating to NFSv4.2

TL;DR

Read on for what each one looked like in the wild and what to do about it.

1. The Proxmox storage layer is a silent NFSv3 survivor

The migration touched every host's /etc/fstab, every NFS export on the server, every modprobe.d file. What it didn't touch — because it didn't have to, in theory — was Proxmox's own storage layer, which auto-mounts shares from /etc/pve/storage.cfg independently of fstab.

A week later, one node in the cluster was still mounting the share as vers=3,soft,timeo=30,retrans=2. Not because anyone undid the migration on that node, but because the storage auto-mount had been generated from the storage config at some earlier point and never refreshed.

findmnt -t nfs,nfs4 --raw 2>/dev/null

Look at the options column. If you see vers=3 mixed with vers=4.2 or soft mixed with hard, you have drift. The fix is either to update /etc/pve/storage.cfg and force a remount via pvesm, or — if you mount NFS directly — verify /etc/fstab against the running mount.

The migration playbook needs an explicit "audit pvesm status on every node" step. We didn't have that step. We do now.

2. nfs4_disable_idmapping=1 makes Domain mismatches invisible

In the original recipe we set nfs4_disable_idmapping=1 on every node, persisted via /etc/modprobe.d/. That correctly forces numeric UID passthrough for sec=sys mounts and avoids the nobody:nogroup failure mode entirely.

It also makes /etc/idmapd.conf's Domain setting irrelevant. So irrelevant that three of five nodes still had it commented out — the default, which falls back to the host's FQDN domain.

This is fine right now. It will be a debugging nightmare on the day someone adds a single Kerberos mount to that cluster. Kerberos forces the idmap path back on for that mount, and the moment the client and server disagree on the Domain, all file ownership appears as nobody:nogroup. The error message tells you nothing about what's wrong.

[General]
Domain = <your-cluster-domain>

Set the same value across the cluster. Put it in your config-management or your runbook. It costs nothing and removes a future class of incidents.

3. ZFS sharenfs and manual /etc/exports.d/ quietly fight

The NFS server's exports were a mix:

• /etc/exports and a hand-written /etc/exports.d/<name>.exports for the migrated share
• /etc/exports.d/zfs.exports (auto-generated) carrying the warning # !!! DO NOT EDIT THIS FILE MANUALLY !!!

The auto-generated file is rewritten any time zfs set sharenfs=... runs on any dataset, on every pool import, and on certain other ZFS events. If a manual edit ever lands in that file (despite the warning), the next ZFS event silently overwrites it.

Worse: if anyone ever sets sharenfs on a dataset whose path also appears in /etc/exports, the two files now share a path with potentially different export options. exportfs -ra reads them in alphabetical order, so the last one wins. Which one depends on the file names.

A quick auditor:

# Server side — find ZFS datasets with sharenfs active
zfs get sharenfs | grep -v ' off '

# And manual exports
ls /etc/exports.d/ | grep -v zfs.exports

# Cross-reference paths to detect overlap

If a path appears in both, pick a side and stick with it.

4. SUNRPC tcp_slot_table_entries reading 2 on modern kernels is not a bug

This one is just an anti-rabbit-hole note. On older kernels, /proc/sys/sunrpc/tcp_slot_table_entries was a static tunable — operators would set it to 128 or so for high-throughput workloads. On kernel 5.3+ the slot tables are per-connection and dynamically sized; the sysctl now reports the minimum number of slots, not the configured count. The actual ceiling is tcp_max_slot_table_entries (default 65536).

Auditors who came up under older kernel docs will see 2 / 65536 and immediately try to "fix" it. There's nothing to fix. The Linux NFS client allocates as many slots as it needs up to the max, and 2 is just where it idles.

cat /proc/sys/sunrpc/tcp_slot_table_entries
cat /proc/sys/sunrpc/tcp_max_slot_table_entries
# Run a heavy NFS load
# Check again — you'll see the same numbers, because the runtime allocation
# is per-connection and not exposed to /proc.

Don't add this to your "things to fix" list. Add it to your "things future-me would otherwise re-discover" runbook.

5. The default 90-second NFSv4 grace period is conservative for fast-restart clusters

After a planned NFS server restart, NFSv4 enters a grace period where clients can reclaim their locks and open files. The default in mainline Linux is 90 seconds. During that window, all NFSv4 opens and locks block on the server.

For applications that actually hold long-lived locks across NFS — databases, certain file-servers — 90 seconds is appropriate, even short. For the workloads we typically run on top of these shares — media streaming, container artifacts, web app file storage — there are no clients that need 90 seconds to reconnect.

The result is that planned NFS server reboots cause longer client-visible stalls than necessary.

# /etc/nfs.conf on the server
[nfsd]
grace-time=45
lease-time=45

Or live, transient:

echo 45 > /proc/fs/nfsd/nfsv4gracetime
echo 45 > /proc/fs/nfsd/nfsv4leasetime

Picking 45 seconds approximately halves the client-block window with no loss of correctness for our workload mix. Pick what's right for yours.

6. mountproto=udp is the NFSv3 default and can fail silently

For the handful of NFSv3 mounts we intentionally retained (one is a high-bandwidth storage path that didn't benefit from migrating), the fstab entries inherited mountproto=udp from the kernel default. That's the protocol used for the mount handshake and portmap lookup, separate from proto= which controls actual data transport.

If a firewall rule between the client and server filters UDP — common on tightly-controlled storage VLANs — the mount fails at boot with a generic timeout. There's no log line saying "UDP filtered." There's just a stuck mount.

server:/path /local nfs vers=3,sec=sys,hard,mountproto=tcp,_netdev 0 0

This costs nothing and removes one silent-fail mode.

What this audit really cost

About forty minutes for the agent doing the read-only sweep across the cluster. About fifteen minutes per finding to sit with what was actually wrong vs what looked wrong. The biggest single insight wasn't any of the six findings — it was that the original migration touched 80% of the surface area and missed 20%, and the missing 20% wasn't visible from any of the obvious places to check.

Storage auto-mounts. Idmapd config. ZFS exports. Kernel tunables that changed semantics. Default values that are right for the kernel's reference workload but wrong for ours. None of these surfaces are part of the standard NFS migration playbook because the standard playbook doesn't know about them.

This is the case for follow-up audits as a category of work. Day-one ship is "the new thing works." Day-seven audit is "the old thing's shadows are still everywhere." Both are necessary.

What's next

The fixes above are simple enough to put in a small follow-up commit — they collectively touch six files, none of them code. The skill in our Claude Code plugin marketplace has been updated to include the audit one-liners; the next agent who triggers the skill on a new cluster gets these checks for free.

If your own cluster has any of the six findings above, the fix in each section is the entire fix. If you find a seventh, please tell us — [email protected] reaches our developer group, and the post will get an update with attribution.