Longhorn: distributed storage for my homelab

Storage is the hardest part of running a Kubernetes homelab. Stateless apps are easy: they crash, they restart, they reschedule. But databases, media libraries, and config files need somewhere persistent to live. And that somewhere needs to survive node failures, reboots, and the general chaos of running hardware in your living room.

I use Longhorn for this. Here's everything I've learned after running it in production for several months, including the time a node went offline for nine days and I had to recover.

What is Longhorn?

Longhorn is a distributed block storage system for Kubernetes, originally built by Rancher. It's now a CNCF incubating project.

It takes the local disks on your Kubernetes nodes and turns them into replicated block devices that pods consume as standard PVCs. Replication is configurable per volume. You get snapshots and incremental backups, thin provisioning so volumes only consume the space they actually use, a web UI for managing volumes, replicas, and backups, and CSI-compliance so it plugs into the standard Kubernetes storage primitives.

No special hardware, no SAN, no NFS. Just the disks already attached to your nodes.

My setup

I run three nodes, each contributing storage to the cluster:

Longhorn runs a manager pod on each node, which handles scheduling replicas, monitoring disk health, and coordinating volume attachments.

GitOps configuration

Everything is managed declaratively via Flux. My Longhorn node configs live in cluster/storage/ as Node custom resources:

# cluster/storage/node-amd-w-minipc.yaml
apiVersion: longhorn.io/v1beta2
kind: Node
metadata:
  name: amd-w-minipc
  namespace: longhorn-system
spec:
  allowScheduling: true
  disks:
    default-disk-nvme:
      allowScheduling: true
      diskType: filesystem
      evictionRequested: false
      path: /var/lib/longhorn
      storageReserved: 107374182400  # 100GB reserved
      tags:
        - nvme
        - fast

This is declarative infrastructure — Flux applies it, Longhorn enforces it. If someone manually tweaks a disk config in the UI, the next sync reverts it. Intentional, but worth knowing if you're debugging and wondering why your changes aren't sticking.

Disk tags

One of Longhorn's best features is disk tags — they let you target specific storage tiers per volume. I use:

Then in your StorageClass, you specify which tags a volume's replicas must land on:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-fast
parameters:
  numberOfReplicas: "2"
  diskSelector: "fast"

Postgres databases land on fast NVMe disks. Large media archives land on cheaper spinning disks. All by picking the right StorageClass at deploy time.

How replication works

When a pod writes to a Longhorn volume:

1. The volume is attached to one node (the "frontend")
2. Longhorn's engine process handles all I/O on that node
3. Writes are replicated synchronously to replica processes on other nodes
4. Each replica is a full, independent copy of the volume data

The default replication factor is 3 for critical data, 2 for less critical. You can set this per StorageClass or override per volume in the UI.

If a node goes down, volumes with surviving replicas stay healthy and keep serving. Volumes where the only replica was on the dead node become faulted — detached, inaccessible, waiting for that node to come back.

This is where things get interesting.

When a node disappears

One of my worker nodes (intel-w-acemagic) went offline unexpectedly. The kubelet stopped posting status, Kubernetes marked it NotReady, and within minutes the damage was visible:

• 12 volumes degraded: lost one replica, still serving from the surviving copy
• 5 volumes faulted: only replica was on the dead node, completely inaccessible
• 20+ pods stuck Terminating, waiting for the node to acknowledge their deletion
• A handful of pods in ContainerCreating, trying to attach volumes that couldn't

The degraded volumes kept running. Redundancy doing its job. The faulted volumes were the problem — pods depending on them sat in ContainerCreating indefinitely.

NAME               STATE      ROBUSTNESS   NODE
pvc-1a589bb4-...   detached   faulted                    ← inaccessible
pvc-365487d4-...   attached   degraded     amd-w-minipc  ← serving, but exposed
pvc-55510f17-...   attached   healthy      amd-w-minipc  ← fully replicated

The node had been offline for nine days by the time I noticed. Nine days without redundancy on half my volumes, and I didn't know.

Once the node came back online, Longhorn recovered automatically:

1. Faulted volumes moved from faulted → unknown → rebuilding
2. Missing replicas rebuilt in the background
3. Stuck pods came up once volumes were available again

The data was intact. Nothing lost. That's the value of replication — even with a single surviving replica on a different node, your data is safe.

Lessons from the outage

Single-replica volumes are a trap. I had five volumes with only one copy of their data. Any workload that matters should have at least two replicas — one replica means one hardware failure equals data inaccessible.

The real risk isn't a volume going faulted. It's a degraded volume (one replica surviving) staying that way long enough for the second node to have trouble. Prometheus alerts on degraded volumes. Don't wait nine days to find out.

Terminating pods block volume reattachment. When a node dies, its pods get stuck Terminating, and StatefulSets with Longhorn volumes need the old pod fully gone before Longhorn can safely attach elsewhere. Force-deleting (--force --grace-period=0) works if you understand what you're doing.

Removing a disk safely

When I physically removed an extra SSD from intel-w-acemagic, Longhorn's admission webhook blocked the config change:

admission webhook "validator.longhorn.io" denied the request:
Delete Disk on node default-disk-nvme error:
Please disable the disk and remove all replicas first

The correct sequence:

# 1. Disable scheduling and request eviction
kubectl patch node.longhorn.io intel-w-acemagic -n longhorn-system \
  --type=merge \
  -p '{"spec":{"disks":{"old-disk":{"allowScheduling":false,"evictionRequested":true}}}}'
 
# 2. Wait for replicas to migrate off the disk
# 3. Then remove the disk from your node manifest and commit

Skipping step 1 and going straight to removing it from the GitOps manifest will block Flux reconciliation until you patch it manually.

Backup strategy

Longhorn supports recurring snapshot and backup jobs via CRDs:

apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: frequent-snapshot
  namespace: longhorn-system
spec:
  cron: "0 */6 * * *"   # Every 6 hours
  task: snapshot
  groups:
    - default
  retain: 8
  concurrency: 2

Snapshots every 6 hours, retaining the last 8. This protects against accidental deletion and allows rollbacks, but snapshots are local — they live on the same nodes as the data itself.

For true disaster recovery, Longhorn can push backups to S3-compatible storage or NFS. That's next on my list.

What works well

The UI is genuinely good. You see replica placement at a glance, trigger manual snapshots, and watch rebuild progress in real time. Most Kubernetes storage tooling is miserable to operate. Longhorn isn't.

Scaling is zero-config: add a new node, configure its Longhorn Node resource, and replicas start landing there automatically. Online volume expansion just works — bump the size in the manifest, Flux applies it, Longhorn resizes without a pod restart for most workloads. And it pairs well with Talos. The immutable OS model fits Longhorn's declarative approach. Disk management is config, not commands.

What to watch out for

Config drift is silent. If your Longhorn node object gets out of sync with your GitOps manifest (auto-discovered disks are the usual culprit), Flux will overwrite the live state on the next sync. Keep manifests current.

Rebuild I/O is real. When a node rejoins after a long absence, Longhorn rebuilds all missing replicas simultaneously, which saturates spinning disks. NVMe handles it much better.

Storage overhead is real too. Longhorn uses disk space for metadata and snapshot data beyond what your volumes actually contain. Budget for it on smaller disks.

The bigger picture

My rule for the homelab: data-recovery over reliability. Services can go down. That's fine. What matters is that the data underneath survives.

Longhorn fits that. Replication means a hardware failure isn't a data loss. Snapshots mean a mistake is recoverable. The distributed architecture means I don't need a NAS or a SAN.

It's not enterprise storage. For a three-node homelab on mini PCs, surviving a nine-day node outage without losing a byte is enough proof it's the right call.