SeaweedFS: Distributed Storage with Commodity Equipment

Maintaining a home lab with a diverse fleet of servers is a constant balancing act, but the real challenge is finding a way to pool mismatched, leftover drives across separate machines without turning storage administration into a second full-time job. Over time, my server farm accumulated an assortment of mismatched disks—a fast NVMe SSD on one machine, a spinning mechanical HDD on another, and a tight OS drive elsewhere. Rather than letting this spare capacity go to waste or buying an expensive, dedicated NAS appliance, I set out to build a unified distributed shared storage layer across my fleet.

My goal was simple: the storage solution had to be low-maintenance, highly efficient with heterogeneous drives, actively developed, and crucially, not a project in itself to manage.

After evaluating the distributed storage landscape, I settled on SeaweedFS. Here is how I built a high-efficiency cluster pooling 840 GB of storage, and the operational lessons I learned along the way.

---

Why SeaweedFS?

Narrowing down the storage options for this farm was an elaborate exercise. Traditional distributed filesystems fell flat for a home lab environment:

• Ceph is incredibly powerful but essentially a career choice to administer. It is far too heavy, requiring deep operational knowledge, constant tuning, and massive memory overhead just to keep the cluster healthy.
• GlusterFS has a heavy configuration footprint and is highly sensitive to network latency, making it bloated for smaller labs.
• MinIO has shifted its focus and license dynamics, making it less attractive for simple, general-purpose mounts.

SeaweedFS fit the bill perfectly because it is designed for speed and simplicity. Written in Go, it compiles to a single static binary with no external database dependencies. Its architecture splits metadata management (handled by the Filer) from block storage (handled by Volume Servers). This makes the entire stack extremely lightweight, requiring less than 50MB of RAM for the master nodes.

More importantly, SeaweedFS excels at managing heterogeneous disk sizes. Instead of striping data across drives (where the smallest disk limits the entire array), SeaweedFS groups data into independent, configurable volume files (which I set to 1 GB). This allows it to pack odd-sized drives to the brim while remaining highly efficient.

---

The Architecture: Pooling the Fleet

I configured the storage pool to span three physical nodes: the orchestrator node (lenovo) running an Alder Lake i7 with 48GB of RAM, the AI inference worker (tiger) hosting a Xeon and a 16GB GPU, and camry running Postgres fuzzing workloads.

Cluster Topology & Capacity

The cluster pools a total of 840 GB of raw capacity across three physical machines running five volume server instances:

Node	Port	Storage Media	Assigned Capacity
lenovo	:8080 :8081	Root NVMe (/var/lib/seaweedfs_data) Data NVMe (/mnt/nv1/seaweedfs_data)	100 GB 400 GB
tiger	:8080 :8081	HDD (/mnt/disk2/seaweedfs_data) SSD (/mnt/ssd/seaweedfs_data)	300 GB 30 GB
camry	:8080	Root SSD (/var/lib/seaweedfs_data)	10 GB (dialed down from 60 GB)

Note: camry is restricted to 10 GB because its root SSD is currently running tight on space. I will scale it up once I upgrade the drive.

The Rack Isolation Strategy: Handling Multi-Disk Nodes

A key detail of this configuration is how SeaweedFS handles nodes with multiple physical disks. Both lenovo and tiger run two separate volume server instances. On lenovo, one instance serves the root NVMe SSD (:8080) while the other serves the larger data NVMe drive (:8081). On tiger, one instance points to a mechanical HDD (:8080) and the other points to a SATA SSD (:8081).

In a default SeaweedFS layout without rack topology, the scheduler treats every volume server as an independent target. If a replication policy of 010 (write one copy to another node in the same data center) is evaluated, SeaweedFS could decide to store the primary copy on lenovo's root SSD volume server and the replica copy on lenovo's data NVMe volume server. While this would protect against a single SSD failure, it does nothing if the entire lenovo motherboard dies, if the machine suffers a power loss, or if the Linux kernel panics.

By explicitly hardcoding the -rack parameter on each systemd unit:

• Both volume servers on lenovo run with -rack=lenovo.
• Both volume servers on tiger run with -rack=tiger.
• The volume server on camry runs with -rack=camry.

I tell SeaweedFS that these volume servers share physical hardware. When the cluster evaluates a rack-level replication rule (like the middle digit in 010 or 020), it is forced to select a volume server in a different rack. This ensures that copies are always placed on physically separate machines, achieving true hardware-level fault tolerance.

Replication Policies

Each path prefix has a tailored replication policy configured via weed shell -> fs.configure:

Path	Policy	Copies	Behavior
/ (Default)	010	2	One extra copy on a different physical machine (different rack).
/photos	020	3	Full replication; one copy on every machine in the fleet.
/scratch	000	1	Single-copy, no redundancy. Used for the ~185 GB Jellyfin movie library.

The /scratch directory allows me to store my movie library with zero redundancy overhead. Since the media can be re-downloaded, accepting single-copy storage is a sensible trade for reclaiming disk space.

---

Operational Lessons: Surviving a Silent Outage

No DIY storage cluster is complete without testing its edges. During a routine reboot of lenovo, I encountered a silent outage that served as a fantastic learning experience for hardening FUSE client mounts.

The Boot-Time Race

When lenovo booted, its local SeaweedFS filer, S3 gateway, and volume servers ran their IP auto-detection routines. Because the systemd units were configured with After=network.target, they launched before the physical network interfaces had actually acquired their LAN IP addresses.

Consequently, weed resolved the local hostname to the loopback address 127.0.1.1 (per Debian's default /etc/hosts mapping) and bound its services to localhost. The master service survived because its peers list explicitly hardcoded the LAN IPs, but the filer and volume servers on lenovo became unreachable to the rest of the network.

This triggered a cascade of failures:

1. Unreachable Filer: Since the filer only listened on 127.0.0.1, FUSE mounts on tiger and camry threw ENOTCONN errors and became stale.
2. FUSE Crash-Loop: The local mount service on lenovo began crash-looping in the background, trying and failing to mount the local filer every 10 seconds.
3. The Samba Blindspot: Samba on lenovo was configured to share the mount point directory /mnt/seaweedfs to my Windows laptop as the Z: drive. Because Samba started successfully and the directory /mnt/seaweedfs still existed, Samba happily served the bare local directory to the network.

Because Samba was serving the bare local mount point, any files written by Samba or automated scripts landed directly on the local host root filesystem, hidden underneath the unmounted folder path.

Discovery & Triage

I detected the issue when running my weekly system inventory script and reviewing the git history. The script captures cluster topology and active mount capacities. The diff immediately flagged a disk contents discrepancy: the files and directories visible on consecutive runs did not match, signaling that the FUSE mounts had silently disappeared.

What was supposed to be the mounted volume was actually just the empty, unmounted /mnt/seaweedfs folder on the host's root filesystem.

Upon realizing this, I triaged the nodes, recovered the orphaned files from the underlying root directories, successfully remounted the cluster, and restored the data back into the active SeaweedFS storage pool.

---

Hardening the Storage Layer

To prevent this boot-race and silent write behavior from happening again, I implemented five structural fixes across all nodes:

1. Hardcoding IPs in Services

I stripped auto-detection out of the systemd unit files. Every service now explicitly binds and advertises its LAN IP:

# /etc/systemd/system/seaweedfs-filer.service
ExecStart=/usr/local/bin/weed filer -ip=192.168.85.70 -ip.bind=0.0.0.0 -port=8888 -s3.port=8333

2. Delaying Until the Network is Truly Online

I updated all SeaweedFS units to wait for network-online.target rather than the weak network.target. This guarantees that IP routing is fully established before the storage layer starts up:

[Unit]
Description=SeaweedFS Filer
After=network-online.target
Wants=network-online.target

3. Fail-Hard Mountpoint Permissions (The chmod 000 Trick)

To stop silent local writes when FUSE mounts fail, I unmounted /mnt/seaweedfs on every node and locked down the bare folders:

sudo umount /mnt/seaweedfs
sudo chmod 000 /mnt/seaweedfs
sudo chown root:root /mnt/seaweedfs

When SeaweedFS is not mounted, /mnt/seaweedfs is owned by root with no read, write, or execute permissions. Any attempt by a script or Samba to write to it immediately throws a loud "Permission Denied" error. When mounted, the FUSE driver overrides these permissions.

4. Active Liveness Watchdog

I deployed a simple systemd timer-based watchdog script. Every hour, it checks the mount status using mountpoint -q and a quick read probe:

#!/bin/bash
# /usr/local/bin/seaweedfs-mount-health.sh
if ! mountpoint -q /mnt/seaweedfs || ! timeout 15 stat /mnt/seaweedfs >/dev/null 2>&1; then
    echo "SeaweedFS mount is unhealthy or stale. Attempting restart..."
    systemctl restart seaweedfs-mount
fi

If the mount fails the probe, the service automatically restarts, restoring the link and kicking Samba to pick up the fresh mount.

5. Prometheus Disk Health Monitoring

Finally, to round out the observability of the storage cluster, I configured Prometheus (backed by VictoriaMetrics) to handle fleet-wide disk health monitoring. By collecting metrics via node_exporter across all three nodes, I can now monitor SMART health, temperature, and disk space utilisation. This provides an essential early-warning layer, alerting me to degrading physical drives or capacity bottlenecks before they have a chance to affect the running volume servers (though I will save the deep dive into the Prometheus configuration and custom alert rules for a future post).

The Economics: Cloud Costs vs. Commodity Hardware

Besides the flexibility of having a local storage pool, there is a strong financial argument for offline storage. Eventually, this local cluster can serve as a direct copy of my cloud storage, saving me the $30 to $50 I spend every month on subscription fees.

When you look at the cost of the commodity hardware in my home lab (much of it sourced from second-hand marketplaces), the economics are hard to argue with:

• camry: Sourced three years ago for a mere $80.
• tiger: A basic server build at $200 (though the RTX 5060 Ti 16 GB inside it was a good grand).
• lenovo: The premium compute node in the farm, which was the most expensive at $800.
• Storage Media: A typical NVMe SSD costs around $100.

For a modest investment in hardware, I get a private storage pool that I own entirely. Yes, running your own storage layer comes with some operational maintenance and the occasional "pain" of triaging configuration races, but the trade-off is massive: zero recurring monthly storage bills, complete data privacy, and local gigabit transfer speeds.

---

Conclusion

SeaweedFS has proven to be a phenomenally fast, lightweight, and low-maintenance distributed storage option. It has allowed me to turn a collection of odd, mismatched disks into a resilient, zero-cost network drive without the operational complexity of Ceph or GlusterFS.

By enforcing strict IP binding, delaying service start until the network is fully online, locking the local directory permissions to 000, and running an active watchdog, I have turned a raw object store into a robust, self-healing, set-and-forget storage layout. It does exactly what home lab storage should do: it runs efficiently in the background, stays out of the way, and lets me focus on the actual work.