SeaweedFS: Distributed Storage with Commodity Equipment

Maintaining a home lab with a diverse fleet of servers is a constant balancing act, but the real challenge is finding a way to pool mismatched, leftover drives across separate machines without turning storage administration into a second full-time job. Over time, my server farm accumulated an assortment of mismatched disks—a fast NVMe SSD on one machine, a spinning mechanical HDD on another, and a tight OS drive elsewhere. Rather than letting this spare capacity go to waste or buying an expensive, dedicated NAS appliance, I set out to build a unified distributed shared storage layer across my fleet.

My goal was simple: the storage solution had to be low-maintenance, highly efficient with heterogeneous drives, actively developed, and crucially, not a project in itself to manage.

After evaluating the distributed storage landscape, I settled on SeaweedFS. Here is how I built a high-efficiency cluster pooling 840 GB of storage, and the operational lessons I learned along the way.

---

Why SeaweedFS?

Narrowing down the storage options for this farm was an elaborate exercise. Traditional distributed filesystems fell flat for a home lab environment:

• Ceph is incredibly powerful but essentially a career choice to administer. It is far too heavy, requiring deep operational knowledge, constant tuning, and massive memory overhead just to keep the cluster healthy.
• GlusterFS has a heavy configuration footprint and is highly sensitive to network latency, making it bloated for smaller labs.
• MinIO has shifted its focus and license dynamics, making it less attractive for simple, general-purpose mounts.

SeaweedFS fit the bill perfectly because it is designed for speed and simplicity. Written in Go, it compiles to a single static binary with no external database dependencies. Its architecture splits metadata management (handled by the Filer) from block storage (handled by Volume Servers). This makes the entire stack extremely lightweight, requiring less than 50MB of RAM for the master nodes.

More importantly, SeaweedFS excels at managing heterogeneous disk sizes. Instead of striping data across drives (where the smallest disk limits the entire array), SeaweedFS groups data into independent, configurable volume files (which I set to 1 GB). This allows it to pack odd-sized drives to the brim while remaining highly efficient.

---

The Architecture: Pooling the Fleet

I configured the storage pool to span three physical nodes: the orchestrator node (lenovo) running an Alder Lake i7 with 48GB of RAM, the AI inference worker (tiger) hosting a Xeon and a 16GB GPU, and camry running Postgres fuzzing workloads.

Cluster Topology & Capacity

The cluster pools a total of 840 GB of raw capacity across three physical machines running five volume server instances:

Node	Port	Storage Media	Assigned Capacity
lenovo	:8080 :8081	Root NVMe (/var/lib/seaweedfs_data) Data NVMe (/mnt/nv1/seaweedfs_data)	100 GB 400 GB
tiger	:8080 :8081	HDD (/mnt/disk2/seaweedfs_data) SSD (/mnt/ssd/seaweedfs_data)	300 GB 30 GB
camry	:8080	Root SSD (/var/lib/seaweedfs_data)	10 GB (dialed down from 60 GB)

Note: camry is restricted to 10 GB because its root SSD is currently running tight on space. I will scale it up once I upgrade the drive.

The Rack Isolation Strategy: Handling Multi-Disk Nodes

A key detail of this configuration is how SeaweedFS handles nodes with multiple physical disks. Both lenovo and tiger run two separate volume server instances. On lenovo, one instance serves the root NVMe SSD (:8080) while the other serves the larger data NVMe drive (:8081). On tiger, one instance points to a mechanical HDD (:8080) and the other points to a SATA SSD (:8081).

In a default SeaweedFS layout without rack topology, the scheduler treats every volume server as an independent target. If a replication policy of 010 (write one copy to another node in the same data center) is evaluated, SeaweedFS could decide to store the primary copy on lenovo's root SSD volume server and the replica copy on lenovo's data NVMe volume server. While this would protect against a single SSD failure, it does nothing if the entire lenovo motherboard dies, if the machine suffers a power loss, or if the Linux kernel panics.

By explicitly hardcoding the -rack parameter on each systemd unit:

• Both volume servers on lenovo run with -rack=lenovo.
• Both volume servers on tiger run with -rack=tiger.
• The volume server on camry runs with -rack=camry.

I tell SeaweedFS that these volume servers share physical hardware. When the cluster evaluates a rack-level replication rule (like the middle digit in 010 or 020), it is forced to select a volume server in a different rack. This ensures that copies are always placed on physically separate machines, achieving true hardware-level fault tolerance.

Replication Policies

Each path prefix has a tailored replication policy configured via weed shell -> fs.configure:

Path	Policy	Copies	Behavior
/ (Default)	010	2	One extra copy on a different physical machine (different rack).
/photos	020	3	Full replication; one copy on every machine in the fleet.
/scratch	000	1	Single-copy, no redundancy. Used for the ~185 GB Jellyfin movie library.

The /scratch directory allows me to store my movie library with zero redundancy overhead. Since the media can be re-downloaded, accepting single-copy storage is a sensible trade for reclaiming disk space.

---

Operational Lessons: Surviving a Silent Outage

No DIY storage cluster is complete without testing its edges. During a routine reboot of lenovo, I encountered a silent outage that served as a fantastic learning experience for hardening FUSE client mounts.

The Boot-Time Race

When lenovo booted, its local SeaweedFS filer, S3 gateway, and volume servers ran their IP auto-detection routines. Because the systemd units were configured with After=network.target, they launched before the physical network interfaces had actually acquired their LAN IP addresses.

Consequently, weed resolved the local hostname to the loopback address 127.0.1.1 (per Debian's default /etc/hosts mapping) and bound its services to localhost. The master service survived because its peers list explicitly hardcoded the LAN IPs, but the filer and volume servers on lenovo became unreachable to the rest of the network.

This triggered a cascade of failures:

1. Unreachable Filer: Since the filer only listened on 127.0.0.1, FUSE mounts on tiger and camry threw ENOTCONN errors and became stale.
2. FUSE Crash-Loop: The local mount service on lenovo began crash-looping in the background, trying and failing to mount the local filer every 10 seconds.
3. The Samba Blindspot: Samba on lenovo was configured to share the mount point directory /mnt/seaweedfs to my Windows laptop as the Z: drive. Because Samba started successfully and the directory /mnt/seaweedfs still existed, Samba happily served the bare local directory to the network.

Because Samba was serving the bare local mount point, any files written by Samba or automated scripts landed directly on the local host root filesystem, hidden underneath the unmounted folder path.

Discovery & Triage

I detected the issue when running my weekly system inventory script and reviewing the git history. The script captures cluster topology and active mount capacities. The diff immediately flagged a disk contents discrepancy: the files and directories visible on consecutive runs did not match, signaling that the FUSE mounts had silently disappeared.

What was supposed to be the mounted volume was actually just the empty, unmounted /mnt/seaweedfs folder on the host's root filesystem.

Upon realizing this, I triaged the nodes, recovered the orphaned files from the underlying root directories, successfully remounted the cluster, and restored the data back into the active SeaweedFS storage pool.

---

Hardening the Storage Layer

To prevent this boot-race and silent write behavior from happening again, I implemented five structural fixes across all nodes:

1. Hardcoding IPs in Services

I stripped auto-detection out of the systemd unit files. Every service now explicitly binds and advertises its LAN IP:

# /etc/systemd/system/seaweedfs-filer.service
ExecStart=/usr/local/bin/weed filer -ip=192.168.85.70 -ip.bind=0.0.0.0 -port=8888 -s3.port=8333

2. Delaying Until the Network is Truly Online

I updated all SeaweedFS units to wait for network-online.target rather than the weak network.target. This guarantees that IP routing is fully established before the storage layer starts up:

[Unit]
Description=SeaweedFS Filer
After=network-online.target
Wants=network-online.target

3. Fail-Hard Mountpoint Permissions (The chmod 000 Trick)

To stop silent local writes when FUSE mounts fail, I unmounted /mnt/seaweedfs on every node and locked down the bare folders:

sudo umount /mnt/seaweedfs
sudo chmod 000 /mnt/seaweedfs
sudo chown root:root /mnt/seaweedfs

When SeaweedFS is not mounted, /mnt/seaweedfs is owned by root with no read, write, or execute permissions. Any attempt by a script or Samba to write to it immediately throws a loud "Permission Denied" error. When mounted, the FUSE driver overrides these permissions.

4. Active Liveness Watchdog

I deployed a simple systemd timer-based watchdog script. Every hour, it checks the mount status using mountpoint -q and a quick read probe:

#!/bin/bash
# /usr/local/bin/seaweedfs-mount-health.sh
if ! mountpoint -q /mnt/seaweedfs || ! timeout 15 stat /mnt/seaweedfs >/dev/null 2>&1; then
    echo "SeaweedFS mount is unhealthy or stale. Attempting restart..."
    systemctl restart seaweedfs-mount
fi

If the mount fails the probe, the service automatically restarts, restoring the link and kicking Samba to pick up the fresh mount.

5. Prometheus Disk Health Monitoring

Finally, to round out the observability of the storage cluster, I configured Prometheus (backed by VictoriaMetrics) to handle fleet-wide disk health monitoring. By collecting metrics via node_exporter across all three nodes, I can now monitor SMART health, temperature, and disk space utilisation. This provides an essential early-warning layer, alerting me to degrading physical drives or capacity bottlenecks before they have a chance to affect the running volume servers (though I will save the deep dive into the Prometheus configuration and custom alert rules for a future post).

The Economics: Cloud Costs vs. Commodity Hardware

Besides the flexibility of having a local storage pool, there is a strong financial argument for offline storage. Eventually, this local cluster can serve as a direct copy of my cloud storage, saving me the $30 to $50 I spend every month on subscription fees.

When you look at the cost of the commodity hardware in my home lab (much of it sourced from second-hand marketplaces), the economics are hard to argue with:

• camry: Sourced three years ago for a mere $80.
• tiger: A basic server build at $200 (though the RTX 5060 Ti 16 GB inside it was a good grand).
• lenovo: The premium compute node in the farm, which was the most expensive at $800.
• Storage Media: A typical NVMe SSD costs around $100.

For a modest investment in hardware, I get a private storage pool that I own entirely. Yes, running your own storage layer comes with some operational maintenance and the occasional "pain" of triaging configuration races, but the trade-off is massive: zero recurring monthly storage bills, complete data privacy, and local gigabit transfer speeds.

---

Conclusion

SeaweedFS has proven to be a phenomenally fast, lightweight, and low-maintenance distributed storage option. It has allowed me to turn a collection of odd, mismatched disks into a resilient, zero-cost network drive without the operational complexity of Ceph or GlusterFS.

By enforcing strict IP binding, delaying service start until the network is fully online, locking the local directory permissions to 000, and running an active watchdog, I have turned a raw object store into a robust, self-healing, set-and-forget storage layout. It does exactly what home lab storage should do: it runs efficiently in the background, stays out of the way, and lets me focus on the actual work.

Taming the Token Burn

Cloud AI is incredibly convenient, but the costs scale rapidly once you start leaning on long-context workflows. I've spent the last year relying on Gemini Pro for everything from Postgres automation to complex scripting, but the honeymoon phase is officially over (for me). As my projects grew in scope, so did the token consumption, eventually hitting a brick wall that forced a rethink of my entire stack.

For most, this is a niche problem. For anyone building high-context agentic workflows, it’s the only problem that matters.

Last year, the ride was smooth. My "AI usage" was minimal and Gemini Pro handled my daily tasks with ease. However, over the last three months, I've started pushing it with "decent-sized" projects—tasks that require keeping dozens of context balls in the air simultaneously. The result? It guzzles through tokens like a V8 engine in a traffic jam.

The breaking point came recently when, during a deep-dive development session, I fed the model a substantial task involving a complex codebase. Within two hours, I burned through 100% of my monthly allocation!!

Three (3) months in a row!

Yeah, that's not going to fly.

Finding the Right Tool for the AI Job

It’s time for a pivot, but not necessarily a total exit from the cloud. I’ve recently taken a Claude subscription as well, and it has been proving to be remarkably good at tasks where Gemini used to stutter. It’s basically been a lesson in finding "the right tool for the job" rather than looking for a single silver bullet.

That said, reducing my reliance on expensive cloud tokens is a priority. The recent NVIDIA RTX 5060 Ti purchase was a steep investment, but nonetheless a strategic move towards local autonomy. The goal is to migrate my heavy-lift agentic AI workflows to my local server farm, reserving premium cloud services for the few tasks where they are truly indispensable—specifically, massive context shifts that can't be easily decomposed. Some key examples being planning complex projects, or tasks that require reasoning over large codebases.

I have a few machines at home (hosting IronClaw, Postgres fuzzing, Jellyfin, Git servers, Obsidian repositories, and a Grafana dashboard to keep an eye on it all), and the goal is to keep costs down while making these tools smarter. The only way I see that happening is if I reduce Cloud API dependence and switch existing tooling to local LLMs.

Another key driver is that I expect some of my newer projects, still in stealth mode, to burn through tokens like nobody's business; they wouldn't see the light of day if I didn't change the status quo.

Stay tuned.

Boost Database Security: Restrict Users to Read Replicas

Only Allow Login to Read-Replicas and Standbys

When you're working with large databases in production, it is incredibly common to use read-replicas to improve performance. These read-replicas are a copy of your primary (main) database and let your applications offload read-heavy queries, which in-turn reduces strain on your primary database, effectively making the application faster and snappier.

Sometimes, you may want to restrict specific database users so they can connect ONLY to these read-replicas, and not to the primary database server. This can be tricky to implement, since any permissions configured for this use-case, whether on the user-level, the database level, the schema-level or even the table level would be quickly replicated to the read-replicas and thus would not work as expected.

This guide will show how to configure a database user to only login successfully on a read-replica. The only requirement is to enable the pg_tle extension [3] on your PostgreSQL database. This is simple to do on your Ubuntu based Laptop (see how to do that here [2]) or virtual-machines offered by your favourite cloud-provider. Furthermore, you could apply your login rules using Pl/PgSQL, PL/v8 or even PL/Rust - See here[1].

Why Restrict Access?

There are many good reasons for restricting users to read-replicas:

• Performance: You can dedicate your primary database server to handling write operations (like updating data), ensuring those operations happen as fast as possible.

• Reporting / Analytics: Production environments often have dedicated users for ancillary tasks, such as monitoring, reporting dashboards, read-only tenants etc. Restricting these database users to read-replica helpsreducing extra load on the primary database.

• Security: In some cases, granting direct access to the primary database might be considered a security risk. Further, you may not be able to force login hygeine for all your database users, and then having a lockdown system to reject those database users to login to primary is crucial for application rollout.

Prerequisites

• An existing PostgreSQL database instance with at least one read-replica.
  • You could also try this on your own Postgres database with pg_tle extension. Read here [2] for more on how to install pg_tle on your Ubuntu system.
• Basic understanding of users and permissions within a database.

Steps

1. Identify Target Database and Users: First we need to define how to implement the restriction. i.e. Which users (and database) are to be restricted to login only to read-replica. In the example below, we would restrict the user standby_only_user to only be able to login to Standbys / Read-Replicas on database prod_db.

psql <<SQL
  \c prod_db
  CREATE EXTENSION pg_tle;
SQL 

2. Ensure that shared_preload_libraries is properly set to allow pg_tle. Also make sure that the pgtle.clientauth_db_name is appropriately set to the desired database (here prod_db):

cat <<EOL >> data/postgresql.conf
  shared_preload_libraries='pg_tle'
  pgtle.enable_clientauth=require
  pgtle.clientauth_db_name=prod_db
  pgtle.clientauth_users_to_skip=robins
  pgtle.clientauth_databases_to_skip=''
EOL

3. Secret Sauce:

Next we create the key pg_tle function that restricts the user standby_only_user to login successfully only if this is a standby / read-replica:

SELECT pgtle.install_extension (
  'standbyusercheck',
  '1.0',
  'Allow some users to login only to standby / read-replicas',
$_pgtle_$
  CREATE SCHEMA standbycheck_schema;

  REVOKE ALL ON SCHEMA standbycheck_schema FROM PUBLIC;
  GRANT USAGE ON SCHEMA standbycheck_schema TO PUBLIC;

  CREATE OR REPLACE FUNCTION standbycheck_schema.standbycheck_hook(port pgtle.clientauth_port_subset, status integer)
  RETURNS void AS $$
    DECLARE
      is_standby bool := TRUE;
    BEGIN
      IF port.user_name = 'standby_only_user' THEN
        SELECT pg_is_in_recovery()
          INTO is_standby;
        IF is_standby THEN
          RAISE NOTICE 'User allowed to login';
        ELSE
          RAISE EXCEPTION 'User can only login to Standby / Read-Replicas';
        END IF;
      END IF;
    END
  $$ LANGUAGE plpgsql SECURITY DEFINER;

  GRANT EXECUTE ON FUNCTION standbycheck_schema.standbycheck_hook TO PUBLIC;
  SELECT pgtle.register_feature('standbycheck_schema.standbycheck_hook', 'clientauth');
  REVOKE ALL ON SCHEMA standbycheck_schema FROM PUBLIC;
$_pgtle_$
);

And now that the function is defined,CREATE EXTENSION would install the function and bind it to future login attempts.

CREATE EXTENSION standbyusercheck;
SHOW pgtle.clientauth_db_name;

4. Test Connection:

• Attempting to connect as a privileged user (here robins) to either of primary or read-replica should succeed.

Logging into Replica as robins
 login  | current_database | pg_is_in_recovery
--------+------------------+-------------------
 robins | prod_db          | t
(1 row)

Logging into Primary as robins
 login  | current_database | pg_is_in_recovery
--------+------------------+-------------------
 robins | prod_db          | f
(1 row)

• However, the user standby_only_user should NOT be able to login to the primary.

Logging into Primary as standby_only_user
psql: error: connection to server at "localhost" (127.0.0.1), port 6432 failed: FATAL:  User can only login to Standby / Read-Replicas

• While the user (standby_only_user) should only be able to login to any read-replica.

Logging into Replica as standby_only_user
       login       | current_database | pg_is_in_recovery
-------------------+------------------+-------------------
 standby_only_user | prod_db          | t
(1 row)

Other important aspects of this feature

• You could force clientauth for all logins by setting the parameter pgtle.enable_clientauth = require

• You could configure some users to always be allowed to login to either of Primary / Read-replica in cases of emergency, by adding that user to the pgtle.clientauth_users_to_skip. Ideally you would want your admin database roles to this list.

• Orthogonally, you could configure some databases to always allow users to skip clientauth by setting the pgtle.clientauth_databases_to_skip feature.

• Note, that both clientauth_databases_to_skip and clientauth_databases_to_skip can be utilised together. This is a good way to ensure that some set of database users (and some databases) are exempt from such a login restriction.

• If pgtle.enable_clientauth is set to on or require and if the database mentioned in pgtle.clientauth_db_name is not configured correctly, postgres would complain with the messsage FATAL: pgtle.enable_clientauth is set to require, but pg_tle is not installed or there are no functions registered with the clientauth feature. This is a good engine check, helping us avoid basic misconfigurations.

• If you're anticipating connection storms, you can also increase the workers (that would help enforce the login restriction) by setting the pgtle.clientauth_num_parallel_workers parameter to greater than 1.

Conclusion

By following the above steps, you've now successfully configured your PostgreSQL environment to restrict certain users to only login to the read-replicas. This helps not just optimize your database performance, but also bolster security.

Let me know if you'd like to explore more advanced scenarios or discuss IAM integration for fine-grained access control!

Reference

1. Clientauth Hook Documentation - https://github.com/aws/pg_tle/blob/main/docs/04_hooks.md'
2. Install pg_tle On Ubuntu - https://www.thatguyfromdelhi.com/2024/04/installing-pgtle-on-ubuntu-quick-guide.html
3. Unlock PostgreSQL Super Powers with pg_tle - https://www.thatguyfromdelhi.com/2024/04/unlock-postgresql-superpowers-with-pgtle.html

Redshift support for psql

Am sure you know that psql doesn't go out of it's way to support Postgres' forks natively. I obviously understand the reasoning, which allowed me to find a gap that I could fill here.

The existing features (in psql) that work with any Postgres fork (like Redshift) are entirely because it is a fork of Postgres. Since I use psql heavily at work, last week I decided to begin maintaining a Postgres fork that better supports (Postgres forks, but initially) Redshift. As always, unless explicitly mentioned, this is entirely an unofficial effort.

The 'redshift' branch of this Postgres code-base, is aimed at supporting Redshift in many ways:

• Support Redshift related artifacts
• Redshift specific SQL Commands / variations
• Redshift Libraries
• Support AWS specific artifacts
• For e.g. AWS Regions
• Support Redshift specific changes
• For e.g. "/d table" etc.

The idea is:

• Maintain this branch for the long-term
• At least as long as I have an accessible Redshift cluster
• Down the line look at whether other Postgres forks (for e.g. RDS Postgres) need such special attention
• Although nothing much stands out yet
• Except some rare exceptions like this or this, which do need to go through an arduous long wait / process of refinement.
• Change the default port to 5439 (or whatever the flavour supports)
• ...with an evil grin ;)
• Additionally, as far as possible:
• Keep submitting Postgres related patches back to Postgres master
• Keep this branch up to date with Postgres master

Update (31st August 2017)

• Currently this branch supports most Redshift specific SQL commands such as
• CREATE LIBRARY
• CREATE TABLE (DISTKEY / DISTSTYLE / ...)
• Returns non-SQL items like
• ENCODINGs (a.k.a. Compressions like ZSTD / LZO etc )
• REGIONs (for e.g. US-EAST-1 etc.)
• Of course some complex variants (for e.g. GRANT SELECT, UPDATE ON ALL TABLES IN SCHEMA TO GROUP xxx ) don't automatically come up with tab-complete feature. This is primarily because psql's tab-complete feature isn't very powerful to cater to all such scenarios which in turn is because psql's auto-complete isn't a full-fledged parser to begin with.
• In a nutshell, this branch is now in a pretty good shape to auto-complete the most common Redshift specific SQL Syntax.
• The best part is that this still merges perfectly with Postgres mainline!

Let me know if you find anything that needs inclusion, or if I missed something.

====================================

$  psql -U redshift_user -h localhost -E -p 5439 db
psql (client-version:11devel, server-version:8.0.2, engine:redshift)
Type "help" for help.

db=#

Reducing Wires

Recently got an additional monitor for my workstation@home and found that the following wires were indispensable:

• USB Mouse
• Monitor VGA / HDMI / DVI cable
• USB Hub cable (Pen Drive etc.)

I was lucky that this ($20 + used) Dell monitor was an awesome buy since it came with a Monitor USB Hub (besides other goodies such as vertical rotate etc).

After a bit of rejigging, this is how things finally panned-out:

• 1 USB Wire (from the laptop) for the MUH (Monitor USB Hub)
• This is usually something like this.
• Use a USB->DVI converter and use that to connect MUH -> Monitor DVI port
• This is usually something like this.
• Plug USB Mouse to MUH
• With things working so well, I also plugged a Wireless Touchpad dongle to the MUH

So now when I need to do some office work, connecting 1 USB wire gets me up and running!

#LoveOneWires :)

Now only if I could find a stable / foolproof Wireless solution here ;)

Using generate_series() in Redshift

Considering that Redshift clearly states that it doesn't support (the commonly used postgres function) generate_series(), it gets very frustrating if you just want to fill a table with a lot of rows and can't without a valid data-source.

Solution (Generates a billion integers on my test-cluster):

--INSERT INTO tbl
WITH x AS (
  SELECT 1 
  FROM stl_connection_log a, stl_connection_log b, stl_connection_log c
  -- LIMIT 100
)
  SELECT row_number() over (order by 1) FROM x;

For a Redshift server with even a basic level of login activity, this should generate enough rows. For e.g. On my test cluster, where I am the only user, this currently generates 4034866688 (4 billion) rows :) !

Interestingly, irrespective of the document, generate_series() actually does work on Redshift:

# select b from generate_series(1,3) as a(b);
┌───┐
│ b │
├───┤
│ 1 │
│ 2 │
│ 3 │
└───┘
(3 rows)

The reason why this wouldn't let you insert any rows to your table though, is that this is a Leader-Node-Only function, whereas INSERTs (on any non-single Redshift Cluster) are run on the Compute Nodes (which don't know about this function).

The reason why the above works, is ROW_NUMBER() and CROSS JOIN allow us to generate a large number of rows, but for that, the initial data-set (here the STL_CONNECTION_LOG System Table) should have at least some rows to multiply on! You could use any other system table (that is available on Compute Nodes) if required, for some other purpose.

Play On!