Why Postgres Doesn't Have synchronous_commit=remote_receive?

In distributed database environments, balancing durability and performance is a constant tug-of-war. PostgreSQL’s synchronous_commit parameter sits at the heart of this, giving administrators a dial to choose exactly when a COMMIT returns success to the client.

The idea of remote_receive was born from a simple question: does skipping the standby's disk write yield a measurable, real-world performance benefit? By waiting only for WAL bytes to reach the standby's memory, could we get a meaningful boost over remote_write? I set out to implement and benchmark this feature to find out.

What followed was a journey through network latency, OS page caches, CPU scheduler thrashing, and benchmarking noise. Here is the breakdown of the implementation, the tests, the initial anomalies, and the final results.

---

1. The Feature: What is remote_receive?

Before this branch, PostgreSQL offered four primary synchronous commit modes:

• off: Fully asynchronous. (Fastest, least safe)
• local: Waits for local disk flush on the primary.
• remote_write: Waits for the standby to write the WAL to its OS buffer cache (pwrite).
• remote_apply: Waits for the standby to fully replay the WAL. (Slowest, most safe)

remote_receive sits directly between local and remote_write. In this mode, the primary guarantees that the WAL bytes have physically arrived at the standby's walreceiver process buffer. It does not wait for the standby to call pwrite().

The Hypothesis: By completely bypassing the standby's disk I/O, remote_receive should deliver lower latency and higher throughput than remote_write, especially on replica hardware with slow disks.

Implementation Details

To build this, I had to modify both the standby and the primary:

1. Standby Status Update: I modified the 34-byte wire message that the standby sends to the primary, adding a new 8-byte receivePtr (creating a 42-byte message, backward compatible).
2. Early Replies: I modified walreceiver.c to send a reply message immediately upon receiving a WAL chunk in memory, before the XLogWalRcvWrite() call executes the pg_pwrite.
3. Primary Wait Logic: I updated syncrep.c and walsender.c to track the SYNC_REP_WAIT_RECEIVE wait queue, releasing waiting backends as soon as the standby's receivePtr advanced.

---

2. Scenario 1: The SSD Baseline (Fast Primary, Fast Replica)

To validate the code, I first set up a baseline test using two fast machines on a gigabit LAN.

The Servers:

• Primary: lenovo (Intel Core i7-12700 12C/20T, 48GB RAM, NVMe SSD)
• Replica: camry (Intel Core i7-4770 4C/8T, 24GB RAM, SATA SSD)

I ran pgbench (Scale 10, 4 clients) for 30 seconds across the different modes.

The Result:

• remote_write: 3,944 TPS (Median)
• remote_receive: 3,946 TPS (Median)

The performance was virtually identical (a 0.06% difference). Why didn't remote_receive pull ahead?

The Reality of pwrite(): On modern operating systems with free RAM, a standard pwrite() to a buffered file does not write to physical disk immediately. It copies the data into the OS page cache (essentially a memory copy taking mere microseconds), leaving the kernel to flush dirty pages asynchronously.

With network round-trip time (RTT) on a gigabit LAN between 0.2ms and 1.0ms, the 5 microseconds saved by bypassing pwrite() is completely dwarfed by network latency. This makes remote_write and remote_receive perform nearly identically in typical conditions.

This fundamental reality explains why core PostgreSQL developers historically questioned the return on investment (RoI) of a memory-only receive mode. Because a standard pwrite() to the OS page cache is already a RAM-speed operation (taking mere microseconds), remote_write is practically as fast as any network-receive-only mode under normal conditions, but with a major durability advantage: it survives a PostgreSQL process crash on the standby (as long as the OS remains running). Bypassing it would reduce durability without providing any real-world performance benefit in typical conditions. In pgsql-hackers discussions—such as the thread on Measuring Replay Lag where distinct write_lag and flush_lag tracking was introduced—it is clear that the network round-trip time dominates the replication pipeline. To find a measurable benefit for a receive-only mode, I needed a replica where disk I/O was a severe enough bottleneck to cause page cache pressure and slow down the pwrite() call itself.

---

3. Scenario 2: The HDD Challenge (Asymmetric Hardware)

For the next test, I replaced the fast camry replica with a much weaker machine.

The Servers:

• Primary: lenovo (i7-12700, SSD)
• Replica: mac (Intel Core i5-2520M 2C/4T, 16GB RAM, 5400RPM HDD)

With a mechanical hard drive and a slow dual-core processor, I expected remote_receive to outpace remote_write. I ran three 30-second runs per mode, but the initial results were unexpected:

The First (Anomalous) Result:

• remote_write: 203.6 TPS (Median)
• remote_receive: 179.0 TPS (Median)

remote_write was ~20 TPS faster than remote_receive. Tracing the walreceiver and walsender loops ruled out code bugs; instead, the bottleneck came down to four factors:

1. The OS Cache Illusion: Even on a slow HDD, pwrite() still writes to RAM. The mechanical disk's extreme latency only hits during fsync or when the page cache fills up, meaning the receive-only advantage remained small.
2. CPU/Scheduler Thrashing: By sending an "early reply" before writing to disk, remote_receive generates twice as many TCP reply packets (one for receive, one later for flush). On the replica's older dual-core CPU, processing this packet flood alongside WAL replay caused high context-switching overhead.
3. Flow Control: remote_write acted as a natural flow-control mechanism. By waiting to write before replying, it throttled the primary slightly, keeping the replica's CPU out of a thrashing state.
4. Statistical Noise: The remote_write runs ranged from 177 to 211 TPS (a 19% spread). A 3-run, 30-second test was simply too noisy to yield a reliable median.

Aside: The Raspberry Pi 4 Attempt Before settling on the Mac Mini HDD, I attempted to use a Raspberry Pi 4 (pi4 — Cortex-A72 4C/4T, 4GB RAM, SD card / USB storage) as the slow replica. However, the Pi 4 was a poor fit for this benchmark. The main issue wasn't simply that the CPU maxed out, but that the low-power ARM CPU could not keep pace with the primary's faster rate of WAL generation. This lag cascaded into secondary issues—such as rapidly mounting replication lag, TCP buffer queues, and process starvation—which completely dominated the environment and masked any storage-level performance differences.

---

4. The Final Test: Eliminating the Noise

In the initial runs, the slow mechanical disk combined with standard kernel buffering created a massive source of noise: the sustained background I/O writes from a just-finished test were still flushing to disk when the next test began. Although an LSN cross-check was already in place (waiting for the replica's replay_lsn to catch up to the primary's current WAL LSN), this only verified database-level catch-up, not physical disk queue clearance. The residual write queue in the OS cache severely penalized the subsequent run, creating artificial variance. To isolate the actual replication performance and eliminate this noise, I overhauled the benchmarking methodology:

1. Interleaved Runs: I interleaved executions (Write, Receive, Write, Receive...) to average out temporal background OS tasks and thermal states.
2. Longer Runs and More Iterations: I ran 10 iterations per mode at 60 seconds per run (20 runs total).
3. Aggressive Cache Flushing: I ran sync on both the primary and replica between runs, followed by a 30-second sleep, to flush the OS page cache to physical disk platters and guarantee clean disk queues.

The Final Results:

Mode	Median TPS	Mean TPS	Median Latency
remote_write	201.58	200.74	20.663 ms
remote_receive	211.44	206.85	19.434 ms

The Results

With the noise eliminated and disk queues flushed, the true behavior emerged: remote_receive outperformed remote_write by ~10 TPS (~4.9% gain) at the median and ~6 TPS (~3.0% gain) at the mean. The small gap is expected: bypassing a RAM-buffered pwrite() on the replica yields microsecond-level gains, while network round-trip time remains the dominant factor.

Conclusion

The remote_receive implementation successfully introduces a granular durability option that guarantees WAL has crossed the network into the standby's memory before committing.

This exercise highlighted a key rule of database benchmarking: the OS page cache masks physical disk latency until it fills up. To accurately benchmark high-variance storage, one must use interleaved runs, higher iteration counts, and rigorous OS-level cache flushes between tests. Otherwise, you are measuring cache behavior and statistical noise rather than raw database throughput.

---

A Note on the Development Process & Resources: While I dabble in C and C++ from time to time, development at this level within PostgreSQL internals would not have been possible without AI assistance. Additionally, all resources for this project—including the Claude subscription, development machines, laptop, and time—were entirely personal and completely independent of my employer.

ALTER TABLE Gotcha related to USING

This is an old gotcha, that has been documented (although probably not much) and is something that we (in my company) almost got caught with recently.

This post, is about documenting the issue that looks like something that isn't going away anytime soon.

Consider this:

CREATE TABLE k (id INTEGER, b text);
CREATE TABLE l (id INTEGER);

CREATE VIEW vw_k AS SELECT k.b FROM k JOIN l USING (id);
TABLE vw_k;
ALTER TABLE l RENAME COLUMN id TO id2;
INSERT INTO k(id, b) VALUES (1, 'abc');
INSERT INTO l(id2)   VALUES (1);
TABLE vw_k;
DROP VIEW vw_k;
CREATE VIEW vw_k AS SELECT k.b FROM k JOIN l USING (id);

This is the output:

rt_testing=# CREATE TABLE k (id INTEGER, b text);
CREATE TABLE
rt_testing=# CREATE TABLE l (id INTEGER);
CREATE TABLE
rt_testing=# CREATE VIEW vw_k AS SELECT k.b FROM k JOIN l USING (id);
CREATE VIEW
rt_testing=# TABLE vw_k;
 b
---
(0 rows)

rt_testing=# ALTER TABLE l RENAME COLUMN id TO id2;
ALTER TABLE
rt_testing=# INSERT INTO k(id, b) VALUES (1, 'abc');
INSERT 0 1
rt_testing=# INSERT INTO l(id2)    VALUES (1);
INSERT 0 1
rt_testing=# TABLE vw_k;
  b
-----
 abc
(1 row)

rt_testing=# DROP VIEW vw_k;
DROP VIEW
rt_testing=# CREATE VIEW vw_k AS SELECT k.b FROM k JOIN l USING (id);
ERROR:  column "id" specified in USING clause does not exist in right table

If you noticed, despite changing the base table's relevant column (i.e. l.id => l.id2), the VIEW kept working without fail, until you actually try to DROP / CREATE the VIEW sometime down the line.

In fact, this gets difficult to trace because the VIEW continues to work normally even after the column name change. For e.g. New rows inserted to the updated table, continue to show up in the VIEW as if its definition was up to date!

To summarize, whenever changing a column name (used in a USING clause on a dependent VIEW), be careful that PostgreSQL may not necessarily complain about the dependency. You'd need to be careful to make that check yourself. Or else its possible that sometime down the line, when you're trying to restore a production backup, you'd start seeing errors related to changes you made a year back!

Why's my database suddenly so big?

In a Product based company, at times DB Developers don't get direct access to production boxes. Instead, an IT resource ends up managing a large swathe of Web / DB boxes. Since such a resource generally has a large field of operation, at times they need quick steps to identify the sudden high-disk usage.

In such a scenario (where Production is silo-ed out of DB Developers), correct triaging of a Database disk-usage spike is especially helpful, because a good bug-report is at times the only help that'd ensure a 1-iteration resolution.

Recently, one of our Production Database boxes hit a Nagios disk-alert and an IT personnel wanted to identify who (and specifically what) was causing this sudden spike.

From the looks of it, it clearly was a time-consuming + space-consuming Job running on the box and not much that could have been done (sans terminating it), but the following steps could have helped to prepare a bug-report of-sorts for the DB developer to identify / correct it before this happens again:

1. Isolate whether the disk increase is because of postgres. i.e. Compare the following two disk-usage outputs:
1. df -h
2. du -h --max-depth=1 /var/lib/postgresql/
2. On the psql prompt
1. SELECT
     datname AS db_name,
     pg_size_pretty(pg_database_size(oid)) AS db_size
   FROM pg_database
   ORDER BY pg_database_size(oid) DESC
   LIMIT 10; 
2. Connect to the Database. (Assuming that there is one large database on the system, and its name is X. In the above output, it'd be the first row, under db_name):
3.  \c  X
4. From Wiki: SELECT nspname || '.' || relname AS "relation",
       pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
     FROM pg_class C
     LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
     WHERE nspname NOT IN ('pg_catalog', 'information_schema')
       AND C.relkind <> 'i'
       AND nspname !~ '^pg_toast'
     ORDER BY pg_total_relation_size(C.oid) DESC
     LIMIT 10; SELECT NOW();
3. A minor addition, but sometimes, a simple thing such as appending the current Server-Timestamp is 'very' beneficial for accurate triaging. For e.g. If a given table is known to grow very big, N minutes into processing, (assuming we have at least basic logging in place) a Developer can easily identify which method to focus on.
4. In most cases, the query above (2.4) would give the list of tables that one is looking for:
1. In most cases, it turns out that the Job is expected to consume processing disk-space, and if so, the DB developer probably needs to request IT for mandatory empty-disk space for each night's periodic processing. That's a simple solution and we're done.
2. Alternatively, its possible that the Job is (unexpectedly) creating some very large tables (and probably removing it post processing). Such cases, may need optimization, which probably got aggravated purely out of DB growth. Possible, but again, we're clear about how to resolve it.
5. However, at times the tables returned doesn't cover 'all' that could consume disk-space:
1. The list of tables generated in Step 2.4 above covers:
1. (Regular) Tables
2. Temporary Tables
3. Unlogged Tables
2. However, it does not cover tables created within another session's transaction. More detailing on that given below:

Since Temporary Tables (within a Transaction) are NOT visible from another session, Query 2.4 when run by an administrator in another psql session, would not be able to show the temporary table. 

To find disk-consuming tables, an alternate approach may yield better results:

$ cd /var/lib/postgres/9.2/data/base/
$ ls -lR | awk '$5 > 1000000' | awk '{print "File: " $9 ",  Size: " $5}' | sed 's/\./ part-/g'
File: t22_3586845404,  Size: 819200000

File: 3586846552,  Size: 630161408
File: t24_3586846545,  Size: 1073741824
File: t24_3586846545 part-1,  Size: 1073741824
File: t24_3586846545 part-2,  Size: 1073741824
File: t24_3586846545 part-3,  Size: 559702016

Lets analyse this Shell command:

• When run from the data/base folder of PostgreSQL it shows which file uses most disk-space across *all* databases. If you already know which database to focus on, you may want to go inside data/base/xxxx folder and run this Shell command there instead.
• In the output we see three things:
• File 3586846552 is a large file pointing to a (non-temporary) table
• File t22_3586845404 is a large file, points to a *temporary* table but is less than 1Gb size
• File t24_3586846545 is a large file, also points to a *temporary* table and is between 3Gb and 4Gb in size, (basically because each file part is a 1Gb volume) and therefore is a good contender to be researched further.

So lets investigate file t24_3586846545 further.

From a psql prompt:
postgres=# \x
Expanded display is on.
postgres=#
WITH x AS (
  SELECT trim('t24_3586846545')::TEXT AS folder_name
),
y AS (
  SELECT
    CASE
      WHEN position('_' in folder_name) = 0
        THEN folder_name::BIGINT
      ELSE substring(folder_name
        FROM (position('_' in folder_name) + 1))::BIGINT
    END AS oid
  FROM x
),
z AS (
  SELECT
    row_to_json(psa.*)::TEXT AS pg_stat_activity_Dump,
    query AS latest_successful_query,
    array_agg(mode) AS modes,
    psa.pid
  FROM y, pg_locks join pg_stat_activity psa
    USING (pid)
  WHERE relation = oid
    AND granted
  GROUP BY psa.pid,row_to_json(psa.*)::TEXT, query
)
  SELECT *
  FROM z

UNION ALL

  SELECT
    'Doesnt look like this folder (' ||
    (SELECT folder_name FROM x) ||
    ') stores data for another session''s transaction'::TEXT,
    NULL, NULL, NULL
  FROM z
  HAVING COUNT(*) = 0;
-[ RECORD 1 ]-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
pg_stat_activity_dump   | {"datid":"2946058308","datname":"rt_ra","pid":16828,"usesysid":"16384","usename":"rms","application_name":"psql","client_addr":null,"client_hostname":null,"client_port":-1,"backend_start":"2014-12-20 18:52:47.702365+05:30","xact_start":"2014-12-20 20:01:12.630708+05:30","query_start":"2014-12-20 20:53:41.733738+05:30","state_change":"2014-12-20 20:53:41.734325+05:30","waiting":false,"state":"idle in transaction","query":"select n.nspname from pg_class c join pg_namespace n on n.oid=c.relnamespace\nwhere c.relname ='a' and n.nspname like 'pg_temp%';"}
latest_successful_query | select n.nspname from pg_class c join pg_namespace n on n.oid=c.relnamespace
                        | where c.relname ='a' and n.nspname like 'pg_temp%';
modes                   | {AccessExclusiveLock,RowExclusiveLock}
pid                     | 16828

postgres=#

The output of this SQL is probably going to be at least of some help, if the rogue table, is a table within another Transaction. I'll try to disect what this query is doing:

• The first column is basically the entire pg_stat_activity row mangled into a JSON object.
• The pid is the PID of the Postgres process that is serving the Connection that currently has a Lock on the table that identifies t24_3586846545. This way, we can know more about the connection that created this (large) table. (Please be very clear, that unless you know what you're doing, you shouldn't *kill* any postgres process from bash).
• The columns 'last_successful_query' and 'modes', are probably uninteresting to the Admin, but may be of big help to the Developer.
  

(Obviously, a hands-on IT admin needs to replace the string (t24_3586846545) in the above SQL, with the file-name that (s)he gets the most number of times, when executing the previous shell-command).

Happy triaging :) !