An end-to-end-encrypted, multi-node replicating database engine that quietly runs everything here — built on the same file architecture as Moneylender.
Open nowOnyx was a natural extension of the encrypted file system I built for Moneylender Professional. As I started new projects, the choice to use a SQL database on a project got harder as my own file system became easier to set up and configure. About a dozen lines of code could stand up a database from scratch, and it was internally multi-threaded and built to answer to multiple concurrent readers and writers.
I started working on the encrypted file system in 2006, and in 2008 the first “production system” built on my new file system went live. The only user of that production system was my mom for free, and that system went offline just a couple years later. But I built more things and used my file system as the main data storage. Moneylender officially switched from the old and very flawed system to this encrypted file system with the release of Moneylender 3 in 2017, and that was pretty much the point where all the systems at Whitman Technological were running on my hand rolled database.
The internal classes that get shared across almost all my projects — the file system, the encrypted network system, and several common converters and utility classes — are grouped into projects with the namespace Nyx. When choosing a name for the formalized database service of the encrypted file system, Onyx seemed quite fitting.
The database file on disk is encrypted. The write-ahead log is encrypted. The topology file is encrypted. Every wire hop — client to server, server to peer, admin to dashboard — runs inside an encrypted connection. At no point does your data exist in the clear outside the endpoints' hands.
Below is the real thing, not a mock-up: your message is AES-encrypted nine separate times by this very server's code-behind — once for every wire and file it touches crossing two Onyx servers on its way to a remote reader, each locus under its own key. Then stand where a thief would stand: hover the wires and the files, and see exactly what a breach would hand them.
Send a message and follow one record through nine separate locks — then stand where a thief would, and see what they'd get.
There is no primary server, and no failover. Servers link pairwise with a PIN ceremony, every replica accepts writes, and conflicts settle deterministically: last writer wins, decided by hybrid-logical-clock version stamps. Two servers can edit the same record at the same moment and land on the same answer.
Every replica accepts writes. Tap a node — or press Q · W · E — to write a record; it gossips to every other replica and lands on each stack.
Every committed edit appends to a segmented, encrypted write-ahead log. Each peer holds a cursor marking what it has confirmed; an interrupted push resumes exactly where it stopped, and once every peer confirms a segment, the whole segment is deleted — no rewriting, no bookkeeping debt.
Each server appends every edit to its encrypted WAL and streams it to its peer. Pull the cable — unsent edits pile in the outbox; reconnect and the backlog drains from each peer's cursor until the stacks clear.
Every thirty minutes, each pair of replicas plays a three-level fingerprint game — one table digest, 256 buckets, then record listings — that costs almost nothing when they match and surgically repairs them when they don't. Replication isn't assumed to work; it is proven, continuously.
Replication isn't assumed to work; it is proven, continuously. Every pass compares record fingerprints — (key, version, tombstone) — so replicas verify each other without reading a single value.
One bad record among 1,048,576 is found in ~268 comparisons — the cost scales with the damage, not the data.
If we want to stand up a new database, how do we get it into the Onyx network so the servers know to share data with each other? What if we have a system that’s getting drilled with users and we want to add more peers to the Onyx network to help shoulder the load?
For a new database, running the app that spawns the database on the first server will generate a database with the target schema in Onyx automatically. Then we can go to a peer server and select the new database for replication and it’ll transmit a snapshot of the database.
The source server can still take read and write requests while sending a snapshot to a remote peer. It tracks those edits in the WAL while the replica is copying. Once the remote peer is up and running, replication starts and the backlog of edits quickly synchronizes the new peer so it immediately reflects all the recent changes. If there are other peers, they will quickly detect the new replica, start a peer cursor in the WAL, and perform an anti-entropy pass to ensure all its local edits are also represented on the new replica.
A new replica isn't configured — it's grown. You run us-east-1; bring eu-west-1 to life, step by step.
Once two servers are linked together, they will monitor that link constantly. If the link is broken, the servers will make periodic attempts to relink until they’re successfully reconnected. The same goes for replication, which rides the server’s link to send replication details between databases. If a link is lost, replication pauses for the missing peer so it knows exactly which new records the other server doesn’t know about yet. When the connection comes back online, each database will resume replication, and the servers quickly catch each other up with all the latest gossip.
Additionally, in a mesh with more than two nodes, servers will check for databases they hold that are also replicated between their peers. Say servers A and B replicate the database “Museum”, and server C links to A and begins replicating Museum also. Once B and C are also linked, B will notice that C has that same “Museum” database and will initiate replication between B and C automatically. In a large mesh of servers, each server only needs to bootstrap the database once from a peer and all the interlinking connections from other peers will be automatically created.
Three servers co-host ‘museum’, every pair streaming. Click any link to cut it — the two ends drift apart (replication doesn't cascade, so their neighbor can't relay). Then watch the enrollment tick re-wire the mesh, every time.
What happens to a cluster of databases when I'm rolling out an update to a service that includes schema changes? Maybe I’m adding a new “credit card number” field to a Customer table. I update one server so it has that field in the record structure, but the other servers are still using the old record format until they also get updated. Sizes and alignment of data are wrong in the stream of record values, and mishandling this situation would quickly corrupt the real records with gibberish. We could bring all the servers down, push updates, and then bring them all back up and hope the new schema lands at the same time on all the servers. But that’s a substantial outage (at least a couple minutes if everything goes smoothly) for something that might happen several times a week.
To solve this, every replication batch carries a schema stamp — a 64-bit fold of every table, field, and type. When a schema change happens on a database, the server wipes its WAL and resets its cursor to clear the old format records that might have been lingering there. Each server shares the schema stamp when sending replication data. If a stamp doesn’t match the recipient server’s expectations, it refuses the records and the other server tries again a little later. Each server is eventually upgraded to the new schema and they all share records again.
But what happens to writes on an outdated server when its peers have already been updated? Or if the server’s under heavy load and there are a lot of unsynchronized records in the WAL? That’s what’s great about the anti-entropy system. It detects that one server knows about certain records and the other servers don’t and synchronizes them automatically. After each schema change, a server will run its anti-entropy process, so those lost old-format edits are immediately discovered and transferred to the neighbors. If the neighbor hasn’t been upgraded yet, it rejects the anti-entropy request. As soon as that neighbor is upgraded, it starts its own anti-entropy pass – and since both servers now match schema, they exchange all the missed records.
All servers remain fully authoritative, replication pauses between mismatched servers, and the rift is immediately healed when the upgrade finishes across all servers.
Roll the v2 upgrade across the mesh in any order you like. A version-mismatched link refuses batches and simply holds — edits pool safely in the WAL until the stamps match again. The worst case is a wait, never a corruption.
A delete leaves a tombstone, so a stale replica can't push a dead record back to life. Once every peer has confirmed the delete and a grace period has passed, the tombstone itself is reclaimed and the space goes back to work.
Race edits against deletes. Delete a record on one server, then try to rescue it on another before the tombstone lands — an edit that out-versions death wins everywhere. Delete too late, and the stale write bounces off the gravestone.
A bare C# class is the schema: every property becomes a stored, queryable field. Point the client at a server and the templates conform the live database file to the code — preserving keys, versions, and tombstones so replicas stay provably comparable across upgrades.
A bare C# class is the schema. Add a property and watch the live database conform itself — columns appear, defaults fill in, and the version stamps never move.
The Onyx layers are a chocolate replication shell around the encrypted file system’s encryption nougat. Each database is a standalone encrypted file on disk. Each database has its own processor threads, so its code runs on the CPU like it was a standalone program. Each database has its own file access streams, index management and retrieval threads, network listener, WAL and peer cursors. That means that every Onyx database gets a fair share of the physical computing resources available, each one is first in it's own line at the OS kiosk for RAM or disk. They can be independently offlined and onlined, maintained, backed up and cloned. Databases aren't mini-sized bars in a bulk Onyx packaging. Onyx is a retail-sized crate of the full-size bars.
Every database on an Onyx server is an island — its own port, users, threads, encryption, WAL. Run month-end on loans, then flip the architecture and run it again. The same flood becomes a private problem.
The standard proof ritual: 15 concurrent writer, rewriter, and deleter threads hammer three servers in thirty-seconds-on, ten-off cycles — then anti-entropy must report the replicas identical, and summing every numeric field on all three machines must agree to the digit. Sometimes we disable replication in the middle so edits from one of the servers remain local only. Sometimes we pull the network cable out of the back of one of the servers to simulate an outage.
Microseconds add up when you're a database and you have to perform operations millions of times. Throughout torture testing, the biggest resource consumers were ruthlessly hunted and eliminated, one-by-one until the system ended up being 3x to 10x faster overall on every operation it supports. After deploying these core updates to our portfolio hosting service, the disk-intensive nightly task that would take about 1500 seconds every morning dropped to 530 seconds.
The numbers from the workshop's own three-server testing setup, with machines ranging from 4 cores to 24: index-heavy edits logging over 9,000 updates per second across the three servers, landing in tables with over two million records each. The test database is over 5 GB of encrypted test records. Three record types were used, a small, medium, and large. All types had multiple indexed fields, but the large record had every combination of data type and index type the file system supports to ensure performance coverage across the entire codebase.
Onyx ends every change to its replication engine the same way: the torture test. Hold the button. Try to break it. Then watch it prove — to the digit — that you didn't.
