How I built PyazDB, a consistent Key-Value Database

Databases sit quietly at the core of almost every system we build. APIs come and go, frameworks change every few years, but databases remain the silent force that decides whether your system feels fast, reliable, or painfully fragile. As developers, we often treat them as black boxes, choosing between options based on benchmarks, blog posts, or what worked last time, without truly understanding why they behave the way they do.

I wanted to change that for myself.

How PyazDB started?

PyazDB started as a curiosity-driven project. I wanted to explore Raft. Raft is a consensus algorithm that provides a practical way to achieve agreement in distributed systems, even in the presence of failures. I had encountered Raft multiple times while reading about systems like etcd and Consul, but it always felt abstract when viewed only through diagrams and papers.

I realized that the best way to truly understand consensus was not to implement Raft in isolation, but to embed it into something real. That “something” became a database.

A key-value store felt like the right starting point. Simple enough to reason about, yet complex enough to expose the hard problems that distributed systems try to solve: consistency, durability, replication, and failure handling. PyazDB was born as a playground where I could experiment freely, break things, and rebuild them with a deeper understanding each time.

What began as an exercise to learn Raft slowly evolved into a consistent key-value database, forcing me to think carefully about log replication, state machines, write paths, and the guarantees a database should actually provide. Along the way, I discovered that many of the design decisions we take for granted in mature databases exist for very good reasons.

How Raft powers PyazDB?

A key-value database by itself is deceptively simple. At the surface, all you need are three operations: get, set, and delete. You can back this with an in-memory map or a simple log on disk and have something working in a few hours.

The real complexity begins the moment you try to distribute it. Distribution is rarely about scale first. It is about fault tolerance, data consistency, and correctness under failure. What happens if a node crashes midway through a write? How do you make sure every replica sees updates in the same order? How do you avoid split-brain scenarios when the network misbehaves?

This is where Raft becomes the backbone of PyazDB.

Raft provides a structured way to maintain a single source of truth across multiple nodes. Instead of allowing every node to accept writes, PyazDB elects a leader using Raft. All write operations flow through this leader, which appends them to a replicated log. Only after a majority of nodes acknowledge the log entry does the write become committed.

Once committed, each node applies the operation to its local state machine, which in PyazDB is the key-value store itself. This ensures that every replica applies the same operations in the same order, giving PyazDB strong consistency guarantees. Reads can be served safely because Raft ensures that the leader’s state reflects the most up-to-date committed log. Even in the presence of node failures or network partitions, PyazDB continues to behave predictably, either making progress or failing safely rather than corrupting data.

By building PyazDB around Raft, I was able to focus on database semantics while relying on a well-defined consensus model to handle the hardest part of distributed systems: agreement.

Consistency, Availability, and Partition Tolerance in PyazDB

Once a database is distributed, it must deal with the trade-offs described by the CAP theorem. When a network partition occurs, a system can choose consistency or availability, but not both.

PyazDB makes this choice explicitly.

PyazDB is a CP system. It prioritizes consistency and partition tolerance over availability, a direct result of building on Raft. Consistency in PyazDB means that all clients observe the same data in the same order. Writes are accepted only by the Raft leader and are committed after a majority of nodes agree. Once acknowledged, a write is durable and immediately visible across the cluster.

Partition tolerance is unavoidable in real systems. Raft allows PyazDB to continue operating as long as a quorum is available. If a majority cannot be formed, PyazDB stops accepting writes instead of risking inconsistency.

This is where availability is intentionally sacrificed. During partitions or failures, PyazDB may reject requests rather than serve stale or conflicting data. This is a conscious design choice that favors correctness over always-on behavior.

By choosing consistency first, PyazDB provides strong, predictable semantics even in the presence of failures.

Write Path in PyazDB

Understanding the write path was one of the most important learning moments while building PyazDB.

When a client issues a write request, it is always routed to the current Raft leader. The leader appends the operation to its log and starts replication to follower nodes. Each follower persists the log entry and acknowledges it back to the leader.

Only after a majority of nodes acknowledge the entry does the leader mark it as committed. At this point, the write is applied to the state machine and the client receives a success response. This ensures that acknowledged writes survive failures and are never partially applied. This strict ordering guarantees linearizability but also highlights why latency increases with replication. Every guarantee comes with a cost.

Read Semantics and Safety

Reads in distributed systems are often trickier than they appear. In PyazDB, reads are served by the leader to ensure they reflect the latest committed state. Before serving a read, the leader confirms it is still authoritative and up to date. This avoids stale reads during leader changes or network partitions.

While this approach slightly limits read scalability, it keeps the semantics simple and predictable. Clients never have to reason about stale data or read inconsistencies.

Failure Handling and Recovery

Failures are not edge cases in distributed systems, they are the norm. PyazDB assumes that nodes will crash, restart, and rejoin the cluster. When a node comes back online, Raft automatically brings it up to date by replaying missing log entries. No special recovery logic is required at the database layer.

Leader failures are handled transparently. If the leader crashes, a new leader is elected and operations resume once a quorum is available. Clients may experience temporary unavailability, but correctness is never compromised.

What PyazDB Does Not Try to Solve

PyazDB is intentionally minimal.

It does not support secondary indexes, complex queries, transactions, or multi-key atomic operations. There is no sharding or horizontal partitioning. These features add significant complexity and would have distracted from the core goal of understanding consensus and replication.

By keeping the scope small, PyazDB remains easy to reason about and serves as a solid foundation for experimentation.

Lessons Learned

Building PyazDB changed how I think about databases.

Consistency is expensive but invaluable. Simplicity is a feature, not a limitation. Most importantly, distributed systems are less about clever algorithms and more about making trade-offs explicit and predictable. Many design decisions in mature databases only make sense once you experience the failure modes firsthand. PyazDB gave me that perspective.

What’s Next for PyazDB

There are many directions PyazDB could evolve in the future: snapshotting to limit log growth, read-only replicas for scalability, sharding for horizontal scale, or persistence optimizations for lower latency. Whether or not those features are added, PyazDB has already achieved its goal. It transformed abstract ideas about consensus and consistency into concrete, lived experience.