Append-only file: Crash-safe writes

Before the code: what crash-safety actually means

Your store is fast because it lives in memory. The price of that speed is that a single power cut wipes everything. This module fixes that. The trick is simple: keep a flight recorder. Every write the server accepts is appended to a file on disk. After a crash, you replay the file from the top and the server is back. That is the entire append-only log idea, used inside Redis, every SQL database, Kafka, even git.

Write to memory, log to disk

Every write goes two places at once: the in-memory dict for fast reads, and the log file so the dict can be rebuilt after a crash.

Quiz: Quiz

Loading practice…

Append-only logs are the workhorse of durability. They have one failure mode (truncated tail) and that mode is recoverable. Database WALs, Kafka topics, git commits, Redis AOF all use the same pattern.

05-aof-persistence/server.py
python
def aof_append(frame):
    with open(AOF_PATH, "ab") as f:
        f.write(encode_array(frame))

def aof_replay():
    if not AOF_PATH.exists(): return
    data = AOF_PATH.read_bytes()
    cursor, n = 0, 0
    while cursor < len(data):
        r = parse_resp(data[cursor:])
        if r is None: break  # partial trailing frame
        frame, consumed = r
        cursor += consumed
        if isinstance(frame, list):
            handle_command(frame, persist=False)
            n += 1
    print(f"  [aof] replayed {n} commands")

aof_append writes one RESP-encoded command. aof_replay reads the whole file on startup, dispatching each frame. Partial trailing bytes are dropped.

We store the exact RESP bytes the client sent. No new format, no new parser. Replay reuses parse_resp. AOF rewrite (compaction) is trivial: walk the current store, emit SET commands, atomically replace the file.

Open in append mode plus write does NOT fsync. Bytes sit in OS page cache until the kernel decides. A power loss can lose those bytes. Real Redis offers three policies: always (fsync after every command, safe + slow), everysec (fsync once per second, the default), no (let the OS decide). Add os.fsync(f.fileno()) after the write for always semantics.

Quiz: Quiz

Loading practice…