SQLite is about as close to “boring, reliable” as a database gets. It sits inside phones, browsers, and countless desktop apps, and most developers never think about it twice. So when Tailscale, a company that runs a mesh VPN used by thousands, started seeing their SQLite databases get corrupted, the last thing anyone expected was a 16-year-old bug at the root of it.
Over six months, Tailscale hit 19 separate instances of database corruption. That is not a normal failure pattern. Databases that quietly rot over months, with no obvious trigger, are the hardest kind to debug because you cannot reproduce them on demand. You can only stare at the wreckage afterward.
What was actually going wrong
The corruption traced back to a data race in SQLite’s Write-Ahead Logging, or WAL, mechanism. WAL is the mode where changes are written to a separate log file before being merged back into the main database during a process called checkpointing. It is a well-trodden path; SQLite has been running WAL mode in production for years.
The bug was a time-of-check to time-of-use race, a TOCTOU for short. Between the moment SQLite checked the state of the WAL and the moment it acted on that state, another thread could slip in and change it. In most workloads the window never opens. Tailscale’s workloads found it.
The key detail is that Tailscale was not doing anything exotic with the database itself. What made it vulnerable was their checkpointing strategy. They were checkpointing aggressively and manually to support quick backups, which put far more pressure on the WAL reset path than the typical embedded use does. Standard usage almost never trips this kind of race. Push the checkpointing hard enough and the 16-year-old bug finally surfaces.
How they found it
What makes this story worth reading is not just the bug. It is how long it took to find, and what finally cracked it.
Tailscale built custom filesystem shims at the VFS layer and instrumented every transaction to log what was happening. That level of observability, transaction logging plus VFS monitoring, is far beyond what you get from normal application logging. Even with all that instrumentation, the team still could not reproduce the corruption on demand.
Eventually they turned to a tool called Antithesis, a platform built for deterministic testing. Antithesis loaded a generic SQLite workload and reproduced the bug in roughly fifteen minutes. Months of forensic debugging collapsed into a quarter hour of automated testing once the right tool was pointed at it.
What the fix looked like
The fix landed in SQLite 3.51.3 and 3.52.0, with a follow-up correction in 3.53.0 after the initial patch introduced a false alarm. If you are running SQLite in WAL mode, this is a reminder to check your version and upgrade if you are on anything older than 3.51.3.
For most people, the practical takeaway is simpler: the bug only bites when you push the database outside the mainstream usage pattern. If you are using SQLite the way everyone else does, with the default checkpointing, you are very unlikely to hit it. The danger lives in the non-standard operations, the aggressive manual checkpointing, the unusual configuration that a confident engineer decided was fine.
What this says about boring technology
There is a real lesson buried here about how we treat mature open source software. SQLite has been around for over two decades and is one of the most tested codebases in existence. And still, a race condition hid in it for 16 years, survivable only because ordinary usage never triggered it.
“Use boring, proven technology” is good advice right up until the day the boring technology turns out to have a hidden edge case nobody has ever hit. The counterweight is observability. When you cannot afford to lose the data, you need enough visibility into the internals to find the problem when it eventually appears, not just enough logging to know that something broke.
Practical takeaways
If you are running SQLite in WAL mode, a few things are worth doing:
Check your SQLite version and confirm you are on 3.51.3 or later. The affected code path has been fixed, but only in current releases.
Reconsider aggressive manual checkpointing. If you are checkpointing on a tight timer or for backup convenience, weigh whether that complexity is earning its keep. The default behavior exists for a reason.
Invest in real observability for critical databases. Transaction-level logging and VFS monitoring are heavy, but they are what turned a months-long investigation into a solvable one.
Know your dependencies. If you are relying on a core open source library in production, understanding its supported configuration and staying current with its releases is part of operating it, not an optional extra.
The Tailscale writeup is a good read, and Antithesis by the company behind it published its own account of reproducing the bug. For anyone who has ever stared at a database that quietly corrupted itself, the takeaway is the same as it always is: you cannot debug what you cannot see, and you cannot fix what you never reproduce.
Upgrade SQLite, simplify the aggressive checkpointing, and add the logging that would have caught this sooner. That is about all any of us can do.