SQLite for Durable Workflows: Hype or Practical Choice?

The Hacker News thread "SQLite is all you need for durable workflows" (496 points, 250 comments) sparked a fierce debate. The article claims SQLite can replace heavier tools like Temporal, Airflow, or custom state machines for many workflow orchestration use cases. Is it hype or a practical choice?

The Core Argument for SQLite Workflows

The original article makes a straightforward case: if your workflows are single-process, or you can accept some downtime for failover, SQLite's durability guarantees are sufficient. The author builds a simple workflow engine on top of SQLite, using its transactional semantics to ensure exactly-once execution, retries, and idempotency. The pitch is avoiding the operational complexity of a dedicated workflow service while still getting reliable execution.

Community Reactions: Simplicity vs Scalability

The thread is a microcosm of the ongoing debate. One camp cheers the reduction in complexity. A commenter wrote:

"The cycle of expertise: ... maybe Y is enough ... I use Y for literally everything"

They highlight how experience often leads back to simpler tools. Another shared their success replacing multiple SaaS tools with Go + SQLite on a single server, slashing costs by 90%.

But skeptics are equally loud. A top-voted counterpoint:

"I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency."

This tension between "it works for me" and "it won't scale" is at the heart of the debate. The article acknowledges trade-offs, but the HN crowd isn't convinced SQLite is a genuine replacement for systems like Temporal, which a commenter noted uses SQLite for local development and offers a rich UI.

When SQLite Works for Durable Workflows

Both sides are right but speaking past each other. The article's claim isn't universal superiority—it's that for durable workflows in many real-world scenarios, SQLite is good enough.

The key insight: workflow engines are primarily about reliability and state management, not distributed consensus. Most business workflows (email notifications, order processing, data pipeline retries) fit into a single process or tolerate brief failover pauses. SQLite gives ACID compliance with zero operational overhead. No dedicated database server needed.

But critics have a point about concurrency. SQLite's write lock serializes transactions, so hundreds of concurrent workflow steps cause contention. The solution isn't abandoning SQLite—it's properly scoping concurrency. Use connection pooling, batch writes, or adopt WAL mode for concurrent reads. For many applications, throughput is perfectly adequate.

Where I see the line: if you need strict horizontal scaling or geographic distribution, SQLite alone isn't enough. But most teams overestimate scaling needs. A single well-tuned server with SQLite can handle thousands of workflows per second.

Practical Advice for Builders

Before reaching for Temporal, Airflow, or a custom Postgres-backed queue, consider these questions:

1. Are your workflows single-process? If you can run on one machine (with failover), SQLite is a strong candidate.
2. Do you need sub-millisecond latencies? SQLite's overhead is minimal; the bottleneck is usually business logic, not the database.
3. Can you partition workflows? Shard by tenant or region using separate SQLite instances. This pattern works well for multi-tenant SaaS.

Here's a minimal example of a durable workflow using SQLite (conceptual):

import sqlite3
import time

def execute_workflow(conn, workflow_id):
    conn.execute("UPDATE workflows SET status = 'running' WHERE id = ?", (workflow_id,))
    try:
        # Do work (e.g., call external API, process data)
        time.sleep(1)
        conn.execute("UPDATE workflows SET status = 'completed' WHERE id = ?", (workflow_id,))
        conn.commit()
    except Exception as e:
        conn.execute("UPDATE workflows SET status = 'failed', error = ? WHERE id = ?", (str(e), workflow_id))
        conn.commit()

By using SQLite transactions, you ensure atomicity between workflow state updates and work execution (or at least idempotency). This is the core idea.

For more on scaling SQLite, explore LiteFS for distributed replication, or Turso for edge-deployed SQLite databases. These projects show SQLite can be the backbone of a resilient architecture.

Final Verdict

Should you care? Yes, if you're building workflow systems and want to reduce operational complexity. No, if you need native cross-region replication or already have a solid Postgres-based solution. The debate is a healthy reminder that the best tool depends on your constraints. Don't let the "SQLite for everything" crowd oversell, but don't dismiss it out of hand. Try it for a small workflow first—you might be surprised how far it takes you.

---

Links: HN Thread | Original Article | SQLite WAL | Temporal | LiteFS | Turso