Background workflows with Hatchet

Long-running operations like cluster initialization and deployment rollouts are now powered by Hatchet, an open-source durable execution engine.

Why Hatchet#

We needed a workflow engine that could handle multi-step infrastructure operations reliably. Hatchet stood out for a few reasons:

• TypeScript-native: Workflows are defined in the same language as the rest of our stack, no YAML or DSLs
• Durable execution: If a step fails mid-workflow, it retries from where it left off rather than restarting from scratch
• DAG-based workflows: Steps can run in parallel or sequence with explicit dependency graphs
• Step-level visibility: You can see exactly which step a workflow is on and what failed

What changed#

Cluster setup, deployment rollouts, and resource cleanup now run as Hatchet workflows. Each step has its own retry policy and timeout, so transient failures (like a cloud API being slow) are handled automatically without failing the entire operation.

Progress is tracked per-step, so the UI always reflects exactly where things stand.