Harden runtime lease recovery and event publishing

[tannerlinsley]

What changed

• heartbeat active run leases while the runtime is driving workflow code
• recover expired running executions during sweep() by replaying durable checkpoints
• add best-effort live event publishing at runtime and per-call scope
• expose recovered sweep results and limits
• cover initialized and pre-initialization crash recovery across runtime and store contracts

Why

The execution-store contract already supported lease heartbeats and stale-run claims, but the runtime driver did not use them. A worker crash could therefore leave a run stuck as running, while long steps could outlive their lease. The runtime also collected core events to completion without exposing core's live publisher hook.

This makes leases operational, gives sweeps a complete crash-recovery path, and provides the live event seam needed by streaming integrations without changing durable event semantics.

Impact

Existing workflows continue to run unchanged. sweep() now returns a recovered collection and accepts maxRecoveredRuns. Valid lease durations behave as before; invalid non-positive or non-finite durations now fail before mutating the store.

Validation

• pnpm test:pr
• core and runtime durability tests
• shared store contract against in-memory, Cloudflare D1, and Drizzle Postgres
• package type checks, lint, docs checks, and builds

Summary by CodeRabbit

• New Features

  • Added recovery for interrupted workflow runs during runtime sweeps.
  • Added automatic lease heartbeats for long-running executions.
  • Added best-effort live event publishing at runtime and per-operation levels.
  • Added recovery limits and recovered-run details to sweep results.
  • Exported the event publisher type for integration use.

• Documentation

  • Updated runtime documentation with lease, recovery, sweep, and live event behavior.

• Tests

  • Added coverage for execution recovery, lease renewal, event publishing, and stale-run handling.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 70089aff-1a1f-448b-be04-48b2cf945e7a

📥 Commits

Reviewing files that changed from the base of the PR and between fa32903 and 0bc426b.

📒 Files selected for processing (10)

• .changeset/bright-runs-recover.md
• docs/api/runtime.md
• docs/guide/runtime-model.md
• packages/workflow-core/src/engine/run-workflow.ts
• packages/workflow-core/tests/engine.durability.test.ts
• packages/workflow-runtime/src/index.ts
• packages/workflow-runtime/src/runtime-driver.ts
• packages/workflow-runtime/src/types.ts
• packages/workflow-runtime/tests/contracts/workflow-execution-store.contract.ts
• packages/workflow-runtime/tests/runtime-driver.test.ts

---

📝 Walkthrough

Walkthrough

The workflow core now supports checkpoint recovery for reclaimed runs. The runtime renews active leases, recovers stale executions during sweeps, exposes recovery limits and results, and supports best-effort live event publishing through configured or per-call publishers.

Changes

Workflow runtime reliability

Layer / File(s)	Summary
Checkpoint recovery contract packages/workflow-core/src/engine/run-workflow.ts, packages/workflow-core/tests/engine.durability.test.ts, docs/guide/runtime-model.md	Recovery dispatch resumes interrupted executions from durable checkpoints, with durability tests covering single execution of checkpointed steps.
Lease normalization and heartbeats packages/workflow-runtime/src/runtime-driver.ts, packages/workflow-runtime/tests/contracts/*, packages/workflow-runtime/tests/runtime-driver.test.ts	Runtime drive paths normalize lease durations, renew active leases, stop heartbeats after completion, and propagate heartbeat failures.
Live event publishing packages/workflow-runtime/src/types.ts, packages/workflow-runtime/src/index.ts, packages/workflow-runtime/src/runtime-driver.ts, packages/workflow-runtime/tests/runtime-driver.test.ts, docs/api/runtime.md, docs/guide/runtime-model.md	Configured and per-call publishers receive live events without requiring event retention in run results; publisher failures are best-effort.
Sweep recovery orchestration packages/workflow-runtime/src/runtime-driver.ts, packages/workflow-runtime/tests/runtime-driver.test.ts, docs/api/runtime.md, docs/guide/runtime-model.md, .changeset/bright-runs-recover.md	Sweeps claim stale executions up to maxRecoveredRuns, replay them with recovery enabled, and include recovered runs in summaries and documentation.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant WorkflowRuntime
  participant ExecutionStore
  participant RunWorkflow
  participant EventPublisher
  WorkflowRuntime->>ExecutionStore: claim stale execution
  WorkflowRuntime->>RunWorkflow: drive with recovery enabled
  RunWorkflow->>ExecutionStore: replay checkpoints
  RunWorkflow-->>WorkflowRuntime: emit workflow events
  WorkflowRuntime->>EventPublisher: best-effort publish events
  WorkflowRuntime-->>WorkflowRuntime: return recovered result

Loading

Possibly related PRs

• TanStack/workflow#2: Introduced the workflow core engine entrypoint that this change extends with recovery dispatch.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: runtime lease recovery plus live event publishing.
Description check	✅ Passed	The description covers the changes, motivation, impact, and validation, though it uses custom headings instead of the template.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch taren/runtime-lease-recovery

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🚀 Changeset Version Preview

2 package(s) bumped directly, 6 bumped as dependents.

🟩 Patch bumps

Package	Version	Reason
@tanstack/workflow-core	0.0.4 → 0.0.5	Changeset
@tanstack/workflow-runtime	0.0.3 → 0.0.4	Changeset
@tanstack/workflow-cloudflare	0.0.3 → 0.0.4	Dependent
@tanstack/workflow-netlify	0.0.4 → 0.0.5	Dependent
@tanstack/workflow-railway	0.0.3 → 0.0.4	Dependent
@tanstack/workflow-store-cloudflare-d1	0.0.4 → 0.0.5	Dependent
@tanstack/workflow-store-drizzle-postgres	0.0.5 → 0.0.6	Dependent
@tanstack/workflow-vercel	0.0.4 → 0.0.5	Dependent

[pkg-pr-new]

More templates

• tanstack-workflow-cloudflare-poc
• tanstack-workflow-cloudflare-d1-poc
• tanstack-workflow-netlify-poc
• tanstack-workflow-vercel-poc
• @tanstack/template-example-react-basic
• @tanstack/template-example-react-devtools
• @tanstack/template-example-solid-basic
• @tanstack/template-example-solid-devtools

@tanstack/react-template

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/react-template@15

@tanstack/react-template-devtools

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/react-template-devtools@15

@tanstack/solid-template

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/solid-template@15

@tanstack/solid-template-devtools

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/solid-template-devtools@15

@tanstack/template

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/template@15

@tanstack/template-devtools

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/template-devtools@15

@tanstack/workflow-cloudflare

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/workflow-cloudflare@15

@tanstack/workflow-core

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/workflow-core@15

@tanstack/workflow-netlify

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/workflow-netlify@15

@tanstack/workflow-railway

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/workflow-railway@15

@tanstack/workflow-runtime

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/workflow-runtime@15

@tanstack/workflow-store-cloudflare-d1

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/workflow-store-cloudflare-d1@15

@tanstack/workflow-store-drizzle-postgres

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/workflow-store-drizzle-postgres@15

@tanstack/workflow-vercel

npm i https://pkg.pr.new/TanStack/workflow/@tanstack/workflow-vercel@15

commit: 0bc426b