Skip to content

fix(orchestrator): preserve conclude DLQ outcomes#426

Draft
albertywu wants to merge 1 commit into
wua/refactor-request-terminationfrom
wua/refactor-dlq-termination
Draft

fix(orchestrator): preserve conclude DLQ outcomes#426
albertywu wants to merge 1 commit into
wua/refactor-request-terminationfrom
wua/refactor-dlq-termination

Conversation

@albertywu

@albertywu albertywu commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes a correctness bug in conclude_dlq recovery and reuses the shared request terminalization helper introduced in #424.

conclude intentionally consumes a BatchID after an upstream controller has persisted the batch's terminal outcome. The message carries only the identity; conclude reloads the batch and maps Succeeded, Failed, or Cancelled to the corresponding request outcome. If conclude exhausts its retries, the same BatchID reaches conclude_dlq while the batch remains terminal.

Previously, conclude_dlq used the generic batch-failure reconciler. That reconciler intentionally skips batches already in Succeeded or Cancelled to avoid overwriting a competing terminal outcome. This behavior is correct for normal pipeline DLQs, but it is a bug for conclude_dlq, where a terminal batch is the expected input. A partially completed conclude fanout could therefore be acknowledged without reconciling the remaining request states or repairing their logs.

This PR gives conclude_dlq outcome-preserving recovery:

  • Succeeded batch -> reconcile member requests to Landed
  • Failed batch -> reconcile member requests to Error
  • Cancelled batch -> reconcile member requests to Cancelled

Normal pipeline DLQs remain failure-oriented: requests reconcile to Error, while batches reconcile to Failed and their member requests to Error.

Scope: This is a targeted bug fix and code deduplication within the existing architecture. It does not introduce new topics, payloads, stages, stores, or request-log paths, so an RFC is not needed.

Before After
DLQs duplicated request terminal-state and log-publication logic. DLQs route request termination through reconcileRequest, which calls request.ReconcileTerminalState; request-scoped DLQs do this once, while batch-scoped DLQs do it once for each member request.
Terminal DLQ logs copied dlq.last_error into LastError, but passed nil for RequestLog.Metadata, so structured delivery context was dropped. DLQs pass a TerminalOutcome to request.ReconcileTerminalState, keeping dlq.last_error in LastError and adding dlq.original_topic, dlq.failure_count, and dlq.failed_at to Metadata when available.
conclude intentionally received IDs for terminal batches, but conclude_dlq used generic batch-failure reconciliation. That logic skipped Succeeded and Cancelled batches, so a partially completed fanout could be acknowledged without reconciling the remaining requests or repairing their logs. concludeBatch treats the existing terminal batch state as the source of truth and calls request.ReconcileTerminalState through reconcileRequest for every member request, completing or repairing the expected fanout.

Helper call path: In every After case above, each request terminal-state reconciliation is performed by request.ReconcileTerminalState. Request-scoped DLQs call it through reconcileRequest; batch and conclude DLQs call reconcileRequest once for each member request. Batch failure still uses BatchStore.UpdateState, while conclude DLQ preserves the batch's existing terminal state.

ReconcileTerminalState usage

DLQ call path Terminal outcome Purpose
Request-scoped DLQ through reconcileRequest Error Terminates the affected request and records available DLQ context.
Batch-scoped DLQ through failBatch Error for each member request Fails the batch and reconciles its request fanout.
Conclude DLQ through concludeBatch Batch Succeeded, Failed, or Cancelled maps to request Landed, Error, or Cancelled Repairs incomplete conclude fanout without changing the batch outcome.

Future callers should use this helper only after deciding that a request must enter a terminal state and needs the matching public request log. The caller owns the business decision and supplies State, LastError, and Metadata; the helper owns the CAS, version advancement, state-to-status mapping, idempotency, and log publication. Do not use it for non-terminal request transitions, batch transitions, or log-only events.

Test Plan

make lint && make check-tidy && make check-gazelle && make test

Revert Plan

Revert this PR. This restores the previous generic conclude_dlq behavior and DLQ-specific request termination logic; #424 can remain independently.

Stack

  1. refactor(orchestrator): centralize request termination #424
  2. @ fix(orchestrator): preserve conclude DLQ outcomes #426

Summary:
Intent:
- Keep DLQ reconciliation consistent with the terminal outcome already selected by conclude.
- Preserve useful failure context in terminal request logs.

Changes:
- Reuse the parent request termination helper across DLQ reconciliation.
- Preserve successful, failed, and cancelled batch outcomes in conclude DLQ fanout.
- Record original topic and failure metadata on DLQ terminal logs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant