Skip to content

Make incident status canonical and derive component status from impacts#398

Draft
jbrooksuk wants to merge 4 commits into
mainfrom
feat/status-model-and-read-policy
Draft

Make incident status canonical and derive component status from impacts#398
jbrooksuk wants to merge 4 commits into
mainfrom
feat/status-model-and-read-policy

Conversation

@jbrooksuk

@jbrooksuk jbrooksuk commented Jul 25, 2026

Copy link
Copy Markdown
Member

Different parts of Cachet answered the same questions differently. This makes them agree, before incident-response features get built on top of a model that contradicts itself.

Is this incident open?

Three definitions could disagree: the status column, an accessor reading the newest update, and a banner query treating an incident as resolved if either said so. An incident could read as resolved on the banner while its own page said otherwise.

The column is now canonical. A SyncIncidentStatus action keeps it in step on create, edit and delete of updates, latestStatus becomes an alias for it, and the latest-update join disappears from Status. Update-driven status changes now apply to every status, not only fixed.

Who may see this incident?

Publication and visibility were decided independently at each read surface, and several forgot one of the two. A single viewableBy scope now decides both, and every surface goes through it — the API, MCP, RSS, the status page, the timeline, the system status and the component overlay.

That closes: embargoed incidents appearing in the RSS feed, hidden incidents being readable by guid and flipping the public banner, an unpublished incident's updates being listable by id, and the component overlay ignoring publication entirely.

What colour is this component?

The badge and the banner were computed by different rules — the badge accounted for incidents, the banner only read the raw column — so every component could show red under "All systems operational".

Component status now splits into a baseline (the column, what operators and monitoring assert) and an effective status (what the page shows). Incidents and maintenance contribute impacts on top of the baseline without ever writing to it, and the most severe wins rather than the most recent, so a minor incident can no longer mask a major one. A maintenance window in progress replaces the baseline, so expected downtime is not reported as an outage, while an incident raised during the window still surfaces. The banner is computed from the same effective statuses, and a single component under maintenance no longer masks a fleet-wide outage.

Resolution therefore restores nothing, because nothing was overwritten: the impact simply stops counting. The rewrite of every pivot to operational on resolution is gone — it destroyed the record of what an incident imposed, and only ran on one of the paths that resolve an incident.

Why did this component change?

The component_status_changed webhook only fired for API and MCP edits, because the event lived inside UpdateComponent and four of the six writers never called it — the dashboard toggle and edit form, the monitor, and the OhDear import all wrote the column directly.

All six now route through a ChangeComponentStatus command, so the event fires wherever a status changes, and every change records what asserted it (manual, monitoring, import, system), who caused it, and an optional reason.

Also

  • Maintenance windows say what each component should look like while they run, and the most severe status wins across overlapping windows. The pivot column has been written since 2016 and read by nothing; the dashboard hardcoded it to operational, so the form now offers a status defaulting to under maintenance. No scheduled job is involved — the window is derived from the clock.
  • Both component pivots gain a unique constraint on their parent and component, with a migration that collapses pre-existing duplicates to their most severe impact.
  • Resolving the current user is now the caller's job: models and actions take a user rather than calling auth(), supplied by the controllers, MCP tools, Filament actions and views entitled to know it. Writes that belong together — a status and the record explaining it, an update and the incident status it drives — are wrapped in transactions, with events dispatched after commit.
  • A null scheduled_at no longer fatals the schedule status accessor, and metric charts order by order rather than by their decimal places.

Compatibility

  • component.status in the API keeps its meaning as the baseline and its wire format. A sibling latest_status is added for the effective status.
  • component_status_changed webhook payloads gain a source field, and old_status may now be null where a component had no prior status.
  • Incidents created through the API or MCP without an explicit visible remain authenticated-only, as they always have. Because impacts are now gated on visibility, such an incident no longer changes the public status page. Pass visible: true for an incident that should be publicly visible.
  • Resolved incidents keep their recorded impact, so GET /incidents?include=components now shows the status the incident imposed rather than operational.

Verification

933 tests passing, Pint and PHPStan clean, migrations verified from a fresh build including the deduplication path against duplicate rows.

New coverage includes severity precedence over recency, the maintenance mask with a failing monitor, per-window maintenance statuses and overlapping windows, exclusion of embargoed, hidden and resolved impacts, parity between eager-loaded and lazy resolution, the read policy per surface, and the status-change log with its source and causer.

Still open

Recording or clearing an impact emits no event — only baseline changes do — so a consumer watching webhooks sees "someone set this component" but not "an incident took it down". Related: because effective status now depends on the clock, a component's public status can change with no write at all when a window opens or an embargo lifts, and nothing fires then either. Both are worth settling before anything starts consuming these events.

🤖 Generated with Claude Code

https://claude.ai/code/session_0188UuLsNZ39q6Yw8BUBVH3b

claude added 4 commits July 25, 2026 09:24
Incident status had three competing definitions: the column, an accessor
that read the newest update, and a banner query that treated an incident
as resolved if either said so. They could disagree, so an incident could
read as resolved on the banner while its own page still said otherwise.
The column is now canonical and every writer keeps it in step through a
SyncIncidentStatus action, so creating, editing and deleting an update
all leave the incident coherent. latestStatus becomes an alias for it.

Component status splits into a baseline and an effective status. The
column stays the baseline — what operators and monitoring assert — while
incidents and maintenance windows contribute impacts on top of it without
ever writing to it. The most severe impact wins rather than the most
recent, so a minor incident can no longer mask a major one, and a
maintenance window in progress replaces the baseline so expected downtime
does not surface as an outage. This also brings schedule_components'
long-dormant status column into use with no scheduled job to drive it.

Resolution therefore restores nothing, because nothing was overwritten:
the impact simply stops counting. The rewrite of every pivot to
operational on resolution is gone — it destroyed the record of what an
incident had imposed, and only ever ran on one of the paths that resolve
an incident.

Publication and visibility now go through a single viewableBy scope so no
read surface can apply one and forget the other. That closes several
leaks: embargoed incidents appeared in the RSS feed, hidden incidents
were readable by guid on the status page and flipped the public banner,
an unpublished incident's updates were listable by id, and both the
component overlay and its incident counts ignored publication entirely.
The system status is computed from effective statuses too, so incident
impacts finally reach the banner instead of it reporting all clear while
every component below it shows red.

Also fixes a null scheduled_at fataling the schedule status accessor, and
metric charts ordering by their decimal places rather than their order.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0188UuLsNZ39q6Yw8BUBVH3b
The component_status_changed webhook only ever fired for API and MCP
edits, because the event was dispatched inside the UpdateComponent action
and four of the six writers never called it — the dashboard toggle and
edit form, the monitor, and the OhDear import all wrote the column
directly. They now all go through a ChangeComponentStatus command, so the
event fires wherever a status changes and every change leaves a record of
what asserted it: manual, monitoring, an import, or the system, with the
user who did it and an optional reason.

The monitor no longer overwrites a component's status while a maintenance
window is in progress. The check is still recorded, but expected downtime
during a window is not an outage, and reporting it as one is what the
window exists to prevent.

Schedules can now say what a component looks like during maintenance. The
pivot column has been written since 2016 and read by nothing; the
dashboard hardcoded it to operational, which under the new severity rules
would never win, so the form now offers a status and defaults it to under
maintenance.

Both pivots gain a unique constraint on their parent and component, since
these rows now decide what the status page displays. Rows that predate it
are collapsed to their most severe impact, which is the one that would
have been shown anyway.

The component API resource gains latest_status alongside status, so
consumers can read the effective status without inferring it. The
existing status field keeps its meaning as the baseline.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0188UuLsNZ39q6Yw8BUBVH3b
Resolving the current user is the caller's job. A model or an action that
reaches for auth() couples domain logic to an HTTP session it should know
nothing about, and makes the same call answer differently depending on
where it runs. ComponentGroup::hasActiveIncident and isExpanded now take
the viewer, and CreateUpdate and UpdateComponent take the user who acted,
supplied by the controllers, MCP tools, Filament actions and views that
are entitled to know it.

Passing the viewer also settles a disagreement the group expansion had
with itself: the eager-loaded incident count is scoped to guests, so it
is only consulted for a guest viewer and anything else is answered with a
query scoped to that viewer.

Writes that belong together are now wrapped in transactions. A component's
status and the record explaining it are written as one, with the event
announced only once both have committed; an update and the incident
status it drives are likewise written together, so a failure part-way
cannot leave an update whose incident disagrees with it.

Maintenance windows now use the status each one asked for, rather than
assuming every window means under maintenance. The picker added for this
was writing a value nothing read, because the component side of the
relation never used the ScheduleComponent pivot and so never cast it. The
most severe status wins across overlapping windows.

Attaching components to a schedule was broken: the attach action's schema
closure took an array, but Filament injects a Schema object for that
parameter, and overriding the schema without restoring the record select
would have removed the component picker even once the type was right.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0188UuLsNZ39q6Yw8BUBVH3b
An incident's status used to live in its updates, and only a fixed update
ever wrote the column. Making the column canonical would therefore have
reverted every in-flight incident on existing installs to the status it
was opened with, so a migration aligns it with the newest status-bearing
update, by the same rule the application now applies. It leaves incidents
without status-bearing updates alone, and does not disturb timestamps.

The MCP incident tools were the one read surface the new scope claimed to
cover but did not: they applied visibility without publication, so any
caller could read an embargoed incident. They now go through the same
scope, and seeing drafts requires a token that can manage incidents —
which is what made it a management surface in the first place.

Duplicate components in a create payload were reaching the new unique
constraint as a 500, with the incident already committed. They are now
rejected in validation, the pivot list is keyed by component so the sync
cannot depend on array position, and incident and schedule creation are
wrapped in transactions so a failure leaves nothing behind.

A single edit was saving the component twice — once for its fields and
once for its status — firing two ComponentUpdated events and two webhook
deliveries. Anything accompanying a status change is now written in the
same statement, which also removes the intermediate null-status row the
Oh Dear import was creating for new components.

The status overview no longer loads every enabled component. The database
counts the baseline, and only components actually carrying an impact are
loaded and moved into their effective bucket. The components API and the
MCP component tools eager-load the two relations that effective status
reads, which were costing two queries per component.

Finally, the status badge no longer links to an incident when the incident
is not what it is reporting: a monitor-set baseline or a maintenance
window can outrank every impact, and the link now reflects that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0188UuLsNZ39q6Yw8BUBVH3b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants