Make incident status canonical and derive component status from impacts#398
Draft
jbrooksuk wants to merge 4 commits into
Draft
Make incident status canonical and derive component status from impacts#398jbrooksuk wants to merge 4 commits into
jbrooksuk wants to merge 4 commits into
Conversation
Incident status had three competing definitions: the column, an accessor that read the newest update, and a banner query that treated an incident as resolved if either said so. They could disagree, so an incident could read as resolved on the banner while its own page still said otherwise. The column is now canonical and every writer keeps it in step through a SyncIncidentStatus action, so creating, editing and deleting an update all leave the incident coherent. latestStatus becomes an alias for it. Component status splits into a baseline and an effective status. The column stays the baseline — what operators and monitoring assert — while incidents and maintenance windows contribute impacts on top of it without ever writing to it. The most severe impact wins rather than the most recent, so a minor incident can no longer mask a major one, and a maintenance window in progress replaces the baseline so expected downtime does not surface as an outage. This also brings schedule_components' long-dormant status column into use with no scheduled job to drive it. Resolution therefore restores nothing, because nothing was overwritten: the impact simply stops counting. The rewrite of every pivot to operational on resolution is gone — it destroyed the record of what an incident had imposed, and only ever ran on one of the paths that resolve an incident. Publication and visibility now go through a single viewableBy scope so no read surface can apply one and forget the other. That closes several leaks: embargoed incidents appeared in the RSS feed, hidden incidents were readable by guid on the status page and flipped the public banner, an unpublished incident's updates were listable by id, and both the component overlay and its incident counts ignored publication entirely. The system status is computed from effective statuses too, so incident impacts finally reach the banner instead of it reporting all clear while every component below it shows red. Also fixes a null scheduled_at fataling the schedule status accessor, and metric charts ordering by their decimal places rather than their order. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188UuLsNZ39q6Yw8BUBVH3b
The component_status_changed webhook only ever fired for API and MCP edits, because the event was dispatched inside the UpdateComponent action and four of the six writers never called it — the dashboard toggle and edit form, the monitor, and the OhDear import all wrote the column directly. They now all go through a ChangeComponentStatus command, so the event fires wherever a status changes and every change leaves a record of what asserted it: manual, monitoring, an import, or the system, with the user who did it and an optional reason. The monitor no longer overwrites a component's status while a maintenance window is in progress. The check is still recorded, but expected downtime during a window is not an outage, and reporting it as one is what the window exists to prevent. Schedules can now say what a component looks like during maintenance. The pivot column has been written since 2016 and read by nothing; the dashboard hardcoded it to operational, which under the new severity rules would never win, so the form now offers a status and defaults it to under maintenance. Both pivots gain a unique constraint on their parent and component, since these rows now decide what the status page displays. Rows that predate it are collapsed to their most severe impact, which is the one that would have been shown anyway. The component API resource gains latest_status alongside status, so consumers can read the effective status without inferring it. The existing status field keeps its meaning as the baseline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188UuLsNZ39q6Yw8BUBVH3b
Resolving the current user is the caller's job. A model or an action that reaches for auth() couples domain logic to an HTTP session it should know nothing about, and makes the same call answer differently depending on where it runs. ComponentGroup::hasActiveIncident and isExpanded now take the viewer, and CreateUpdate and UpdateComponent take the user who acted, supplied by the controllers, MCP tools, Filament actions and views that are entitled to know it. Passing the viewer also settles a disagreement the group expansion had with itself: the eager-loaded incident count is scoped to guests, so it is only consulted for a guest viewer and anything else is answered with a query scoped to that viewer. Writes that belong together are now wrapped in transactions. A component's status and the record explaining it are written as one, with the event announced only once both have committed; an update and the incident status it drives are likewise written together, so a failure part-way cannot leave an update whose incident disagrees with it. Maintenance windows now use the status each one asked for, rather than assuming every window means under maintenance. The picker added for this was writing a value nothing read, because the component side of the relation never used the ScheduleComponent pivot and so never cast it. The most severe status wins across overlapping windows. Attaching components to a schedule was broken: the attach action's schema closure took an array, but Filament injects a Schema object for that parameter, and overriding the schema without restoring the record select would have removed the component picker even once the type was right. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188UuLsNZ39q6Yw8BUBVH3b
An incident's status used to live in its updates, and only a fixed update ever wrote the column. Making the column canonical would therefore have reverted every in-flight incident on existing installs to the status it was opened with, so a migration aligns it with the newest status-bearing update, by the same rule the application now applies. It leaves incidents without status-bearing updates alone, and does not disturb timestamps. The MCP incident tools were the one read surface the new scope claimed to cover but did not: they applied visibility without publication, so any caller could read an embargoed incident. They now go through the same scope, and seeing drafts requires a token that can manage incidents — which is what made it a management surface in the first place. Duplicate components in a create payload were reaching the new unique constraint as a 500, with the incident already committed. They are now rejected in validation, the pivot list is keyed by component so the sync cannot depend on array position, and incident and schedule creation are wrapped in transactions so a failure leaves nothing behind. A single edit was saving the component twice — once for its fields and once for its status — firing two ComponentUpdated events and two webhook deliveries. Anything accompanying a status change is now written in the same statement, which also removes the intermediate null-status row the Oh Dear import was creating for new components. The status overview no longer loads every enabled component. The database counts the baseline, and only components actually carrying an impact are loaded and moved into their effective bucket. The components API and the MCP component tools eager-load the two relations that effective status reads, which were costing two queries per component. Finally, the status badge no longer links to an incident when the incident is not what it is reporting: a monitor-set baseline or a maintenance window can outrank every impact, and the link now reflects that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188UuLsNZ39q6Yw8BUBVH3b
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Different parts of Cachet answered the same questions differently. This makes them agree, before incident-response features get built on top of a model that contradicts itself.
Is this incident open?
Three definitions could disagree: the
statuscolumn, an accessor reading the newest update, and a banner query treating an incident as resolved if either said so. An incident could read as resolved on the banner while its own page said otherwise.The column is now canonical. A
SyncIncidentStatusaction keeps it in step on create, edit and delete of updates,latestStatusbecomes an alias for it, and the latest-update join disappears fromStatus. Update-driven status changes now apply to every status, not onlyfixed.Who may see this incident?
Publication and visibility were decided independently at each read surface, and several forgot one of the two. A single
viewableByscope now decides both, and every surface goes through it — the API, MCP, RSS, the status page, the timeline, the system status and the component overlay.That closes: embargoed incidents appearing in the RSS feed, hidden incidents being readable by guid and flipping the public banner, an unpublished incident's updates being listable by id, and the component overlay ignoring publication entirely.
What colour is this component?
The badge and the banner were computed by different rules — the badge accounted for incidents, the banner only read the raw column — so every component could show red under "All systems operational".
Component status now splits into a baseline (the column, what operators and monitoring assert) and an effective status (what the page shows). Incidents and maintenance contribute impacts on top of the baseline without ever writing to it, and the most severe wins rather than the most recent, so a minor incident can no longer mask a major one. A maintenance window in progress replaces the baseline, so expected downtime is not reported as an outage, while an incident raised during the window still surfaces. The banner is computed from the same effective statuses, and a single component under maintenance no longer masks a fleet-wide outage.
Resolution therefore restores nothing, because nothing was overwritten: the impact simply stops counting. The rewrite of every pivot to
operationalon resolution is gone — it destroyed the record of what an incident imposed, and only ran on one of the paths that resolve an incident.Why did this component change?
The
component_status_changedwebhook only fired for API and MCP edits, because the event lived insideUpdateComponentand four of the six writers never called it — the dashboard toggle and edit form, the monitor, and the OhDear import all wrote the column directly.All six now route through a
ChangeComponentStatuscommand, so the event fires wherever a status changes, and every change records what asserted it (manual, monitoring, import, system), who caused it, and an optional reason.Also
operational, so the form now offers a status defaulting to under maintenance. No scheduled job is involved — the window is derived from the clock.auth(), supplied by the controllers, MCP tools, Filament actions and views entitled to know it. Writes that belong together — a status and the record explaining it, an update and the incident status it drives — are wrapped in transactions, with events dispatched after commit.scheduled_atno longer fatals the schedule status accessor, and metric charts order byorderrather than by their decimal places.Compatibility
component.statusin the API keeps its meaning as the baseline and its wire format. A siblinglatest_statusis added for the effective status.component_status_changedwebhook payloads gain asourcefield, andold_statusmay now be null where a component had no prior status.visibleremain authenticated-only, as they always have. Because impacts are now gated on visibility, such an incident no longer changes the public status page. Passvisible: truefor an incident that should be publicly visible.GET /incidents?include=componentsnow shows the status the incident imposed rather thanoperational.Verification
933 tests passing, Pint and PHPStan clean, migrations verified from a fresh build including the deduplication path against duplicate rows.
New coverage includes severity precedence over recency, the maintenance mask with a failing monitor, per-window maintenance statuses and overlapping windows, exclusion of embargoed, hidden and resolved impacts, parity between eager-loaded and lazy resolution, the read policy per surface, and the status-change log with its source and causer.
Still open
Recording or clearing an impact emits no event — only baseline changes do — so a consumer watching webhooks sees "someone set this component" but not "an incident took it down". Related: because effective status now depends on the clock, a component's public status can change with no write at all when a window opens or an embargo lifts, and nothing fires then either. Both are worth settling before anything starts consuming these events.
🤖 Generated with Claude Code
https://claude.ai/code/session_0188UuLsNZ39q6Yw8BUBVH3b