Rebuild the metrics ingest, storage and read paths#397
Open
jbrooksuk wants to merge 2 commits into
Open
Conversation
Metrics stay what they are: a small set of operator-curated, human-readable
series that give context during an incident. The mechanics behind them are
brought up to date.
Ingest no longer discards data. CreateMetricPoint used to fetch the latest
point, and if it fell inside the metric's threshold, increment its counter
and throw the submitted value away — for an average-type metric that was
data loss by design, and the read-then-branch had no lock, so concurrent
writers left duplicate rows behind. Observations now accumulate into a
fixed-width bucket through a single upsert statement that maintains a
running sum_value alongside counter.
calc_type finally means something: it is how observations combine within a
bucket. A point resolves to the total of its observations, or their mean,
and calculated_value returns that instead of the unrendered value * counter.
For sum metrics the number is unchanged; for average metrics it is now
correct.
Behaviour changes, both deliberate:
- Metric points recorded through the ingest path dispatch no model events,
so MetricPointCreated and the webhook behind it no longer fire per
observation. The write is an upsert and loads no model. Counter
increments already fired nothing, so consumers only ever saw an
arbitrary subsample of what was submitted.
- Metric points are pruned after cachet.metrics.retention_days, 90 by
default, by the existing model:prune schedule. Set it to null to keep
every point forever and let the table grow without bound.
Also in this change:
- A RecordsMetricObservations contract, taking MetricObservation DTOs
singly or in batch, with Cachet's bucketing recorder bound as the
default and a POST metrics/{metric}/points/batch endpoint in front of
it. Monitoring packages that need to record elsewhere, or bucket
differently, rebind the contract.
- A unique (metric_id, created_at) index, which replaces the metric_id
index every ingest and render was range-scanning, and enforces one
point per bucket. Rows duplicated by the old racy write are collapsed
first, summing their totals into the oldest of them.
- The bucket width is frozen at write time. Editing a metric's threshold
changes how later observations are bucketed and leaves the buckets
already written alone.
- The status page drew all four chart windows from 30 days of raw points
inlined into the HTML. The hour and day windows now come from 24 hours
of raw buckets and the week and month windows from hourly totals rolled
up in the database, behind the same 30 second cache. No rollup table:
with the composite index and retention in place, GROUP BY at read time
is enough until hosted telemetry says otherwise.
- Charts ordered by places, the decimal-places column, where Filament and
MCP order by order. Fixed. The metric points endpoint had no default
sort at all, which the new index exposed; it now sorts by id.
- ?include=points loaded the unbounded relation. It is capped at
cachet.metrics.max_included_points, 100 by default; the paginated
metric points endpoint still walks the full history.
- Metrics take an optional component_id, which groups a chart under the
component it describes. Display only: component health has its own
pruned telemetry in component_checks, and metrics never drive component
status. Checks are operational, metrics are communicative.
- Cachet's cache keys are documented as tenant-agnostic — they identify
the audience a value was built for, never the installation — so hosts
sharing a cache store separate installations with the store's prefix
rather than each feature growing its own hook.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BP5RBEiV6NoeQsGPQLC1s1
Static analysis failed on the dependency floor. Collection generics are invariant, and the two supported Laravel versions disagree about what groupBy() returns: on 11.x it propagates the callback's int, on 13.x it widens to array-key. No single annotation satisfies both, so rawTotals() was correct on one and wrong on the other. Both helpers now build a plain array keyed by metric id with a foreach, which types identically on every supported version. hourlyTotals() moves with them, returning the resolved bucket shape rather than raw rows, so the view component no longer needs to normalise what it reads back. Verified against the failing matrix cell — PHP 8.3, Laravel 11.x, prefer-lowest — as well as the ceiling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BP5RBEiV6NoeQsGPQLC1s1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Metrics stay what they are: a small set of operator-curated, human-readable
series that give context during an incident. The mechanics behind them are
brought up to date.
Ingest no longer discards data. CreateMetricPoint used to fetch the latest
point, and if it fell inside the metric's threshold, increment its counter
and throw the submitted value away — for an average-type metric that was
data loss by design, and the read-then-branch had no lock, so concurrent
writers left duplicate rows behind. Observations now accumulate into a
fixed-width bucket through a single upsert statement that maintains a
running sum_value alongside counter.
calc_type finally means something: it is how observations combine within a
bucket. A point resolves to the total of its observations, or their mean,
and calculated_value returns that instead of the unrendered value * counter.
For sum metrics the number is unchanged; for average metrics it is now
correct.
Behaviour changes, both deliberate:
so MetricPointCreated and the webhook behind it no longer fire per
observation. The write is an upsert and loads no model. Counter
increments already fired nothing, so consumers only ever saw an
arbitrary subsample of what was submitted.
default, by the existing model:prune schedule. Set it to null to keep
every point forever and let the table grow without bound.
Also in this change:
singly or in batch, with Cachet's bucketing recorder bound as the
default and a POST metrics/{metric}/points/batch endpoint in front of
it. Monitoring packages that need to record elsewhere, or bucket
differently, rebind the contract.
index every ingest and render was range-scanning, and enforces one
point per bucket. Rows duplicated by the old racy write are collapsed
first, summing their totals into the oldest of them.
changes how later observations are bucketed and leaves the buckets
already written alone.
inlined into the HTML. The hour and day windows now come from 24 hours
of raw buckets and the week and month windows from hourly totals rolled
up in the database, behind the same 30 second cache. No rollup table:
with the composite index and retention in place, GROUP BY at read time
is enough until hosted telemetry says otherwise.
MCP order by order. Fixed. The metric points endpoint had no default
sort at all, which the new index exposed; it now sorts by id.
cachet.metrics.max_included_points, 100 by default; the paginated
metric points endpoint still walks the full history.
component it describes. Display only: component health has its own
pruned telemetry in component_checks, and metrics never drive component
status. Checks are operational, metrics are communicative.
the audience a value was built for, never the installation — so hosts
sharing a cache store separate installations with the store's prefix
rather than each feature growing its own hook.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01BP5RBEiV6NoeQsGPQLC1s1