Skip to content

Rebuild the metrics ingest, storage and read paths#397

Open
jbrooksuk wants to merge 2 commits into
mainfrom
claude/irm-metrics-architecture-r6nxkk
Open

Rebuild the metrics ingest, storage and read paths#397
jbrooksuk wants to merge 2 commits into
mainfrom
claude/irm-metrics-architecture-r6nxkk

Conversation

@jbrooksuk

Copy link
Copy Markdown
Member

Metrics stay what they are: a small set of operator-curated, human-readable
series that give context during an incident. The mechanics behind them are
brought up to date.

Ingest no longer discards data. CreateMetricPoint used to fetch the latest
point, and if it fell inside the metric's threshold, increment its counter
and throw the submitted value away — for an average-type metric that was
data loss by design, and the read-then-branch had no lock, so concurrent
writers left duplicate rows behind. Observations now accumulate into a
fixed-width bucket through a single upsert statement that maintains a
running sum_value alongside counter.

calc_type finally means something: it is how observations combine within a
bucket. A point resolves to the total of its observations, or their mean,
and calculated_value returns that instead of the unrendered value * counter.
For sum metrics the number is unchanged; for average metrics it is now
correct.

Behaviour changes, both deliberate:

  • Metric points recorded through the ingest path dispatch no model events,
    so MetricPointCreated and the webhook behind it no longer fire per
    observation. The write is an upsert and loads no model. Counter
    increments already fired nothing, so consumers only ever saw an
    arbitrary subsample of what was submitted.
  • Metric points are pruned after cachet.metrics.retention_days, 90 by
    default, by the existing model:prune schedule. Set it to null to keep
    every point forever and let the table grow without bound.

Also in this change:

  • A RecordsMetricObservations contract, taking MetricObservation DTOs
    singly or in batch, with Cachet's bucketing recorder bound as the
    default and a POST metrics/{metric}/points/batch endpoint in front of
    it. Monitoring packages that need to record elsewhere, or bucket
    differently, rebind the contract.
  • A unique (metric_id, created_at) index, which replaces the metric_id
    index every ingest and render was range-scanning, and enforces one
    point per bucket. Rows duplicated by the old racy write are collapsed
    first, summing their totals into the oldest of them.
  • The bucket width is frozen at write time. Editing a metric's threshold
    changes how later observations are bucketed and leaves the buckets
    already written alone.
  • The status page drew all four chart windows from 30 days of raw points
    inlined into the HTML. The hour and day windows now come from 24 hours
    of raw buckets and the week and month windows from hourly totals rolled
    up in the database, behind the same 30 second cache. No rollup table:
    with the composite index and retention in place, GROUP BY at read time
    is enough until hosted telemetry says otherwise.
  • Charts ordered by places, the decimal-places column, where Filament and
    MCP order by order. Fixed. The metric points endpoint had no default
    sort at all, which the new index exposed; it now sorts by id.
  • ?include=points loaded the unbounded relation. It is capped at
    cachet.metrics.max_included_points, 100 by default; the paginated
    metric points endpoint still walks the full history.
  • Metrics take an optional component_id, which groups a chart under the
    component it describes. Display only: component health has its own
    pruned telemetry in component_checks, and metrics never drive component
    status. Checks are operational, metrics are communicative.
  • Cachet's cache keys are documented as tenant-agnostic — they identify
    the audience a value was built for, never the installation — so hosts
    sharing a cache store separate installations with the store's prefix
    rather than each feature growing its own hook.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01BP5RBEiV6NoeQsGPQLC1s1

claude added 2 commits July 25, 2026 06:03
Metrics stay what they are: a small set of operator-curated, human-readable
series that give context during an incident. The mechanics behind them are
brought up to date.

Ingest no longer discards data. CreateMetricPoint used to fetch the latest
point, and if it fell inside the metric's threshold, increment its counter
and throw the submitted value away — for an average-type metric that was
data loss by design, and the read-then-branch had no lock, so concurrent
writers left duplicate rows behind. Observations now accumulate into a
fixed-width bucket through a single upsert statement that maintains a
running sum_value alongside counter.

calc_type finally means something: it is how observations combine within a
bucket. A point resolves to the total of its observations, or their mean,
and calculated_value returns that instead of the unrendered value * counter.
For sum metrics the number is unchanged; for average metrics it is now
correct.

Behaviour changes, both deliberate:

- Metric points recorded through the ingest path dispatch no model events,
  so MetricPointCreated and the webhook behind it no longer fire per
  observation. The write is an upsert and loads no model. Counter
  increments already fired nothing, so consumers only ever saw an
  arbitrary subsample of what was submitted.
- Metric points are pruned after cachet.metrics.retention_days, 90 by
  default, by the existing model:prune schedule. Set it to null to keep
  every point forever and let the table grow without bound.

Also in this change:

- A RecordsMetricObservations contract, taking MetricObservation DTOs
  singly or in batch, with Cachet's bucketing recorder bound as the
  default and a POST metrics/{metric}/points/batch endpoint in front of
  it. Monitoring packages that need to record elsewhere, or bucket
  differently, rebind the contract.
- A unique (metric_id, created_at) index, which replaces the metric_id
  index every ingest and render was range-scanning, and enforces one
  point per bucket. Rows duplicated by the old racy write are collapsed
  first, summing their totals into the oldest of them.
- The bucket width is frozen at write time. Editing a metric's threshold
  changes how later observations are bucketed and leaves the buckets
  already written alone.
- The status page drew all four chart windows from 30 days of raw points
  inlined into the HTML. The hour and day windows now come from 24 hours
  of raw buckets and the week and month windows from hourly totals rolled
  up in the database, behind the same 30 second cache. No rollup table:
  with the composite index and retention in place, GROUP BY at read time
  is enough until hosted telemetry says otherwise.
- Charts ordered by places, the decimal-places column, where Filament and
  MCP order by order. Fixed. The metric points endpoint had no default
  sort at all, which the new index exposed; it now sorts by id.
- ?include=points loaded the unbounded relation. It is capped at
  cachet.metrics.max_included_points, 100 by default; the paginated
  metric points endpoint still walks the full history.
- Metrics take an optional component_id, which groups a chart under the
  component it describes. Display only: component health has its own
  pruned telemetry in component_checks, and metrics never drive component
  status. Checks are operational, metrics are communicative.
- Cachet's cache keys are documented as tenant-agnostic — they identify
  the audience a value was built for, never the installation — so hosts
  sharing a cache store separate installations with the store's prefix
  rather than each feature growing its own hook.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BP5RBEiV6NoeQsGPQLC1s1
Static analysis failed on the dependency floor. Collection generics are
invariant, and the two supported Laravel versions disagree about what
groupBy() returns: on 11.x it propagates the callback's int, on 13.x it
widens to array-key. No single annotation satisfies both, so rawTotals()
was correct on one and wrong on the other.

Both helpers now build a plain array keyed by metric id with a foreach,
which types identically on every supported version. hourlyTotals() moves
with them, returning the resolved bucket shape rather than raw rows, so
the view component no longer needs to normalise what it reads back.

Verified against the failing matrix cell — PHP 8.3, Laravel 11.x,
prefer-lowest — as well as the ceiling.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BP5RBEiV6NoeQsGPQLC1s1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants