prometheus: fix scrape duration growing unbounded (#13667)#13678
Open
PrashantBhanage wants to merge 1 commit into
Open
prometheus: fix scrape duration growing unbounded (#13667)#13678PrashantBhanage wants to merge 1 commit into
PrashantBhanage wants to merge 1 commit into
Conversation
Three changes scoped exclusively to the prometheus exporter plugin: 1. Bounded HTTP executor — give the HttpServer a FixedThreadPool(2) instead of the JDK default single-threaded executor; shut it down cleanly in stop(). 2. TTL guard on updateMetrics() — add a synchronized guard with a last-run timestamp so scrapes arriving faster than the configurable minimum interval (prometheus.exporter.metrics.min.refresh.interval, default 5 s) reuse the previously computed metrics instead of triggering a new recomputation. 3. Timing instrumentation — wrap updateMetrics() body with System.nanoTime() start/end and log elapsed wall-clock time at info level on every run (including exception path), so slow sub-collectors can be identified from logs. Fixes: apache#13667 Ref: apache#13586
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes unbounded growth of Prometheus
/metricsscrape durationagainst the CloudStack management server (root cause tracked in #13586,
improvements requested in #13667).
Before: the exporter's
HttpServerused the JDK default single-threadedexecutor, so a slow scrape serialized/queued every other scrape behind it.
Additionally,
updateMetrics()had no guard against concurrent or rapidrecomputation, so scrape frequency could multiply backend load
indefinitely. Only a full management server restart reset the growth.
After: three targeted, contained changes, all inside
plugins/integrations/prometheus/:PrometheusExporterServerImpl#start()nowsets an explicit
Executors.newFixedThreadPool(2)as the HttpServer'sexecutor instead of relying on the JDK default, and shuts it down
cleanly in
stop().PrometheusExporterImpl#updateMetrics()isnow
synchronizedand checks alastMetricsUpdateTimetimestampagainst a new dynamic global setting,
prometheus.exporter.metrics.min.refresh.interval(default 5 seconds).Scrapes arriving within that window reuse the previously computed
metrics instead of triggering a fresh, expensive recomputation.
updateMetrics()now logs its wall-clockexecution time in milliseconds at
infolevel on every run (includingon the exception path), so operators/CI can identify which
sub-collector is slow and confirm the fix closes the growth.
This PR deliberately does not touch
AlertManagerImplor thecapacity-calculation executor in
server/— that code already creates afresh thread pool per invocation and reads
capacity.calculate.workersdynamically on
main, and rewriting it was flagged in the issue commentsas unnecessary scope creep that introduced its own bugs (stale config,
swallowed shutdown exceptions, thread-pool leaks).
Fixes: #13667
Ref: #13586
Types of changes
Feature/Enhancement Scale or Bug Severity
Bug Severity
Screenshots (if appropriate):
N/A — backend-only change, no UI impact.
How Has This Been Tested?
PrometheusExporterImplTest:testUpdateMetricsTTLGuardSkipsSecondCall— two rapid calls toupdateMetrics()within the TTL window result in only onerecomputation.
testUpdateMetricsTTLGuardAllowsAfterInterval— after the configuredinterval elapses, a subsequent call triggers a fresh recomputation.
mvn -pl plugins/integrations/prometheus test→ 6/6 tests pass (4existing + 2 new).
mvn -pl plugins/integrations/prometheus -am install→ build success./metricsreuse cachedoutput within the TTL window, and that logged
updateMetrics()durationstays flat instead of climbing across successive scrapes.
How did you try to break this feature and the system with this change?
/metricsto confirm thebounded executor and
synchronizedguard prevent overlappingrecomputation instead of deadlocking or throwing.
stop()shuts down the new executor cleanly with no resourceleak warnings.
prometheus.exporter.metrics.min.refresh.intervalsetting is dynamically adjustable at runtime without a restart, and
that a value of
0effectively disables the guard (falls back tooriginal per-scrape recomputation behavior) for anyone who wants that.
AlertManagerImpl,CapacityManagerImpl,etc.) was touched, keeping this change fully isolated to the exporter
plugin.