Add exponential histogram support to CloudWatch PMD Exporter #1677

dricross · 2025-05-08T20:55:23Z

Description of the issue

The Cloudwatch/PMD exporter currently drops all exponential histogram metrics.

Description of changes

Add support for exponential histogram to the ec2tagger processor
Add support for exponential histogram to the CloudWatch/PMD exporter

Note

See companion PR for updating cumulativetodelta processor: amazon-contributing/opentelemetry-collector-contrib#331

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Note

See companion PR for integration test: aws/amazon-cloudwatch-agent-test#558

Integration test run: https://github.com/aws/amazon-cloudwatch-agent/actions/runs/16371811763
Histogram test: https://github.com/aws/amazon-cloudwatch-agent/actions/runs/16371811763/job/46261959442

Requirements

Before commit the code, please do the following steps.

Run make fmt and make fmt-sh
Run make lint

metric/distribution/exph/exph.go

jefchien · 2025-05-15T22:01:30Z

metric/distribution/exph/exph.go

+	return d.AddEntryWithUnit(value, weight, "")
+}
+
+func (d *ExpHistogramDistribution) AddDistribution(other *ExpHistogramDistribution) {


Are we trying to match the Distribution interface? It doesn't look like this would satisfy the interface as it is now. We would have to make the Distribution interface generic.

Yeah, you're right it doesn't satisfy the interface. Initially I was trying to do so but it ended up not working out because the Distribution interface is specific too HistogramDataPoint. In hindsight, I don't think it's even possible to combine exp histograms and classic histograms anyways so not really necessary at all. I already had this code though and didn't think it was worth refactoring everything to make it fit nicely together so I left it as is for now

plugins/outputs/cloudwatch/convert_otel.go

jefchien · 2025-05-20T18:00:05Z

plugins/outputs/cloudwatch/convert_otel.go

+		}
+		// Assume function pointer is valid.
+		ad.expHistDistribution = exph.NewExpHistogramDistribution()
+		ad.expHistDistribution.ConvertFromOtel(dp, unit)


nit: If we store the unit in the MetricDatum already, why do we need to store it in the distribution?

This is based off of how it currently works for the regular histograms. The unit is given to the distribution so that it can check that similar distributions are being combined. If the unit for distribution does match the unit for the incoming distribution to combine it with, a debug log is printed.

jefchien · 2025-05-20T18:01:08Z

plugins/outputs/cloudwatch/cloudwatch.go

+	return datums
+}
+
+func (c *CloudWatch) buildMetricDataumExph(metric *aggregationDatum, dimensionsList [][]*cloudwatch.Dimension) []*cloudwatch.MetricDatum {


nit: Typo

Suggested change

func (c *CloudWatch) buildMetricDataumExph(metric *aggregationDatum, dimensionsList [][]*cloudwatch.Dimension) []*cloudwatch.MetricDatum {

func (c *CloudWatch) buildMetricDatumExph(metric *aggregationDatum, dimensionsList [][]*cloudwatch.Dimension) []*cloudwatch.MetricDatum {

github-actions · 2025-05-28T00:13:31Z

This PR was marked stale due to lack of activity.

github-actions · 2025-07-07T00:14:53Z

This PR was marked stale due to lack of activity.

lisguo · 2025-07-18T17:03:57Z

go.mod

@@ -4,6 +4,10 @@ go 1.24.4

 replace github.com/influxdata/telegraf => github.com/aws/telegraf v0.10.2-0.20250113150713-a2dfaa4cdf6d

+replace collectd.org v0.4.0 => github.com/collectd/go-collectd v0.4.0


are these dependencies of cumulativetodeltaprocessor? I don't see go-collectd or clock used in the new code

I was hitting some issues with downloading the collectd dependency after clearing my local go cache and I needed to redirect to github to pick it up. I'm not exactly sure what happened but it looks like collectd.org stopped vending their package via collectd.org. The issue may have been resolved by now though, so I can try again.

../../../.gvm/pkgsets/go1.22.7/global/pkg/mod/github.com/aws/[email protected]/plugins/parsers/collectd/parser.go:8:2: unrecognized import path "collectd.org": https fetch: Get "https://collectd.org/?go-get=1": dial tcp: lookup collectd.org on 10.4.4.10:53: read udp 10.169.109.191:52627->10.4.4.10:53: i/o timeout

lisguo · 2025-07-18T17:10:31Z

metric/distribution/exph/exph.go

+
+func (d *ExpHistogramDistribution) Size() int {
+	size := len(d.negativeBuckets) + len(d.positiveBuckets)
+	if d.zeroCount > 0 {


what is zeroCount? the number of datapoints with 0?

Yes, pretty much. OTLP exponential histograms splits the data into three section: negative values, zero values, positive values. Positive and negative buckets are defined separately by a series of buckets+counts. The zero values don't have any buckets so its just a counter stored in the histogram structure. Side note: the definition of "0" in OTLP exponential histograms is loose. Datapoints with a magnitude less than the configurable "zero threshold" is treated as 0.

lisguo · 2025-07-18T18:20:10Z

metric/distribution/exph/exph.go

+}
+
+func (d *ExpHistogramDistribution) Resize(_ int) []*ExpHistogramDistribution {
+	// TODO: split data points into separate PMD requests if the number of buckets exceeds the API limit


what happens if we exceed the API limit?

Based on the API documentation, I believe the PMD request will be rejected with error code 400 InvalidParameterValue, though I haven't tried.

lisguo · 2025-07-21T19:44:44Z

metric/distribution/exph/exph.go

+
+// ValuesAndCounts outputs two arrays representing the midpoints of each exponential histogram bucket and the
+// counter of datapoints within the corresponding exponential histogram buckets
+func (d *ExpHistogramDistribution) ValuesAndCounts() ([]float64, []float64) {


nit: we can make the name of the function more descriptive. Maybe GetMidpointsAndCounts?

This naming scheme was based off of the existing function in the Distribution interface. I think ValuesAndCounts is a fairly descriptive name as that's what is actually pushed to CloudWatch in the PMD request (an array of values and an array of counts).

lisguo · 2025-07-21T19:49:19Z

metric/distribution/exph/exph.go

+	d.min = min(d.min, value)
+	d.max = max(d.max, value)
+
+	if math.Abs(value) > d.zeroThreshold {


so if my zero threshold is 1, then both values 2 and -2 will be counted in the zero bucket?

Good catch, this logic is incorrect. In fact this function is left over work from trying to adhere to the Distribution interface that's not used nor needed. I'll remove this function entirely.

lisguo · 2025-07-21T19:49:42Z

metric/distribution/exph/exph.go

+
+	if math.Abs(value) > d.zeroThreshold {
+		d.zeroCount += uint64(weight)
+	} else if value > d.zeroThreshold {


would this ever happen? Wouldn't we always hit the first case?

Incorrect logic, but will be removing this function.

lisguo · 2025-07-21T20:01:57Z

plugins/outputs/cloudwatch/cloudwatch.go

-		distList = resize(metric.distribution, c.config.MaxValuesPerDatum)
-	}
+		datums = c.buildMetricDatumDist(metric, dimensionsList)
+	} else if metric.expHistDistribution != nil {


is there ever a chance we will have both distribution and expHistDistribution not nil? Probably shouldn't happen...

No (at least they shouldn't...). The agent will convert the different OTLP datatypes into aggregationDatums in ConvertOtelMetrics. The existing ConvertOtelHistogramDataPoints sets distribution field and the new ConvertOtelExponentialHistogramDataPoints sets the expHistDistribution field.

not quite complete. need more unit tests

* Move OTLP implementation to separate file * Simplify map key sorting

dricross · 2025-07-23T17:43:00Z

plugins/outputs/cloudwatch/util.go

@@ -78,34 +77,6 @@ func setNewDistributionFunc(maxValuesPerDatumLimit int) {
 	}
 }

-func resize(dist distribution.Distribution, listMaxSize int) (distList []distribution.Distribution) {


refactored as functions on each distribution

dricross requested a review from a team as a code owner May 8, 2025 20:55

jefchien reviewed May 15, 2025

View reviewed changes

dricross force-pushed the dricross/exponentialhistogram branch 2 times, most recently from 97c4271 to 995902e Compare May 16, 2025 21:08

jefchien reviewed May 20, 2025

View reviewed changes

github-actions bot added the Stale label May 28, 2025

github-actions bot removed the Stale label Jun 28, 2025

github-actions bot added the Stale label Jul 7, 2025

dricross force-pushed the dricross/exponentialhistogram branch from 995902e to 936253b Compare July 17, 2025 17:15

dricross mentioned this pull request Jul 17, 2025

Add exponential histograms to histograms integration test aws/amazon-cloudwatch-agent-test#558

Open

dricross force-pushed the dricross/exponentialhistogram branch from 07375ab to 5362dc8 Compare July 17, 2025 22:52

github-actions bot removed the Stale label Jul 18, 2025

lisguo reviewed Jul 21, 2025

View reviewed changes

dricross added 11 commits July 23, 2025 08:12

A dumb commit

2d29845

adding support for exphistograms

c0e0713

not quite complete. need more unit tests

Do not split exph metrics into separate PMD requests

94a95ef

fixup tests

19cc625

linting fixup. remove dead code

4935327

Fix typo in test

4a3ec70

PR feedback

5498274

* Move OTLP implementation to separate file * Simplify map key sorting

Update ec2tagger to support exp histograms

2046e5c

Use modified c2d processor

900d0c5

Remove dead code

f7a1f76

Fix typo

d71cfd4

dricross force-pushed the dricross/exponentialhistogram branch from 5362dc8 to d71cfd4 Compare July 23, 2025 12:24

Move functions to unexported to fix lint issue

da167ee

dricross commented Jul 23, 2025

View reviewed changes

	func (c CloudWatch) buildMetricDataumExph(metric aggregationDatum, dimensionsList [][]cloudwatch.Dimension) []cloudwatch.MetricDatum {
	func (c CloudWatch) buildMetricDatumExph(metric aggregationDatum, dimensionsList [][]cloudwatch.Dimension) []cloudwatch.MetricDatum {

		@@ -4,6 +4,10 @@ go 1.24.4

		replace github.com/influxdata/telegraf => github.com/aws/telegraf v0.10.2-0.20250113150713-a2dfaa4cdf6d

		replace collectd.org v0.4.0 => github.com/collectd/go-collectd v0.4.0

Add exponential histogram support to CloudWatch PMD Exporter #1677

Are you sure you want to change the base?

Add exponential histogram support to CloudWatch PMD Exporter #1677

Conversation

dricross commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of the issue

Description of changes

License

Tests

Requirements

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 28, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dricross commented May 8, 2025 •

edited

Loading