Add exponential histogram support to CloudWatch PMD Exporter #1677

dricross · 2025-05-08T20:55:23Z

Description of the issue

The Cloudwatch/PMD exporter currently drops all exponential histogram metrics.

Description of changes

Add support for exponential histogram to the ec2tagger processor
Add support for exponential histogram to the CloudWatch/PMD exporter

Note

See companion PR for updating cumulativetodelta processor: amazon-contributing/opentelemetry-collector-contrib#331

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Note

See companion PR for integration test: aws/amazon-cloudwatch-agent-test#558

Integration test run: https://github.com/aws/amazon-cloudwatch-agent/actions/runs/16526626714
Histogram test: https://github.com/aws/amazon-cloudwatch-agent/actions/runs/16526626714/job/46741673921

The data we are sending as part of the test is defined here: https://github.com/aws/amazon-cloudwatch-agent-test/blob/dricross/exphistograms/test/histograms/resources/otlp_metrics.json#L188. where CUMULATIVE_HIST_COUNT and CUMULATIVE_HIST_SUM are increasing by 2 every 10 seconds. Important things to note:

Scale=0
IndexOffset = 1
Buckets = [0, 2] (where 2 is growing by 2 every 10 seconds)
With Scale=0, Base = 2 (Base = 2^(2^(-scale)) = 2^(2^0) = 2^1)

Formula for mapping bucket index to value: (Base^(index+offset), Base^(index+offset+1]

The first bucket (with 0 count) maps to range (2,4] ((2^(0+1), 2^(1+1+1])
The second bucket (with a count of 2) maps to range (4, 8] ((2^(1+1), 2^(1+1+1] = (4, 8])

Since we take the midpoints of the buckets when sending to exponential histograms to CloudWatch (consistent with the EMF exporter), the values/counts we send to CloudWatch is:

Values=[6,3]
Counts[2,0]

That list of values and counts is sent 6 times per minute

CloudWatch metrics:

EMF Logs from test as verification (this is existing behavior but demonstrates the intended output)

{
    "AWS.ServiceNameSource": "Instrumentation",
    "CloudWatchMetrics": [
        {
            "Namespace": "CWAgent",
            "Dimensions": [
                [
                    "service.name",
                    "my.exponential.histogram.attr"
                ]
            ],
            "Metrics": [
                {
                    "Name": "my.delta.exponential.histogram",
                    "Unit": "",
                    "StorageResolution": 60
                }
            ]
        }
    ],
    "EC2.InstanceId": "i-04282f146a9432a17",
    "Environment": "ec2:default",
    "PlatformType": "AWS::EC2",
    "Service": "my.service",
    "Timestamp": "1753710063060",
    "Version": "0",
    "my.exponential.histogram.attr": "some value",
    "service.name": "my.service",
    "my.delta.exponential.histogram": {
        "Values": [
            6,
            0
        ],
        "Counts": [
            2,
            1
        ],
        "Max": 5,
        "Min": 0,
        "Count": 3,
        "Sum": 10
    }
}
{
    "AWS.ServiceNameSource": "Instrumentation",
    "CloudWatchMetrics": [
        {
            "Namespace": "CWAgent",
            "Dimensions": [
                [
                    "my.cumulative.exponential.histogram.attr",
                    "service.name"
                ]
            ],
            "Metrics": [
                {
                    "Name": "my.cumulative.exponential.histogram",
                    "Unit": "",
                    "StorageResolution": 60
                }
            ]
        }
    ],
    "EC2.InstanceId": "i-04282f146a9432a17",
    "Environment": "ec2:default",
    "PlatformType": "AWS::EC2",
    "Service": "my.service",
    "Timestamp": "1753710063060",
    "Version": "0",
    "my.cumulative.exponential.histogram.attr": "some value",
    "service.name": "my.service",
    "my.cumulative.exponential.histogram": {
        "Values": [
            6
        ],
        "Counts": [
            2
        ],
        "Max": 0,
        "Min": 0,
        "Count": 2,
        "Sum": 2
    }
}

Requirements

Before commit the code, please do the following steps.

Run make fmt and make fmt-sh
Run make lint

metric/distribution/exph/exph.go

jefchien · 2025-05-15T22:01:30Z

metric/distribution/exph/exph.go

+	return d.AddEntryWithUnit(value, weight, "")
+}
+
+func (d *ExpHistogramDistribution) AddDistribution(other *ExpHistogramDistribution) {


Are we trying to match the Distribution interface? It doesn't look like this would satisfy the interface as it is now. We would have to make the Distribution interface generic.

Yeah, you're right it doesn't satisfy the interface. Initially I was trying to do so but it ended up not working out because the Distribution interface is specific too HistogramDataPoint. In hindsight, I don't think it's even possible to combine exp histograms and classic histograms anyways so not really necessary at all. I already had this code though and didn't think it was worth refactoring everything to make it fit nicely together so I left it as is for now

plugins/outputs/cloudwatch/convert_otel.go

plugins/outputs/cloudwatch/cloudwatch.go

github-actions · 2025-05-28T00:13:31Z

This PR was marked stale due to lack of activity.

github-actions · 2025-07-07T00:14:53Z

This PR was marked stale due to lack of activity.

lisguo · 2025-07-18T17:03:57Z

go.mod

@@ -4,6 +4,10 @@ go 1.24.4

 replace github.com/influxdata/telegraf => github.com/aws/telegraf v0.10.2-0.20250113150713-a2dfaa4cdf6d

+replace collectd.org v0.4.0 => github.com/collectd/go-collectd v0.4.0


are these dependencies of cumulativetodeltaprocessor? I don't see go-collectd or clock used in the new code

I was hitting some issues with downloading the collectd dependency after clearing my local go cache and I needed to redirect to github to pick it up. I'm not exactly sure what happened but it looks like collectd.org stopped vending their package via collectd.org. The issue may have been resolved by now though, so I can try again.

../../../.gvm/pkgsets/go1.22.7/global/pkg/mod/github.com/aws/[email protected]/plugins/parsers/collectd/parser.go:8:2: unrecognized import path "collectd.org": https fetch: Get "https://collectd.org/?go-get=1": dial tcp: lookup collectd.org on 10.4.4.10:53: read udp 10.169.109.191:52627->10.4.4.10:53: i/o timeout

lisguo · 2025-07-18T17:10:31Z

metric/distribution/exph/exph.go

+
+func (d *ExpHistogramDistribution) Size() int {
+	size := len(d.negativeBuckets) + len(d.positiveBuckets)
+	if d.zeroCount > 0 {


what is zeroCount? the number of datapoints with 0?

Yes, pretty much. OTLP exponential histograms splits the data into three section: negative values, zero values, positive values. Positive and negative buckets are defined separately by a series of buckets+counts. The zero values don't have any buckets so its just a counter stored in the histogram structure. Side note: the definition of "0" in OTLP exponential histograms is loose. Datapoints with a magnitude less than the configurable "zero threshold" is treated as 0.

lisguo · 2025-07-18T18:20:10Z

metric/distribution/exph/exph.go

+}
+
+func (d *ExpHistogramDistribution) Resize(_ int) []*ExpHistogramDistribution {
+	// TODO: split data points into separate PMD requests if the number of buckets exceeds the API limit


what happens if we exceed the API limit?

Based on the API documentation, I believe the PMD request will be rejected with error code 400 InvalidParameterValue, though I haven't tried.

lisguo · 2025-07-21T19:44:44Z

metric/distribution/exph/exph.go

+
+// ValuesAndCounts outputs two arrays representing the midpoints of each exponential histogram bucket and the
+// counter of datapoints within the corresponding exponential histogram buckets
+func (d *ExpHistogramDistribution) ValuesAndCounts() ([]float64, []float64) {


nit: we can make the name of the function more descriptive. Maybe GetMidpointsAndCounts?

This naming scheme was based off of the existing function in the Distribution interface. I think ValuesAndCounts is a fairly descriptive name as that's what is actually pushed to CloudWatch in the PMD request (an array of values and an array of counts).

metric/distribution/exph/exph.go

plugins/outputs/cloudwatch/cloudwatch.go

dricross · 2025-07-23T17:43:00Z

plugins/outputs/cloudwatch/util.go

@@ -78,34 +77,6 @@ func setNewDistributionFunc(maxValuesPerDatumLimit int) {
 	}
 }

-func resize(dist distribution.Distribution, listMaxSize int) (distList []distribution.Distribution) {


refactored as functions on each distribution

not quite complete. need more unit tests

* Move OTLP implementation to separate file * Simplify map key sorting

dricross requested a review from a team as a code owner May 8, 2025 20:55

jefchien reviewed May 15, 2025

View reviewed changes

dricross force-pushed the dricross/exponentialhistogram branch 2 times, most recently from 97c4271 to 995902e Compare May 16, 2025 21:08

jefchien reviewed May 20, 2025

View reviewed changes

github-actions bot added the Stale label May 28, 2025

github-actions bot removed the Stale label Jun 28, 2025

github-actions bot added the Stale label Jul 7, 2025

dricross force-pushed the dricross/exponentialhistogram branch from 995902e to 936253b Compare July 17, 2025 17:15

dricross mentioned this pull request Jul 17, 2025

Add exponential histograms to histograms integration test aws/amazon-cloudwatch-agent-test#558

Open

dricross force-pushed the dricross/exponentialhistogram branch from 07375ab to 5362dc8 Compare July 17, 2025 22:52

github-actions bot removed the Stale label Jul 18, 2025

lisguo reviewed Jul 21, 2025

View reviewed changes

dricross force-pushed the dricross/exponentialhistogram branch from 5362dc8 to d71cfd4 Compare July 23, 2025 12:24

dricross commented Jul 23, 2025

View reviewed changes

dricross force-pushed the dricross/exponentialhistogram branch from da167ee to eb11f44 Compare July 23, 2025 18:01

dricross added 12 commits July 24, 2025 13:50

A dumb commit

f3afa8a

adding support for exphistograms

971a0b6

not quite complete. need more unit tests

Do not split exph metrics into separate PMD requests

14aa119

fixup tests

51d915f

linting fixup. remove dead code

208c8aa

Fix typo in test

6e56335

PR feedback

ed49d91

* Move OTLP implementation to separate file * Simplify map key sorting

Update ec2tagger to support exp histograms

d2510af

Use modified c2d processor

7ac688c

Remove dead code

ce5c4d4

Fix typo

6a940b5

Move functions to unexported to fix lint issue

dffa311

Minor cleanup work from PR review

f7f02fe

dricross force-pushed the dricross/exponentialhistogram branch from eb11f44 to f7f02fe Compare July 24, 2025 18:18

dricross added the ready for testing Indicates this PR is ready for integration tests to run label Jul 24, 2025

lint fixup

5f6f635

		@@ -4,6 +4,10 @@ go 1.24.4

		replace github.com/influxdata/telegraf => github.com/aws/telegraf v0.10.2-0.20250113150713-a2dfaa4cdf6d

		replace collectd.org v0.4.0 => github.com/collectd/go-collectd v0.4.0

Add exponential histogram support to CloudWatch PMD Exporter #1677

Are you sure you want to change the base?

Add exponential histogram support to CloudWatch PMD Exporter #1677

Uh oh!

Conversation

dricross commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of the issue

Description of changes

License

Tests

Requirements

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 28, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dricross commented May 8, 2025 •

edited

Loading