Instrument http add on operator #1328

elieserr · 2025-08-24T22:43:59Z

Added instrumentation for promethues compatible endpoints and otel.
Added tests and e2e tests.

I went forward this ways as I was trying to get familiar with codebase. Happy to go in any direction reviewers advise.

Checklist

Commits are signed with Developer Certificate of Origin (DCO)
Changelog has been updated and is aligned with our changelog requirements
Any necessary documentation is added, such as:

Part of #965

elieserr · 2025-08-25T11:46:12Z

Hi folks, trying to get some eyes here but not really sure who to tag, thanks for any pointer in advance
@JorTurFer @rickbrouwer @zroubalik tagging you 3 since helped me with previous unrelated PR kedacore/keda#6990

wozniakjan · 2025-08-28T08:35:52Z

operator/controllers/http/httpscaledobject_controller.go

+func (r *HTTPScaledObjectReconciler) updatePromMetricsOnDelete(ctx context.Context, scaledObject *httpv1alpha1.HTTPScaledObject, namespacedName string) {
+	logger := log.FromContext(ctx, "updatePromMetricsOnDelete", namespacedName)
+	logger.Info("updatePromMetricsOnDelete")
+	metrics.RecordDeleteHTTPScaledObjectCount(namespacedName)
+}


looks like this is never called, wouldn't this result in always growing metric?

you are right, forgot to call this, will fix

wozniakjan · 2025-08-28T08:40:04Z

tests/checks/operator_otel_metrics/operator_otel_metrics_test.go

+	assert.True(t, ok, "operator_http_scaled_object_count_total is available")
+
+	requestCount := getMetricsValue(val)
+	assert.GreaterOrEqual(t, requestCount, float64(1))


when the whole e2e suite is executed, multiple e2e tests can run in parallel (iirc it could be like 3?). Perhaps the test could create 100 HSOs check if the metric is equal or over 100. And then we may want to delete that 100, wait for the metric to propagate and check again if it's under some reasonably low number, e.g. 10.

That's a good point, meanwhile we don't execute sequential tests (I'd like to avoid it tbh, but maybe we have to 🤔)
I guess that we can just execute both checks at once in the same tests, something like:

spawn 20 HTTPScaledObjects

check prometheus

check otel

remove the HTTPScaledObjects

check prometheus

check otel

I'm updating the e2e to do some sequencing for testing metric going up and down. also including testing for http and grpc for otel

JorTurFer · 2025-08-28T18:13:43Z

config/operator/deployment.yaml

        ports:
        - name: metrics
-          containerPort: 8080
+          containerPort: 2223


why this? Personally, I'd prefer to not change an already existing port

JorTurFer · 2025-08-28T18:15:27Z

docs/operate.md

+This endpoint can be enabled by setting the `OTEL_PROM_EXPORTER_ENABLED` environment variable to `true` on the operator deployment (`true` by default) and by setting `OTEL_PROM_EXPORTER_PORT` to an unused port for the endpoint to be made avaialble on (`2223` by default).
+
+### Configuring the OTEL HTTP exporter
+When configured, the ioperator can export metrics to a OTEL HTTP collector.


Suggested change

When configured, the ioperator can export metrics to a OTEL HTTP collector.

When configured, the operator can export metrics to a OTEL HTTP collector.

JorTurFer · 2025-08-28T18:15:55Z

docs/operate.md

+### Configuring the OTEL HTTP exporter
+When configured, the ioperator can export metrics to a OTEL HTTP collector.
+
+The OTEL exporter can be enabled by setting the `OTEL_EXPORTER_OTLP_METRICS_ENABLED` environment variable to `true` on the operator deployment (`false` by default). When enabled the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable must also be configured so the exporter knows what collector to send the metrics to (e.g. http://opentelemetry-collector.open-telemetry-system:4318).


Suggested change

The OTEL exporter can be enabled by setting the `OTEL_EXPORTER_OTLP_METRICS_ENABLED` environment variable to `true` on the operator deployment (`false` by default). When enabled the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable must also be configured so the exporter knows what collector to send the metrics to (e.g. http://opentelemetry-collector.open-telemetry-system:4318).

The OTEL exporter can be enabled by setting the `OTEL_EXPORTER_OTLP_METRICS_ENABLED` environment variable to `true` on the operator deployment (`false` by default). When enabled, the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable must also be configured so the exporter knows what collector to send the metrics to (e.g. http://opentelemetry-collector.open-telemetry-system:4318).

JorTurFer · 2025-08-28T18:19:56Z

operator/main.go

+	if metricsCfg.OtelPrometheusExporterEnabled {
+		// start the prometheus compatible metrics server
+		// serves a prometheus compatible metrics endpoint on the configured port
+		eg.Go(func() error {
+			if err := runMetricsServer(ctx, ctrl.Log, metricsCfg); !util.IsIgnoredErr(err) {
+				setupLog.Error(err, "could not start the Prometheus metrics server")
+				return err
+			}
+
+			return nil
+		})
+	}


The operator is already exposing a metrics server in prom format. Let's use it instead of starting another one, you can register more metrics on that server, for example, that's what we do for KEDA -> https://github.com/kedacore/keda/blob/main/pkg/metricscollector/prommetrics.go#L161

yeah that make sense, will use it

JorTurFer · 2025-08-28T18:21:48Z

operator/metrics/otelmetrics.go

+func NewOtelMetrics(options ...metric.Option) *OtelMetrics {
+	ctx := context.Background()
+
+	exporter, err := otlpmetrichttp.New(ctx)


Opentelemetry supports HTTP and gRCP protocols, I prefer if we support both tbh. This is how we do it in KEDA, you can use it as example https://github.com/kedacore/keda/blob/main/pkg/metricscollector/opentelemetry.go#L67-L86

including it in upcoming changes, thnaks

JorTurFer · 2025-08-28T18:24:40Z

operator/metrics/otelmetrics.go

+	provider := metric.NewMeterProvider(options...)
+	meter := provider.Meter(meterName)
+
+	httpScaledObjectCounter, err := meter.Int64UpDownCounter("operator_http_scaled_object_count", api.WithDescription("a counter of http_scaled_objects processed by the operator"))


in opentelemetry context, dots are used instead of underscores

Suggested change

httpScaledObjectCounter, err := meter.Int64UpDownCounter("operator_http_scaled_object_count", api.WithDescription("a counter of http_scaled_objects processed by the operator"))

httpScaledObjectCounter, err := meter.Int64UpDownCounter("keda.http.scaled.object.count", api.WithDescription("a counter of HttpScaledObjects processed by the operator"))

JorTurFer · 2025-08-28T18:26:00Z

operator/metrics/prommetrics.go

+	)
+	meter := provider.Meter(meterName)
+
+	httpScaledObjectCounter, err := meter.Int64UpDownCounter("operator_http_scaled_object_count", api.WithDescription("a counter of http_scaled_objects processed by the operator"))


Suggested change

httpScaledObjectCounter, err := meter.Int64UpDownCounter("operator_http_scaled_object_count", api.WithDescription("a counter of http_scaled_objects processed by the operator"))

httpScaledObjectCounter, err := meter.Int64UpDownCounter("keda_http_scaled_object_total", api.WithDescription("a counter of http_scaled_objects processed by the operator"))

JorTurFer · 2025-08-28T18:28:47Z

tests/checks/operator_otel_metrics/operator_otel_metrics_test.go

+	assert.True(t, ok, "operator_http_scaled_object_count_total is available")
+
+	requestCount := getMetricsValue(val)
+	assert.GreaterOrEqual(t, requestCount, float64(1))


That's a good point, meanwhile we don't execute sequential tests (I'd like to avoid it tbh, but maybe we have to 🤔)
I guess that we can just execute both checks at once in the same tests, something like:

spawn 20 HTTPScaledObjects

check prometheus

check otel

remove the HTTPScaledObjects

check prometheus

check otel

JorTurFer · 2025-08-28T18:29:12Z

Thanks a lot for the contribution!

zroubalik · 2025-09-18T15:57:32Z

@elieser1101 could you please check those failing e2e tests?

Execution of tests/checks/interceptor_otel_tracing/interceptor_otel_tracing_test.go, has failed after 3 tries

https://productionresultssa7.blob.core.windows.net/actions-results/26098fe6-b81f-40fd-a510-2a1de926e4ca/workflow-job-run-ede9479a-8054-5533-84ce-0cba19fe1088/logs/job/job-logs.txt?rsct=text%2Fplain&se=2025-09-18T16%3A06%3A24Z&sig=UYhSwGEyITYO9%2F4%2B6TwjmNd%2FI9WZOIUDfJonhMAyaUg%3D&ske=2025-09-19T02%3A15%3A03Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2025-09-18T14%3A15%3A03Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-11-05&sp=r&spr=https&sr=b&st=2025-09-18T15%3A56%3A19Z&sv=2025-11-05

Signed-off-by: Elieser Pereira <[email protected]> Signed-off-by: Elieser Pereira <[email protected]>

Signed-off-by: Elieser Pereira <[email protected]>

…o fit meaning Signed-off-by: Elieser Pereira <[email protected]>

Signed-off-by: Elieser Pereira [email protected] Signed-off-by: Elieser Pereira <[email protected]>

Signed-off-by: Elieser Pereira <[email protected]>

elieserr · 2025-09-23T20:23:14Z

I was able to reproduce the error in my local env. Was because of misconfig on go.mod. Also I rebased latest changes from main so this should work. lets see how e2e goes.

@zroubalik @wozniakjan @JorTurFer for workflow approval.

elieserr · 2025-09-24T15:26:46Z

e2e went good on previous run, so fixing lints and some broken links from docs

Signed-off-by: Elieser Pereira <[email protected]>

Copilot

Pull Request Overview

This PR adds instrumentation for Prometheus-compatible endpoints and OpenTelemetry (OTEL) metrics to the KEDA HTTP Add-on operator. The implementation provides monitoring capabilities for HTTP scaled objects through both metrics collection methods.

Added metrics collection infrastructure with Prometheus and OTEL exporters
Implemented HTTP scaled object count tracking in the operator controller
Added comprehensive end-to-end tests for both metrics collection methods

Reviewed Changes

Copilot reviewed 21 out of 22 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
`operator/metrics/`	New package containing metrics collection infrastructure with Prometheus and OTEL exporters
`operator/controllers/http/httpscaledobject_controller.go`	Updated controller to track HTTP scaled object metrics with thread-safe map
`operator/controllers/http/config/config.go`	Added metrics configuration structure with environment variable parsing
`operator/main.go`	Integrated metrics collectors initialization into main function
`tests/checks/operator_*_metrics/`	Comprehensive E2E tests for both Prometheus and OTEL metrics
`docs/`	Updated documentation with metrics configuration instructions
`config/`	Added Kubernetes manifests for metrics service and E2E test configurations

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

tests/checks/operator_prometheus_metrics/operator_prometheus_metrics_test.go

tests/checks/operator_otel_metrics/operator_otel_metrics_test.go

docs/operate.md

CHANGELOG.md

operator/controllers/http/httpscaledobject_controller.go

Copilot · 2025-10-03T07:54:15Z

operator/controllers/http/httpscaledobject_controller.go

+	namespacedName := scaledObject.Name + scaledObject.Namespace
+	metricsData, ok := httpScaledObjectPromMetricsMap[namespacedName]
+	if ok {
+		metrics.RecordDeleteHTTPScaledObjectCount(scaledObject.Namespace)


When the map entry already exists, the code calls RecordDeleteHTTPScaledObjectCount before RecordHTTPScaledObjectCount, which could lead to incorrect metrics if the namespace changed. This logic appears to be attempting to handle namespace changes, but it's unclear and potentially buggy.

Suggested change

metrics.RecordDeleteHTTPScaledObjectCount(scaledObject.Namespace)

// Record deletion for the old namespace before updating

metrics.RecordDeleteHTTPScaledObjectCount(metricsData.namespace)

Signed-off-by: Jan Wozniak <[email protected]>

wozniakjan reviewed Aug 28, 2025

View reviewed changes

JorTurFer reviewed Aug 28, 2025

View reviewed changes

elieserr force-pushed the instrument-http-add-on-operator branch from 723bbb9 to 176de5d Compare September 6, 2025 12:58

elieserr added 9 commits September 23, 2025 17:11

basic instrumentation for the operator

c7c7717

Signed-off-by: Elieser Pereira <[email protected]> Signed-off-by: Elieser Pereira <[email protected]>

add e2e tests and update config/operator to suport e2e

ff77d80

Signed-off-by: Elieser Pereira <[email protected]>

document how to use prometheus and otel metrics. change matric type t…

b8767b8

…o fit meaning Signed-off-by: Elieser Pereira <[email protected]>

register metrics collector to use existing metrics server

e46bce8

Signed-off-by: Elieser Pereira [email protected] Signed-off-by: Elieser Pereira <[email protected]>

update metrics names and dots are used instead of underscores for otel

015b5c3

Signed-off-by: Elieser Pereira <[email protected]>

proper management for metrics updates + working e2e

a1c5a02

Signed-off-by: Elieser Pereira <[email protected]>

operator prometheus e2e creates and removes multiple httpscaledobjects

0072119

Signed-off-by: Elieser Pereira <[email protected]>

add suport for grpc for otel

1e610a3

Signed-off-by: Elieser Pereira <[email protected]>

fix go.mod

f56c1bd

Signed-off-by: Elieser Pereira <[email protected]>

elieserr force-pushed the instrument-http-add-on-operator branch from 176de5d to f56c1bd Compare September 23, 2025 20:17

elieserr added 2 commits September 24, 2025 12:31

fix broken links for linkinator

cbcb3f9

Signed-off-by: Elieser Pereira <[email protected]>

lint fixes

bb94b8f

Signed-off-by: Elieser Pereira <[email protected]>

elieserr force-pushed the instrument-http-add-on-operator branch from bbad4f2 to bb94b8f Compare September 24, 2025 15:32

wozniakjan requested a review from Copilot October 3, 2025 07:51

Copilot AI reviewed Oct 3, 2025

View reviewed changes

fix typos and correctly keying HTTPSO resource for otel metrics

1e8e24a

Signed-off-by: Jan Wozniak <[email protected]>

	When configured, the ioperator can export metrics to a OTEL HTTP collector.
	When configured, the operator can export metrics to a OTEL HTTP collector.

	The OTEL exporter can be enabled by setting the `OTEL_EXPORTER_OTLP_METRICS_ENABLED` environment variable to `true` on the operator deployment (`false` by default). When enabled the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable must also be configured so the exporter knows what collector to send the metrics to (e.g. http://opentelemetry-collector.open-telemetry-system:4318).
	The OTEL exporter can be enabled by setting the `OTEL_EXPORTER_OTLP_METRICS_ENABLED` environment variable to `true` on the operator deployment (`false` by default). When enabled, the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable must also be configured so the exporter knows what collector to send the metrics to (e.g. http://opentelemetry-collector.open-telemetry-system:4318).

	httpScaledObjectCounter, err := meter.Int64UpDownCounter("operator_http_scaled_object_count", api.WithDescription("a counter of http_scaled_objects processed by the operator"))
	httpScaledObjectCounter, err := meter.Int64UpDownCounter("keda.http.scaled.object.count", api.WithDescription("a counter of HttpScaledObjects processed by the operator"))

	metrics.RecordDeleteHTTPScaledObjectCount(scaledObject.Namespace)
	// Record deletion for the old namespace before updating
	metrics.RecordDeleteHTTPScaledObjectCount(metricsData.namespace)

Instrument http add on operator #1328

Are you sure you want to change the base?

Instrument http add on operator #1328

Conversation

elieserr commented Aug 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

elieserr commented Aug 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JorTurFer commented Aug 28, 2025

Uh oh!

zroubalik commented Sep 18, 2025

Uh oh!

elieserr commented Sep 23, 2025

Uh oh!

elieserr commented Sep 24, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elieserr commented Aug 24, 2025 •

edited

Loading