[Bug] sdk metrics do not include tags added with "telemetry.global_tags"

[antmendoza]

What are you really trying to do?

Add custom tags to SDK metrics

Describe the bug

sdk metrics no longer include tags added with telemetry.global_tags , which appear to be the expected behavior https://python.temporal.io/temporalio.runtime.TelemetryConfig.html#global_tags

running the same with version 1.3 works as expected

Minimal Reproduction

Environment/Versions

• OS and processor: [e.g. M1 Mac, x86 Windows, Linux]
• Temporal Version: [e.g. 1.14.0?] and/or SDK version
• Are you using Docker or Kubernetes or building Temporal from source?

Additional context

[cretz]

To clarify, issue #380 inadvertently didn't set global_tags on Rust side. We need to make sure it is set and add a test or alter test_different_runtimes to confirm this setting (and maybe any others we want coverage for).

[cretz]

The issue here is that we use OTel's Prometheus exporting which takes the approach at https://prometheus.io/docs/guides/opentelemetry/#including-resource-attributes-at-query-time. Namely the global_tags are on target_info which you can combine with the rest at query time. I have opened temporalio/sdk-core#882 to have us put the labels on every metric by default (with an option to restore to today's behavior).

[cretz]

Note, just for linking, this is caused by an upstream OTel Rust issue (our Rust core relies on this): open-telemetry/opentelemetry-rust#2858