[Feature Request] Reclassify Benign Application errors in OpenTelemetry

[gbxl]

Is your feature request related to a problem? Please describe.

When using the suggested polling style leveraging raise ApplicationError(category=BENIGN) and activity retries, Temporal seem to properly catch that it's not really an error and log it at a debug level. Unfortunately, OpenTelemetry still reports the exception as an error just as dire as any other one. This is an issue because then libraries like logfire that plug into OpenTelemetry will trigger alerts for "expected behavior"

Describe the solution you'd like

It'd be great to have a possible configuration or have an update to the OpenTelemetry contrib so that benign errors aren't treated as a real error in OpenTelemetry spans

Additional context

I ended up adding a custom interceptor like this:

class BenignAppErrorInterceptor(Interceptor):
    """Interceptor to handle benign application errors in activities."""

    def intercept_activity(
        self,
        next: ActivityInboundInterceptor,  # noqa: A002
    ) -> ActivityInboundInterceptor:
        """Intercept activity execution to handle benign application errors."""
        return _BenignAppErrorActivityInboundInterceptor(next)


class _BenignAppErrorActivityInboundInterceptor(ActivityInboundInterceptor):
    def __init__(self, next: ActivityInboundInterceptor) -> None:  # noqa: A002
        super().__init__(next)

    async def execute_activity(self, input, *args, **kwargs):  # noqa: A002, ANN002, ANN003, ANN001, ANN202
        try:
            return await super().execute_activity(input, *args, **kwargs)
        except ApplicationError as e:
            msg = str(e)
            category = getattr(e, "category", None) or ""
            if category == "BENIGN":
                span = trace.get_current_span()
                if span and span.is_recording():
                    span.set_attribute("temporal.error.benign", value=True)
                    span.add_event(
                        "benign_application_error_filtered",
                        {
                            "exception.type": "ApplicationError",
                            "exception.message": msg,
                            "exception.category": str(category),
                        },
                    )
                    span.set_status(Status(StatusCode.OK))
                raise

But that seems really unideal to have to do that on the user side.