[bug] Tracing starts even after disabling during guardrails configure

[prnvkv]

Describe the bug
Even after disabling anonymous tracing during guardrails configure, during the program execution, the stack traces are visible for trace collection

ERROR:opentelemetry.sdk._shared_internal:Exception while exporting Span.
Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/urllib3/connection.py", line 203, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File ".venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 497, in _make_request
    conn.request(
  File ".venv/lib/python3.12/site-packages/urllib3/connection.py", line 395, in request
    self.endheaders()
  File "/usr/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.12/http/client.py", line 1091, in _send_output
    self.send(msg)
  File "/usr/lib/python3.12/http/client.py", line 1035, in send
    self.connect()
  File ".venv/lib/python3.12/site-packages/urllib3/connection.py", line 243, in connect
    self.sock = self._new_conn()
                ^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/urllib3/connection.py", line 218, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x79f16c0022d0>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 845, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x79f16c0022d0>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 139, in _export
    resp = self._session.post(
           ^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x79f16c0022d0>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/urllib3/connection.py", line 203, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File ".venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 497, in _make_request
    conn.request(
  File ".venv/lib/python3.12/site-packages/urllib3/connection.py", line 395, in request
    self.endheaders()
  File "/usr/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.12/http/client.py", line 1091, in _send_output
    self.send(msg)
  File "/usr/lib/python3.12/http/client.py", line 1035, in send
    self.connect()
  File ".venv/lib/python3.12/site-packages/urllib3/connection.py", line 243, in connect
    self.sock = self._new_conn()
                ^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/urllib3/connection.py", line 218, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x79f16c002a50>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 845, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x79f16c002a50>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/opentelemetry/sdk/_shared_internal/__init__.py", line 152, in _export
    self._exporter.export(
  File ".venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 204, in export
    return self._export_serialized_spans(serialized_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 174, in _export_serialized_spans
    resp = self._export(serialized_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 147, in _export
    resp = self._session.post(
           ^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x79f16c002a50>: Failed to establish a new connection: [Errno 111] Connection refused'))

To Reproduce
Steps to reproduce the behavior:
Just configure guardrails by disabling anonymous tracing
and use Guard class.

Expected behavior
Should not start tracing

Library version:
Version (e.g. 0.6.0)

Additional context
Add any other context about the problem here.

[CalebCourier]

From the stack trace you posted, this does not look like the anonymous metrics we collect. That is trying to connect to localhost, not our collector endpoint. I also don't see any mention of the guardrails-ai package in the stack trace.

We do have support for collecting your own telemetry and sending it to your own sink (see here for details: https://www.guardrailsai.com/docs/concepts/telemetry). From the limited information, it is possible these errors are coming from that path if it is improperly configured but I cannot confirm that either without more detail. I also cannot reproduce this issue using the latest guardrails-ai and python 3.12 versions.

Please share more detail on you setup and any parts of the stack trace that identify these exceptions as coming from the guardrails-ai package.

[akranga]

Note, current latest package seems not to be the latest. Stanza for my workaround in the dockerfile (less verbose logging now)

pip install --system --upgrade --prerelease=allow "guardrails-ai[api]==0.6.6"
pip install --system --upgrade 'guardrails-api' --index-url 'https://github.com/guardrails-ai/guardrails-api.git'

• Upgrade to the latest guardrails-ai package from pre-release (0.6.6. instead of 0.5.*)
• Upgrade guardrails-api it comes as outdated v0.0.3 by transitivity. Upgrade to the latest from Github (0.1.0a)

[prnvkv]

Hi @CalebCourier , thanks for responding to this.

I am currently using this setup in langgraph. I see this issue only when an guardrail validators raises exception. For example:

guard = Guard().use_many(
    DetectPII(on_fail="exception"),
    ToxicLanguage(
        threshold=0.75,
        on_fail="exception"),
    DetectJailbreak(threshold=0.9,on_fail="exception"),

When one of these throws exceptions on failing the validation, i see the stack trace.

When i dont use the langgraph nodes i created for guardrails, i dont see this issue.
I will enable a tracing service to get more details on this.

[CalebCourier]

@prnvkv are you also using the guardrails-api package? e.g. via the guardrails start command? Another issue has been filed that demonstrates this behaviour in the api package: #1294

[prnvkv]

No @CalebCourier . I am currently just using guardrails and guardrails-hub.

[CalebCourier]

@prnvkv I was finally able to reproduce the issue you're reporting. For now you can disable this via the following:

from guardrails import Guard, settings

settings.disable_tracing = True

guard = Guard().use_many(...)

CC: @zsimjee @dtam @AlejandroEsquivel The underlying cause is likely the same as in #1294. I think the first question to answer is what the intended default bevaiour is. We currently default settings.disable_tracing to None which should yield a truthy value when negated leading to this same behvaviour all the time. It appears though, that the trace provider may no longer return a NonRecordingSpan when initialized without config.

I can confirm that this issue does not arise if older versions of the otel dependencies are installed:

opentelemetry-api==1.25.0
opentelemetry-exporter-otlp-proto-common==1.25.0
opentelemetry-exporter-otlp-proto-grpc==1.25.0
opentelemetry-exporter-otlp-proto-http==1.25.0
opentelemetry-instrumentation==0.46b0
opentelemetry-instrumentation-flask==0.46b0
opentelemetry-instrumentation-wsgi==0.46b0
opentelemetry-proto==1.25.0
opentelemetry-sdk==1.25.0
opentelemetry-semantic-conventions==0.46b0
opentelemetry-util-http==0.46b0

These issues do arise with newer versions though e.g. 1.35.x

[prnvkv]

Thanks for confirming this @CalebCourier .
I was able to workaround the problem by setting OTEL_SDK_DISABLED=true.

I will try your solution