Distributed tracing (SQS & Lambda)

[axxander]

Hi 👋 ,

I'm trying to implement some basic distributed tracing across a couple of services.

I have a web backend (flask), SQS queue and a lambda that consumes it. I've been using the flask and lambda integrations provided.

In order to trace across the services and ultimately monitor the processing time, especially as it pertains to how long a message sits on the queue, I implemented the approach outlined in the docs. However, I'm a little lost.

My understanding is that a trace consists of one or more transactions. Then a transaction zero to many spans. And of course, spans can have some arbitrary number of child spans.

The issue I'm facing is that because the lambda integration creates a transaction when it initialises, I've ended up with:

• Flask transaction
• AWS lambda transaction with no reference/context to the parent trace
• Transaction within the lambda wrapping the processing of the queue, that is tied to the parent trace (docs outlines it)

So, when I'm in the UI, it is a little bit of a mess when it comes to looking at end to end flows.

So my question is, how do I implement the distributed tracing in a way such that I have just the two transactions (flask backend and lambda), so that they belong to the same trace.

• Only real solution I see if not using the lambda integration and simple starting the transaction myself with the sentry-trace and baggage, but wondered if there is a way to get the current transaction and overwrite some given fields

Any help/guidance would really be appreciated 🙌

[alexander-alderman-webb]

Hi @axxander,

I agree that manual trace propagation is cumbersome in your case because you must disable the lambda integration for the tracing context to be preserved.

There is no API to mutate the auto-instrumented transactions because the metadata should be immutable, and the transaction for the lambda function is created by the lambda integration before you have programmatic access to attach trace ID and baggage information. We also do not have access to relevant trace context in the lambda integration, since we wrap the lambda execution from the beginning, before the queue is accessed in the function body.

You can see more about our SDK development guidelines on this topic here: https://develop.sentry.dev/sdk/telemetry/traces/.

For future reference and for others that may stumble on this discussion, you can attach the sentry-trace and baggage like so

sqs.send_message(
    QueueUrl=queue_url,
    MessageBody="...",
    MessageAttributes={
        "sentry-trace": {
            "StringValue": sentry_sdk.get_traceparent(),
            "DataType": "String"
        },
        "baggage": {
            "StringValue": sentry_sdk.get_baggage(),
            "DataType": "String"
        }
    }
)

and start a span in the lambda with the context of the Flask transaction with

response = sqs.receive_message(
    QueueUrl=queue_url,
    MessageAttributeNames=["All"],
)

attributes = response["Messages"][0]["MessageAttributes"]
traceparent = attributes["sentry-trace"]["StringValue"]
baggage = attributes["baggage"]["StringValue"]

headers = {"sentry-trace": traceparent, "baggage": baggage}
transaction = sentry_sdk.continue_trace(headers)
with sentry_sdk.start_transaction(transaction):
    with sentry_sdk.start_span(
        op="queue.process",
        name="queue_consumer",
    ) as span:
      	...