Configure our opentelemetry tracing so we can sample it and enable opentelemetry tracing in prod

[bisgaard-itis]

In the simcore stack tracing can either be enabled or disabled and it is enabled on master and staging deployments while disabled on production deployments. We should allow for a sampling based trace collection strategy in osparc simcore and enable tracing in production.

Steps to be done

• Implement a sampling strategy. As detailed in https://uptrace.dev/opentelemetry/sampling there are essentially 3 sampling strategies for tracing. The simplest and most reliable is head-based sampling. I suggest we go for this one. This would mean adding a few lines of code to https://github.com/ITISFoundation/osparc-simcore/blob/master/packages/service-library/src/servicelib/aiohttp/tracing.py and https://github.com/ITISFoundation/osparc-simcore/blob/master/packages/service-library/src/servicelib/fastapi/tracing.py. Essentially what needs to be done is

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

# Sample 10% of new traces; child spans inherit their parent’s decision
tracer_provider = TracerProvider(
    sampler=ParentBased(root=TraceIdRatioBased(0.1))
)

where the 0.1 is the sampling ratio, which should be an environment variable. I would suggest to keep that value 1.0 on master and staging deployments and 0.1 or 0.2 on production deployments.

• We should check that the above works with a sampling ratio of, say 0.1, on master before this moves to prod.
• We need to add environment variables in the different repo.configs to set the wanted sampling rate.
• Together with the above code modification I would also enable tracing in the local deployment by default .env-devel on osparc-simcore as it is very useful for local debugging.