Observability: Trace context propagation within DPF

[marcgs]

What this PR changes/adds

Add trace context propagation at different points within DPF:

• DataPlaneManagerImpl: trace context is propagated to the corresponding async worker threads through the "DataFlowRequest" entities stored in the processing queue.
• ParallelSink: trace context is propagated to the async workers that process parts. No entity is persisted in this case, the trace context is propagated in memory.

Additionally TracingIntegrationTest was extended to verify traces propagation within DPF is working properly.

Why it does that

In order to have traces spanning across asynchronous calls it is required to propagate the tracing context to worker threads. See resulting trace visualization in Jaeger:

Linked Issue(s)

Closes #936

Checklist

• added appropriate tests?
• performed checkstyle check locally?
• added/updated copyright headers?
• documented public classes/methods?
• added/updated relevant documentation?
• added relevant details to the changelog? (skip with label no-changelog)
• formatted title correctly? (take a look at the CONTRIBUTING and styleguide for details)

[bscholtes1A]

Hi @marcgs
I am a bit embarassed with the fact of adding a dependency on opentelemetry into the *-spi. For example data-plane-transfer has a dependency on data-plane-spi. With your PR, this would therefore pull the opentelemetry dependency into data-plane-transfer.

Is there any way to limit dependencies on external libraries to non-spi submodules?

[marcgs]

Hi @marcgs I am a bit embarassed with the fact of adding a dependency on opentelemetry into the *-spi. For example data-plane-transfer has a dependency on data-plane-spi. With your PR, this would therefore pull the opentelemetry dependency into data-plane-transfer.

Is there any way to limit dependencies on external libraries to non-spi submodules?

HI @bscholtes1A , thanks for your comment :-). All open-telemetry dependencies have implementation scope, which actually prevents from the situation you have described. This PR is actually not adding any dependencies in the spi module either, these changes were already discussed and agreed with @jimmarino and @paullatzelsperger in previous PRs and documented in a decision record.

[marcgs]

@bscholtes1A @jimmarino Could we merge this PR as it already has 2 approvals?