Skip to content
fernet.consultores

Reference scenarioSplunk Observability CloudAPM and logsTechnical · 3 min read

We have logs, metrics and traces… but they don't talk to each other.

Written by Fernando Mateo Ruiz · Splunk Enterprise Certified Architect · Splunk O11y Cloud Certified Metrics User · Reviewed on 28 October 2026

Reference scenario: it describes a common situation and how we tackle it. It is not a client project and includes no measured results.

In 30 seconds

The problem: Metrics live in one place, traces in another and logs in Splunk; moving between them means copying a timestamp and searching “somewhere around there”.

What solves it: Share context between signals: trace_id and span_id in the logs and the same resource attributes everywhere.

What changes: From an anomalous metric to its trace, and from the trace to the exact logs of that request, in a few clicks.

Sound familiar?

  • You copy the time from a chart and search the logs for “roughly that time”.
  • The logs carry no request identifier.
  • Each tool names the same service differently.
  • Investigations depend on whoever knows the system best.

The question to answer

For this specific slow request, what happened in each service, and what did each one write to its logs?

Why it matters

Correlating by hand is what drags investigations out the most, and it makes them depend on specific people.

Signals needed

  • EssentialTraces (APM)
  • EssentialLogs with trace_id and span_id
  • RecommendedMetrics
  • EssentialCommon resource attributes

Architecture

Reference scenario: the architecture is generic and does not belong to any client.

How it’s implemented

  1. Agree on common attributes for every signal: service.name, deployment.environment and service.version.
  2. The Splunk Java agent adds trace_id, span_id and trace_flags to the logging context (MDC), but they aren't written to the file until the log pattern includes them.
  3. Index the logs in Splunk Cloud Platform (or Splunk Enterprise) with those fields extracted.
  4. Connect that Splunk platform to Splunk Observability Cloud with Log Observer Connect: from a trace, Related Content takes you to the logs that share its trace_id.
Logback pattern with trace context · example
<pattern>%d{yyyy-MM-dd HH:mm:ss} %-5level %logger{36} - %msg trace_id=%X{trace_id} span_id=%X{span_id} trace_flags=%X{trace_flags}%n</pattern>

Step-by-step investigation

An illustrative example of how it would be investigated, from signal to action.

  1. 01 SignalThe p99 latency of the checkout service doubles.
  2. 02 ContextIn APM, all the slow traces go through the inventory service.
  3. 03 CorrelationFrom the trace, the related logs show the inventory entries with that same trace_id.
  4. 04 CauseThe logs show retries against a cache that isn't responding.
  5. 05 DecisionThe problem is the cache, not inventory or the network.
  6. 06 ActionRestore the cache, create a detector on its availability, and turn the retries into a metric.

Common mistakes

  • Assuming the trace_id is already in the logs: the agent puts it in the context, but someone has to write it out.
  • Different service names in logs and traces.
  • Sources in different time zones.
  • Sending the same logs to two platforms and doubling the cost for no reason.

What this case doesn’t solve

  • It doesn't correlate services that aren't instrumented.
  • It doesn't replace well-structured logs.

Expected result

From a metric or a trace you reach the exact logs of that request, without searching by approximate time.

Observed result

This is a reference scenario: we don’t publish measured results until it has been tested in our lab or in an authorised project.

Requirements

Platform

Splunk Observability Cloud with APM and Log Observer Connect

Logs

Splunk Cloud Platform or Splunk Enterprise, with trace_id extracted

Instrumentation

OpenTelemetry and a log pattern with trace context

Related services

Questions we’d ask you

  1. Do your logs include any request identifier today?
  2. Where do your logs, metrics and traces live right now?
  3. How long does your team take to find the log of a specific request?
Let’s talk about your case