Skip to content
fernet.consultores

Reference scenarioSplunk Observability CloudOpenTelemetryTechnical · 3 min read

Our charts have gaps and we don't know if the Collector is losing data.

Written by Fernando Mateo Ruiz · Splunk Enterprise Certified Architect · Splunk O11y Cloud Certified Metrics User · Reviewed on 28 October 2026

Reference scenario: it describes a common situation and how we tackle it. It is not a client project and includes no measured results.

In 30 seconds

The problem: Charts show gaps or traces arrive incomplete, and nobody can say whether the application, the network or the Collector is to blame.

What solves it: Treat the Collector as a production service: watch its internal metrics and alert when it refuses data, fails to send it or its queue fills up.

What changes: A gap becomes a diagnosis: which Collector, which pipeline and why.

Sound familiar?

  • Gaps appear in the charts during peaks.
  • After a network change, some hosts stop reporting.
  • The gateways restart for no obvious reason.
  • No detector watches the agents or the gateways.

The question to answer

Is every Collector receiving, processing and sending everything it should and, if not, where is data being lost?

Why it matters

If the pipeline loses data silently, every chart and detector built on it is wrong without anyone noticing.

Signals needed

  • EssentialCollector internal metrics (otelcol_*)
  • EssentialCollector logs
  • RecommendedHost metrics for agents and gateways
  • OptionalLoad balancer metrics

Architecture

Reference scenario: the architecture is generic and does not belong to any client.

How it’s implemented

  1. Check that every Collector sends its own metrics. Splunk's default configuration collects them with the prometheus/internal receiver on port 8888 and sends them in the metrics/internal pipeline.
  2. Compare what comes in with what goes out, per Collector: otelcol_receiver_accepted_* against otelcol_exporter_sent_*.
  3. Create detectors for losses: otelcol_receiver_refused_*, otelcol_exporter_send_failed_* and otelcol_exporter_enqueue_failed_* above zero.
  4. Watch the queue: when otelcol_exporter_queue_size approaches otelcol_exporter_queue_capacity, the destination can't keep up.
  5. Size memory and keep the memory_limiter processor: when it acts, the Collector refuses data on purpose so it doesn't run out of memory.
  6. For gateways behind a load balancer, check the health check and that traffic is spread across all of them.
Memory limit, queue and retries · example
processors:
  memory_limiter:
    check_interval: 2s
    limit_mib: ${SPLUNK_MEMORY_LIMIT_MIB}
exporters:
  otlp_http:
    # Keep your current endpoint and headers
    sending_queue:
      enabled: true
    retry_on_failure:
      enabled: true
# Size the queue for your peaks and watch otelcol_exporter_queue_size.

Step-by-step investigation

An illustrative example of how it would be investigated, from signal to action.

  1. 01 SignalDashboards show gaps on several hosts between 09:00 and 09:20.
  2. 02 ContextThe hosts were up, and their agents' send-failure counters rise in that window.
  3. 03 CorrelationOn the gateways, the queue reaches its capacity and otelcol_exporter_enqueue_failed_* starts counting.
  4. 04 CauseOne gateway had dropped out of the load balancer pool and the other couldn't absorb the peak.
  5. 05 DecisionIt isn't an application problem: the gateway layer is undersized and has a single point of failure.
  6. 06 ActionFix the health check, size both gateways for the peak and add detectors on queue usage and send failures.

Common mistakes

  • Not monitoring the Collector because “it's just an agent”.
  • Huge queues that hide the problem until memory runs out.
  • Removing memory_limiter so it “stops refusing data”.
  • Health checks that test the port but not the pipeline.

What this case doesn’t solve

  • It doesn't recover data that has already been lost.
  • It doesn't replace capacity planning for the gateways.

Expected result

Every gap in a chart has an explanation, and an alert fires when a Collector refuses, fails to send or queues data, before anyone notices missing data.

Observed result

This is a reference scenario: we don’t publish measured results until it has been tested in our lab or in an authorised project.

Requirements

Platform

Splunk Observability Cloud

Access

Configuration of agents, gateways and load balancer

Version

A supported version of the Splunk distribution of the OpenTelemetry Collector

Related services

Questions we’d ask you

  1. How many Collectors do you run, and in which mode (agent, gateway)?
  2. Does any detector watch the Collectors today?
  3. What happens to your data if a gateway goes down?
Let’s talk about your case