Reference scenarioSplunk Observability CloudOpenTelemetryTechnical · 3 min read
Our charts have gaps and we don't know if the Collector is losing data.
Written by Fernando Mateo Ruiz · Splunk Enterprise Certified Architect · Splunk O11y Cloud Certified Metrics User · Reviewed on 28 October 2026
Reference scenario: it describes a common situation and how we tackle it. It is not a client project and includes no measured results.
In 30 seconds
The problem: Charts show gaps or traces arrive incomplete, and nobody can say whether the application, the network or the Collector is to blame.
What solves it: Treat the Collector as a production service: watch its internal metrics and alert when it refuses data, fails to send it or its queue fills up.
What changes: A gap becomes a diagnosis: which Collector, which pipeline and why.
Sound familiar?
- Gaps appear in the charts during peaks.
- After a network change, some hosts stop reporting.
- The gateways restart for no obvious reason.
- No detector watches the agents or the gateways.
The question to answer
Is every Collector receiving, processing and sending everything it should and, if not, where is data being lost?
Why it matters
If the pipeline loses data silently, every chart and detector built on it is wrong without anyone noticing.
Signals needed
- EssentialCollector internal metrics (otelcol_*)
- EssentialCollector logs
- RecommendedHost metrics for agents and gateways
- OptionalLoad balancer metrics
Architecture
Reference scenario: the architecture is generic and does not belong to any client.
How it’s implemented
- Check that every Collector sends its own metrics. Splunk's default configuration collects them with the
prometheus/internalreceiver on port 8888 and sends them in themetrics/internalpipeline. - Compare what comes in with what goes out, per Collector:
otelcol_receiver_accepted_*againstotelcol_exporter_sent_*. - Create detectors for losses:
otelcol_receiver_refused_*,otelcol_exporter_send_failed_*andotelcol_exporter_enqueue_failed_*above zero. - Watch the queue: when
otelcol_exporter_queue_sizeapproachesotelcol_exporter_queue_capacity, the destination can't keep up. - Size memory and keep the
memory_limiterprocessor: when it acts, the Collector refuses data on purpose so it doesn't run out of memory. - For gateways behind a load balancer, check the health check and that traffic is spread across all of them.
processors:
memory_limiter:
check_interval: 2s
limit_mib: ${SPLUNK_MEMORY_LIMIT_MIB}
exporters:
otlp_http:
# Keep your current endpoint and headers
sending_queue:
enabled: true
retry_on_failure:
enabled: true
# Size the queue for your peaks and watch otelcol_exporter_queue_size.
Step-by-step investigation
An illustrative example of how it would be investigated, from signal to action.
- 01 SignalDashboards show gaps on several hosts between 09:00 and 09:20.
- 02 ContextThe hosts were up, and their agents' send-failure counters rise in that window.
- 03 CorrelationOn the gateways, the queue reaches its capacity and
otelcol_exporter_enqueue_failed_*starts counting. - 04 CauseOne gateway had dropped out of the load balancer pool and the other couldn't absorb the peak.
- 05 DecisionIt isn't an application problem: the gateway layer is undersized and has a single point of failure.
- 06 ActionFix the health check, size both gateways for the peak and add detectors on queue usage and send failures.
Common mistakes
- Not monitoring the Collector because “it's just an agent”.
- Huge queues that hide the problem until memory runs out.
- Removing
memory_limiterso it “stops refusing data”. - Health checks that test the port but not the pipeline.
What this case doesn’t solve
- It doesn't recover data that has already been lost.
- It doesn't replace capacity planning for the gateways.
Expected result
Every gap in a chart has an explanation, and an alert fires when a Collector refuses, fails to send or queues data, before anyone notices missing data.
Observed result
This is a reference scenario: we don’t publish measured results until it has been tested in our lab or in an authorised project.
Requirements
Platform
Splunk Observability Cloud
Access
Configuration of agents, gateways and load balancer
Version
A supported version of the Splunk distribution of the OpenTelemetry Collector
Related services
Questions we’d ask you
- How many Collectors do you run, and in which mode (agent, gateway)?
- Does any detector watch the Collectors today?
- What happens to your data if a gateway goes down?