Skip to content
fernet.consultores

Reference scenarioSplunk Observability CloudAlerts and SLOsExecutive and technical · 3 min read

Hundreds of CPU alerts, and none of them tells us the customer is affected.

Written by Fernando Mateo Ruiz · Splunk Enterprise Certified Architect · Splunk O11y Cloud Certified Metrics User · Reviewed on 28 October 2026

Reference scenario: it describes a common situation and how we tackle it. It is not a client project and includes no measured results.

In 30 seconds

The problem: On-call gets paged for a host at 85 % CPU, while real outages arrive without any alert, because nothing measures what the user experiences.

What solves it: Service level objectives (SLOs) on request success and latency from APM, with burn-rate alerts that page only when the error budget is at risk.

What changes: Fewer pages, each one tied to user impact, and a reliability target that business and IT can both read.

Sound familiar?

  • The alerts channel is muted.
  • A host at 90 % CPU wakes someone up at night and nothing is wrong.
  • The last real outage didn't trigger a single alert.
  • Nobody remembers why half the detectors exist.

The question to answer

Is the service meeting the reliability we promised and, at the current rate, when will it stop meeting it?

Why it matters

An alert that doesn't reflect user impact teaches the team to ignore alerts. An SLO turns reliability into a figure the business understands.

Signals needed

  • EssentialAPM requests, errors and latency
  • RecommendedSynthetic test results
  • RecommendedInfrastructure metrics
  • OptionalDeployment events

Architecture

Reference scenario: the architecture is generic and does not belong to any client.

How it’s implemented

  1. Review the detectors that exist today: which fired in recent weeks, which led to an action and which were ignored.
  2. Choose the 2 or 3 journeys or endpoints that matter and define an indicator for each: request success or request latency.
  3. In Alerts › Service level objectives, create the SLO with the Service & endpoint type, a target (for example 99.5 %) and a compliance window.
  4. Page with the Burn rate and Error budget alerts. The Breach event alert tells you when the target has already been missed, so it's better for reporting.
  5. Move CPU, memory and disk detectors to a channel that doesn't page and use them for capacity, unless they predict an imminent failure (a disk that will fill up within hours).
  6. Add an owner and a runbook link to every detector that pages someone.
Indicator with a custom metric (SignalFlow) · example
# Only if the Service & endpoint type doesn't fit your case.
# APM metric: requests of one service in production.
good = data('service.request.count', filter=filter('sf_service', 'checkout') and filter('sf_environment', 'prod') and filter('sf_error', 'false')).sum()
total = data('service.request.count', filter=filter('sf_service', 'checkout') and filter('sf_environment', 'prod')).sum()

Step-by-step investigation

An illustrative example of how it would be investigated, from signal to action.

  1. 01 SignalThe burn-rate alert for the checkout SLO fires: at the current rate, the month's error budget would run out in two days.
  2. 02 ContextHost CPU is normal and no infrastructure detector has fired.
  3. 03 CorrelationIn APM, the errors come from POST /api/checkout and start right after the 10:40 deployment.
  4. 04 CauseThe new version calls an endpoint of the pricing service that no longer exists and gets a 404.
  5. 05 DecisionRoll back: the budget is being spent too fast to wait for a fix.
  6. 06 ActionRoll back, add the dependency check to the deployment checklist and review the target with the business.

Common mistakes

  • Paging on resource thresholds that don't mean user impact.
  • Setting a 100 % target: there's no error budget left and every blip becomes an incident.
  • Creating SLOs for everything instead of the journeys that matter.
  • Waiting for the breach event, when the budget is already spent.

What this case doesn’t solve

  • It doesn't set the target for you: that's an agreement with the business.
  • It doesn't cover services without instrumentation or without traffic.

Expected result

Fewer pages, each one linked to user impact and to an owner, and a monthly reliability figure for each critical service.

Observed result

This is a reference scenario: we don’t publish measured results until it has been tested in our lab or in an authorised project.

Requirements

Platform

Splunk Observability Cloud with APM

Instrumentation

OpenTelemetry in the services behind the journeys

Agreement

A target agreed with the business and an on-call owner

Related services

Questions we’d ask you

  1. How many alerts paged someone last month, and how many needed action?
  2. Which journeys must never fail, and what reliability do you promise for them?
  3. Who sets the reliability target today: IT, the business or nobody?
Let’s talk about your case