Skip to content
fernet.consultores

Reference scenarioSplunk Observability CloudSynthetic MonitoringExecutive and technical · 3 min read

We find out about outages from our customers.

Written by Fernando Mateo Ruiz · Splunk Enterprise Certified Architect · Splunk O11y Cloud Certified Metrics User · Reviewed on 28 October 2026

Reference scenario: it describes a common situation and how we tackle it. It is not a client project and includes no measured results.

In 30 seconds

The problem: Everything looks green on the inside, but users can't log in or complete a payment, and the first warning comes from customer support.

What solves it: Continuously test the journeys that matter from the outside, alert on symptoms that affect users, and use APM to reach the cause.

What changes: The team detects the failure before the customer does and knows which step and which service it's in.

Sound familiar?

  • The first warning of an outage comes from customer support or social media.
  • The monitor says the server responds, but the form doesn't work.
  • An expired certificate or a DNS change took the service down without a single alert.
  • The failure is only visible from the outside: CDN, firewall or an external provider.

The question to answer

Can a user complete the actions that matter right now, from where they are, and how long does it take?

Why it matters

Every minute between an outage and its detection is paid for by users. And customers being the first to report it damages trust more than the outage itself.

Signals needed

  • EssentialSynthetic test results
  • RecommendedBack-end APM traces
  • OptionalReal user sessions (RUM)
  • OptionalInfrastructure metrics

Architecture

Reference scenario: the architecture is generic and does not belong to any client.

How it’s implemented

  1. Pick 3 to 5 journeys that must never fail (for example: login, search and payment) and use test accounts that don't create real transactions.
  2. Use the right test type: uptime for simple endpoints, API for services, and browser for multi-step journeys.
  3. Run the tests from locations close to your users, and from a private location if the application is internal.
  4. Alert when the test fails in several locations at once, not just one, so a problem at the test point isn't mistaken for an outage.
  5. Instrument the back end with OpenTelemetry so you can go from the test result to the related APM traces. API and HTTP test results can link directly to the correlated APM trace.

Step-by-step investigation

An illustrative example of how it would be investigated, from signal to action.

  1. 01 SignalThe browser test for the payment journey fails in three of four locations.
  2. 02 ContextThe page loads; the failing step is “confirm payment”.
  3. 03 CorrelationIn APM, the POST /api/pay endpoint has been returning 5xx errors for six minutes.
  4. 04 CauseThe payments service isn't getting a timely response from its external provider.
  5. 05 DecisionIt's neither the website nor your own infrastructure: it's an external dependency. Scaling won't fix it.
  6. 06 ActionTrigger customer communication and contact the provider, and create a dedicated detector for that dependency.

Common mistakes

  • Only checking that the home page returns a 200, not that the whole journey works.
  • Alerting from a single location: a network issue at the test point looks like an outage.
  • Tests that use real accounts or place real orders.
  • Forgetting certificates and DNS, which fail without warning.

What this case doesn’t solve

  • It doesn't measure every real user's experience: that's what RUM is for.
  • It covers the journeys you define, not every possible one.

Expected result

The team is alerted by a symptom that affects users, with the exact failing step and the related back-end traces, before the first complaint arrives.

Observed result

This is a reference scenario: we don’t publish measured results until it has been tested in our lab or in an authorised project.

Requirements

Platform

Splunk Observability Cloud with Synthetic Monitoring and APM

Instrumentation

OpenTelemetry in the back-end services

Tests

Test accounts; a private location if the application is internal

Related services

Questions we’d ask you

  1. What was the last incident you heard about from a customer?
  2. Which three journeys must never fail?
  3. Are your applications public, internal or both?
Let’s talk about your case