Everbridge · xMatters · 2020 – 2022
Service Intelligence
Four hundred alerts, one cause.
When a service degrades, monitoring reports every service that noticed — not the one that broke. I designed the service dependency map so operations and SRE teams could trace an incident to its cause, and see what was about to be hit next.
At a glance
- Role
- Lead designer — concept through shipped experience
- Team
- The CPO, the CTO and engineering leads
- Users
- IT operations, SRE and on-call engineers during live incidents
- Shipped in
- xMatters On-Demand, and extended in every release after
Context
xMatters was good at response: route the alert to the right on-call person, escalate when nobody answers, get people onto a bridge. What it could not do was say what was broken. It handled alerts, and an alert is an event — restoring a service means reasoning about the service.
Problem
Microservices fail in cascades. One degraded dependency produces hundreds of alerts from everything downstream, so the responder’s first job is not fixing anything — it is telling the cause from the echo.
- The cause was inferred, not shown. The dependency graph lived in senior engineers’ heads, so triage speed depended on who was on call.
- Blast radius was invisible. Nobody could answer “what else is about to break” until it already had.
- Remediation aimed at hosts, not services. Fixes landed on whatever the loudest alert pointed at, which is usually a victim.
Make the service the object
Zero to one. I led it from concept to release, working directly with the CPO, the CTO and the engineering leads — necessary, because I was proposing a new first-class object in the data model, not a screen.
A service knows who owns it, what it depends on, what depends on it, and whether it is healthy. Once that object exists, dependency is something the product can draw instead of something the responder has to remember.
Designing the map
Show the neighborhood, not the network
The whole graph is the obvious version and an unusable one — a hairball at 3am looks like information without being any. The map opens on the impacted service and its direct dependencies, then expands one hop at a time, upstream or downstream. Expansion follows a hypothesis, so the map grows along the investigation.
Let people take services off the map
Expansion alone is a ratchet — three hops in, the map is dense again. So each service carries controls to hide what sits upstream or downstream of it. Collapsing a branch mid-incident means choosing to stop looking at something; previewing the cost is what makes that safe enough to do.
When the responder already has a name
Expanding is for when you do not know where you are going. Often the responder already has a suspect — a change ticket, a colleague, a hunch — and the only question is whether the two are connected. Search takes a service name and traces the path to it, turning a scan into a lookup. The same field adds a service to the map, so the picture gets corrected where it is read.
Let responders arrange it
Auto-layout is a starting position, not an answer. Services drag where the responder wants them, and selecting one darkens its own connections so it stays legible in a crowd. Minimap, zoom and fit-to-screen cover the case the default view is trying to avoid: a map that has outgrown the screen.
Direction carries the meaning
Upstream and downstream are not two flavors of “related”. Upstream is where the cause probably is; downstream is who is about to feel it — the responder’s question and the incident commander’s. The map answers both without either having to translate.
The map is a place to act
A diagram you have to leave costs a context switch nobody has during an incident. From a service on the map you can open its info card, engage the group that owns it, and run automations scoped to it — health check, rollback, status-page update — without leaving the thing that told you where to run them.
Because an automation belongs to a service, the incident already knows which ones matter — ranked against the impacted services instead of listing every runbook in the company.
Tradeoffs I made
Defaulted to direct dependencies instead of the full graph.
Real signal often sits two or three hops out, and the default view hides it. An engineer who already knows the topology has to click to get back to where they started.
What I gave up: completeness on arrival. What I got: a first screen readable by someone who is not an expert in that estate. The experts lose a click; everyone else gets to participate.
Built services into the platform instead of shipping a visualization.
Drawing a map from data we already had would have shipped months sooner. A first-class object meant modeling, ownership, permissions and migration before anything reached a screen.
What I gave up: months, and visible progress for most of them. What I got: the map, the impacted-services list, the mobile views and the service-scoped automations all reading the same object — instead of a fifth vocabulary in a product that already had four.
Outcome
Root cause became something responders look at rather than deduce, and the services about to be hit became visible while there was still time to act — ahead of the cascade instead of behind it.
Services then spread through the product: impacted services in the incident console and the mobile apps, resolvers targeted by service name rather than by group, automations scoped to the service they fix. A feature gets used; a good object gets reused.
What I’d change
The map is only as honest as its dependency data, and that data is maintained by people who are not in the incident. I designed the moment of use and under-designed the moment of upkeep.
I would put the correction path inside the incident. When a responder finds a dependency the map did not know about, that is the best data the system will ever get — and it arrives with proof. Catching it there beats asking service owners to review their configuration.