How StackGen’s SRE Team Identifies Root Signals Across Cascading Alerts

Background

StackGen’s agent, Aiden, works across infrastructure provisioning, CI/CD deployments, incident response, and observability. As our customer base and internal environments have grown, so has the volume of alerts investigated by our DevOps and SRE teams. Our multi-cloud Kubernetes estate spans AWS, GCP, and Azure, including three internal development clusters that feed alerts into the same channels.

Our SRE team uses Aiden to investigate these alerts. Synced from Grafana, each alert can represent either the original infrastructure failure or its effect on a dependent service. Identifying the right starting point is therefore an important part of the investigation.

Challenges

• Cascading Alerts from a Single Failure: A single infrastructure failure can trigger alerts across several dependent services. The alert channel then fills with multiple messages that appear to describe separate problems, even when they originate from the same failure.
• Identifying the Root Signal: We still had to determine which alert was closest to the underlying failure and which alerts were downstream effects. Reviewing each alert independently made it harder to establish where the investigation should begin.

Manual Prioritization Across 400 Daily Alerts

On a typical day, our team receives around 400 alerts. Much of that volume comes from five categories:

• Kubernetes pods in a stuck state
• APM HTTP 4xx errors
• Kubernetes ReplicaSet mismatches
• Kubernetes nodes in a NotReady state
• High CPU load on hosts

These alerts span different systems and levels of severity. We had to review the technical context of each one before deciding what required immediate investigation. Alerts containing error-budget thresholds or other monitoring details added another interpretation step before we could act.

Solution

• Root and Downstream Alert Classification: Aiden classifies each alert as either a root signal or a downstream effect. We can use this classification to select the signal closest to the underlying failure and begin the investigation from there, rather than reviewing every alert in the order it arrived.
• Alert Refresh After Resolution: Once we resolve the root cause, we re-sync the alerts from Grafana. This removes the resolved alerts from Aiden and pulls in new alerts from Grafana that still need to be actioned. The team is then left with a current view of the alerts that require attention.
• Severity-Based Prioritization: Aiden automatically classifies alerts by severity and creates an initial order for investigation. For example, it prioritizes an alert for a Kubernetes node that is down over an alert for a service that is unable to scale up. A node failure can cause an outage and broader disruption, while the scalability issue has a lower immediate impact. This helps us begin with the alerts most likely to affect service availability. If the assigned severity does not reflect the operational context, we can re-categorize the alert before proceeding.

Technical Alert Interpretation

Some alerts are written in monitoring syntax and include thresholds, time windows, and reliability objectives. An SRE has to read and translate these details to understand what the alert means. Aiden converts the description into a human-readable summary while preserving the original technical values required for the investigation.

For example:

<table class="case-table is-compare"><thead><tr><th>Raw alert</th><th>Aiden’s summary</th></tr></thead><tbody><tr><td>“Error budget burning at 3.0x over 6h and 3d (objective 99.0%, threshold 0.0100000000).”</td><td>“Your service is experiencing errors three times faster than allowed over both the past 6 hours and 3 days, putting your 99% reliability goal at risk.”</td></tr></tbody></table>

Before vs After

<table class="case-table is-compare"><thead><tr><th>Before</th><th>With Aiden</th></tr></thead><tbody><tr><td>The team reviewed alerts individually to determine which one represented the underlying failure.</td><td>Aiden classifies alerts as root signals or downstream effects.</td></tr><tr><td>Engineers compared alert severity and context before deciding where to begin.</td><td>Aiden prioritizes alerts by severity, with the option to re-categorize them based on operational context.</td></tr><tr><td>Technical descriptions required engineers to interpret thresholds and monitoring terminology before proceeding.</td><td>Aiden summarizes the alert while retaining the technical values needed for the investigation.</td></tr></tbody></table>