{"id":14458,"date":"2021-09-03T20:17:00","date_gmt":"2021-09-03T14:47:00","guid":{"rendered":"http:\/\/ismiletechnologies.com\/?p=14458"},"modified":"2022-10-06T16:47:19","modified_gmt":"2022-10-06T11:17:19","slug":"incident-alerts-tending-to-alerts-that-matter","status":"publish","type":"post","link":"https:\/\/ismiletechnologies.com\/en_us\/cloud\/incident-alerts-tending-to-alerts-that-matter\/","title":{"rendered":"Incident Alerts \u2014 Tending to alerts that matter"},"content":{"rendered":"\n

According to Google’s SRE book, “At least 50% of each SRE’s time should be spent on engineering project work that will either reduce future toil or add service features. Feature development typically focuses on improving reliability, performance, or utilization, which reduces toil as a second-order effect. We share this 50% goal because toil expands if left unchecked and can quickly fill 100% of everyone’s time. The work of reducing toil and scaling up services is the “Engineering” in Site Reliability Engineering. Engineering work is what enables the SRE organization to scale up sublinearly with service size and to manage services more efficiently than either a pure Dev team or a pure Ops team.”<\/em>  <\/p>\n\n\n\n

A major challenge for an SRE team when optimizing activities between feature development and reducing toil is managing the alert noise. False alarms often lead to resources being pulled away from important issues that must be addressed immediately. The alert can be noisy for the SRE team if  <\/p>\n\n\n\n