Err on the side of Noisy Alerts 🔔

Tue, 02 Sep 2025 00:00:00 +0000

In the world of IT, alerts 🔔 are very important to know that there is something going on that is not normal behaviour.

General alerts: 🔗

Application performance degraded 📉
CPU utilisation is too high 💻
Memory consumed is greater than the specified threshold, like 80% 🧠
and many more… ♾️

So what do we generally do? 🔗

We create a rule that whenever a certain criterion is met, it should send an alert. Okay, we test in lower environments by simulating the real-time scenario.

We kept on adding alerts for all the possible scenarios that can fail, avoiding any corner cases. Thinking we should not be deprived of alerts of anything unusual happening.

So we did it. Yeah!! Pat on our backs. 👏

With this, we assume that we have covered everything. Yet unknowingly, we introduced a bigger problem. 😬

Alerts are implemented in the actual PRODUCTION. Now we see alerts coming now and then. Thus, overloading the mailboxes. 📧

Now, what happens next is that the team starts ignoring those alerts, considering them as usual. Or setting up rules to automatically route to some other folder, or even auto-delete. 🗑️

Now, when the real issue occurs, an alert is sent, but is ignored gracefully by the rules in our mailboxes. 🤫

Thus, chaos starts after the application crashes. 💥

So what went wrong here? 🔗

Alerts were there, the team was notified, but why were they ignored? 🤔

Having too many alerts makes it hard for the team to take any action. 😟

Now what to do? 🤷 🔗

Removing alerting is also not a valid option. ❌

The only viable options here are:

To optimise the alerting in such a way that it can reduce the number of alerts. 🛠️
Fixing the apps to avoid known failures and ignoring of the alerts. Thus, reducing overall number of alerts🔧
Never categorise any alert as a KNOWN alert. 🚨 Most of time, when we see a series of same KNOWN alert coming continuously, the known alert becomes a ALERT of CAUTION. ⚠️

Original Post: LinkedIn

What are your thoughts or feedback on this? I’d love to hear about your experiences! 👇

DevOps on DecipherMiddleware

Err on the side of Noisy Alerts 🔔

General alerts: 🔗

So what do we generally do? 🔗

So what went wrong here? 🔗

Now what to do? 🤷 🔗