Using Slack for Error Alerts in Your Data Pipeline

Data

Using Slack for Error Alerts in Your Data Pipeline

//

Any data pipeline will fail from time to time. Errors can come from hardware failures, networking issues or simple bugs. Sometimes, an error can occur because somebody forgot to update a password or pay the bill for a cloud service. To keep things running, you need a mechanism for logging errors and sending notifications to those who can deal with the problem immediately.

Why Not Use Email?

Now that smartphones are everywhere, system-generated emails have become the most common way for a failing process to call for help. Notifications can reach individuals or groups wherever they are within a few seconds. It’s easy for individuals to keep a record of past alerts because … well, it’s email.

Still, there are some problems with this email-based approach that can catch you out:

  • If you have lots of different systems, each might have a different way of managing who gets an email. Some may have a formal publisher/subscriber model, but others may simply have a hardcoded list of email addresses tucked away in a configuration file somewhere. It can be hard to keep track of them all and keep them consistent.
  • Emails sent automatically to multiple addresses can easily get caught in spam filters. Even if a mail message has worked once, it doesn’t mean it will always get through as spam filtering rules change all the time. The more email addresses you have in a distribution list, the more likely the message will be quarantined.
  • If someone new joins the group, they will have no visibility of past notifications or how they were handled.
  • Emails sent to groups can cause everyone to react, or worse, cause everyone to think that someone else will pick up the problem. There’s no easy way for the members of the group to coordinate a response (except by sending more emails).

If you’ve used Slack, you’ll know how Slack channels provide a really neat way of organising messages by topic and making them visible to interested groups. If a message is sent to a Slack group, anyone who is a member of the group will see it and any previous messages sent to the group. Anyone can add comments, or just little “I got this” emoji to keep the whole group updated. Like email, Slack can be accessed from your phone when you’re away from your desk.

Integrated Notifications with Slack

The good news is that in Slack, you can set up new channels, each with their own email address. Any email sent to that address will appear in the channel. It’s simple to set up. Here’s how:

1. Open the Slack desktop application.

2. Starting with the organisation at the top left, click the down arrow on the organisation name then Create a channel:

3. Enter a name and description for the channel:

4. Go to the channel and select Send emails to the channel:

5. If you like, you can customise how emails sent to the channel will appear:

6. Copy the email address and try sending a test email to it. Any emails sent to that address will now appear in the Slack channel:

7. Now, just configure your ETL tool to send emails to that address, and invite people to join the Slack channel.

More Helpful Resources

Thanks to Jonathan Solomon at Rivery, who wrote an earlier article on sending Rivery alerts to Slack and my colleague Igor Garlowski who highlighted this approach.

Below are links to articles walking you through various implementations of notices in Slack:

Configuring Email Notifications in Various ETL/ELT Tools

More Details on Sending Emails to Slack

More About the Author

Mike Oldroyd

Data Architect
Recapping Our Talk with the Consultancy Club at Imperial College London Robin Bergmans and I greatly enjoyed presenting to around 25 members of the “Consultancy Club” at Imperial College London ...
What Makes Good Analytics: The Beauty & Danger of Scalable Systems In this post, we’re launching a new blog series that will take a closer look at what makes good analytics. We’ll attempt to ...

See more from this author →

Subscribe to our newsletter

  • I understand that InterWorks will use the data provided for the purpose of communication and the administration my request. InterWorks will never disclose or sell any personal data except where required to do so by law. Finally, I understand that future communications related topics and events may be sent from InterWorks, but I can opt-out at any time.
  • This field is for validation purposes and should be left unchanged.

InterWorks uses cookies to allow us to better understand how the site is used. By continuing to use this site, you consent to this policy. Review Policy OK

×

Interworks GmbH
Ratinger Straße 9
40213 Düsseldorf
Germany
Geschäftsführer: Mel Stephenson

Kontaktaufnahme: markus@interworks.eu
Telefon: +49 (0)211 5408 5301

Amtsgericht Düsseldorf HRB 79752
UstldNr: DE 313 353 072