Any data pipeline will fail from time to time. Errors can come from hardware failures, networking issues or simple bugs. Sometimes, an error can occur because somebody forgot to update a password or pay the bill for a cloud service. To keep things running, you need a mechanism for logging errors and sending notifications to those who can deal with the problem immediately.

Why Not Use Email?

Now that smartphones are everywhere, system-generated emails have become the most common way for a failing process to call for help. Notifications can reach individuals or groups wherever they are within a few seconds. It’s easy for individuals to keep a record of past alerts because … well, it’s email.

Still, there are some problems with this email-based approach that can catch you out:

• If you have lots of different systems, each might have a different way of managing who gets an email. Some may have a formal publisher/subscriber model, but others may simply have a hardcoded list of email addresses tucked away in a configuration file somewhere. It can be hard to keep track of them all and keep them consistent.
• Emails sent automatically to multiple addresses can easily get caught in spam filters. Even if a mail message has worked once, it doesn’t mean it will always get through as spam filtering rules change all the time. The more email addresses you have in a distribution list, the more likely the message will be quarantined.
• If someone new joins the group, they will have no visibility of past notifications or how they were handled.
• Emails sent to groups can cause everyone to react, or worse, cause everyone to think that someone else will pick up the problem. There’s no easy way for the members of the group to coordinate a response (except by sending more emails).

If you’ve used Slack, you’ll know how Slack channels provide a really neat way of organising messages by topic and making them visible to interested groups. If a message is sent to a Slack group, anyone who is a member of the group will see it and any previous messages sent to the group. Anyone can add comments, or just little “I got this” emoji to keep the whole group updated. Like email, Slack can be accessed from your phone when you’re away from your desk.

Integrated Notifications with Slack

The good news is that in Slack, you can set up new channels, each with their own email address. Any email sent to that address will appear in the channel. It’s simple to set up. Here’s how:

1. Open the Slack desktop application.

2. Starting with the organisation at the top left, click the down arrow on the organisation name then Create a channel:

3. Enter a name and description for the channel:

4. Go to the channel and select Send emails to the channel:

5. If you like, you can customise how emails sent to the channel will appear:

6. Copy the email address and try sending a test email to it. Any emails sent to that address will now appear in the Slack channel:

7. Now, just configure your ETL tool to send emails to that address, and invite people to join the Slack channel.

More Helpful Resources

Thanks to Jonathan Solomon at Rivery, who wrote an earlier article on sending Rivery alerts to Slack and my colleague Igor Garlowski who highlighted this approach.

Below are links to articles walking you through various implementations of notices in Slack:

Configuring Email Notifications in Various ETL/ELT Tools

• Fivetran
• Rivery
• Matillion (via SNS, via Python script)
• Amazon SNS
• Python

More Details on Sending Emails to Slack

• Explore all the possible options.