Comment on page
Sending Email Alerts in Apache Airflow with Sendgrid
Airflow is the automation tool used to create and manage reproducible data pipelines. There are specific mechanisms to send alerts to users when there is an error when executing these pipelines.
One of the most common ways of sending these alerts is email. Other's include slack, PagerDuty, or Zapier integration. In one of the blogs, I have covered different ways of sending an email with Airflow. This one is specific to SendGrid.
Before we use SendGrid with Airflow, you to an account with Sendgrid, go ahead and register yourself. You get 100 emails/day on the free plan, which is good enough for this exercise.
Next, you need to authenticate your domain.

You will get a set of DNS records to add to the DNS provider on adding your domain name.
- 1.Web API
- 2.SMTP Relay
Web API providers SDK in different languages for you to integrate email sending logic to the code.

SMTP Relay is used when you already have an email sending app and want to configure the SMTP server.
Since Airflow already has the email sending logic, it needs the SMTP details to start sending emails.
Setting up SMTP details is pretty straightforward. You have to give a name for the API key, and your key will be generated.

Now it's only a matter of adding it to Airflow.
There are two ways you can integrate SendGrid with Airflow.
With default SMTP configuration airflow uses the default SMTP email backend
airflow.utils.email.send_email_smtp
to send the email. Set the following airflow configurations and you're good to goAIRFLOW__SMTP__SMTP_HOST=smtp.sendgrid.net
AIRFLOW__SMTP__SMTP_STARTTLS=False
AIRFLOW__SMTP__SMTP_SSL=False
AIRFLOW__SMTP__SMTP_USER=apikey
AIRFLOW__SMTP__SMTP_PASSWORD=<generated-api-key>
AIRFLOW__SMTP__SMTP_PORT=587
AIRFLOW__SMTP__SMTP_MAIL_FROM=<your-from-email>
The equivalent
airflow.cfg
looks like this.[smtp]
smtp_host=smtp.sendgrid.net
smtp_starttls=False
smtp_ssl=False
smtp_ssl=apikey
smtp_password=<generated-api-key>
smtp_port=587
smtp_email=<your-from-email>
Airflow also has a Sendgrid specific provider, which you can use as follows.
- 1.Install
pip install apache-airflow-providers-sendgrid
- 2.Set the following mandatory param
AIRFLOW__EMAIL__EMAIL_BACKEND=airflow.providers.sendgrid.utils.emailer.send_email
AIRFLOW__EMAIL__EMAIL_CONN_ID=sendgrid_default
- 1.From the Airflow UI, create a Connection of type Email and add the Sendgrid SMTP parameters

That's it. After configuring, you can use any email sending method, but the whole point of this blog is to send alerts on failure. The following DAG has a
PythonOperator
, which will fail no matter what.def my_custom_function():
raise Exception
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email_on_failure': True,
'email':"<to-email>"
}
with DAG('example_dag',
start_date=datetime(2022, 1, 1),
max_active_runs=1,
schedule_interval=None,
default_args=default_args,
) as dag:
tn = PythonOperator(
task_id=f'python_print_date_1',
python_callable=my_custom_function
)
Add this to your Airflow DAGs, and you will get an email on failure. You can also set
email_on_retry
as True
to send emails on retry.Airflow currently does not support
email_on_sucess
. The only way to achieve this is via EmailOperator
or a custom PythonOperator.
--
Newsletter embed
Newsletter embed