Sending Airflow DAG Errors to VictorOps

By WikiExplain on 2023-03-14 · 3 mins read

If you are running your Airflow DAGs(Directed Acyclic Graph) in Google Cloud Composer , you would think you could use a GCP Monitoring Metrics. However, at the time of this writing, there is not an existing default metric we could use for DAGs or Tasks errors. Don't worry there is an easy way around it, leveraging Airflow natural support for email notification.

Setting up Airflow with VictorOps and Slack

Airflow Email Notification System

As show in the image above, we need to set up an email notification system. Luckily for us, we do not need to build anything only configure the different pieces of the puzzle. Follow the steps below to configure the corresponding service as needed.

I. Integrate Slack to VictorOps

Note: The official guide detailed requirements and helpful tools if you looking for a more in depth instructions

Requirements:

Steps:

Navigate to VictorOps web portal
Select Integrations
Select 3rd Party Integrations
Search for Slack
Click Enable Integration
Enter the desired workspace and log in
Follow the instruction and authorize the application
Select a default Slack channel
II. Create VictorOps Alert Routing Key

Note: Navigate to Settings >> Routing keys and verify if you already have a Slack channel routing key

Requirements:

Steps:

Navigate to VictorOps web portal
Select Settings*
Select Routing Keys*
Click Add Key button
Type the Routing Key name
Assign the Routing Key to an Escalation Policy for your team
III. Enable Email Integration in VictorOps

VictorOps allows you to send emails from your monitoring service or email provider to a dedicated VictorOps address. This way you will be creating, acknowledging, or resolving incidents in your VictorOps wall, I mean timeline :)

Requirements:

Able to modify the subject line of the source email. Plain and simple, you need to be able to add any of the expected key words critical, warning, info, acknowledgement, recovery. Otherwise, refer to Legacy Email Systems or reach out to VictorOps support team
VictorOps account with Administrative Privileges

Steps:

Log into VictorOps web portal
Select Integrations
Search Email Generic
Click Enable Integration button
Replace $routing_key with your Routing Key from the previous step

IV. Configure Airflow in Google Composer with SendGrid

Log into your Google Cloud console account
Select START WITH THE FREE PLANto create a SendGrid account
Fill up the form in the popped up window
Select Go Back to Google
Select Manage API keys on SendGrid website
Type a name for the key
Select Web API
Select Create API Key
Copy the generated key
Go back to your Cloud Composer environment
Select ENVIRONMENT VARIABLES
Select Edit
Select ADD ENVIRONMENT VARIABLES
Enter SENDGRID_API_KEY for the name
Pasted the generated SendGrid key for the value
Select ADD ENVIRONMENT VARIABLES to add another variable
Enter SENDGRID_MAIL_FROM for the name and your email from for the value. It could be the same one you use to create the SendGrid account or something such as noreply-composer@.
Select Save and grab a drink :)

Configure Airflow

Steps

Open your DAG file
Create a function or a module for the on_failure_callback property with the email_operator. Your function or module should look like this:

from airflow.operators.email_operator import EmailOperator
def report_failure(context):
   subject = 'Airflow critical: {{ti}}'
   html_content = (
      'Exception:<br>{{exception_html}}<br>')
   """Send custom email alerts."""
   return EmailOperator(
     task_id='task_id',
     to='your-destination-email@email.com',
     subject=subject,
     html_content=html_content
   ).execute(context)

Configure the fall back property in the DAG arguments.

default_args = {
   ....
   'on_failure_callback': report_failure,
   'email_on_failure': False,
   'email_on_retry': False}

Configure Airflow with AWS

Enable AWS SES service
Select SMTP Settings in the left side
Select Create My SMTP Credentials
Copy your credentials or click the "Download Credentials"
Go to your code base and open the file airflow.cfg and replace the following properties with your downloaded/copied AWS credentials and you are done :)

[smtp]
smtp_host = YOUR-EMAIL-HOST-SERVER-FROM-SES
smtp_starttls = True
smtp_ssl = False
smtp_user = SES-user-from-previous-step
smtp_password = SES-password-from-previous-step
smtp_port = 587
smtp_mail_from = @email.com

\ Note: Port will be provided by the SMTP provider typically is 25, 587

Configure Airflow with SendGrid SMTP Relay

Sign in to SendGrid
Select under Integrate using our Web API or SMTP relay
Select Choose under SMTP Relay
Type your API key name
Select Create Key
Select Next: Verify Integration
Verify the properties
Open airflow.cfg in your code repo
Replace properties with your smtp sendgrid generated values.
Set smtp_host to smtp.sendgrid.ne t
Set your smtp_user to apikey.
Set smtp_password to the API key generated in your SMTP account.
Set smtp_port to 587
Set smtp_mail_from to your account email

Configure Airflow with Gmail

Login to Gmail
Go to App passwords in your account security section
Select App
Select your device
Generate your password
Make sure you copy the generated password in somewhere you won't be able to see it again
Open airflow.cfg
Set smtp_host to smtp.gmail.com
Set smtp_user to your Gmail address
Set smtp_password to the generated 16 digits App passwords
Set smtp_port to 587
Set smtp_mail_from to your Gmail address

Show me the code!

[smtp]
smtp_host = smtp.gmail.com
smtp_starttls = True
smtp_ssl = False
smtp_user = Your-Gmail@gmail.com
smtp_password = Your-[App passwords](https://security.google.com/settings/security/apppasswords)
smtp_port = 587
smtp_mail_from = Your-Gmail@gmail.com

Another option can be to configure Airflow with esuite Relay