Skip to main content

Trigger PagerDuty alarms when dbt Cloud jobs fail

Updated
Webhooks
Advanced
Menu

    Introduction​

    This guide will teach you how to build and host a basic Python app which will monitor dbt Cloud jobs and create PagerDuty alarms based on failure. To do this, when a dbt Cloud job completes it will:

    • Check for any failed nodes (e.g. non-passing tests or errored models), and
    • create a PagerDuty alarm based on those nodes by calling the PagerDuty Events API. Events are deduplicated per run ID.

    Screenshot of the PagerDuty UI, showing an alarm created by invalid SQL in a dbt model

    In this example, we will use fly.io for hosting/running the service. fly.io is a platform for running full stack apps without provisioning servers etc. This level of usage should comfortably fit inside of the Free tier. You can also use an alternative tool such as AWS Lambda or Google Cloud Run.

    Prerequisites​

    This guide assumes some familiarity with:

    • dbt Cloud Webhooks
    • CLI apps
    • Deploying code to a serverless code runner like fly.io or AWS Lambda

    Clone the dbt-cloud-webhooks-pagerduty repo​

    This repository contains the sample code for validating a webhook and creating events in PagerDuty.

    Install flyctl and sign up for fly.io​

    Follow the directions for your OS in the fly.io docs, then from your command line, run the following commands:

    Switch to the directory containing the repo you cloned in step 1:

    #example: replace with your actual path
    cd ~/Documents/GitHub/dbt-cloud-webhooks-pagerduty

    Sign up for fly.io:

    flyctl auth signup

    Your console should show successfully logged in as YOUR_EMAIL when you're done, but if it doesn't then sign in to fly.io from your command line:

    flyctl auth login

    Launch your fly.io app​

    Launching your app publishes it to the web and makes it ready to catch webhook events:

    flyctl launch

    You will see a message saying that an existing fly.toml file was found. Type y to copy its configuration to your new app.

    Choose an app name of your choosing, such as YOUR_COMPANY-dbt-cloud-webhook-pagerduty, or leave blank and one will be generated for you. Note that your name can only contain numbers, lowercase letters and dashes.

    Choose a deployment region, and take note of the hostname that is generated (normally APP_NAME.fly.dev).

    When asked if you would like to set up Postgresql or Redis databases, type n for each.

    Type y when asked if you would like to deploy now.

    Sample output from the setup wizard:

    joel@Joel-Labes dbt-cloud-webhooks-pagerduty % flyctl launch
    An existing fly.toml file was found for app dbt-cloud-webhooks-pagerduty
    ? Would you like to copy its configuration to the new app? Yes
    Creating app in /Users/joel/Documents/GitHub/dbt-cloud-webhooks-pagerduty
    Scanning source code
    Detected a Dockerfile app
    ? Choose an app name (leave blank to generate one): demo-dbt-cloud-webhook-pagerduty
    automatically selected personal organization: Joel Labes
    Some regions require a paid plan (fra, maa).
    See https://fly.io/plans to set up a plan.
    ? Choose a region for deployment: [Use arrows to move, type to filter]
    ? Choose a region for deployment: Sydney, Australia (syd)
    Created app dbtlabs-dbt-cloud-webhook-pagerduty in organization personal
    Admin URL: https://fly.io/apps/demo-dbt-cloud-webhook-pagerduty
    Hostname: demo-dbt-cloud-webhook-pagerduty.fly.dev
    ? Would you like to set up a Postgresql database now? No
    ? Would you like to set up an Upstash Redis database now? No
    Wrote config file fly.toml
    ? Would you like to deploy now? Yes

    Create a PagerDuty integration application​

    See PagerDuty's guide for full instructions.

    Make note of the integration key for later.

    Configure a new webhook in dbt Cloud​

    See Create a webhook subscription for full instructions. Your event should be Run completed.

    Set the webhook URL to the host name you created earlier (APP_NAME.fly.dev)

    Make note of the Webhook Secret Key for later.

    Do not test the endpoint; it won't work until you have stored the auth keys (next step)

    Store secrets​

    The application requires three secrets to be set, using these names:

    • DBT_CLOUD_SERVICE_TOKEN: a dbt Cloud user token or service account token with at least the Metdata Only permission.
    • DBT_CLOUD_AUTH_TOKEN: the Secret Key for the dbt Cloud webhook you created earlier.
    • PD_ROUTING_KEY: the integration key for the PagerDuty integration you created earlier.

    Set these secrets as follows, replacing abc123 etc with actual values:

    flyctl secrets set DBT_CLOUD_SERVICE_TOKEN=abc123 DBT_CLOUD_AUTH_TOKEN=def456 PD_ROUTING_KEY=ghi789

    Deploy your app​

    After you set your secrets, fly.io will redeploy your application. When it has completed successfully, go back to the dbt Cloud webhook settings and click Test Endpoint.

    0