Job scheduler

The job scheduler is the backbone of running jobs in dbt Cloud, bringing power and simplicity to building data pipelines in both continuous integration and production contexts. The scheduler frees teams from having to build and maintain their own infrastructure, and ensures the timeliness and reliability of data transformations.

The scheduler enables both cron-based and event-driven execution of dbt commands in the user’s data platform. Specifically, it handles:

• Cron-based execution of dbt Cloud jobs that run on a predetermined cadence
• Event-driven execution of dbt Cloud CI jobs triggered by pull requests to the dbt repo
• Event-driven execution of dbt Cloud jobs triggered by API
• Event-driven execution of dbt Cloud jobs manually triggered by a user to "Run Now"

The scheduler handles various tasks including queuing jobs, creating temporary environments to run the dbt commands required for those jobs, providing logs for debugging and remediation, and storing dbt artifacts for direct consumption/ingestion by the Discovery API.

The scheduler powers running dbt in staging and production environments, bringing ease and confidence to CI/CD workflows and enabling observability and governance in deploying dbt at scale.

Scheduler terms

Familiarize yourself with these useful terms to help you understand how the job scheduler works.

Term	Definition
Scheduler	The dbt Cloud engine that powers job execution. The scheduler queues scheduled or API-triggered job runs, prepares an environment to execute job commands in your cloud data platform, and stores and serves logs and artifacts that are byproducts of run execution.
Job	A collection of run steps, settings, and a trigger to invoke dbt commands against a project in the user's cloud data platform.
Job queue	The job queue acts as a waiting area for job runs when they are scheduled or triggered to run; runs remain in queue until execution begins. More specifically, the Scheduler checks the queue for runs that are due to execute, ensures the run is eligible to start, and then prepares an environment with appropriate settings, credentials, and commands to begin execution. Once execution begins, the run leaves the queue.
Over-scheduled job	A situation when a cron-scheduled job's run duration becomes longer than the frequency of the job’s schedule, resulting in a job queue that will grow faster than the scheduler can process the job’s runs.
Deactivated job	A situation where a job has reached 100 consecutive failing runs.
Prep time	The time dbt Cloud takes to create a short-lived environment to execute the job commands in the user's cloud data platform. Prep time varies most significantly at the top of the hour when the dbt Cloud Scheduler experiences a lot of run traffic.
Run	A single, unique execution of a dbt job.
Run slot	Run slots control the number of jobs that can run concurrently. Developer plans have a fixed number of run slots, while Enterprise and Team plans have unlimited run slots. Each running job occupies a run slot for the duration of the run. Team and Developer plans are limited to one project each. For additional projects, consider upgrading to the Enterprise plan.
Threads	When dbt builds a project's DAG, it tries to parallelize the execution by using threads. The thread count is the maximum number of paths through the DAG that dbt can work on simultaneously. The default thread count in a job is 4.
Wait time	Amount of time that dbt Cloud waits before running a job, either because there are no available slots or because a previous run of the same job is still in progress.

Scheduler queue

The scheduler queues a deployment job to be processed when it's triggered to run by a set schedule, a job completed, an API call, or manual action.

Before the job starts executing, the scheduler checks these conditions to determine if the run can start executing:

• Is there a run slot that's available on the account for use? — If all run slots are occupied, the queued run will wait. The wait time is displayed in dbt Cloud. If there are long wait times, upgrading to Enterprise can provide more run slots and allow for higher job concurrency.

• Does this same job have a run already in progress? — The scheduler executes distinct runs of the same dbt Cloud job serially to avoid model build collisions. If there's a job already running, the queued job will wait, and the wait time will be displayed in dbt Cloud.

If there is an available run slot and there isn't an actively running instance of the job, the scheduler will prepare the job to run in your cloud data platform. This prep involves readying a Kubernetes pod with the right version of dbt installed, setting environment variables, loading data platform credentials, and Git provider authorization, amongst other environment-setting tasks. The time it takes to prepare the job is displayed as Prep time in the UI.

An overview of a dbt Cloud job run

Treatment of CI jobs

When compared to deployment jobs, the scheduler behaves differently when handling continuous integration (CI) jobs. It queues a CI job to be processed when it's triggered to run by a Git pull request, and the conditions the scheduler checks to determine if the run can start executing are also different:

• Will the CI run consume a run slot? — CI runs don't consume run slots and will never block production runs.
• Does this same job have a run already in progress? — CI runs can execute concurrently (in parallel). CI runs build into unique temporary schemas, and CI checks execute in parallel to help increase team productivity. Teammates never have to wait to get a CI check review.

Treatment of merge jobs

When triggered by a merged Git pull request, the scheduler queues a merge job to be processed.

• Will the merge job run consume a run slot? — Yes, merge jobs do consume run slots.
• Does this same job have a run already in progress? — A merge job can only have one run in progress at a time. If there are multiple runs queued up, the scheduler will enqueue the most recent run and cancel all the other runs. If there is a run in progress, it will wait until the run completes before queuing the next run.

Job memory

In dbt Cloud, the setting to provision memory available to a job is defined at the account-level and applies to each job running in the account; the memory limit cannot be customized per job. If a running job reaches its memory limit, the run is terminated with a "memory limit error" message.

Jobs consume a lot of memory in the following situations:

• A high thread count was specified
• Custom dbt macros attempt to load data into memory instead of pushing compute down to the cloud data platform
• Having a job that generates dbt project documentation for a large and complex dbt project.
  • To prevent problems with the job running out of memory, we recommend generating documentation in a separate job that is set aside for that task and removing dbt docs generate from all other jobs. This is especially important for large and complex projects.

Refer to dbt Cloud architecture for an architecture diagram and to learn how the data flows.

Run cancellation for over-scheduled jobs

Scheduler won't cancel API-triggered jobs

The scheduler will not cancel over-scheduled jobs triggered by the API.

The dbt Cloud scheduler prevents too many job runs from clogging the queue by canceling unnecessary ones. If a job takes longer to run than its scheduled frequency, the queue will grow faster than the scheduler can process the runs, leading to an ever-expanding queue with runs that don’t need to be processed (called over-scheduled jobs).

The scheduler prevents queue clog by canceling runs that aren't needed, ensuring there is only one run of the job in the queue at any given time. If a newer run is queued, the scheduler cancels any previously queued run for that job and displays an error message.

The cancelled runs display an error message explaining why the run was cancelled and recommendations

To prevent over-scheduling, users will need to take action by either refactoring the job so it runs faster or modifying its schedule.

Deactivation of jobs beta

To reduce unnecessary resource consumption and reduce contention for run slots in your account, dbt Cloud will deactivate a deploy job or a CI job if it reaches 100 consecutive failing runs and indicate this through the use of banners. When this happens, scheduled and triggered-to-run jobs will no longer be enqueued.

To reactivate a deactivated job, you can either:

• Update the job's settings to fix the issue and save the job (recommended)
• Perform a manual run by clicking Run now on the job's page

Example of deactivation banner on job's page:

Example of deactivation banner on job's page

FAQs

I'm receiving a 'This run exceeded your account's run memory limits' error in my failed job

If you're receiving a This run exceeded your account's run memory limits error in your failed job, it means that the job exceeded the memory limits set for your account. All dbt Cloud accounts have a pod memory of 600Mib and memory limits are on a per run basis. They're typically influenced by the amount of result data that dbt has to ingest and process, which is small but can become bloated unexpectedly by project design choices.

Common reasons

Some common reasons for higher memory usage are:

• dbt run/build: Macros that capture large result sets from run query may not all be necessary and may be memory inefficient.
• dbt docs generate: Source or model schemas with large numbers of tables (even if those tables aren't all used by dbt) cause the ingest of very large results for catalog queries.

Resolution

There are various reasons why you could be experiencing this error but they are mostly the outcome of retrieving too much data back into dbt. For example, using the run_query() operations or similar macros, or even using database/schemas that have a lot of other non-dbt related tables/views. Try to reduce the amount of data / number of rows retrieved back into dbt by refactoring the SQL in your run_query() operation using group, where, or limit clauses. Additionally, you can also use a database/schema with fewer non-dbt related tables/views.

Video example

As an additional resource, check out this example video, which demonstrates how to refactor the sample code by reducing the number of rows returned.

If you've tried the earlier suggestions and are still experiencing failed job runs with this error about hitting the memory limits of your account, please reach out to support. We're happy to help!

Additional resources

• Blog post on how we shaved 90 mins off

Related docs

• dbt Cloud architecture
• Job commands
• Job notifications
• Webhooks
• dbt Cloud continuous integration