Using dbt with Dagster, part two: Load dbt models as Dagster assets#

At this point, you should have a fully-configured dbt project that's ready to work with Dagster.

In this step, you'll finally begin integrating dbt with Dagster! To do this, you'll:

• Load the dbt models into Dagster as assets
• Add a Dagster repository and supply a dbt resource
• View the assets in Dagit

Step 1: Load the dbt models as assets#

In this step, you'll load the dbt models into Dagster as assets using the dagster-dbt library.

Open the __init__.py file, located in /tutorial_template/tutorial_dbt_dagster/assets, and add the following code:

from dagster_dbt import load_assets_from_dbt_project

from dagster import file_relative_path


DBT_PROJECT_PATH = file_relative_path(__file__, "../../jaffle_shop")
DBT_PROFILES = file_relative_path(__file__, "../../jaffle_shop/config")

dbt_assets = load_assets_from_dbt_project(
    project_dir=DBT_PROJECT_PATH, profiles_dir=DBT_PROFILES, key_prefix=["jaffle_shop"]
)

Let's discuss what this example is doing, specifically the load_assets_from_dbt_project function. This function loads dbt models into Dagster as assets, creating one Dagster asset for each model.

When invoked, this function:

1. Compiles your dbt project,
2. Parses the metadata provided by dbt, and
3. Generates a set of software-defined assets reflecting the models in the project. These assets share the same underlying op, which will invoke dbt to run the models represented by the loaded assets.

Let's take a look at the arguments we've supplied:

• project_dir, which is the path to the dbt project
• profiles_dir, which is the path to the dbt project's connection profiles
• key_prefix, which is a prefix to apply to all models in the dbt project

Step 2: Add a Dagster repository and supply a dbt resource#

Next, you'll create a Dagster repository to supply resources to the dbt assets from the previous step.

Assets loaded from dbt require a dbt resource, which is responsible for firing off dbt CLI commands. Using the dbt_cli_resource resource, we can supply a dbt resource to the dbt project.

Open the repository.py file, located in /tutorial_template/tutorial_dbt_dagster, and add the following code:

import os

from dagster_dbt import dbt_cli_resource
from tutorial_dbt_dagster import assets
from tutorial_dbt_dagster.assets import DBT_PROFILES, DBT_PROJECT_PATH

from dagster import load_assets_from_package_module, repository, with_resources


@repository
def tutorial_dbt_dagster():
    return with_resources(
        load_assets_from_package_module(assets),
        {
            "dbt": dbt_cli_resource.configured(
                {
                    "project_dir": DBT_PROJECT_PATH,
                    "profiles_dir": DBT_PROFILES,
                },
            ),
        },
    )

Let's take a look at what's happening here:

• Using with_resources, we've provided resources to the assets in the repository. In this example, that's the dbt_cli_resource resource.
• Using load_assets_from_package_module, we've imported all assets in the assets module into the repository. This approach allows any new assets we create to be automatically added to the repository instead of needing to manually add them one by one.

Step 3: View the assets in Dagit#

In this step you'll start Dagit, Dagster's web interface.

1. To start Dagit, run the following in /tutorial_template:

   dagit
   

   Which will result in output similar to:

   Serving dagit on http://127.0.0.1:3000 in process 70635
   

2. In your browser, navigate to http://127.0.0.1:3000. The page will display the assets:

   

What's next?#

At this point, you've loaded your dbt models into Dagster as assets, supplied them with a dbt resource, and viewed them in Dagit's asset graph. The next step is to add upstream Dagster assets and kick off a run that materializes them.