👩💻dbt Core integration
Import your dbt Core models into Count
Last updated
Import your dbt Core models into Count
Last updated
In an existing database connection, select Manage connection, then select the dbt tab and choose the dbt Core source.
You can import your dbt Core models by installing the Count GitHub app, and/or by executing dbt locally and uploading the manifest.json artifact file.
manifest.jsonClicking the Upload artifact button will display an upload form, where the required fields are
Project name - the name of this dbt project. Any macros with this package name will be available in the global scope within Count cells.
Manifest file - a manifest.json artifact with the correct format
Manifest identifier - an arbitrary name to refer to this manifest by in Count
Once one or more manifests have been uploaded, available artifacts will be listed below.
The Count GitHub integration simplifies the process of retrieving dbt Core metadata, and also allows committing changes back to your repository. To connect to GitHub, you'll be prompted to install the Count GitHub app, and give it access to one or more of your repositories.
Once the app has been installed, you will be redirected back to Count and prompted to choose:
Repository - the GitHub repository containing your dbt project
Branch - the default branch to use
Schema - the name of the database schema that contains your dbt models (this is usually defined in the appropriate profiles.yml file)
dbt version - the version of dbt to parse your project with
Once these details are confirmed, Count will then check your repository and attempt to determine:
Project name - the name of your dbt project
Project file location - where in your repository the dbt_project.yml file is located
Default model path - the default location for new model files
You may also define:
Environment variables (optional) - any environment variables required by dbt to parse your project files. Once set, non-secret variables are accessible from Jinja using the env_var() macro.
With the dbt Core integration, is possible to access different versions of your dbt metadata in different canvases.
In the table pane, a metadata selector button will appear which allows selecting:
GitHub branches (if the GitHub app is installed)
Uploaded files (if any manifest files have been uploaded)
Clicking the status indicator shows additional information about your dbt metadata, including:
Which commit it was generated for (if applicable)
Some information on the contents of the dbt manifest
If a branch is selected that has never had any metadata generated, then a warning will appear. To generate metadata for this branch, click the Refresh now button.
Any cell can be annotated as a dbt model using the toggle in the cell controls. If enabled, a dbt icon will also appear in the corner of the cell.
Once a cell has been marked as a model, any changes to its contents will be included in commits to GitHub. To signify the deletion of a model, just remove the contents of the cell with the model toggle enabled.
Model cells behave in all other respects like other cells - they can be referred to by their names or by the ref() macro, and renamed or deleted.
Once changes are ready to commit, clicking the Commit to GitHub button will open the model diff viewer. Here it is possible to:
Choose the branch to commit to
Choose a commit message
Choose the files to update
(Optionally) open a pull request after committing
When generating diffs, Count compares cell names to model names. If no model with that name exists, the cell is understood as a new model, otherwise as a change to an existing model.
When adding a new model the directory defaults to the one set in the connection dbt settings. To choose a different directory and/or file name, click the folder icon:
One step to be aware of when viewing diffs is the process of model compilation. Here Count will analyse the dependency graph of your cells, and ensure that any diffs it generates are compatible with dbt. Consider the following case that involves a mix of model and non-model cells:
The changes made in this example when compiling models include:
my_new_model refers to cell upstream_cell, so upstream_cell is included in my_new_model as a CTE.
my_new_model_2 refers to model upstream_model by name. To be compatible with dbt, this reference is converted to a ref().
Other changes may include:
Replacing any jinja expression involving the Count-only cells variable with the compiled output of that expression.
Hoisting and deduplicating any shared macro or set expressions.
The changes made here are the same as those made when exporting SQL and selecting the SQL + Jinja option.
When Count has made changes to your model definition, those changes will be listed above the diff:
Once the GitHub app has been installed on a repository, it is used for a number of tasks:
When refreshing your dbt metadata, the app is used to retrieve a private URL to download your repository as a .zip file. This URL is then sent to a temporary isolated virtual machine (VM), which downloads the zip file, unpacks the contents into a temporary directory, and executes dbt parse. The generated artifact files (like manifest.json) are then extracted from the VM, and the VM is deleted.
When committing to GitHub, the app is used to generate commits and open pull requests in your repository.
(Optionally) if you choose to change your commit authorship to your GitHub account, the app is used to verify your identity and view your GitHub email address.
The permissions required by the app are therefore:
Read access to repository contents - for task 1
Write access to repository contents - for task 2
Read and write access to pull requests - for task 2
Read access to your personal email addresses - for task 3
