dbt flags: precedence, behavior change flags and flags in Jinja

1.Where flags are set and which setting wins

Flags change how dbt runs, not what it runs. Selecting models and setting materializations is the job of resource configs. Flags, also called global configs, control things like log output, whether warnings become errors, and whether a run stops at the first failure. They are called global because every dbt command accepts them and they can be set in several places. That is why debugging them is tricky. If a run behaves differently in CI than on your laptop, the flag may come from a place you didn't check.

Each of the four places has its own purpose.

- CLI options set behavior for a single invocation. - Environment variables vary behavior by runtime environment (development, production, CI) or by developer. - Project-level flags in the flags: dictionary of dbt_project.yml are version-controlled defaults for everyone who runs the project. They are also where you opt in to or out of behavior changes. - User settings in ~/.dbt/user_settings.yml hold personal preferences that apply to every project on your machine. dbt login writes this file.

The most specific setting wins. If none of the four sets the flag, dbt uses its built-in default. The fail_fast flag shows the pattern.

# dbt_project.yml
flags:
  # set default for running this project -- anywhere, anytime, by anyone
  fail_fast: true

# set this environment variable to 'True' (bash syntax)
export DBT_ENGINE_FAIL_FAST=1
dbt run

dbt run --fail-fast # set to True for this specific invocation
dbt run --no-fail-fast # set to False

The environment variable prefix depends on the version. dbt v1.11 and later use DBT_ENGINE_, and v1.10 and earlier use DBT_. If an old CI script sets a flag with the old prefix after an upgrade, check that the variable name matches the version you run. Boolean flags come in pairs: --fail-fast turns the flag on and --no-fail-fast turns it off. Non-boolean flags take a value, as in --printer-width=80.

Put CLI flags after the subcommand. Write dbt run --no-populate-cache, not dbt --no-populate-cache run. Passing flags before the subcommand is legacy behavior that dbt Labs can remove at any time, and passing the same flag on both sides of the subcommand is not supported. User settings sit at the bottom of the order. If user_settings.yml sets manage_state: true and you run dbt run --no-manage-state, the CLI option wins for that run.

Sources123

2.What the flags dictionary can and cannot hold

You can set most flags in all of these places, but there are two important exceptions. The first is file paths. Flags such as log_path and state point at locations used during execution, and you cannot set them in dbt_project.yml. To change them, pass a CLI option such as --log-path or --state, or set an environment variable such as DBT_ENGINE_LOG_PATH. Settings that tell dbt where to find project resources, like model-paths, do go in dbt_project.yml. They are top-level keys outside the flags: dictionary, because they are meant to stay fixed whatever command or environment runs.

The second exception runs the other way, and the next section covers it. Behavior change flags can only be set in dbt_project.yml. Some flags also work differently from the rest. empty is a CLI option or environment variable and has no project setting. full_refresh and store_failures can appear in the project only as resource configs, not as flags: entries. To check a flag before you put it in a file, look at the 'In project?' column of the flags table.

Sources1

3.Behavior change flags: opting in, opting out, and handling the warnings

A behavior change is a deliberate update that makes the same project code and the same commands give a different result. Examples include a new validation error, a changed signature on a built-in macro that your project reimplemented, or an adapter method removed from the {{ adapter }} Jinja object. dbt puts each of these behind a flag so you decide when to adopt it. A bug fix doesn't count as a behavior change. Neither does a new warning or reworded log text. When an upgrade makes a project that used to work start failing validation, a behavior change flag is a likely cause.

Every behavior change flag goes through three phases. When introduced, dbt supports both the old and new behavior, and the flag is disabled by default. When mature, the default switches to the new behavior. You can still keep the old behavior, but you may see deprecation warnings. When removed, the old behavior is gone from the codebase. Several flags are marked as removed in dbt v2, where the new behavior is always on. For example, skip_nodes_if_on_run_start_fails can stop a build, and state_modified_compare_more_unrendered_values changes which nodes get selected, which can affect CI.

You can set these flags only in the flags: dictionary of dbt_project.yml. No environment variable or CLI option sets them. They are tied closely to project code, so a change goes through a pull request with the same testing and peer review as any other code change, and the value should not differ between environments or users for long. To opt out, set the flag to false. Opting out does not silence anything. You keep seeing warnings for the legacy behavior, and you keep that behavior even after the maturity date, until you either migrate and set the flag to true or silence the warning.

Two flags control how warnings are handled. warn_error turns every warning into an error. warn_error_options gives you finer control through three parameters: error lists the warnings to promote to errors, warn keeps listed warnings as warnings when error is all, and silence ignores warnings. The two flags are mutually exclusive. Setting both, even in different places such as an environment variable and a CLI flag, gives a usage error. Be careful with --warn-error in production: a warning added in a future dbt version will then fail your job. List only the warnings you mean to promote.

flags:
  warn_error_options:
    error: # Previously called "include"
    warn: # Previously called "exclude"
     ? : # To silence or ignore warnings
      - NoNodesForSelectionCriteria

Sources45

4.Reading flag values in Jinja

Your own Jinja can read flags through the flags context variable. Use it when logic should depend on how dbt was invoked. Common cases are materialization logic for different run modes, which read flags.FULL_REFRESH, flags.STORE_FAILURES and flags.EMPTY, and hooks that run only for certain commands, which read flags.WHICH. A materialization that drops and rebuilds on a full refresh checks the flag directly. What --empty does to a model is covered in [dry runs with --empty](lesson:1.12). Here, flags.EMPTY is just a value your code can test.

{% if flags.FULL_REFRESH %}
drop table ...
{% else %}
-- no-op
{% endif %}

# dbt_project.yml

on-run-start:
  - '{{ log("I will stop at the first sign of trouble", info = true) if flags.FAIL_FAST }}'

Use flags only in code that runs at execution time. Don't use them in anything dbt resolves while parsing, such as configs, properties, or ref and source dependencies. Flag values can change on every invocation. With partial parsing on, dbt reuses earlier parse results, so a value read at parse time goes stale and gives wrong results on later runs. If a config seems to keep the full-refresh setting from an earlier run, look for a flag read at parse time.

Sources16

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.You can opt in to a behavior change for one CI run with an environment variable or a CLI option.Why is that wrong?

   Behavior change flags can only be set in the flags dictionary of dbt_project.yml, so changes go through version control and review.

   Covered in Behavior change flags: opting in, opting out, and handling the warnings

2. 2.Setting a behavior change flag to false makes its deprecation warnings go away.Why is that wrong?

   Opting out keeps the legacy behavior, but the warnings continue until you migrate and set the flag to true, or silence them with warn_error_options.silence.

   Covered in Behavior change flags: opting in, opting out, and handling the warnings

3. 3.log_path or state can be set under flags: in dbt_project.yml like any other default.Why is that wrong?

   Runtime file-path flags are excluded from dbt_project.yml. Set them with a CLI option or an environment variable.

   Covered in What the flags dictionary can and cannot hold

4. 4.--warn-error and --warn-error-options can be combined, for example one in an environment variable and one on the command line.Why is that wrong?

   The two are mutually exclusive wherever they are set. Specifying both gives a usage error.

   Covered in Behavior change flags: opting in, opting out, and handling the warnings