Implicit override for built-in materializations from installed packages in dbt-core
Description
Malicious dbt packages can override core components, leading to potential data exfiltration; fixed in versions 1.8.0, 1.6.14, and 1.7.14.
AI Insight
LLM-synthesized narrative grounded in this CVE's description and references.
Malicious dbt packages can override core components, leading to potential data exfiltration; fixed in versions 1.8.0, 1.6.14, and 1.7.14.
Root
Cause
CVE-2024-40637 affects dbt, a data transformation tool. The vulnerability stems from the fact that when a user installs a dbt package, it can override macros, materializations, and other core components of dbt by design [1][2]. This feature allows packages to extend functionality, but it also means a malicious package could override these components with harmful code [3].
Exploitation
An attacker could create a malicious dbt package that overrides built-in materializations or macros with code designed to exfiltrate data. If a victim installs and runs this package, the malicious code would execute in the context of the victim's dbt project, potentially allowing the attacker to access sensitive data [1][3][4]. The attack does not require additional authentication beyond normal package installation, as dbt trusts packages to override components [2].
Impact
A successful attacker could exfiltrate sensitive data, such as data stored in BigQuery or other data warehouses, by having the victim unintentionally run malicious SQL on the attacker's behalf [4]. This could lead to data theft or unauthorized data access, as the victim's credentials are used to access and move data to the attacker's writable dataset [4].
Mitigation
The vulnerability has been fixed in dbt versions 1.8.0, 1.6.14, and 1.7.14. Users are advised to upgrade immediately [3]. For those upgrading to 1.6.14 or 1.7.14, a configuration flag flags.require_explicit_package_overrides_for_builtin_materializations must be set to False in dbt_project.yml to maintain existing package behavior [1][3]. No workarounds are available [3].
AI Insight generated on May 20, 2026. Synthesized from this CVE's description and the cited reference URLs; citations are validated against the source bundle.
Affected packages
Versions sourced from the GitHub Security Advisory.
| Package | Affected versions | Patched versions |
|---|---|---|
dbt-corePyPI | >= 1.6.0, < 1.6.14 | 1.6.14 |
dbt-corePyPI | >= 1.7.0, < 1.7.14 | 1.7.14 |
Affected products
2Patches
419ad148bfeeef71a5d5c42493c82a0296d22deprecate materialization overrides from imported packages (#9971) (#10008)
39 files changed · +1737 −1514
.changes/unreleased/Features-20231218-195854.yaml+6 −0 added@@ -0,0 +1,6 @@ +kind: Features +body: Move flags from UserConfig in profiles.yml to flags in dbt_project.yml +time: 2023-12-18T19:58:54.075811-05:00 +custom: + Author: gshank + Issue: "9183"
.changes/unreleased/Features-20240422-173703.yaml+6 −0 added@@ -0,0 +1,6 @@ +kind: Features +body: Add require_explicit_package_overrides_for_builtin_materializations to dbt_project.yml flags, which can be used to opt-out of overriding built-in materializations from packages +time: 2024-04-22T17:37:03.892268-04:00 +custom: + Author: michelleark + Issue: "10007"
.changes/unreleased/Under the Hood-20240418-172528.yaml+6 −0 added@@ -0,0 +1,6 @@ +kind: Under the Hood +body: Raise deprecation warning if installed package overrides built-in materialization +time: 2024-04-18T17:25:28.37886-04:00 +custom: + Author: michelleark + Issue: "9971"
core/dbt/cli/flags.py+30 −13 modified@@ -3,6 +3,7 @@ from dataclasses import dataclass from importlib import import_module from multiprocessing import get_context +from pathlib import Path from pprint import pformat as pf from typing import Any, Callable, Dict, List, Optional, Set, Union @@ -11,8 +12,8 @@ from dbt.cli.exceptions import DbtUsageException from dbt.cli.resolvers import default_log_path, default_project_dir from dbt.cli.types import Command as CliCommand -from dbt.config.profile import read_user_config -from dbt.contracts.project import UserConfig +from dbt.config.project import read_project_flags +from dbt.contracts.project import ProjectFlags from dbt.exceptions import DbtInternalError from dbt.deprecations import renamed_env_var from dbt.helper_types import WarnErrorOptions @@ -24,7 +25,8 @@ FLAGS_DEFAULTS = { "INDIRECT_SELECTION": "eager", "TARGET_PATH": None, - # Cli args without user_config or env var option. + "WARN_ERROR": None, + # Cli args without project_flags or env var option. "FULL_REFRESH": False, "STRICT_MODE": False, "STORE_FAILURES": False, @@ -76,7 +78,7 @@ class Flags: """Primary configuration artifact for running dbt""" def __init__( - self, ctx: Optional[Context] = None, user_config: Optional[UserConfig] = None + self, ctx: Optional[Context] = None, project_flags: Optional[ProjectFlags] = None ) -> None: # Set the default flags. @@ -201,27 +203,40 @@ def _assign_params( invoked_subcommand_ctx, params_assigned_from_default, deprecated_env_vars ) - if not user_config: + if not project_flags: + project_dir = getattr(self, "PROJECT_DIR", str(default_project_dir())) profiles_dir = getattr(self, "PROFILES_DIR", None) - user_config = read_user_config(profiles_dir) if profiles_dir else None + if profiles_dir and project_dir: + project_flags = read_project_flags(project_dir, profiles_dir) + else: + project_flags = None # Add entire invocation command to flags object.__setattr__(self, "INVOCATION_COMMAND", "dbt " + " ".join(sys.argv[1:])) - # Overwrite default assignments with user config if available. - if user_config: + if project_flags: + # Overwrite default assignments with project flags if available. param_assigned_from_default_copy = params_assigned_from_default.copy() for param_assigned_from_default in params_assigned_from_default: - user_config_param_value = getattr(user_config, param_assigned_from_default, None) - if user_config_param_value is not None: + project_flags_param_value = getattr( + project_flags, param_assigned_from_default, None + ) + if project_flags_param_value is not None: object.__setattr__( self, param_assigned_from_default.upper(), - convert_config(param_assigned_from_default, user_config_param_value), + convert_config(param_assigned_from_default, project_flags_param_value), ) param_assigned_from_default_copy.remove(param_assigned_from_default) params_assigned_from_default = param_assigned_from_default_copy + # Add project-level flags that are not available as CLI options / env vars + for ( + project_level_flag_name, + project_level_flag_value, + ) in project_flags.project_only_flags.items(): + object.__setattr__(self, project_level_flag_name.upper(), project_level_flag_value) + # Set hard coded flags. object.__setattr__(self, "WHICH", invoked_subcommand_name or ctx.info_name) object.__setattr__(self, "MP_CONTEXT", get_context("spawn")) @@ -235,9 +250,11 @@ def _assign_params( # Starting in v1.5, if `log-path` is set in `dbt_project.yml`, it will raise a deprecation warning, # with the possibility of removing it in a future release. if getattr(self, "LOG_PATH", None) is None: - project_dir = getattr(self, "PROJECT_DIR", default_project_dir()) + project_dir = getattr(self, "PROJECT_DIR", str(default_project_dir())) version_check = getattr(self, "VERSION_CHECK", True) - object.__setattr__(self, "LOG_PATH", default_log_path(project_dir, version_check)) + object.__setattr__( + self, "LOG_PATH", default_log_path(Path(project_dir), version_check) + ) # Support console DO NOT TRACK initiative. if os.getenv("DO_NOT_TRACK", "").lower() in ("1", "t", "true", "y", "yes"):
core/dbt/config/__init__.py+1 −1 modified@@ -1,4 +1,4 @@ # all these are just exports, they need "noqa" so flake8 will not complain. -from .profile import Profile, read_user_config # noqa +from .profile import Profile # noqa from .project import Project, IsFQNResource, PartialProject # noqa from .runtime import RuntimeConfig # noqa
core/dbt/config/profile.py+1 −38 modified@@ -8,7 +8,7 @@ from dbt.clients.system import load_file_contents from dbt.clients.yaml_helper import load_yaml_text from dbt.contracts.connection import Credentials, HasCredentials -from dbt.contracts.project import ProfileConfig, UserConfig +from dbt.contracts.project import ProfileConfig from dbt.exceptions import ( CompilationError, DbtProfileError, @@ -19,7 +19,6 @@ ) from dbt.events.types import MissingProfileTarget from dbt.events.functions import fire_event -from dbt.utils import coerce_dict_str from .renderer import ProfileRenderer @@ -51,27 +50,13 @@ def read_profile(profiles_dir: str) -> Dict[str, Any]: return {} -def read_user_config(directory: str) -> UserConfig: - try: - profile = read_profile(directory) - if profile: - user_config = coerce_dict_str(profile.get("config", {})) - if user_config is not None: - UserConfig.validate(user_config) - return UserConfig.from_dict(user_config) - except (DbtRuntimeError, ValidationError): - pass - return UserConfig() - - # The Profile class is included in RuntimeConfig, so any attribute # additions must also be set where the RuntimeConfig class is created # `init=False` is a workaround for https://bugs.python.org/issue45081 @dataclass(init=False) class Profile(HasCredentials): profile_name: str target_name: str - user_config: UserConfig threads: int credentials: Credentials profile_env_vars: Dict[str, Any] @@ -80,7 +65,6 @@ def __init__( self, profile_name: str, target_name: str, - user_config: UserConfig, threads: int, credentials: Credentials, ): @@ -89,7 +73,6 @@ def __init__( """ self.profile_name = profile_name self.target_name = target_name - self.user_config = user_config self.threads = threads self.credentials = credentials self.profile_env_vars = {} # never available on init @@ -106,12 +89,10 @@ def to_profile_info(self, serialize_credentials: bool = False) -> Dict[str, Any] result = { "profile_name": self.profile_name, "target_name": self.target_name, - "user_config": self.user_config, "threads": self.threads, "credentials": self.credentials, } if serialize_credentials: - result["user_config"] = self.user_config.to_dict(omit_none=True) result["credentials"] = self.credentials.to_dict(omit_none=True) return result @@ -124,7 +105,6 @@ def to_target_dict(self) -> Dict[str, Any]: "name": self.target_name, "target_name": self.target_name, "profile_name": self.profile_name, - "config": self.user_config.to_dict(omit_none=True), } ) return target @@ -246,7 +226,6 @@ def from_credentials( threads: int, profile_name: str, target_name: str, - user_config: Optional[Dict[str, Any]] = None, ) -> "Profile": """Create a profile from an existing set of Credentials and the remaining information. @@ -255,20 +234,13 @@ def from_credentials( :param threads: The number of threads to use for connections. :param profile_name: The profile name used for this profile. :param target_name: The target name used for this profile. - :param user_config: The user-level config block from the - raw profiles, if specified. :raises DbtProfileError: If the profile is invalid. :returns: The new Profile object. """ - if user_config is None: - user_config = {} - UserConfig.validate(user_config) - user_config_obj: UserConfig = UserConfig.from_dict(user_config) profile = cls( profile_name=profile_name, target_name=target_name, - user_config=user_config_obj, threads=threads, credentials=credentials, ) @@ -316,7 +288,6 @@ def from_raw_profile_info( raw_profile: Dict[str, Any], profile_name: str, renderer: ProfileRenderer, - user_config: Optional[Dict[str, Any]] = None, target_override: Optional[str] = None, threads_override: Optional[int] = None, ) -> "Profile": @@ -328,8 +299,6 @@ def from_raw_profile_info( disk as yaml and its values rendered with jinja. :param profile_name: The profile name used. :param renderer: The config renderer. - :param user_config: The global config for the user, if it - was present. :param target_override: The target to use, if provided on the command line. :param threads_override: The thread count to use, if @@ -338,9 +307,6 @@ def from_raw_profile_info( target could not be found :returns: The new Profile object. """ - # user_config is not rendered. - if user_config is None: - user_config = raw_profile.get("config") # TODO: should it be, and the values coerced to bool? target_name, profile_data = cls.render_profile( raw_profile, profile_name, target_override, renderer @@ -361,7 +327,6 @@ def from_raw_profile_info( profile_name=profile_name, target_name=target_name, threads=threads, - user_config=user_config, ) @classmethod @@ -396,13 +361,11 @@ def from_raw_profiles( if not raw_profile: msg = f"Profile {profile_name} in profiles.yml is empty" raise DbtProfileError(INVALID_PROFILE_MESSAGE.format(error_string=msg)) - user_config = raw_profiles.get("config") return cls.from_raw_profile_info( raw_profile=raw_profile, profile_name=profile_name, renderer=renderer, - user_config=user_config, target_override=target_override, threads_override=threads_override, )
core/dbt/config/project.py+74 −15 modified@@ -16,7 +16,7 @@ from dbt.flags import get_flags from dbt import deprecations -from dbt.constants import DEPENDENCIES_FILE_NAME, PACKAGES_FILE_NAME +from dbt.constants import DEPENDENCIES_FILE_NAME, PACKAGES_FILE_NAME, DBT_PROJECT_FILE_NAME from dbt.clients.system import path_exists, resolve_path_from_base, load_file_contents from dbt.clients.yaml_helper import load_yaml_text from dbt.contracts.connection import QueryComment @@ -31,12 +31,13 @@ from dbt.helper_types import NoValue from dbt.semver import VersionSpecifier, versions_compatible from dbt.version import get_installed_version -from dbt.utils import MultiDict, md5 +from dbt.utils import MultiDict, md5, coerce_dict_str from dbt.node_types import NodeType from dbt.config.selectors import SelectorDict from dbt.contracts.project import ( Project as ProjectContract, SemverString, + ProjectFlags, ) from dbt.contracts.project import PackageConfig, ProjectPackageMetadata from dbt.dataclass_schema import ValidationError @@ -77,8 +78,8 @@ """ MISSING_DBT_PROJECT_ERROR = """\ -No dbt_project.yml found at expected path {path} -Verify that each entry within packages.yml (and their transitive dependencies) contains a file named dbt_project.yml +No {DBT_PROJECT_FILE_NAME} found at expected path {path} +Verify that each entry within packages.yml (and their transitive dependencies) contains a file named {DBT_PROJECT_FILE_NAME} """ @@ -183,16 +184,20 @@ def value_or(value: Optional[T], default: T) -> T: def load_raw_project(project_root: str) -> Dict[str, Any]: project_root = os.path.normpath(project_root) - project_yaml_filepath = os.path.join(project_root, "dbt_project.yml") + project_yaml_filepath = os.path.join(project_root, DBT_PROJECT_FILE_NAME) # get the project.yml contents if not path_exists(project_yaml_filepath): - raise DbtProjectError(MISSING_DBT_PROJECT_ERROR.format(path=project_yaml_filepath)) + raise DbtProjectError( + MISSING_DBT_PROJECT_ERROR.format( + path=project_yaml_filepath, DBT_PROJECT_FILE_NAME=DBT_PROJECT_FILE_NAME + ) + ) project_dict = _load_yaml(project_yaml_filepath) if not isinstance(project_dict, dict): - raise DbtProjectError("dbt_project.yml does not parse to a dictionary") + raise DbtProjectError(f"{DBT_PROJECT_FILE_NAME} does not parse to a dictionary") return project_dict @@ -307,21 +312,21 @@ def get_rendered( selectors_dict=rendered_selectors, ) - # Called by Project.from_project_root (not PartialProject.from_project_root!) + # Called by Project.from_project_root which first calls PartialProject.from_project_root def render(self, renderer: DbtProjectYamlRenderer) -> "Project": try: rendered = self.get_rendered(renderer) return self.create_project(rendered) except DbtProjectError as exc: if exc.path is None: - exc.path = os.path.join(self.project_root, "dbt_project.yml") + exc.path = os.path.join(self.project_root, DBT_PROJECT_FILE_NAME) raise def render_package_metadata(self, renderer: PackageRenderer) -> ProjectPackageMetadata: packages_data = renderer.render_data(self.packages_dict) packages_config = package_config_from_data(packages_data) if not self.project_name: - raise DbtProjectError("Package dbt_project.yml must have a name!") + raise DbtProjectError(f"Package defined in {DBT_PROJECT_FILE_NAME} must have a name!") return ProjectPackageMetadata(self.project_name, packages_config.packages) def check_config_path( @@ -332,7 +337,7 @@ def check_config_path( msg = ( "{deprecated_path} and {expected_path} cannot both be defined. The " "`{deprecated_path}` config has been deprecated in favor of `{expected_path}`. " - "Please update your `dbt_project.yml` configuration to reflect this " + f"Please update your `{DBT_PROJECT_FILE_NAME}` configuration to reflect this " "change." ) raise DbtProjectError( @@ -404,11 +409,11 @@ def create_project(self, rendered: RenderComponents) -> "Project": docs_paths: List[str] = value_or(cfg.docs_paths, all_source_paths) asset_paths: List[str] = value_or(cfg.asset_paths, []) - flags = get_flags() + global_flags = get_flags() - flag_target_path = str(flags.TARGET_PATH) if flags.TARGET_PATH else None + flag_target_path = str(global_flags.TARGET_PATH) if global_flags.TARGET_PATH else None target_path: str = flag_or(flag_target_path, cfg.target_path, "target") - log_path: str = str(flags.LOG_PATH) + log_path: str = str(global_flags.LOG_PATH) clean_targets: List[str] = value_or(cfg.clean_targets, [target_path]) packages_install_path: str = value_or(cfg.packages_install_path, "dbt_packages") @@ -545,6 +550,12 @@ def from_project_root( packages_specified_path, ) = package_and_project_data_from_root(project_root) selectors_dict = selector_data_from_root(project_root) + + if "flags" in project_dict: + # We don't want to include "flags" in the Project, + # it goes in ProjectFlags + project_dict.pop("flags") + return cls.from_dicts( project_root=project_root, project_dict=project_dict, @@ -681,7 +692,6 @@ def to_project_config(self, with_packages=False): "exposures": self.exposures, "vars": self.vars.to_dict(), "require-dbt-version": [v.to_version_string() for v in self.dbt_version], - "config-version": self.config_version, "restrict-access": self.restrict_access, "dbt-cloud": self.dbt_cloud, } @@ -745,3 +755,52 @@ def get_macro_search_order(self, macro_namespace: str): def project_target_path(self): # If target_path is absolute, project_root will not be included return os.path.join(self.project_root, self.target_path) + + +def read_project_flags(project_dir: str, profiles_dir: str) -> ProjectFlags: + try: + project_flags: Dict[str, Any] = {} + # Read project_flags from dbt_project.yml first + # Flags are instantiated before the project, so we don't + # want to throw an error for non-existence of dbt_project.yml here + # because it breaks things. + project_root = os.path.normpath(project_dir) + project_yaml_filepath = os.path.join(project_root, DBT_PROJECT_FILE_NAME) + if path_exists(project_yaml_filepath): + try: + project_dict = load_raw_project(project_root) + if "flags" in project_dict: + project_flags = project_dict.pop("flags") + except Exception: + # This is probably a yaml load error.The error will be reported + # later, when the project loads. + pass + + from dbt.config.profile import read_profile + + profile = read_profile(profiles_dir) + profile_project_flags: Optional[Dict[str, Any]] = {} + if profile: + profile_project_flags = coerce_dict_str(profile.get("config", {})) + + if project_flags and profile_project_flags: + raise DbtProjectError( + f"Do not specify both 'config' in profiles.yml and 'flags' in {DBT_PROJECT_FILE_NAME}. " + "Using 'config' in profiles.yml is deprecated." + ) + + if profile_project_flags: + # This can't use WARN_ERROR or WARN_ERROR_OPTIONS because they're in + # the config that we're loading. Uses special "warn" method. + deprecations.warn("project-flags-moved") + project_flags = profile_project_flags + + if project_flags is not None: + ProjectFlags.validate(project_flags) + return ProjectFlags.from_dict(project_flags) + except (DbtProjectError) as exc: + # We don't want to eat the DbtProjectError for UserConfig to ProjectFlags + raise exc + except (DbtRuntimeError, ValidationError): + pass + return ProjectFlags()
core/dbt/config/runtime.py+1 −3 modified@@ -20,7 +20,7 @@ from dbt.config.project import load_raw_project from dbt.contracts.connection import AdapterRequiredConfig, Credentials, HasCredentials from dbt.contracts.graph.manifest import ManifestMetadata -from dbt.contracts.project import Configuration, UserConfig +from dbt.contracts.project import Configuration from dbt.contracts.relation import ComponentName from dbt.dataclass_schema import ValidationError from dbt.events.functions import warn_or_error @@ -176,7 +176,6 @@ def from_parts( profile_env_vars=profile.profile_env_vars, profile_name=profile.profile_name, target_name=profile.target_name, - user_config=profile.user_config, threads=profile.threads, credentials=profile.credentials, args=args, @@ -428,7 +427,6 @@ def _connection_keys(self): class UnsetProfile(Profile): def __init__(self): self.credentials = UnsetCredentials() - self.user_config = UserConfig() # This will be read in _get_rendered_profile self.profile_name = "" self.target_name = "" self.threads = -1
core/dbt/constants.py+1 −0 modified@@ -9,6 +9,7 @@ "https://docs.getdbt.com/docs/package-management#section-specifying-package-versions" ) +DBT_PROJECT_FILE_NAME = "dbt_project.yml" PACKAGES_FILE_NAME = "packages.yml" DEPENDENCIES_FILE_NAME = "dependencies.yml" MANIFEST_FILE_NAME = "manifest.json"
core/dbt/contracts/connection.py+0 −8 modified@@ -181,17 +181,9 @@ def __post_serialize__(self, dct): return dct -class UserConfigContract(Protocol): - send_anonymous_usage_stats: bool - use_colors: Optional[bool] = None - partial_parse: Optional[bool] = None - printer_width: Optional[int] = None - - class HasCredentials(Protocol): credentials: Credentials profile_name: str - user_config: UserConfigContract target_name: str threads: int
core/dbt/contracts/graph/manifest.py+48 −5 modified@@ -23,7 +23,6 @@ ) from typing_extensions import Protocol from uuid import UUID - from dbt.contracts.graph.nodes import ( BaseNode, Documentation, @@ -59,7 +58,7 @@ from dbt.events.contextvars import get_node_info from dbt.node_types import NodeType, AccessType from dbt.flags import get_flags, MP_CONTEXT -from dbt import tracking +from dbt import tracking, deprecations import dbt.utils @@ -562,11 +561,29 @@ def __lt__(self, other: object) -> bool: class CandidateList(List[M]): - def last(self) -> Optional[Macro]: + def last_candidate( + self, valid_localities: Optional[List[Locality]] = None + ) -> Optional[MacroCandidate]: + """ + Obtain the last (highest precedence) MacroCandidate from the CandidateList of any locality in valid_localities. + If valid_localities is not specified, return the last MacroCandidate of any locality. + """ if not self: return None self.sort() - return self[-1].macro + + if valid_localities is None: + return self[-1] + + for candidate in reversed(self): + if candidate.locality in valid_localities: + return candidate + + return None + + def last(self) -> Optional[Macro]: + last_candidate = self.last_candidate() + return last_candidate.macro if last_candidate is not None else None def _get_locality(macro: Macro, root_project_name: str, internal_packages: Set[str]) -> Locality: @@ -850,7 +867,33 @@ def find_materialization_macro_by_name( for specificity, atype in enumerate(self._get_parent_adapter_types(adapter_type)) ) ) - return candidates.last() + core_candidates = [ + candidate for candidate in candidates if candidate.locality == Locality.Core + ] + + materialization_candidate = candidates.last_candidate() + # If an imported materialization macro was found that also had a core candidate, fire a deprecation + if ( + materialization_candidate is not None + and materialization_candidate.locality == Locality.Imported + and core_candidates + ): + # preserve legacy behaviour - allow materialization override + if ( + get_flags().require_explicit_package_overrides_for_builtin_materializations + is False + ): + deprecations.warn( + "package-materialization-override", + package_name=materialization_candidate.macro.package_name, + materialization_name=materialization_name, + ) + else: + materialization_candidate = candidates.last_candidate( + valid_localities=[Locality.Core, Locality.Root] + ) + + return materialization_candidate.macro if materialization_candidate else None def get_resource_fqns(self) -> Mapping[str, PathSet]: resource_fqns: Dict[str, Set[Tuple[str, ...]]] = {}
core/dbt/contracts/project.py+9 −3 modified@@ -1,5 +1,5 @@ from dbt.contracts.util import Replaceable, Mergeable, list_str, Identifier -from dbt.contracts.connection import QueryComment, UserConfigContract +from dbt.contracts.connection import QueryComment from dbt.helper_types import NoValue from dbt.dataclass_schema import ( dbtClassMixin, @@ -248,7 +248,7 @@ def validate(cls, data): @dataclass -class UserConfig(ExtensibleDbtClassMixin, Replaceable, UserConfigContract): +class ProjectFlags(ExtensibleDbtClassMixin, Replaceable): cache_selected_only: Optional[bool] = None debug: Optional[bool] = None fail_fast: Optional[bool] = None @@ -260,6 +260,7 @@ class UserConfig(ExtensibleDbtClassMixin, Replaceable, UserConfigContract): partial_parse: Optional[bool] = None populate_cache: Optional[bool] = None printer_width: Optional[int] = None + require_explicit_package_overrides_for_builtin_materializations: bool = False send_anonymous_usage_stats: bool = DEFAULT_SEND_ANONYMOUS_USAGE_STATS static_parser: Optional[bool] = None use_colors: Optional[bool] = None @@ -270,12 +271,17 @@ class UserConfig(ExtensibleDbtClassMixin, Replaceable, UserConfigContract): warn_error_options: Optional[Dict[str, Union[str, List[str]]]] = None write_json: Optional[bool] = None + @property + def project_only_flags(self) -> Dict[str, Any]: + return { + "require_explicit_package_overrides_for_builtin_materializations": self.require_explicit_package_overrides_for_builtin_materializations, + } + @dataclass class ProfileConfig(HyphenatedDbtClassMixin, Replaceable): profile_name: str = field(metadata={"preserve_underscore": True}) target_name: str = field(metadata={"preserve_underscore": True}) - user_config: UserConfig = field(metadata={"preserve_underscore": True}) threads: int # TODO: make this a dynamic union of some kind? credentials: Optional[Dict[str, Any]]
core/dbt/deprecations.py+21 −0 modified@@ -96,6 +96,25 @@ class CollectFreshnessReturnSignature(DBTDeprecation): _event = "CollectFreshnessReturnSignature" +class ProjectFlagsMovedDeprecation(DBTDeprecation): + _name = "project-flags-moved" + _event = "ProjectFlagsMovedDeprecation" + + def show(self, *args, **kwargs) -> None: + if self.name not in active_deprecations: + event = self.event(**kwargs) + # We can't do warn_or_error because the ProjectFlags + # is where that is set up and we're just reading it. + dbt.events.functions.fire_event(event) + self.track_deprecation_warn() + active_deprecations.add(self.name) + + +class PackageMaterializationOverrideDeprecation(DBTDeprecation): + _name = "package-materialization-override" + _event = "PackageMaterializationOverrideDeprecation" + + def renamed_env_var(old_name: str, new_name: str): class EnvironmentVariableRenamed(DBTDeprecation): _name = f"environment-variable-renamed:{old_name}" @@ -134,6 +153,8 @@ def warn(name, *args, **kwargs): ConfigLogPathDeprecation(), ConfigTargetPathDeprecation(), CollectFreshnessReturnSignature(), + ProjectFlagsMovedDeprecation(), + PackageMaterializationOverrideDeprecation(), ] deprecations: Dict[str, DBTDeprecation] = {d.name: d for d in deprecations_list}
core/dbt/events/types_pb2.py+886 −877 modifiedcore/dbt/events/types.proto+20 −0 modified@@ -394,6 +394,26 @@ message CollectFreshnessReturnSignatureMsg { CollectFreshnessReturnSignature data = 2; } +// D013 +message ProjectFlagsMovedDeprecation { +} + +message ProjectFlagsMovedDeprecationMsg { + EventInfo info = 1; + ProjectFlagsMovedDeprecation data = 2; +} + +// D016 +message PackageMaterializationOverrideDeprecation { + string package_name = 1; + string materialization_name = 2; +} + +message PackageMaterializationOverrideDeprecationMsg { + EventInfo info = 1; + PackageMaterializationOverrideDeprecation data = 2; +} + // E - DB Adapter // E001
core/dbt/events/types.py+23 −0 modified@@ -797,6 +797,29 @@ def message(self) -> str: return line_wrap_message(warning_tag(msg)) +class ProjectFlagsMovedDeprecation(WarnLevel): + def code(self) -> str: + return "D013" + + def message(self) -> str: + description = ( + "User config should be moved from the 'config' key in profiles.yml to the 'flags' " + "key in dbt_project.yml." + ) + # Can't use line_wrap_message here because flags.printer_width isn't available yet + return warning_tag(f"Deprecated functionality\n\n{description}") + + +class PackageMaterializationOverrideDeprecation(WarnLevel): + def code(self) -> str: + return "D016" + + def message(self) -> str: + description = f"Installed package '{self.package_name}' is overriding the built-in materialization '{self.materialization_name}'. Overrides of built-in materializations from installed packages will be deprecated in future versions of dbt. Please refer to https://docs.getdbt.com/reference/global-configs/legacy-behaviors#require_explicit_package_overrides_for_builtin_materializations for detailed documentation and suggested workarounds." + + return line_wrap_message(warning_tag(description)) + + # ======================================================= # I - Project parsing # =======================================================
core/dbt/flags.py+9 −8 modified@@ -39,23 +39,24 @@ def get_flags(): return GLOBAL_FLAGS -def set_from_args(args: Namespace, user_config): +def set_from_args(args: Namespace, project_flags): global GLOBAL_FLAGS from dbt.cli.main import cli from dbt.cli.flags import Flags, convert_config - # we set attributes of args after initialize the flags, but user_config + # we set attributes of args after initialize the flags, but project_flags # is being read in the Flags constructor, so we need to read it here and pass in - # to make sure we use the correct user_config - if (hasattr(args, "PROFILES_DIR") or hasattr(args, "profiles_dir")) and not user_config: - from dbt.config.profile import read_user_config + # to make sure we use the correct project_flags + profiles_dir = getattr(args, "PROFILES_DIR", None) or getattr(args, "profiles_dir", None) + project_dir = getattr(args, "PROJECT_DIR", None) or getattr(args, "project_dir", None) + if profiles_dir and project_dir: + from dbt.config.project import read_project_flags - profiles_dir = getattr(args, "PROFILES_DIR", None) or getattr(args, "profiles_dir") - user_config = read_user_config(profiles_dir) + project_flags = read_project_flags(project_dir, profiles_dir) # make a dummy context to get the flags, totally arbitrary ctx = cli.make_context("run", ["run"]) - flags = Flags(ctx, user_config) + flags = Flags(ctx, project_flags) for arg_name, args_param_value in vars(args).items(): args_param_value = convert_config(arg_name, args_param_value) object.__setattr__(flags, arg_name.upper(), args_param_value)
core/dbt/include/starter_project/dbt_project.yml+0 −1 modified@@ -4,7 +4,6 @@ # name or the intended use of these models name: '{project_name}' version: '1.0.0' -config-version: 2 # This setting configures which "profile" dbt uses for this project. profile: '{profile_name}'
core/dbt/tests/fixtures/project.py+1 −1 modified@@ -142,7 +142,6 @@ def profiles_config_update(): @pytest.fixture(scope="class") def dbt_profile_data(unique_schema, dbt_profile_target, profiles_config_update): profile = { - "config": {"send_anonymous_usage_stats": False}, "test": { "outputs": { "default": {}, @@ -181,6 +180,7 @@ def dbt_project_yml(project_root, project_config_update): project_config = { "name": "test", "profile": "test", + "flags": {"send_anonymous_usage_stats": False}, } if project_config_update: if isinstance(project_config_update, dict):
core/dbt/tracking.py+0 −1 modified@@ -471,7 +471,6 @@ def process(self, record): def initialize_from_flags(send_anonymous_usage_stats, profiles_dir): - # Setting these used to be in UserConfig, but had to be moved here global active_user if send_anonymous_usage_stats: active_user = User(profiles_dir)
core/dbt/utils.py+1 −1 modified@@ -623,7 +623,7 @@ def _connection_exception_retry(fn, max_attempts: int, attempt: int = 0): def args_to_dict(args): var_args = vars(args).copy() # update the args with the flags, which could also come from environment - # variables or user_config + # variables or project_flags flag_dict = flags.get_flag_dict() var_args.update(flag_dict) dict_args = {}
tests/functional/basic/test_mixed_case_db.py+0 −1 modified@@ -16,7 +16,6 @@ def models(): def dbt_profile_data(unique_schema): return { - "config": {"send_anonymous_usage_stats": False}, "test": { "outputs": { "default": {
tests/functional/basic/test_project.py+6 −1 modified@@ -77,11 +77,16 @@ def test_dbt_cloud(self, project): conf = yaml.safe_load( Path(os.path.join(project.project_root, "dbt_project.yml")).read_text() ) - assert conf == {"name": "test", "profile": "test"} + assert conf == { + "name": "test", + "profile": "test", + "flags": {"send_anonymous_usage_stats": False}, + } config = { "name": "test", "profile": "test", + "flags": {"send_anonymous_usage_stats": False}, "dbt-cloud": { "account_id": "123", "application": "test",
tests/functional/configs/test_disabled_configs.py+0 −1 modified@@ -9,7 +9,6 @@ class TestDisabledConfigs(BaseConfigProject): @pytest.fixture(scope="class") def dbt_profile_data(self, unique_schema): return { - "config": {"send_anonymous_usage_stats": False}, "test": { "outputs": { "default": {
tests/functional/dependencies/test_local_dependency.py+4 −9 modified@@ -253,6 +253,10 @@ class TestSimpleDependencyNoVersionCheckConfig(BaseDependencyTest): @pytest.fixture(scope="class") def project_config_update(self): return { + "flags": { + "send_anonymous_usage_stats": False, + "version_check": False, + }, "models": { "schema": "dbt_test", }, @@ -261,15 +265,6 @@ def project_config_update(self): }, } - @pytest.fixture(scope="class") - def profiles_config_update(self): - return { - "config": { - "send_anonymous_usage_stats": False, - "version_check": False, - } - } - @pytest.fixture(scope="class") def macros(self): return {"macro.sql": macros__macro_override_schema_sql}
tests/functional/deprecations/test_deprecations.py+30 −1 modified@@ -2,7 +2,8 @@ from dbt import deprecations import dbt.exceptions -from dbt.tests.util import run_dbt +from dbt.tests.util import run_dbt, write_file +import yaml models__already_exists_sql = """ @@ -157,3 +158,31 @@ def test_exposure_name_fail(self, project): exc_str = " ".join(str(exc.value).split()) # flatten all whitespace expected_msg = "Starting in v1.3, the 'name' of an exposure should contain only letters, numbers, and underscores." assert expected_msg in exc_str + + +class TestPrjectFlagsMovedDeprecation: + @pytest.fixture(scope="class") + def profiles_config_update(self): + return { + "config": {"send_anonymous_usage_stats": False}, + } + + @pytest.fixture(scope="class") + def dbt_project_yml(self, project_root, project_config_update): + project_config = { + "name": "test", + "profile": "test", + } + write_file(yaml.safe_dump(project_config), project_root, "dbt_project.yml") + return project_config + + @pytest.fixture(scope="class") + def models(self): + return {"my_model.sql": "select 1 as fun"} + + def test_profile_config_deprecation(self, project): + deprecations.reset_deprecations() + assert deprecations.active_deprecations == set() + run_dbt(["parse"]) + expected = {"project-flags-moved"} + assert expected == deprecations.active_deprecations
tests/functional/fail_fast/test_fail_fast_run.py+3 −3 modified@@ -44,15 +44,15 @@ def test_fail_fast_run( class TestFailFastFromConfig(FailFastBase): @pytest.fixture(scope="class") - def profiles_config_update(self): + def project_config_update(self): return { - "config": { + "flags": { "send_anonymous_usage_stats": False, "fail_fast": True, } } - def test_fail_fast_run_user_config( + def test_fail_fast_run_project_flags( self, project, models, # noqa: F811
tests/functional/init/test_init.py+3 −12 modified@@ -67,9 +67,7 @@ def test_init_task_in_project_with_existing_profiles_yml( with open(os.path.join(project.profiles_dir, "profiles.yml"), "r") as f: assert ( f.read() - == """config: - send_anonymous_usage_stats: false -test: + == """test: outputs: dev: dbname: test_db @@ -369,9 +367,7 @@ def test_init_task_in_project_with_invalid_profile_template( with open(os.path.join(project.profiles_dir, "profiles.yml"), "r") as f: assert ( f.read() - == """config: - send_anonymous_usage_stats: false -test: + == """test: outputs: dev: dbname: test_db @@ -408,7 +404,6 @@ class TestInitOutsideOfProject(TestInitOutsideOfProjectBase): @pytest.fixture(scope="class") def dbt_profile_data(self, unique_schema): return { - "config": {"send_anonymous_usage_stats": False}, "test": { "outputs": { "default2": { @@ -491,9 +486,7 @@ def test_init_task_outside_of_project( with open(os.path.join(project.profiles_dir, "profiles.yml"), "r") as f: assert ( f.read() - == f"""config: - send_anonymous_usage_stats: false -{project_name}: + == f"""{project_name}: outputs: dev: dbname: test_db @@ -538,7 +531,6 @@ def test_init_task_outside_of_project( # name or the intended use of these models name: '{project_name}' version: '1.0.0' -config-version: 2 # This setting configures which "profile" dbt uses for this project. profile: '{project_name}' @@ -657,7 +649,6 @@ def test_init_provided_project_name_and_skip_profile_setup( # name or the intended use of these models name: '{project_name}' version: '1.0.0' -config-version: 2 # This setting configures which "profile" dbt uses for this project. profile: '{project_name}'
tests/functional/materializations/conftest.py+24 −0 modified@@ -325,6 +325,21 @@ {%- endmaterialization -%} """ +custom_materialization_dep__dbt_project_yml = """ +name: custom_materialization_default +macro-paths: ['macros'] +""" + +custom_materialization_sql = """ +{% materialization custom_materialization, default %} + {%- set target_relation = this.incorporate(type='table') %} + {% call statement('main') -%} + select 1 as column1 + {%- endcall %} + {{ return({'relations': [target_relation]}) }} +{% endmaterialization %} +""" + @pytest.fixture(scope="class") def override_view_adapter_pass_dep(project_root): @@ -368,3 +383,12 @@ def override_view_return_no_relation(project_root): }, } write_project_files(project_root, "override-view-return-no-relation", files) + + +@pytest.fixture(scope="class") +def custom_materialization_dep(project_root): + files = { + "dbt_project.yml": custom_materialization_dep__dbt_project_yml, + "macros": {"custom_materialization.sql": custom_materialization_sql}, + } + write_project_files(project_root, "custom-materialization-dep", files)
tests/functional/materializations/test_custom_materialization.py+165 −3 modified@@ -1,7 +1,7 @@ import pytest from dbt.tests.util import run_dbt - +from dbt import deprecations models__model_sql = """ {{ config(materialized='view') }} @@ -10,34 +10,196 @@ """ +models_custom_materialization__model_sql = """ +{{ config(materialized='custom_materialization') }} +select 1 as id + +""" + + @pytest.fixture(scope="class") def models(): return {"model.sql": models__model_sql} +@pytest.fixture(scope="class") +def set_up_deprecations(): + deprecations.reset_deprecations() + assert deprecations.active_deprecations == set() + + class TestOverrideAdapterDependency: # make sure that if there's a dependency with an adapter-specific # materialization, we honor that materialization @pytest.fixture(scope="class") def packages(self): return {"packages": [{"local": "override-view-adapter-dep"}]} - def test_adapter_dependency(self, project, override_view_adapter_dep): + def test_adapter_dependency(self, project, override_view_adapter_dep, set_up_deprecations): + run_dbt(["deps"]) + # this should error because the override is buggy + run_dbt(["run"], expect_pass=False) + + # overriding a built-in materialization scoped to adapter from package is deprecated + assert deprecations.active_deprecations == {"package-materialization-override"} + + +class TestOverrideAdapterDependencyDeprecated: + # make sure that if there's a dependency with an adapter-specific + # materialization, we honor that materialization + @pytest.fixture(scope="class") + def packages(self): + return {"packages": [{"local": "override-view-adapter-dep"}]} + + @pytest.fixture(scope="class") + def project_config_update(self): + return { + "flags": { + "require_explicit_package_overrides_for_builtin_materializations": True, + }, + } + + def test_adapter_dependency_deprecate_overrides( + self, project, override_view_adapter_dep, set_up_deprecations + ): + run_dbt(["deps"]) + # this should pass because the override is buggy and unused + run_dbt(["run"]) + + # no deprecation warning -- flag used correctly + assert deprecations.active_deprecations == set() + + +class TestOverrideAdapterDependencyLegacy: + # make sure that if there's a dependency with an adapter-specific + # materialization, we honor that materialization + @pytest.fixture(scope="class") + def packages(self): + return {"packages": [{"local": "override-view-adapter-dep"}]} + + @pytest.fixture(scope="class") + def project_config_update(self): + return { + "flags": { + "require_explicit_package_overrides_for_builtin_materializations": False, + }, + } + + def test_adapter_dependency(self, project, override_view_adapter_dep, set_up_deprecations): run_dbt(["deps"]) # this should error because the override is buggy run_dbt(["run"], expect_pass=False) + # overriding a built-in materialization scoped to adapter from package is deprecated + assert deprecations.active_deprecations == {"package-materialization-override"} + class TestOverrideDefaultDependency: @pytest.fixture(scope="class") def packages(self): return {"packages": [{"local": "override-view-default-dep"}]} - def test_default_dependency(self, project, override_view_default_dep): + def test_default_dependency(self, project, override_view_default_dep, set_up_deprecations): + run_dbt(["deps"]) + # this should error because the override is buggy + run_dbt(["run"], expect_pass=False) + + # overriding a built-in materialization from package is deprecated + assert deprecations.active_deprecations == {"package-materialization-override"} + + +class TestOverrideDefaultDependencyDeprecated: + @pytest.fixture(scope="class") + def packages(self): + return {"packages": [{"local": "override-view-default-dep"}]} + + @pytest.fixture(scope="class") + def project_config_update(self): + return { + "flags": { + "require_explicit_package_overrides_for_builtin_materializations": True, + }, + } + + def test_default_dependency_deprecated( + self, project, override_view_default_dep, set_up_deprecations + ): + run_dbt(["deps"]) + # this should pass because the override is buggy and unused + run_dbt(["run"]) + + # overriding a built-in materialization from package is deprecated + assert deprecations.active_deprecations == set() + + +class TestOverrideDefaultDependencyLegacy: + @pytest.fixture(scope="class") + def packages(self): + return {"packages": [{"local": "override-view-default-dep"}]} + + @pytest.fixture(scope="class") + def project_config_update(self): + return { + "flags": { + "require_explicit_package_overrides_for_builtin_materializations": False, + }, + } + + def test_default_dependency(self, project, override_view_default_dep, set_up_deprecations): + run_dbt(["deps"]) + # this should error because the override is buggy + run_dbt(["run"], expect_pass=False) + + # overriding a built-in materialization from package is deprecated + assert deprecations.active_deprecations == {"package-materialization-override"} + + +root_view_override_macro = """ +{% materialization view, default %} + {{ return(view_default_override.materialization_view_default()) }} +{% endmaterialization %} +""" + + +class TestOverrideDefaultDependencyRootOverride: + @pytest.fixture(scope="class") + def packages(self): + return {"packages": [{"local": "override-view-default-dep"}]} + + @pytest.fixture(scope="class") + def macros(self): + return {"my_view.sql": root_view_override_macro} + + def test_default_dependency_with_root_override( + self, project, override_view_default_dep, set_up_deprecations + ): run_dbt(["deps"]) # this should error because the override is buggy run_dbt(["run"], expect_pass=False) + # using an package-overriden built-in materialization in a root matereialization is _not_ deprecated + assert deprecations.active_deprecations == set() + + +class TestCustomMaterializationDependency: + @pytest.fixture(scope="class") + def models(self): + return {"model.sql": models_custom_materialization__model_sql} + + @pytest.fixture(scope="class") + def packages(self): + return {"packages": [{"local": "custom-materialization-dep"}]} + + def test_custom_materialization_deopendency( + self, project, custom_materialization_dep, set_up_deprecations + ): + run_dbt(["deps"]) + # custom materilization is valid + run_dbt(["run"]) + + # using a custom materialization is from an installed package is _not_ deprecated + assert deprecations.active_deprecations == set() + class TestOverrideAdapterDependencyPassing: @pytest.fixture(scope="class")
tests/functional/metrics/test_metric_deferral.py+0 −1 modified@@ -23,7 +23,6 @@ def setup(self, project): @pytest.fixture(scope="class") def dbt_profile_data(self, unique_schema): return { - "config": {"send_anonymous_usage_stats": False}, "test": { "outputs": { "default": {
tests/functional/run_operations/test_run_operations.py+0 −1 modified@@ -28,7 +28,6 @@ def macros(self): @pytest.fixture(scope="class") def dbt_profile_data(self, unique_schema): return { - "config": {"send_anonymous_usage_stats": False}, "test": { "outputs": { "default": {
tests/unit/test_cli_flags.py+76 −31 modified@@ -9,7 +9,7 @@ from dbt.cli.flags import Flags from dbt.cli.main import cli from dbt.cli.types import Command -from dbt.contracts.project import UserConfig +from dbt.contracts.project import ProjectFlags from dbt.exceptions import DbtInternalError from dbt.helper_types import WarnErrorOptions from dbt.tests.util import rm_file, write_file @@ -27,8 +27,8 @@ def run_context(self) -> click.Context: return self.make_dbt_context("run", ["run"]) @pytest.fixture - def user_config(self) -> UserConfig: - return UserConfig() + def project_flags(self) -> ProjectFlags: + return ProjectFlags() def test_which(self, run_context): flags = Flags(run_context) @@ -110,35 +110,35 @@ def test_anonymous_usage_state( flags = Flags(run_context) assert flags.SEND_ANONYMOUS_USAGE_STATS == expected_anonymous_usage_stats - def test_empty_user_config_uses_default(self, run_context, user_config): - flags = Flags(run_context, user_config) + def test_empty_project_flags_uses_default(self, run_context, project_flags): + flags = Flags(run_context, project_flags) assert flags.USE_COLORS == run_context.params["use_colors"] - def test_none_user_config_uses_default(self, run_context): + def test_none_project_flags_uses_default(self, run_context): flags = Flags(run_context, None) assert flags.USE_COLORS == run_context.params["use_colors"] - def test_prefer_user_config_to_default(self, run_context, user_config): - user_config.use_colors = False + def test_prefer_project_flags_to_default(self, run_context, project_flags): + project_flags.use_colors = False # ensure default value is not the same as user config - assert run_context.params["use_colors"] is not user_config.use_colors + assert run_context.params["use_colors"] is not project_flags.use_colors - flags = Flags(run_context, user_config) - assert flags.USE_COLORS == user_config.use_colors + flags = Flags(run_context, project_flags) + assert flags.USE_COLORS == project_flags.use_colors - def test_prefer_param_value_to_user_config(self): - user_config = UserConfig(use_colors=False) + def test_prefer_param_value_to_project_flags(self): + project_flags = ProjectFlags(use_colors=False) context = self.make_dbt_context("run", ["--use-colors", "True", "run"]) - flags = Flags(context, user_config) + flags = Flags(context, project_flags) assert flags.USE_COLORS - def test_prefer_env_to_user_config(self, monkeypatch, user_config): - user_config.use_colors = False + def test_prefer_env_to_project_flags(self, monkeypatch, project_flags): + project_flags.use_colors = False monkeypatch.setenv("DBT_USE_COLORS", "True") context = self.make_dbt_context("run", ["run"]) - flags = Flags(context, user_config) + flags = Flags(context, project_flags) assert flags.USE_COLORS def test_mutually_exclusive_options_passed_separately(self): @@ -163,14 +163,14 @@ def test_mutually_exclusive_options_from_cli(self): Flags(context) @pytest.mark.parametrize("warn_error", [True, False]) - def test_mutually_exclusive_options_from_user_config(self, warn_error, user_config): - user_config.warn_error = warn_error + def test_mutually_exclusive_options_from_project_flags(self, warn_error, project_flags): + project_flags.warn_error = warn_error context = self.make_dbt_context( "run", ["--warn-error-options", '{"include": "all"}', "run"] ) with pytest.raises(DbtUsageException): - Flags(context, user_config) + Flags(context, project_flags) @pytest.mark.parametrize("warn_error", ["True", "False"]) def test_mutually_exclusive_options_from_envvar(self, warn_error, monkeypatch): @@ -182,14 +182,16 @@ def test_mutually_exclusive_options_from_envvar(self, warn_error, monkeypatch): Flags(context) @pytest.mark.parametrize("warn_error", [True, False]) - def test_mutually_exclusive_options_from_cli_and_user_config(self, warn_error, user_config): - user_config.warn_error = warn_error + def test_mutually_exclusive_options_from_cli_and_project_flags( + self, warn_error, project_flags + ): + project_flags.warn_error = warn_error context = self.make_dbt_context( "run", ["--warn-error-options", '{"include": "all"}', "run"] ) with pytest.raises(DbtUsageException): - Flags(context, user_config) + Flags(context, project_flags) @pytest.mark.parametrize("warn_error", ["True", "False"]) def test_mutually_exclusive_options_from_cli_and_envvar(self, warn_error, monkeypatch): @@ -202,15 +204,15 @@ def test_mutually_exclusive_options_from_cli_and_envvar(self, warn_error, monkey Flags(context) @pytest.mark.parametrize("warn_error", ["True", "False"]) - def test_mutually_exclusive_options_from_user_config_and_envvar( - self, user_config, warn_error, monkeypatch + def test_mutually_exclusive_options_from_project_flags_and_envvar( + self, project_flags, warn_error, monkeypatch ): - user_config.warn_error = warn_error + project_flags.warn_error = warn_error monkeypatch.setenv("DBT_WARN_ERROR_OPTIONS", '{"include": "all"}') context = self.make_dbt_context("run", ["run"]) with pytest.raises(DbtUsageException): - Flags(context, user_config) + Flags(context, project_flags) @pytest.mark.parametrize( "cli_colors,cli_colors_file,flag_colors,flag_colors_file", @@ -319,10 +321,10 @@ def test_log_format_interaction( assert flags.LOG_FORMAT_FILE == flag_log_format_file def test_log_settings_from_config(self): - """Test that values set in UserConfig for log settings will set flags as expected""" + """Test that values set in ProjectFlags for log settings will set flags as expected""" context = self.make_dbt_context("run", ["run"]) - config = UserConfig(log_format="json", log_level="warn", use_colors=False) + config = ProjectFlags(log_format="json", log_level="warn", use_colors=False) flags = Flags(context, config) @@ -334,11 +336,11 @@ def test_log_settings_from_config(self): assert flags.USE_COLORS_FILE is False def test_log_file_settings_from_config(self): - """Test that values set in UserConfig for log *file* settings will set flags as expected, leaving the console + """Test that values set in ProjectFlags for log *file* settings will set flags as expected, leaving the console logging flags with their default values""" context = self.make_dbt_context("run", ["run"]) - config = UserConfig(log_format_file="json", log_level_file="warn", use_colors_file=False) + config = ProjectFlags(log_format_file="json", log_level_file="warn", use_colors_file=False) flags = Flags(context, config) @@ -369,6 +371,14 @@ def test_global_flag_at_child_context(self): assert flags_a.USE_COLORS == flags_b.USE_COLORS + def test_set_project_only_flags(self, project_flags, run_context): + flags = Flags(run_context, project_flags) + + for project_only_flag, project_only_flag_value in project_flags.project_only_flags.items(): + assert getattr(flags, project_only_flag) == project_only_flag_value + # sanity check: ensure project_only_flag is not part of the click context + assert project_only_flag not in run_context.params + def _create_flags_from_dict(self, cmd, d): write_file("", "profiles.yml") result = Flags.from_dict(cmd, d) @@ -409,3 +419,38 @@ def test_from_dict_0_value(self): args_dict = {"log_file_max_bytes": 0} flags = Flags.from_dict(Command.RUN, args_dict) assert flags.LOG_FILE_MAX_BYTES == 0 + + +def test_project_flag_defaults(): + flags = ProjectFlags() + # From # 9183: Let's add a unit test that ensures that: + # every attribute of ProjectFlags that has a corresponding click option + # in params.py should be set to None by default (except for anon user + # tracking). Going forward, flags can have non-None defaults if they + # do not have a corresponding CLI option/env var. These will be used + # to control backwards incompatible interface or behaviour changes. + + # List of all flags except send_anonymous_usage_stats + project_flags = [ + "cache_selected_only", + "debug", + "fail_fast", + "indirect_selection", + "log_format", + "log_format_file", + "log_level", + "log_level_file", + "partial_parse", + "populate_cache", + "printer_width", + "static_parser", + "use_colors", + "use_colors_file", + "use_experimental_parser", + "version_check", + "warn_error", + "warn_error_options", + "write_json", + ] + for flag in project_flags: + assert getattr(flags, flag) is None
tests/unit/test_config.py+98 −131 modified@@ -119,12 +119,17 @@ class BaseConfigTest(unittest.TestCase): """ def setUp(self): + # Write project + self.project_dir = normalize(tempfile.mkdtemp()) self.default_project_data = { "version": "0.0.1", "name": "my_test_project", "profile": "default", - "config-version": 2, } + self.write_project(self.default_project_data) + + # Write profile + self.profiles_dir = normalize(tempfile.mkdtemp()) self.default_profile_data = { "default": { "outputs": { @@ -176,6 +181,8 @@ def setUp(self): }, "empty_profile_data": {}, } + self.write_profile(self.default_profile_data) + self.args = Namespace( profiles_dir=self.profiles_dir, cli_vars={}, @@ -203,13 +210,6 @@ def assertRaisesOrReturns(self, exc): else: return self.assertRaises(exc) - -class BaseFileTest(BaseConfigTest): - def setUp(self): - self.project_dir = normalize(tempfile.mkdtemp()) - self.profiles_dir = normalize(tempfile.mkdtemp()) - super().setUp() - def tearDown(self): try: shutil.rmtree(self.project_dir) @@ -248,11 +248,6 @@ def write_empty_profile(self): class TestProfile(BaseConfigTest): - def setUp(self): - self.profiles_dir = "/invalid-path" - self.project_dir = "/invalid-project-path" - super().setUp() - def from_raw_profiles(self): renderer = empty_profile_renderer() return dbt.config.Profile.from_raw_profiles(self.default_profile_data, "default", renderer) @@ -262,8 +257,6 @@ def test_from_raw_profiles(self): self.assertEqual(profile.profile_name, "default") self.assertEqual(profile.target_name, "postgres") self.assertEqual(profile.threads, 7) - self.assertTrue(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) self.assertTrue(isinstance(profile.credentials, PostgresCredentials)) self.assertEqual(profile.credentials.type, "postgres") self.assertEqual(profile.credentials.host, "postgres-db-hostname") @@ -273,29 +266,6 @@ def test_from_raw_profiles(self): self.assertEqual(profile.credentials.schema, "postgres-schema") self.assertEqual(profile.credentials.database, "postgres-db-name") - def test_config_override(self): - self.default_profile_data["config"] = { - "send_anonymous_usage_stats": False, - "use_colors": False, - } - profile = self.from_raw_profiles() - self.assertEqual(profile.profile_name, "default") - self.assertEqual(profile.target_name, "postgres") - self.assertFalse(profile.user_config.send_anonymous_usage_stats) - self.assertFalse(profile.user_config.use_colors) - - def test_partial_config_override(self): - self.default_profile_data["config"] = { - "send_anonymous_usage_stats": False, - "printer_width": 60, - } - profile = self.from_raw_profiles() - self.assertEqual(profile.profile_name, "default") - self.assertEqual(profile.target_name, "postgres") - self.assertFalse(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) - self.assertEqual(profile.user_config.printer_width, 60) - def test_missing_type(self): del self.default_profile_data["default"]["outputs"]["postgres"]["type"] with self.assertRaises(dbt.exceptions.DbtProfileError) as exc: @@ -337,7 +307,7 @@ def test_extra_path(self): } ) with self.assertRaises(dbt.exceptions.DbtProjectError) as exc: - project_from_config_norender(self.default_project_data) + project_from_config_norender(self.default_project_data, project_root=self.project_dir) self.assertIn("source-paths and model-paths", str(exc.exception)) self.assertIn("cannot both be defined.", str(exc.exception)) @@ -403,11 +373,7 @@ def test_invalid_env_vars(self): self.assertIn("Could not convert value 'hello' into type 'number'", str(exc.exception)) -class TestProfileFile(BaseFileTest): - def setUp(self): - super().setUp() - self.write_profile(self.default_profile_data) - +class TestProfileFile(BaseConfigTest): def from_raw_profile_info(self, raw_profile=None, profile_name="default", **kwargs): if raw_profile is None: raw_profile = self.default_profile_data["default"] @@ -438,8 +404,6 @@ def test_profile_simple(self): self.assertEqual(profile.profile_name, "default") self.assertEqual(profile.target_name, "postgres") self.assertEqual(profile.threads, 7) - self.assertTrue(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) self.assertTrue(isinstance(profile.credentials, PostgresCredentials)) self.assertEqual(profile.credentials.type, "postgres") self.assertEqual(profile.credentials.host, "postgres-db-hostname") @@ -464,8 +428,6 @@ def test_profile_override(self): self.assertEqual(profile.profile_name, "other") self.assertEqual(profile.target_name, "other-postgres") self.assertEqual(profile.threads, 3) - self.assertTrue(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) self.assertTrue(isinstance(profile.credentials, PostgresCredentials)) self.assertEqual(profile.credentials.type, "postgres") self.assertEqual(profile.credentials.host, "other-postgres-db-hostname") @@ -485,8 +447,6 @@ def test_env_vars(self): self.assertEqual(profile.profile_name, "default") self.assertEqual(profile.target_name, "with-vars") self.assertEqual(profile.threads, 1) - self.assertTrue(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) self.assertEqual(profile.credentials.type, "postgres") self.assertEqual(profile.credentials.host, "env-postgres-host") self.assertEqual(profile.credentials.port, 6543) @@ -505,8 +465,6 @@ def test_env_vars_env_target(self): self.assertEqual(profile.profile_name, "default") self.assertEqual(profile.target_name, "with-vars") self.assertEqual(profile.threads, 1) - self.assertTrue(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) self.assertEqual(profile.credentials.type, "postgres") self.assertEqual(profile.credentials.host, "env-postgres-host") self.assertEqual(profile.credentials.port, 6543) @@ -537,8 +495,6 @@ def test_cli_and_env_vars(self): self.assertEqual(profile.profile_name, "default") self.assertEqual(profile.target_name, "cli-and-env-vars") self.assertEqual(profile.threads, 1) - self.assertTrue(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) self.assertEqual(profile.credentials.type, "postgres") self.assertEqual(profile.credentials.host, "cli-postgres-host") self.assertEqual(profile.credentials.port, 6543) @@ -567,18 +523,19 @@ def test_profile_with_empty_profile_data(self): def project_from_config_norender( - cfg, packages=None, path="/invalid-root-path", verify_version=False + cfg, packages=None, project_root="/invalid-root-path", verify_version=False ): if packages is None: packages = {} partial = dbt.config.project.PartialProject.from_dicts( - path, + project_root, project_dict=cfg, packages_dict=packages, selectors_dict={}, verify_version=verify_version, ) - # no rendering + # no rendering ... Why? + partial.project_dict["project-root"] = project_root rendered = dbt.config.project.RenderComponents( project_dict=partial.project_dict, packages_dict=partial.packages_dict, @@ -590,14 +547,14 @@ def project_from_config_norender( def project_from_config_rendered( cfg, packages=None, - path="/invalid-root-path", + project_root="/invalid-root-path", verify_version=False, packages_specified_path=PACKAGES_FILE_NAME, ): if packages is None: packages = {} partial = dbt.config.project.PartialProject.from_dicts( - path, + project_root, project_dict=cfg, packages_dict=packages, selectors_dict={}, @@ -608,18 +565,14 @@ def project_from_config_rendered( class TestProject(BaseConfigTest): - def setUp(self): - self.profiles_dir = "/invalid-profiles-path" - self.project_dir = "/invalid-root-path" - super().setUp() - self.default_project_data["project-root"] = self.project_dir - def test_defaults(self): - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.project_name, "my_test_project") self.assertEqual(project.version, "0.0.1") self.assertEqual(project.profile_name, "default") - self.assertEqual(project.project_root, "/invalid-root-path") + self.assertEqual(project.project_root, self.project_dir) self.assertEqual(project.model_paths, ["models"]) self.assertEqual(project.macro_paths, ["macros"]) self.assertEqual(project.seed_paths, ["seeds"]) @@ -645,30 +598,38 @@ def test_defaults(self): str(project) def test_eq(self): - project = project_from_config_norender(self.default_project_data) - other = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) + other = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project, other) def test_neq(self): - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertNotEqual(project, object()) def test_implicit_overrides(self): self.default_project_data.update( { "model-paths": ["other-models"], - "target-path": "other-target", } ) - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual( set(project.docs_paths), set(["other-models", "seeds", "snapshots", "analyses", "macros"]), ) - self.assertEqual(project.clean_targets, ["other-target"]) def test_hashed_name(self): - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.hashed_name(), "754cd47eac1d6f50a5f7cd399ec43da4") def test_all_overrides(self): @@ -682,7 +643,6 @@ def test_all_overrides(self): "analysis-paths": ["other-analyses"], "docs-paths": ["docs"], "asset-paths": ["other-assets"], - "target-path": "other-target", "clean-targets": ["another-target"], "packages-install-path": "other-dbt_packages", "quoting": {"identifier": False}, @@ -731,19 +691,19 @@ def test_all_overrides(self): {"git": "git@example.com:dbt-labs/dbt-utils.git", "revision": "test-rev"}, ], } - project = project_from_config_norender(self.default_project_data, packages=packages) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir, packages=packages + ) self.assertEqual(project.project_name, "my_test_project") self.assertEqual(project.version, "0.0.1") self.assertEqual(project.profile_name, "default") - self.assertEqual(project.project_root, "/invalid-root-path") self.assertEqual(project.model_paths, ["other-models"]) self.assertEqual(project.macro_paths, ["other-macros"]) self.assertEqual(project.seed_paths, ["other-seeds"]) self.assertEqual(project.test_paths, ["other-tests"]) self.assertEqual(project.analysis_paths, ["other-analyses"]) self.assertEqual(project.docs_paths, ["docs"]) self.assertEqual(project.asset_paths, ["other-assets"]) - self.assertEqual(project.target_path, "other-target") self.assertEqual(project.clean_targets, ["another-target"]) self.assertEqual(project.packages_install_path, "other-dbt_packages") self.assertEqual(project.quoting, {"identifier": False}) @@ -815,11 +775,12 @@ def test_string_run_hooks(self): def test_invalid_project_name(self): self.default_project_data["name"] = "invalid-project-name" with self.assertRaises(dbt.exceptions.DbtProjectError) as exc: - project_from_config_norender(self.default_project_data) + project_from_config_norender(self.default_project_data, project_root=self.project_dir) self.assertIn("invalid-project-name", str(exc.exception)) def test_no_project(self): + os.remove(os.path.join(self.project_dir, "dbt_project.yml")) renderer = empty_project_renderer() with self.assertRaises(dbt.exceptions.DbtProjectError) as exc: dbt.config.Project.from_project_root(self.project_dir, renderer) @@ -829,12 +790,12 @@ def test_no_project(self): def test_invalid_version(self): self.default_project_data["require-dbt-version"] = "hello!" with self.assertRaises(dbt.exceptions.DbtProjectError): - project_from_config_norender(self.default_project_data) + project_from_config_norender(self.default_project_data, project_root=self.project_dir) def test_unsupported_version(self): self.default_project_data["require-dbt-version"] = ">99999.0.0" # allowed, because the RuntimeConfig checks, not the Project itself - project_from_config_norender(self.default_project_data) + project_from_config_norender(self.default_project_data, project_root=self.project_dir) def test_none_values(self): self.default_project_data.update( @@ -884,7 +845,9 @@ def test_query_comment_disabled(self): "query-comment": None, } ) - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.query_comment.comment, "") self.assertEqual(project.query_comment.append, False) @@ -893,12 +856,16 @@ def test_query_comment_disabled(self): "query-comment": "", } ) - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.query_comment.comment, "") self.assertEqual(project.query_comment.append, False) def test_default_query_comment(self): - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.query_comment, QueryComment()) def test_default_query_comment_append(self): @@ -907,7 +874,9 @@ def test_default_query_comment_append(self): "query-comment": {"append": True}, } ) - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.query_comment.comment, DEFAULT_QUERY_COMMENT) self.assertEqual(project.query_comment.append, True) @@ -917,7 +886,9 @@ def test_custom_query_comment_append(self): "query-comment": {"comment": "run by user test", "append": True}, } ) - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.query_comment.comment, "run by user test") self.assertEqual(project.query_comment.append, True) @@ -939,17 +910,13 @@ def test_packages_from_dependencies(self): assert git_package.git == "{{ env_var('some_package') }}" -class TestProjectFile(BaseFileTest): - def setUp(self): - super().setUp() - self.write_project(self.default_project_data) - # and after the fact, add the project root - self.default_project_data["project-root"] = self.project_dir - +class TestProjectFile(BaseConfigTest): def test_from_project_root(self): renderer = empty_project_renderer() project = dbt.config.Project.from_project_root(self.project_dir, renderer) - from_config = project_from_config_norender(self.default_project_data) + from_config = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project, from_config) self.assertEqual(project.version, "0.0.1") self.assertEqual(project.project_name, "my_test_project") @@ -966,12 +933,7 @@ def run(self): pass -class TestConfiguredTask(BaseFileTest): - def setUp(self): - super().setUp() - self.write_project(self.default_project_data) - self.write_profile(self.default_profile_data) - +class TestConfiguredTask(BaseConfigTest): def tearDown(self): super().tearDown() # These tests will change the directory to the project path, @@ -990,15 +952,13 @@ def test_configured_task_dir_change_with_bad_path(self): InheritsFromConfiguredTask.from_args(self.args) -class TestVariableProjectFile(BaseFileTest): +class TestVariableProjectFile(BaseConfigTest): def setUp(self): super().setUp() self.default_project_data["version"] = "{{ var('cli_version') }}" self.default_project_data["name"] = "blah" self.default_project_data["profile"] = "{{ env_var('env_value_profile') }}" self.write_project(self.default_project_data) - # and after the fact, add the project root - self.default_project_data["project-root"] = self.project_dir def test_cli_and_env_vars(self): renderer = dbt.config.renderer.DbtProjectYamlRenderer(None, {"cli_version": "0.1.2"}) @@ -1015,15 +975,11 @@ def test_cli_and_env_vars(self): class TestRuntimeConfig(BaseConfigTest): - def setUp(self): - self.profiles_dir = "/invalid-profiles-path" - self.project_dir = "/invalid-root-path" - super().setUp() - self.default_project_data["project-root"] = self.project_dir - def get_project(self): return project_from_config_norender( - self.default_project_data, verify_version=self.args.version_check + self.default_project_data, + project_root=self.project_dir, + verify_version=self.args.version_check, ) def get_profile(self): @@ -1072,14 +1028,6 @@ def test_str(self): # to make sure nothing terrible happens str(config) - def test_validate_fails(self): - project = self.get_project() - profile = self.get_profile() - # invalid - must be boolean - profile.user_config.use_colors = 100 - with self.assertRaises(dbt.exceptions.DbtProjectError): - dbt.config.RuntimeConfig.from_parts(project, profile, {}) - def test_supported_version(self): self.default_project_data["require-dbt-version"] = ">0.0.0" conf = self.from_parts() @@ -1203,7 +1151,9 @@ def setUp(self): } def get_project(self): - return project_from_config_norender(self.default_project_data, verify_version=True) + return project_from_config_norender( + self.default_project_data, project_root=self.project_dir, verify_version=True + ) def get_profile(self): renderer = empty_profile_renderer() @@ -1235,14 +1185,7 @@ def test__warn_for_unused_resource_config_paths(self): assert expected_msg in msg -class TestRuntimeConfigFiles(BaseFileTest): - def setUp(self): - super().setUp() - self.write_profile(self.default_profile_data) - self.write_project(self.default_project_data) - # and after the fact, add the project root - self.default_project_data["project-root"] = self.project_dir - +class TestRuntimeConfigFiles(BaseConfigTest): def test_from_args(self): with temp_cd(self.project_dir): config = dbt.config.RuntimeConfig.from_args(self.args) @@ -1272,7 +1215,7 @@ def test_from_args(self): self.assertEqual(config.project_name, "my_test_project") -class TestVariableRuntimeConfigFiles(BaseFileTest): +class TestVariableRuntimeConfigFiles(BaseConfigTest): def setUp(self): super().setUp() self.default_project_data.update( @@ -1304,9 +1247,6 @@ def setUp(self): } ) self.write_project(self.default_project_data) - self.write_profile(self.default_profile_data) - # and after the fact, add the project root - self.default_project_data["project-root"] = self.project_dir def test_cli_and_env_vars(self): self.args.target = "cli-and-env-vars" @@ -1380,3 +1320,30 @@ def test_lookups(self): for node, key, expected_value in expected: value = vars_provider.vars_for(node, "postgres").get(key) assert value == expected_value + + +class TestMultipleProjectFlags(BaseConfigTest): + def setUp(self): + super().setUp() + + self.default_project_data.update( + { + "flags": { + "send_anonymous_usage_data": False, + } + } + ) + self.write_project(self.default_project_data) + + self.default_profile_data.update( + { + "config": { + "send_anonymous_usage_data": False, + } + } + ) + self.write_profile(self.default_profile_data) + + def test_setting_multiple_flags(self): + with pytest.raises(dbt.exceptions.DbtProjectError): + set_from_args(self.args, None)
tests/unit/test_events.py+4 −0 modified@@ -138,6 +138,10 @@ def test_event_codes(self): types.ConfigLogPathDeprecation(deprecated_path=""), types.ConfigTargetPathDeprecation(deprecated_path=""), types.CollectFreshnessReturnSignature(), + types.ProjectFlagsMovedDeprecation(), + types.PackageMaterializationOverrideDeprecation( + package_name="my_package", materialization_name="view" + ), # E - DB Adapter ====================== types.AdapterEventDebug(), types.AdapterEventInfo(),
tests/unit/test_flags.py+0 −340 removed@@ -1,340 +0,0 @@ -import os -from unittest import TestCase -from argparse import Namespace -import pytest - -from dbt import flags -from dbt.contracts.project import UserConfig -from dbt.graph.selector_spec import IndirectSelection -from dbt.helper_types import WarnErrorOptions - -# Skip due to interface for flag updated -pytestmark = pytest.mark.skip - - -class TestFlags(TestCase): - def setUp(self): - self.args = Namespace() - self.user_config = UserConfig() - - def test__flags(self): - - # use_experimental_parser - self.user_config.use_experimental_parser = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.USE_EXPERIMENTAL_PARSER, True) - os.environ["DBT_USE_EXPERIMENTAL_PARSER"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.USE_EXPERIMENTAL_PARSER, False) - setattr(self.args, "use_experimental_parser", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.USE_EXPERIMENTAL_PARSER, True) - # cleanup - os.environ.pop("DBT_USE_EXPERIMENTAL_PARSER") - delattr(self.args, "use_experimental_parser") - flags.USE_EXPERIMENTAL_PARSER = False - self.user_config.use_experimental_parser = None - - # static_parser - self.user_config.static_parser = False - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.STATIC_PARSER, False) - os.environ["DBT_STATIC_PARSER"] = "true" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.STATIC_PARSER, True) - setattr(self.args, "static_parser", False) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.STATIC_PARSER, False) - # cleanup - os.environ.pop("DBT_STATIC_PARSER") - delattr(self.args, "static_parser") - flags.STATIC_PARSER = True - self.user_config.static_parser = None - - # warn_error - self.user_config.warn_error = False - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WARN_ERROR, False) - os.environ["DBT_WARN_ERROR"] = "true" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WARN_ERROR, True) - setattr(self.args, "warn_error", False) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WARN_ERROR, False) - # cleanup - os.environ.pop("DBT_WARN_ERROR") - delattr(self.args, "warn_error") - flags.WARN_ERROR = False - self.user_config.warn_error = None - - # warn_error_options - self.user_config.warn_error_options = '{"include": "all"}' - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WARN_ERROR_OPTIONS, WarnErrorOptions(include="all")) - os.environ["DBT_WARN_ERROR_OPTIONS"] = '{"include": []}' - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WARN_ERROR_OPTIONS, WarnErrorOptions(include=[])) - setattr(self.args, "warn_error_options", '{"include": "all"}') - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WARN_ERROR_OPTIONS, WarnErrorOptions(include="all")) - # cleanup - os.environ.pop("DBT_WARN_ERROR_OPTIONS") - delattr(self.args, "warn_error_options") - self.user_config.warn_error_options = None - - # write_json - self.user_config.write_json = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WRITE_JSON, True) - os.environ["DBT_WRITE_JSON"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WRITE_JSON, False) - setattr(self.args, "write_json", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WRITE_JSON, True) - # cleanup - os.environ.pop("DBT_WRITE_JSON") - delattr(self.args, "write_json") - - # partial_parse - self.user_config.partial_parse = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.PARTIAL_PARSE, True) - os.environ["DBT_PARTIAL_PARSE"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.PARTIAL_PARSE, False) - setattr(self.args, "partial_parse", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.PARTIAL_PARSE, True) - # cleanup - os.environ.pop("DBT_PARTIAL_PARSE") - delattr(self.args, "partial_parse") - self.user_config.partial_parse = False - - # use_colors - self.user_config.use_colors = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.USE_COLORS, True) - os.environ["DBT_USE_COLORS"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.USE_COLORS, False) - setattr(self.args, "use_colors", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.USE_COLORS, True) - # cleanup - os.environ.pop("DBT_USE_COLORS") - delattr(self.args, "use_colors") - - # debug - self.user_config.debug = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.DEBUG, True) - os.environ["DBT_DEBUG"] = "True" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.DEBUG, True) - os.environ["DBT_DEBUG"] = "False" - setattr(self.args, "debug", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.DEBUG, True) - # cleanup - os.environ.pop("DBT_DEBUG") - delattr(self.args, "debug") - self.user_config.debug = None - - # log_format -- text, json, default - self.user_config.log_format = "text" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.LOG_FORMAT, "text") - os.environ["DBT_LOG_FORMAT"] = "json" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.LOG_FORMAT, "json") - setattr(self.args, "log_format", "text") - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.LOG_FORMAT, "text") - # cleanup - os.environ.pop("DBT_LOG_FORMAT") - delattr(self.args, "log_format") - self.user_config.log_format = None - - # version_check - self.user_config.version_check = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.VERSION_CHECK, True) - os.environ["DBT_VERSION_CHECK"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.VERSION_CHECK, False) - setattr(self.args, "version_check", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.VERSION_CHECK, True) - # cleanup - os.environ.pop("DBT_VERSION_CHECK") - delattr(self.args, "version_check") - - # fail_fast - self.user_config.fail_fast = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.FAIL_FAST, True) - os.environ["DBT_FAIL_FAST"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.FAIL_FAST, False) - setattr(self.args, "fail_fast", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.FAIL_FAST, True) - # cleanup - os.environ.pop("DBT_FAIL_FAST") - delattr(self.args, "fail_fast") - self.user_config.fail_fast = False - - # send_anonymous_usage_stats - self.user_config.send_anonymous_usage_stats = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.SEND_ANONYMOUS_USAGE_STATS, True) - os.environ["DBT_SEND_ANONYMOUS_USAGE_STATS"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.SEND_ANONYMOUS_USAGE_STATS, False) - setattr(self.args, "send_anonymous_usage_stats", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.SEND_ANONYMOUS_USAGE_STATS, True) - os.environ["DO_NOT_TRACK"] = "1" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.SEND_ANONYMOUS_USAGE_STATS, False) - # cleanup - os.environ.pop("DBT_SEND_ANONYMOUS_USAGE_STATS") - os.environ.pop("DO_NOT_TRACK") - delattr(self.args, "send_anonymous_usage_stats") - - # printer_width - self.user_config.printer_width = 100 - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.PRINTER_WIDTH, 100) - os.environ["DBT_PRINTER_WIDTH"] = "80" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.PRINTER_WIDTH, 80) - setattr(self.args, "printer_width", "120") - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.PRINTER_WIDTH, 120) - # cleanup - os.environ.pop("DBT_PRINTER_WIDTH") - delattr(self.args, "printer_width") - self.user_config.printer_width = None - - # indirect_selection - self.user_config.indirect_selection = "eager" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.INDIRECT_SELECTION, IndirectSelection.Eager) - self.user_config.indirect_selection = "cautious" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.INDIRECT_SELECTION, IndirectSelection.Cautious) - self.user_config.indirect_selection = "buildable" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.INDIRECT_SELECTION, IndirectSelection.Buildable) - self.user_config.indirect_selection = None - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.INDIRECT_SELECTION, IndirectSelection.Eager) - os.environ["DBT_INDIRECT_SELECTION"] = "cautious" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.INDIRECT_SELECTION, IndirectSelection.Cautious) - setattr(self.args, "indirect_selection", "cautious") - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.INDIRECT_SELECTION, IndirectSelection.Cautious) - # cleanup - os.environ.pop("DBT_INDIRECT_SELECTION") - delattr(self.args, "indirect_selection") - self.user_config.indirect_selection = None - - # quiet - self.user_config.quiet = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.QUIET, True) - # cleanup - self.user_config.quiet = None - - # no_print - self.user_config.no_print = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.NO_PRINT, True) - # cleanup - self.user_config.no_print = None - - # cache_selected_only - self.user_config.cache_selected_only = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.CACHE_SELECTED_ONLY, True) - os.environ["DBT_CACHE_SELECTED_ONLY"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.CACHE_SELECTED_ONLY, False) - setattr(self.args, "cache_selected_only", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.CACHE_SELECTED_ONLY, True) - # cleanup - os.environ.pop("DBT_CACHE_SELECTED_ONLY") - delattr(self.args, "cache_selected_only") - self.user_config.cache_selected_only = False - - # target_path/log_path - flags.set_from_args(self.args, self.user_config) - self.assertIsNone(flags.LOG_PATH) - os.environ["DBT_LOG_PATH"] = "a/b/c" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.LOG_PATH, "a/b/c") - setattr(self.args, "log_path", "d/e/f") - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.LOG_PATH, "d/e/f") - # cleanup - os.environ.pop("DBT_LOG_PATH") - delattr(self.args, "log_path") - - def test__flags_are_mutually_exclusive(self): - # options from user config - self.user_config.warn_error = False - self.user_config.warn_error_options = '{"include":"all"}' - with pytest.raises(ValueError): - flags.set_from_args(self.args, self.user_config) - # cleanup - self.user_config.warn_error = None - self.user_config.warn_error_options = None - - # options from args - setattr(self.args, "warn_error", False) - setattr(self.args, "warn_error_options", '{"include":"all"}') - with pytest.raises(ValueError): - flags.set_from_args(self.args, self.user_config) - # cleanup - delattr(self.args, "warn_error") - delattr(self.args, "warn_error_options") - - # options from environment - os.environ["DBT_WARN_ERROR"] = "false" - os.environ["DBT_WARN_ERROR_OPTIONS"] = '{"include": []}' - with pytest.raises(ValueError): - flags.set_from_args(self.args, self.user_config) - # cleanup - os.environ.pop("DBT_WARN_ERROR") - os.environ.pop("DBT_WARN_ERROR_OPTIONS") - - # options from user config + args - self.user_config.warn_error = False - setattr(self.args, "warn_error_options", '{"include":"all"}') - with pytest.raises(ValueError): - flags.set_from_args(self.args, self.user_config) - # cleanup - self.user_config.warn_error = None - delattr(self.args, "warn_error_options") - - # options from user config + environ - self.user_config.warn_error = False - os.environ["DBT_WARN_ERROR_OPTIONS"] = '{"include": []}' - with pytest.raises(ValueError): - flags.set_from_args(self.args, self.user_config) - # cleanup - self.user_config.warn_error = None - os.environ.pop("DBT_WARN_ERROR_OPTIONS") - - # options from args + environ - setattr(self.args, "warn_error", False) - os.environ["DBT_WARN_ERROR_OPTIONS"] = '{"include": []}' - with pytest.raises(ValueError): - flags.set_from_args(self.args, self.user_config) - # cleanup - delattr(self.args, "warn_error") - os.environ.pop("DBT_WARN_ERROR_OPTIONS")
tests/unit/test_graph.py+2 −1 modified@@ -16,6 +16,7 @@ import dbt.parser.manifest from dbt import tracking from dbt.contracts.files import SourceFile, FileHash, FilePath +from dbt.contracts.project import ProjectFlags from dbt.contracts.graph.manifest import MacroManifest, ManifestStateCheck from dbt.graph import NodeSelector, parse_difference from dbt.events.functions import setup_event_logger @@ -140,7 +141,7 @@ def get_config(self, extra_cfg=None): cfg.update(extra_cfg) config = config_from_parts_or_dicts(project=cfg, profile=self.profile) - dbt.flags.set_from_args(Namespace(), config) + dbt.flags.set_from_args(Namespace(), ProjectFlags()) setup_event_logger(dbt.flags.get_flags()) object.__setattr__(dbt.flags.get_flags(), "PARTIAL_PARSE", False) return config
tests/unit/test_graph_selection.py+2 −2 modified@@ -13,9 +13,9 @@ from dbt import flags from argparse import Namespace -from dbt.contracts.project import UserConfig +from dbt.contracts.project import ProjectFlags -flags.set_from_args(Namespace(), UserConfig()) +flags.set_from_args(Namespace(), ProjectFlags()) def _get_graph():
tests/unit/test_manifest.py+176 −1 modified@@ -1224,7 +1224,7 @@ def test_find_generate_macros_by_name(macros, expectations): FindMaterializationSpec = namedtuple("FindMaterializationSpec", "macros,adapter_type,expected") -def _materialization_parameter_sets(): +def _materialization_parameter_sets_legacy(): # inject the plugins used for materialization parameter tests with mock.patch("dbt.adapters.base.plugin.project_name_from_path") as get_name: get_name.return_value = "foo" @@ -1371,12 +1371,187 @@ def id_mat(arg): return "_".join(arg) +@pytest.mark.parametrize( + "macros,adapter_type,expected", + _materialization_parameter_sets_legacy(), + ids=id_mat, +) +def test_find_materialization_by_name_legacy(macros, adapter_type, expected): + set_from_args( + Namespace( + SEND_ANONYMOUS_USAGE_STATS=False, + REQUIRE_EXPLICIT_PACKAGE_OVERRIDES_FOR_BUILTIN_MATERIALIZATIONS=False, + ), + None, + ) + + manifest = make_manifest(macros=macros) + result = manifest.find_materialization_macro_by_name( + project_name="root", + materialization_name="my_materialization", + adapter_type=adapter_type, + ) + if expected is None: + assert result is expected + else: + expected_package, expected_adapter_type = expected + assert result.adapter_type == expected_adapter_type + assert result.package_name == expected_package + + +def _materialization_parameter_sets(): + # inject the plugins used for materialization parameter tests + with mock.patch("dbt.adapters.base.plugin.project_name_from_path") as get_name: + get_name.return_value = "foo" + FooPlugin = AdapterPlugin( + adapter=mock.MagicMock(), + credentials=mock.MagicMock(), + include_path="/path/to/root/plugin", + ) + FooPlugin.adapter.type.return_value = "foo" + inject_plugin(FooPlugin) + + BarPlugin = AdapterPlugin( + adapter=mock.MagicMock(), + credentials=mock.MagicMock(), + include_path="/path/to/root/plugin", + dependencies=["foo"], + ) + BarPlugin.adapter.type.return_value = "bar" + inject_plugin(BarPlugin) + + sets = [ + FindMaterializationSpec(macros=[], adapter_type="foo", expected=None), + ] + + # default only, each project + sets.extend( + FindMaterializationSpec( + macros=[MockMaterialization(project, adapter_type=None)], + adapter_type="foo", + expected=(project, "default"), + ) + for project in ["root", "dep", "dbt"] + ) + + # other type only, each project + sets.extend( + FindMaterializationSpec( + macros=[MockMaterialization(project, adapter_type="bar")], + adapter_type="foo", + expected=None, + ) + for project in ["root", "dep", "dbt"] + ) + + # matching type only, each project + sets.extend( + FindMaterializationSpec( + macros=[MockMaterialization(project, adapter_type="foo")], + adapter_type="foo", + expected=(project, "foo"), + ) + for project in ["root", "dep", "dbt"] + ) + + sets.extend( + [ + # matching type and default everywhere + FindMaterializationSpec( + macros=[ + MockMaterialization(project, adapter_type=atype) + for (project, atype) in product(["root", "dep", "dbt"], ["foo", None]) + ], + adapter_type="foo", + expected=("root", "foo"), + ), + # default in core, override is in dep, and root has unrelated override + # should find the dbt default because default materializations cannot be overwritten by packages. + FindMaterializationSpec( + macros=[ + MockMaterialization("root", adapter_type="bar"), + MockMaterialization("dep", adapter_type="foo"), + MockMaterialization("dbt", adapter_type=None), + ], + adapter_type="foo", + expected=("dbt", "default"), + ), + # default in core, unrelated override is in dep, and root has an override + # should find the root override. + FindMaterializationSpec( + macros=[ + MockMaterialization("root", adapter_type="foo"), + MockMaterialization("dep", adapter_type="bar"), + MockMaterialization("dbt", adapter_type=None), + ], + adapter_type="foo", + expected=("root", "foo"), + ), + # default in core, override is in dep, and root has an override too. + # should find the root override. + FindMaterializationSpec( + macros=[ + MockMaterialization("root", adapter_type="foo"), + MockMaterialization("dep", adapter_type="foo"), + MockMaterialization("dbt", adapter_type=None), + ], + adapter_type="foo", + expected=("root", "foo"), + ), + # core has default + adapter, dep has adapter, root has default + # should find the default adapter implementation, because it's the most specific + # and default materializations cannot be overwritten by packages + FindMaterializationSpec( + macros=[ + MockMaterialization("root", adapter_type=None), + MockMaterialization("dep", adapter_type="foo"), + MockMaterialization("dbt", adapter_type=None), + MockMaterialization("dbt", adapter_type="foo"), + ], + adapter_type="foo", + expected=("dbt", "foo"), + ), + ] + ) + + # inherit from parent adapter + sets.extend( + FindMaterializationSpec( + macros=[MockMaterialization(project, adapter_type="foo")], + adapter_type="bar", + expected=(project, "foo"), + ) + for project in ["root", "dep", "dbt"] + ) + sets.extend( + FindMaterializationSpec( + macros=[ + MockMaterialization(project, adapter_type="foo"), + MockMaterialization(project, adapter_type="bar"), + ], + adapter_type="bar", + expected=(project, "bar"), + ) + for project in ["root", "dep", "dbt"] + ) + + return sets + + @pytest.mark.parametrize( "macros,adapter_type,expected", _materialization_parameter_sets(), ids=id_mat, ) def test_find_materialization_by_name(macros, adapter_type, expected): + set_from_args( + Namespace( + SEND_ANONYMOUS_USAGE_STATS=False, + REQUIRE_EXPLICIT_PACKAGE_OVERRIDES_FOR_BUILTIN_MATERIALIZATIONS=True, + ), + None, + ) + manifest = make_manifest(macros=macros) result = manifest.find_materialization_macro_by_name( project_name="root",
87ac4deb00cc[Backport] deprecate materialization overrides from imported packages (#9998)
38 files changed · +1779 −1557
.changes/unreleased/Features-20231218-195854.yaml+6 −0 added@@ -0,0 +1,6 @@ +kind: Features +body: Move flags from UserConfig in profiles.yml to flags in dbt_project.yml +time: 2023-12-18T19:58:54.075811-05:00 +custom: + Author: gshank + Issue: "9183"
.changes/unreleased/Features-20240422-173703.yaml+6 −0 added@@ -0,0 +1,6 @@ +kind: Features +body: Add require_explicit_package_overrides_for_builtin_materializations to dbt_project.yml flags, which can be used to opt-out of overriding built-in materializations from packages +time: 2024-04-22T17:37:03.892268-04:00 +custom: + Author: michelleark + Issue: "10007"
.changes/unreleased/Under the Hood-20240418-172528.yaml+6 −0 added@@ -0,0 +1,6 @@ +kind: Under the Hood +body: Raise deprecation warning if installed package overrides built-in materialization +time: 2024-04-18T17:25:28.37886-04:00 +custom: + Author: michelleark + Issue: "9971"
core/dbt/cli/flags.py+29 −13 modified@@ -3,6 +3,7 @@ from dataclasses import dataclass from importlib import import_module from multiprocessing import get_context +from pathlib import Path from pprint import pformat as pf from typing import Any, Callable, Dict, List, Optional, Set, Union @@ -11,8 +12,8 @@ from dbt.cli.exceptions import DbtUsageException from dbt.cli.resolvers import default_log_path, default_project_dir from dbt.cli.types import Command as CliCommand -from dbt.config.profile import read_user_config -from dbt.contracts.project import UserConfig +from dbt.config.project import read_project_flags +from dbt.contracts.project import ProjectFlags from dbt.exceptions import DbtInternalError from dbt.deprecations import renamed_env_var from dbt.helper_types import WarnErrorOptions @@ -25,7 +26,7 @@ "INDIRECT_SELECTION": "eager", "TARGET_PATH": None, "WARN_ERROR": None, - # Cli args without user_config or env var option. + # Cli args without project_flags or env var option. "FULL_REFRESH": False, "STRICT_MODE": False, "STORE_FAILURES": False, @@ -77,7 +78,7 @@ class Flags: """Primary configuration artifact for running dbt""" def __init__( - self, ctx: Optional[Context] = None, user_config: Optional[UserConfig] = None + self, ctx: Optional[Context] = None, project_flags: Optional[ProjectFlags] = None ) -> None: # Set the default flags. for key, value in FLAGS_DEFAULTS.items(): @@ -200,27 +201,40 @@ def _assign_params( invoked_subcommand_ctx, params_assigned_from_default, deprecated_env_vars ) - if not user_config: + if not project_flags: + project_dir = getattr(self, "PROJECT_DIR", str(default_project_dir())) profiles_dir = getattr(self, "PROFILES_DIR", None) - user_config = read_user_config(profiles_dir) if profiles_dir else None + if profiles_dir and project_dir: + project_flags = read_project_flags(project_dir, profiles_dir) + else: + project_flags = None # Add entire invocation command to flags object.__setattr__(self, "INVOCATION_COMMAND", "dbt " + " ".join(sys.argv[1:])) - # Overwrite default assignments with user config if available. - if user_config: + if project_flags: + # Overwrite default assignments with project flags if available. param_assigned_from_default_copy = params_assigned_from_default.copy() for param_assigned_from_default in params_assigned_from_default: - user_config_param_value = getattr(user_config, param_assigned_from_default, None) - if user_config_param_value is not None: + project_flags_param_value = getattr( + project_flags, param_assigned_from_default, None + ) + if project_flags_param_value is not None: object.__setattr__( self, param_assigned_from_default.upper(), - convert_config(param_assigned_from_default, user_config_param_value), + convert_config(param_assigned_from_default, project_flags_param_value), ) param_assigned_from_default_copy.remove(param_assigned_from_default) params_assigned_from_default = param_assigned_from_default_copy + # Add project-level flags that are not available as CLI options / env vars + for ( + project_level_flag_name, + project_level_flag_value, + ) in project_flags.project_only_flags.items(): + object.__setattr__(self, project_level_flag_name.upper(), project_level_flag_value) + # Set hard coded flags. object.__setattr__(self, "WHICH", invoked_subcommand_name or ctx.info_name) object.__setattr__(self, "MP_CONTEXT", get_context("spawn")) @@ -234,9 +248,11 @@ def _assign_params( # Starting in v1.5, if `log-path` is set in `dbt_project.yml`, it will raise a deprecation warning, # with the possibility of removing it in a future release. if getattr(self, "LOG_PATH", None) is None: - project_dir = getattr(self, "PROJECT_DIR", default_project_dir()) + project_dir = getattr(self, "PROJECT_DIR", str(default_project_dir())) version_check = getattr(self, "VERSION_CHECK", True) - object.__setattr__(self, "LOG_PATH", default_log_path(project_dir, version_check)) + object.__setattr__( + self, "LOG_PATH", default_log_path(Path(project_dir), version_check) + ) # Support console DO NOT TRACK initiative. if os.getenv("DO_NOT_TRACK", "").lower() in ("1", "t", "true", "y", "yes"):
core/dbt/config/__init__.py+1 −1 modified@@ -1,4 +1,4 @@ # all these are just exports, they need "noqa" so flake8 will not complain. -from .profile import Profile, read_user_config # noqa +from .profile import Profile # noqa from .project import Project, IsFQNResource, PartialProject # noqa from .runtime import RuntimeConfig # noqa
core/dbt/config/profile.py+1 −38 modified@@ -8,7 +8,7 @@ from dbt.clients.system import load_file_contents from dbt.clients.yaml_helper import load_yaml_text from dbt.contracts.connection import Credentials, HasCredentials -from dbt.contracts.project import ProfileConfig, UserConfig +from dbt.contracts.project import ProfileConfig from dbt.exceptions import ( CompilationError, DbtProfileError, @@ -19,7 +19,6 @@ ) from dbt.events.types import MissingProfileTarget from dbt.events.functions import fire_event -from dbt.utils import coerce_dict_str from .renderer import ProfileRenderer @@ -51,27 +50,13 @@ def read_profile(profiles_dir: str) -> Dict[str, Any]: return {} -def read_user_config(directory: str) -> UserConfig: - try: - profile = read_profile(directory) - if profile: - user_config = coerce_dict_str(profile.get("config", {})) - if user_config is not None: - UserConfig.validate(user_config) - return UserConfig.from_dict(user_config) - except (DbtRuntimeError, ValidationError): - pass - return UserConfig() - - # The Profile class is included in RuntimeConfig, so any attribute # additions must also be set where the RuntimeConfig class is created # `init=False` is a workaround for https://bugs.python.org/issue45081 @dataclass(init=False) class Profile(HasCredentials): profile_name: str target_name: str - user_config: UserConfig threads: int credentials: Credentials profile_env_vars: Dict[str, Any] @@ -80,7 +65,6 @@ def __init__( self, profile_name: str, target_name: str, - user_config: UserConfig, threads: int, credentials: Credentials, ) -> None: @@ -89,7 +73,6 @@ def __init__( """ self.profile_name = profile_name self.target_name = target_name - self.user_config = user_config self.threads = threads self.credentials = credentials self.profile_env_vars = {} # never available on init @@ -106,12 +89,10 @@ def to_profile_info(self, serialize_credentials: bool = False) -> Dict[str, Any] result = { "profile_name": self.profile_name, "target_name": self.target_name, - "user_config": self.user_config, "threads": self.threads, "credentials": self.credentials, } if serialize_credentials: - result["user_config"] = self.user_config.to_dict(omit_none=True) result["credentials"] = self.credentials.to_dict(omit_none=True) return result @@ -124,7 +105,6 @@ def to_target_dict(self) -> Dict[str, Any]: "name": self.target_name, "target_name": self.target_name, "profile_name": self.profile_name, - "config": self.user_config.to_dict(omit_none=True), } ) return target @@ -246,7 +226,6 @@ def from_credentials( threads: int, profile_name: str, target_name: str, - user_config: Optional[Dict[str, Any]] = None, ) -> "Profile": """Create a profile from an existing set of Credentials and the remaining information. @@ -255,20 +234,13 @@ def from_credentials( :param threads: The number of threads to use for connections. :param profile_name: The profile name used for this profile. :param target_name: The target name used for this profile. - :param user_config: The user-level config block from the - raw profiles, if specified. :raises DbtProfileError: If the profile is invalid. :returns: The new Profile object. """ - if user_config is None: - user_config = {} - UserConfig.validate(user_config) - user_config_obj: UserConfig = UserConfig.from_dict(user_config) profile = cls( profile_name=profile_name, target_name=target_name, - user_config=user_config_obj, threads=threads, credentials=credentials, ) @@ -316,7 +288,6 @@ def from_raw_profile_info( raw_profile: Dict[str, Any], profile_name: str, renderer: ProfileRenderer, - user_config: Optional[Dict[str, Any]] = None, target_override: Optional[str] = None, threads_override: Optional[int] = None, ) -> "Profile": @@ -328,8 +299,6 @@ def from_raw_profile_info( disk as yaml and its values rendered with jinja. :param profile_name: The profile name used. :param renderer: The config renderer. - :param user_config: The global config for the user, if it - was present. :param target_override: The target to use, if provided on the command line. :param threads_override: The thread count to use, if @@ -338,9 +307,6 @@ def from_raw_profile_info( target could not be found :returns: The new Profile object. """ - # user_config is not rendered. - if user_config is None: - user_config = raw_profile.get("config") # TODO: should it be, and the values coerced to bool? target_name, profile_data = cls.render_profile( raw_profile, profile_name, target_override, renderer @@ -361,7 +327,6 @@ def from_raw_profile_info( profile_name=profile_name, target_name=target_name, threads=threads, - user_config=user_config, ) @classmethod @@ -396,13 +361,11 @@ def from_raw_profiles( if not raw_profile: msg = f"Profile {profile_name} in profiles.yml is empty" raise DbtProfileError(INVALID_PROFILE_MESSAGE.format(error_string=msg)) - user_config = raw_profiles.get("config") return cls.from_raw_profile_info( raw_profile=raw_profile, profile_name=profile_name, renderer=renderer, - user_config=user_config, target_override=target_override, threads_override=threads_override, )
core/dbt/config/project.py+73 −14 modified@@ -20,6 +20,7 @@ DEPENDENCIES_FILE_NAME, PACKAGES_FILE_NAME, PACKAGE_LOCK_HASH_KEY, + DBT_PROJECT_FILE_NAME, ) from dbt.clients.system import path_exists, load_file_contents from dbt.clients.yaml_helper import load_yaml_text @@ -35,12 +36,13 @@ from dbt.helper_types import NoValue from dbt.semver import VersionSpecifier, versions_compatible from dbt.version import get_installed_version -from dbt.utils import MultiDict, md5 +from dbt.utils import MultiDict, md5, coerce_dict_str from dbt.node_types import NodeType from dbt.config.selectors import SelectorDict from dbt.contracts.project import ( Project as ProjectContract, SemverString, + ProjectFlags, ) from dbt.contracts.project import PackageConfig, ProjectPackageMetadata from dbt.dataclass_schema import ValidationError @@ -81,8 +83,8 @@ """ MISSING_DBT_PROJECT_ERROR = """\ -No dbt_project.yml found at expected path {path} -Verify that each entry within packages.yml (and their transitive dependencies) contains a file named dbt_project.yml +No {DBT_PROJECT_FILE_NAME} found at expected path {path} +Verify that each entry within packages.yml (and their transitive dependencies) contains a file named {DBT_PROJECT_FILE_NAME} """ @@ -199,16 +201,20 @@ def value_or(value: Optional[T], default: T) -> T: def load_raw_project(project_root: str) -> Dict[str, Any]: project_root = os.path.normpath(project_root) - project_yaml_filepath = os.path.join(project_root, "dbt_project.yml") + project_yaml_filepath = os.path.join(project_root, DBT_PROJECT_FILE_NAME) # get the project.yml contents if not path_exists(project_yaml_filepath): - raise DbtProjectError(MISSING_DBT_PROJECT_ERROR.format(path=project_yaml_filepath)) + raise DbtProjectError( + MISSING_DBT_PROJECT_ERROR.format( + path=project_yaml_filepath, DBT_PROJECT_FILE_NAME=DBT_PROJECT_FILE_NAME + ) + ) project_dict = _load_yaml(project_yaml_filepath) if not isinstance(project_dict, dict): - raise DbtProjectError("dbt_project.yml does not parse to a dictionary") + raise DbtProjectError(f"{DBT_PROJECT_FILE_NAME} does not parse to a dictionary") return project_dict @@ -323,21 +329,21 @@ def get_rendered( selectors_dict=rendered_selectors, ) - # Called by Project.from_project_root (not PartialProject.from_project_root!) + # Called by Project.from_project_root which first calls PartialProject.from_project_root def render(self, renderer: DbtProjectYamlRenderer) -> "Project": try: rendered = self.get_rendered(renderer) return self.create_project(rendered) except DbtProjectError as exc: if exc.path is None: - exc.path = os.path.join(self.project_root, "dbt_project.yml") + exc.path = os.path.join(self.project_root, DBT_PROJECT_FILE_NAME) raise def render_package_metadata(self, renderer: PackageRenderer) -> ProjectPackageMetadata: packages_data = renderer.render_data(self.packages_dict) packages_config = package_config_from_data(packages_data, self.packages_dict) if not self.project_name: - raise DbtProjectError("Package dbt_project.yml must have a name!") + raise DbtProjectError(f"Package defined in {DBT_PROJECT_FILE_NAME} must have a name!") return ProjectPackageMetadata(self.project_name, packages_config.packages) def check_config_path( @@ -348,7 +354,7 @@ def check_config_path( msg = ( "{deprecated_path} and {expected_path} cannot both be defined. The " "`{deprecated_path}` config has been deprecated in favor of `{expected_path}`. " - "Please update your `dbt_project.yml` configuration to reflect this " + f"Please update your `{DBT_PROJECT_FILE_NAME}` configuration to reflect this " "change." ) raise DbtProjectError( @@ -420,11 +426,11 @@ def create_project(self, rendered: RenderComponents) -> "Project": docs_paths: List[str] = value_or(cfg.docs_paths, all_source_paths) asset_paths: List[str] = value_or(cfg.asset_paths, []) - flags = get_flags() + global_flags = get_flags() - flag_target_path = str(flags.TARGET_PATH) if flags.TARGET_PATH else None + flag_target_path = str(global_flags.TARGET_PATH) if global_flags.TARGET_PATH else None target_path: str = flag_or(flag_target_path, cfg.target_path, "target") - log_path: str = str(flags.LOG_PATH) + log_path: str = str(global_flags.LOG_PATH) clean_targets: List[str] = value_or(cfg.clean_targets, [target_path]) packages_install_path: str = value_or(cfg.packages_install_path, "dbt_packages") @@ -569,6 +575,11 @@ def from_project_root( ) = package_and_project_data_from_root(project_root) selectors_dict = selector_data_from_root(project_root) + if "flags" in project_dict: + # We don't want to include "flags" in the Project, + # it goes in ProjectFlags + project_dict.pop("flags") + return cls.from_dicts( project_root=project_root, project_dict=project_dict, @@ -709,7 +720,6 @@ def to_project_config(self, with_packages=False): "exposures": self.exposures, "vars": self.vars.to_dict(), "require-dbt-version": [v.to_version_string() for v in self.dbt_version], - "config-version": self.config_version, "restrict-access": self.restrict_access, "dbt-cloud": self.dbt_cloud, } @@ -773,3 +783,52 @@ def get_macro_search_order(self, macro_namespace: str): def project_target_path(self): # If target_path is absolute, project_root will not be included return os.path.join(self.project_root, self.target_path) + + +def read_project_flags(project_dir: str, profiles_dir: str) -> ProjectFlags: + try: + project_flags: Dict[str, Any] = {} + # Read project_flags from dbt_project.yml first + # Flags are instantiated before the project, so we don't + # want to throw an error for non-existence of dbt_project.yml here + # because it breaks things. + project_root = os.path.normpath(project_dir) + project_yaml_filepath = os.path.join(project_root, DBT_PROJECT_FILE_NAME) + if path_exists(project_yaml_filepath): + try: + project_dict = load_raw_project(project_root) + if "flags" in project_dict: + project_flags = project_dict.pop("flags") + except Exception: + # This is probably a yaml load error.The error will be reported + # later, when the project loads. + pass + + from dbt.config.profile import read_profile + + profile = read_profile(profiles_dir) + profile_project_flags: Optional[Dict[str, Any]] = {} + if profile: + profile_project_flags = coerce_dict_str(profile.get("config", {})) + + if project_flags and profile_project_flags: + raise DbtProjectError( + f"Do not specify both 'config' in profiles.yml and 'flags' in {DBT_PROJECT_FILE_NAME}. " + "Using 'config' in profiles.yml is deprecated." + ) + + if profile_project_flags: + # This can't use WARN_ERROR or WARN_ERROR_OPTIONS because they're in + # the config that we're loading. Uses special "warn" method. + deprecations.warn("project-flags-moved") + project_flags = profile_project_flags + + if project_flags is not None: + ProjectFlags.validate(project_flags) + return ProjectFlags.from_dict(project_flags) + except (DbtProjectError) as exc: + # We don't want to eat the DbtProjectError for UserConfig to ProjectFlags + raise exc + except (DbtRuntimeError, ValidationError): + pass + return ProjectFlags()
core/dbt/config/runtime.py+1 −3 modified@@ -20,7 +20,7 @@ from dbt.config.project import load_raw_project from dbt.contracts.connection import AdapterRequiredConfig, Credentials, HasCredentials from dbt.contracts.graph.manifest import ManifestMetadata -from dbt.contracts.project import Configuration, UserConfig +from dbt.contracts.project import Configuration from dbt.contracts.relation import ComponentName from dbt.dataclass_schema import ValidationError from dbt.events.functions import warn_or_error @@ -178,7 +178,6 @@ def from_parts( profile_env_vars=profile.profile_env_vars, profile_name=profile.profile_name, target_name=profile.target_name, - user_config=profile.user_config, threads=profile.threads, credentials=profile.credentials, args=args, @@ -432,7 +431,6 @@ def _connection_keys(self): class UnsetProfile(Profile): def __init__(self): self.credentials = UnsetCredentials() - self.user_config = UserConfig() # This will be read in _get_rendered_profile self.profile_name = "" self.target_name = "" self.threads = -1
core/dbt/contracts/connection.py+0 −8 modified@@ -178,17 +178,9 @@ def __post_serialize__(self, dct): return dct -class UserConfigContract(Protocol): - send_anonymous_usage_stats: bool - use_colors: Optional[bool] = None - partial_parse: Optional[bool] = None - printer_width: Optional[int] = None - - class HasCredentials(Protocol): credentials: Credentials profile_name: str - user_config: UserConfigContract target_name: str threads: int
core/dbt/contracts/graph/manifest.py+49 −4 modified@@ -25,6 +25,7 @@ from typing_extensions import Protocol from uuid import UUID + from dbt.contracts.graph.nodes import ( BaseNode, Documentation, @@ -67,7 +68,7 @@ from dbt.events.contextvars import get_node_info from dbt.node_types import NodeType, AccessType from dbt.flags import get_flags, MP_CONTEXT -from dbt import tracking +from dbt import tracking, deprecations import dbt.utils @@ -616,11 +617,29 @@ def __lt__(self, other: object) -> bool: class CandidateList(List[M]): - def last(self) -> Optional[Macro]: + def last_candidate( + self, valid_localities: Optional[List[Locality]] = None + ) -> Optional[MacroCandidate]: + """ + Obtain the last (highest precedence) MacroCandidate from the CandidateList of any locality in valid_localities. + If valid_localities is not specified, return the last MacroCandidate of any locality. + """ if not self: return None self.sort() - return self[-1].macro + + if valid_localities is None: + return self[-1] + + for candidate in reversed(self): + if candidate.locality in valid_localities: + return candidate + + return None + + def last(self) -> Optional[Macro]: + last_candidate = self.last_candidate() + return last_candidate.macro if last_candidate is not None else None def _get_locality(macro: Macro, root_project_name: str, internal_packages: Set[str]) -> Locality: @@ -914,7 +933,33 @@ def find_materialization_macro_by_name( for specificity, atype in enumerate(self._get_parent_adapter_types(adapter_type)) ) ) - return candidates.last() + core_candidates = [ + candidate for candidate in candidates if candidate.locality == Locality.Core + ] + + materialization_candidate = candidates.last_candidate() + # If an imported materialization macro was found that also had a core candidate, fire a deprecation + if ( + materialization_candidate is not None + and materialization_candidate.locality == Locality.Imported + and core_candidates + ): + # preserve legacy behaviour - allow materialization override + if ( + get_flags().require_explicit_package_overrides_for_builtin_materializations + is False + ): + deprecations.warn( + "package-materialization-override", + package_name=materialization_candidate.macro.package_name, + materialization_name=materialization_name, + ) + else: + materialization_candidate = candidates.last_candidate( + valid_localities=[Locality.Core, Locality.Root] + ) + + return materialization_candidate.macro if materialization_candidate else None def get_resource_fqns(self) -> Mapping[str, PathSet]: resource_fqns: Dict[str, Set[Tuple[str, ...]]] = {}
core/dbt/contracts/project.py+9 −3 modified@@ -1,5 +1,5 @@ from dbt.contracts.util import Replaceable, Mergeable, list_str, Identifier -from dbt.contracts.connection import QueryComment, UserConfigContract +from dbt.contracts.connection import QueryComment from dbt.helper_types import NoValue from dbt.dataclass_schema import ( dbtClassMixin, @@ -283,7 +283,7 @@ def validate(cls, data): @dataclass -class UserConfig(ExtensibleDbtClassMixin, Replaceable, UserConfigContract): +class ProjectFlags(ExtensibleDbtClassMixin, Replaceable): cache_selected_only: Optional[bool] = None debug: Optional[bool] = None fail_fast: Optional[bool] = None @@ -295,6 +295,7 @@ class UserConfig(ExtensibleDbtClassMixin, Replaceable, UserConfigContract): partial_parse: Optional[bool] = None populate_cache: Optional[bool] = None printer_width: Optional[int] = None + require_explicit_package_overrides_for_builtin_materializations: bool = False send_anonymous_usage_stats: bool = DEFAULT_SEND_ANONYMOUS_USAGE_STATS static_parser: Optional[bool] = None use_colors: Optional[bool] = None @@ -305,12 +306,17 @@ class UserConfig(ExtensibleDbtClassMixin, Replaceable, UserConfigContract): warn_error_options: Optional[Dict[str, Union[str, List[str]]]] = None write_json: Optional[bool] = None + @property + def project_only_flags(self) -> Dict[str, Any]: + return { + "require_explicit_package_overrides_for_builtin_materializations": self.require_explicit_package_overrides_for_builtin_materializations, + } + @dataclass class ProfileConfig(dbtClassMixin, Replaceable): profile_name: str target_name: str - user_config: UserConfig threads: int # TODO: make this a dynamic union of some kind? credentials: Optional[Dict[str, Any]]
core/dbt/deprecations.py+21 −0 modified@@ -96,6 +96,25 @@ class CollectFreshnessReturnSignature(DBTDeprecation): _event = "CollectFreshnessReturnSignature" +class ProjectFlagsMovedDeprecation(DBTDeprecation): + _name = "project-flags-moved" + _event = "ProjectFlagsMovedDeprecation" + + def show(self, *args, **kwargs) -> None: + if self.name not in active_deprecations: + event = self.event(**kwargs) + # We can't do warn_or_error because the ProjectFlags + # is where that is set up and we're just reading it. + dbt.events.functions.fire_event(event) + self.track_deprecation_warn() + active_deprecations.add(self.name) + + +class PackageMaterializationOverrideDeprecation(DBTDeprecation): + _name = "package-materialization-override" + _event = "PackageMaterializationOverrideDeprecation" + + def renamed_env_var(old_name: str, new_name: str): class EnvironmentVariableRenamed(DBTDeprecation): _name = f"environment-variable-renamed:{old_name}" @@ -134,6 +153,8 @@ def warn(name, *args, **kwargs): ConfigLogPathDeprecation(), ConfigTargetPathDeprecation(), CollectFreshnessReturnSignature(), + ProjectFlagsMovedDeprecation(), + PackageMaterializationOverrideDeprecation(), ] deprecations: Dict[str, DBTDeprecation] = {d.name: d for d in deprecations_list}
core/dbt/events/types_pb2.py+930 −921 modifiedcore/dbt/events/types.proto+20 −0 modified@@ -415,6 +415,26 @@ message CollectFreshnessReturnSignatureMsg { CollectFreshnessReturnSignature data = 2; } +// D013 +message ProjectFlagsMovedDeprecation { +} + +message ProjectFlagsMovedDeprecationMsg { + EventInfo info = 1; + ProjectFlagsMovedDeprecation data = 2; +} + +// D016 +message PackageMaterializationOverrideDeprecation { + string package_name = 1; + string materialization_name = 2; +} + +message PackageMaterializationOverrideDeprecationMsg { + EventInfo info = 1; + PackageMaterializationOverrideDeprecation data = 2; +} + // E - DB Adapter // E001
core/dbt/events/types.py+23 −0 modified@@ -796,6 +796,29 @@ def message(self) -> str: return line_wrap_message(warning_tag(msg)) +class ProjectFlagsMovedDeprecation(WarnLevel): + def code(self) -> str: + return "D013" + + def message(self) -> str: + description = ( + "User config should be moved from the 'config' key in profiles.yml to the 'flags' " + "key in dbt_project.yml." + ) + # Can't use line_wrap_message here because flags.printer_width isn't available yet + return warning_tag(f"Deprecated functionality\n\n{description}") + + +class PackageMaterializationOverrideDeprecation(WarnLevel): + def code(self) -> str: + return "D016" + + def message(self) -> str: + description = f"Installed package '{self.package_name}' is overriding the built-in materialization '{self.materialization_name}'. Overrides of built-in materializations from installed packages will be deprecated in future versions of dbt. Please refer to https://docs.getdbt.com/reference/global-configs/legacy-behaviors#require_explicit_package_overrides_for_builtin_materializations for detailed documentation and suggested workarounds." + + return line_wrap_message(warning_tag(description)) + + # ======================================================= # I - Project parsing # =======================================================
core/dbt/flags.py+9 −8 modified@@ -39,23 +39,24 @@ def get_flags(): return GLOBAL_FLAGS -def set_from_args(args: Namespace, user_config): +def set_from_args(args: Namespace, project_flags): global GLOBAL_FLAGS from dbt.cli.main import cli from dbt.cli.flags import Flags, convert_config - # we set attributes of args after initialize the flags, but user_config + # we set attributes of args after initialize the flags, but project_flags # is being read in the Flags constructor, so we need to read it here and pass in - # to make sure we use the correct user_config - if (hasattr(args, "PROFILES_DIR") or hasattr(args, "profiles_dir")) and not user_config: - from dbt.config.profile import read_user_config + # to make sure we use the correct project_flags + profiles_dir = getattr(args, "PROFILES_DIR", None) or getattr(args, "profiles_dir", None) + project_dir = getattr(args, "PROJECT_DIR", None) or getattr(args, "project_dir", None) + if profiles_dir and project_dir: + from dbt.config.project import read_project_flags - profiles_dir = getattr(args, "PROFILES_DIR", None) or getattr(args, "profiles_dir") - user_config = read_user_config(profiles_dir) + project_flags = read_project_flags(project_dir, profiles_dir) # make a dummy context to get the flags, totally arbitrary ctx = cli.make_context("run", ["run"]) - flags = Flags(ctx, user_config) + flags = Flags(ctx, project_flags) for arg_name, args_param_value in vars(args).items(): args_param_value = convert_config(arg_name, args_param_value) object.__setattr__(flags, arg_name.upper(), args_param_value)
core/dbt/include/starter_project/dbt_project.yml+0 −1 modified@@ -4,7 +4,6 @@ # name or the intended use of these models name: '{project_name}' version: '1.0.0' -config-version: 2 # This setting configures which "profile" dbt uses for this project. profile: '{profile_name}'
core/dbt/tests/fixtures/project.py+1 −1 modified@@ -142,7 +142,6 @@ def profiles_config_update(): @pytest.fixture(scope="class") def dbt_profile_data(unique_schema, dbt_profile_target, profiles_config_update): profile = { - "config": {"send_anonymous_usage_stats": False}, "test": { "outputs": { "default": {}, @@ -181,6 +180,7 @@ def dbt_project_yml(project_root, project_config_update): project_config = { "name": "test", "profile": "test", + "flags": {"send_anonymous_usage_stats": False}, } if project_config_update: if isinstance(project_config_update, dict):
core/dbt/tracking.py+0 −1 modified@@ -471,7 +471,6 @@ def process(self, record): def initialize_from_flags(send_anonymous_usage_stats, profiles_dir): - # Setting these used to be in UserConfig, but had to be moved here global active_user if send_anonymous_usage_stats: active_user = User(profiles_dir)
core/dbt/utils.py+1 −1 modified@@ -631,7 +631,7 @@ def _connection_exception_retry(fn, max_attempts: int, attempt: int = 0): def args_to_dict(args): var_args = vars(args).copy() # update the args with the flags, which could also come from environment - # variables or user_config + # variables or project_flags flag_dict = flags.get_flag_dict() var_args.update(flag_dict) dict_args = {}
tests/functional/basic/test_mixed_case_db.py+0 −1 modified@@ -16,7 +16,6 @@ def models(): def dbt_profile_data(unique_schema): return { - "config": {"send_anonymous_usage_stats": False}, "test": { "outputs": { "default": {
tests/functional/basic/test_project.py+6 −1 modified@@ -77,11 +77,16 @@ def test_dbt_cloud(self, project): conf = yaml.safe_load( Path(os.path.join(project.project_root, "dbt_project.yml")).read_text() ) - assert conf == {"name": "test", "profile": "test"} + assert conf == { + "name": "test", + "profile": "test", + "flags": {"send_anonymous_usage_stats": False}, + } config = { "name": "test", "profile": "test", + "flags": {"send_anonymous_usage_stats": False}, "dbt-cloud": { "account_id": "123", "application": "test",
tests/functional/configs/test_disabled_configs.py+0 −1 modified@@ -9,7 +9,6 @@ class TestDisabledConfigs(BaseConfigProject): @pytest.fixture(scope="class") def dbt_profile_data(self, unique_schema): return { - "config": {"send_anonymous_usage_stats": False}, "test": { "outputs": { "default": {
tests/functional/dependencies/test_local_dependency.py+4 −9 modified@@ -243,6 +243,10 @@ class TestSimpleDependencyNoVersionCheckConfig(BaseDependencyTest): @pytest.fixture(scope="class") def project_config_update(self): return { + "flags": { + "send_anonymous_usage_stats": False, + "version_check": False, + }, "models": { "schema": "dbt_test", }, @@ -251,15 +255,6 @@ def project_config_update(self): }, } - @pytest.fixture(scope="class") - def profiles_config_update(self): - return { - "config": { - "send_anonymous_usage_stats": False, - "version_check": False, - } - } - @pytest.fixture(scope="class") def macros(self): return {"macro.sql": macros__macro_override_schema_sql}
tests/functional/deprecations/test_deprecations.py+30 −1 modified@@ -2,7 +2,8 @@ from dbt import deprecations import dbt.exceptions -from dbt.tests.util import run_dbt +from dbt.tests.util import run_dbt, write_file +import yaml models__already_exists_sql = """ @@ -157,3 +158,31 @@ def test_exposure_name_fail(self, project): exc_str = " ".join(str(exc.value).split()) # flatten all whitespace expected_msg = "Starting in v1.3, the 'name' of an exposure should contain only letters, numbers, and underscores." assert expected_msg in exc_str + + +class TestPrjectFlagsMovedDeprecation: + @pytest.fixture(scope="class") + def profiles_config_update(self): + return { + "config": {"send_anonymous_usage_stats": False}, + } + + @pytest.fixture(scope="class") + def dbt_project_yml(self, project_root, project_config_update): + project_config = { + "name": "test", + "profile": "test", + } + write_file(yaml.safe_dump(project_config), project_root, "dbt_project.yml") + return project_config + + @pytest.fixture(scope="class") + def models(self): + return {"my_model.sql": "select 1 as fun"} + + def test_profile_config_deprecation(self, project): + deprecations.reset_deprecations() + assert deprecations.active_deprecations == set() + run_dbt(["parse"]) + expected = {"project-flags-moved"} + assert expected == deprecations.active_deprecations
tests/functional/fail_fast/test_fail_fast_run.py+3 −3 modified@@ -44,15 +44,15 @@ def test_fail_fast_run( class TestFailFastFromConfig(FailFastBase): @pytest.fixture(scope="class") - def profiles_config_update(self): + def project_config_update(self): return { - "config": { + "flags": { "send_anonymous_usage_stats": False, "fail_fast": True, } } - def test_fail_fast_run_user_config( + def test_fail_fast_run_project_flags( self, project, models, # noqa: F811
tests/functional/init/test_init.py+3 −13 modified@@ -70,9 +70,7 @@ def test_init_task_in_project_with_existing_profiles_yml( with open(os.path.join(project.profiles_dir, "profiles.yml"), "r") as f: assert ( f.read() - == """config: - send_anonymous_usage_stats: false -test: + == """test: outputs: dev: dbname: test_db @@ -391,9 +389,7 @@ def test_init_task_in_project_with_invalid_profile_template( with open(os.path.join(project.profiles_dir, "profiles.yml"), "r") as f: assert ( f.read() - == """config: - send_anonymous_usage_stats: false -test: + == """test: outputs: dev: dbname: test_db @@ -430,7 +426,6 @@ class TestInitOutsideOfProject(TestInitOutsideOfProjectBase): @pytest.fixture(scope="class") def dbt_profile_data(self, unique_schema): return { - "config": {"send_anonymous_usage_stats": False}, "test": { "outputs": { "default2": { @@ -513,9 +508,7 @@ def test_init_task_outside_of_project( with open(os.path.join(project.profiles_dir, "profiles.yml"), "r") as f: assert ( f.read() - == f"""config: - send_anonymous_usage_stats: false -{project_name}: + == f"""{project_name}: outputs: dev: dbname: test_db @@ -560,7 +553,6 @@ def test_init_task_outside_of_project( # name or the intended use of these models name: '{project_name}' version: '1.0.0' -config-version: 2 # This setting configures which "profile" dbt uses for this project. profile: '{project_name}' @@ -679,7 +671,6 @@ def test_init_provided_project_name_and_skip_profile_setup( # name or the intended use of these models name: '{project_name}' version: '1.0.0' -config-version: 2 # This setting configures which "profile" dbt uses for this project. profile: '{project_name}' @@ -766,7 +757,6 @@ def test_init_task_outside_of_project_with_specified_profile( # name or the intended use of these models name: '{project_name}' version: '1.0.0' -config-version: 2 # This setting configures which "profile" dbt uses for this project. profile: 'test'
tests/functional/materializations/conftest.py+24 −0 modified@@ -325,6 +325,21 @@ {%- endmaterialization -%} """ +custom_materialization_dep__dbt_project_yml = """ +name: custom_materialization_default +macro-paths: ['macros'] +""" + +custom_materialization_sql = """ +{% materialization custom_materialization, default %} + {%- set target_relation = this.incorporate(type='table') %} + {% call statement('main') -%} + select 1 as column1 + {%- endcall %} + {{ return({'relations': [target_relation]}) }} +{% endmaterialization %} +""" + @pytest.fixture(scope="class") def override_view_adapter_pass_dep(project_root): @@ -368,3 +383,12 @@ def override_view_return_no_relation(project_root): }, } write_project_files(project_root, "override-view-return-no-relation", files) + + +@pytest.fixture(scope="class") +def custom_materialization_dep(project_root): + files = { + "dbt_project.yml": custom_materialization_dep__dbt_project_yml, + "macros": {"custom_materialization.sql": custom_materialization_sql}, + } + write_project_files(project_root, "custom-materialization-dep", files)
tests/functional/materializations/test_custom_materialization.py+165 −3 modified@@ -1,7 +1,7 @@ import pytest from dbt.tests.util import run_dbt - +from dbt import deprecations models__model_sql = """ {{ config(materialized='view') }} @@ -10,34 +10,196 @@ """ +models_custom_materialization__model_sql = """ +{{ config(materialized='custom_materialization') }} +select 1 as id + +""" + + @pytest.fixture(scope="class") def models(): return {"model.sql": models__model_sql} +@pytest.fixture(scope="class") +def set_up_deprecations(): + deprecations.reset_deprecations() + assert deprecations.active_deprecations == set() + + class TestOverrideAdapterDependency: # make sure that if there's a dependency with an adapter-specific # materialization, we honor that materialization @pytest.fixture(scope="class") def packages(self): return {"packages": [{"local": "override-view-adapter-dep"}]} - def test_adapter_dependency(self, project, override_view_adapter_dep): + def test_adapter_dependency(self, project, override_view_adapter_dep, set_up_deprecations): + run_dbt(["deps"]) + # this should error because the override is buggy + run_dbt(["run"], expect_pass=False) + + # overriding a built-in materialization scoped to adapter from package is deprecated + assert deprecations.active_deprecations == {"package-materialization-override"} + + +class TestOverrideAdapterDependencyDeprecated: + # make sure that if there's a dependency with an adapter-specific + # materialization, we honor that materialization + @pytest.fixture(scope="class") + def packages(self): + return {"packages": [{"local": "override-view-adapter-dep"}]} + + @pytest.fixture(scope="class") + def project_config_update(self): + return { + "flags": { + "require_explicit_package_overrides_for_builtin_materializations": True, + }, + } + + def test_adapter_dependency_deprecate_overrides( + self, project, override_view_adapter_dep, set_up_deprecations + ): + run_dbt(["deps"]) + # this should pass because the override is buggy and unused + run_dbt(["run"]) + + # no deprecation warning -- flag used correctly + assert deprecations.active_deprecations == set() + + +class TestOverrideAdapterDependencyLegacy: + # make sure that if there's a dependency with an adapter-specific + # materialization, we honor that materialization + @pytest.fixture(scope="class") + def packages(self): + return {"packages": [{"local": "override-view-adapter-dep"}]} + + @pytest.fixture(scope="class") + def project_config_update(self): + return { + "flags": { + "require_explicit_package_overrides_for_builtin_materializations": False, + }, + } + + def test_adapter_dependency(self, project, override_view_adapter_dep, set_up_deprecations): run_dbt(["deps"]) # this should error because the override is buggy run_dbt(["run"], expect_pass=False) + # overriding a built-in materialization scoped to adapter from package is deprecated + assert deprecations.active_deprecations == {"package-materialization-override"} + class TestOverrideDefaultDependency: @pytest.fixture(scope="class") def packages(self): return {"packages": [{"local": "override-view-default-dep"}]} - def test_default_dependency(self, project, override_view_default_dep): + def test_default_dependency(self, project, override_view_default_dep, set_up_deprecations): + run_dbt(["deps"]) + # this should error because the override is buggy + run_dbt(["run"], expect_pass=False) + + # overriding a built-in materialization from package is deprecated + assert deprecations.active_deprecations == {"package-materialization-override"} + + +class TestOverrideDefaultDependencyDeprecated: + @pytest.fixture(scope="class") + def packages(self): + return {"packages": [{"local": "override-view-default-dep"}]} + + @pytest.fixture(scope="class") + def project_config_update(self): + return { + "flags": { + "require_explicit_package_overrides_for_builtin_materializations": True, + }, + } + + def test_default_dependency_deprecated( + self, project, override_view_default_dep, set_up_deprecations + ): + run_dbt(["deps"]) + # this should pass because the override is buggy and unused + run_dbt(["run"]) + + # overriding a built-in materialization from package is deprecated + assert deprecations.active_deprecations == set() + + +class TestOverrideDefaultDependencyLegacy: + @pytest.fixture(scope="class") + def packages(self): + return {"packages": [{"local": "override-view-default-dep"}]} + + @pytest.fixture(scope="class") + def project_config_update(self): + return { + "flags": { + "require_explicit_package_overrides_for_builtin_materializations": False, + }, + } + + def test_default_dependency(self, project, override_view_default_dep, set_up_deprecations): + run_dbt(["deps"]) + # this should error because the override is buggy + run_dbt(["run"], expect_pass=False) + + # overriding a built-in materialization from package is deprecated + assert deprecations.active_deprecations == {"package-materialization-override"} + + +root_view_override_macro = """ +{% materialization view, default %} + {{ return(view_default_override.materialization_view_default()) }} +{% endmaterialization %} +""" + + +class TestOverrideDefaultDependencyRootOverride: + @pytest.fixture(scope="class") + def packages(self): + return {"packages": [{"local": "override-view-default-dep"}]} + + @pytest.fixture(scope="class") + def macros(self): + return {"my_view.sql": root_view_override_macro} + + def test_default_dependency_with_root_override( + self, project, override_view_default_dep, set_up_deprecations + ): run_dbt(["deps"]) # this should error because the override is buggy run_dbt(["run"], expect_pass=False) + # using an package-overriden built-in materialization in a root matereialization is _not_ deprecated + assert deprecations.active_deprecations == set() + + +class TestCustomMaterializationDependency: + @pytest.fixture(scope="class") + def models(self): + return {"model.sql": models_custom_materialization__model_sql} + + @pytest.fixture(scope="class") + def packages(self): + return {"packages": [{"local": "custom-materialization-dep"}]} + + def test_custom_materialization_deopendency( + self, project, custom_materialization_dep, set_up_deprecations + ): + run_dbt(["deps"]) + # custom materilization is valid + run_dbt(["run"]) + + # using a custom materialization is from an installed package is _not_ deprecated + assert deprecations.active_deprecations == set() + class TestOverrideAdapterDependencyPassing: @pytest.fixture(scope="class")
tests/functional/metrics/test_metric_deferral.py+0 −1 modified@@ -23,7 +23,6 @@ def setup(self, project): @pytest.fixture(scope="class") def dbt_profile_data(self, unique_schema): return { - "config": {"send_anonymous_usage_stats": False}, "test": { "outputs": { "default": {
tests/functional/run_operations/test_run_operations.py+0 −1 modified@@ -28,7 +28,6 @@ def macros(self): @pytest.fixture(scope="class") def dbt_profile_data(self, unique_schema): return { - "config": {"send_anonymous_usage_stats": False}, "test": { "outputs": { "default": {
tests/unit/test_cli_flags.py+76 −31 modified@@ -9,7 +9,7 @@ from dbt.cli.flags import Flags from dbt.cli.main import cli from dbt.cli.types import Command -from dbt.contracts.project import UserConfig +from dbt.contracts.project import ProjectFlags from dbt.exceptions import DbtInternalError from dbt.helper_types import WarnErrorOptions from dbt.tests.util import rm_file, write_file @@ -27,8 +27,8 @@ def run_context(self) -> click.Context: return self.make_dbt_context("run", ["run"]) @pytest.fixture - def user_config(self) -> UserConfig: - return UserConfig() + def project_flags(self) -> ProjectFlags: + return ProjectFlags() def test_which(self, run_context): flags = Flags(run_context) @@ -110,35 +110,35 @@ def test_anonymous_usage_state( flags = Flags(run_context) assert flags.SEND_ANONYMOUS_USAGE_STATS == expected_anonymous_usage_stats - def test_empty_user_config_uses_default(self, run_context, user_config): - flags = Flags(run_context, user_config) + def test_empty_project_flags_uses_default(self, run_context, project_flags): + flags = Flags(run_context, project_flags) assert flags.USE_COLORS == run_context.params["use_colors"] - def test_none_user_config_uses_default(self, run_context): + def test_none_project_flags_uses_default(self, run_context): flags = Flags(run_context, None) assert flags.USE_COLORS == run_context.params["use_colors"] - def test_prefer_user_config_to_default(self, run_context, user_config): - user_config.use_colors = False + def test_prefer_project_flags_to_default(self, run_context, project_flags): + project_flags.use_colors = False # ensure default value is not the same as user config - assert run_context.params["use_colors"] is not user_config.use_colors + assert run_context.params["use_colors"] is not project_flags.use_colors - flags = Flags(run_context, user_config) - assert flags.USE_COLORS == user_config.use_colors + flags = Flags(run_context, project_flags) + assert flags.USE_COLORS == project_flags.use_colors - def test_prefer_param_value_to_user_config(self): - user_config = UserConfig(use_colors=False) + def test_prefer_param_value_to_project_flags(self): + project_flags = ProjectFlags(use_colors=False) context = self.make_dbt_context("run", ["--use-colors", "True", "run"]) - flags = Flags(context, user_config) + flags = Flags(context, project_flags) assert flags.USE_COLORS - def test_prefer_env_to_user_config(self, monkeypatch, user_config): - user_config.use_colors = False + def test_prefer_env_to_project_flags(self, monkeypatch, project_flags): + project_flags.use_colors = False monkeypatch.setenv("DBT_USE_COLORS", "True") context = self.make_dbt_context("run", ["run"]) - flags = Flags(context, user_config) + flags = Flags(context, project_flags) assert flags.USE_COLORS def test_mutually_exclusive_options_passed_separately(self): @@ -163,14 +163,14 @@ def test_mutually_exclusive_options_from_cli(self): Flags(context) @pytest.mark.parametrize("warn_error", [True, False]) - def test_mutually_exclusive_options_from_user_config(self, warn_error, user_config): - user_config.warn_error = warn_error + def test_mutually_exclusive_options_from_project_flags(self, warn_error, project_flags): + project_flags.warn_error = warn_error context = self.make_dbt_context( "run", ["--warn-error-options", '{"include": "all"}', "run"] ) with pytest.raises(DbtUsageException): - Flags(context, user_config) + Flags(context, project_flags) @pytest.mark.parametrize("warn_error", ["True", "False"]) def test_mutually_exclusive_options_from_envvar(self, warn_error, monkeypatch): @@ -182,14 +182,16 @@ def test_mutually_exclusive_options_from_envvar(self, warn_error, monkeypatch): Flags(context) @pytest.mark.parametrize("warn_error", [True, False]) - def test_mutually_exclusive_options_from_cli_and_user_config(self, warn_error, user_config): - user_config.warn_error = warn_error + def test_mutually_exclusive_options_from_cli_and_project_flags( + self, warn_error, project_flags + ): + project_flags.warn_error = warn_error context = self.make_dbt_context( "run", ["--warn-error-options", '{"include": "all"}', "run"] ) with pytest.raises(DbtUsageException): - Flags(context, user_config) + Flags(context, project_flags) @pytest.mark.parametrize("warn_error", ["True", "False"]) def test_mutually_exclusive_options_from_cli_and_envvar(self, warn_error, monkeypatch): @@ -202,15 +204,15 @@ def test_mutually_exclusive_options_from_cli_and_envvar(self, warn_error, monkey Flags(context) @pytest.mark.parametrize("warn_error", ["True", "False"]) - def test_mutually_exclusive_options_from_user_config_and_envvar( - self, user_config, warn_error, monkeypatch + def test_mutually_exclusive_options_from_project_flags_and_envvar( + self, project_flags, warn_error, monkeypatch ): - user_config.warn_error = warn_error + project_flags.warn_error = warn_error monkeypatch.setenv("DBT_WARN_ERROR_OPTIONS", '{"include": "all"}') context = self.make_dbt_context("run", ["run"]) with pytest.raises(DbtUsageException): - Flags(context, user_config) + Flags(context, project_flags) @pytest.mark.parametrize( "cli_colors,cli_colors_file,flag_colors,flag_colors_file", @@ -319,10 +321,10 @@ def test_log_format_interaction( assert flags.LOG_FORMAT_FILE == flag_log_format_file def test_log_settings_from_config(self): - """Test that values set in UserConfig for log settings will set flags as expected""" + """Test that values set in ProjectFlags for log settings will set flags as expected""" context = self.make_dbt_context("run", ["run"]) - config = UserConfig(log_format="json", log_level="warn", use_colors=False) + config = ProjectFlags(log_format="json", log_level="warn", use_colors=False) flags = Flags(context, config) @@ -334,11 +336,11 @@ def test_log_settings_from_config(self): assert flags.USE_COLORS_FILE is False def test_log_file_settings_from_config(self): - """Test that values set in UserConfig for log *file* settings will set flags as expected, leaving the console + """Test that values set in ProjectFlags for log *file* settings will set flags as expected, leaving the console logging flags with their default values""" context = self.make_dbt_context("run", ["run"]) - config = UserConfig(log_format_file="json", log_level_file="warn", use_colors_file=False) + config = ProjectFlags(log_format_file="json", log_level_file="warn", use_colors_file=False) flags = Flags(context, config) @@ -369,6 +371,14 @@ def test_global_flag_at_child_context(self): assert flags_a.USE_COLORS == flags_b.USE_COLORS + def test_set_project_only_flags(self, project_flags, run_context): + flags = Flags(run_context, project_flags) + + for project_only_flag, project_only_flag_value in project_flags.project_only_flags.items(): + assert getattr(flags, project_only_flag) == project_only_flag_value + # sanity check: ensure project_only_flag is not part of the click context + assert project_only_flag not in run_context.params + def _create_flags_from_dict(self, cmd, d): write_file("", "profiles.yml") result = Flags.from_dict(cmd, d) @@ -409,3 +419,38 @@ def test_from_dict_0_value(self): args_dict = {"log_file_max_bytes": 0} flags = Flags.from_dict(Command.RUN, args_dict) assert flags.LOG_FILE_MAX_BYTES == 0 + + +def test_project_flag_defaults(): + flags = ProjectFlags() + # From # 9183: Let's add a unit test that ensures that: + # every attribute of ProjectFlags that has a corresponding click option + # in params.py should be set to None by default (except for anon user + # tracking). Going forward, flags can have non-None defaults if they + # do not have a corresponding CLI option/env var. These will be used + # to control backwards incompatible interface or behaviour changes. + + # List of all flags except send_anonymous_usage_stats + project_flags = [ + "cache_selected_only", + "debug", + "fail_fast", + "indirect_selection", + "log_format", + "log_format_file", + "log_level", + "log_level_file", + "partial_parse", + "populate_cache", + "printer_width", + "static_parser", + "use_colors", + "use_colors_file", + "use_experimental_parser", + "version_check", + "warn_error", + "warn_error_options", + "write_json", + ] + for flag in project_flags: + assert getattr(flags, flag) is None
tests/unit/test_config.py+98 −131 modified@@ -119,12 +119,17 @@ class BaseConfigTest(unittest.TestCase): """ def setUp(self): + # Write project + self.project_dir = normalize(tempfile.mkdtemp()) self.default_project_data = { "version": "0.0.1", "name": "my_test_project", "profile": "default", - "config-version": 2, } + self.write_project(self.default_project_data) + + # Write profile + self.profiles_dir = normalize(tempfile.mkdtemp()) self.default_profile_data = { "default": { "outputs": { @@ -176,6 +181,8 @@ def setUp(self): }, "empty_profile_data": {}, } + self.write_profile(self.default_profile_data) + self.args = Namespace( profiles_dir=self.profiles_dir, cli_vars={}, @@ -203,13 +210,6 @@ def assertRaisesOrReturns(self, exc): else: return self.assertRaises(exc) - -class BaseFileTest(BaseConfigTest): - def setUp(self): - self.project_dir = normalize(tempfile.mkdtemp()) - self.profiles_dir = normalize(tempfile.mkdtemp()) - super().setUp() - def tearDown(self): try: shutil.rmtree(self.project_dir) @@ -248,11 +248,6 @@ def write_empty_profile(self): class TestProfile(BaseConfigTest): - def setUp(self): - self.profiles_dir = "/invalid-path" - self.project_dir = "/invalid-project-path" - super().setUp() - def from_raw_profiles(self): renderer = empty_profile_renderer() return dbt.config.Profile.from_raw_profiles(self.default_profile_data, "default", renderer) @@ -262,8 +257,6 @@ def test_from_raw_profiles(self): self.assertEqual(profile.profile_name, "default") self.assertEqual(profile.target_name, "postgres") self.assertEqual(profile.threads, 7) - self.assertTrue(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) self.assertTrue(isinstance(profile.credentials, PostgresCredentials)) self.assertEqual(profile.credentials.type, "postgres") self.assertEqual(profile.credentials.host, "postgres-db-hostname") @@ -273,29 +266,6 @@ def test_from_raw_profiles(self): self.assertEqual(profile.credentials.schema, "postgres-schema") self.assertEqual(profile.credentials.database, "postgres-db-name") - def test_config_override(self): - self.default_profile_data["config"] = { - "send_anonymous_usage_stats": False, - "use_colors": False, - } - profile = self.from_raw_profiles() - self.assertEqual(profile.profile_name, "default") - self.assertEqual(profile.target_name, "postgres") - self.assertFalse(profile.user_config.send_anonymous_usage_stats) - self.assertFalse(profile.user_config.use_colors) - - def test_partial_config_override(self): - self.default_profile_data["config"] = { - "send_anonymous_usage_stats": False, - "printer_width": 60, - } - profile = self.from_raw_profiles() - self.assertEqual(profile.profile_name, "default") - self.assertEqual(profile.target_name, "postgres") - self.assertFalse(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) - self.assertEqual(profile.user_config.printer_width, 60) - def test_missing_type(self): del self.default_profile_data["default"]["outputs"]["postgres"]["type"] with self.assertRaises(dbt.exceptions.DbtProfileError) as exc: @@ -337,7 +307,7 @@ def test_extra_path(self): } ) with self.assertRaises(dbt.exceptions.DbtProjectError) as exc: - project_from_config_norender(self.default_project_data) + project_from_config_norender(self.default_project_data, project_root=self.project_dir) self.assertIn("source-paths and model-paths", str(exc.exception)) self.assertIn("cannot both be defined.", str(exc.exception)) @@ -403,11 +373,7 @@ def test_invalid_env_vars(self): self.assertIn("Could not convert value 'hello' into type 'number'", str(exc.exception)) -class TestProfileFile(BaseFileTest): - def setUp(self): - super().setUp() - self.write_profile(self.default_profile_data) - +class TestProfileFile(BaseConfigTest): def from_raw_profile_info(self, raw_profile=None, profile_name="default", **kwargs): if raw_profile is None: raw_profile = self.default_profile_data["default"] @@ -438,8 +404,6 @@ def test_profile_simple(self): self.assertEqual(profile.profile_name, "default") self.assertEqual(profile.target_name, "postgres") self.assertEqual(profile.threads, 7) - self.assertTrue(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) self.assertTrue(isinstance(profile.credentials, PostgresCredentials)) self.assertEqual(profile.credentials.type, "postgres") self.assertEqual(profile.credentials.host, "postgres-db-hostname") @@ -464,8 +428,6 @@ def test_profile_override(self): self.assertEqual(profile.profile_name, "other") self.assertEqual(profile.target_name, "other-postgres") self.assertEqual(profile.threads, 3) - self.assertTrue(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) self.assertTrue(isinstance(profile.credentials, PostgresCredentials)) self.assertEqual(profile.credentials.type, "postgres") self.assertEqual(profile.credentials.host, "other-postgres-db-hostname") @@ -485,8 +447,6 @@ def test_env_vars(self): self.assertEqual(profile.profile_name, "default") self.assertEqual(profile.target_name, "with-vars") self.assertEqual(profile.threads, 1) - self.assertTrue(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) self.assertEqual(profile.credentials.type, "postgres") self.assertEqual(profile.credentials.host, "env-postgres-host") self.assertEqual(profile.credentials.port, 6543) @@ -505,8 +465,6 @@ def test_env_vars_env_target(self): self.assertEqual(profile.profile_name, "default") self.assertEqual(profile.target_name, "with-vars") self.assertEqual(profile.threads, 1) - self.assertTrue(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) self.assertEqual(profile.credentials.type, "postgres") self.assertEqual(profile.credentials.host, "env-postgres-host") self.assertEqual(profile.credentials.port, 6543) @@ -537,8 +495,6 @@ def test_cli_and_env_vars(self): self.assertEqual(profile.profile_name, "default") self.assertEqual(profile.target_name, "cli-and-env-vars") self.assertEqual(profile.threads, 1) - self.assertTrue(profile.user_config.send_anonymous_usage_stats) - self.assertIsNone(profile.user_config.use_colors) self.assertEqual(profile.credentials.type, "postgres") self.assertEqual(profile.credentials.host, "cli-postgres-host") self.assertEqual(profile.credentials.port, 6543) @@ -567,18 +523,19 @@ def test_profile_with_empty_profile_data(self): def project_from_config_norender( - cfg, packages=None, path="/invalid-root-path", verify_version=False + cfg, packages=None, project_root="/invalid-root-path", verify_version=False ): if packages is None: packages = {} partial = dbt.config.project.PartialProject.from_dicts( - path, + project_root, project_dict=cfg, packages_dict=packages, selectors_dict={}, verify_version=verify_version, ) - # no rendering + # no rendering ... Why? + partial.project_dict["project-root"] = project_root rendered = dbt.config.project.RenderComponents( project_dict=partial.project_dict, packages_dict=partial.packages_dict, @@ -590,14 +547,14 @@ def project_from_config_norender( def project_from_config_rendered( cfg, packages=None, - path="/invalid-root-path", + project_root="/invalid-root-path", verify_version=False, packages_specified_path=PACKAGES_FILE_NAME, ): if packages is None: packages = {} partial = dbt.config.project.PartialProject.from_dicts( - path, + project_root, project_dict=cfg, packages_dict=packages, selectors_dict={}, @@ -608,18 +565,14 @@ def project_from_config_rendered( class TestProject(BaseConfigTest): - def setUp(self): - self.profiles_dir = "/invalid-profiles-path" - self.project_dir = "/invalid-root-path" - super().setUp() - self.default_project_data["project-root"] = self.project_dir - def test_defaults(self): - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.project_name, "my_test_project") self.assertEqual(project.version, "0.0.1") self.assertEqual(project.profile_name, "default") - self.assertEqual(project.project_root, "/invalid-root-path") + self.assertEqual(project.project_root, self.project_dir) self.assertEqual(project.model_paths, ["models"]) self.assertEqual(project.macro_paths, ["macros"]) self.assertEqual(project.seed_paths, ["seeds"]) @@ -645,30 +598,38 @@ def test_defaults(self): str(project) def test_eq(self): - project = project_from_config_norender(self.default_project_data) - other = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) + other = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project, other) def test_neq(self): - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertNotEqual(project, object()) def test_implicit_overrides(self): self.default_project_data.update( { "model-paths": ["other-models"], - "target-path": "other-target", } ) - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual( set(project.docs_paths), set(["other-models", "seeds", "snapshots", "analyses", "macros"]), ) - self.assertEqual(project.clean_targets, ["other-target"]) def test_hashed_name(self): - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.hashed_name(), "754cd47eac1d6f50a5f7cd399ec43da4") def test_all_overrides(self): @@ -682,7 +643,6 @@ def test_all_overrides(self): "analysis-paths": ["other-analyses"], "docs-paths": ["docs"], "asset-paths": ["other-assets"], - "target-path": "other-target", "clean-targets": ["another-target"], "packages-install-path": "other-dbt_packages", "quoting": {"identifier": False}, @@ -731,19 +691,19 @@ def test_all_overrides(self): {"git": "git@example.com:dbt-labs/dbt-utils.git", "revision": "test-rev"}, ], } - project = project_from_config_norender(self.default_project_data, packages=packages) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir, packages=packages + ) self.assertEqual(project.project_name, "my_test_project") self.assertEqual(project.version, "0.0.1") self.assertEqual(project.profile_name, "default") - self.assertEqual(project.project_root, "/invalid-root-path") self.assertEqual(project.model_paths, ["other-models"]) self.assertEqual(project.macro_paths, ["other-macros"]) self.assertEqual(project.seed_paths, ["other-seeds"]) self.assertEqual(project.test_paths, ["other-tests"]) self.assertEqual(project.analysis_paths, ["other-analyses"]) self.assertEqual(project.docs_paths, ["docs"]) self.assertEqual(project.asset_paths, ["other-assets"]) - self.assertEqual(project.target_path, "other-target") self.assertEqual(project.clean_targets, ["another-target"]) self.assertEqual(project.packages_install_path, "other-dbt_packages") self.assertEqual(project.quoting, {"identifier": False}) @@ -822,11 +782,12 @@ def test_string_run_hooks(self): def test_invalid_project_name(self): self.default_project_data["name"] = "invalid-project-name" with self.assertRaises(dbt.exceptions.DbtProjectError) as exc: - project_from_config_norender(self.default_project_data) + project_from_config_norender(self.default_project_data, project_root=self.project_dir) self.assertIn("invalid-project-name", str(exc.exception)) def test_no_project(self): + os.remove(os.path.join(self.project_dir, "dbt_project.yml")) renderer = empty_project_renderer() with self.assertRaises(dbt.exceptions.DbtProjectError) as exc: dbt.config.Project.from_project_root(self.project_dir, renderer) @@ -836,12 +797,12 @@ def test_no_project(self): def test_invalid_version(self): self.default_project_data["require-dbt-version"] = "hello!" with self.assertRaises(dbt.exceptions.DbtProjectError): - project_from_config_norender(self.default_project_data) + project_from_config_norender(self.default_project_data, project_root=self.project_dir) def test_unsupported_version(self): self.default_project_data["require-dbt-version"] = ">99999.0.0" # allowed, because the RuntimeConfig checks, not the Project itself - project_from_config_norender(self.default_project_data) + project_from_config_norender(self.default_project_data, project_root=self.project_dir) def test_none_values(self): self.default_project_data.update( @@ -891,7 +852,9 @@ def test_query_comment_disabled(self): "query-comment": None, } ) - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.query_comment.comment, "") self.assertEqual(project.query_comment.append, False) @@ -900,12 +863,16 @@ def test_query_comment_disabled(self): "query-comment": "", } ) - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.query_comment.comment, "") self.assertEqual(project.query_comment.append, False) def test_default_query_comment(self): - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.query_comment, QueryComment()) def test_default_query_comment_append(self): @@ -914,7 +881,9 @@ def test_default_query_comment_append(self): "query-comment": {"append": True}, } ) - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.query_comment.comment, DEFAULT_QUERY_COMMENT) self.assertEqual(project.query_comment.append, True) @@ -924,7 +893,9 @@ def test_custom_query_comment_append(self): "query-comment": {"comment": "run by user test", "append": True}, } ) - project = project_from_config_norender(self.default_project_data) + project = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project.query_comment.comment, "run by user test") self.assertEqual(project.query_comment.append, True) @@ -946,17 +917,13 @@ def test_packages_from_dependencies(self): assert git_package.git == "{{ env_var('some_package') }}" -class TestProjectFile(BaseFileTest): - def setUp(self): - super().setUp() - self.write_project(self.default_project_data) - # and after the fact, add the project root - self.default_project_data["project-root"] = self.project_dir - +class TestProjectFile(BaseConfigTest): def test_from_project_root(self): renderer = empty_project_renderer() project = dbt.config.Project.from_project_root(self.project_dir, renderer) - from_config = project_from_config_norender(self.default_project_data) + from_config = project_from_config_norender( + self.default_project_data, project_root=self.project_dir + ) self.assertEqual(project, from_config) self.assertEqual(project.version, "0.0.1") self.assertEqual(project.project_name, "my_test_project") @@ -973,12 +940,7 @@ def run(self): pass -class TestConfiguredTask(BaseFileTest): - def setUp(self): - super().setUp() - self.write_project(self.default_project_data) - self.write_profile(self.default_profile_data) - +class TestConfiguredTask(BaseConfigTest): def tearDown(self): super().tearDown() # These tests will change the directory to the project path, @@ -997,15 +959,13 @@ def test_configured_task_dir_change_with_bad_path(self): InheritsFromConfiguredTask.from_args(self.args) -class TestVariableProjectFile(BaseFileTest): +class TestVariableProjectFile(BaseConfigTest): def setUp(self): super().setUp() self.default_project_data["version"] = "{{ var('cli_version') }}" self.default_project_data["name"] = "blah" self.default_project_data["profile"] = "{{ env_var('env_value_profile') }}" self.write_project(self.default_project_data) - # and after the fact, add the project root - self.default_project_data["project-root"] = self.project_dir def test_cli_and_env_vars(self): renderer = dbt.config.renderer.DbtProjectYamlRenderer(None, {"cli_version": "0.1.2"}) @@ -1022,15 +982,11 @@ def test_cli_and_env_vars(self): class TestRuntimeConfig(BaseConfigTest): - def setUp(self): - self.profiles_dir = "/invalid-profiles-path" - self.project_dir = "/invalid-root-path" - super().setUp() - self.default_project_data["project-root"] = self.project_dir - def get_project(self): return project_from_config_norender( - self.default_project_data, verify_version=self.args.version_check + self.default_project_data, + project_root=self.project_dir, + verify_version=self.args.version_check, ) def get_profile(self): @@ -1079,14 +1035,6 @@ def test_str(self): # to make sure nothing terrible happens str(config) - def test_validate_fails(self): - project = self.get_project() - profile = self.get_profile() - # invalid - must be boolean - profile.user_config.use_colors = 100 - with self.assertRaises(dbt.exceptions.DbtProjectError): - dbt.config.RuntimeConfig.from_parts(project, profile, {}) - def test_supported_version(self): self.default_project_data["require-dbt-version"] = ">0.0.0" conf = self.from_parts() @@ -1210,7 +1158,9 @@ def setUp(self): } def get_project(self): - return project_from_config_norender(self.default_project_data, verify_version=True) + return project_from_config_norender( + self.default_project_data, project_root=self.project_dir, verify_version=True + ) def get_profile(self): renderer = empty_profile_renderer() @@ -1242,14 +1192,7 @@ def test__warn_for_unused_resource_config_paths(self): assert expected_msg in msg -class TestRuntimeConfigFiles(BaseFileTest): - def setUp(self): - super().setUp() - self.write_profile(self.default_profile_data) - self.write_project(self.default_project_data) - # and after the fact, add the project root - self.default_project_data["project-root"] = self.project_dir - +class TestRuntimeConfigFiles(BaseConfigTest): def test_from_args(self): with temp_cd(self.project_dir): config = dbt.config.RuntimeConfig.from_args(self.args) @@ -1279,7 +1222,7 @@ def test_from_args(self): self.assertEqual(config.project_name, "my_test_project") -class TestVariableRuntimeConfigFiles(BaseFileTest): +class TestVariableRuntimeConfigFiles(BaseConfigTest): def setUp(self): super().setUp() self.default_project_data.update( @@ -1311,9 +1254,6 @@ def setUp(self): } ) self.write_project(self.default_project_data) - self.write_profile(self.default_profile_data) - # and after the fact, add the project root - self.default_project_data["project-root"] = self.project_dir def test_cli_and_env_vars(self): self.args.target = "cli-and-env-vars" @@ -1387,3 +1327,30 @@ def test_lookups(self): for node, key, expected_value in expected: value = vars_provider.vars_for(node, "postgres").get(key) assert value == expected_value + + +class TestMultipleProjectFlags(BaseConfigTest): + def setUp(self): + super().setUp() + + self.default_project_data.update( + { + "flags": { + "send_anonymous_usage_data": False, + } + } + ) + self.write_project(self.default_project_data) + + self.default_profile_data.update( + { + "config": { + "send_anonymous_usage_data": False, + } + } + ) + self.write_profile(self.default_profile_data) + + def test_setting_multiple_flags(self): + with pytest.raises(dbt.exceptions.DbtProjectError): + set_from_args(self.args, None)
tests/unit/test_events.py+4 −0 modified@@ -145,6 +145,10 @@ def test_event_codes(self): types.ConfigLogPathDeprecation(deprecated_path=""), types.ConfigTargetPathDeprecation(deprecated_path=""), types.CollectFreshnessReturnSignature(), + types.ProjectFlagsMovedDeprecation(), + types.PackageMaterializationOverrideDeprecation( + package_name="my_package", materialization_name="view" + ), # E - DB Adapter ====================== types.AdapterEventDebug(), types.AdapterEventInfo(),
tests/unit/test_flags.py+0 −340 removed@@ -1,340 +0,0 @@ -import os -from unittest import TestCase -from argparse import Namespace -import pytest - -from dbt import flags -from dbt.contracts.project import UserConfig -from dbt.graph.selector_spec import IndirectSelection -from dbt.helper_types import WarnErrorOptions - -# Skip due to interface for flag updated -pytestmark = pytest.mark.skip - - -class TestFlags(TestCase): - def setUp(self): - self.args = Namespace() - self.user_config = UserConfig() - - def test__flags(self): - - # use_experimental_parser - self.user_config.use_experimental_parser = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.USE_EXPERIMENTAL_PARSER, True) - os.environ["DBT_USE_EXPERIMENTAL_PARSER"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.USE_EXPERIMENTAL_PARSER, False) - setattr(self.args, "use_experimental_parser", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.USE_EXPERIMENTAL_PARSER, True) - # cleanup - os.environ.pop("DBT_USE_EXPERIMENTAL_PARSER") - delattr(self.args, "use_experimental_parser") - flags.USE_EXPERIMENTAL_PARSER = False - self.user_config.use_experimental_parser = None - - # static_parser - self.user_config.static_parser = False - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.STATIC_PARSER, False) - os.environ["DBT_STATIC_PARSER"] = "true" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.STATIC_PARSER, True) - setattr(self.args, "static_parser", False) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.STATIC_PARSER, False) - # cleanup - os.environ.pop("DBT_STATIC_PARSER") - delattr(self.args, "static_parser") - flags.STATIC_PARSER = True - self.user_config.static_parser = None - - # warn_error - self.user_config.warn_error = False - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WARN_ERROR, False) - os.environ["DBT_WARN_ERROR"] = "true" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WARN_ERROR, True) - setattr(self.args, "warn_error", False) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WARN_ERROR, False) - # cleanup - os.environ.pop("DBT_WARN_ERROR") - delattr(self.args, "warn_error") - flags.WARN_ERROR = False - self.user_config.warn_error = None - - # warn_error_options - self.user_config.warn_error_options = '{"include": "all"}' - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WARN_ERROR_OPTIONS, WarnErrorOptions(include="all")) - os.environ["DBT_WARN_ERROR_OPTIONS"] = '{"include": []}' - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WARN_ERROR_OPTIONS, WarnErrorOptions(include=[])) - setattr(self.args, "warn_error_options", '{"include": "all"}') - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WARN_ERROR_OPTIONS, WarnErrorOptions(include="all")) - # cleanup - os.environ.pop("DBT_WARN_ERROR_OPTIONS") - delattr(self.args, "warn_error_options") - self.user_config.warn_error_options = None - - # write_json - self.user_config.write_json = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WRITE_JSON, True) - os.environ["DBT_WRITE_JSON"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WRITE_JSON, False) - setattr(self.args, "write_json", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.WRITE_JSON, True) - # cleanup - os.environ.pop("DBT_WRITE_JSON") - delattr(self.args, "write_json") - - # partial_parse - self.user_config.partial_parse = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.PARTIAL_PARSE, True) - os.environ["DBT_PARTIAL_PARSE"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.PARTIAL_PARSE, False) - setattr(self.args, "partial_parse", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.PARTIAL_PARSE, True) - # cleanup - os.environ.pop("DBT_PARTIAL_PARSE") - delattr(self.args, "partial_parse") - self.user_config.partial_parse = False - - # use_colors - self.user_config.use_colors = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.USE_COLORS, True) - os.environ["DBT_USE_COLORS"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.USE_COLORS, False) - setattr(self.args, "use_colors", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.USE_COLORS, True) - # cleanup - os.environ.pop("DBT_USE_COLORS") - delattr(self.args, "use_colors") - - # debug - self.user_config.debug = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.DEBUG, True) - os.environ["DBT_DEBUG"] = "True" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.DEBUG, True) - os.environ["DBT_DEBUG"] = "False" - setattr(self.args, "debug", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.DEBUG, True) - # cleanup - os.environ.pop("DBT_DEBUG") - delattr(self.args, "debug") - self.user_config.debug = None - - # log_format -- text, json, default - self.user_config.log_format = "text" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.LOG_FORMAT, "text") - os.environ["DBT_LOG_FORMAT"] = "json" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.LOG_FORMAT, "json") - setattr(self.args, "log_format", "text") - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.LOG_FORMAT, "text") - # cleanup - os.environ.pop("DBT_LOG_FORMAT") - delattr(self.args, "log_format") - self.user_config.log_format = None - - # version_check - self.user_config.version_check = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.VERSION_CHECK, True) - os.environ["DBT_VERSION_CHECK"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.VERSION_CHECK, False) - setattr(self.args, "version_check", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.VERSION_CHECK, True) - # cleanup - os.environ.pop("DBT_VERSION_CHECK") - delattr(self.args, "version_check") - - # fail_fast - self.user_config.fail_fast = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.FAIL_FAST, True) - os.environ["DBT_FAIL_FAST"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.FAIL_FAST, False) - setattr(self.args, "fail_fast", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.FAIL_FAST, True) - # cleanup - os.environ.pop("DBT_FAIL_FAST") - delattr(self.args, "fail_fast") - self.user_config.fail_fast = False - - # send_anonymous_usage_stats - self.user_config.send_anonymous_usage_stats = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.SEND_ANONYMOUS_USAGE_STATS, True) - os.environ["DBT_SEND_ANONYMOUS_USAGE_STATS"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.SEND_ANONYMOUS_USAGE_STATS, False) - setattr(self.args, "send_anonymous_usage_stats", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.SEND_ANONYMOUS_USAGE_STATS, True) - os.environ["DO_NOT_TRACK"] = "1" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.SEND_ANONYMOUS_USAGE_STATS, False) - # cleanup - os.environ.pop("DBT_SEND_ANONYMOUS_USAGE_STATS") - os.environ.pop("DO_NOT_TRACK") - delattr(self.args, "send_anonymous_usage_stats") - - # printer_width - self.user_config.printer_width = 100 - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.PRINTER_WIDTH, 100) - os.environ["DBT_PRINTER_WIDTH"] = "80" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.PRINTER_WIDTH, 80) - setattr(self.args, "printer_width", "120") - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.PRINTER_WIDTH, 120) - # cleanup - os.environ.pop("DBT_PRINTER_WIDTH") - delattr(self.args, "printer_width") - self.user_config.printer_width = None - - # indirect_selection - self.user_config.indirect_selection = "eager" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.INDIRECT_SELECTION, IndirectSelection.Eager) - self.user_config.indirect_selection = "cautious" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.INDIRECT_SELECTION, IndirectSelection.Cautious) - self.user_config.indirect_selection = "buildable" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.INDIRECT_SELECTION, IndirectSelection.Buildable) - self.user_config.indirect_selection = None - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.INDIRECT_SELECTION, IndirectSelection.Eager) - os.environ["DBT_INDIRECT_SELECTION"] = "cautious" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.INDIRECT_SELECTION, IndirectSelection.Cautious) - setattr(self.args, "indirect_selection", "cautious") - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.INDIRECT_SELECTION, IndirectSelection.Cautious) - # cleanup - os.environ.pop("DBT_INDIRECT_SELECTION") - delattr(self.args, "indirect_selection") - self.user_config.indirect_selection = None - - # quiet - self.user_config.quiet = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.QUIET, True) - # cleanup - self.user_config.quiet = None - - # no_print - self.user_config.no_print = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.NO_PRINT, True) - # cleanup - self.user_config.no_print = None - - # cache_selected_only - self.user_config.cache_selected_only = True - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.CACHE_SELECTED_ONLY, True) - os.environ["DBT_CACHE_SELECTED_ONLY"] = "false" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.CACHE_SELECTED_ONLY, False) - setattr(self.args, "cache_selected_only", True) - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.CACHE_SELECTED_ONLY, True) - # cleanup - os.environ.pop("DBT_CACHE_SELECTED_ONLY") - delattr(self.args, "cache_selected_only") - self.user_config.cache_selected_only = False - - # target_path/log_path - flags.set_from_args(self.args, self.user_config) - self.assertIsNone(flags.LOG_PATH) - os.environ["DBT_LOG_PATH"] = "a/b/c" - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.LOG_PATH, "a/b/c") - setattr(self.args, "log_path", "d/e/f") - flags.set_from_args(self.args, self.user_config) - self.assertEqual(flags.LOG_PATH, "d/e/f") - # cleanup - os.environ.pop("DBT_LOG_PATH") - delattr(self.args, "log_path") - - def test__flags_are_mutually_exclusive(self): - # options from user config - self.user_config.warn_error = False - self.user_config.warn_error_options = '{"include":"all"}' - with pytest.raises(ValueError): - flags.set_from_args(self.args, self.user_config) - # cleanup - self.user_config.warn_error = None - self.user_config.warn_error_options = None - - # options from args - setattr(self.args, "warn_error", False) - setattr(self.args, "warn_error_options", '{"include":"all"}') - with pytest.raises(ValueError): - flags.set_from_args(self.args, self.user_config) - # cleanup - delattr(self.args, "warn_error") - delattr(self.args, "warn_error_options") - - # options from environment - os.environ["DBT_WARN_ERROR"] = "false" - os.environ["DBT_WARN_ERROR_OPTIONS"] = '{"include": []}' - with pytest.raises(ValueError): - flags.set_from_args(self.args, self.user_config) - # cleanup - os.environ.pop("DBT_WARN_ERROR") - os.environ.pop("DBT_WARN_ERROR_OPTIONS") - - # options from user config + args - self.user_config.warn_error = False - setattr(self.args, "warn_error_options", '{"include":"all"}') - with pytest.raises(ValueError): - flags.set_from_args(self.args, self.user_config) - # cleanup - self.user_config.warn_error = None - delattr(self.args, "warn_error_options") - - # options from user config + environ - self.user_config.warn_error = False - os.environ["DBT_WARN_ERROR_OPTIONS"] = '{"include": []}' - with pytest.raises(ValueError): - flags.set_from_args(self.args, self.user_config) - # cleanup - self.user_config.warn_error = None - os.environ.pop("DBT_WARN_ERROR_OPTIONS") - - # options from args + environ - setattr(self.args, "warn_error", False) - os.environ["DBT_WARN_ERROR_OPTIONS"] = '{"include": []}' - with pytest.raises(ValueError): - flags.set_from_args(self.args, self.user_config) - # cleanup - delattr(self.args, "warn_error") - os.environ.pop("DBT_WARN_ERROR_OPTIONS")
tests/unit/test_graph.py+2 −1 modified@@ -16,6 +16,7 @@ import dbt.parser.manifest from dbt import tracking from dbt.contracts.files import SourceFile, FileHash, FilePath +from dbt.contracts.project import ProjectFlags from dbt.contracts.graph.manifest import MacroManifest, ManifestStateCheck from dbt.graph import NodeSelector, parse_difference from dbt.events.functions import setup_event_logger @@ -130,7 +131,7 @@ def get_config(self, extra_cfg=None): cfg.update(extra_cfg) config = config_from_parts_or_dicts(project=cfg, profile=self.profile) - dbt.flags.set_from_args(Namespace(), config) + dbt.flags.set_from_args(Namespace(), ProjectFlags()) setup_event_logger(dbt.flags.get_flags()) object.__setattr__(dbt.flags.get_flags(), "PARTIAL_PARSE", False) return config
tests/unit/test_graph_selection.py+2 −2 modified@@ -13,9 +13,9 @@ from dbt import flags from argparse import Namespace -from dbt.contracts.project import UserConfig +from dbt.contracts.project import ProjectFlags -flags.set_from_args(Namespace(), UserConfig()) +flags.set_from_args(Namespace(), ProjectFlags()) def _get_graph():
tests/unit/test_manifest.py+176 −1 modified@@ -1239,7 +1239,7 @@ def test_find_generate_macros_by_name(macros, expectations): FindMaterializationSpec = namedtuple("FindMaterializationSpec", "macros,adapter_type,expected") -def _materialization_parameter_sets(): +def _materialization_parameter_sets_legacy(): # inject the plugins used for materialization parameter tests with mock.patch("dbt.adapters.base.plugin.project_name_from_path") as get_name: get_name.return_value = "foo" @@ -1386,12 +1386,187 @@ def id_mat(arg): return "_".join(arg) +@pytest.mark.parametrize( + "macros,adapter_type,expected", + _materialization_parameter_sets_legacy(), + ids=id_mat, +) +def test_find_materialization_by_name_legacy(macros, adapter_type, expected): + set_from_args( + Namespace( + SEND_ANONYMOUS_USAGE_STATS=False, + REQUIRE_EXPLICIT_PACKAGE_OVERRIDES_FOR_BUILTIN_MATERIALIZATIONS=False, + ), + None, + ) + + manifest = make_manifest(macros=macros) + result = manifest.find_materialization_macro_by_name( + project_name="root", + materialization_name="my_materialization", + adapter_type=adapter_type, + ) + if expected is None: + assert result is expected + else: + expected_package, expected_adapter_type = expected + assert result.adapter_type == expected_adapter_type + assert result.package_name == expected_package + + +def _materialization_parameter_sets(): + # inject the plugins used for materialization parameter tests + with mock.patch("dbt.adapters.base.plugin.project_name_from_path") as get_name: + get_name.return_value = "foo" + FooPlugin = AdapterPlugin( + adapter=mock.MagicMock(), + credentials=mock.MagicMock(), + include_path="/path/to/root/plugin", + ) + FooPlugin.adapter.type.return_value = "foo" + inject_plugin(FooPlugin) + + BarPlugin = AdapterPlugin( + adapter=mock.MagicMock(), + credentials=mock.MagicMock(), + include_path="/path/to/root/plugin", + dependencies=["foo"], + ) + BarPlugin.adapter.type.return_value = "bar" + inject_plugin(BarPlugin) + + sets = [ + FindMaterializationSpec(macros=[], adapter_type="foo", expected=None), + ] + + # default only, each project + sets.extend( + FindMaterializationSpec( + macros=[MockMaterialization(project, adapter_type=None)], + adapter_type="foo", + expected=(project, "default"), + ) + for project in ["root", "dep", "dbt"] + ) + + # other type only, each project + sets.extend( + FindMaterializationSpec( + macros=[MockMaterialization(project, adapter_type="bar")], + adapter_type="foo", + expected=None, + ) + for project in ["root", "dep", "dbt"] + ) + + # matching type only, each project + sets.extend( + FindMaterializationSpec( + macros=[MockMaterialization(project, adapter_type="foo")], + adapter_type="foo", + expected=(project, "foo"), + ) + for project in ["root", "dep", "dbt"] + ) + + sets.extend( + [ + # matching type and default everywhere + FindMaterializationSpec( + macros=[ + MockMaterialization(project, adapter_type=atype) + for (project, atype) in product(["root", "dep", "dbt"], ["foo", None]) + ], + adapter_type="foo", + expected=("root", "foo"), + ), + # default in core, override is in dep, and root has unrelated override + # should find the dbt default because default materializations cannot be overwritten by packages. + FindMaterializationSpec( + macros=[ + MockMaterialization("root", adapter_type="bar"), + MockMaterialization("dep", adapter_type="foo"), + MockMaterialization("dbt", adapter_type=None), + ], + adapter_type="foo", + expected=("dbt", "default"), + ), + # default in core, unrelated override is in dep, and root has an override + # should find the root override. + FindMaterializationSpec( + macros=[ + MockMaterialization("root", adapter_type="foo"), + MockMaterialization("dep", adapter_type="bar"), + MockMaterialization("dbt", adapter_type=None), + ], + adapter_type="foo", + expected=("root", "foo"), + ), + # default in core, override is in dep, and root has an override too. + # should find the root override. + FindMaterializationSpec( + macros=[ + MockMaterialization("root", adapter_type="foo"), + MockMaterialization("dep", adapter_type="foo"), + MockMaterialization("dbt", adapter_type=None), + ], + adapter_type="foo", + expected=("root", "foo"), + ), + # core has default + adapter, dep has adapter, root has default + # should find the default adapter implementation, because it's the most specific + # and default materializations cannot be overwritten by packages + FindMaterializationSpec( + macros=[ + MockMaterialization("root", adapter_type=None), + MockMaterialization("dep", adapter_type="foo"), + MockMaterialization("dbt", adapter_type=None), + MockMaterialization("dbt", adapter_type="foo"), + ], + adapter_type="foo", + expected=("dbt", "foo"), + ), + ] + ) + + # inherit from parent adapter + sets.extend( + FindMaterializationSpec( + macros=[MockMaterialization(project, adapter_type="foo")], + adapter_type="bar", + expected=(project, "foo"), + ) + for project in ["root", "dep", "dbt"] + ) + sets.extend( + FindMaterializationSpec( + macros=[ + MockMaterialization(project, adapter_type="foo"), + MockMaterialization(project, adapter_type="bar"), + ], + adapter_type="bar", + expected=(project, "bar"), + ) + for project in ["root", "dep", "dbt"] + ) + + return sets + + @pytest.mark.parametrize( "macros,adapter_type,expected", _materialization_parameter_sets(), ids=id_mat, ) def test_find_materialization_by_name(macros, adapter_type, expected): + set_from_args( + Namespace( + SEND_ANONYMOUS_USAGE_STATS=False, + REQUIRE_EXPLICIT_PACKAGE_OVERRIDES_FOR_BUILTIN_MATERIALIZATIONS=True, + ), + None, + ) + manifest = make_manifest(macros=macros) result = manifest.find_materialization_macro_by_name( project_name="root",
Vulnerability mechanics
Generated on May 9, 2026. Inputs: CWE entries + fix-commit diffs from this CVE's patches. Citations validated against bundle.
References
11- github.com/advisories/GHSA-p3f3-5ccg-83xqghsaADVISORY
- nvd.nist.gov/vuln/detail/CVE-2024-40637ghsaADVISORY
- docs.getdbt.com/docs/build/packagesghsax_refsource_MISCWEB
- docs.getdbt.com/reference/global-configs/legacy-behaviorsghsax_refsource_MISCWEB
- github.com/dbt-labs/dbt-core/commit/3c82a0296d227cb1be295356df314c11716f4ff6ghsax_refsource_MISCWEB
- github.com/dbt-labs/dbt-core/commit/87ac4deb00cc9fe334706e42a365903a1d581624ghsax_refsource_MISCWEB
- github.com/dbt-labs/dbt-core/security/advisories/GHSA-p3f3-5ccg-83xqghsax_refsource_CONFIRMWEB
- github.com/pypa/advisory-database/tree/main/vulns/dbt-core/PYSEC-2024-66.yamlghsaWEB
- tempered.works/posts/2024/07/06/preventing-data-theft-with-gcp-service-controlsghsax_refsource_MISCWEB
- www.elementary-data.com/post/are-dbt-packages-secure-the-answer-lies-in-your-dwh-policiesghsax_refsource_MISCWEB
- www.equalexperts.com/blog/tech-focus/are-you-at-risk-from-this-critical-dbt-vulnerabilityghsax_refsource_MISCWEB
News mentions
0No linked articles in our index yet.