Version: 1.0 prerelease

AbstractDataContext

class great_expectations.data_context.AbstractDataContext(runtime_environment: Optional[dict] = None)#

Base class for all Data Contexts that contains shared functionality.

The class encapsulates most store / core components and convenience methods used to access them, meaning the majority of Data Context functionality lives here.

One of the primary responsibilities of the DataContext is managing CRUD operations for core GX objects:

Supported CRUD Methods#
	Stores	Datasources	ExpectationSuites	Checkpoints
get	❌	✅	✅	✅
add	✅	✅	✅	✅
update	❌	✅	✅	✅
add_or_update	❌	✅	✅	✅
delete	✅	✅	✅	✅

add_checkpoint(name: str = None, expectation_suite_name: str | None = None, batch_request: dict | None = None, action_list: Sequence[ActionDict] | None = None, suite_parameters: dict | None = None, runtime_configuration: dict | None = None, validations: list[CheckpointValidationDefinition] | list[dict] | None = None, id: str | None = None, expectation_suite_id: str | None = None, default_validation_id: str | None = None, validator: Validator | None = None, checkpoint: None = None) → Checkpoint#

add_checkpoint(name: None = None, expectation_suite_name: None = None, batch_request: None = None, action_list: Sequence[ActionDict] | None = None, suite_parameters: None = None, runtime_configuration: None = None, validations: None = None, id: None = None, expectation_suite_id: None = None, default_validation_id: None = None, validator: Validator | None = None, checkpoint: Checkpoint = None) → Checkpoint

Add a Checkpoint to the DataContext.

Relevant Documentation Links

Checkpoint

Parameters:

name – The name to give the checkpoint.
expectation_suite_name – The expectation suite name to use in generating this checkpoint.
batch_request – The batch request to use in generating this checkpoint.
action_list – The action list to use in generating this checkpoint.
suite_parameters – The suite parameters to use in generating this checkpoint.
runtime_configuration – The runtime configuration to use in generating this checkpoint.
validations – The validations to use in generating this checkpoint.
id – The ID to use in generating this checkpoint.
expectation_suite_id – The expectation suite ID to use in generating this checkpoint.
default_validation_id – The default validation ID to use in generating this checkpoint.
validator –
An existing validator used to generate a validations list.
New in version 0.16.15: Pass in an existing validator instead of individual validations
checkpoint –
An existing checkpoint you wish to persist.
New in version 0.15.48: Pass in an existing checkpoint instead of individual constructor args

Returns:

The Checkpoint object created.

add_data_docs_site(site_name: str, site_config: DataDocsSiteConfigTypedDict) → None#

Add a new Data Docs Site to the DataContext.

New in version 0.17.2.

Example site config dicts can be found in our “Host and share Data Docs” guides.

Parameters:

site_name – New site name to add.
site_config – Config dict for the new site.

add_datasource(name: str = None, initialize: bool = True, datasource: None = None, **kwargs) → BaseDatasource | FluentDatasource | LegacyDatasource | None#

Add a new Datasource to the data context, with configuration provided as kwargs.

Relevant Documentation Links

Datasource

Parameters:

name – the name of the new Datasource to add
initialize – if False, add the Datasource to the config, but do not initialize it, for example if a user needs to debug database connectivity.
datasource –
an existing Datasource you wish to persist
New in version 0.15.49: Pass in an existing Datasource instead of individual constructor arguments
kwargs – the configuration for the new Datasource

Returns:

Datasource instance added.

add_expectation_suite(expectation_suite_name: str, id: str | None = None, expectations: list[dict | ExpectationConfiguration] | None = None, suite_parameters: dict | None = None, execution_engine_type: Type[ExecutionEngine] | None = None, meta: dict | None = None, expectation_suite: None = None) → great_expectations.core.expectation_suite.ExpectationSuite#

add_expectation_suite(expectation_suite_name: None = None, id: str | None = None, expectations: list[dict | ExpectationConfiguration] | None = None, suite_parameters: dict | None = None, execution_engine_type: Type[ExecutionEngine] | None = None, meta: dict | None = None, expectation_suite: great_expectations.core.expectation_suite.ExpectationSuite = None) → great_expectations.core.expectation_suite.ExpectationSuite

Build a new ExpectationSuite and save it utilizing the context’s underlying ExpectationsStore.

New in version 0.15.48.

Note that this method can be called by itself or run within the get_validator workflow.

When run with create_expectation_suite():

expectation_suite_name="genres_movies.fkey"context.create_expectation_suite(expectation_suite_name,overwrite_existing=True)batch=context.get_batch_list(expectation_suite_name=expectation_suite_name)[0]

When run as part of get_validator():

validator=context.get_validator(datasource_name="my_datasource",data_connector_name="whole_table",data_asset_name="my_table",create_expectation_suite_with_name="my_expectation_suite",)validator.expect_column_values_to_be_in_set("c1",[4,5,6])

Parameters:

expectation_suite_name – The name of the suite to create.
id – Identifier to associate with this suite.
expectations – Expectation Configurations to associate with this suite.
suite_parameters – Suite parameters to be substituted when evaluating Expectations.
execution_engine_type – Name of the execution engine type.
meta – Metadata related to the suite.

Returns:

A new ExpectationSuite built with provided input args.

Raises:

DataContextError – A suite with the same name already exists (and overwrite_existing is not enabled).
ValueError – The arguments provided are invalid.

add_or_update_checkpoint(name: str = None, id: str | None = None, expectation_suite_name: str | None = None, batch_request: dict | None = None, action_list: Sequence[ActionDict] | None = None, suite_parameters: dict | None = None, runtime_configuration: dict | None = None, validations: list[dict] | None = None, expectation_suite_id: str | None = None, default_validation_id: str | None = None, validator: Validator | None = None, checkpoint: None = None) → Checkpoint#

add_or_update_checkpoint(name: None = None, id: None = None, expectation_suite_name: None = None, batch_request: None = None, action_list: Sequence[ActionDict] | None = None, suite_parameters: None = None, runtime_configuration: None = None, validations: None = None, expectation_suite_id: None = None, default_validation_id: None = None, validator: Validator | None = None, checkpoint: Checkpoint = None) → Checkpoint

Add a new Checkpoint or update an existing one on the context depending on whether it already exists or not.

New in version 0.15.48.

Parameters:

name – The name to give the checkpoint.
id – The ID to associate with this checkpoint.
expectation_suite_name – The expectation suite name to use in generating this checkpoint.
batch_request – The batch request to use in generating this checkpoint.
action_list – The action list to use in generating this checkpoint.
suite_parameters – The suite parameters to use in generating this checkpoint.
runtime_configuration – The runtime configuration to use in generating this checkpoint.
validations – The validations to use in generating this checkpoint.
expectation_suite_id – The expectation suite GE Cloud ID to use in generating this checkpoint.
default_validation_id – The default validation ID to use in generating this checkpoint.
validator –
An existing validator used to generate a validations list.
New in version 0.16.15: Pass in an existing validator instead of individual validations
checkpoint – An existing checkpoint you wish to persist.

Returns:

A new Checkpoint or an updated once (depending on whether or not it existed before this method call).

add_or_update_datasource(name: str = None, datasource: None = None, **kwargs) → BaseDatasource | FluentDatasource | LegacyDatasource#

add_or_update_datasource(name: None = None, datasource: BaseDatasource | FluentDatasource | LegacyDatasource = None, **kwargs) → BaseDatasource | FluentDatasource | LegacyDatasource

Add a new Datasource or update an existing one on the context depending on whether

New in version 0.15.48.

it already exists or not. The configuration is provided as kwargs.

Parameters:

name – The name of the Datasource to add or update.
datasource – an existing Datasource you wish to persist.
kwargs – Any relevant keyword args to use when adding or updating the target Datasource named name.

Returns:

The Datasource added or updated by the input kwargs.

add_or_update_expectation_suite(expectation_suite_name: str, id: str | None = None, expectations: list[dict | ExpectationConfiguration] | None = None, suite_parameters: dict | None = None, execution_engine_type: Type[ExecutionEngine] | None = None, meta: dict | None = None, expectation_suite: None = None) → great_expectations.core.expectation_suite.ExpectationSuite#

add_or_update_expectation_suite(expectation_suite_name: None = None, id: str | None = None, expectations: list[dict | ExpectationConfiguration] | None = None, suite_parameters: dict | None = None, execution_engine_type: Type[ExecutionEngine] | None = None, meta: dict | None = None, expectation_suite: great_expectations.core.expectation_suite.ExpectationSuite = None) → great_expectations.core.expectation_suite.ExpectationSuite

Add a new ExpectationSuite or update an existing one on the context depending on whether it already exists or not.

New in version 0.15.48.

Parameters:

expectation_suite_name – The name of the suite to create or replace.
id – Identifier to associate with this suite (ignored if updating existing suite).
expectations – Expectation Configurations to associate with this suite.
suite_parameters – Suite parameters to be substituted when evaluating Expectations.
execution_engine_type – Name of the Execution Engine type.
meta – Metadata related to the suite.
expectation_suite – The ExpectationSuite object you wish to persist.

Returns:

The persisted ExpectationSuite.

add_store(store_name: str, store_config: StoreConfigTypedDict) → Store#

Add a new Store to the DataContext.

Parameters:

store_name – the name to associate with the created store.
store_config – the config to use to construct the store.

Returns:

The instantiated Store.

build_data_docs(site_names: list[str] | None = None, resource_identifiers: list | None = None, dry_run: bool = False, build_index: bool = True) → dict#

Build Data Docs for your project.

Relevant Documentation Links

Data Docs

Parameters:

site_names – if specified, build data docs only for these sites, otherwise, build all the sites specified in the context’s config
resource_identifiers – a list of resource identifiers (ExpectationSuiteIdentifier, ValidationResultIdentifier). If specified, rebuild HTML (or other views the data docs sites are rendering) only for the resources in this list. This supports incremental build of data docs sites (e.g., when a new validation result is created) and avoids full rebuild.
dry_run – a flag, if True, the method returns a structure containing the URLs of the sites that would be built, but it does not build these sites.
build_index – a flag if False, skips building the index page

Returns:

A dictionary with the names of the updated data documentation sites as keys and the location info of their index.html files as values

Raises:

ClassInstantiationError – Site config in your Data Context config is not valid.

delete_data_docs_site(site_name: str)#

Delete an existing Data Docs Site.

New in version 0.17.2.

Parameters:

site_name – Site name to delete.

delete_datasource(datasource_name: Optional[str])→ None#

Delete a given Datasource by name.

Note that this method causes deletion from the underlying DatasourceStore.

Parameters:

datasource_name – The name of the target datasource.

Raises:

ValueError – The datasource_name isn’t provided or cannot be found.

delete_expectation_suite(expectation_suite_name: str | None = None, id: str | None = None) → None#

Delete specified expectation suite from data_context expectation store.

Parameters:

expectation_suite_name – The name of the expectation suite to delete
id – The identifier of the expectation suite to delete

Returns:

True for Success and False for Failure.

delete_store(store_name: str)→ None#

Delete an existing Store from the DataContext.

New in version 0.15.48.

Parameters:

store_name – The name of the Store to be deleted.

Raises:

StoreConfigurationError if the target Store is not found. –

Inspect datasource and batch kwargs generators to provide available data_asset objects.

Parameters:

datasource_names – List of datasources for which to provide available data asset name objects. If None, return available data assets for all datasources.
batch_kwargs_generator_names – List of batch kwargs generators for which to provide available data_asset_name objects.

Returns:

Dictionary describing available data assets

Return type:

data_asset_names

Raises:

ValueError – datasource_names is not None, a string, or list of strings.

get_batch_list(datasource_name: Optional[str] = None, data_connector_name: Optional[str] = None, data_asset_name: Optional[str] = None, batch_request: Optional[BatchRequestBase] = None, batch_data: Optional[Any] = None, data_connector_query: Optional[dict] = None, batch_identifiers: Optional[dict] = None, limit: Optional[int] = None, index: Optional[Union[int, list, tuple, slice, str]] = None, custom_filter_function: Optional[Callable] = None, sampling_method: Optional[str] = None, sampling_kwargs: Optional[dict] = None, partitioner_method: Optional[str] = None, partitioner_kwargs: Optional[dict] = None, runtime_parameters: Optional[dict] = None, query: Optional[str] = None, path: Optional[str] = None, batch_filter_parameters: Optional[dict] = None, batch_spec_passthrough: Optional[dict] = None, batch_parameters: Optional[Union[dict, BatchParameters]] = None, **kwargs: Optional[dict]) → List[Batch]#

Get the list of zero or more batches, based on a variety of flexible input types.

get_batch_list is the main user-facing API for getting batches. In contrast to virtually all other methods in the class, it does not require typed or nested inputs. Instead, this method is intended to help the user pick the right parameters

This method attempts to return any number of batches, including an empty list.

Parameters:

datasource_name – The name of the Datasource that defines the Data Asset to retrieve the batch for
data_connector_name – The Data Connector within the datasource for the Data Asset
data_asset_name – The name of the Data Asset within the Data Connector
batch_request – Encapsulates all the parameters used here to retrieve a BatchList. Use either batch_request or the other params (but not both)
batch_data – Provides runtime data for the batch; is added as the key batch_data to the runtime_parameters dictionary of a BatchRequest
query – Provides runtime data for the batch; is added as the key query to the runtime_parameters dictionary of a BatchRequest
path – Provides runtime data for the batch; is added as the key path to the runtime_parameters dictionary of a BatchRequest
runtime_parameters – Specifies runtime parameters for the BatchRequest; can includes keys batch_data, query, and path
data_connector_query – Used to specify connector query parameters; specifically batch_filter_parameters, limit, index, and custom_filter_function
batch_identifiers – Any identifiers of batches for the BatchRequest
batch_filter_parameters – Filter parameters used in the data connector query
limit – Part of the data_connector_query, limits the number of batches in the batch list
index – Part of the data_connector_query, used to specify the index of which batch to return. Negative numbers retrieve from the end of the list (ex: -1 retrieves the last or latest batch)
custom_filter_function – A Callable function that accepts batch_identifiers and returns a bool
sampling_method – The method used to sample Batch data (see: Partitioning and Sampling)
sampling_kwargs – Arguments for the sampling method
partitioner_method – The method used to partition the Data Asset into Batches
partitioner_kwargs – Arguments for the partitioning method
batch_spec_passthrough – Arguments specific to the ExecutionEngine that aid in Batch retrieval
batch_parameters – Options for FluentBatchRequest
**kwargs – Used to specify either batch_identifiers or batch_filter_parameters

Returns:

(Batch) The list of requested Batch instances

Raises:

DatasourceError – If the specified datasource_name does not exist in the DataContext
TypeError – If the specified types of the batch_request are not supported, or if the datasource_name is not a str
ValueError – If more than one exclusive parameter is specified (ex: specifing more than one of batch_data, query or path)

get_datasource(datasource_name: str = 'default')→ BaseDatasource | FluentDatasource | LegacyDatasource#

Retrieve a given Datasource by name from the context’s underlying DatasourceStore.

Parameters:

datasource_name – The name of the target datasource.

Returns:

The target datasource.

Raises:

ValueError – The input datasource_name is None.

get_expectation_suite(expectation_suite_name: str | None = None, include_rendered_content: bool | None = None, id: str | None = None) → ExpectationSuite#

Get an Expectation Suite by name.

Parameters:

expectation_suite_name (str) – The name of the Expectation Suite
include_rendered_content (bool) – Whether to re-populate rendered_content for each ExpectationConfiguration.
id (str) – The GX Cloud ID for the Expectation Suite (unused)

Returns:

An existing ExpectationSuite

Raises:

DataContextError – There is no expectation suite with the name provided

get_validator(datasource_name: Optional[str] = None, data_connector_name: Optional[str] = None, data_asset_name: Optional[str] = None, batch: Optional[Batch] = None, batch_list: Optional[List[Batch]] = None, batch_request: Optional[Union[BatchRequestBase, FluentBatchRequest]] = None, batch_request_list: Optional[List[BatchRequestBase]] = None, batch_data: Optional[Any] = None, data_connector_query: Optional[Union[IDDict, dict]] = None, batch_identifiers: Optional[dict] = None, limit: Optional[int] = None, index: Optional[Union[int, list, tuple, slice, str]] = None, custom_filter_function: Optional[Callable] = None, sampling_method: Optional[str] = None, sampling_kwargs: Optional[dict] = None, partitioner_method: Optional[str] = None, partitioner_kwargs: Optional[dict] = None, runtime_parameters: Optional[dict] = None, query: Optional[str] = None, path: Optional[str] = None, batch_filter_parameters: Optional[dict] = None, batch_spec_passthrough: Optional[dict] = None, expectation_suite_id: Optional[str] = None, expectation_suite_name: Optional[str] = None, expectation_suite: Optional[ExpectationSuite] = None, create_expectation_suite_with_name: Optional[str] = None, include_rendered_content: Optional[bool] = None, **kwargs) → Validator#

Retrieve a Validator with a batch list and an ExpectationSuite.

get_validator first calls get_batch_list to retrieve a batch list, then creates or retrieves an ExpectationSuite used to validate the Batches in the list.

Parameters:

datasource_name – The name of the Datasource that defines the Data Asset to retrieve the batch for
data_connector_name – The Data Connector within the datasource for the Data Asset
data_asset_name – The name of the Data Asset within the Data Connector
batch – The Batch to use with the Validator
batch_list – The List of Batches to use with the Validator
batch_request – Encapsulates all the parameters used here to retrieve a BatchList. Use either batch_request or the other params (but not both)
batch_request_list – A List of BatchRequest to use with the Validator
batch_data – Provides runtime data for the batch; is added as the key batch_data to the runtime_parameters dictionary of a BatchRequest
query – Provides runtime data for the batch; is added as the key query to the runtime_parameters dictionary of a BatchRequest
path – Provides runtime data for the batch; is added as the key path to the runtime_parameters dictionary of a BatchRequest
runtime_parameters – Specifies runtime parameters for the BatchRequest; can includes keys batch_data, query, and path
data_connector_query – Used to specify connector query parameters; specifically batch_filter_parameters, limit, index, and custom_filter_function
batch_identifiers – Any identifiers of batches for the BatchRequest
batch_filter_parameters – Filter parameters used in the data connector query
limit – Part of the data_connector_query, limits the number of batches in the batch list
index – Part of the data_connector_query, used to specify the index of which batch to return. Negative numbers retrieve from the end of the list (ex: -1 retrieves the last or latest batch)
custom_filter_function – A Callable function that accepts batch_identifiers and returns a bool
sampling_method – The method used to sample Batch data (see: Partitioning and Sampling)
sampling_kwargs – Arguments for the sampling method
partitioner_method – The method used to partition the Data Asset into Batches
partitioner_kwargs – Arguments for the partitioning method
batch_spec_passthrough – Arguments specific to the ExecutionEngine that aid in Batch retrieval
expectation_suite_id – The identifier of the ExpectationSuite to retrieve from the DataContext (can be used in place of expectation_suite_name)
expectation_suite_name – The name of the ExpectationSuite to retrieve from the DataContext
expectation_suite – The ExpectationSuite to use with the validator
create_expectation_suite_with_name – Creates a Validator with a new ExpectationSuite with the provided name
include_rendered_content – If True the ExpectationSuite will include rendered content when saved
**kwargs – Used to specify either batch_identifiers or batch_filter_parameters

Returns:

A Validator with the specified Batch list and ExpectationSuite

Return type:

Validator

Raises:

DatasourceError – If the specified datasource_name does not exist in the DataContext
TypeError – If the specified types of the batch_request are not supported, or if the datasource_name is not a str
ValueError – If more than one exclusive parameter is specified (ex: specifing more than one of batch_data, query or path), or if the ExpectationSuite cannot be created or retrieved using either the provided name or identifier

list_checkpoints()→ Union[List[str], List[great_expectations.data_context.types.resource_identifiers.ConfigurationIdentifier]]#

List existing Checkpoint identifiers on this context.

Returns:: Either a list of strings or ConfigurationIdentifiers depending on the environment and context type.

list_data_docs_sites()→ dict[str, DataDocsSiteConfigTypedDict]#: List all Data Docs Sites with configurations.
New in version 0.17.2.

list_datasources()→ List[dict]#

List the configurations of the datasources associated with this context.

Note that any sensitive values are obfuscated before being returned.

Returns:: A list of dictionaries representing datasource configurations.

list_expectation_suite_names()→ List[str]#

Lists the available expectation suite names.

Returns:: A list of suite names (sorted in alphabetic order).

save_expectation_suite(expectation_suite: great_expectations.core.expectation_suite.ExpectationSuite, expectation_suite_name: Optional[str] = None, overwrite_existing: bool = True, include_rendered_content: Optional[bool] = None, **kwargs: Optional[dict]) → None#

Save the provided ExpectationSuite into the DataContext using the configured ExpectationStore.

Deprecated since version 0.15.48: Part of the deprecated DataContext CRUD API

Parameters:

expectation_suite – The ExpectationSuite to save.
expectation_suite_name – The name of this ExpectationSuite. If no name is provided, the name will be read from the suite.
overwrite_existing – Whether to overwrite the suite if it already exists.
include_rendered_content – Whether to save the prescriptive rendered content for each expectation.
kwargs – Additional parameters, unused

Returns:

None

Raises:

DataContextError – If a suite with the same name exists and overwrite_existing is set to False.

test_yaml_config(yaml_config: str, name: Optional[str] = None, class_name: Optional[str] = None, runtime_environment: Optional[dict] = None, pretty_print: bool = True, shorten_tracebacks: bool = False)#

Convenience method for testing yaml configs.

test_yaml_config is a convenience method for configuring the moving parts of a Great Expectations deployment. It allows you to quickly test out configs for system components, especially Datasources, Checkpoints, and Stores.

For many deployments of Great Expectations, these components (plus Expectations) are the only ones you’ll need.

test_yaml_config is mainly intended for use within notebooks and tests.

Relevant Documentation Links

Data Context

Parameters:

yaml_config – A string containing the yaml config to be tested
name – Optional name of the component to instantiate
class_name – Optional, overridden if provided in the config
runtime_environment – Optional override for config items
pretty_print – Determines whether to print human-readable output
shorten_tracebacks – If true, catch any errors during instantiation and print only the last element of the traceback stack. This can be helpful for rapid iteration on configs in a notebook, because it can remove the need to scroll up and down a lot.

Returns:

The instantiated component (e.g. a Datasource) OR a json object containing metadata from the component’s self_check method. The returned object is determined by return_mode.

update_checkpoint(checkpoint: Checkpoint)→ Checkpoint#

Update a Checkpoint that already exists.

New in version 0.15.48.

Parameters:

checkpoint – The checkpoint to use to update.

Raises:

DataContextError – A suite with the given name does not already exist.

Returns:

The updated Checkpoint.

update_data_docs_site(site_name: str, site_config: DataDocsSiteConfigTypedDict) → None#

Update an existing Data Docs Site.

New in version 0.17.2.

Example site config dicts can be found in our “Host and share Data Docs” guides.

Parameters:

site_name – Site name to update.
site_config – Config dict that replaces the existing.

update_datasource(datasource: BaseDatasource | FluentDatasource | LegacyDatasource)→ BaseDatasource | FluentDatasource | LegacyDatasource#

Updates a Datasource that already exists in the store.

Parameters:

datasource – The Datasource object to update.

Returns:

The updated Datasource.

update_expectation_suite(expectation_suite: great_expectations.core.expectation_suite.ExpectationSuite)→ great_expectations.core.expectation_suite.ExpectationSuite#

Update an ExpectationSuite that already exists.

New in version 0.15.48.

Parameters:

expectation_suite – The suite to use to update.

Raises:

DataContextError – A suite with the given name does not already exist.

update_project_config(project_config: DataContextConfig | Mapping)→ DataContextConfig#

Update the context’s config with the values from another config object.

Parameters:

project_config – The config to use to update the context’s internal state.

Returns:

The updated project config.

view_validation_result(result: CheckpointResult)→ None#

New in version 0.16.15.

Opens a validation result in a browser.

Parameters:

result – The result of a Checkpoint run.