dbt

class DbtManager(data_store_id: int)

Bases: object

Create a dbt manager that handles the sending of a manifest file to Tree Schema, waits for the parsing to complete, caches the result output from the parsing and submits results to be saved in Tree Schema

Parameters

data_store_id – the ID for the data store

get_manifest_parse_status() → str

Retrieves the manifest parse status from Tree Schema as well as the results from the parse, if the parse has been completed.

Returns

The manifest parse status

parse_dbt_manifest(manifest: [<class 'str'>, <class 'bytes'>]) → str

Sends a manifest file to Tree Schema to be parsed.

Parameters

manifest – The dbt manifest file to send to Tree Schema. This can be a string which points to a location where the file can be read, or it can be in the form of bytes where the user would have already read the object into memory, for example if the object resides on an external file store.

Returns

a string, the dbt_process_id, representing the unique parse instance

>>> my_data_store = ts.data_store('my data store')
>>> my_data_store.dbt.parse_dbt_manifest('./path/to/target/manifest.json')
save_parse_results(add_schemas_fields: bool = False, update_descriptions: bool = False, update_tags: bool = True, add_lineage: bool = True) → bool

Saves the parsing results into Tree Schema. Allows the user to specify the options for saving the results. Users can choose to add the schemas & fields, descriptions tags and data lineage indepdently.

Parameters
  • add_schemas_fields – whether or not to add the schemas and fields, it is suggested to set this to False for most use-cases and instead to use the automated crawlers within Tree Schema to extract your schemas and fields first

  • update_descriptions – whether or not to update descriptions of your data assets based off of your dbt manifest file

  • update_tags – whether or not to update tags of your data assets based off of your dbt manifest file

  • update_tags – whether or not to add lineage to your data assets based off of your dbt manifest file

Returns

boolean, True if the API invocation was successful

>>> my_data_store = ts.data_store('my data store')
>>> my_data_store.dbt.parse_dbt_manifest('./path/to/target/manifest.json')
>>> my_data_store.dbt.wait_for_parse_complete()
>>> my_data_store.dbt.save_parse_results(update_descriptions=True, add_lineage=True)
wait_for_parse_complete(max_seconds=360) → str

Parsing takes place asynchronously, this function acts as a waiter to try and retrieve the status of the parse for up to max seconds (default 5 minutes) before timing out.

Parameters

max_seconds – the maximum number of seconds to wait while waiting for the results

Returns

the final parse status

>>> my_data_store = ts.data_store('my data store')
>>> my_data_store.dbt.parse_dbt_manifest('./path/to/target/manifest.json')
>>> my_data_store.dbt.wait_for_parse_complete()