Lineage Import

Lineage Import is a tool that Admins can use to import a set of queries that is used to build out field-level lineage between datasets in the project you have connected to. You can find lineage import by navigating to "Mission Control > Lineage Import." From this page, click "Upload" and fill out the form.

Based on the set of queries that match the form, we will build dataset-to-dataset lineage that you can explore in our app on an entity's details page.

📘

Query History Logs

Note that this feature assumes you are exporting your query history logs in a standard format. For example, if you have connected to a BigQuery project, then you will need to be exporting your BigQuery query logs using a process like this.

If you perform multiple "Lineage Imports," then we will build lineage based on the union of queries across the imports. For example:

  • You upload your query history from January 1.
  • The next day you upload your query history from January 2.

After the last upload, your lineage will be based on the union of queries from January 1 and January 2.

Issues

  • We can only build dataset-to-dataset lineage based on the set of queries provided. For example, if the set of queries never references a particular dataset in your project, then we will not be able to derive any lineage for that dataset - we do not know where it comes from or where it is used.