Lineage Import

Lineage Import is a tool that Admins can use to import a set of queries that is used to build out field-level lineage between datasets in the project you have connected to. You can find lineage import by navigating to "Mission Control > Lineage Import." From this page, click "Upload" and fill out the form:

• select a user's email to filter the queries by the user, and select a time range.
• choose a table and apply any additional filters. Filters include:
  • INTERACTIVE (queries run interactively)
  • BATCH (queries run as part of a batch process)
  • HIGH (queries run at a high priority level).

Note that this form assumes that you are exporting your logs in a standard format, e.g., with BigQuery using a process like this. Non-standard tables will not appear in the tables list.

Based on the set of queries that match the form, we will build dataset-to-dataset lineage that you can explore in our app on an entity's details page. For more, see our Data Lineage page.

If you perform multiple "Lineage Imports," then we will build lineage based on the union of queries across the imports. For example:

• You upload your query history from January 1.
• The next day you upload your query history from January 2.

After the last upload, your lineage will be based on the union of queries from January 1 and January 2.

FAQ

• We can only build dataset-to-dataset lineage based on the set of queries provided. For example, if the set of queries never references a particular dataset in your project, then we will not be able to derive any lineage for that dataset - we do not know where it comes from or where it is used.