Section 1.1: Understanding ETL and ELT

For those unfamiliar with the Data Transfer Service (DTS), it’s essential to understand its role in data warehousing. Here’s a brief overview of the ETL and ELT processes facilitated by DTS.

ETL vs. ELT Source: fivetran [1]

In the ETL (Extract, Transform, Load) model, data is transformed within the integration tool before being loaded into the target system. Conversely, the ELT (Extract, Load, Transform) approach loads data into the target system first and then performs transformations. Nowadays, ELT is often preferred due to its simplicity and speed. The DTS accommodates both methods, allowing for data integration from source systems (E and L), as well as transformations directly within BigQuery, which can be executed using SQL and scheduled automatically.

Section 1.2: New Security Features in DTS

What’s New?

The latest updates to the BigQuery Data Transfer Service include support for Audit Logging, Cloud Logging, and Cloud Monitoring [2]. These features enable users to track the service's health—monitoring factors such as operational status, data transfer volumes, and active transfers. Additionally, these metrics facilitate the creation of alerts, thereby enhancing the security of data pipelines.