ETL Pipelines (page 1)
Create pipelines to extract, transform, and load data. Use Big Data tools like Apache Spark and Kafka to build horizontally scalable ETL pipelines.
Data Warehouses
Simplify BigQuery ETL jobs using SQLAlchemy
Extract and move data between BigQuery and relational databases using PyBigQuery: a connector for SQLAlchemy.
Apache
Manage Data Pipelines with Apache Airflow
Use Apache Airflow to build and monitor better data pipelines.
Apache
Becoming Familiar with Apache Kafka and Message Queues
Getting to know Apache Kafka: a horizontally scalable event streaming platform. Learn what makes Kafka critical to high-volume low-latency data pipelines.
Spark
Learning Apache Spark with PySpark & Databricks
Get started with Apache Spark in part 1 of our series, where we leverage Databricks and PySpark.
Data Engineering
Building an ETL Pipeline: From JIRA's REST API to SQL
Build a pipeline which extracts raw data from the JIRA's Cloud API, transforms it, and loads the data into a SQL database.