Data Engineering (page 1)
Collect and transform data on a large scale. Build data pipelines, work with a horizontally scalable architecture, or simply scrape and collect data.
Google BigQuery's Python SDK: Creating Tables Programmatically
Use Google Cloud's Python SDK to insert large datasets into Google BigQuery, enjoy the benefits of schema detection, and manipulating data programmatically.
Scrape Structured Data with Python and Extruct
Supercharge your scraper to extract quality page metadata by parsing JSON-LD data via Python's extruct library.
Simplify BigQuery ETL jobs using SQLAlchemy
Extract and move data between BigQuery and relational databases using PyBigQuery: a connector for SQLAlchemy.
Using Amazon Redshift as your Data Warehouse
Get the most out of Redshift by performance tuning your cluster and learning how to query your data optimally.
Join and Aggregate PySpark DataFrames
Perform SQL-like joins and aggregations on your PySpark DataFrames.
Working with PySpark RDDs
Working with Spark's original data structure API: Resilient Distributed Datasets.
Manage Data Pipelines with Apache Airflow
Use Apache Airflow to build and monitor better data pipelines.
Structured Streaming in PySpark
Become familiar with building a structured stream in PySpark using the Databricks interface.
Becoming Familiar with Apache Kafka and Message Queues
Getting to know Apache Kafka: a horizontally scalable event streaming platform. Learn what makes Kafka critical to high-volume low-latency data pipelines.
Cleaning PySpark DataFrames
Easy DataFrame cleaning techniques ranging from dropping rows to selecting important data.