Data Engineering

The systematic collection and transformation of data via the creation of tools and pipelines.

Using Amazon Redshift as your Data Warehouse

Get the most out of Redshift by performance tuning your cluster and learning how to query your data optimally.

Perform SQL-like joins and aggregations on your PySpark DataFrames.

Working with Spark's original data structure API: Resilient Distributed Datasets.

Use Apache Airflow to build and monitor better data pipelines.

Become familiar with building a structured stream in PySpark using the Databricks interface.

Continuing to apply transformations to Spark DataFrames using PySpark.

An overview of how Kafka works, as well as equivalent message brokers.

Using PySpark to apply transformations to real datasets.