Big Data

Work with unstructured data across file types and schemas. Tools such as data warehouses, Hadoop, Spark, BigQuery, etc.
Google BigQuery's Python SDK: Creating Tables Programmatically

Google BigQuery's Python SDK: Creating Tables Programmatically

Create tables in Google BigQuery, auto-generate their schemas, and retrieve said schemas.

GCP is on the rise, and it's getting harder and harder to have conversations around data without addressing the 500-pound gorilla in the room: Google BigQuery. With most enterprises comfortably settled into their Apache-based Big Data stacks, BigQuery rattles the cages of convention for many. Luckily, Hackers And Slackers is no such enterprise. Thus, we aren't afraid to ask the

Big Data Todd Birchard avatarTodd Birchard Todd Birchard avatar
toddFeb 23
Feb 02
Read
From CSVs to Tables: Infer Data Types From Raw Spreadsheets

From CSVs to Tables: Infer Data Types From Raw Spreadsheets

The quest to never explicitly set a table schema ever again.

Back in August of last year (roughly 8 months ago), I hunched over my desk at 4 am desperate to fire off a post before boarding a flight the next morning. The article was titled Creating Database Schemas: a Job for Robots, or Perhaps Pandas. It was my intent at the time to solve a common annoyance: creating database tables

Big Data Todd Birchard avatarTodd Birchard Todd Birchard avatar
toddFeb 23
Jan 23
Read
Data Could Save Humanity if it Weren't for Humanity

Data Could Save Humanity if it Weren't for Humanity

A compelling case for robot overlords.

A decade has passed since I stumbled into technical product development. Looking back, I've spent that time almost exclusively in the niche of data-driven products and engineering. While it seems obvious now, I realized in the 2000s that you could generally create two types of product: you could either build a (likely uninspired) UI for existing data, or you could

Read