Data Engineering

The systematic collection and transformation of data via the creation of tools and pipelines.
MySQL, Google Cloud, and a REST API that Generates Itself

MySQL, Google Cloud, and a REST API that Generates Itself

Deploy a MySQL database that auto-creates endpoints for itself.

It wasn’t too long ago that I haphazardly forced us down a journey of exploring Google Cloud’s cloud SQL service. The focus of this exploration was Google’s accompanying REST API for all of its cloud SQL instances. That API turned out to be a relatively disappointing administrative API which did little to extend the features you’d

MySQL Todd Birchard avatarTodd Birchard Todd Birchard avatar
toddFeb 23
Oct 23
Read
Create Google Cloud Functions Running Python 3.7

Create Google Cloud Functions Running Python 3.7

GCP scores another victory by trivializing serverless functions.

The more I explore Google Cloud's endless catalog of cloud services, the more I really like Google Cloud. This is why before moving forward, I'd like to be transparent that this blog has become little more than thinly veiled Google propaganda, where I will henceforth bombard you with persuasive and subtle messaging to sell your soul to Google. Let's be

Google Cloud Todd Birchard avatarTodd Birchard Todd Birchard avatar
toddFeb 23
Oct 18
Read
Extract Nested Data From Complex JSON

Extract Nested Data From Complex JSON

Never manually walk through complex JSON objects again by using this function.

We're all data people here, so you already know the scenario: it happens perhaps once a day, perhaps 5, or even more. There's an API you're working with, and it's great. It contains all the information you're looking for, but there's just one problem: the complexity of nested JSON objects is endless, and suddenly the job you love needs to

Python Todd Birchard avatarTodd Birchard Todd Birchard avatar
toddFeb 23
Oct 10
Read
Reading and Writing to CSVs in Python

Reading and Writing to CSVs in Python

Playing with tabular data the native Python way.

Tables. Cells. Two-dimensional data. We here at Hackers & Slackers know how to talk dirty, but there's one word we'll be missing from our vocabulary today: Pandas.Before the remaining audience closes their browser windows in fury, hear me out. We love Pandas; so much so that tend to recklessly gunsling this 30mb library to perform simple tasks. This isn't

Python Todd Birchard avatarTodd Birchard Todd Birchard avatar
toddFeb 23
Sep 27
Read
Hacking Your Tableau Linux Server

Hacking Your Tableau Linux Server

Cracking Tableau's master Postgres account.

Let's say you're a Data Scientist. Well, maybe not a data scientist- I mean, those online data analysis courses were definitely worth it, and you'd made it this far without being quizzed on Bayesian linear regression. So maybe you're an analyst or something, but whatever:  you use Tableau, So you must be a Scientist™.

I've admitted a few times in

Tableau Todd Birchard avatarTodd Birchard Todd Birchard avatar
toddFeb 23
Jul 26
Read
Using Pandas with AWS Lambda Functions

Using Pandas with AWS Lambda Functions

Forcefully use the Pandas library in your AWS Lambda functions.

In one corner we have Pandas: Python's beloved data analysis library. In the other, AWS: the unstoppable cloud provider we're obligated to use for all eternity. We should have know this day would come.

While not the prettiest workflow, uploaded Python package dependencies for usage in AWS Lambda is typically straightforward. We install the packages locally to a virtual env,

Read
Working with XML tree data in Python

Working with XML tree data in Python

Make use of Python's native XML library to walk through and extract data.

Life is filled with things we don't want to do; you're a developer so you probably understand this to a higher degree than most people. Sometimes we waste weeks of our lives thanks to an unreasonable and unknowledgeable stakeholder. Other times, we need to deal with XML trees.

At some point or another you're going to need to work with

Python Todd Birchard avatarTodd Birchard Todd Birchard avatar
toddFeb 23
Jun 19
Read
Using PyMySQL: Python's MySQL Library

Using PyMySQL: Python's MySQL Library

Lightweight Python library to interact with MySQL.

It's almost Friday night, and the squad at H+S is ready to get cooking. Dim down the lights and slip into something more comfortable as we take you on this 100% organic flavor extravaganza. Tonight's menu? A Python MySQL library: PyMySQL.

PyMySQL is lightweight and perfect for fulfilling MySQL queries. If you want bells and whistles, you're probably barking

Python Todd Birchard avatarTodd Birchard Todd Birchard avatar
toddFeb 23
Jun 15
Read
Accessing Self-Hosted MySQL  Externally

Accessing Self-Hosted MySQL Externally

Connecting to MySQL instances hosted on a VPS.

In the previous post we got familiar with the basics of creating and navigating MySQL databases. This leads us to the next most logic thing to ask: how can I use this in any meaningful way?

MySQL installations default to refusing connections outside of the local machine's IP address, as we should expect. That said, relational databases aren't usually being

MySQL Todd Birchard avatarTodd Birchard Todd Birchard avatar
toddFeb 23
Apr 22
Read
Tree Hierarchies with Treelib

Tree Hierarchies with Treelib

Using Python to visualize heirarchy trees.

The first part of understanding any type of software is taking a glance at its file structure. It may seem like an outlandish and redundant statement to make to a generation who grew up on GUIs. GitHub is essentially no more than a GUI for Git, so it’s unsurprisingly that one of the largest company to follow a similar

Python Todd Birchard avatarTodd Birchard Todd Birchard avatar
toddFeb 23
Nov 17
Read