REST APIs

Get the most out of REST APIs by example, or build your own. Discover new APIs and how to interact with them, regardless of proffered programming language.
Building an ETL Pipeline: From JIRA to SQL

Building an ETL Pipeline: From JIRA to SQL

An example data pipeline which extracts data from the JIRA Cloud API and loads it to a SQL database.

Something we haven't done just yet on this site is walking through the humble process of creating data pipelines: the art of taking a bunch of data, changing said data, and putting it somewhere else. It's kind of a weird thing to be into, hence why the MoMA has been rejecting my submissions of Github repositories. Don't worry; I'll keep at it.

Something you don't see every day are people sharing their pipelines, which is understandable. Presumably, the other people who do this kind of stuff do it for work; nobody is happily building stupid pipelines in their free time

Making API Requests With node-fetch

Making API Requests With node-fetch

Using the lightweight node-fetch library for REST API requests in NodeJS.

If you're the type of person to read technical Javascript posts in your free time (you are), you don't need me to tell you that JQuery is dead. JQuery themselves have proclaimed JQuery to be dead. The only cool thing about JQuery is who can remove it from their legacy stack the fastest, which begs the question: why is the third most popular page on this site an old post about JQuery?

Maintaining a blog of tutorials has taught me a lot about the gap between perception and reality. While we content publishers sling Medium posts from our ivory towers,

The Hostile Extraction of Tableau Server Data

The Hostile Extraction of Tableau Server Data

Say no to proprietary software constraints. Say no to vendor lock. Say yes to freedom.

I try my best not to hate on Tableau. It was the software’s combination of power and ease-of-use that drove me to purchase a license in the first place. Ever since then, I’m finding new and exciting ways Tableau intentionally locks users out of their data.

I gave the Tableau Server Client Python library a spin recently in hopes of finding something useful. I decided to (sigh, once more) allow Tableau the benefit of the doubt: after pushing four updates in a single month, maybe things had changed. On the contrary, the Tableau business strategy stands strong: to

Tableau's REST API: Turning Tableau into an ETL Pipeline GUI

Tableau's REST API: Turning Tableau into an ETL Pipeline GUI

Organizing a heist on Tableau Server to reclaim workbook data.

There's nothing I love more than exposing expensive enterprise software.

It may not seem obvious, but most SaaS products have an underlying core goal: shackle businesses to depend on proprietary, closed-source, costly software. When you pair a surplus of money with a reluctance to work, you've arrived at Corporate America: a prime victim yearning to marry itself to any vendor with a nice pitch deck and a vague promise.

In the case of Tableau, this becomes obvious when you attempt to do anything besides create visuals. I don't like spending hours of my time cleaning data to be rewarded with

Complex Features in MongoDB Cloud: Add Image Tags with AI

Complex Features in MongoDB Cloud: Add Image Tags with AI

Using functions, webhooks, and values to utilize external APIs.

Friends, family, and most importantly, strangers: I approach you today with a tale of renewed inspiration. After loudly broadcasting my own confusion and mediocre ability to actually implement an effective cloud via MongoDB Stitch, my ineptitude has been answered with an early Christmas gift.

My incessant complaining gained some acknowledgement from a couple of folks over at MongoDB. Perhaps the timing is simply by chance, but since then I've begun noticing something some subtleties in the Stitch documentation; namely that if you look hard enough, some of it begins to make sense. Either way, I'm chalking this one up as

Python-Lambda: The Essential Library for AWS Cloud Functions

Python-Lambda: The Essential Library for AWS Cloud Functions

Deploy AWS Lambda functions with ease by using a powerful Python library.

In our series about building AWS APIs, we've covered a lot of ground around learning the AWS ecosystem. Now that we're all feeling a bit more comfortable, it may be time to let everybody in on the world's worst-kept secret: Almost nobody builds architecture by interacting with the AWS UI directly. There are plenty examples of how this is done, with the main example being HashiCorp: an entire business model based around the premise that AWS has a shitty UI, to the point where it's easier to write code to make things which will host your code. What a world.

Building an API with Amazon's API Gateway

Building an API with Amazon's API Gateway

Leverage AWS' API Gateway to create an API which fetches records from an RDS database.

In our last adventure, we ventured off to create our very own cloud database by using Amazon's RDS service. We've also briefly covered the general concept behind what Lambda functions. In case you've already forgotten, Lambdas are basically just chunks of code in the cloud; think of them as tiny virtual servers, which have already been configured (and locked down) to serve one specific purpose. Because that's literally what it is.

The data being stored in RDS is ultimately what we're targeting, and Lambdas serve as the in-between logic to serve up, modify, or add to the proper data. The

Extract Nested Data From Complex JSON

Extract Nested Data From Complex JSON

Never manually walk through complex JSON objects again by using this function.

We're all data people here, so you already know the scenario: it happens perhaps once a day, perhaps 5, or even more. There's an API you're working with, and it's great. It contains all the information you're looking for, but there's just one problem: the complexity of nested JSON objects is endless, and suddenly the job you love needs to be put on hold to painstakingly retrieve the data you actually want, and it's 5 levels deep in a nested JSON hell. Nobody feels like much of a "scientist" or an "engineer" when half their day becomes dealing with key

Recreate JIRA Service Desk in Python & Flask

Recreate JIRA Service Desk in Python & Flask

When SaaS doesn't cut it, beat it down and take everything its got.

When it comes to SaaS products in the realm of Service desks, JIRA Service Desk is at the very least, just as powerful as the next solution (Zendesk comes to mind). This naturally begs the question: Why is JIRA Service Desk's pricing model roughly 1/10th of that of it's competitor?

The answer lies within ease of use, but more importantly, presentation. While Atlassian's cloud offering is slowly playing catchup, Atlassian as a company has never quite seemed to nail down the importance of a friendly user interface, nor the importance of this when it comes to worldwide adoption. To

More API Columns, More Problems: Easily Adding DB Variables To Match New Input

More API Columns, More Problems: Easily Adding DB Variables To Match New Input

Python, Pandas, & Functional Programming!

APIs.  They're wonderful.  For every headache they've given me, I'm glad I live in the age where hitting an API endpoint is a standard way of retrieving data - I recently had to crawl a bunch of records from the Brazilian census in 200, and that was an ordeal (but this isn't about that!).

The thing about APIs is that you generally shouldn't be hitting them all day long - generally you should be doing regular imports to a database (or whatever).  And the other thing about APIs is that they're not quite as fussy about names as databases.  There's

Hacking Tableau to Handle ETL Workflows

Hacking Tableau to Handle ETL Workflows

Weaponizing APIs against tyrannical software.

Before we get into the specifics of how to sadistically abuse Tableau, let's clear the air: there's something about inaccessible, expensive, proprietary enterprise software that tends to put me in a touchy mood. As we know, B2B software pricing has nothing to do with code quality or even value-add, but rather the tendency of businesses to create time-based urgencies without warning; the kinds of urgencies which may be solved by, say, a tool of sorts.

My first interaction with Tableau actually took place after I had committed myself to the cult of Python's Pandas library and all that comes with

Automagically Turn JSON into Pandas DataFrames

Automagically Turn JSON into Pandas DataFrames

Let Pandas do the heavy lifting for you when turning JSON into a DataFrame.

In his post about extracting data from APIs, Todd demonstrated a nice way to massage JSON into a pandas DataFrame. This method works great when our JSON response is flat, because dict.keys() only gets the keys on the first "level" of a dictionary. It gets a little trickier when our JSON starts to become nested though, as I experienced when working with Spotify's API via the Spotipy library. For example, take a look at a response from their https://api.spotify.com/v1/tracks/{id} endpoint:

import spotipy
from spotipy.oauth2 import SpotifyClientCredentials

spotify_client_id = 'YOUR_ID'
spotify_
Extract Massive Amounts of Data from APIs in Python

Extract Massive Amounts of Data from APIs in Python

Abusing REST APIs for all they’re worth.

Taxation without representation. Colonialism. Not letting people eat cake. Human beings rightfully meet atrocities with action in an effort to change the worked for the better. Cruelty by mankind justifies revolution, and it is this writer’s opinion that API limitations are one such cruelty.

The data we need and crave is stashed in readily available APIs all around us. It’s as though we have the keys to the world, but that power often comes with a few caveats:

  • Your “key” only lasts a couple of hours, and if you want another one, you’ll have to use some
Working with XML tree data in Python

Working with XML tree data in Python

Make use of Python's native XML library to walk through and extract data.

Life is filled with things we don't want to do; you're a developer so you probably understand this to a higher degree than most people. Sometimes we waste weeks of our lives thanks to an unreasonable and unknowledgeable stakeholder. Other times, we need to deal with XML trees.

At some point or another you're going to need to work with an API that returns information in XML format. "Sure," we might think, "I'll just import the standard Python XML package, pick up some syntax nuances, and be on my way." That's what I thought too. Today we're going to look