Last episode we covered every programming noob's favorite 'A-ha' moment: making GET requests using AJAX. Stepping stones such as these can serve as great turning points in a career, but they also expose how little we still know. For instance, when we integrated the functional logic of APIs on the client side, we actually broke a cardinal rule: storing and passing private keys on the client side like an idiot. Does that make everything we learned useless? Not entirely, but kinda yeah.

Today we'll do the equivalent in Python by using the requests library. Requests is successor to Urllib, both of which are simple tools to retrieve or modify information on the web, most commonly in the case of APIs.

We'll be using JIRA's API as an example of how to format GET and POST requests. JIRA's API in particular is an excellent example of a powerful and useful API. There's a ton we can do, thus a perfect demonstration of how much power one library can give you.

Batteries Not Included

Even I sometimes forget that requests is not a built-in Python library. Make sure requests is installed in your environment via pip install requests.

Create a file in your directory called config.py to store your credentials. Make sure to add that file to your .gitignore if you plan on committing anything any time soon.

# creds.py
username = fake.user
password = securepassword123
config.py

The only libraries we need to import are requests and json. Make sure you import your credentials from the file you created earlier.

import requests
import json
from creds import username, password
main.py

GET Requests

As long as you have a URL, you can make a GET request. The requests library will return the content of any page it hits; if you make a request to an HTML page, your response will be that page's HTML source.

When we know what sort of data we're expecting to receive back, we can specify the expected content type by passing the headers argument, and specifying the Content-Type. Authentication is handled via passing arguments as well, specifically the auth argument. Take a look at what you can pass in a GET request:

Common GET Arguments

  • url: The URL we will either retrieve or pass the information along to.
  • parameters (optional):  Depending on the API, some URLs can accept a dictionary of variables to be passed along with the URL. These are called query strings; you notice these all the time whenever you come across a URL that looks like nonsense... that nonsense is information!
  • headers (optional): A collection of metadata sent along with the request. Our browsers send HTTP headers every time we visit a site, but the scope of what a header value might cover ranges from tokens to content types.
  • auth (optional):  Method for logging in if needed. Basic/Digest/Custom HTTP Auth.

Let’s GET Some

We're going to make a relatively simple request to pull open tickets from a JIRA project called EXM.

This request will:

  • Accept our destination's base URL
  • Append 'search/' (the endpoint for searching issues)
  • Pass two parameters:  A query to return issues A flag to show the issue history
  • Authenticate with our username/password
  • Print the result
import requests
import json
from creds import username
from creds import password

base_url = 'https://examplejira.com/rest/api/2/'
headers = {'Content-Type': 'application/json'}
params = {
    'jql': 'project = EXM AND resolution is not EMPTY',
    'expand': 'changelog',
}

req = requests.get(base_url + 'search/', headers=headers, params=params, auth=(username, password))

print(req.content)
main.py

Notice that setting a variable equal to the request will equal the result of that request. Printing r alone would return a numerical status code (200, 404, etc). The response that comes back from request such as r are actually complex objects — printing r.json() will display the contents of the response as a JSON object. Alternatively, r.text returns the raw response as a string.

If your response comes back with an error, remember that you can always debug your requests via Postman.

If all went well with our request, r.json() should return something similar to the following:

{  
   "expand":"schema,names",
   "startAt":0,
   "maxResults":50,
   "total":63,
   "issues":[  
      {  
         "expand":"operations,versionedRepresentations,editmeta,changelog,renderedFields",
         "id":"10558",
         "self":"https://hackersandslackers.atlassian.net/rest/api/2/issue/10558",
         "key":"HSB-63",
         "fields":{  
            "issuetype":{  
               "self":"https://hackersandslackers.atlassian.net/rest/api/2/issuetype/10007",
               "id":"10007",
               "description":"Non-development related content task",
               "iconUrl":"https://hackersandslackers.atlassian.net/secure/viewavatar?size=xsmall&avatarId=10306&avatarType=issuetype",
               "name":"Content",
               "subtask":false,
               "avatarId":10306
            },
         }
    ]
}
Response JSON

The entirety of the request is probably much longer (depending on how many issues you have). Notice how JIRA will only return a maximum of 50 results unless otherwise specified (this is one of the parameters they accept). Feel free to check out JIRA's API documentation to see what else you can do, but be warned: their docs kind of suck.

Retrieving information is cool, but modifying it is even better. Here's a use case which might be immediately useful: creating a user.

POST Requests

In addition to the arguments GET requests can receive, POST requests can also accept arguments like as data. This is where we tell the API the specifics of what we're trying to do.

Common POST Arguments

  • url: Endpoint URL.
  • params (optional): Dictionary of variables to be passed as parameters of a query string.
  • body (optional): A JSON or  ML object sent in the body of the Request.
  • headers (optional):  Dictionary of HTTP Headers to send with the Request.
  • auth (optional):  Auth to enable Basic/Digest/Custom HTTP Auth.

Let There be Users

The main difference between this request and the last will be what we pass via the data argument. For example's sake we'll be creating a user named bro with the appropriate broiest details.

Take special note of json.dumps(userdata). If an endpoint is expecting JSON (it probably is) we need to explicitly convert our dictionary of values to JSON before making this request.

import requests
import json
from creds import username
from creds import password


base_url = "https://examplejira.com/rest/api/2/"
headers = {'Content-Type': 'application/json'}
userdata = {
  'username': 'bro',
  'name': 'Bro',
  'password': '32456456',
  'email': 'bro@broiest.com',
  "notification" : "true"
}

req = requests.post(base_url + 'user/', data=json.dumps(userdata), headers=headers, auth=(username, password))

print(req.content)
main.py

You just created a user. That's basically like giving birth to a child. Congratulations.

Advanced POST Requests

As fun as it is to create bro users in JIRA instances, one-off usage of APIs like this isn't really useful. We haven't done anything that we couldn't have just done ourselves via the UI.

To spice things up, here's a very real use case: importing a list of users via a CSV. As we speak, people in corporations around the world are manually adding thousands of users by hand to internal SaaS products. Don't be that person.

This request will do the following:

  • Use pandas to open users.csv  (presumably this CSV should have columns for name, email, etc)  
  • Generate a random password using secrets
  • Use the CSV to create accounts with each user's information
  • Output the result to users_created.csv
"""JIRA User Import"""
import pandas as pd
import requests
import secrets
import json

# store credentials
from creds import username
from creds import password

# dataframe from csv
user_df = pd.read_csv('users.csv')

# store results of import
rows_list = []

headers = {'Content-Type': 'application/json'}
base_url = "https://examplejira.com/rest/api/2/"

# generate 20-character password
def generate_password():
    alphabet = string.ascii_letters + string.digits
    password = ''.join(secrets.choice(alphabet) for i in range(20))
    return password

# iterate and create users
for index, row in user_df.iterrows():
    userdata = {
        "name": row['email'].split('@')[0],
        "password": generate_password(),
        "emailAddress": row['email'],
        "displayName": row['name'],
        "notification" : "true"
    }
    req = requests.post(base_url + 'user/', data=json.dumps(userdata), headers=headers, auth=(jirauser, password))
    rows_list.append(userdata) # adds row to array to be tracked
    # create & export results to a csv
    users_imported_df = pd.DataFrame(rows_list)
    users_imported_df.to_csv('users_created.csv')
main.py

If this worked for you, take a moment to put something in perspective: you just automated somebody's entire 9-5 job in a few minutes.

Also feel free to reflect on our purpose as a species. If automating this was so straightforward, why do so many of us choose not to automate more tasks? Is our entire economy a hoax created to grant the masses an illusion of free will? Are we running around in circles trying to solve problems we create ourselves, to pay the bills which come with being employed? Finally: if robots are clearly this superior, is there a purpose for the human race at all?

Now you're asking the real questions. Hail Megatron.