AWS

Cloud behemoth and mother of microservices. Become familiar with AWS products, account administration, security practices, and tips to make it all easier.
Python
18 Feb 2019

S3 File Management With The Boto3 Python SDK

Modify and manipulate thousands of files in your S3 (or DigitalOcean) Bucket.
S3 File Management With The Boto3 Python SDK

It's incredible the things human beings can adapt to in life-or-death circumstances, isn't it? In this particular case it wasn't my personal life in danger, but rather the life of this very blog. I will allow for a brief pause while the audience shares gasps of disbelief. We must stay strong and collect ourselves from such distress.

Like most things I despise, the source of this unnecessary headache was a SaaS product. I won't name any names here, but it was Cloudinary. Yep, totally them. We'd been using their (supposedly) free service for hosting our blog's images for about a

Continue Reading
AWS
11 Nov 2018

Reselling AWS Load Balancing

Providing Cloud Load Balancing for your customers; My ultimatum.
Reselling AWS Load Balancing

Let's say we have a hosting service for users who bring their own domain name. In this scenario we'd like to be able to service customers no matter who manages their DNS records. Be it GoDaddy, Namecheap, Google, Hostgator, some offshore place, etc.

At the same time, we'd also like to provide Load balancing so no one-user can overload any of our systems. This means, instead of having a customer's domain name point directly to the system where their webserver or app resides, it will point the HTTP connection to a Load Balancer which is prepared to handle serious connection

Continue Reading
AWS
07 Nov 2018

Python-Lambda: The Essential Library for AWS Cloud Functions

Deploy AWS Lambda functions with ease with the help of a single Python library.
Python-Lambda: The Essential Library for AWS Cloud Functions

In our series about building AWS APIs, we've covered a lot of ground around learning the AWS ecosystem. Now that we're all feeling a bit more comfortable, it may be time to let everybody in on the world's worst-kept secret: Almost nobody builds architecture by interacting with the AWS UI directly. There are plenty examples of how this is done, with the main example being HashiCorp: an entire business model based around the premise that AWS has a shitty UI, to the point where it's easier to write code to make things which will host your code. What a world.

Continue Reading
AWS
29 Oct 2018

Create a REST API Endpoint Using AWS Lambda

Use Python and MySQL to build a serverless endpoint.
Create a REST API Endpoint Using AWS Lambda

Now that you know your way around API Gateway, you have the power to create vast collections of endpoints. If only we could get those endpoints to actually receive and return some stuff.

We'll create a GET function which will solve the common task of retrieving data from a database. The sequence will look something like:

  • Connect to the database
  • Execute the relevant SQL query
  • Map values returned by the query to a key/value dictionary
  • Return a response body containing the prepared response

To get started, create a project on your local machine (this is necessary as we'll need

Continue Reading
AWS
29 Oct 2018

Building an API with Amazon's API Gateway

Building APIs: The final frontier of cool-stuff-to-do-in-AWS.
Building an API with Amazon's API Gateway

In our last adventure, we ventured off to create our very own cloud database by using Amazon's RDS service. We've also briefly covered the general concept behind what Lambda functions. In case you've already forgotten, Lambdas are basically just chunks of code in the cloud; think of them as tiny virtual servers, which have already been configured (and locked down) to serve one specific purpose. Because that's literally what it is.

The data being stored in RDS is ultimately what we're targeting, and Lambdas serve as the in-between logic to serve up, modify, or add to the proper data. The

Continue Reading
DevOps
03 Oct 2018

Creating an AMI with HashiCorp Packer

HashiCorp's version control for infrastructure .
Creating an AMI with HashiCorp Packer

Why use Packer? Infrastructure as code has become part of the buzzword bingo surrounding operational teams and their desired optimal workflows.

One could theoretically just start with a base AMI and manually update it and then re-save it as a new AMI, but this process is not repeatable.  We can check in our desired infrastructure states as code to version control.  This is good practice for change control management.  We can readily see what worked before and what was changed in the latest update.  If something catastrophic happens or we encounter unforeseen issues, we can always rollback to a previous

Continue Reading
NodeJS
22 Jun 2018

Read and Write to S3 Buckets via NodeJS

Node’s most popular package interacting with the most popular file store on the world’s most popular cloud.
Read and Write to S3 Buckets via NodeJS

We here at H+S are dedicated to one simple cause: creating posts about oddly specific programming scenarios. Somewhere in the world as sad soul is looking to programmatically access files from an S3 server while keeping their bucket private. To that person: we heard you.

There are plenty of reasons you'd want to access files in S3. For example, let's say you read that post about using Pandas in a Lambda function. Since you're already familiar with PyMySQL, you may hypothetically be in a position to export data from a DB query to a CSV saved in S3. I

Continue Reading
AWS
21 Jun 2018

Using Pandas with AWS Lambda Functions

Forcefully use the Pandas library in your AWS Lambda functions.
Using Pandas with AWS Lambda Functions

In one corner we have Pandas: Python's beloved data analysis library. In the other, AWS: the unstoppable cloud provider we're obligated to use for all eternity. We should have known this day would come.

While not the prettiest workflow, uploaded Python package dependencies for usage in AWS Lambda is typically straightforward. We install the packages locally to a virtual env, package them with our app logic, and upload a neat CSV to Lambda. In some cases this doesn't always work: some packages result in a cryptic error message with absolutely no helpful instruction. Pandas is one of those packages.

Why

Continue Reading
AWS
06 Jun 2018

AWS Adventures: Why Can't I SSH Into My Damn EC2 Instance?

Watch as I struggle with the cloud! My pain, your gain!
AWS Adventures: Why Can't I SSH Into My Damn EC2 Instance?

Sometimes I find myself responsible for setting up and maintaining my own infrastructure for doing data stuff. In light of this, I've taken it upon myself to learn more about AWS in order for this experience to be less terrifying. I've been following a course on Udemy for the AWS Solutions Architect Certification. So far, so good.

Cut to the EC2 section. Has you spin up a little EC2 instance and SSH into it. I would find myself SSHing in, then after a few minutes my connection would be reset and then any further attempts would be met with a

Continue Reading
REST APIs
06 May 2018

Preparing your AWS Project to Build an API

Get your AWS instance configured, and become familiar with the services needed to build APIs.
Preparing your AWS Project to Build an API

There comes a surreal moment in nearly every profession in which perspective violently forces itself into our self-awareness. People with cooler jobs probably have that moment when they save their first patient or launch their first rocket. For me, the idea of building an API was this moment in software development. All those past black boxes which spat out results your life depended on: we can make those now.

Feel free to remain unfazed by this as I'm sure most are... for those of us who can't remember how they became an "engineer" in the first place, API design feels

Continue Reading
MySQL
30 Apr 2018

MySQL on the Cloud with AWS RDS

Spinning up a standalone MySQL Database with Amazon.
MySQL on the Cloud with AWS RDS

Last time we became familiar with the handiwork of setting up MySQL locally, navigating databases via command line, and exposing your database to external access. While badass, it has come to my attention that most people don't bother doing things this way. Unless you're getting deep into some heavy architecture, most people opt to use cloud services such as AWS to set up databases which are intended to be interacted with by multiple services.

A perfect example is one we ran into over the weekend while working on this very blog. We're running a Ghost instance, which is respectably complex

Continue Reading