Hackers and Slackers

Read/Write to Private S3 Buckets in NodeJS

On the off-chance you were wondering how to do this

We here at H+S are dedicated to one simple cause: creating posts about oddly specific programming scenarios. Somewhere in the world as sad soul is looking to programmatically access files from an S3 server while keeping their bucket private. To that person: we heard you.

There are plenty of reasons you'd want to access files in S3. For example, let's say you read that post about using pandas in a Lambda function. Since you're already familiar with PyMySQL, you may hypothetically be in a position to export data from a DB query to a CSV saved in S3. I bet you can guess what I've been doing lately.

Configure AWS on your VPS

The easiest way to get this is done is via the AWS CLI:

pip3 install awscli

This allows us to run AWS configure:

aws configure
AWS Access Key ID [None]: YOURACCESSKEY
AWS Secret Access Key [None]: YOURSECRETKEY
Default region name [None]: us-east-2
Default output format [None]: json

This creates a couple config files for us which are globally accessible by all applications on our VPS. This is great because it enables us to keep our keys out of our source, thus avoiding any accidental commits.

If we never need to modify thee files, they can be found here:

vim ~/.aws/credentials
vim ~/.aws/config

Node Time

We'll assume you have an app set up with some basic routing, such as the barebones ExpressJS set up.

In your app we'll need to add 2 dependencies:

npm install --save aws-sdk
npm install --save aws-config

Now we'll create a route.

var awsConfig = require('aws-config');
var AWS = require('aws-sdk');

router.get('/export', function(req, res, next) {
    var file = 'df.csv';
    console.log('Trying to download file', fileKey);

    var s3 = new AWS.S3({});

    var options = {
        Bucket: 'your-bucket-name',
        Key: file,
    };

    s3.getObject(options, function(err, data) {
      res.attachment(file);
      res.send(data.Body);
  });
});

Setting new AWS.S3({}) will automatically look to our AWS credentials file for our access and secret keys. This can be overridden if needed:

var s3 = new AWS.S3({
    'AccessKeyID': 'YOURACCESSKEY', 
    'SecretAccessKey': 'YOURSECRETACCESSKEY', 
    'Region': 'YOUR REGION'
});

Honestly that's pretty much it: this route will prompt a download of the target file upon hitting the route. As much as I'm sure we'd all love to sit here and go through more complicated use cases, let's just avoid Callback Hell altogether and enjoy the rest of our day.

Hell will have to wait until next time.

Author image
New York City Website
Product manager turned engineer with ongoing identity crisis. Breaks everything before learning best practices. Completely normal and emotionally stable.

Product manager turned engineer with ongoing identity crisis. Breaks everything before learning best practices. Completely normal and emotionally stable.