Cloud providers have made a killing from neatly-packaged managed services for years. Whether it be databases or message brokers, developers like ourselves don't seem to have a problem paying a bit extra to have things taken care of. But wait, aren't we typically the last people to opt for less optimization and less control? Why is this the time we decide otherwise? If I had to make a guess, I'd wager it's because server-side DevOps kind of sucks.

As a developer, configuring or debugging a VPS is usually work which is unaccounted for, and it isn't particularly rewarding. At best, your application will probably end up running the same as your local environment. How could we make this inevitable part of our jobs better? Well, we could automate  it.

paramiko and scp are two Python libraries we can use together to automate tasks we'd want to run on a remote host such as restarting services, making updates, or grabbing log files. We're going to take a look at what scripting with these libraries looks like. Fair warning: there's a sizable amount of code in this tutorial, which tends to make me excited enough to coerce others into my manic code-tutorial episodes. If you ever begin to feel lost, the full repo can be found here:

hackersandslackers/paramiko-tutorial
📡🐍SSH & SCP in Python with Paramiko. Contribute to hackersandslackers/paramiko-tutorial development by creating an account on GitHub.

Setting up SSH Keys

To authenticate an SSH connection, we need to set up a private RSA SSH key (not to be confused with OpenSSH). We can generate a key using the following command:

$ ssh-keygen -t rsa

This will prompt us to provide a name for our key. Name it whatever you like:

Generating a public/private rsa key pair.
Enter the file in which you wish to save they key (i.e., /home/username/.ssh/id_rsa):

Next, you'll be prompted to provide a password (feel free to leave this blank).

Now that we have our key, we need to copy this to our remote host. The easiest way to do this is by using ssh-copy-id:

$ ssh-copy-id -i ~/.ssh/mykey username@my_remote_host.org

Verifying our SSH Key

If you'd like to check which keys you already have, these can be found in your system's .ssh directory:

$ cd ~/.ssh

We're looking for keys which begin with the following header:

-----BEGIN RSA PRIVATE KEY-----

...

-----END RSA PRIVATE KEY-----
id_rsa

Feel free to do the same on your FPS.

Starting our Script

Let's install our libraries. Fire up whichever virtual environment you prefer and let em rip:

$ pip3 install paramiko scp

Just one more thing before we write some meaningful Python code! Create a config file to hold the variables we'll need to connect to our host. Here are the barebones of what we need to get into our server:

  • Host: The IP address or URL of the remote host we're trying to access.
  • Username: This is the username you use to SSH into your server.
  • Passphrase (optional): If you specified a passphrase when you created your ssh key, specify that here. Remember that your SSH key passphrase is not the same as your user's password.
  • SSH Key: The file path of the key we created earlier. On OSX, these live in your system's ~/.ssh folder. SSH key we're targeting must have an accompanying key with a .pub file extension. This is our public key; if you were following along earlier, this should have already been generated for you.

If you're trying to upload or download files from your remote host, you'll need to include two more variables:

  • Remote Path: The path to the remote directory we're looking to target for file transfers. We can either upload things to this folder or download the contents of it.
  • Local Path: Same idea as above, but the reverse. For our convenience, the local path we'll be using is simply /data, and contains pictures of cute fox gifs.

Now we have everything we need to make a respectable config.py file:

"""Configuration module."""
from os import environ

host = environ.get('REMOTE_HOST')
user = environ.get('REMOTE_USERNAME')
ssh_key_filepath = environ.get('SSH_KEY')

remote_path = environ.get('REMOTE_PATH')
local_path = '/data'
config.py

Creating an SSH Client

We're going to create a class called RemoteClient to handle the interactions we'll be having with our remote host. Before we get too fancy, let's just start things off by instantiating the RemoteClient class with the variables we created in config.py:

"""Remote host object to handle connections and actions."""

class RemoteClient:
    """Client to interact with a remote host via SSH & SCP."""

    def __init__(self, host, user, ssh_key_filepath, remote_path):
        self.host = host
        self.user = user
        self.ssh_key_filepath = ssh_key_filepath
        self.remote_path = remote_path
client.py

Nothing impressive so far: we've just set some variables and passed them into a useless class. Let's kick things up a notch without leaving our constructor:

"""Remote host object to handle connections and actions."""
from paramiko import SSHClient, AutoAddPolicy, RSAKey
from paramiko.auth_handler import AuthenticationException, SSHException


class RemoteClient:
    """Client to interact with a remote host via SSH & SCP."""

    def __init__(self, host, user, ssh_key_filepath, remote_path):
        self.host = host
        self.user = user
        self.ssh_key_filepath = ssh_key_filepath
        self.remote_path = remote_path
        self.client = None
        self.scp = None
        self.__upload_ssh_key()
client.py

We've added three new things to be instantiated with our class:

  • self.client = None: self.client will ultimately serve as the connection objection in our class, similar to how you have dealt with terminology like conn in database libraries. Our connection will be None until we explicitly connect to our remote host.
  • self.scp = None Similar to self.client, but exclusively handles connections for transferring files.
  • self.__upload_ssh_key() isn't a variable, but rather a function to be run automatically whenever our client is instantiated. Calling __upload_ssh_key() is telling our RemoteClient object to check for local ssh keys immediately upon creation so that we can try to pass them to our remote host. Otherwise, we wouldn't be able to establish a connection at all.

Uploading SSH Keys to a Remote Host

We've reached the section of this exercise where we need to knock out some devastatingly inglorious boilerplate code. This is typically where emotionally inferior individuals succumb to the sheer dull obscurity of understanding SSH keys and maintaining connections. Make no mistake: authenticating and managing connections to anything programmatically is overwhelmingly dull... unless your tour guide happens to be an enchanting wordsmith, serving as your loving protector through perilous obscurity. Some people call this post a tutorial. I intend to call it art.

RemoteClient will start with two private methods: __get_ssh_key()  and __upload_ssh_key(). The former will fetch a locally stored public key, and if successful, the latter will deliver this public key to our remote host as an olive branch of access. Once a locally created public key exists on a remote machine, that machine will then forever trust us our requests to connect to it: no passwords required. We'll be including proper logging along the way, just in case we run into any trouble:

import sys
from loguru import logger
from os import system
from paramiko import SSHClient, AutoAddPolicy, RSAKey
from paramiko.auth_handler import AuthenticationException, SSHException
from scp import SCPClient, SCPException


logger.add(sys.stderr,
           format="{time} {level} {message}",
           filter="client",
           level="INFO")
logger.add('logs/log_{time:YYYY-MM-DD}.log',
           format="{time} {level} {message}",
           filter="client",
           level="ERROR")


class RemoteClient:
    """Client to interact with a remote host via SSH & SCP."""

    ...

    def __get_ssh_key(self):
        """Fetch locally stored SSH key."""
        try:
            self.ssh_key = RSAKey.from_private_key_file(self.ssh_key_filepath)
            logger.info(f'Found SSH key at self {self.ssh_key_filepath}')
        except SSHException as error:
            logger.error(error)
        return self.ssh_key

    def __upload_ssh_key(self):
        try:
            system(f'ssh-copy-id -i {self.ssh_key_filepath} {self.user}@{self.host}>/dev/null 2>&1')
            system(f'ssh-copy-id -i {self.ssh_key_filepath}.pub {self.user}@{self.host}>/dev/null 2>&1')
            logger.info(f'{self.ssh_key_filepath} uploaded to {self.host}')
        except FileNotFoundError as error:
            logger.error(error)
client.py

__get_ssh_key() is quite simple: it verifies that an SSH key exists at the path we specified in our config to be used for connecting to our host. If the file does in fact exist, we happily set our self.ssh_key variable, so this key can be uploaded and used by our client from here forward. Paramiko provides us with a submodule called RSAKey to easily handle all things RSA key related, like parsing a private key file into a usable connection authentication. That's what we get here:

 RSAKey.from_private_key_file(self.ssh_key_filepath)

If our RSA key were incomprehensible nonsense instead of a real key, Paramiko's SSHException would have caught this and raised an exception early on explaining just that. Properly utilizing a library's error handling takes a lot of the guesswork out of "what went wrong," especially in cases like where there's potential for numerous unknowns in a niche space neither of us mess with often.

__upload_ssh_key() is where we get to jam our SSH key down the throat of our remote server while shouting, "LOOK! YOU CAN TRUST ME FOREVER NOW!" To accomplish this, I go a bit "old school" by passing bash commands via Python's os.system. Unless somebody makes me aware of a cleaner approach in the comments, I'll assume this is the most badass way to handle passing keys to a remote server.

The standard non-Python way of passing keys to a host looks like this:

ssh-copy-id -i ~/.ssh/mykey user@host

This is precisely what we accomplish in our function in Python, which looks like this:

system(f'ssh-copy-id -i {self.ssh_key_filepath} {self.user}@{self.host}>/dev/null 2>&1')

I suppose you won't let me slip that /dev/null 2>&1 bit by you? Fine. If you must know, here's some guy on StackOverflow explaining it better than I can:

> is for redirect /dev/null is a black hole where any data sent, will be discarded. 2 is the file descriptor for Standard Error. > is for redirect. & is the symbol for file descriptor (without it, the following 1 would be considered a filename). 1 is the file descriptor for Standard O.

So we're basically telling our remote server we're giving it something, and it's all like "where do I put this thing," to which we reply "nowhere in physical in space, as this is not an object, but rather an eternal symbol of our friendship.  Our remote host is then flooded with gratitude and emotion, because yes, computers do have emotions, but we can't be bothered by that right now.

Connecting to our Client

We'll add a method to our client called connect() to handle connecting to our host:

...

class RemoteClient:
    """Client to interact with a remote host via SSH & SCP."""

    ...

    def __connect(self):
        """Open connection to remote host."""
        try:
            self.client = SSHClient()
            self.client.load_system_host_keys()
            self.client.set_missing_host_key_policy(AutoAddPolicy())
            self.client.connect(self.host,
                                username=self.user,
                                key_filename=self.ssh_key_filepath,
                                look_for_keys=True,
                                timeout=5000)
            self.scp = SCPClient(self.client.get_transport())  # For later
        except AuthenticationException as error:
            logger.info('Authentication failed: did you remember to create an SSH key?')
            logger.error(error)
            raise error
        finally:
            return self.client
client.py

Let's break this down:

  • client = SSHClient() sets the stage for creating an object representing our SSH client. The following lines will configure this object to make it more useful.
  • load_system_host_keys() instructs our client to look for all the hosts we've connected to in the past by looking at our system's known_hosts file and finding the SSH keys our host is expecting. We've never connected to our host in the past, so we need to specify our SSH key explicitly.
  • set_missing_host_key_policy() tells Paramiko what to do in the event of an unknown key pair. This is expecting a "policy" built-in to Paramiko, to which we're going to specific AutoAddPolicy(). Setting our policy to "auto-add" means that if we attempt to connect to an unrecognized host, Paramiko will automatically add the missing key locally.
  • connect() is SSHClient's most important method (as you might imagine). We're finally able to pass our host, user, and SSH key to achieve what we've all been waiting for: a glorious SSH connection into our server! The connect() method allows for a ton of flexibility via a vast array of optional keyword arguments too. I happen to pass a few here: setting look_for_keys to True gives Paramiko permission to look around in our ~/.ssh folder to discover SSH keys on its own, and setting timeout will automatically close connections we'll probably forget to close. We could even pass variables for things like port and password, if we had elected to connect to our host this way.

Disconnecting

We should close connections to our remote host whenever we're done using them. Failing to do so might not necessarily be disastrous, but I've had a few instances where enough hanging connections would eventually max out inbound traffic on port 22. Regardless of whether your use case might consider a reboot to be a disaster or mild inconvenience, let's just close our damn connections like adults as though we were wiping our butts after pooping. No matter your connection hygiene, I advocate for setting a timeout variable (as we saw earlier). Anyway. voila:

class RemoteClient:
    ...

    def disconnect(self):
        """Close ssh connection."""
        self.client.close()
        self.scp.close()  # Coming later
client.py

Fun fact: setting self.client.close() actually sets self.client to equal None, which is useful in cases where you might want to check if a connection is already open.

Executing Unix Commands

We now have a wonderful Python class that can find RSA keys, connect, and disconnect. It does lack the ability to do, well, anything useful.

We can fix this and finally begin doing "stuff" with a brand new method to execute commands, which I'll aptly dub execute_commands() (that's correct, "commands" as in potentially-more-than-one, we'll touch on that in a moment). The legwork of all this is done by the Paramiko client's built-in exec_command() method, which accepts a single string as a command and executes it:

class RemoteClient:
    ...

    def execute_commands(self, commands):
        """Execute multiple commands in succession."""
        if self.client is None:
            self.client = self.__connect()
        for cmd in commands:
            stdin, stdout, stderr = self.client.exec_command(cmd)
            stdout.channel.recv_exit_status()
            response = stdout.readlines()
            for line in response:
                logger.info(f'INPUT: {cmd} | OUTPUT: {line}')
client.py

The function we just created execute_commands() expects a list of strings to execute as commands. That's partially for convenience, but it's also because Paramiko won't run any "state" changes (like changing directories) between commands, so each command we pass to Paramiko should assume we're working out of our server's root. I took the liberty of passing three such commands like so:

remote.execute_commands(['cd /var/www/ && ls',
                        'tail /var/log/nginx/access.log',
                        'ps aux | grep node'])
init.py

I can view the contents of a directory by chaining cd path/to/dir && ls, but running cd path/to/dir followed by ls would result in nothingness because ls the second time returns the list of files in our server's root.

You'll notice client.exec_command(cmd) returns three values as opposed to one: this can be useful to see which input produced which output. For example, here are the full logs for the example I provided where I passed three commands to remote.execute_commands():

2020-01-02 23:20:16.103 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: django
2020-01-02 23:20:16.103 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: ghost
2020-01-02 23:20:16.103 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: hackers-hbs
2020-01-02 23:20:16.103 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: html
2020-01-02 23:20:16.104 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: hustlers
2020-01-02 23:20:16.104 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: pizza
2020-01-02 23:20:16.104 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: toddbirchard

2020-01-02 23:20:16.196 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 134.209.37.49 - - [03/Jan/2020:03:42:34 +0000] "GET / HTTP/2.0" 404 139 "-" "Mozilla/5.0 (compatible; NetcraftSurv
eyAgent/1.0; +info@netcraft.com)"
2020-01-02 23:20:16.196 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 54.36.148.187 - - [03/Jan/2020:03:43:49 +0000] "GET /robots.txt HTTP/1.1" 404 149 "-" "Mozilla/5.0 (compatible; Ah
refsBot/6.1; +http://ahrefs.com/robot/)"
2020-01-02 23:20:16.196 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 54.36.150.104 - - [03/Jan/2020:03:43:50 +0000] "GET / HTTP/1.1" 404 139 "-" "Mozilla/5.0 (compatible; AhrefsBot/6.
1; +http://ahrefs.com/robot/)"
2020-01-02 23:20:16.196 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 46.229.168.146 - - [03/Jan/2020:03:48:49 +0000] "GET /robots.txt HTTP/1.1" 200 92 "-" "Mozilla/5.0 (compatible; Se
mrushBot/6~bl; +http://www.semrush.com/bot.html)"
2020-01-02 23:20:16.197 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 46.229.168.153 - - [03/Jan/2020:03:48:50 +0000] "GET /the-art-of-technical-documentation/ HTTP/1.1" 200 9472 "-" "
Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)"
2020-01-02 23:20:16.197 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 157.55.39.171 - - [03/Jan/2020:03:50:20 +0000] "GET /robots.txt HTTP/1.1" 200 94 "-" "Mozilla/5.0 (compatible; bin
gbot/2.0; +http://www.bing.com/bingbot.htm)"
2020-01-02 23:20:16.197 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 40.77.167.220 - - [03/Jan/2020:03:50:24 +0000] "GET / HTTP/1.1" 200 3791 "-" "Mozilla/5.0 (compatible; bingbot/2.0
; +http://www.bing.com/bingbot.htm)"
2020-01-02 23:20:16.197 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 54.36.150.64 - - [03/Jan/2020:03:56:47 +0000] "GET /the-ruin-of-feeling-powerless/ HTTP/2.0" 200 9605 "-" "Mozilla
/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)"
2020-01-02 23:20:16.197 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 54.36.150.187 - - [03/Jan/2020:04:05:01 +0000] "GET /bigquery-and-sql-databases/ HTTP/2.0" 404 146 "-" "Mozilla/5.
0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)"
2020-01-02 23:20:16.197 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 46.229.168.149 - - [03/Jan/2020:04:13:41 +0000] "GET /robots.txt HTTP/1.1" 502 182 "-" "Mozilla/5.0 (compatible; S
emrushBot/6~bl; +http://www.semrush.com/bot.html)"

2020-01-02 23:20:16.322 | paramiko_tutorial.client:execute_cmd:95 - INPUT: ps aux | grep node | OUTPUT: ghost     1354 39.2  3.4 1223844 140172 ?      Sl   04:20   0:05 /usr/bin/node current/index.js
2020-01-02 23:20:16.323 | paramiko_tutorial.client:execute_cmd:95 - INPUT: ps aux | grep node | OUTPUT: ghost     1375 36.8  3.3 1217696 135548 ?      Sl   04:20   0:04 /usr/bin/node current/index.js
2020-01-02 23:20:16.323 | paramiko_tutorial.client:execute_cmd:95 - INPUT: ps aux | grep node | OUTPUT: ghost     1395 46.8  3.6 1229824 147384 ?      Sl   04:20   0:06 /usr/bin/node current/index.js
2020-01-02 23:20:16.323 | paramiko_tutorial.client:execute_cmd:95 - INPUT: ps aux | grep node | OUTPUT: ghost     1410 37.7  3.2 1216320 132912 ?      Sl   04:20   0:04 /usr/bin/node current/index.js
2020-01-02 23:20:16.323 | paramiko_tutorial.client:execute_cmd:95 - INPUT: ps aux | grep node | OUTPUT: root      1848  0.0  0.0  13312  3164 ?        Ss   04:20   0:00 bash -c ps aux | grep node
2020-01-02 23:20:16.323 | paramiko_tutorial.client:execute_cmd:95 - INPUT: ps aux | grep node | OUTPUT: root      1850  0.0  0.0  14856  1104 ?        S    04:20   0:00 grep node
output

Some beautiful stuff here. Now you can see which sites are on my server, which bots are spamming me, and how many node processes I'm running.

I don't want to waste much more time on the art of executing commands, but it's worth mentioning the presence of why we call stdout.channel.recv_exit_status() after each command. Waiting for recv_exit_status() to come back after running client.exec_command() forces our commands to be run synchronously, otherwise there's a likely chance our remote machine won't be about to decipher commands as fast as we pass them along.

Uploading (and Downloading) Files via SCP

SCP refers to both the protocol for copying files to remote machines (secure copy protocol) as well as the Python library, which utilizes this. We've already installed the SCP library, so import that shit.

The SCP and Paramiko libraries complement one another to make uploading via SCP super easy. SCPClient() creates an object which expects  "transport" from Paramiko, which we provide with self.conn.get_transport(). Creating an SCP connection piggybacks off of our SSH client in terms of syntax, but these connections are separate. It's possible to close an SSH connection and leave an SCP connection open, so don't do that. Open an SCP connection like this:

self.scp = SCPClient(self.client.get_transport())

Uploading a single file is boring, so let's upload an entire directory of files instead. bulk_upload() accepts a list of file paths, and then calls __upload_single_file()

class RemoteClient:
    ...

    def bulk_upload(self, files):
        """Upload multiple files to a remote directory."""
        if self.client is None:
            self.client = self.__connect()
        uploads = [self.__upload_single_file(file) for file in files]
        logger.info(f'Uploaded {len(uploads)} files to {self.remote_path} on {self.host}')

    def __upload_single_file(self, file):
        """Upload a single file to a remote directory."""
        try:
            self.scp.put(file,
                         recursive=True,
                         remote_path=self.remote_path)
        except SCPException as error:
            logger.error(error)
            raise error
        finally:
            return file
client.py

Our method is expecting to receive two strings: the first being the local path to our file, and the second being the path of the remote directory we'd like to upload to.

SCP's put() method will upload a local file to our remote host. This will replace existing files with the same name if they happen to exist at the destination we specify. That's all it takes!

Downloading Files

The counterpart to SCP's put() is the get() method:

class RemoteClient:

    ...

    def download_file(self, file):
        """Download file from remote host."""
        if self.conn is None:
            self.conn = self.connect()
        self.scp.get(file)
client.py

Our Big Beautiful Script

We now have a sick Python class to handle SSH and SCP with a remote host... let's put it to work! The following snippet is a quick way to test what we've built so far. In short, this script looks for a local folder filled with files (in my case, I filled the folder with fox gifs 🦊).

Check out how easy it is to create a main.py that handles complex tasks on remote machines thanks to our RemoteClient class:

"""Perform tasks against a remote host."""
from config import (host,
                    user,
                    ssh_key_filepath,
                    local_file_directory,
                    remote_path)
from .files import fetch_local_files
from .client import RemoteClient


def main():
    """Initialize remote host client and execute actions."""
    remote = RemoteClient(host, user, ssh_key_filepath, remote_path)
    upload_files_to_remote(remote)
    execute_command_on_remote(remote)
    remote.disconnect()


def upload_files_to_remote(remote):
    """Upload files to remote via SCP."""
    files = fetch_local_files(local_file_directory)
    remote.bulk_upload(files)


def execute_command_on_remote(remote):
    """Execute UNIX command on the remote host."""
    remote.execute_cmd('cd /var/www/ghost ls')

main.py

Here's the output of the our upload function:

2020-01-03 00:00:27.215 | paramiko_tutorial.client:__upload_single_file:85 - Uploaded data/fox1.gif to /uploads/
2020-01-03 00:00:27.985 | paramiko_tutorial.client:__upload_single_file:85 - Uploaded data/fox2.gif to /uploads/
2020-01-03 00:00:30.015 | paramiko_tutorial.client:__upload_single_file:85 - Uploaded data/fox3.gif to /uploads/
2020-01-03 00:00:30.015 | paramiko_tutorial.client:bulk_upload:73 - Finished uploading 3 files to /uploads/ on 149.433.117.1425
Foxes uploaded successfully

It worked! Don't believe me? Why don't we check for ourselves by running remote.execute_commands(['cd /var/www/ && ls'])?

2020-01-03 00:08:55.955 | paramiko_tutorial.client:execute_commands:96 - INPUT: cd /uploads/ && ls | OUTPUT: fox1.gif
2020-01-03 00:08:55.955 | paramiko_tutorial.client:execute_commands:96 - INPUT: cd /uploads/ && ls | OUTPUT: fox2.gif
2020-01-03 00:08:55.956 | paramiko_tutorial.client:execute_commands:96 - INPUT: cd /uploads/ && ls | OUTPUT: fox3.gif
Output of cd /var/www/ && ls

There you have it. Straight from the fox's mouth.

Take It And Run With It

This is where I'd like to take a moment to thank all of you, and apologize that you're still here. I swore an oath to myself to stop posting tutorials over two thousand words long, and this one is looking to push five thousand words of nonsense. I'll work on that. New year, new me.

For your convenience, I've uploaded the source for this tutorial to Github. Feel free to take this and run with it! To close things out, I'll leave you with the meat and potatoes of the Client class we put together:

"""Remote host object to handle connections and actions."""
import sys
from loguru import logger
from os import system
from paramiko import SSHClient, AutoAddPolicy, RSAKey
from paramiko.auth_handler import AuthenticationException, SSHException
from scp import SCPClient, SCPException


logger.add(sys.stderr,
           format="{time} {message}",
           filter="client",
           level="INFO")
logger.add('logs/log_{time:YYYY-MM-DD}.log',
           format="{time} {level} {message}",
           filter="client",
           level="ERROR")


class RemoteClient:
    """Client to interact with a remote host via SSH & SCP."""

    def __init__(self, host, user, ssh_key_filepath, remote_path):
        self.host = host
        self.user = user
        self.ssh_key_filepath = ssh_key_filepath
        self.remote_path = remote_path
        self.client = None
        self.scp = None
        self.__upload_ssh_key()

    # Fetch and transfer SSH Keys.
    # ------------------------------------------------------
    def __get_ssh_key(self):
        """Fetch locally stored SSH key."""
        try:
            self.ssh_key = RSAKey.from_private_key_file(self.ssh_key_filepath)
            logger.info(f'Found SSH key at self {self.ssh_key_filepath}')
        except SSHException as error:
            logger.error(error)
        return self.ssh_key

    def __upload_ssh_key(self):
        try:
            system(f'ssh-copy-id -i {self.ssh_key_filepath} {self.user}@{self.host}>/dev/null 2>&1')
            system(f'ssh-copy-id -i {self.ssh_key_filepath}.pub {self.user}@{self.host}>/dev/null 2>&1')
            logger.info(f'{self.ssh_key_filepath} uploaded to {self.host}')
        except FileNotFoundError as error:
            logger.error(error)

    # Open and close remote SSH and SCP connections.
    # ------------------------------------------------------
    def __connect(self):
        """Open connection to remote host."""
        try:
            self.client = SSHClient()
            self.client.load_system_host_keys()
            self.client.set_missing_host_key_policy(AutoAddPolicy())
            self.client.connect(self.host,
                                username=self.user,
                                key_filename=self.ssh_key_filepath,
                                look_for_keys=True,
                                timeout=5000)
            self.scp = SCPClient(self.client.get_transport())
        except AuthenticationException as error:
            logger.info('Authentication failed: did you remember to create an SSH key?')
            logger.error(error)
            raise error
        finally:
            return self.client

    def disconnect(self):
        """Close ssh connection."""
        self.client.close()
        self.scp.close()

    # Upload or download files from host.
    # ------------------------------------------------------
    def bulk_upload(self, files):
        """Upload multiple files to a remote directory."""
        if self.client is None:
            self.client = self.__connect()
        uploads = [self.__upload_single_file(file) for file in files]
        logger.info(f'Finished uploading {len(uploads)} files to {self.remote_path} on {self.host}')

    def __upload_single_file(self, file):
        """Upload a single file to a remote directory."""
        try:
            self.scp.put(file,
                         recursive=True,
                         remote_path=self.remote_path)
        except SCPException as error:
            logger.error(error)
            raise error
        finally:
            logger.info(f'Uploaded {file} to {self.remote_path}')

    def download_file(self, file):
        """Download file from remote host."""
        if self.conn is None:
            self.conn = self.connect()
        self.scp.get(file)

    # Execute commands on your remote host.
    # ------------------------------------------------------
    def execute_commands(self, commands):
        """Execute multiple commands in succession."""
        if self.client is None:
            self.client = self.__connect()
        for cmd in commands:
            stdin, stdout, stderr = self.client.exec_command(cmd)
            stdout.channel.recv_exit_status()
            response = stdout.readlines()
            for line in response:
                logger.info(f'INPUT: {cmd} | OUTPUT: {line}')
client.py

The full source code for this tutorial can be found here:

hackersandslackers/paramiko-tutorial
📡🐍SSH & SCP in Python with Paramiko. Contribute to hackersandslackers/paramiko-tutorial development by creating an account on GitHub.