Cloud providers have made a killing from neatly-packaged managed services for years. Whether it be databases or message brokers, developers like ourselves don't seem to have a problem paying a bit extra to have things taken care of. But wait, aren't we typically the last people to opt for less optimization and less control? Why is this the time we decide otherwise? If I had to make a guess, I'd wager it's because server-side devops kind of sucks.
As a developer, configuring or debugging a VPS is usually work which is unaccounted for, and it isn't particularly rewarding. At best, your application will probably end up running the same as your local environment. How could we make this inevitable part of our jobs better? Well, we could automate it.
scp are two Python libraries we can use together to automate tasks we'd want to run on a remote host such as restarting services, making updates, or grabbing log files. We're going to take a look at what scripting with these libraries looks like.
Setting up SSH Keys
To authenticate an SSH connection, we need to set up a private RSA SSH key (not to be confused with OpenSSH). We can generate a key using the following command:
$ ssh-keygen -t rsa
This will prompt us to provide a name for our key. Name it whatever you like:
Generating a public/private rsa key pair. Enter the file in which you wish to save they key (i.e., /home/username/.ssh/id_rsa):
Next, you'll be prompted to provide a password (feel free to leave this blank).
Now that we have our key, we need to copy this to our remote host. The easiest way to do this is by using
$ ssh-copy-id -i ~/.ssh/mykey username@my_remote_host.org
Verifying our SSH Key
If you'd like to check which keys you already have, these can be found in your system's
$ cd ~/.ssh
We're looking for keys which begin with the following header:
-----BEGIN RSA PRIVATE KEY----- ...
Feel free to do the same on your FPS.
Starting our Script
Let's install our libraries. Fire up whichever virtual environment you prefer and let em rip:
$ pip3 install paramiko scp
Just one more thing before we write some meaningful Python code! Create a config file to hold the variables we'll need to connect to our host. We'll need to have three values handy:
- Host: The IP address or URL of the remote host we're trying to access.
- Username: This is the username you use to SSH into your server.
- Passphrase: If you specified a passphrase when you created your ssh key, specify that here. Remember that your SSH key passphrase is not the same as your user's password.
- SSH Key: The file path of the key we created earlier. On OSX, these live in your system's ~/.ssh folder.
Feel free to use the following to grab these variables from environment variables:
Creating an SSH Client
We're going to create a class called Client to handle the interactions we'll be having with our remote host. We'll assume the Config class is getting passed in to instantiate our client:
remote_ssh_key are self-explanatory: these are the variables we'll utilize to connect to our host in a bit.
We're going to store the object representing our client as
self.client. By storing this as a high-level variable in our class, we can create class methods that reuse our client.
self.pkey is what we'll be using to grab the SSH key we need to connect to our host. This will be the first method we add to our client:
Connecting to our Client
We'll add a method to our client called
connect() to handle connecting to our host:
Let's break this down:
client = SSHClient(): This sets the stage for creating an object representing our SSH client. The following lines will configure this object to make it more useful.
load_system_host_keys(): "Loading system keys" instructs our client to look for all the hosts we've connected to in the past by looking at our system's known_hosts file and finding the SSH keys our host is expecting. If we've never connected to our host in the past, we're going to have to specify our SSH key explicitly.
set_missing_host_key_policy(): We can tell Paramiko what to do in the event of an unknown key pair. This is expecting a "policy" built-in to Paramiko, to which we're going to specific
AutoAddPolicy(). Setting our policy to "auto-add" means that if we attempt to connect to an unrecognized host, Paramiko will automatically add the missing key locally.
Now we can connect to our client! If Paramiko was able to resolve our SSH key, all we need to pass here is our host URL and username. We could also pass variables for things like port and password, if we had elected to connect to our host this way. If there's trouble finding our key, we'll need to set this explicitly.
Getting Our SSH Key
The process of grabbing our SSH key isn't fun or interesting, but we can make it quick. We'll handle grabbing our SSH key in a new method called
There are three things happening:
- First, we grab the contents of our SSH key by opening our key file using
open(self.remote_ssh_key, 'r'), and read the contents with
- Next we stream the contents of this file using StringIO:
keyfile = StringIO(s).
- Finally, we parse the contents of the file as private RSA key using
Feel free not to dwell on this aspect of our script. The result of this is we can now modify our connect logic to explicitly pass the correct private key:
client.connect(self.remote_url, username=self.remote_user, pkey=self.pkey)
Disconnect From Our Client
Let's definitely not forget to add a
disconnect() method next:
Executing Unix Commands
Let's actually do some stuff! Now that we can connect, we can start interacting with our remote host using Python. To accomplish this, we'll add an
The first thing we want to do is check if we have an open connection to our client, and if not, go ahead and open one. This gives us the benefit of being able to execute commands without explicitly connecting each time.
cmd will be a string containing whatever unix command you'd like to execute. This can be anything you'd like! Feel free to mess around with restarting services, checking logs, or whatever it is you'd like to automate. The output returned by this command will be stored as a variable called
stout, which we'll return. Since we're going to mess with uploading files via SCP next, a good start might be creating a directory with
mkdir and setting user permissions on the resulting directory using
Depending on what you're looking to achieve, you may actually already have enough to take the ball and run with it.
exec_command() in itself gives us a lot of power!
Uploading & Downloading Files via SCP
SCP refers to the protocol for copying files to hosts (secure copy protocol) as well as the Python library which utilizes this. We've already installed the SCP library, so we can go ahead and import this. Let's see how we'd upload files to our host this way:
Our method is expecting to receive two strings: the first being the local path to our file, and the second being the path of the remote directory we'd like to upload to.
The SCP and Paramiko were meant to complement one another, which makes uploading via SCP super easy. SCPClient() creates an object which expects "transport" from Paramiko which we provide with
put() method will upload a local file to our remote host. This will replace existing files with the same name if they happen to exist at the destination we specify. That's all it takes!
The counterpart to SCP's
put() is the
Tracking Upload/Download Progress
SCP lets us see the progress of our file transfers via command-line easily. First, we need to create a new
To utilize this method, all we need to do is go back to where we set our
SCPClient() and pass an extra argument:
scp = SCPClient(self.conn.get_transport(), progress=self.__progress)
Putting Our Script to the Test
We now have a pretty sick class to handle SSH and SCP with a remote host. Let's put it to work! The following snippet is a quick way to test what we've built so far. In short, this script looks for a local folder filled with files (in my case, I filled the folder with fox gifs 🦊).
We'll be accomplishing this with the final addition to our script, main.py:
Here's the output:
Uploaded fox1.gif to 22.214.171.124:/uploads. Uploaded fox2.gif to 126.96.36.199:/uploads. Uploaded fox3.gif to 188.8.131.52:/uploads.
Take It And Run With It
For your convenience, I've uploaded the source for this tutorial to Github. Feel free to take this and run with it! To close things out, I'll leave you with the meat and potatoes of the Client class we put together:
from paramiko import SSHClient, AutoAddPolicy, RSAKey from paramiko.auth_handler import AuthenticationException from scp import SCPClient, SCPException from io import StringIO class Client: def __init__(self, config): self.remote_url = config.remote_url self.remote_user = config.remote_user self.remote_ssh_key = config.remote_ssh_key self.pkey = self.__get_ssh_key() self.client = None def __get_ssh_key(self): """Get our SSh key.""" f = open(self.remote_ssh_key, 'r') s = f.read() keyfile = StringIO(s) pkey = RSAKey.from_private_key(keyfile) return pkey def __connect(self): """Connect to remote.""" if self.client is None: try: client = SSHClient() client.load_system_host_keys() client.set_missing_host_key_policy(AutoAddPolicy()) client.connect(self.remote_url, username=self.remote_user, pkey=self.pkey) except AuthenticationException: raise AuthenticationException('Authentication failed: did you remember to create an SSH key?') finally: return client return self.client def upload(self, file, remote_directory): """Upload a single file to a remote directory""" self.client = self.__connect() scp = SCPClient(self.client.get_transport()) try: scp.put(file, recursive=True, remote_path=remote_directory) except SCPException: raise SCPException.message finally: scp.close() def execute(self, cmd): """Executes a single unix command.""" self.client = self.__connect() stdin, stdout, stderr = self.client.exec_command(cmd) return stdout.readlines() def disconnect(self): """Close ssh connection.""" self.client.close()