Package Python Projects the Proper Way with Poetry

Managing software dependencies is typically a short and straightforward topic to cover; unless we're talking about Python, that is. Python has a myriad of options when it comes to virtual environments, which is arguably not a good thing. Virtualenv, Pipenv, Conda, and Poetry all aim to keep Python libraries project-specific, and they each do an adequate job of accomplishing this... but why do we need a vast catalog of solutions to solve the same problem? If modern programming languages such as NodeJS manage to sidestep environments entirely (via the /node_modules folder), why couldn't the Python community settle on a standard?

Python isn't exactly an "old" programming language (relatively speaking), but a lot has changed in programming since 1991. This was an era before language-specific package management systems had been conceived. Both Pip and Virtualenv were introduced to Python in 2011- that's 20 years where installing dependencies meant manually downloading packaged source code into a system folder. Using Virtualenv in 2020 feels archaic, but the concept of isolating packages on a per-project level was a somewhat novel concept at the time. Once this minimal implementation of package management became the norm, it didn't take long for developers to realize ways to improve this workflow. Early revelations resulted in the creation of virtual environment solutions, which seemed monumentally better than the last. In retrospect, these improvements were incremental to reaching the state of virtual environments today. This is why we find Python bloated with environment managers accumulated over time as the concept matured.

Poetry is arguably Python's most sophisticated dependency management option available today. Poetry goes far beyond dependencies, with features like generating .lock files, generating project scaffolding, and a ton of configuration options, all of which are handled via a simple CLI. If you're unsure how to cleanly and effectively structure and manage your Python projects, do yourself a favor and use Poetry.

More Than Just Dependencies

Proper dependency management is only one dimension of preparing a production-ready application. For a Python project to be considered a valid package to be listed on PyPi, the Python Packaging Authority has a list of guidelines outlining various other requirements involving licensing, setup configuration files, manifests, and so forth. The allure of Poetry is that it is the first tool that handles all of these things cleanly. The Poetry Github repository states its purpose as follows:

Packaging systems and dependency management in Python are rather convoluted and hard to understand for newcomers. Even for seasoned developers it might be cumbersome at times to create all files needed in a Python project: setup.py, requirements.txt, setup.cfg, MANIFEST.in and the newly added Pipfile. So I wanted a tool that would limit everything to a single configuration file to do: dependency management, packaging and publishing.

It sounds crazy, but Poetry makes covering all bases of a managing a Python project easy via its CLI and a single configuration file.

Setting up Poetry On OSX/Unix

I don't typically reiterate installation instructions from official docs, I've run into issues following the existing install instructions word-for-word.

I'm assuming we're all using Python3 here since we aren't absolute savages. It's critically important that we install Poetry for python3 as opposed to Python, contrary to what the official docs would have you copy-and-paste. Mistakenly installing Poetry for the wrong version of Python is a massive headache, so please use this:

$ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3

Poetry install script

This script should install Poetry and automatically modify your system PATH to include the installation, as the success message states. Go ahead and confirm:

$ poetry --version

Confirm Poetry was installed correctly.

If the above command returns an error, make sure your .bash_profile (or .zshrc) was updated with the following, and restart your terminal:

export PATH="$HOME/.poetry/bin:${PATH}"

Add poetry to system path

Create a Python Project with Poetry

Enough with the foreplay, let's see how Poetry streamlines not only dependency management but nearly everything that goes into structuring a Python project. Poetry has a robust CLI, which allows us to create and configure Python projects easily. Here's what getting started fresh looks like:

$ poetry new poetry-tutorial-project

Create a Python project via Poetry CLI.

This is an absurdly convenient way to generate a standard Python folder structure for our new project named poetry-tutorial-project:

/poetry-tutorial-project
├── README.md
├── poetry_tutorial_project
│   └── __init__.py
├── pyproject.toml
└── tests
    ├── __init__.py
    └── test_poetry_tutorial_project.py

Contents of our new project: /poetry-tutorial-project

This saves us the trouble of manually creating this standard folder structure ourselves. Most of the file contents are empty, with one exception: pyproject.toml.

One Configuration to Rule Them All

The secret sauce of every Poetry project is contained in a file called pyproject.toml. This is where we define everything from our project's metadata, dependencies, scripts, and more. If you're familiar with Node, think of pyproject.toml as the Python equivalent of package.json.

Starting a new Poetry project automatically creates a minimal version of this file. Here's what mine looks like:

[tool.poetry]
name = "poetry-tutorial-project"
version = "0.1.0"
description = ""
authors = ["Todd Birchard <todd@example.com>"]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]
pytest = "^4.6"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

pyproject.toml

The above config has some basic information, but it isn't enough to be useful just yet. A complete pyproject.toml file would look something like this:

[tool.poetry]
name = "poetry_tutorial_project"
version = "0.1.0"
description = "Simple Python project built with Poetry."
authors = ["Todd Birchard <toddbirchard@gmail.com>"]
maintainers = ["Todd Birchard <toddbirchard@gmail.com>"]
license = "MIT"
readme = "README.md"
homepage = ""
repository = "https://github.com/hackersandslackers/python-poetry-tutorial/"
documentation = "https://hackersandslackers.com/python-poetry/"
keywords = [
    "Poetry",
    "Virtual Environments",
    "Tutorial",
    "Packages",
    "Packaging"
]

[tool.poetry.dependencies]
python = "^3.8"
loguru = "*"
psutil = "*"

[tool.poetry.dev-dependencies]
pytest = "*"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[tool.poetry.scripts]
run = "wsgi:main"

[tool.poetry.urls]
issues = "https://github.com/hackersandslackers/python-poetry-tutorial/issues"

pyproject.toml

Now we're cookin' with gas! Our .toml file is now comprised of 6 "sections," where each section contains config values for our project:

[tool.poetry]: The first section of pyproject.toml is simply informational metadata about our package, such as the package name, description, author details, etc. Most of the config values here are optional unless you're planning on publishing this project as an official PyPi package. Providing values for repository and keywords isn't going to matter if you're not distributing this package. Still, this sort of metadata would be critical if you ever hope to distribute your package.
[tool.poetry.dependencies]: This is where we define dependencies our application absolutely must download to run. You may specify specific version numbers for required packages (such as Flask = "1.0.0"), or if you simply want to grab the latest version, setting the version to "*" will do just that. You'll also notice that the version of Python we're targeting for our project is provided here as well: this is specifying the minimum version required to run our app. In the example above, a user running Python 3.6 will not be able to run this app, as we specify that Python 3.7 is the absolute lowest version required.
[tool.poetry.dev-dependencies]: Dev dependencies are packages that contributing developers should download to iterate on this project. Dev dependencies are not required to run the app, and won't be downloaded when the app is built by default.
[build-system]: This is rarely a section you'll need to touch unless you upgrade your version of Poetry.
[tool.poetry.scripts]: This is where we specify where our app entry point(s) is by assigning function within modules to the name of a script to be run. The example run = "wsgi:main" is specifying that we want to create a command called "run," which will look in wsgi.py for a function called main(). With this set, we can then launch our app via the Poetry CLI by typing poetry run (more on this in a bit).
[tool.poetry.urls]: This is a completely optional section where you can add any number of helpful links or resources that somebody downloading this package might find useful.

A config like the one above is more than sufficient to have a clean, functioning, packaged app. Poetry supports other types of config values as well, although chances are you'll rarely need most of them. In case you're curious, find the full list here:

The pyproject.toml file | Documentation | Poetry - Python dependency management and packaging made easy.

Official documentation of Poetry

Python dependency management and packaging made easy.

Poetry CLI

Poetry's command-line interface is impressively simplistic for the scope of what it achieves. The equivalent functionality of both Pipenv and setup.py are covered by Poetry, as well as numerous other features related to configuration management and package publishing. We'll start with installing and managing the dependencies we just set in pyproject.toml.

Installing & Managing Dependencies

poetry shell: The first time this command is run in your project directory, Poetry creates a Python virtual environment which will forever be associated with this project. Instead of creating a folder containing your dependency libraries (as virtualenv does), Poetry creates an environment on a global system path, therefore separating dependency source code from your project. Once this virtual environment is created, it can be activated again at any time by simply running poetry shell in your project directory in the future. Try comparing the output which python before and after activating your project shell to see how Poetry handles virtual environments.
poetry install: Installs the dependencies specified in pyproject.toml. The first time a project's dependencies are installed, a .lock file is created, which contains the actual version numbers of each package that was installed (i.e.: if Flask = "*" resulted in downloading Flask version 1.0.0, the actual version number would be stored in .lock). If a .lock file is present, the version numbers in .lock will always be prioritized over what is in pyproject.toml.
poetry update: Mimics the functionality of install, with the exception that version numbers in .lock will NOT be respected. If newer versions exist for packages in pyproject.toml, newer versions will be installed, and .lock will be updated accordingly.
poetry add [package-name]: A shortcut for adding a dependency to pyproject.toml. The package is installed immediately upon being added.
poetry remove [package-name]: The opposite of the above.
poetry export -f requirements.txt > requirements.txt: Exports the contents of your project's .lock file to requirements.txt. It comes in handy when handing work off to developers who still use requirements.txt for some reason.

Configuration

poetry config: "Config" refers to environment-level configuration, such as the paths of the current virtual environment, or environment variables. Passing the --list option will return your environment's current config values.
poetry check: Checks pyproject.toml for errors.
poetry show: Returns a breakdown of all packages currently installed to the project, including dependencies of dependencies.

Executing Applications

poetry run [script-name]: Executes a script defined in the [tool.poetry.scripts] section of pyproject.toml.

Building and Publishing

poetry build: Builds the source and wheels archives.
poetry publish: Publishes the output of the previous build to the project's external repository (likely PyPi).

Become a Poet

I'm well aware that tooling/architectural decisions in software are subjective preferences. Developers are not defined by their tools, blah blah, but look... at the risk of coming off as an ignorant novice, Poetry is objectively the best way to create, manage, and package Python projects. I don't have a horse in this race, but I'd go so far as to say that not embracing Poetry is a waste of your time.

It's not much, but I've posted the code from this tutorial on Github: