Application Dependencies

Application dependencies are the libraries other than your project code that are required to create and run your application.

Why are application dependencies important?

Python web applications are built upon the work done by thousands of open source programmers. Application dependencies include not only web frameworks but also libraries for scraping, parsing, processing, analyzing, visualizing, and many other tasks. Python's ecosystem facilitates discovery, retrieval and installation so applications are easier for developers to create.

Finding libraries

Python libraries are stored in a central location known as the Python Package Index. PyPi contains search functionality with results weighted by usage and relevance based on keyword terms.

Besides PyPi there are numerous resources that list common or "must-have" libraries. Ultimately the decision for which application dependencies are necessary for your project is up to you and the functionality you're looking to build. However, it's useful to browse through these lists in case you come across a library to solve a problem by reusing the code instead of writing it all yourself. A few of the best collections of Python libraries are

Python.org's useful modules which groups modules into categories.
GitHub Explore Trending repositories shows the open source Python projects trending today, this week, and this month.
This list of 20 Python libraries you can’t live without is a wide-ranging collection from data analysis to testing tools.
Wikipedia actually has an extensive page dedicated to Python libraries grouped by categories.

Isolating dependencies

Dependencies are installed separately from system-level packages to prevent library version conflicts. The most common isolation method is virtualenv. Each virtualenv is its own copy of the Python interpreter and dependencies in the site-packages directory. To use a virtualenv it must first be created with the virtualenv command and then activated.

The virtualenv stores dependencies in an isolated environment. The web application then relies only on that virtualenv instance which has a separate copy of the Python interpreter and site-packages directory. A high level of how a server configured with virtualenv can look is shown in the picture below.

How the virtualenv separates dependencies on the server.

Installing Python dependencies

The recommended way to install Python library dependencies is with the pip command when a virtualenv is activated.

Pip and virtualenv work together and have complementary responsibilities. Pip downloads and installs application dependencies from the central PyPi repository.

requirements.txt

The pip convention for specifying application dependencies is with a requirements.txt file. When you build a Python web application you should include requirements.txt in the base directory of your project.

Python projects' dependencies for a web application should be specified with pegged dependencies like the following:

django==1.11.0
bpython==0.12
django-braces==0.2.1
django-model-utils==1.1.0
logutils==0.3.3
South==0.7.6
requests==1.2.0
stripe==1.9.1
dj-database-url==0.2.1
django-oauth2-provider==0.2.4
djangorestframework==2.3.1

Pegged dependencies with precise version numbers or Git tags are important because otherwise the latest version of a dependency will be used. While it may sound good to stay up to date, there's no telling if your application actually works with the latest versions of all dependencies. Developers should deliberately upgrade and test to make sure there were no backwards-incompatible modifications in newer dependency library versions.

setup.py

There is another type of dependency specification for Python libraries known as setup.py. Setup.py is a standard for distributing and installing Python libraries. If you're building a Python library, such as requests or underwear you must include setup.py so a dependency manager can correctly install both the library as well as additional dependencies for the library. There's still quite a bit of confusion in the Python community over the difference between requirements.txt and setup.py, so read this well written post for further clarification.

Application dependency resources

The following links provide advice on how to use Python packages as well as package your own dependencies for projects or consumption by other developers.

Python Packaging Is Good Now is a wonderfully written blog post. It provides historical context on why Python's code library packaging was painful for a long time, and what's been fixed to make building and installing application dependencies so much better.
Python's New Package Landscape covers the history of Python packaging tools and examines the problems with dependency isolation and the dependency graphs that newer tools such as Pipenv, Poetry, Hatch and pipsi aim to solve.
Python Packaging User Guide provides a collection of resources to understand how to package and distribute Python code libraries.
Alice in Python projectland is an amazing post that takes the reader from simple Python script into a complete Python package.
A non-magical introduction to virtualenv and pip breaks down what problems these tools solve and how to use them.
There’s no magic: virtualenv edition breaks open the virtual environment "black box" to show you what the tool is doing when you use its commands.
Testing & Packaging examines a configuration for ensuring tests run against your code and how to properly package your project.
Python Application Dependency Management in 2018 presents some critical analysis and critique oof the existing Python dependency management tools including newer ones such as pipenv and Poetry.
Occasionally arguments about using Python's dependency manager versus one of Linux's dependency managers comes up. This provides one perspective on that debate.
Open source trust scaling is a good piece for the Python community (and other programming communities) that is based on the left-pad NPM situation that broke many dependent packages in the Node.js community.
Major speed improvements were made in pip 7 over previous versions. Read this article about the differences and be sure to upgrade.
Typosquatting programming language package managers shows how many packages on centralized dependency servers for Python, Node.js and Ruby can be vulnerable to "typosquatting" where a developer either confuses a fake package for the correct one or simply makes a typo when specifying her dependency list.
Perils of packaging covers several edge cases that come up when trying to put together pieces like Travis CI, PyPI and conda. The post walks through the errors and how to get around them until they are smoothed out by updates to the tools.
The Many Layers of Packaging goes up and down the packaging stack and even covers bits about virtual environments and security. It's well worth investing some time to read this post to get an overview of the many layers involved in dependency packaging.
How to Publish Your Package on PyPI is for developers who have created a code library they would like to share and make installable for other developers.

Open source app dependency projects

pip and venv are part of Python 3's standard library as of version 3.3. However, there are numerous other open source libraries that can be helpful when managing application dependencies in your projects, as listed below.

Autoenv is a tool for activating environment variables stored in a .env file in your projects' home directories. Environment variables aren't managed by virtualenv and although virtualenvwrapper has some hooks for handling them, it's often easiest to use a shell script or .env file to set them in a development environment.
Pipenv is a newer Python packaging and dependency management library that has seen some adoption in place of the standard pip library.
Pipreqs searches through a project for dependencies based on imports. It then generates a requirements.txt file based on the libraries necessary to run those dependencies. Note though that while this could come in handy with a legacy project, the version numbers for those libraries will not be generated with the output.
pip-check presents a nicely-formatted list of all your installed dependencies and the status of whether or not updates are available for each of them.
pip-name is a straightforward library that looks up package names on PyPI and tells you whether or not the library name is already taken.