Python: distribution systems world

At the very beginning Python packaging world may seem too confusing. But only at first sight.

There are a lot of different packages package formats, libraries and tools for distributing and  packages management in Python world. There is also PyPI - the Python Package Index is a repository of software for the Python.

Most often used formats are tarbolls (tar.gz or zip files), eggs and system packages (like rpm, deb or other). Tarbolls are just source files archives. rpm or deb are binary files built with some system tools like rpmbuild. Eggs can  be thought of as a kind of zipped source code with some meta information (for instance, dependencies).

There are also various libraries: distutils, setuptools, distribute and zc.buildout.

distutils

It’s a built-in library. To distribute package developer need to write simple setup.py file which will be used to built your package in various formats.  It’s definitely lacks some features but clean and easy to use.
Simple example of setup.py:

from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)

For instance, building RPM:

python setup.py bdist_rpm

That is all.

setuptools

It is a third-party software based on distutils. setuptools was created to make Python packaging more powerful then it was  possible with distutils (for instance, dealing with dependencies).

Basic example of setup.py:

from setuptools import setup, find_packages
setup(
name = "HelloWorld",
version = "0.1",
packages = find_packages(),
)

Like distutils, setuptools supports different formats (bdist_rpm, bdist_wininst, etc.). For a long time, setuptools was widely used as one of the most popular packaging system.  For instance, redis-py by Andy McCurdy uses setuptools and many others do.

distribute

It’s also a third-patty library and a fork of setuptools.  It was born after setuptools was not maintained for a long time. The main goal of distribute is ‘to replace setuptools as the standard method for working with Python module distributions.’  It supports both Python 2 and Python 3.  distribute is developing in two branches: 0.6.x (provides a setuptools-0.6cX compatible version and also fixes a lot of bugs which were not fixed in setuptools)  and 0.7.x (will be refactored version with a lot of changes).

zc.buildbot

Project was started by Jim Fulton in 2006  and is based on setuptools and easy_install. It is possibly more complicated then others building systems but it’s also more ambitious.  Buildbot uses additional configurations files (e.g., boostrap.py,   bootstrap.cfg). It’s Python-based and works with eggs. Buildbot also introduces concept of recipes - plugins which aim to add new functionalities to software building, like it’s stated in official documentation.  Recipes  can installed from PyPI.  It allows zc.buildbot to extend possibilities since recipes can be used for different things, like setting up Apache or managing Cron tasks and so on. It’s also possible to use pip with buildbot.

And finally, packages management tools: easy_install and pip.

easy_install

It is a part of setuptools and is a package manager. It’s really easy to install packages with it. It allows to download, build, install, and manage Python packages. It can also be used for installing packages in form of Eggs and packages from PyPI. easy_install was a main tool for installing third-party packages before pip was developed. On the dark side, it can’t be used for packages uninstalling. easy_install will be deprecated in 0.7.x of distribute.

Here is an example if package installation with easy_install:

easy_install install foobar

pip

It’s a tool for Python packages management created by Ian Bicking. Primary it’s used for installing and management packages from PyPI. Pip is intended to replace easy_install and contains extra features (e.g., unlikely easy_install it can be used for uninstalling ). It can not be used for installing eggs but it seems it is not really a problem. pip uses setuptools or distribute. pip will be automatically installed into each virtual environment created with virtualenv.

Example of package installation with pip:

pip install foobar

One more important tool you should know about is a virtualenv. It’s widely used for creating isolated Python environments which can be useful in both production and development. We will cover it later.

At the time of writing (Aug 2011),  the most preferable way to install packages is pip. And distribute is for packaging. easy_install and setuptool can be considered as deprecated.

Now you can easily understand what co-creator of Django Jacob Kaplan-Moss meant:

Python has one package distribution system: source files and setup.py install.
And easy_install.
Python has two package distribution systems: setup.py install and easy_install. And zc.buildout.
Python has three package distribution systems: setup.py install, easy_install, and zc.buildout. And pip.
Amongst Python’s package distribution are such diverse elements as…

Future reading

  1. On packaging, Why I like pip, both by James Bennett, Django release manager
  2. Chapter 16. Packaging Python Libraries from Dive Into Python 3 by Mark Pilgrim
  3. A small introduction to Python Eggs by by Christian Scholz
  4. Python Packaging, Distribution, and Deployment: Volume 1
  5. A Few Corrections To “On Packaging” by Ian Bicking
  6. A history of Python packaging by Martijn Faassen
  7. Developing Django apps with zc.buildout by Jacob Kaplan-Moss
  8. Chapter 14. Python Packaging by Tarek Ziadé

Didn’t find the answer to your question? Ask it our administrators to reply we will publish on website.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>