Python – pip and virtualenv on Ubuntu

TL;DR;

If you are developing in Python on ubuntu – ensure that you use the Ubuntu provided pip and virtualenv:

sudo apt-get install python-pip

sudo apt-get install python-virtualenv

sudo apt-get install virtualenvwrapper  (assuming you like virtualenvwrapper)

sudo pip install virtualenvwrapper (the ubuntu provided virtualenvwrapper is too horribly out of date to use)

So I have just spent two evenings battling with one particular wrinkle about Python on Ubuntu that I found a little hard to resolve – even after copious googling – so wanted to blog my case here in case it helps others.

So this is related to doing Python (2) development on Ubuntu. I have done some Python development previously on Ubuntu and had followed setup guides such as this one: http://docs.python-guide.org/en/latest/starting/install/linux/ 

This worked fine for me. virtualenv and pip all seemed to work fine and life was good. However I had already fallen into one trap and I was about to fall into another.

The second more minor trap I was about to fall into was accidentally running ‘pip install’ when not inside a virtualenv. This installs into the global scope – rather than the virtualenv – which just ends up being messy. Even worse is running ‘sudo pip install’ which ends up installing things differently again. So the MUST DO from all of this is to add the following to my .bashrc:

export PIP_REQUIRE_VIRTUALENV=true

This essentially means that if you try and run pip when not inside your virtualenv you get an error message. This helps keep your environments much cleaner. This seems particularly important on Ubuntu where Python is used by the system for so many things – and so trying to keep the dependencies of your project clean is much harder.

But the real tricky one was that the Ubuntu Python and the ‘official Python’ put their packages in different locations (/usr/lib/python2.7 versus /usr/local/lib/python2.7/dist-packages). Mostly this did not cause me any issues until I came across the scenario that I wanted to install and access a library that I wanted to develop on alongside a virtualenv.

Following standard practice I forked/cloned the library from git and ran:

python setup.py develop

(This was in a directory outside my main project folder since I planned to use this library in other projects). However I was not able to ‘import project’ within my virtualenv. It couldn’t be found. I could have manually added paths to my PYTHONPATH to resolve this but it “felt wrong”. After much digging, the reason essentially boiled down to the fact that my ‘Ubuntu provided’ python had installed a link to the library in /usr/local/lib/Python – when my ‘pip installed’ virtualenv was expecting it to be in /usr/lib/Python.

Once I realised this there was a bunch of pain involved in removing the stuff that I had ‘sudo pip installed’ or ‘pip installed’ outside of the virtualenv, then installing the Ubuntu versions (see TL;DR;).