I have been playing with word2vec on and off since it was released in 2013. It is a neural network that scans vast amounts of text and produces a vector representation of each word that can do some useful things. There are quite a few intros posted online – like here [http://radimrehurek.com/2014/12/making-sense-of-word2vec/]. If you are not familiar with word2vec – go read one of these first then come back. What I wanted to talk about here was just some of the practical issues I came across when trying to get up and running.

Which code?

The canonical implementation of word2vec is the C code released by Tomas Mikolov [https://code.google.com/p/word2vec/] – the original author of word2vec. Gensim by Radim Rehurek [http://radimrehurek.com/gensim/index.html] is a fantastic implementation. Python makes ‘playing with’ these models very easy – and Gensim is robust and suprisingly fast python implementation. (There are also a number of other implementations but I don’t have any experience of them.)

Using a pretrained model

The easiest way to start working with word2vec is just to load up a pretrained model. The original authors released a model trained on around 100 billion words! It is readily accessible and of high quality. Loading it up is pretty easy – however be warned that it requires at least 4GB of RAM.

Understanding vectors

In some intro material to word vectors they explain word vectors with an example based on the king-man+woman example that goes something like this: Imagine that the vector for king has one co-ordinate correlated with ‘royalty’ and another correlated with ‘maleness’. So by subtracting ‘maleness’ and ‘adding ‘femaleness’ we can end up with a word like queen.[REF TODO] That is ok as a starting point if you are not used to thinking about word vectors. However be warned – the co-ordinates of each vector do not actually work like this. I spent a long time thinking like this. It leads to questions like ‘can we find words that are half male?’, ‘can we find words that are equal part male and female?’, ‘what do words with a negative male coordinate represent?’ As far as I can tell these are all meaningless questions. In reality ‘maleness’ is in some way distributed across all of the co-ordinates of the vector.

The curse of dimensionality

Words are vectors in high dimensional spaces. For word2vec 300-500 dimensions are not uncommon. These sorts of spaces are very hard to reason about (see especially’The Curse of Dimensionality’ [http://en.wikipedia.org/wiki/Curse_of_dimensionality].) I can’t really give much practical advice other than to say it is pretty normal for your head to hurt when trying to think about it.

Training your own model

When you want to start training your own model (rather than using a pretrained model). There are a few things to watch for. Firstly, it is important to appreciate that there is some inherent randomness in the models – so no two models will be the same. This makes reproducibility and comparing different iterations a little bit difficult. There seem to be some efforts within gensim to manage this a bit – but I haven’t really looked into this aspect of it. There are a number of critical thing to decide to train a model:

• What corpus to use? What data will you use to train your model.
• What algorithms will you use (word2vec actually implements a couple of different algorithms)
• What parameters will you use when training?
• How will you evaluate your model?

Mikolov’s released ‘big-data.sh’ late last year (2014) which gives a good baseline for training your own model. My comments are all related to training a general purpose English language model.

Corpus

Word2vec really relies on a large amount of text. It took me a while to appreciate how large this needs to be. If you don’t have a large amount of data – then the word2vec tool supports multiple passes or iterations over your data to try and improve the quality of the model.

• text8 – (from here [http://mattmahoney.net/dc/textdata]) is a good starting point – not really big enough for anything serious but training is quite quick and gives ok results which makes it good for prototyping, sanity checks and so on
• text9 contains about 120M words. It is big enough that you start to get some meaningful results. Training this model (with Gensim) takes at least 8G of RAM. A model can be trained in 6-12 hours on a 2013 quad core i5-3580
• Mikolov’s big data corpus contains about 8.5B words. It requires at least 11G RAM to build with Gensim. It took over 4 days to train on my i5 using the CBOW algorithm (and 3 iterations) and almost 8 days to train using the skipgram algorithm

Iterations and Corpus Size

One of the basic premises of Word2Vec seems to be that data beats algorithms – in the sense that a simpler algorithm that can be trained on more data is better than a more sophisticated algorithm that takes longer to train. So in a sense choice of algorithms needs to be measured against training time. For example, when training a model on the text9 data set, I used both the ‘skipgram’ and the ‘hierarchical softmax’ algorithms simultaneously (with 1 iteration) and performance against the ‘questions-words’ analogy task improved significantly. However training time also improved significantly. If I simply used the skipgram algorithm with more iterations, the performance was even better again for the same training time. As a rule of thumb (which admittedly is pretty arbitrary) – I would suggest that you need at least 1 billion words to train a reasonable model (again – this is for general purpose English only). In the case of the text9 corpus with about 100M words, this suggests at least 10 iterations over the data are needed. (I did not find much improvement beyond 10 iterations with text9.) Mikolov suggests 3 iterations of 8.5b words each in his big-data script.

TL;DR;

If you are developing in Python on ubuntu – ensure that you use the Ubuntu provided pip and virtualenv:

sudo apt-get install python-pip

sudo apt-get install python-virtualenv

sudo apt-get install virtualenvwrapper  (assuming you like virtualenvwrapper)

sudo pip install virtualenvwrapper (the ubuntu provided virtualenvwrapper is too horribly out of date to use)

So I have just spent two evenings battling with one particular wrinkle about Python on Ubuntu that I found a little hard to resolve – even after copious googling – so wanted to blog my case here in case it helps others.

So this is related to doing Python (2) development on Ubuntu. I have done some Python development previously on Ubuntu and had followed setup guides such as this one: http://docs.python-guide.org/en/latest/starting/install/linux/ 

This worked fine for me. virtualenv and pip all seemed to work fine and life was good. However I had already fallen into one trap and I was about to fall into another.

The second more minor trap I was about to fall into was accidentally running ‘pip install’ when not inside a virtualenv. This installs into the global scope – rather than the virtualenv – which just ends up being messy. Even worse is running ‘sudo pip install’ which ends up installing things differently again. So the MUST DO from all of this is to add the following to my .bashrc:

export PIP_REQUIRE_VIRTUALENV=true

This essentially means that if you try and run pip when not inside your virtualenv you get an error message. This helps keep your environments much cleaner. This seems particularly important on Ubuntu where Python is used by the system for so many things – and so trying to keep the dependencies of your project clean is much harder.

But the real tricky one was that the Ubuntu Python and the ‘official Python’ put their packages in different locations (/usr/lib/python2.7 versus /usr/local/lib/python2.7/dist-packages). Mostly this did not cause me any issues until I came across the scenario that I wanted to install and access a library that I wanted to develop on alongside a virtualenv.

Following standard practice I forked/cloned the library from git and ran:

python setup.py develop

(This was in a directory outside my main project folder since I planned to use this library in other projects). However I was not able to ‘import project’ within my virtualenv. It couldn’t be found. I could have manually added paths to my PYTHONPATH to resolve this but it “felt wrong”. After much digging, the reason essentially boiled down to the fact that my ‘Ubuntu provided’ python had installed a link to the library in /usr/local/lib/Python – when my ‘pip installed’ virtualenv was expecting it to be in /usr/lib/Python.

Once I realised this there was a bunch of pain involved in removing the stuff that I had ‘sudo pip installed’ or ‘pip installed’ outside of the virtualenv, then installing the Ubuntu versions (see TL;DR;).

It seems that it is the ‘new normal’ to bemoan how easy it is to overdo passive consumption of information on the internet. Facebook, Twitter, random blog posts such as this one – all consume our time but don’t always ad a lot of value. This post is the most recent one that I came across. All true – but almost a tru-ism. Even the comments over on Hacker News were pretty predictable. One of the ‘predictable’ comments compared this sort of low quality information with junk food. Most of these articles seem to end with general exhortations to ‘watch what we read’.

To take the ‘junk food’ analogy further, the invention of ‘junk food’ (low quality – easy to eat food) eventually led to the creation of ‘Nutrition labels’ that at least make it visible to people what the nutritional content of the food is. So what would ‘Nutrition labels’ on our ‘information’ look like? Is this a useful analogy?

I have spent some time recently working with Angular.js and thought I would record some thoughts. In particular I was working on an RSS Reader type application with ‘infinite scrolling’ functionality. So in dot points:

• It was very quick to become somewhat productive. The ‘phonecat’ example app and on line documentation are thorough. The documentation itself is pretty dense – so it is best to work through the tutorial first.
• In my experience the two way data binding was both the greatest strength and ultimate downfall of Angular.js in my application. Two way data binding meant getting a functional application up was extremely quick… a lot of typing was saved coupling models to views. On the other hand, for a predominantly ‘read’ application, there seemed to be some performance degradation with maintaining this two way data binding.
• I did manage to get an interesting ‘infinite scrolling’ implementation which I hope to share in a later post
• The $resource service made working with a json RESTful api a breeze

So overall, for a dynamic application involving both reading and writing of dynamic data Angular.js could well make you life a lot easier. For predominantly ‘read only’ applications it may not be the right fit.

There seems to be a debate about advertising as a business model. I have been thinking about this on an off long but was inspired to write this post by Fred Wilson’s recent post. My comment is to ask a question: Why is it that content publishers get to sell access to my eyeballs to advertisers? I understand that for ‘traditional mass media’ there is no practical way for me as a consumer to sell that access directly to an advertiser….. but in the enlightened internet age, why can’t I have a direct relationship with the advertisers and decide who gets access to my eyeballs… why does a content publisher get this exclusive right?