Techempowers Web Framework Benchmarks

I have been following the ‘Web Framework Benchmarks’ being run by Techempower over the last couple of months. Last year I spent a  bit of time playing with some Python frameworks so I was interested to see how they stacked up.

Web benchmarks

Many people loath benchmarks – because it is so hard to be equitable between the candidates – and even then it is very hard to interpret their significance for a given application. (In other words – the results can be highly dependant upon the specifics of the tests and your particular application requirements may vary significantly from the benchmark.)

There is a place though for benchmarks to provide a ‘level if somewhat subjective’ playing field. And further the best way to do a benchmark is in the open so that every framework/community can give it their best shot and see how they go. It seemed to me that the Techempower guys were approaching this in the right spirit. It has been very instructive to see the same relatively simple tasks coded in different languages and frameworks.

Python Frameworks

I am a part time hacker. Each year I tend to find a project to spend a bit of time on… two years ago it was a Rails/Ruby project and last year it was a Flask/Python one. This year I was toying with the idea of trying out either a functional language (Scala/Haskell/Clojure) or trying Go. And then I saw this benchmark! At first the results of the Java based languages (e.g. Clojure) looked really promising and Go showed flashes of greatness. I was tempted to jump on the Go bandwagon – but I felt that the Python frameworks were not being given much attention – so instead I spent a bit of time on them.

SQLALchemy sucks cycles

I am pretty comfortable working in SQL – so I had always been in two minds about the value of an ORM. But looking at the Techempower benchmarks one thing that jumps out is that SQLAlchemy does incur a performance penalty. So I guess the moral is that if you want a well documented, well rounded, fully featured ORM for Python – use SQLAlchemy, but if you are comfortable in SQL – then just use that instead. (Disclaimer: The non-ORM test for Bottle still uses SQLAlchemy to provide connection pooling. It really does seem as though it is the ORM in SQLAlchemy and not the Core that incurs the hit in these tests.)

PyPy on Amazon EC2

In the most recent round 5 tests though – a community contribution added PyPy support for the Flask tests. For tests running in Amazon EC2, PyPy produced some pretty significant – in some cases staller – performance improvements. Unfortunately this did not seem to be the case on dedicated i7 hardware – further proving the ‘horses for courses’ nature of these benchmarks and the need to always run your own tests on your own setup.

First appearances are deceiving

The very first appearance of Python on these tests was a Django test and it appeared 4th from the bottom of the list showing only 5.9% of the throughput of the best performer (single db query on EC2). At this point it was easy to write off Python as a dynamic, interpreted language that was always going to be slow and was ‘yesterdays news’. However in the most recent (round 5) tests, the best performing Python framework is at 32.8% on the same test. That is an improvement of almost 600%!!! Optimisation can suffer from diminishing returns – however at least for Python the ‘obvious’ optimisations can make a big difference. In almost all tests, Python ended up within a factor of 2 of the best Go-based framework (revel and webgo were included in Round 5).

Bottle surprises

I remember seeing Bottle when I first came across Flask. It looked very neat – but didn’t have the mindshare of Bottle so initially I went with Flask. However on these benchmarks Bottle showed a clear performance advantage. I will definitely be looking into it further.

Python FTW

So initially I entered this thinking that Python would struggle to compete with the likes of Go and Java on these tests and that this exercise would be the ‘nail in the coffin’ for Python. And I am not claiming that Python is anywhere close to beating these platforms. However the performance gap between them is not an order of magnitude. Typically within a factor of 2 times slower – but not 10 or 20 times as it seemed in the early tests. Against this you have to trade off the maturity of existing libraries.

Having said that there are definitely many ‘tricks’ to know to get the most performance out of Python web apps. From database connections, to choice of json library to choice of web server and python interpreter. From the discussion I have seen – the same is true for all the platforms/languages – and why it has been so useful/interesting to watch the results over multiple rounds for these benchmarks as each test is optimised. Given the maturity of Python, some of these choices can be pretty confusing though. There is a load of information available – but not all of it is up to date – and what was the best choice 3 years ago is probably not the best choice today.

Conclusion

Ultimately these benchmarks have reinforced almost any of the languages (Python, Go, Scala, PHP, Clojure, Java etc) are capable of performing ‘performant enough’ applications. I guess the main point of this post is that I was surprised how competitive Python was.