Weblog-Software Benchmark II · 22. Mai 2005, 17:19

So finally here is the promised follow-up to my first benchmark (Wordpress vs. Textpattern – a quick performance benchmark) which will compare Wordpress, Textpattern, Serendipity and Lightpress, the last one (currently) only being a frontend for Wordpress.

Before I go on let me make one thing clear: These are not absolute numbers, it makes absolutely no sense to compare it to numbers from a different benchmark. To illustrate this: The same Textpattern was about 10 times as fast on Jason’s machine than on my old Windows machine which the first benchmark ran on (3.63 vs ~35 [#/sec]), the main reason for the difference being hardware (the second being the OS). I would not be mentioning this triviality, if some people hadn’t overlooked this fact last time (intentionally no links here).

For a more thorough introduction and more disclaimers take a look at the above mentioned first benchmark.

The set-up

Quite a few changes in the setup with respect to the last benchmark. While last time everything ran on my (old) Windows machine, this time I had 2 machines:
Server: Celeron 1.7Ghz, 512 RAM, Ubuntu5.04, Apache2 (pre-fork), Mysql 4.0, PHP5.0.4 (SAPI, w/ eaccelerator)
Client: Pentium 933 Mhz; 256MB RAM, WinXP

Here the respective versions of the software packages used:

I started out installing Wordpress and made 10 posts (total of 12,306 characters). Then I installed the other packages and used their wordpress import scripts which worked fine. I also have to compliment the packages for the straightforward installation/import procedures, where ironically Serendipity with the lowest version number has the most mature installation procedure, but I digress. I switched all software to a Kubrick-Layout in the hopes that I would get similar page sizes, but that didn’t quite work, so I padded the templates of the smaller sites with some bogus ipsums to get around 40kb page size. In Serendipity I turned off the plugin calendar and did not enable gzip-compression, to adjust it to the rest of the field.

The speed benchmark

I ran apache bench on the front-page of each blog. I divived the actual test into two parts. *(1)* First I tested as is, *(2)* then I installed caching plugins where available : txp -> my own asy_jpcache 0.9; wp -> wp-cache 2.0; s9y -> garvin’s cachesimple. Apache bench was run with concurrency of 10 for 500 requests:

ab.exe -c10 -n500 http://site/

Blog (1) as is (2) with caches   
[#/sec] time/req. [#/sec] time/req. cache
wp 5.26 # 190.2 ms 118.3 # 8.5 ms wp-cache 2.0
txp 10.99 # 91.2 ms 123.9 # 8.1 ms asy_jpcache 0.91
s9y 6.49 # 154.1 ms 27.03 # 37.0 ms cachesimple
lp 18.06 # 55.4 ms - - -

1 see below A few words on the caches.

A few words on the caches

The features and properties of the different caching plugins would well deserve their own article. For the sake of comparability in txp’s asy_jpcache I turned gzip off which usually caches a compressed version of the page. When it was on, and ab ran with the additional header to request gzipped content the numbers were considerably higher: 183.56 [#/sec] and 5.44 ms. This is probably more relevant in the real world, since most browsers support gzip. On the other hand wp-cache preserves the option to include php pages when serving the cached pages allowing partly dynamic pages, which is not possible with asy_jpcache. However that is possible with zem_cache whose performance is similar to Staticize (wp), both of which are not included in this benchmark (but were somewhere around the performance of garvin’s cachesimple). zem_cache is the easiest when mixing static and dynamic content, all plugins and/or features of the weblog-software could be continued to use if they were not within the part that was cached, whereas with Staticize and wp-cache one can only dynamically include external .php files. s9y’s cachesimple still allowed for some event-based plugins to run, but none that would change the outputted page (please correct me if I am wrong).

Another thing you have to keep in mind when looking at the numbers for the caches, is that they are only measuring cache-hits, i.e. that’s how fast it would be if you never had to generate the page, never had to write cache-files and never had to delete cache-files. For practical purposes this is pretty unrealistic. But since the ratio of cache-hits to cache-writes is dependant on a lot of different things, I won’t (or rather can’t) tell you how to adjust these theoretical results. If you’re slashdotted (you wish ;)) you’re going to have have a high “hits per write”-ratio. If you have very many loyal readers that write a lot of comments, then you’re going to have a very lousy “hits per write”-ratio. And if those loyal readers are the only people hitting your blog, you may be better off without a cache at all.

The memory check

Last time I mentioned how speed alone doesn’t tell you much about scalability (also see hare vs. tortoise), so this time I decided to install xdebug2.0.0beta and take a look at how much memory was consumed – only for php, not taking into account mysql. Since I heard that the opcode cache can irritate xdebug, but wasn’t able to verify that information, I ran tests both, with eaccelerator enabled and disabled.

eaccelerator disabled:

Blog (1) as is (2) with caches
wp 4,173 KB 250 KB wp-cache 2.0
txp 2,012 KB 208 KB asy_jpcache 0.9
s9y 5,286 KB 4,600 KB cachesimple
lp 1,117 KB - -

eaccelerator enabled:

Blog (1) as is (2) with caches
wp 235 KB 42 KB wp-cache 2.0
txp 352 KB 58 KB asy_jpcache 0.9
s9y 700 KB2 317 KB cachesimple
lp 360 KB3 - -

2 S9y only peeks at 700kb for one instruction, and would otherwise peak at 669kb.

3 Lightpress only peeks at 360kb for one instruction, and would otherwise peak at around 310kb.

Without eaccelerator and cache-plugins the differences are pretty clear, they result mostly from the size of the codebase, more code, means more file i/o and more parsing. With eaccelerator and/or cache-plugins it will mostly depend on what kind of webserver you are running. Each apache process had a memory footprint of roughly ~ 11MB (top -> resident), [update (6.7.05): however I have read that with lighttpd the memory footprint of the webserver process will get down to 350kb. So with lighttpd or similar low memory footprint webservers the differences in the rangeof x00 KB (and lower) can be relevant, but on Apache it won’t be that noticable. after trying out lighttpd: it’s a single process webserver (threaded) taking about 1,5 MB of memory. But the relevant part is the numerous php-processes which take at least a few MB each (3,5 MB for me, but depends on the used modules). So I think it is pretty reasonable to conclude, that if there is a php-accelerator installed the memory usage of the scripts will be more or less neglectable.]

My personal Conclusions

  1. If your server is relatively fast (and running an opcode cache for php) and not under heavy load, from a visitors perspective each software was comparably fast, you certainly couldn’t guess the software by how fast the page loads in your browser.
  2. JPCache, which I based my asy_jpcache plugin for Textpatter on, is a genereal pluggable caching solution. It could be used for each of the software-packages above as is, with a time-based cache. And it can be adapted to event-based cache-cleaning (so your users never get stale pages) with some effort, as I did for Textpattern. Maybe someone will do it for Lightpress for which I couldn’t find any caching plugin yet, and for s9y.
  3. If you are a Wordpress user, the only reason for not using the Lightpress frontend is maybe certain plugins (though some are supported in lp) or if you don’t know any html and like the many ready-made templates for Wordpress (lp uses a different, IMHO easier templating system). Both of these points may change in the future.
  4. Caching plugins improve performance considerably, at least if you have a good cache-hit vs. cache-write ratio and can live with the side-effects. But even in a perfect scenario, they are not nearly as fast as static pages (those were served at ~ 380 requests/second), and they are not quite “hundreds of times faster” (best relative performance gain was for wp -> 23 times as fast compared to no cache)
  5. Serendipity seems to be a really nice weblog-software and I really hope nobody just skips it (or any of the other packages for that matter) as an option because of these numbers. Each does some things differently and some better than the other packages, and if you are looking for a weblog-software a lot of things will be a lot more important than performance (except for people on slow or crowded servers).