Varnish + Zope – Multiple zope instances behind a single varnish cache

I run multiple Zope instances on one server. Each Zope instance listens on a different port (localhost:100xx). Historically I’ve just used Apache as a front end which forwards requests to the Zope instance.

Unfortunately there are periods of the year when one site gets a deluge of requests (for example; when hosting a school site, if it snows overnight, all the parents will check the site in the morning at around about 8am).

Zope is not particularly quick on it’s own – Apache’s “ab” reports that a dual core server with plenty of RAM can manage about 7-14 requests per second – which isn’t that many when you consider each page on a Plone site will have a large number of dependencies (css/js/png’s etc).

Varnish is a reverse HTTP proxy – meaning it sits in-front of the real web server, caching content.

So, as I’m using Debian Lenny….

  1. apt-get install -t lenny-backports varnish
  2. Edit /etc/varnish/default.vcl
  3. Edit Apache virtual hosts to route requests through varnish (rather than directly to Zope)
  4. I didn’t need to change /etc/default/varnish.

In my case there are a number of Zope instances on the same server, but I only wanted to have one instance of varnish running. This is possible – but it requires me to look at the URL requested to determine which Zope instance to route through to.

So, for example, SiteA runs on a Zope instance on localhost:10021/sites/sitea. My original Apache configuration would contain something like :

<IfModule mod_rewrite.c>
   RewriteEngine on
   RewriteRule ^/(.*)$1 [L,P]

To use varnish, I’ll firstly need to tell Varnish how to recognise requests for sitea (and other sites), so it can forward a cache miss to the right place, and then reconfigure Apache – so it sends requests into varnish and not directly to Zope.

So, firstly, in Varnish’s configuration (/etc/varnish/default.vcl), we need to define the different backend server’s we want varnish to proxy / cache. In my case they’re on the same server –

backend zope1 {
.host = "";
.port = "10021";
backend zope2 {
.host = "";
.port = "10022";
Then, in the 'sub vcl_recv' section, use logic like :
if ( req.url ~ "/sites/sitea/VirtualHostRoot") {
   set req.backend = zope1;
if ( req.url ~ "/siteb/VirtualHostRoot") {
    set req.backend = zope2;

With the above in place, I can now just tell Apache to rewrite Sitea to :

RewriteRule ^/(.*)$1 [L,P]

Instead….. and now we’ll find that our site is much quicker 🙂 (This assumes your varnish listens on localhost:6081).

There are a few additional snippets I found – in the vcl_fetch { … } block, I’ve told Varnish to always cache items for 30 seconds, and to also overwrite the default Server header given out by Apache etc, namely :

sub vcl_fetch {

    # ..... <snip> <snip>

    # force minimum ttl for objects

    if (obj.ttl < 30s) {

        set obj.ttl = 30s;


    # ... <snip> <snip>

    unset obj.http.Server;

    set obj.http.Server = "Apache/2 Varnish";

    return (deliver);

I'm happy anyway. :)
Use 'varnishlog', 'varnishtop' and 'varnishhist' to monitor varnish.

Twitter Weekly Updates for 2010-01-10

  • Snowing. B61. 2/10 #
  • Feast lodge, bromsgrove: Don't bother. Food better at Johns mega balti. Take away fail. How can you screw up naan bread ffs? #
  • Entered the Shakespeare marathon (April 28ish). #Stratford #running #marathon #
  • I sometimes wish my toddler was a camel. Then he might not wake at 5am to demand food. #
  • Tried the thick winter duvet last night for the first time this year. I was too hot in, and any parts sticking out froze. #fail #
  • Bacon and salad baguette. Nom nom. Mince pie. Nom nom. Caramel shortbread. Nom nom. </lunch> #
  • 20 pressups. Surprised i could do that many. Prob need to work on breathing. #
  • Pondering doing the 100 pressups thing. Iphone app purchased. Step 1 done. #
  • I (well technically work) bought windows 7 professional today. First windows I've bought since win2k 10(?) years ago. #
  • couchdb is behaving itself now; 3 node continuous replication works well. #
  • Not impressed with couchdb on Karmic 🙁 Can't stop it (crap init script?); can't tell couchdb to listen on a different IP #fail #ubuntu #
  • Facebook iPhone app finally does push notificatons too it seems. #
  • Facebook iPhone app now does contact picture syncing. Seems to have missed a few people in the sync. #facebook #iPhone #
  • Perhaps i should find some smashing pumpkins to listen to on my iphone. GnR's Get In the Ring seems to be the highlight of my day so far :-/ #
  • *wishes he could be sledging / throwing snow at Rachel etc* #
  • 13th june. Mtb enduro event. Kington, Herefordshire. Excellent route. Shame I can't do it – kat's due on 11th 🙁 #
  • Don't panic – IPhone works again after holding both buttons for 10+ seconds. Catastrophe averted. #
  • RT @steve_parkes RT @mezzle: @fzzpop #hack <– Awesomest machine ever. Homer Simpson button presser? #
  • Hooray. Sledging here I come. First time in 10+ years. It's good being the boss #
  • Snow snow 🙂 #uksnow b61 10/10 🙂 #
  • Snow is falling. B61 #uksnow 1/10 #
  • Hmm snow. Sledges. Perhaps. #
  • Surprised no one has tweeted about the google homepage and it's falling apple. Is this a new feature, or old hat that's now boring? #google #
  • I <3 Chrome; it seems so much more responsive and more nimble than fat Firefox. Shame it lacks extensions though. #
  • At least I got to run merrily past all the poor people in cars going back to work – queuing on the Birmingham road :). So long suckers. #
  • Hat, gloves, traccy trousers (no shorts) and two top layers. Just about warm enough running. Weather app thinks -5 to -7'c. #ukcold b61 #
  • Good walk around Dodford in circles getting lost this afternoon. Shame it was so cold – Rowan suffered 🙁 #
  • Some stupid American keeps trying to retrieve the password for gingerdog@gmail. FFS it's not yours. Stop using it for airmiles & shopping #

Twitter Weekly Updates for 2010-01-03

  • RT @4eversleepless Safe Danger 3: "Ice". If you enjoy it, please re-tweet etc. Happy New Year! #
  • I am a mince pie seeking missile. #
  • Patch, The Jack Russell illustrated his fine breeding and training by catching a partridge. Was it road kill or canine skill? #
  • I must be getting out of shape – Bonzo the fat Labrador who is never walked kept up over 3-4 miles running :-/ #
  • Hello 2010 #
  • RT @greensql Joined to the SQL Injected web sites.. #
  • Snow in ld8 #uksnow #
  • I seem incapable of resisting the @asda dark chocolate covered almonds. Good thing they're on offer 🙂 #
  • RT @loudmouthman oooh Look the real Drobo Killer #
  • A Geek Xmas Story #
  • I am in an igloo playing with rowan. [Un]fortunately it's not made of snow/ice. Where is the snow goddammit?… Weather forecast fail! #
  • There is a distinct lack of snow here … 🙁 #
  • Watching the great escape. Toddler doesn't seem impressed. #
  • Just created a @rowangoodwin Twitter account so I can stop confusing people when I pretend to be him on this account. #
  • My favourite Xmas toy, on it's 'rails' … It makes a great noise my parents love. #

Twitter Weekly Updates for 2009-12-27

  • I appear to be tweaking. #
  • And we're back home. As I'm sure everyone will be fascinated to know. Joy. (who stole all the #uksnow ?!?) #
  • Operation keep @thegingerdog awake with coke taste test: regular – ok, diet – yuck, diet citrus – horse semen would prob taste better. #
  • No tcx078l you may not have your luggage. Pls wait more now. #
  • Waiting to board. Rowan seems determined to remain awake. Grr #
  • My last siesta, during which I dreamt of crocodiles daddy! (toddler) (dangerous sand croc) #
  • Festive sandcastle in st Agustin #
  • .@chairmummiaow says I'm turning into my father in law. Perhaps I need to work on the beer gut etc though. #
  • Rowan now greets people with 'hola'. He's doing better than me – "two please…. TWO PLEASE… Thanks!" #
  • Xmas nearly over. Toddler nearly asleep. Beach tomorrow and then eventually home. I miss my bed. Won't miss the bitey insect things. #
  • Getting sun burnt. Day trip a partial failure as there is f. All to do in this town and a v bored toddler. Boat trip cancelled (bad weather) #
  • At Mogan market. Can we barter. Errr no. #
  • Can you hide your feet? #
  • Park done. Lots of animals seen. Lovely scenery / setting. Tourist tat bought for toddles and nieces #
  • I can't sleep… I can't talk…. Feeling hot, hot. hot…. either sunburnt face or virus :-/ palmitos park today – #
  • #Mtb – Good route but guide/group slow. Fed up braking. Mud scared them. Heavy rain made for interesting journey. #
  • Surprised how little bandwidth Twitter /fb/email are using. Must use iPhone more to get value from o2. £50 for 50mb :/. Used ~15mb so far. #
  • On bus, waiting for them to pick up other mtb'ers. Why was I the first? :-/ off to different route, like I care / know any different. #
  • Crossing fingers and hoping I can go
    Mountain biking tomorrow. Got NO kit with me, so I'll suffer and be uncomfortable. Weather permitting!! #
  • Post cards sent, toddler asleep. Weather lovely (sun, warm, wind). To the beach with the sand eater later I think. #
  • I eat sand. Nom nom. The parents couldn't stop laughing. #
  • After 2.5 hours of flying, we finally had this thankfullyhe slept to landing despite other kids screams on descent #
  • Fyi parents – I like ice cream, but only with sprinkles AND only from mummy's bowl. #

Twitter Weekly Updates for 2009-12-20

  • Xmas SMS sent to annoy lots of people. Now to find food in gatwick. #
  • Waiting in duty free. Nothing seems like a bargain or any cheaper. #
  • It's a world of fun, when you're a toddler #
  • Stupid car frozen on the inside too…. Hurry up holiday. Don't be closed gatwick. #
  • Snow. Paranoid wife thinking we won't get to go on holiday…. #
  • Toddler vomit; Chinese flavour. Not the best start to bed time. #
  • RT @codepo8: A single sperm has 37.5MB of DNA information in it. That means a normal ejaculation represents a data transfer of 1,587.5TB! #
  • hmm.. sugar rush…. Better not stop eating though #
  • I have an xmas biscuit addiction. #
  • Rowans ELC musical keyboard has dead keys and the microphone does not work. Great. Reminds me of my first computer – zx spectrum +2 (DOA) #
  • RT @garywkfung Ping! is FREE AGAIN! Get your friends on Ping! iPhone-2-iPhone msging (*Please RT!*) #
  • RT @grifferz (NSFW language, via prh) #

Twitter Weekly Updates for 2009-12-13

  • /me hopes tonight is peaceful and uninterrupted. #
  • The toddler lost the battle. Now what do I so with no earphones on the sofa with him asleep on me? #
  • Rowan isn't too keen on the idea of sleeping :-/ #
  • Interesting 2 days training the devs behind the BBC's glow javascript library in php / zend framework etc. Now home for the weekend 🙂 #
  • Stupid woman. Trying to pay for the bus with an inadequate supply of what seems to ve 5p coins. #
  • I wish $idiot would stop trying to recover the password for gingerdog @ gmail. While I'm at it – stop using my address to signup to stuff. #
  • Well that rocked. Thanks dominion theatre and cast 🙂 #
  • We will, we will…. Rock you. (waiting for the performance to start) #
  • Well I jumped into the river, too many times to make it home, I'm here on my own, drifting all alone….. #gnr #
  • Fuckit – water bottle has leaked in my bag; laptop + adaptor appears to have escaped TFFT. 2nd time for this to happen. Had better learn! #
  • Support call with mr paranoid cookie hater *sigh*. #
  • s/Firefox/Chromium/g perhaps…. seems much quicker at least, and now has required extensions. Shame it didn't import passwords from FF. #
  • Wonders if anyone on bromsgrove freecycle passed GCSE English. #
  • My nose should win an award for the volume, various colours,range of consistencies and stamina in snot production. #fedup #wanttransplant #
  • Well at least rowan seems awake and happy this morning. #
  • My iPhone is only 4(?) months old yet the case is cracked. Think I need to encase it in something rigid. #
  • I hate post office queues #
  • Right own up! Who gave Rowan speed? He's been hyper hyper super hyper toddler since being home from nursery. #
  • Think I, or bromsgrove missed some heavy rain today. No great loss. #
  • iPhone gun app discovered. Suspect we may not get to use our phones much when we next see the nieces. (literal) Banana gun vs bazooka…. #
  • Good episode of stargate universe (s1e10). #

Twitter Weekly Updates for 2009-12-06

  • Mcvities light chocolate digestives are rubbish. Biscuit too hard. #sundaylunchfail #
  • The instore asda radio started telling me about @asda today. Is Twitter too mainstream? Does it matter? Might as well 'subscribe' for now. #
  • Aiming a virtual kick at a London data centre. #
  • Stupid m6 and m1. All crawling #
  • Arrived in milton Keynes; I still couldn't find my way without the gps. Too many roundabouts. #
  • Number 2 is due for 11th June 2010. Now you all know. There will be no more. #
  • RT @greensql GreenSQL-FW: 1.2.0 has just been released! Now with #postgresql support. #security #
  • Daiseychain nursery fail. #
  • Haha <marquee> hahaha #
  • I hate hardware. Stupid motherboard with a broken SAta controller. Grrr #
  • Christmas shopping nearly complete. Thank —- #
  • Sleeping on a towel …. *sigh* stupid virus and sweat eager body #

wapiti – web application vulnerability scanner (super quick review/intro)

Today, I finally looked at Wapiti, which is a web application vulnerability scanner. It operates on a black box basis (i.e. it doesn’t see the underlying PHP/ASP/Java source code), and effectively tries to ‘break’ any forms on a page.

In order to get it to do anything useful, you’ll probably need to provide it with a cookie file to use. Unfortunately, I couldn’t originally get the provided ‘’ file to work, as the application in question just posted the login form details to ” (i.e. <form action=” method=’post’>)…. after a bit of hacking I fixed this, but it took some time.

Installation is relatively easy – download the .zip file, extract it and change directory into it (e.g. cd wapiti-2.0.X)

Anyway, given we have “webapp” installed at http://orange/webapp, and we wish to test it, we might do something like the following :

  1. cd src/net
  2. python ~/cookie.txt http://orange/webapp/login.php
  3. Enter username/password etc as required to complete the login form
  4. Script exists, check the contents of ~/cookie.txt – it will look something like :

Set-Cookie3: PHPSESSID=3d20841af5de43c718732d80e5d78fe3; path=”/”; domain=”orange”; path_spec; expires=”2010-01-04 22:42:47Z”; version=0

Now we can use wapiti to test any urls ‘behind’ the login screen (as it were) :

wapiti http://orange/webapp/search.php –cookie ~/cookie.txt -v 2 -o ~/report -x http://orange/webapp/logout.php

(We need to exclude the logout page, else our session will get destroyed when wapiti spiders that page…)

Depending on how good the application is, you may see output like :

Found permament XSS in http://orange/webapp/search.php
attacked by http://orange/webapp/search.php?area=on&client_id=on&county=on with fields county=crzbl79tqr&status=x57cjl7m14&website=vk59qqbgmp&name=<script>alert(’11byq04xd1′)</script>&client_id=on&region=on

and similar for the other vulnerabilities.

If I point my web browser at file:///home/david/report I’ll see a nice HTML report listing the vulnerabilites and so on – similar to the below…

report output etc
report output etc

Wapiti appears to detect:

  • SQL Injection holes
  • Cross Site Scripting (XSS) holes
  • File inclusion (local/remote)
  • Command execution vulnerabilities
  • and others

I’m a bit annoyed I’ve only found this tool now – but also glad I’ve finally found it. I’ve been looking for something that can pick up XSS holes for ages (SQL Injection stuff I could already test using SQLMap, and ensuring I only ever used prepared statements).

Update (July 2011) – cookie file format has changed to xml –

<?xml version="1.0" encoding="UTF-8"?>
  <domain name="uk">
    <domain name="co">
      <domain name="palepurple">
        <domain name="david">
            <cookie name="PHPSESSID" path="/" value="vmabdv5giph334aq33vb0add67" version="0"/>
            <cookie name="globdisc" path="/" value="yes" version="0"/>

Twitter Weekly Updates for 2009-11-29

  • Perhaps I should clean the car more often – latest 'find' a delicious ripe Banana skin #
  • Webbs 'garden centre' – where you buy christmas tat which people don't want #
  • New mobile carrier – – free in-network calls? #
  • ASUS EeePC 900 – broken screen – extra battery+memory / spare parts? #
  • Beans on toast with marmite was surprisingly good last night. What else can i do with marmite? #
  • Cough cough cough …. Cough cough cough go the wife and toddler. #
  • Ah. Can now sign up for spotify. It didn't like me having a 1/1/1890 DoB tho the form allows it. No error msg given. #fail #
  • Hoping an excess of fruit will scare off this virus. #
  • RT @garywkfung *BLACK FRI SALE* Ping! iPhone-2-iPhone messaging is free. Get your friends pinging for FREE! #
  • What a crap night. Now to run and cough my guts up. #
  • I've got my tea sorted daddy! #
  • Spotify clearly doesn't want my money. Stupid signup form gives no error msg feedback but keeps redisplaying itself. #fail #
  • Look – proof I can cook (first time in 9+ months) #
  • #phpuk2010 tickets bought for all employees … 26th Feb 2010 … You should be there too – #php #london #
  • wishes $previous_developer had discovered fputcsv() rather than doing a /lot/ of unsafe string concatenation("," . $foo . "," is WRONG) #php #
  • Excessive screen estate ? 2×24" monitors – too much? #
  • RT @garywkfung Ping! 1.2 now live! Includes address book, in-app purchase for sending photos and other fixes #
  • Well trains at least you picked today to be crap, rather than one where I had less leeway. #
  • Hello White city. #
  • Stupid signal failure is causing trouble for this train. Good thing I've got loads of 'spare' time. #
  • RT @Openrightsgroup BBC: ORG supporters jump by 20% as protests grow: #threestrikes #
  • But I *had* to buy that massive bag of yummy licquorice and a triple choc muffin to have change for the bus!! #
  • My way, your way, anything goes tonight……. Hmm. Need more music. #
  • Confused. I thought rebooting iPhone jailbroken with blackra1n would result in it loosing jailbreak. This does not seem to be the case. :~/ #
  • Pondering trying spotify for a month or two. Thoughts lazyweb? #
  • Rowan is still asleep. 13 hours and counting. A new record. Shame it takes illness to cause this :-/ #
  • RT @Openrightsgroup No 10 petition now stands at 6.8k sigs thanks to @stephenfry @glinner #threestrikes #
  • failing to jailbreak my iPhone… (3gs v3.1.2 etc). Stupid tools. #


This will end my fit of blogging diarrhea. Honest.

On Saturday, I ran to Kidderminster (21 miles in total). It went quite well, although my left thigh ached a little and I got a sore groin. Afterwards I also noticed my feet were aching on the outside of my sole (they don’t normally)….

Yesterday morning I went running again, only for 30ish minutes and found my thigh seemed worse and my right knee was unhappy too. And my lower back aches a little.

I’m wondering if my new running shoes are responsible – or if it’s just because I somehow pushed myself too far on Saturday (considering my running routine has been a mess for the last month with me rarely managing to run more than twice a week (i.e ~8-10 miles if optimistic)).

Stay tuned. Or not. Today and tomorrow will be run-free days in the hope something will repair itself.