varnish throttling

I came across the varnish throttle module the other day – which seems quite useful – and certainly gives better control over abusive requests than using fail2ban (in that, only specific URLs/request types can be targeted and blocked with the throttle module, while fail2ban tends to trigger the blocking of any traffic from a client which can be more painful).

Fail2ban —

• – tends to block all traffic from a host, at least on a specific port.
• – works across multiple protocols – so one fail2ban instance can protect everything on a server (it just requires a log file and regexp to look for to match abusive hosts)
• – but once a limit is hit the end user can often just not connect – they’re not shown any useful error message

Anyway, with the Throttle module, you can combine blocking based on some or all of :

• Client IP address
• Request method (GET/POST)
• Some sort of http header (backend, or client provided)
• Request URL (e.g wp-login.php)
• For one or many virtual hosts/websites/domains
• Characteristics of a URL (e.g. if it has an API key in it)

based on how you construct the ‘throttle key’ (see is_allowed(key, duration) calls below) – so for example:

1. by omitting a client_ip from the throttle key, you could protect against a distributed attack,
2. or by omitting a Http Host header, you could protect against for attacks against multiple sites on one server (if you’re hosting many wordpress blogs, for example).

Presumably, you could provide clients with a way of unblocking themselves if it’s in error.

Anyway, in my trial configuration (see segments below) I’ve specified two time periods – so combining blocking a short spike in traffic, or excessive slower longer term abuse — so e.g. 10 matches in 15 seconds or 500 matches in an hour. In my case, I’m protecting individual websites/domains, and from excessive HTTP POST requests to WordPress’s xmlrpc.php or wp-login page(s).

(Let’s hope no other websites have ‘xmlrpc.php’ files!)

Installation

Installation isn’t difficult – just follow the instructions, perhaps a bit like the below….

apt-src build varnish
export VARNISHSRC=$(pwd)/varnish-3.0.2
git clone https://github.com/nand2/libvmod-throttle.git
cd libvmod-throttle
bash autogen.sh
mkdir build
export VMODDIR=$(pwd)/build
./configure
make
make install
cp build/libvmod_throttle.so /usr/lib/x86_64-linux-gnu/varnish/vmods/

varnish – default.vcl

Random varnish.vcl snippet below (for varnish 3.0.x).

import throttle;
import std;

....
sub vcl_pass {
    ....
    std.log("PASS : " + req.http.host + " : " + req.url );

    // some arbitrary condition/check/url to match against...
    if(req.url ~ "(xmlrpc|wp-login).php" && req.request == "POST") {

        // log some info about our buckets 
        std.log("XXX Rate limit bucket: 10req/m : " + throttle.remaining_calls(req.http.Host + "wp1", "10req/m"));
        std.log("XXX Rate limit bucket: 50req/h : " + throttle.remaining_calls(req.http.Host + "wp2", "50req/h"));

        // the check ....for 10 req/minute
        if(throttle.is_allowed(req.http.Host + "wp1", "10req/m") > 0s) {
                std.log("XXX Rate limit 10req/m throttling : " + req.http.Host + " too many requests for " + req.url + " / " + req.http.X-Forwarded-For );
                error 429 "Calm down";
        }

        // the check for 50 req/hour.
        if(throttle.is_allowed(req.http.Host + "wp2", "50req/h") > 0s) {
                std.log("XXX Rate limit 50req/h throttling : " + req.http.Host + " too many requests for " + req.url + " / " + req.http.X-Forwarded-For );
                error 429 "Calm down";
        }
    }
    return (pass);
}

....

sub vcl_error {
    set obj.http.content-type = "text/html; charset=utf-8";
    std.log("XXX Error page - URL:" + req.url + " Host " + req.http.host + " Status " + obj.status);
    if ( obj.status == 429 ) {
        synthetic std.fileread("/etc/varnish/error-429.html");
    }
    else {
        synthetic std.fileread("/etc/varnish/error-503.html");
    }
    return ( deliver ) ;
}