Script to fix NFS (Debian Squeeze + Backports bits)

I have a NFS server running Debian Squeeze. Additionally it’s using the 3.2.x kernel from backports, and the nfs-kernel-server from backports too.

Sometimes NFS breaks, and gives helpful messages like :

mount.nfs: connection timed out

or just:

Stale NFS handle on clients.


While I’m confident that my /etc/exports and other configuration files are correct, it still insists on misbehaving.

Below is a random shell script I seem to have created to fix the NFS server –

set -e
/etc/init.d/nfs-kernel-server stop
/etc/init.d/nfs-common stop
/etc/init.d/rpcbind stop

rm -Rf /var/lib/nfs
mkdir /var/lib/nfs
mkdir /var/lib/nfs/v4recovery /var/lib/nfs/rpc_pipefs

for f in /var/lib/nfs/etab \
/var/lib/nfs/rmtab \
/var/lib/nfs/xtab; do
[ -e $f ] || touch $f

/etc/init.d/rpcbind start
sleep 2
/etc/init.d/nfs-common start
sleep 2
/etc/init.d/nfs-kernel-server start

echo "NFS may now work"

exportfs -f

Yes… “NFS may now work” … that sums it up about right.

Virtualbox 4.2 VM autostart on Debian Squeeze & Wheezy

One new feature of VirtualBox 4.2 is that it has support for auto-starting vm’s on bootup of the host server (via init etc). This means I can remove my hackish ‘su – vbox -c “VBoxHeadless –startvm VMName &”‘ additions in /etc/rc.local, and the VM’s will also hopefully be terminated gracefully on shutdown.

The docs/guides online which I could find were a bit cryptic, or incomplete, so here’s what I ended up doing :

Continue reading “Virtualbox 4.2 VM autostart on Debian Squeeze & Wheezy”

fsck paranoid?

Some random hints :

  1. Ensure the final field / column in /etc/fstab is non-zero for other filesystems you have mounted; if it’s 0 then fsck will never run on them.
  2. fsck -Cccy /dev/blah1 does a read-write (non-destructive test). Works well on SSDs 🙂
Example from /etc/fstab:
/dev/md0  /mount/point ext3 defaults 0 2

When looking at the various boxes we have in our office, I found one server had the following (run dumpe2fs /dev/whatever1):

  • Mount count:              62
  • Maximum mount count:      39
  • Last checked:             Wed Jul  9 16:09:17 2008
  • Next check after:         Mon Jan  5 15:09:17 2009
Today is 8th June 2012. Ooops.

Interestingly when I did run fsck on it, there were no errors. Is perhaps the default ext3 setting of checking every 20-30 mounts too paranoid?  It’s certainly very painful running fsck on large ‘rotating’ volumes – waiting over an hour for a server to come up is not fun.


Sponge – Shell command

Today, my sed kung-foo seemed to be lacking, so I ended up having to split the sed command over a zillion lines…

Normally I’d do something like :

sed 's/foo/bar/g' tmp.txt > tmp2.txt
sed 's/fo2/blah/g' tmp2.txt > tmp3.txt

But this obviously gets painful after a time, a different approach would be to use sponge where we can do :

sed 's/foo/bar/g' tmp.txt | sponge tmp.txt
sed 's/fo2/blah/g' tmp.txt | sponge tmp.txt

Whereby ‘sponge’ soaks up standard input and when there’s no more, opens the output file. This gets around the obvious problem that :

sed 's/foo/bar/g' tmp.txt > tmp.txt

doesn’t work because the shell opens (and overwrites) tmp.txt  before sed’s had a chance to do anything.

Checking varnish configuration syntax

If you’ve updated your varnish server’s configuration, there doesn’t seem to be an equivalent of ‘apachectl configtest’ for it, but you can do :

varnishd -C -f /etc/varnish/default.vcl

If everything is correct, varnish will then dump out the generated configuration. Otherwise you’ll get an error message pointing you to a specific line number.

yum changelog (Want to know what you’re about to upgrade on CentOS/RHEL?)

Want to see what changes you’re about to apply when doing a ‘yum update’ ? Similar-ish to how ‘apt-listchanges’ works…

On CentOS 5.6, try :

  • yum install yum-changelog python-dateutil

Note, python-dateutil seems to be an unmarked dependency – i.e. you get an error message like : “Dateutil module not available, so can’t parse dates” when trying to run ‘yum changelog all updates’.

Note, /etc/yum/pluginconf.d/changelog.conf (but this didn’t seem to need changing to me).

Now you can do :

  • yum changelog all updates
  • yum changelog all mysql-server (or whatever package you’re interested in).


Getting a kvm serial console with grub2

I’ve a few kvm guest servers, which I’ve been accessing using vnc – but this is a bit of a pain (getting port forwarding setup etc). Host and guests run Debian Squeeze with Grub2 installed/in use.

So, here’s how to do the ‘virsh console ‘ thing …

  1. Edit /etc/default/grub, specify
    GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"
    GRUB_CMDLINE_LINUX="text console=tty0 console=ttyS0,115200n8"
  2. Run update-grub
  3. Edit /etc/inittab and enable ttyS0 for logins.
  4. Reboot
  5. ‘virsh console servername’ on the kvm host.

The libvirt config files I have already have the appropriate bits in them –

<serial type='pty'><target port='0'/></serial>
<console type='pty'><target type='serial' port='0'/></console>

Varnish + Zope – Multiple zope instances behind a single varnish cache

I run multiple Zope instances on one server. Each Zope instance listens on a different port (localhost:100xx). Historically I’ve just used Apache as a front end which forwards requests to the Zope instance.

Unfortunately there are periods of the year when one site gets a deluge of requests (for example; when hosting a school site, if it snows overnight, all the parents will check the site in the morning at around about 8am).

Zope is not particularly quick on it’s own – Apache’s “ab” reports that a dual core server with plenty of RAM can manage about 7-14 requests per second – which isn’t that many when you consider each page on a Plone site will have a large number of dependencies (css/js/png’s etc).

Varnish is a reverse HTTP proxy – meaning it sits in-front of the real web server, caching content.

So, as I’m using Debian Lenny….

  1. apt-get install -t lenny-backports varnish
  2. Edit /etc/varnish/default.vcl
  3. Edit Apache virtual hosts to route requests through varnish (rather than directly to Zope)
  4. I didn’t need to change /etc/default/varnish.

In my case there are a number of Zope instances on the same server, but I only wanted to have one instance of varnish running. This is possible – but it requires me to look at the URL requested to determine which Zope instance to route through to.

So, for example, SiteA runs on a Zope instance on localhost:10021/sites/sitea. My original Apache configuration would contain something like :

<IfModule mod_rewrite.c>
   RewriteEngine on
   RewriteRule ^/(.*)$1 [L,P]

To use varnish, I’ll firstly need to tell Varnish how to recognise requests for sitea (and other sites), so it can forward a cache miss to the right place, and then reconfigure Apache – so it sends requests into varnish and not directly to Zope.

So, firstly, in Varnish’s configuration (/etc/varnish/default.vcl), we need to define the different backend server’s we want varnish to proxy / cache. In my case they’re on the same server –

backend zope1 {
.host = "";
.port = "10021";
backend zope2 {
.host = "";
.port = "10022";
Then, in the 'sub vcl_recv' section, use logic like :
if ( req.url ~ "/sites/sitea/VirtualHostRoot") {
   set req.backend = zope1;
if ( req.url ~ "/siteb/VirtualHostRoot") {
    set req.backend = zope2;

With the above in place, I can now just tell Apache to rewrite Sitea to :

RewriteRule ^/(.*)$1 [L,P]

Instead….. and now we’ll find that our site is much quicker 🙂 (This assumes your varnish listens on localhost:6081).

There are a few additional snippets I found – in the vcl_fetch { … } block, I’ve told Varnish to always cache items for 30 seconds, and to also overwrite the default Server header given out by Apache etc, namely :

sub vcl_fetch {

    # ..... <snip> <snip>

    # force minimum ttl for objects

    if (obj.ttl < 30s) {

        set obj.ttl = 30s;


    # ... <snip> <snip>

    unset obj.http.Server;

    set obj.http.Server = "Apache/2 Varnish";

    return (deliver);

I'm happy anyway. :)
Use 'varnishlog', 'varnishtop' and 'varnishhist' to monitor varnish.