Tag: ec2

Posted on

A little over ten years ago ….

A little over 10 years ago, in a previous role/company, I designed and implemented a website hosting environment (with a catchy name of “w 3 p cloud”) to ….

  • support WordPress/LAMP like environments
  • have some sort of process/file isolation between sites, so a malware infection in one shouldn’t be able to spread/reach other sites
  • have resource limits in place (the business also liked the idea of charging for more “firepower”, I just wanted to try and stop one site from doing a denial of service on others)
  • be hosted in AWS (EC2) because it was cool to be moving to the cloud (despite the cost)

Eventually, I settled on using LXC containers with a Varnish server as a HTTP frontend router. Hosting within AWS (EC2) was basically a non-negotiable requirement and there weren’t many alternatives either.

A crude web UI was added for managing sites, which was quickly adapted to have a JSON API on top. Then background tasks were added – involving a job queue (originally gearman, later on beanstalk) and some management of iptables rules.

Fast forward to 2026, and it’s still (just about) in use.

As a programmer/developer, we often don’t think too much about the distant future when creating something. We’ve got immediate deadlines, and thinking more than 2-3 years is difficult. There are plenty of uncertainties in life afterall!

Over the years, there have been multiple upgrades of various bits (Varnish, PHP, Debian release, Linux kernels etc).

I think it had between 5-8k sites at it’s peak, and I often found it amusing when I realised I was ordering/using a website hosted on it as a member of the public.

Anyway, all good things come to an end, I guess …. and finally a migration to something bigger/better/faster/shinier has begun, as this count of sites being hosted in it shows:

graph of number of sites being hosted over time, showing a recent decline

Posted on

Automated snapshot backup of an Amazon EBS volume

I found the following Python script online, but it didn’t really work too well :

http://aws-musings.com/manage-ebs-snapshots-with-a-python-script/

EBS – Elastic Block Storage …

I had to easy_install boto, to get it to work.

I’m not sure the Debian python-boto package in Lenny is up to date.

Anyway, $server now has :

from boto.ec2.connection import EC2Connection
from boto.ec2.regioninfo import RegionInfo

from datetime import datetime
import sys

# Substitute your access key and secret key here
aws_access_key = 'MY_AWS_ACCESS_KEY'
aws_secret_key = 'MY_AWS_SECRET_KEY'
# Change to your region/endpoint...
region = RegionInfo(endpoint='eu-west-1.ec2.amazonaws.com', name='eu-west-1')

if len(sys.argv) < 3:
    print "Usage: python manage_snapshots.py volume_id number_of_snapshots_to_keep description"     
    print "volume id and number of snapshots to keep are required. description is optional"
    sys.exit(1) 

vol_id = sys.argv[1] 
keep = int(sys.argv[2]) 
conn = EC2Connection(aws_access_key, aws_secret_key, region=region) 
volumes = conn.get_all_volumes([vol_id]) 
print "%s" % repr(volumes) 
volume = volumes[0] 
description = 'Created by manage_snapshots.py at ' + datetime.today().isoformat(' ') 
if len(sys.argv) > 3:
    description = sys.argv[3]

if volume.create_snapshot(description):
    print 'Snapshot created with description: ' + description

snapshots = volume.snapshots()
snapshot = snapshots[0]

def date_compare(snap1, snap2):
    if snap1.start_time < snap2.start_time:
        return -1
    elif snap1.start_time == snap2.start_time:
        return 0
    return 1

snapshots.sort(date_compare)
delta = len(snapshots) - keep
for i in range(delta):
    print 'Deleting snapshot ' + snapshots[i].description
    snapshots[i].delete()

And then plonk something like the following in /etc/cron.daily/backup_ebs :

for volume in vol-xxxx vol-yyyyy vol-zzzz
do
	/path/to/above/python/script.py $volume 7 "Backup of $volume on $(date +%F-%H:%m)"
done

Which keeps 7 backups for each volume with a time/date stamp in each description.