Python script to backup mysql databases on Debian

Here’s a short python script I must have knocked up some time ago – and totally forgotten – hopefully it’ll be of some use to others….

Purpose: backup all MySQL databases, one in each file with a timestamp on the end. You’ll probably want to have a secondary cron job which does something like :

find /backups/mysql -mtime +5 -print | xargs -r rm

to delete old copies… changing +5 to how ever many days history you wish to have.

Method: Read /etc/mysql/debian.cnf to get login details for MySQL, connect to MySQL and ask it for a list of all databases, go through this list calling mysqldump on each one.

Code:

(Last updated: 2012/10/10 – skip trying to backup the performance_schema).

#!/usr/bin/env python
import ConfigParser
import os
import time

# On Debian, /etc/mysql/debian.cnf contains 'root' a like login and password.
config = ConfigParser.ConfigParser()
config.read("/etc/mysql/debian.cnf")
username = config.get('client', 'user')
password = config.get('client', 'password')
hostname = config.get('client', 'host')

filestamp = time.strftime('%Y-%m-%d')

# Get a list of databases with :
database_list_command="mysql -u %s -p%s -h %s --silent -N -e 'show databases'" % (username, password, hostname)
for database in os.popen(database_list_command).readlines():
    database = database.strip()
    if database == 'information_schema':
        continue
    if database == 'performance_schema':
        continue
    filename = "/backups/mysql/%s-%s.sql" % (database, filestamp)
    os.popen("mysqldump -u %s -p%s -h %s -e --opt -c %s | gzip -c > %s.gz" % (username, password, hostname, database, filename))

Twitter Weekly Updates for 2010-03-14

  • Tcp over Dns seems a success – just slow, really slow. 33kbps old skopl slow. Hurry up bethere I want those 10mbps #
  • And no real golf involved either in last tweet. Not sure I'll visit all the watering holes. Getting to 3 holes would make me happy. #
  • Beer and running don't mix too well. At least I avoided the drinking games. 9 hole golf next weekend #bromsgrove #hockey #
  • Beer and skittles. Not too interested in the skittles part tbh. #hockey #bromsgrove #
  • What's with all the little hot air balloons people seen intent on launching at night? #lazyweb #
  • Nealry walked into some idiot who was too absorbed in his phone. Muppet….. *ahem* #
  • Dog picked up by someone and now in evenjobb. *sigh*. #
  • All aboard the toddler train! #
  • finds his dad drinking coffee on Google street view #
  • Set up http://analogbit.com/tcp-over-dns_howto -initial 'test' appears to show it working; hope it saves me from being screwed in the future #
  • Discovered perl's quotemeta() and php's preg_quote() …. Wondering how it's taken so long for me to realise they exist! #
  • Notes a darkening sky and that he has no coat. This could be sub optimal. #
  • I think patch wants to be nocturnal. #
  • Off to hockey #bromsgrove #
  • Congratulates @moobert on purchasing tolberone cookies from @asda nom nom nom nom #
  • Step 1: Chocolate, Beer and Pizza. Step 2: …. Step 3: Sleep #
  • Office game of the day – http://www.cantyouseeimbusy.com/games/breakdown/indexFS.php?cname=Your%20Company #
  • Today looks like it'll be busy. #
  • Today has been more productive… Next up #phpwm zend form etc #zf http://phpwm.org #
  • I think a long hard sweaty morning fu^h^hrun is needed. Toodle-pip. #
  • Despite sending essays via SMS lately (or so it seems) I've still got 292 left to cover 9 days. Now who do I annoy next…. :-> #
  • Most unproductive day ever. Meh. #
  • Wasted 2 minutes of my life – went to MySpace; eventually remembered login details; realised it's still crap -> closing the browser window. #

Twitter Weekly Updates for 2010-03-07

  • Sweet. The now show is back. Good times. #nowshow #radio4 #BBC #
  • Lovely run up Lickey hill; http://favoriterun.com/294345 – 7:45mins/mile; 12.57 miles. Now to eat breakfast and visit/annoy @carolinegoodwin #
  • Congratulations Rowan. You've slept well. Don't rush to get up; I don't mind. When you do wake, we'll eventually go and have Lots of fun. #
  • My ankle seems worse this morning. Time to wear boots :-/ #
  • Hockey good. Left ankle not so good. Feels f–ked and swelling. Boo. #
  • Hockey time. #bromsgrove At least it's dry #
  • I seem to be in a good mood today; a welcome change. Hockey tonight, Dead mech podcast to listen to, new ear phones and a sunny day. #
  • I seem to be in a good mood today; a welcome change. Hockey tonight too #
  • Interesting day. Hyper toddler. Poor patch. #
  • Signed up to the #dyfienduro – #474 Perhaps I need to do some cycling now. Or not. #
  • fixed http://www.whoateallthepies.tv – silly rinetd and so on. Perhaps I ought to do some /paying/ work today…. #
  • It's probably a good thing i haven't got last.fm recording what I listen to on my iPhone. Guns and roses estranged would be top. #
  • Surprise surprise galaxy chocolate is now the same price in @asda as poundland. £1 for 125g. Was £1.26 in asda. #
  • Lovely sunny morning in bromsgrove – one of the rare times I miss farming. #

Twitter Weekly Updates for 2010-03-07

  • Sweet. The now show is back. Good times. #nowshow #radio4 #BBC #
  • Lovely run up Lickey hill; http://favoriterun.com/294345 – 7:45mins/mile; 12.57 miles. Now to eat breakfast and visit/annoy @carolinegoodwin #
  • Congratulations Rowan. You've slept well. Don't rush to get up; I don't mind. When you do wake, we'll eventually go and have Lots of fun. #
  • My ankle seems worse this morning. Time to wear boots :-/ #
  • Hockey good. Left ankle not so good. Feels f–ked and swelling. Boo. #
  • Hockey time. #bromsgrove At least it's dry #
  • I seem to be in a good mood today; a welcome change. Hockey tonight, Dead mech podcast to listen to, new ear phones and a sunny day. #
  • I seem to be in a good mood today; a welcome change. Hockey tonight too #
  • Interesting day. Hyper toddler. Poor patch. #
  • Signed up to the #dyfienduro – #474 Perhaps I need to do some cycling now. Or not. #
  • fixed http://www.whoateallthepies.tv – silly rinetd and so on. Perhaps I ought to do some /paying/ work today…. #
  • It's probably a good thing i haven't got last.fm recording what I listen to on my iPhone. Guns and roses estranged would be top. #
  • Surprise surprise galaxy chocolate is now the same price in @asda as poundland. £1 for 125g. Was £1.26 in asda. #
  • Lovely sunny morning in bromsgrove – one of the rare times I miss farming. #

Google News Sitemap + WordPress

Annoyingly the current version of the google-news-sitemap plugin for WordPress (v1.4) doesn’t work with some silly XML namespace error reported by google.

See http://wordpress.org/support/topic/364929 and effectively the ‘patch’ on the Google Support forum thing which works fine (there are two bits of the plugin which need updating – whcih correlate to the two parts mentioned in the posting etc)

Bit annoyed that the fix is so easy – yet the plugin hasn’t been updated yet. Grr.

Silly SoapClient

Sam made some changes to a SOAP service one customer has – and suddenly our automated tests kept failing. “WTF?” we thought.

We persistently got the same error (e.g. Fatal error: SOAP-ERROR: Encoding: object hasn’t ‘SortResults’ property in ….) yet the generated WSDL file (when viewed through a web browser etc) no longer has SortResults in it.

We checked :

  • Proxy settings,
  • Apache configuration (and restarted Apache)
  • svn reverted files
  • DNS and more.

Then it turned out the SoapClient PHP object caches the WSDL file by default in e.g. /tmp/wsdl-blahblahblah. (Where blahblahblah looks very much like an md5). And it doesn’t make much of an effort to check it’s validity. Obviously the documentation does state this, but it does seem like the wrong default behaviour to me.

Solution: Add something to the automated tests to delete /tmp/wsdl-*. OR pass into the SoapClient constructor an array of options like :

$client = new SoapClient('http://somewhere/blah.php?wsdl", array(cache_wsdl => WSDL_CACHE_NONE)

Time wasted: Too much.

Trying to not delve into wordpress

I don’t normally do anything with WordPress from a work point of view – I’ve always left such work to ‘designer’ types…

Anyway, yesterday I had a referral for someone who has two fairly busy websites (anorak.co.ukwhoateallthepies.tv) sat on a fairly beefy server (8 core, 16g of ram… oh in a few years that’ll be entry level… but I digress)… anyway, they were having performance difficulties with one site – a bit of investigation found the problem to be related to their migration from one server to another – rinetd was directing traffic from the old server, but had filled the filesystem up and was consuming all cpu time …… Easy enough to fix. Job done. Everything started working again.

After a bit more investigation I found that the two sites needed updates applying and plugins upgraded, and they had no backup job in place *doh* …. Clone the site, whizz through the wordpress upgrade routine on the clone, get the customer to OK it (he did) and then we did it on the live server…. and it looked like a success. Until an hour after I’d done the update and the customer realised part of his front page was missing….

Great. Just what I’d hoped to avoid – delving into wordpress’s code.

On opening up the theme’s index.php file it was easy to see where the content should be –  add in some debugging on the clone – and “Oh look – that ‘thing’ is empty.. it should contain ‘stuff’….”

Turns out there’s a WP_Query class; and it seems WordPress 2.9.x treats it’s query slightly differently to previous versions – ‘they’ used category_name=blah as a parameter – this no longer works, instead it needed changing to cat=1234 .. bingo, data returned; site fixed; customer happy.

I breathed a big sigh of relief. I was worried that they previous developers had made some weird customisation to wordpress core which I’d have to forward port and debug/fix.

Being the nice chap I am, I also installed xcache onto the server to help PHP out – I suspect they could cut their hardware ‘allocation’ by half and still have ample capacity to serve the sites. A few days with munin running and I’ll know for certain. Perhaps they’ll appreciate the cost saving?

Twitter Weekly Updates for 2010-02-28

  • Back from long run. Good but body now protesting; dog caused me to loose timings etc so no idea of time. http://favoriterun.com/302054 #
  • RT @bromsgrove Winner of the best curry award named: spice valley, worcs rd. http://bit.ly/amuzbx <- I agree. #
  • First ice cream van of the year just drove past, and now he's back again. Surely it's just not warm enough yet? #bromsgrove #
  • Toddler appears to be awake; I'd better sneak a shower before he finds me 😉 #
  • Back home – after an epic tube journey and drive back from #phpuk2010 – now onto a json 2 soap proxy for upcoming iphone app #
  • Thanks for a great day #phpuk2010 ! #
  • RT @andygirvan Check to see if your desired username is available at dozens of social networking sites w/ Namechk – http://namechk.com/ #
  • Aaaah phpillow api looks good. #phpuk2010 #
  • How would you deal with stale reads from varnish infront of couchdb? #phpuk2010 #
  • Nice lunch – thanks #phpuk2010 – more cakes would be good though 😉 #
  • Finally on the tube; will be slightly late for #phpuk2010 …. Oh well. Perhaps we won't need to queue when there 😉 #
  • I failed to find either #php t-shirt to wear; so opted for a #python one instead. #traitor #phpuk2010 #
  • Survived hockey 😉 v. friendly bunch; might get into the 4ths easily. Can't hit the ball properly though; blaming stick as my body's perfect #
  • Off to play hockey (hopefully) for the first time in 11(?) years. I don't remember the rules…. #bromsgrove #hockey #
  • Dyfi enduro email arrives; I contemplate doing it – when I had previously decided to give up enduros months ago. #mtb #notcycledinayear :-/ #
  • Annoyance #3 startssl.com offer free ssl certs – but only if you're willing to wait hours for 'checks' to complete … #
  • Annoyance #2 editors which reformat / reindent code. Thanks for the merge conflict. Really. #
  • Annoyance #1 subverion merging still sucks (using 1.6 etc). Ffs that file moved directory – you should still cope #
  • The dog agrees – I need to eat a sandwich or two before going to bed as it's late. Clever dog. #
  • Wondering what's wrong with his local council and who put this notice up…. http://twitpic.com/151o1e #bromsgrove #
  • How a Web Design Goes Straight to Hell http://theoatmeal.com/comics/design_hell from @oatmeal #
  • What a website – http://www.inspirationnw.co.uk/index.html …… just missing the "best viewed in netscape/ie4 button" #backtothe90s #fail #
  • Can't resist eating one more handful of Bombay mix….. Must…. Stop…. Nom nom. Crikey – all gone. Which greedy sod ate it??… Oh. Oops #
  • Finally almost worked myself up to going to #Bromsgrove #hockey club (Thurs evenings). Might have to buy a stick tho – not played in 12 yrs #
  • I need to update PalePurple's website with a few half decent examples of our PHP development work… #EveryonesSiteIsBetterThanOurs #
  • I'm happy now I've found the Gruffalo's website and there is also a film. http://www.gruffalo.com/ *YEY!* #
  • RT @guardiantech Young men are hooked on the web – even in bed http://bit.ly/9JY5Y5 – oops, that's me #
  • Stonehenge Bacon Wrapped French Toast Sticks – Bacon Worshipping http://bit.ly/9BUlol #thisiswhyyourefat #
  • Amused that customer expected a mock iphone app to have working 'bits' beyond static content. #mondayfun #
  • Action packed dog walk – Rowan fell over and cut his finger, then splashed in puddles and mud. Then Patch escaped chasing (a) cat(s)…. #

Random notes from installing postfixadmin 2.3 deb

Server: Debian Lenny

Installed .deb from here

# wget http://sourceforge.net/projects/postfixadmin/files/postfixadmin/postfixadmin_2.3_all.deb/download
....
# dpkg -i postfixadmin_2.3_all.deb
... dpkg moans about missing dependencies
Selecting previously deselected package postfixadmin.(Reading database ... 38632 files and directories currently installed.)Unpacking postfixadmin (from postfixadmin_2.3_all.deb) ...dpkg: dependency problems prevent configuration of postfixadmin: postfixadmin depends on dbconfig-common; however:  Package dbconfig-common is not installed. postfixadmin depends on wwwconfig-common; however:  Package wwwconfig-common is not installed. postfixadmin depends on php5-imap; however:  Package php5-imap is not installed.dpkg: error processing postfixadmin (--install): dependency problems - leaving unconfiguredErrors were encountered while processing: postfixadmin
# apt-get -f install
(dependencies now get installed...)

…. Goes into the Postfixadmin .deb installer thing.

  • Tell it to use Apache2 (in my case) as the webserver,
  • Give it the ‘root’ user’s database password (if MySQL).
  • Tell it to generate a password for the postfixadmin user…
  • Tell it to use the package maintainers version of /etc/postfixadmin/config.inc.php (well I did).

I think something went wrong for me – as I needed to do this afterwards; perhaps you’ll have better luck.

# mv /etc/postfixadmin/config.inc.php.ucf-dist /etc/postfixadmin/config.inc.php

Next, goto http://yourserver/postfixadmin/setup.php – you should see lots of text saying how it’s updating the database to version xyz, xyz+1 etc.

Choose a password to protect the setup.php page; press submit, and you’ll be presented with a ‘hash’ – copy and paste this into the config.inc.php file – so you get something like this on line 32 :

$CONF[‘setup_password’] = ‘my long hash thingy goes in here’;

Next, create a super admin account, using the setup.php page – remembering to type in the setup password you used to create the hash above.

That’s it.

(Well, now you need to configure Postfix and/or courier and/or dovecot etc etc)

You might want to read my other article which covers this…

Twitter Weekly Updates for 2010-02-21

  • pondering a long run tomorrow – http://favoriterun.com/302054 – 19.3 miles. Hope the weather's ok. #
  • UK Drivers – when on the motorway, if there is adequate space on your left, pull over and let those behind through. #fsckingBMWidiotontheM6 #
  • All stop. M42….. #
  • Only 8 hours in the car today. 'should' have been closer to 4 hours. #couldhavecycledtherequicker #
  • Uh oh. Snow! Cold fluffy snow. We can't go over it, we can't go under it. We've got to go through it. #
  • I've noticed that when I turn the car off, the traffic starts to move again. Git driving past in the field. *jealous*. 2 miles in 2 hours 🙁 #
  • Hmm slow slow slow #
  • Engine off. Handbrake on. Stupid m1. #
  • Spice valley, bromsgrove – nice food, shame about the lack of baby / toddler facilities. One of the better local curry houses at least. #
  • Silly cron, you really shouldn't run cron.d files which are chmod'ed 000. Oh well one mystery solved at least. #debian #bug #lenny #
  • .@Bendihossan nah. I'm cheap, dirty and run an interpreted language like php. Nat evolution not intelligent design ftw 🙂 I'd silently fail in reply to Bendihossan #
  • Soon I shall be 11111. Go me! (overflow error next year, or hardware upgrade?) #
  • It's cold. Specks of snow falling. But I Think it's not going to be a White birthday tomorrow … Boo #
  • "…. then bake mysterious things" #cake #ftw #
  • Garage: "We can't work on your car until it's been valeted", they did drill a hole in the boot (bye water) and give it two new tires… #
  • Guess I'd better give this sleep thing another try. In other news, the car interior might dry up after tomorrow (yey!) #
  • spent the morning debugging Python, stupid variable casting requirement…. #