dotdeb – apt package pinning

As of last night, Debian Security released PHP 5.4.44 for Wheezy. Wheezy shipped with PHP 5.4.12 or something like that.

DotDeb is currently on 5.4.43, and if you’ve been using it based on the assumption that it has a newer version of a package over Debian, then an upgrade will leave your PHP install in a mess (e.g. no php5-gearman or php5-imagick).

To fix this, the following in e.g. /etc/apt/preferences.d/dotdeb will help :

Package: *
Pin: origin packages.dotdeb.org
Pin-Priority: 1001

This should make apt choose dotdeb packages over Debian, even if Debian contains a newer version.

i.e. stop apt relying on just the package version number, and previously dotdeb always had a higher one.

PHP 5.4 + xcache -> PHP 5.6 + Zend OpCache memory usage

In relatively unscientific tests, it seems moving from PHP5.4 with xcache to PHP 5.6 with Zend’s OpCache can lead to approximately 50% memory reduction.

(xcache was set to have a 64mb size, Zend OpCache is also using 64mb of memory).

Continue reading “PHP 5.4 + xcache -> PHP 5.6 + Zend OpCache memory usage”

PostgreSQL unbuffered queries and PHP (cursors)

From using MySQL, I’ve used the ‘unbuffered queries‘ feature a number of times. It’s where you don’t fetch the entire resultset into memory at once – which is necessary if you’re retrieving more data than you have memory available. If’s often also generally gets results/data back to you sooner.
Continue reading “PostgreSQL unbuffered queries and PHP (cursors)”

Fixing REMOTE_ADDR when behind a proxy/varnish server

I had an annoyance where varnish proxy infront of a LAMP server and the LAMP server therefore thought all clients were from the varnish proxy – rather than the client’s real IP address – i.e. $_SERVER[‘REMOTE_ADDR’] was set to the IP address of the Varnish proxy and not that of the client’s actual IP address.

Obviously, Varnish adds the X_HTTP_FORWARDED_FOR HTTP header in when a connection comes through it; so my initial thought was to just overwrite PHP’s $_SERVER[‘REMOTE_ADDR’] setting. A bit of a hack and annoying – as I’d need to fix all sites, or have some sort of global prepend file (which is horrible).

I then discovered something which sorts the problem out  – RPAF

  • apt-get install libapache2-mod-rpaf
  • Edit /etc/apache2/mods-enabled/rpaf.conf and ensure your proxy server’s IP address is listed on the RPAFproxy_ips line (e.g. RPAFproxy_ips 127.0.0.1 89.16.176.x).
  • Restart Apache, and you’ll then find that the $_SERVER[‘REMOTE_ADDR’] value will be correct.

 

 

Checking PHP code for compatibility issues

One project I occassionally hack on is Xerte Toolkits.

Yesterday on the mailing list it came up that someone was trying to use XOT with PHP4.

After getting over some initial shock that people still use PHP4 (it was end-of-lifed in August 2008) I wondered how easy it would be to check the status of a code base to find how incompatible with PHP4 it now is.

My initial thought was to find a list of functions which had been added with PHP5 and then just grep the code for them, but it turns out there is a much nicer approach – PHP_CompatInfo

Installation was fairly straight forward – like :

pear channel-discover bartlett.laurent-laville.org
pear install bartlett/PHP_CompatInfo

Annoyingly the documentation seemed well hidden – but once I found it (http://php5.laurent-laville.org/compatinfo/manual/2.3/en/index.html#_documentation) it was pretty easy to use, and the ‘phpci’ command did all I needed –

Examples :

1. List global variables in use :

$ phpci print --reference PHP5 --report global -R . 
436 / 436 [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++>] 100.00%
BASE: /home/david/src/XOT/trunk

-------------------------------------------------------------------------------
PHP COMPAT INFO GLOBAL SUMMARY
-------------------------------------------------------------------------------
  GLOBAL                                                  VERSION         COUNT
-------------------------------------------------------------------------------
                                        $_GET             4.1.0               1
  data                                  $_GET             4.1.0               2
  debug                                 $_GET             4.1.0               2
  export                                $_GET             4.1.0               2
  file                                  $_GET             4.1.0               1
  firstname                             $_GET             4.1.0               1
....

2. Find all PHP5 functions in use :

$ phpci print  --report function -R . | grep 5.
436 / 436 [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++>] 100.00%
  spl_autoload_register                 SPL               5.1.2               1
  simplexml_load_file                   SimpleXML         5.0.0               1
  iconv_set_encoding                    iconv             4.0.5               1
  iconv_strlen                          iconv             5.0.0              10
  iconv_strpos                          iconv             5.0.0              38
  iconv_strrpos                         iconv             5.0.0               3
  iconv_substr                          iconv             5.0.0              33
  dirname                               standard          4.0.0              53
  fclose                                standard          4.0.0              51
  file_put_contents                     standard          5.0.0               6
  fopen                                 standard          4.0.0              55
  fread                                 standard          4.0.0              57
  fwrite                                standard          4.0.0              50
  htmlentities                          standard          5.2.3               1
  md5                                   standard          4.0.0               1
  scandir                               standard          5.0.0               1
  str_split                             standard          5.0.0               3
REQUIRED PHP 5.2.3 (MIN) 
Time: 0 seconds, Memory: 28.25Mb

and finally,

3. Class usage :

$ phpci print  --report class -R . | grep 5.
436 / 436 [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++>] 100.00%
  Exception                             SPL               5.1.0               1
  InvalidArgumentException              SPL               5.1.0               2
REQUIRED PHP 5.1.0 (MIN)

4. All class usage :

i.e. without me grep’ping the results.

$ phpci print  --report class -R . 
436 / 436 [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++>] 100.00%
BASE: /home/david/src/XOT/trunk

-------------------------------------------------------------------------------
PHP COMPAT INFO CLASS SUMMARY
-------------------------------------------------------------------------------
  CLASS                                 EXTENSION         VERSION         COUNT
-------------------------------------------------------------------------------
  Exception                             SPL               5.1.0               1
  InvalidArgumentException              SPL               5.1.0               2
  PHP_CompatInfo                                          4.0.0               1
  Snoopy                                                  4.0.0               2
  StdClass                                                4.0.0               2
  Xerte_Authentication_Abstract                           4.0.0               6
  Xerte_Authentication_Factory                            4.0.0               4
  Xerte_Authentication_Guest                              4.0.0               1
  Xerte_Authentication_Ldap                               4.0.0               1
  Xerte_Authentication_Moodle                             4.0.0               1
  Xerte_Authentication_Static                             4.0.0               1
  Xerte_Authetication_Db                                  4.0.0               1
  Zend_Exception                                          4.0.0               2
  Zend_Locale                                             4.0.0               7
  Zend_Locale_Data                                        4.0.0              19
  Zend_Locale_Data_Translation                            4.0.0               6
  Zend_Locale_Exception                                   4.0.0              28
  Zend_Locale_Format                                      4.0.0               3
  Zend_Locale_Math                                        4.0.0              14
  Zend_Locale_Math_Exception                              4.0.0               9
  Zend_Locale_Math_PhpMath                                4.0.0              11
  archive                                                 4.0.0               3
  bzip_file                                               4.0.0               1
  dUnzip2                                                 4.0.0               3
  gzip_file                                               4.0.0               1
  tar_file                                                4.0.0               3
  toolkits_session_handler                                4.0.0               1
  zip_file                                                4.0.0               2
-------------------------------------------------------------------------------
A TOTAL OF 28 CLASS(S) WERE FOUND
REQUIRED PHP 5.1.0 (MIN) 
-------------------------------------------------------------------------------
Time: 0 seconds, Memory: 27.50Mb
-------------------------------------------------------------------------------

Which answers my question(s) and so on.

Netbeans vs Vim … round 1 … fight

So, I think I’ve changed ‘editor’. Perhaps this is a bit like an engineer changing their calculator or something.

For the last 10 years, I’ve effectively only used ‘vim‘ for development of any PHP code I work on.

I felt I was best served using something like vim – where the interface was uncluttered, everything was a keypress away and I could literally fill my entire monitor with code. This was great if my day consisted of writing new code.

Unfortunately, this has rarely been the case for the last few years. I’ve increasingly found myself dipping in and out of projects – or needing to navigate through a complex set of dependencies to find methods/definitions/functions – thanks to  the likes of PSR0. Suffice to say, Vim doesn’t really help me do this.

Perhaps, I’ve finally learnt that ‘raw’ typing speed is not the only measure of productivity – navigation through the codebase, viewing inline documentation or having a debugger at my fingertips is also important.

So, last week, while working on one project, I eventually got fed up of juggling between terminals and fighting with tab completion that I re-installed netbeans – so, while I’m sure vim can probably do anything netbeans can – if you have the right plugin installed and super flexible fingers.

So, what have I gained/lost :

  • a 10 second slow down – that’s how long it takes to start Netbeans – even with an SSD and a desktop that should be quick enough (perhaps this is justification for a faster desktop?). Lesson learnt – do not close Netbeans.
  • indexing of the code – although this takes some time when importing an existing project – it does make it very quick to ‘find uses’ of function/class X
  • the ability to easily jump to a function/class definition (right click, find uses …)
  • a list of ‘todo’ items – from my code comments … this has led to me removing some old legacy cruft…
  • the ‘refactor’ functionality – i.e. rename function old_name() and change all instances of it to new_name()
  • code indentation seems to work better under netbeans
  • inline ‘warnings’ (i.e. unused variable, previously undefined variable[x], syntax errors etc) – so I don’t have to wait for them to appear in a browser or elsewhere.
  • inline documentation etc while typing (i.e. type in in_array( and it reminds me where the needle and haystack are)
  • with the right them, a nicer editor Window (mine is based on BrandonDark)

x – Fails with global variables on legacy projects though – in that netbeans doesn’t realise the variable has been sucked in through a earlier ‘require’ call.

I did briefly look at sublime a few weeks ago, but couldn’t see what the fuss was about – it didn’t seem to do very much – apart from have multiple tabs open for the various files I was editing.

A week of fire fighting (aka why you should <3 unit tests).

I feel like I’ve spent most of this week debugging some PHP code, and writing unit tests for it. Thankfully I think this firefighting exercise is nearly over.

Perhaps the alarm bells should have been going off a bit more in my head when it was implied that the code was being written in a quick-and-dirty manner (“as long as it works….”) and the customer kept adding in additional requirements – to the extent it is no longer clear where the end of the project is.

“It’s really simple – you just need to get a price back from two APIs”

then they added in :

“Some customers need to see the cheapest, some should only see some specific providers …”

and then :

“We want a global markup to be applied to all quotes”

and then :

“We also want a per customer markup”

and so on….

And the customer didn’t provide us with any verified test data (i.e. for a quote consisting of X it should cost A+B+C+D=Y).

The end result is an application which talks to two remote APIs to retrieve quotes. Users are asked at least 6 questions per quote (so there are a significant number of variations).

Experience made me slightly paranoid about editing the code base – I was worried I’d fix a bug in one pathway only to break another. On top of which, I initially didn’t really have any idea of whether it was broken or not – because I didn’t know what the correct (£) answers were.

Anyway, to start with, it was a bit like :

  • Deploy through some weird copy+pasting manner due to Windows file permissions
  • No unit test coverage
  • No logging
  • Apparently finished
  • Apparently working (but the customer kept asking for more changes)

Now:

  • Deployment through git; Stupid Windows file permissions fixed.
  • Merged in changes by third party graphics designer – should be much easier to track any changes he makes in the future
  • ~80% test code coverage. I’ve had to modify the ‘real’ scripts to accept a new parameter, which would make them read a local XML file (rather than the remote API/service) – in order for the tests to be reproducible (and quick to run)
  • Logging in place, so there’s some traceability
  • Better error handing
  • Calculations manually checked and confirmed.

Interesting things I didn’t like about the code :

  • Premature attempt at dependency injection – the few functions there are/were have a $db variable being passed in – but often don’t use it.
  • There is significant duplication in the code base still.
  • The code didn’t (and still doesn’t really) support testing particularly well – I’m having to retrieve output through a Zend_Http_Client call (i.e. mimicking a browser) – which means I can’t get code test coverage stats easily.
  • In some places a variable was being set to 0 (e.g. $speed) which would be used to determine a special case (if $speed == 0). Having multiple meanings in/on the same variable makes it difficult to follow what’s going on – and is a pain when the customer requests it behaves a bit differently. Really a separate parameter should have been used.

SQL Injection with added magic_quotes assistance (the joys of legacy code maintenance)

 

Sometimes you really have to laugh (or shoot yourself) when you come across legacy code / the mess some other developer(s) left behind. (Names slightly changed to protect the innocent)

class RocketShip {

    function rahrah() {
        $sql = "insert into foo (rah,rahrah,...) 
            values ( '" . $this->escape_str($this->meh) . "', ...... )";
        mysqli_query($this->db_link, $sql) or 
            die("ERROR: " . mysqli_error($this->db_link));
        $this->id = mysqli_insert_id($this->db_link);
    }

    function escape_str($str)
    {
        if(get_magic_quotes_gpc())
           { $str = stripslashes($str);}
        //echo $str;
        //$clean = mysqli_real_escape_string($this->db_link,$str);
        //echo $clean;
       return $str;
    }
// ....
    function something_else() {
         mysqli_query($this->db_link, 
            sprintf("insert into fish(field1,field2) values('%s', '%s')", 
            $this->escape_str($this->field1), 
            $this->escape_str($this->field2));

    }
}

You’ve got to just love the :

  1. Lack of Error handling / logging.
  2. Functionality of the escape_str function which is only making matters worse (and could never have worked due to the variable names)
  3. Use of sprintf  and %s ….(obviously %d could be useful)
  4. Documentation?

Dare I uncomment the mysqi_real_escape_string and fix escape_str’s behaviour?

In other news, see this tweet – 84% of web apps are insecure; that’s a bit damning. But perhaps not surprising given code has a far longer lifespan than you expect….