git config –global gpg.format ssh
git config –global user.signingkey ~/.ssh/id_ed25519.pub
git config –global commit.gpgsign true
Hopefully that’ll result in my github commits being signed…. and when I forget how to do it …
docker proxy image download
Docker doesn’t like me for some reason, and I often get really bad download speeds from my home IP address.
One crude fix, is use an external server I can access as a SOCKS proxy.
To do so, edit / create /etc/systemd/system/docker.service.d/http-proxy.conf and put in it :
[Service] Environment="HTTP_PROXY=socks5://localhost:8888"
Then, restart/reload systemd ( systemctl daemon-reload ; service docker restart )
Then, setup your socks tunnel using some remote server you have SSH access on –
ssh -D 8888 david@some.remote.server
Hopefully now when you do a ‘docker-compose up‘ it won’t take forever.
A crude way of speed testing it, is to try running this bash snippet :
token=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/mysql:pull" | jq -r .token) curl -v https://registry-1.docker.io/v2/library/mysql/blobs/sha256:2a72cbf407d67c7a7a76dd48e432091678e297140dce050ad5eccad918a9f8d6 -H "Authorization: Bearer $token" -L > /dev/null
Mockery (test doubles/mocking dependencies)
[This is a relatively old post I think I forgot to publish….]
Previously, I’d only used PHPUnit’s mock implementation; however lately I’ve been exposed to Mockery.
While they both achieve broadly the same result (at least from my point of view), here’s an example of how to mock dependencies with Mockery.
Class to test:
class TestMe {
private $db;
public function __construct(PDO $db) {
$this->db = $db;
}
public function doesSomethingImportant(array $data) : bool {
$stmt = $this->db->prepare("UPDATE my_table SET field2 = :field2 WHERE field1 = :field1");
return $stmt->execute(['field1' => $data['field1'], 'field2' => $data['field2']);
}
}
And to test ….
....
use Mockery as m;
class MyTest extends \PHPUnit\Framework\TestCase {
public function tearDown() {
m::close();
parent::tearDown();
}
public function testDoesSomethingImpotant() {
$fakePdo = m::mock(\PDO::class);
$stmt = m::mock(\PDOStatement::class);
$stmt
->shouldReceive('execute')
->withArgs(['field1' => 'hello', 'field2' => 'world'])
->once()
->andReturn(true);
$fakePdo
->shouldReceive('prepare')
->withArgs(['UPDATE my table SET field2 = :field2 WHERE field1 = :field1'])
->once()
->andReturn($stmt);
$testClass = new TestMe($fakePdo);
$this->assertTrue(
$testClass->updateSomething(['field1' => 'hello', 'field2' => 'world']);
}
}
So that’s all well and good, and with a little imagination you can see how a method that does some calculation could be tested to ensure it does the calculation correctly and performs the appropriate database query. It obviously requires you inject all dependencies in to the class (else you can’t pass in the appropriate mocks!)
While this test is isolated from the underlying database, it doesn’t ensure you code will work – what if someone’s changed the database schema – your test will still (incorrectly) pass …
You can also create ‘fake’ errors throughout your code which might help give you a higher code coverage score 🙂
$pdo = m::mock(PDO::class);
$pdo->shouldReceive('prepare')
->withArgs(['bad sql'])
->andThrow(PDOException::class);
$pdo->shouldReceive('prepare')
->withArgs(['whatever good sql'])
->once()
->andReturn($stmt);
$pdo->shouldReceive('prepare')
->withArgs(['more good sql'])
->once()
->andReturn($stmt);
....
curl, jq and slightly dynamic input to a service
I keep forgetting the syntax for these two things, so there’s a chance writing it here will help me remember.
Possibly of use/relevance for: elasticsearch or Debezium….
Continue reading “curl, jq and slightly dynamic input to a service”
PostgreSQL unbuffered queries and PHP (cursors)
From using MySQL, I’ve used the ‘unbuffered queries‘ feature a number of times. It’s where you don’t fetch the entire resultset into memory at once – which is necessary if you’re retrieving more data than you have memory available. If’s often also generally gets results/data back to you sooner.
Continue reading “PostgreSQL unbuffered queries and PHP (cursors)”
vagrant commands for the lazy and forgetful
Vagrant commands :
- vagrant up  – aka “clocking on at work, time to boot up the VM”
- vagrant status – aka “did I leave that VM running last night?”
- vagrant halt – aka “time to go home or change project”
- vagrant destroy – aka “VM is broken; kill it with fire and deploy a virgin VM to start again”
- vagrant package default –output /path/to/output/file.box – aka “I like this VM, I think I’ll keep a copy around somewhere for future purposes”
- vagrant init – aka “Get this project started …”
Obviously requires you have a reasonably good ‘Vagrantfile’ to start with…..
Checking PHP code for compatibility issues
One project I occassionally hack on is Xerte Toolkits.
Yesterday on the mailing list it came up that someone was trying to use XOT with PHP4.
After getting over some initial shock that people still use PHP4 (it was end-of-lifed in August 2008) I wondered how easy it would be to check the status of a code base to find how incompatible with PHP4 it now is.
My initial thought was to find a list of functions which had been added with PHP5 and then just grep the code for them, but it turns out there is a much nicer approach – PHP_CompatInfo
Installation was fairly straight forward – like :
pear channel-discover bartlett.laurent-laville.org
pear install bartlett/PHP_CompatInfo
Annoyingly the documentation seemed well hidden – but once I found it (http://php5.laurent-laville.org/compatinfo/manual/2.3/en/index.html#_documentation) it was pretty easy to use, and the ‘phpci’ command did all I needed –
Examples :
1. List global variables in use :
$ phpci print --reference PHP5 --report global -R .
436 / 436 [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++>] 100.00%
BASE: /home/david/src/XOT/trunk
-------------------------------------------------------------------------------
PHP COMPAT INFO GLOBAL SUMMARY
-------------------------------------------------------------------------------
GLOBAL VERSION COUNT
-------------------------------------------------------------------------------
$_GET 4.1.0 1
data $_GET 4.1.0 2
debug $_GET 4.1.0 2
export $_GET 4.1.0 2
file $_GET 4.1.0 1
firstname $_GET 4.1.0 1
....
2. Find all PHP5 functions in use :
$ phpci print --report function -R . | grep 5. 436 / 436 [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++>] 100.00% spl_autoload_register SPL 5.1.2 1 simplexml_load_file SimpleXML 5.0.0 1 iconv_set_encoding iconv 4.0.5 1 iconv_strlen iconv 5.0.0 10 iconv_strpos iconv 5.0.0 38 iconv_strrpos iconv 5.0.0 3 iconv_substr iconv 5.0.0 33 dirname standard 4.0.0 53 fclose standard 4.0.0 51 file_put_contents standard 5.0.0 6 fopen standard 4.0.0 55 fread standard 4.0.0 57 fwrite standard 4.0.0 50 htmlentities standard 5.2.3 1 md5 standard 4.0.0 1 scandir standard 5.0.0 1 str_split standard 5.0.0 3 REQUIRED PHP 5.2.3 (MIN) Time: 0 seconds, Memory: 28.25Mb
and finally,
3. Class usage :
$ phpci print --report class -R . | grep 5. 436 / 436 [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++>] 100.00% Exception SPL 5.1.0 1 InvalidArgumentException SPL 5.1.0 2 REQUIRED PHP 5.1.0 (MIN)
4. All class usage :
i.e. without me grep’ping the results.
$ phpci print --report class -R . 436 / 436 [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++>] 100.00% BASE: /home/david/src/XOT/trunk ------------------------------------------------------------------------------- PHP COMPAT INFO CLASS SUMMARY ------------------------------------------------------------------------------- CLASS EXTENSION VERSION COUNT ------------------------------------------------------------------------------- Exception SPL 5.1.0 1 InvalidArgumentException SPL 5.1.0 2 PHP_CompatInfo 4.0.0 1 Snoopy 4.0.0 2 StdClass 4.0.0 2 Xerte_Authentication_Abstract 4.0.0 6 Xerte_Authentication_Factory 4.0.0 4 Xerte_Authentication_Guest 4.0.0 1 Xerte_Authentication_Ldap 4.0.0 1 Xerte_Authentication_Moodle 4.0.0 1 Xerte_Authentication_Static 4.0.0 1 Xerte_Authetication_Db 4.0.0 1 Zend_Exception 4.0.0 2 Zend_Locale 4.0.0 7 Zend_Locale_Data 4.0.0 19 Zend_Locale_Data_Translation 4.0.0 6 Zend_Locale_Exception 4.0.0 28 Zend_Locale_Format 4.0.0 3 Zend_Locale_Math 4.0.0 14 Zend_Locale_Math_Exception 4.0.0 9 Zend_Locale_Math_PhpMath 4.0.0 11 archive 4.0.0 3 bzip_file 4.0.0 1 dUnzip2 4.0.0 3 gzip_file 4.0.0 1 tar_file 4.0.0 3 toolkits_session_handler 4.0.0 1 zip_file 4.0.0 2 ------------------------------------------------------------------------------- A TOTAL OF 28 CLASS(S) WERE FOUND REQUIRED PHP 5.1.0 (MIN) ------------------------------------------------------------------------------- Time: 0 seconds, Memory: 27.50Mb -------------------------------------------------------------------------------
Which answers my question(s) and so on.
Netbeans vs Vim … round 1 … fight
So, I think I’ve changed ‘editor’. Perhaps this is a bit like an engineer changing their calculator or something.
For the last 10 years, I’ve effectively only used ‘vim‘ for development of any PHP code I work on.
I felt I was best served using something like vim – where the interface was uncluttered, everything was a keypress away and I could literally fill my entire monitor with code. This was great if my day consisted of writing new code.
Unfortunately, this has rarely been the case for the last few years. I’ve increasingly found myself dipping in and out of projects – or needing to navigate through a complex set of dependencies to find methods/definitions/functions – thanks to  the likes of PSR0. Suffice to say, Vim doesn’t really help me do this.
Perhaps, I’ve finally learnt that ‘raw’ typing speed is not the only measure of productivity – navigation through the codebase, viewing inline documentation or having a debugger at my fingertips is also important.
So, last week, while working on one project, I eventually got fed up of juggling between terminals and fighting with tab completion that I re-installed netbeans – so, while I’m sure vim can probably do anything netbeans can – if you have the right plugin installed and super flexible fingers.
So, what have I gained/lost :
- a 10 second slow down – that’s how long it takes to start Netbeans – even with an SSD and a desktop that should be quick enough (perhaps this is justification for a faster desktop?). Lesson learnt – do not close Netbeans.
- indexing of the code – although this takes some time when importing an existing project – it does make it very quick to ‘find uses’ of function/class X
- the ability to easily jump to a function/class definition (right click, find uses …)
- a list of ‘todo’ items – from my code comments … this has led to me removing some old legacy cruft…
- the ‘refactor’ functionality – i.e. rename function old_name() and change all instances of it to new_name()
- code indentation seems to work better under netbeans
- inline ‘warnings’ (i.e. unused variable, previously undefined variable[x], syntax errors etc) – so I don’t have to wait for them to appear in a browser or elsewhere.
- inline documentation etc while typing (i.e. type in in_array( and it reminds me where the needle and haystack are)
- with the right them, a nicer editor Window (mine is based on BrandonDark)
x – Fails with global variables on legacy projects though – in that netbeans doesn’t realise the variable has been sucked in through a earlier ‘require’ call.
I did briefly look at sublime a few weeks ago, but couldn’t see what the fuss was about – it didn’t seem to do very much – apart from have multiple tabs open for the various files I was editing.
SQL Injection with added magic_quotes assistance (the joys of legacy code maintenance)
Sometimes you really have to laugh (or shoot yourself) when you come across legacy code / the mess some other developer(s) left behind. (Names slightly changed to protect the innocent)
class RocketShip {
function rahrah() {
$sql = "insert into foo (rah,rahrah,...)
values ( '" . $this->escape_str($this->meh) . "', ...... )";
mysqli_query($this->db_link, $sql) or
die("ERROR: " . mysqli_error($this->db_link));
$this->id = mysqli_insert_id($this->db_link);
}
function escape_str($str)
{
if(get_magic_quotes_gpc())
{ $str = stripslashes($str);}
//echo $str;
//$clean = mysqli_real_escape_string($this->db_link,$str);
//echo $clean;
return $str;
}
// ....
function something_else() {
mysqli_query($this->db_link,
sprintf("insert into fish(field1,field2) values('%s', '%s')",
$this->escape_str($this->field1),
$this->escape_str($this->field2));
}
}
You’ve got to just love the :
- Lack of Error handling / logging.
- Functionality of the escape_str function which is only making matters worse (and could never have worked due to the variable names)
- Use of sprintf and %s ….(obviously %d could be useful)
- Documentation?
Dare I uncomment the mysqi_real_escape_string and fix escape_str’s behaviour?
In other news, see this tweet – 84% of web apps are insecure; that’s a bit damning. But perhaps not surprising given code has a far longer lifespan than you expect….
Zend_Cache – automatic cache cleaning can be bad, mmkay?
$customer uses Zend_Cache in their codebase – and I noticed that every so often a page request would take ~5 seeconds (for no apparent reason), while normally they take < 1 second …
Some rummaging and profiling with xdebug showed that some requests looked like :
Note how there are 25,000 or so calls for various Zend_Cache_Backend_File thingys (fetch meta data, load contents, flock etc etc).
This alternative rendering might make it more clear – especially when compared with the image afterwards :
while a normal request should look more like :
Zend_Cache has a ‘automatic_cleaning_mode’ frontend parameter – which is by default set to 10 (i.e. 10% of all write requests to the cache result in it checking if there is anything to garbage collect/clean). Since we’re nearly always writing something to the cache, this results in 10% of requests triggering the cleaning logic.
See http://framework.zend.com/manual/en/zend.cache.frontends.html.
The cleaning is now run via a cron job something like :
$cache_instance->clean(Zend_Cache::CLEANING_MODE_OLD);