rsyslog filtering (with loggly)

Filtering rsyslog output being sent to loggly (e.g. from an apache imfile input)

If you’re a bit slow on the uptake, like me … this might help.

Basic logging to Loggly is simple enough –

References : https://www.loggly.com/docs/rsyslog-tls-configuration/ gets you to add in an omfwd action and a template with auth details in …

However, when you also want to mix in sending Apache logs to loggly, and at the same time want to suppress sending some lines ….. life becomes a bit harder.

Here’s what worked for me anyway… replace MAGIC_AUTH_TOKEN_HERE with your loggly auth details.

Place this in /etc/rsyslog.d/loggly.conf.

# Setup disk assisted queues
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1     # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g       # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on     # save messages to disk on shutdown
$ActionQueueType LinkedList       # run asynchronously
$ActionResumeRetryCount -1        # infinite retries if host is down

#RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/ca.d/logs-01.loggly.com_sha12.crt


$ActionSendStreamDriver gtls # use gtls netstream driver
$ActionSendStreamDriverMode 1 # require TLS
$ActionSendStreamDriverAuthMode x509/name # authenticate by hostname
$ActionSendStreamDriverPermittedPeer *.loggly.com

template(name="LogglyFormat" type="string"
string="< %pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [MAGIC_AUTH_TOKEN_HERE tag=\"Syslog\"] %msg%\n"
)


module(load="imfile") 

# Apache file inputs :

input(type="imfile"
    File="/var/log/apache2/access.log"
    Tag="apache-access"
    Severity="info"
    Facility="local7")

input(type="imfile"
    File="/var/log/apache2/error.log"
    Tag="apache-error"
    Severity="error"
    Facility="local7")


# Format for Apache things.
$template LogglyFormatApache,"< %pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [MAGIC_AUTH_TOKEN_HERE  tag=\"apache\" ] %msg%\n"

if ( $programname == 'apache-access' ) and not ( $msg contains "/something-to-skip/" ) then {
     action(
        type="omfwd" 
        protocol="tcp" 
        target="logs-01.loggly.com" 
        port="6514" template="LogglyFormatApache" 
        StreamDriver="gtls" 
        StreamDriverMode="1" 
        StreamDriverAuthMode="x509/name" 
        StreamDriverPermittedPeers="*.loggly.com"
    )
    stop
} 

# no further processing of apache-access things 
if ( $programname == 'apache-access') then stop

if ( $programname == 'apache-error' ) then {
         action(
                type="omfwd" 
                protocol="tcp" 
                target="logs-01.loggly.com" 
                port="6514" template="LogglyFormatApache" 
                StreamDriver="gtls" 
                StreamDriverMode="1" 
                StreamDriverAuthMode="x509/name" 
                StreamDriverPermittedPeers="*.loggly.com"
        )
    stop
} 

if ( $programname == 'apache-error') then stop

# Anything else ... sent to loggly.
action(
    type="omfwd" 
    protocol="tcp" 
    target="logs-01.loggly.com" 
    port="6514" template="LogglyFormatApache" 
    StreamDriver="gtls" 
    StreamDriverMode="1" 
    StreamDriverAuthMode="x509/name" 
    StreamDriverPermittedPeers="*.loggly.com"
)

Using hitch with varnish on Debian Jessie

I ended up needing to install hitch on a server recently, so the https:// traffic could be routed through Varnish (along with the existing ‘http’ stuff) for performance reasons.

The server only runs WordPress sites, so there are WordPress specific things in the Varnish configuration (vcl) file below.

Versions: Varnish 5.2, Hitch 1.4.4, Apache 2.4 and Debian Jessie.

Continue reading “Using hitch with varnish on Debian Jessie”

Trac and Git on Debian Lenny

Random Brain dump – Trac 0.11 with Git on Debian Lenny; this worked for me …

We’ll use /var/git/ as the location where our git repositories live – e.g. /var/git/repository1, /var/git/repository2 etc.

So, assuming you have git-core installed, create the Git repository:

mkdir -p /var/git/repository

cd /var/git/repository

git init –bare

Next, install the trac-git extension so trac can do ‘git’ like things:

apt-get install trac-git

Ensure the WebDAV FS module is enabled in Apache:

a2enmod dav_fs

And Expose where the Git repository is on the web server – e.g.

cd /var/www
ln -s /var/git git

And add something like :

<Location /git>
     DAV on
</Location>
To which ever virtual host file has /var/www as it’s document root (probably ‘default’); this should then mean that any git repositories you create in the future will automatically be exposed via Apache. It should go without saying that you should put some sort of Apache authentication check on this location.

Next, let’s create the Trac Repository:

trac-admin /var/trac/repository initenv \
     repository sqlite:db/trac.db git /var/git/repository

Configure Trac to do Git things:

Edit /var/trac/repository/conf/trac.ini and make sure it contains something like :

[components]
tracext.git.* = enabled
[git]
cached_repository = false
git_bin = /usr/bin/git
persistent_cache = false
shortrev_len = 7

Finally, just make sure permissions are correct:

chown -R www-data /var/trac/repository
chown -R www-data /var/git/repository

Then finally, restart Apache, point your web browser at the trac repository (assuming you’ve already setup Trac via e.g mod_python or similar) and you’ll probably seen an AssertionFailed error (with no helpful message). This seems to be a one off when the repository is empty… so try :

mkdir ~/src/tmp
cd ~/src/tmp
git init
echo 'test' > hello.txt
git add hello.txt
git commit
git config remote.upload.url https://user@remote.host/git/repository/
git push upload master

(If this fails with some unhelpful message like :

orange:~/src/tmp $ git push upload master

....

error: Cannot access URL https://david@remote.host/git/repository/,
      return code 60

error: failed to push some refs to
      'https://david@remote.host/git/repository/'
It’s probably moaning about you having an invalid (or at least non-trusted) SSL Certificate (as I happen to) – create ~/.gitconfig and set it to contain :
[http]
sslVerify = false
You might also wish to read this kernel.org doc on git setup with Apache