rsyslog filtering (with loggly)

Filtering rsyslog output being sent to loggly (e.g. from an apache imfile input)

If you’re a bit slow on the uptake, like me … this might help.

Basic logging to Loggly is simple enough –

References : https://www.loggly.com/docs/rsyslog-tls-configuration/ gets you to add in an omfwd action and a template with auth details in …

However, when you also want to mix in sending Apache logs to loggly, and at the same time want to suppress sending some lines ….. life becomes a bit harder.

Here’s what worked for me anyway… replace MAGIC_AUTH_TOKEN_HERE with your loggly auth details.

Place this in /etc/rsyslog.d/loggly.conf.

# Setup disk assisted queues
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1     # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g       # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on     # save messages to disk on shutdown
$ActionQueueType LinkedList       # run asynchronously
$ActionResumeRetryCount -1        # infinite retries if host is down

#RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/ca.d/logs-01.loggly.com_sha12.crt


$ActionSendStreamDriver gtls # use gtls netstream driver
$ActionSendStreamDriverMode 1 # require TLS
$ActionSendStreamDriverAuthMode x509/name # authenticate by hostname
$ActionSendStreamDriverPermittedPeer *.loggly.com

template(name="LogglyFormat" type="string"
string="< %pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [MAGIC_AUTH_TOKEN_HERE tag=\"Syslog\"] %msg%\n"
)


module(load="imfile") 

# Apache file inputs :

input(type="imfile"
    File="/var/log/apache2/access.log"
    Tag="apache-access"
    Severity="info"
    Facility="local7")

input(type="imfile"
    File="/var/log/apache2/error.log"
    Tag="apache-error"
    Severity="error"
    Facility="local7")


# Format for Apache things.
$template LogglyFormatApache,"< %pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [MAGIC_AUTH_TOKEN_HERE  tag=\"apache\" ] %msg%\n"

if ( $programname == 'apache-access' ) and not ( $msg contains "/something-to-skip/" ) then {
     action(
        type="omfwd" 
        protocol="tcp" 
        target="logs-01.loggly.com" 
        port="6514" template="LogglyFormatApache" 
        StreamDriver="gtls" 
        StreamDriverMode="1" 
        StreamDriverAuthMode="x509/name" 
        StreamDriverPermittedPeers="*.loggly.com"
    )
    stop
} 

# no further processing of apache-access things 
if ( $programname == 'apache-access') then stop

if ( $programname == 'apache-error' ) then {
         action(
                type="omfwd" 
                protocol="tcp" 
                target="logs-01.loggly.com" 
                port="6514" template="LogglyFormatApache" 
                StreamDriver="gtls" 
                StreamDriverMode="1" 
                StreamDriverAuthMode="x509/name" 
                StreamDriverPermittedPeers="*.loggly.com"
        )
    stop
} 

if ( $programname == 'apache-error') then stop

# Anything else ... sent to loggly.
action(
    type="omfwd" 
    protocol="tcp" 
    target="logs-01.loggly.com" 
    port="6514" template="LogglyFormatApache" 
    StreamDriver="gtls" 
    StreamDriverMode="1" 
    StreamDriverAuthMode="x509/name" 
    StreamDriverPermittedPeers="*.loggly.com"
)

First steps with a Pixelbook

So, my 2009 MacBook Pro decided to slowly die … and after dithering for about 3 years over what to buy to replace it …. I chose a Google Pixelbook (i7 variant, 500Gb NVME disk etc) (via eBay)…..

So, my 2009 MacBook Pro decided to slowly die … and after dithering for about 3 years over what to buy to replace it …. I chose a Google Pixelbook (i7 variant, 500Gb NVME disk etc) (via eBay).

Here are some findings …

  • Installing Linux within the supported VM environment is straight forward (see docs) but it’s a 4.14 kernel with Debian Stretch. Given it’s using BTRFS I’d prefer a newer kernel (or at least the ability to choose what kernel the VM boots…)
  • I can’t seem to find a way of getting a clipboard manager that works across all applications (so I can copy+paste multiple things between windows). I’ve been using ClipIt for years on my main desktop.
  • Sharing files between the Linux environment and native ChromeOS is kind of annoying (go into the Files app, and drag/drop the file(s) around). The UI hints at there being shared folders, but I’m guessing they’ll be enabled in a future release.
  • Sound from a Linux app doesn’t work (when running vlc within the Linux VM, there is no sound); apparently a known bug so I’ll hope it’ll get fixed soon.
  • It’s fast. Especially browsing the web.
  • It’s not burnt my lap yet (unlike the MBP)
  • It’s possible to get sound to stutter from e.g. Play Music, if you’re doing a reasonable amount of I/O (like PHPStorm rebuilding it’s indexes)
  • Installing PHPStorm (and other Linux apps) was fairly straight forward (either via apt or however I’d normally do it in Linux) and generally works fine …
  • There’s no “right click” for the mouse pad; instead you do a double finger tap. You can ctrl+click or use a two finger tap.
  • Tablet mode is great for Android Apps – I’ve tried a couple of toddler apps and they just worked fine.
  • Not all Android apps work properly – e.g. using Authenticator Plus for 2FA auth codes – doesn’t seem to be able to sync with my Google Drive backup and when opening it, there are always two windows for some reason.
  • Thankfully you can “right click” on the launcher tray and configure it to auto-hide and pin apps you use often.

I’m toying with the idea of replacing ChromeOS with a native Linux install; but I’ve not yet seen enough evidence to suggest that it’ll work well.

Hopefully the Campfire project will have a release soon …. Until then I’ll be watching https://www.reddit.com/r/pixelbook etc

Using hitch with varnish on Debian Jessie

I ended up needing to install hitch on a server recently, so the https:// traffic could be routed through Varnish (along with the existing ‘http’ stuff) for performance reasons.

The server only runs WordPress sites, so there are WordPress specific things in the Varnish configuration (vcl) file below.

Versions: Varnish 5.2, Hitch 1.4.4, Apache 2.4 and Debian Jessie.

Continue reading “Using hitch with varnish on Debian Jessie”

compiling a vanilla kernel to a .deb

As I keep losing this….

Creating a .deb kernel package from a vanilla kernel.org kernel :

  1. wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.4.24.tar.xz
  2. tar -xf linux-4.4.24.tar.xz
  3. cd linux-4.4.24
  4. cp /boot/config-whatever .config
  5. Optionally: edit and set: CONFIG_DEBUG_INFO=n to stop the *dbg* package being generated
  6. make olddefconfig
  7. make deb-pkg -j6 LOCALVERSION=-dg1