I've been looking for a dtrace alternative on linux for a while and recently came accross sysdig which looks promising.
A few interting one liners:

get all the threads id of a running process (mysqld):

sysdig -p %thread.tid proc.name=mysqld| sort -u

Show the activity of a single thread:

sysdig thread.tid=5445 proc.name=mysqld

The intersting point of sysdig compared to other is the ability to script 'chisel' allowing to write a complex probe, the default install comes with a few

Here's one of mine, getting the top allocating threads of a process:

--[[
Author: Jérémie Banier
Contact: jbanier@gmail.com
Date: 08 Sep 2014
 
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License version 2 as
published by the Free Software Foundation.


This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see .
--]]

-- The number of items to show
TOP_NUMBER = 30

-- Chisel description
description = "Show the top " .. TOP_NUMBER .. " threads allocating memory. You can use filters to restrict this to a specific process, thread or file."
short_description = "Top threads by memory allocation"
category = "Performance"

-- Chisel argument list
args = {}

-- Argument notification callback
function on_set_arg(name, val)
 return false
end

-- Initialization callback
function on_init()
 chisel.exec("table_generator", 
  "thread.tid",
  "Thread ID",
  "evt.count",
  "# Calls",
  "evt.type=brk",
  "" .. TOP_NUMBER,
  "none")
 return true
end

(save it in ~/.chisels then invoke it with sysdig -c topmemory proc.name=mysqld)

Sadly it still miss a few dtrace feature like enabling a probe when you got into specific user land function or the ability to display a stack trace but the tool is still young :)

There's a JSON output format available which means you could trace all reads and writes, send them to kibana and graph them or trace all syscall failing with ENOMEM, EAGAIN, ... graph and know when you're getting behind in terms of capacity. all kind of cool stuff :)

A journey in the country of Winnie the malware

A couple of weeks ago I've deployed a honeypot in our network perimeter to have an idea of how aggressive the peoples scanning our network really are. I've decided to install kippo a medium interaction honeypot, medium interaction means you can log on and run a set of limited commands and it acts like a *real* machine, it traces everything you do, save everything you downloads ... To make sure the joke wasn't on me, I run kippo in a chroot as a non privileged user it listen on port 2222 and iptables does the forwarding to kippo for those not on my network, pretty neat.

1st catch:

After a few hours, the first connection starts to appears and the 1st users try to login. The funny thing is, they all use sftp and not the expected ssh ? The reason for that is kippo is a popular honeypot and it _doesn't_ support sftp yet, so the attackers use sftp to avoid falling into honeypot ! Luckily a patched version exist that support sftp, once I used that one people stick around a bit more but not that long, the next trick in the attacker sleeve is 'iptables', kippo doesn't implement the command soooo ... you get the idea. Good news again, kippo is easy to extend and simply adding a text file containing the output of the iptables command to "txtcmds/sbin/iptables" is enough to lure some automated scanner into the trap (until next week or so)

Passwords!:

One of the interesting intel to collect is what password do the attacker try ? Well here's a small sample of the most popular passwords you shouldn't use:

     18 [admin/123123]

     18 [admin/1234567890]

     18 [admin/12345678]

     18 [admin/1234]

     18 [admin/123qwe!@#]

     18 [admin/142536]

     18 [admin/1qaz2wsx]

     18 [admin/data]

     18 [admin/qweasd]

     18 [admin/rootme]

     19 [admin/123456]

     19 [admin/P@ssw0rd]

     19 [admin/admin123]

     19 [admin/passw0rd]

     19 [admin/qwe123]

     19 [admin/root123]

     19 [admin/root@123]

     21 [admin/12345]

     22 [admin/password]

     23 [admin/root]

     27 [admin/admin]

     32 [root/root]

    214 [root/admin]

Note the high score of the root/admin combo, the classics never dies or so it seems.

Malware collection:

Another cool feature of kippo, is that it will backup anything the attacker downloads by curl, wget and so on and again pretty quickly you get a few samples so far I have received:

822dd344bfa3ab37ebc968140f5f6296  http___mdb7_cn_8081_star 1.1M

5cdf87129e45d9a3132b7b4840237190 http___121_40_196_12_65533_wawa 834K

I'll try to reverse engineer those samples as time allows (not so much I'm afraid) but I can already provide a few info:

star:

by running 'string' on the 1st sample (star) I find a large list of ip addresses, likely compromised hosts used for C&C:

61.132.163.68

202.102.192.68

202.102.213.68

202.102.200.101

58.242.2.2

202.38.64.1

211.91.88.129

211.138.180.2

218.104.78.2

[...] 

The executable is not stripped and contains lots of mangled symbols indicating that it has been coded in C++, ldd show now external dependencies meaning that for portability it was statically linked, all in all it seems pretty neat !

wawa:

The seconds sample looks a bit more elaborate, like star it is statically linked but this times all symbols have been stripped and running strings on it reveals the following:

$Info: This file is packed with the UPX executable packer http://upx.sf.net $

$Id: UPX 3.91 Copyright (C) 1996-2013 the UPX Team. All Rights Reserved. $

Which means  that it was packed to make the work of potential reverse engineers more difficult but not impossible since UPX is open source. 

I hope to give it a go one of these days and try out http://www.radare.org at the same time, in the mean time if you have details on those malware or want more info on them don't hesitate to drop me a note.

Elasticsearch housekeeping

Logstash is very cool but the underlying Elasticsearch engine can take up a lot of space, so I wrote a small cleaning up script that runs daily to either discard older than 30 days data or optimize active tables.

#!/usr/bin/python

import pycurl
import json
import StringIO
from datetime import datetime, timedelta

retentionDays = 30

c = pycurl.Curl()
b = StringIO.StringIO()

c.setopt(c.URL, 'http://127.0.0.1:9200/_status')
c.setopt(pycurl.WRITEFUNCTION, b.write)
c.perform()

blob = json.loads( b.getvalue() )

for index in blob['indices']:
 if 'logstash' in index:
  old = datetime.now() - timedelta(days = retentionDays)
  indexDate = datetime.strptime(index, "logstash-%Y.%m.%d")
  if old > indexDate:
   print "delete", index
   c.setopt(pycurl.CUSTOMREQUEST, "DELETE")
   c.setopt(c.URL, ('http://127.0.0.1:9200/%s').format(index))
   c.perform()
  else:
   print "optimize", index
   c.setopt(c.URL, ('http://127.0.0.1:9200/%s/_optimize').format(index))
   c.perform()

Turns out there is a much better tool to do all Elasticsearch related housekeeping called curator but anyway sometimes it's nice to make your own scripts :-)

Scapy - Rmcp / ipmi fuzzer

J'avais déjà joué avec scapy  dans un billet précédent et je me suis demandé à quel point c'est difficile de rajouter un nouveau protocole... Suite à ça j'ai cherché un truc raisonnablement simple et sympa et je me suis penché sur Rmcp / ipmi ...
Ipmi d'après Wikipedia est: "L'Interface de gestion intelligente de matériel, (ou IPMI, Intelligent Platform Management Interface) est un ensemble de spécifications d'interfaces communes avec du matériel informatique (principalement des serveurs) permettant de surveiller certains composants (ventilateur, sonde de température, ...), mais également de contrôler l'ordinateur à distance, reboot, interrupteur, console à distance."
En gros ça se présente sous la forme d'un chip intégré à la carte réseau d'un serveur et ça écoute pour recevoir des commande bas niveau de type combien de courant consomme tu ? coupe l'alimentation, effectue un redémarrage par acpi et ainsi de suite ... Les firmwares embarqués sont toujours de bon candidat au fuzzing car pas souvent mis à jour et mis à jour pénible et c'est pas comme si ça n'avait pas posé des problèmes dans le passé ...

#! /usr/bin/env python
#vim: set fileencoding=latin-1
# Author: Jérémie Banier
# Date: Oct. 29 2013
# Purpose: implement / test ipmi protocol with scapy
# Based on test add-ons sample 
# usage:

import logging
# Set log level to benefit from Scapy warnings
logging.getLogger("scapy").setLevel(1)

from scapy.all import *

class Rmcp(Packet):
    name = "Remote Management Control Protocol"
    fields_desc=[ LEShortField("Version",0x06),
        ByteField("Sequence", 0xFF) ,
        XByteField("Type and Class", 0x07) , 
        ByteEnumField("Authentication type", 0, {0:'None', 6:'RMCP+'}), ]

bind_layers( UDP, Rmcp, sport=623 )
bind_layers( UDP, Rmcp, dport=623 )

class IPMISessionLayer(Packet):
    name = "IPMI Session Wrapper"
    fields_desc=[ IntField("Session sequence number", 0),
            XIntField("Session ID", 0),
            ByteField("Message length", 0), ]

class IPMISessionLayer2(Packet):
    name = "IPMI Session Wrapper v2.0+"
    fields_desc=[ ByteEnumField("Payload type", 0x10, 
                {0x10:"Open session request", 0x11:"Open session response",
                    0x12:"RAKP Message 1", 0x13:"RAKP Message 2"
                    }),
            IntField("Session sequence number", 0),
            XIntField("Session ID", 0),
            ByteField("Message length", 0), ]

bind_layers( Rmcp, IPMISessionLayer, {'Authentication type':0} )
bind_layers( Rmcp, IPMISessionLayer2, {'Authentication type':6} )

class IPMILayer(Packet):
    name = "Intelligent Platform Management Interface"
    fields_desc = [ ByteField("Target address", 0x20), ByteEnumField("Target LUN", 0x18, {0x18:"NetFN Application Request"}),
            ByteField("Header checksum", 0xc8), ByteField("Source address", 0x81),
            ByteField("Source LUN", 0x00), 
            ByteEnumField("Command", 0x38, {0x38:"Get Channel Auth. cap."}),
            ByteEnumField("Version compat.", 0x0e, {0x0e:"IPMI v2.0+"}),
            ByteEnumField("Requested privilege level", 0x04, {0x04:"Administrator"}),
            ByteField("Data checksum", 0xb5) ]

bind_layers( IPMISessionLayer, IPMILayer )

if __name__ == "__main__":
    interact(mydict=globals(), mybanner="IPMI fuzzer")

Le script en est encore à l'état d'ébauche mais on peut déjà l'utiliser pour tester l'API de Scapy:

└───> ./Rmcp.py
WARNING: No route found for IPv6 destination :: (no default route?)
Welcome to Scapy (2.2.0)
IPMI fuzzer
>>> t= rdpcap('ipmi2.pcap')
>>> t[16].show()
###[ cooked linux ]###
  pkttype= sent-by-us
  lladdrtype= 0x1
  lladdrlen= 6
  src= '\x00\x1d\tlg,'
  proto= IPv4
###[ IP ]###
     version= 4L
     ihl= 5L
     tos= 0x0
     len= 76
     id= 9063
     flags= DF
     frag= 0L
     ttl= 64
     proto= udp
     chksum= 0x5c0a
     src= 10.200.82.208
     dst= 10.201.82.207
     \options\
###[ UDP ]###
        sport= 41227
        dport= asf_rmcp
        len= 56
        chksum= 0xbb79
###[ Remote Management Control Protocol ]###
           Version= 6
           Sequence= 255
           Type and Class= 0x7
           Authentication type= RMCP+
###[ IPMI Session Wrapper v2.0+ ]###
              Payload type= Open session request
              Session sequence number= 0
              Session ID= 0x0
              Message length= 32

Il me reste encore à implémenter l'ouverture de session, ce qui permettrais de faire du brute force sur le mot de passe admin par exemple (encore qu'il faudrait pour cela que le chip ipmi soit accessible depuis internet ce qui serait pas un très bonne idée pour dire ça poliment)
Une affaire à suivre avec heureusement des détails croustillants :P et une fin heureuse ...

Using bcfg2 data to generate IDS config file.

We use Suricata as and IDS and our typical configuration file define list of hosts sharing the same firewall rules, the firewall rules are function of the purpose of the box and since our configuration management tool (BCFG2) knows all about that, I was curious to see if I could generate part of the configuration using bcfg2 ...

#!/usr/bin/python

""" the idea is to use the bcfg2 backend to generate a list of ip's and associated with heir security groups
 like bcfg2-info groups give the list of groups and bcfg2-client query -g "group" gives the list of bcfg2-client
then resolve it like SER = "1.1.1.1,1.1.1.2,...." and so on
this should allow us to genereate normal rules for every security group and stuff that into the ids to spot any non compliance with the
firewall ruleset, possibly some other shit as well ? """

import os
import re
import socket
import sys
import cmd
import getopt
import fnmatch
import logging
import lxml.etree
import traceback
from code import InteractiveConsole
import Bcfg2.Logger
import Bcfg2.Options
import Bcfg2.Server.Core
import Bcfg2.Server.Plugin
import Bcfg2.Client.Tools.POSIX


def main():
    optinfo = dict(profile=Bcfg2.Options.CORE_PROFILE,
        command_timeout=Bcfg2.Options.CLIENT_COMMAND_TIMEOUT)

    optinfo.update(Bcfg2.Options.INFO_COMMON_OPTIONS)
    setup = Bcfg2.Options.OptionParser(optinfo)
    setup.hm = "\n".join(["prepareIdsMetadata","Options:",setup.buildHelpMessage()])
    setup.parse(sys.argv[1:])

    Bcfg2.Logger.setup_logging('prepareIdsMetadata', to_syslog=False, level=0)

    bcfg2 = Bcfg2.Server.Core.BaseCore(setup)
    bcfg2.load_plugins()
    bcfg2.block_for_fam_events(handle_events=True)
    f = open("group.txt", "w")
    for group in list(bcfg2.metadata.groups.keys()):
        hosts =  bcfg2.metadata.get_client_names_by_groups([group])

        first = True
        if hosts :
            f.write(group.upper() + '="')
            for h in hosts:
                try:
                    (hostname, aliases, ips) = socket.gethostbyname_ex(h)
                    if first:
                        f.write(ips[0])
                        first = False
                    else:
                        f.write(',' + ips[0])
                except:
                    pass
            f.write('"\n')
    f.close()

if __name__ == '__main__':
    sys.exit(main())

Which will generate a text file (group.txt) containing something like that:

DEBIAN-SQUEEZE="10.5.1.21,10.5.1.22,10.5.1.23,10.5.1.24"
SIP="10.21.81.249"
MYSQLNDBAPI="10.4.16.101,10.4.16.102,10.4.16.103,10.4.16.104"

Which I can then add as groups in my suricata yaml config file, and then build rules specific to those host purpose :-) Some host appears in several groups since some of our groups are OS based (CentOS/Debian-Wheezy ... ), geographic (BE, US, ...) and purpose based (WEB, PROXY, DB, ...)

The obvious advantages are to keep everything consistent, get rid of duplicate information and automatic updates of the IDS whenever you add an hosts. bcfg2 also has a plugin for Nagios which is nice to use too to make sure no host is left behind.

Thanks to the people on #bcfg2 on freenode for their support.