setup 'bro' to log into elasticsearch

A recurring question on the bro mailling list is, "how do you setup bro to ship logs to elasticsearch?"
I'll explain how I've setup my bro network monitor and hopefully it will be useful to others as well.

Setup bro to log in json format:

There are others recipes where you can use grok to parse the text logs to json-ify it, but I find it easier and more stable if the application can log directly into json.

In you local.bro add the following lines:

redef LogAscii::json_timestamps = JSON::TS_ISO8601;
@load tuning/json-logs

The 1st line changes the format of the time stamp from UNIX epoch to Iso 8601, that make it easier for logstash to parse the date into @timestamp, the 2nd line loads a tunning script that will turn your logs into json.

restart bro with broctl restart --clean and you should be set.

Setup logstash input to feed on the logs:

Note that a few assumptions are made here.
I like to organise my logstash conf in different files, named [input, output, filter]-<purpose>.conf, this make it easy for templating and debugging. Off course you can always stuff it into a gigantic file, that entirely up to you :-) and in the output-default.conf, you will need to change the cluster name to whatever name your cluster has. you will also need to change the input path to match yours and so on ...
final gotcha; on Debian you will need to apt-get install logstash-contrib to have the translate plugin.

In /etc/logstash/conf.d/input-bro.conf, we specify the source files, the type and that the logs are in json.

input {
    file {
       type => 'bro_logs' 
       path => [ '/opt/spool/manager/*.log' ]
        codec => "json"
    }
}

In /etc/logstash/conf.d/filter-bro.conf, the translate will add a field named conn_state_full with the full text based on the content of conn_state, the grok will add a field named bro_type with the type of bro logs (conn, weird, dns, ssl ...) based on the file name, we could do that in the input part as well by giving all the file name and adding a specific type but I'm lazy and I always forget one file so ...

filter {
        if [type] == "bro_logs" {
                date { 
                        match => [ "ts", "ISO8601" ]
                }
                translate { field => "conn_state" destination => "conn_state_full" dictionary => [ "S0", "Attempt", "S1", "Established", "S2", "Originator close only", "S3", "Responder close only", "SF", "SYN/FIN completion", "REJ", "Rejected", "RSTO", "Originator aborted", "RSTR", "Responder aborted", "RSTOS0", "Originator SYN +  RST", "RSTRH", "Responder SYN ACK + RST", "SH", "Originator SYN + FIN", "SHR", "Responder SYN ACK + FIN", "OTH", "Midstream traffic" ] }
                grok {
                        match => { "path" => ".*\/(?<bro_type>[a-zA-Z0-9]+)\.log$"} 
                } 
        }
}

In /etc/logstash/conf.d/output-default.conf

output { 
        elasticsearch {
                cluster => 'elasticsearch'
        }
        #stdout { codec => rubydebug } 
}

restart logstash.

Conclusion:

By now, you should have logs in json and logstash shipping them to elasticsearch, you can now start kibana and visualize our bro logs:

pretty pictures ^_^

Honeypot malware collection

The setup

I've been toying with the idea of installing a kippo honeypot in my company network for a while but 6 months ago I had some time to make a basic setup.
By basic, I mean that kippo runs:

• as a captive user
• in a chroot
• listen to a non privileged port (I used iptables forwarding to redirect external traffic targeted to port 22 to kippo's active port) 
• on a host that's blacklisted in the IDS and iptables config of the rest of the machine (to prevent a lateral move)

The non basic version would involve selinux and strong auditing policy, in case you wonder.

All that is to avoid some skilled people to be able to exploit a vulnerability in kippo and move further on in my network.

And then I let it run for a while ...

The deception

It turns out that, kippo is very popular and that means that most scanning scripts have evolved to detect kippo and quickly run away. 

Some scanner will flag you as honeypot if the password is too easy (root/root), some will try to run commands that kippo doesn't support, many use sftp to try to login ...

I had to use this fork instead, note that this fork also take care of json logging (logstash!) and ssh algorithm fingerprint which is also popular

The hit

After switching to a more capable version of kippo, it was like I opened a big malware tap, I collected 2898 samples, most of them (2101) are ELF-32 executable, 382 are 64 bit executable around 200 perl IRC bot and so on, all in all pretty good !

The 10 most common ones account for nearly 2/3 of the total:

183 1a8712007f9ef593044350226b829a9fb25f91ad ==> elf32
186 acbe528883175ce934df4edd4fff045a0e2d2d8f ==> elf32
187 bcf7c4b4621a6452f8ace5e1c0df78c71f7ae4bb ==> "C" source jessica_biel_naked_in_my_bed.c
187 dc063902fc457a2d13b0d91ebc4d508bf6bfd118 ==> elf32
188 ba61480ec4062c3386a7b1e559dab2f0baf5e98f ==> elf32 (always fdsfsfvff, left handed?)
188 ec22fac0510d0dc2c29d56c55ff7135239b0aeee ==> elf32
189 059964612c1ac7c928b81a99da67aed3f3a41865 ==> elf64 (always rewgtf3er4t)
189 44e569a191a5d7bd720c7af06c2fd81a501a245b ==> elf32 to replace udevd
190 0e76f4c72295fe851b775dac8c49ec53108f1df6 ==> elf64
236 27e67a31ffc2797340a02133a4bfab5584faa65d ==> elf32

There is a very popular vmsplice exploit (jessical biel ...), so far nothing spectacular all the sha1's existed on Virus total around the same date I received my 'copy'

The weird

Notable mention, some automated tools assumed that because the username / password combination succeeded the device on which it just log must be the expected ones. 

As an amusing side effect, I now have malware build for mips and ARM:

25dc278de8f8b80cea05e9afc4faac6df9b0638b  ==> MIPS32
2a63299784407db16cd3168a197fe57070b1ff83 ==> ARM
3e73c1a31580a6d0e65c4c8a436ec4be8f00c496 ==> MIPS32
4d90877a832ae21befd5a5556b2bfec3c2404c35  ==> ARM
7130128fc2bcafa4b4fe0ba6159399432d833dfd  ==> MIPS-I
7d0ab04aa3c835956d3fe6549ec7bf0223931468  ==> ARM
7e54ec563e186f225b2d04e0f8f1d28dffdd6fb1  ==> ARM
8103179432bcc189f81810bacd191e4440f3a0a3  ==> ARM
9b3b2c7eecad5ab0bf5cb37f094ceb951d7ae52c  ==> ARM
c421bfebe129aa4179e769b56fccb37bb026a1b2  ==> ARM (not stripped)
db79126d667109c3df6138d55b8669fe1a1f10f4  ==> MIPS32

Targeted to routers, ADSL modems, IP cameras and other embedded devices ...

The end?

I'm still working on that project as time allows, I'm very curious to understand the purpose of the MIPS/ARM malware, I suspect some of them change the dns settings to redirect / mitm the traffic of the appliance but I suspect there's more than that.