Caveats with Logstash udp input type and event splitting
What I noticed with logstash version 1.1.5 and even 1.1.1, is that when using the UDP input type in logstash, each line of input is not split into a separate event. The multiline filter expects each line of input to be a separate event, otherwise it won’t work properly.
Therefore, before using the multiline filter, I had to use the split filter.
The split should not hurt other supported inputs, like stdin, of the same type since data coming from stdin will already have lines of input that is splitted into separate events so a split filter won’t have any negative impact on that input.
Here is an example of what I had to do which made everything work as it did before:
input {
udp {
port => 9999
type => "some_rails_input"
}
}
filter {
# splitting each line into multiple events since UDP input creates one multiline event instead of breaking each line as an event
split {
type => "some_rails_input"
}
multiline {
type => "some_rails_input"
stream_identity => "%{@source_host}.%{@type}"
pattern => "###"
negate => true
what => "previous"
}
}
output {
stdout { debug => true debug_format => "json"}
}
If you’re wondering why I’m using the ‘###’ pattern as an indicator for the end of the multiline, it’s because I modified the default rails log so that Logstash can work with it. Logstash’s multiline filter currently doesn’t work with the default Rails log format.
I submitted this as a bug in logstash. Hopefully they’ll fix the issue. In the meantime, use the technique I’ve described at http://ionrails.com/2012/10/11/collecting-aggregating-rails-log-stacktrace-with-logstash-shipper-agent/
