Logstash Rails – Collecting / Aggregating Rails Log Stacktrace with Logstash shipper agent

How to prepare and aggregate Rails logs for Logstash:

Before tagging your stacktrace events from your Rails production.log, for example, you will want to write a multiline filter in Logstash which will include each stacktrace event as a separate entry. Each section of your Rails log starts with the word “Processing” and ends with two empty lines. Refer to your own production.log to see the pattern. I had a bit of trouble with Logstash’s multiline filter and the rails production.log since Logstash’s pattern matching for the multiline filter is designed to match against content within the same line. However, the pattern which separates multiline stacktrace events in Rails are indeed two newlines.

Logstash’s multiline filter’s goal, according to one of the engineers working on the Logstash project, “is to evaluate ONE line to see if it should be grouped with others. So indeed your pattern cannot match (and for info you want a true multiline regex you should put (?m) in front of it).”

Modify the default Rails log format to work with Logstash:

So, what I decided was to open up the standard logging in Rails ActiveSupport::BufferedLogger class and modify it to add my own separator of ### (3 hash symbols) on the same line which to me would indicate the end of an event request block stacktrace.

Here is what that code looks like in a new initializer file in config/initializers/modify_log_format.rb:

For Rails 3.2.x (Tested on 3.2.9):  (See below for what to do in Rails 3.0.x and Rails 2.3.x)

#
# ActiveSupport patches
#
module ActiveSupport

class BufferedLogger

def flush
 @log_dest.write("###\n") if Rails.env.production?
 @log_dest.flush
 end

def respond_to?(method, include_private = false)
 super
 end
 end
end

Essentially Rails 3.2 has deprecated the ActiveSupport::BufferedReader#flush method and what I’ve done above is to have it again respond to the flush method, append ### followed by a newline to each request / event in the log and then flush out the buffer. Do the above, only if it makes sense for you to do it. Hopefully there will be a better solution in future versions of Rails which would take into consideration logstash and not interweave events in the logs as well. The solution above may fix that issue but I haven’t verified that yet.

For Rails 3.0.x: (Tested on 3.0.3. See below for what to do in Rails 2.3.x)

#
# ActiveSupport patches
#
module ActiveSupport

class BufferedLogger

# Here is where the rails production.log is modified to work with logstash.
 def flush
 @guard.synchronize do
 unless buffer.empty?
 old_buffer = buffer
 all_content = StringIO.new
 old_buffer.each do |content|

# Rails logs add unnecessary newlines so stripping all empty characters and then adding one newline per line
 all_content.puts content.strip
 end

# For Logstash - adding this delimeter to log file so that logstash multiline filter can pattern match
 all_content << "###\n" if Rails.env.production?

@log.write(all_content.string)
 end

# Important to do this even if buffer was empty or else @buffer will
 # accumulate empty arrays for each request where nothing was logged.
 clear_buffer
 end
 end
 end
end

For Rails 2.3.4:

#
# ActiveSupport patches
#
module ActiveSupport

# Format the buffered logger with timestamp/severity info.
 class BufferedLogger
def flush
@guard.synchronize do
unless buffer.empty?
old_buffer = buffer
@log.write(old_buffer.join)

# adding this delimeter to log file so that logstash multiline filter can pattern match
@log.write("###\n")
end

# Important to do this even if buffer was empty or else @buffer will
# accumulate empty arrays for each request where nothing was logged.
clear_buffer
end
 end
 #
 end
end

Logstash multiline filter which knows how to parse your modified Rails log:

Here we create a multiline filter which will include everything before the standard Rails double new line which separates each request in the rails production.log. The pattern means ‘a line that starts with a hash’. The ‘negate’ means exclude everything that matches the pattern. The ‘what’ means that we want everything ‘previous’ to the pattern which is everything that came before it.

Your filter would look something like this:

multiline {
type => "a_type_which_matches_your_input_and_output"
pattern => "#\s*"
negate => "true"
what => "previous"
 }

Logstash Rails - Collecting / Aggregating Rails Log Stacktrace with Logstash shipper agent, 3.0 out of 5 based on 2 ratings