Description
I'm attempting to use Flume 1.6 and a spooldir source to stream a directory of logs through a broker and into logstash kafka input. One of the header values passed from the Flume spooldir source is the file path and name of the file picked up. I'm expecting this to be exposed somehow upon reception in logstash so that I can extract it and make parsing and routing decisions based on the logfile type.
Are the metadata options exposed with decorate turned on all that's available? How can I see the kafka message and its content at a deeper level in logstash? I don't see it with debug on.
Looking a the plugin code leads me to think I could add things here, but not sure if there are other options available. I added one that showed me the cryptic kafka message, but that doesn't help until it's decoded.
event['kafka'] = {'msg_size' => message_and_metadata.message.size,
'topic' => message_and_metadata.topic,
'consumer_group' => @group_id,
'partition' => message_and_metadata.partition,
'message' => message_and_metadata.message, <-- I added this
'key' => message_and_metadata.key}
end
Thanks for pointers. I'd think this would be a common use case for folks using Flume 1.6 to ship logs to logstash using kafka.
Doug