When testing aggregate plugin get a ConfigurationError for a valid configuration

[apple-corps]

Filed a bug against logstash-filter-aggregate plugin but perhaps I should have filed it here. Have been struggling to test my logstash configurations using logstash-devutils. Testing based on this post https://gquintana.github.io/2016/09/07/Testing-Logstash-configuration.html . I define a valid logstash configuration, test it's validity via the logstash tool, run it, all good, etc.

When testing with logstash-devutils get a ConfigurationError but I am only defining one timeout. Even when slimming down my example shown in 119 to have only a single aggregate block in the filter section get the same error. It's really a shame because I was looking forward to learning more about the logstash development ecosystem. But I've lost many days trying to write simple tests.

Aggregate plugin: For task_id pattern '%{eventId}', there are more than one filter which defines timeout options. All timeout options have to be defined in only one aggregate filter per task_id pattern. Timeout options are : timeout, inactivity_timeout, timeout_code, push_map_as_event_on_timeout, push_previous_map_as_event, timeout_timestamp_field, timeout_task_id_field, timeout_tags

logstash-plugins/logstash-filter-aggregate#119

[apple-corps]

Perhaps the problem is it's being executed with multiple workers. Then I'll try to figure out how to set the workers to 1 via the tests. Furthermore there are many tests and types / styles of tests included in the project. But it's hard to navigate . It would be nice if there was some more documentation on the testing and more examples provided.

[apple-corps]

I tried adding the following settings to my test with no idea if they work or are applied. However don't have a clue if they are applied. Without a debugger, etc. this is very hard to progress against.

require 'logstash/config/pipeline_config'
  let(:worker_thread_count)     { 1 }
  let(:safe_thread_count)       { 1 }
  let(:override_thread_count)   { 1 }
  let(:pipeline_settings_obj) { LogStash::SETTINGS.clone } 
  let(:pipeline_settings) { {} }

It did not resolve the ConfigurationError

[apple-corps]

Reproduction steps: cameronkerrnz/logstash-plugin-dev#5 (comment)

[UXabre]

Hi, I've encountered this exact problem myself and manage to fix it by installing a newer devpackage
/opt/logstash/bin/ruby -S gem update logstash-devutils --ignore-dependencies

As the version installed in 7.x.x logstash installations of the devutils is 1.3.6 and this indeed does multiple instantiations of the same filter, which causes the error you have (it also took me several days to get there).

In Logstash 8.xx I think the package has been upgraded by default so should work from there on as expected.

As for your tests; are you planning on using timeouts in your aggregate filter? If so, I'd be very curious to learn how you'll write a test for that; as I'm not a ruby programmer if have unfortunately zero knowledge on this or how I can fake a timeout even. The only thing I could come up with, was setting timeout_timestamp_field => "@timestamp" and just sending some extra messages to progress time; but that feels a bit cumbersome...