Support for multi workers?

[vikin91]

I am currently unable to run this plugin in a multi-worker fluentd setup.

(...) [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Plugin 'http_ext' does not support multi workers configuration (Fluent::HTTPOutput)"

Are there any plans to add support for multi workers?

[cosmo0920]

Upstream plugin had supported multi-worker Fluentd setup.
Is it enough for you?

[vikin91]

Thanks for the reply! There was a reason I switch to this plugin after trying many others. Shame that I don't remember which feature was it - maybe raising an exception in some configurable situations?

I though that adding support for multi workers can be done with just adding 3 lines to mark that this plugin supports it (see example).

The checklist for supporting multi-workers is quite short (from https://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released):

• Plugin MUST be implemented with v0.14 API natively
• Plugin MUST have #multi_workers_ready? method and return true
  • the three lines to add that I mentioned
• Plugin MUST NOT use raw sockets to listen network ports
  • here I am not 100% sure, but this should not be the problem for this plugin

[cosmo0920]

The checklist for supporting multi-workers is quite short (from https://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released):

• Plugin MUST be implemented with v0.14 API natively

Basically, Yes. Even though, through v0.12 compatibility layers, Fluentd can handle multi workers when #multi_workers_ready? returns true. But, we recommend to use v1 API natively.

• Plugin MUST have #multi_workers_ready? method and return true
  * the three lines to add that I mentioned

Yup. It is needed fix.

• Plugin MUST NOT use raw sockets to listen network ports
  * here I am not 100% sure, but this should not be the problem for this plugin

Yep. ServerEngine will distribute front socket connection into running workers under supervisor.
Then, plugin should use server plugin helper to use ServerEngine powered utility methods.

I'm maintaining Upstream plugin and also Fluentd maintainer.
Feel free to ask anyquestion about Fluentd. 😄