Buffer support

[adie]

Hello, any plans to add buffering support? Without it the messages cannot be retried if the destination becomes temporary unavailable, which means that we will lose logs.

[zarqman]

No active plans for it.

In high volume environments, it seems of limited use as any buffer would be almost immediately exhausted if the connection didn't recover quickly. If it does recover quickly, then the only message(s) lost now are for the current connection attempt. Fluent's event loop runs every 0.1sec, so after a failure, there's 100ms for the remote to recover before further messages have the potential to be discarded.

For low log volumes, an argument could be made, but I'm hesitant to add too much complexity if that's the only use-case.

I have in my notes to someday explore Fluent's buffered writes to see if Fluent could be made to take on the buffering, but I haven't confirmed whether it preserves or discards the current batch of messages upon an error. Even if it preserves and retries, I haven't evaluated that path for any other potential problems or added complexity.

[adie]

The default retry timeout is 72h https://docs.fluentd.org/configuration/buffer-section#retries-parameters
So if we have enough space (we use file buffer) or memory we could support up to 3 days of syslog server unavailability.
Right now we use custom-built syslog plugin based on fluent-plugin-remote_syslog, but were thinking to use yours as it looks to have everything we need except buffering. And having buffering greatly helps when syslog server starts to have some problems and gets back online after some time, the chunks get retried and delivered eventually.

[aashishadhh]

Without Buffer, there is loss of logs if fluentd restarts.
File buffer could help solve that as well.