KAFKA-19893: Reduce tiered storage redundancy with delayed upload (KIP-1241)

[jiafu1115]

JIRA:19893
KIP:1241

Currently, Kafka uploads all non-active local log segments to remote
storage even when they are still within the local retention period,
resulting in redundant storage of the same data in both tiers. This
wastes storage capacity (cost) without providing immediate
benefits,since reads during the retention window prioritize local data.

However, some users/topics rely on remote storage for real-time
analytics and need the latest data to be available as soon as possible
(In fact, it only tries to stay as up-to-date as possible, but it still
can’t include the latest data because the active segment hasn’t been
uploaded yet.). Therefore, this optimization is offered as a topic's
optional configuration rather than the default behavior.

Here are some additional thoughts/considerations.

1. Local files won’t be deleted until they’ve been uploaded to the
   remote storage, so this change is very safe—you don’t need to worry
   about files being cleaned up before they be upload to the remote.
2. Considering the latency of remote storage, the local retention period
   won’t be set too short. For example, in our production environment, we
   keep 1 day of local data alongside 3-7 days in remote storage, so
   there’s still 1 day of redundancy.

Example for the goal:

[jiafu1115]

Attach test result:
[Precondition]
Create one topic enable remote stroage in Kafka (3 brokers + 3 controller)

local storage time: 20 minutes
remote stroage time: 40 minutes
partition:  3
segement.bytes: 10M

[Steps]

1. Deploy this code patch into one broker only and restart the broker
2. Keep sending a lot of messages to the topic
3. Check the disk sizes on both local and remote storage at two points in time: 20 minutes before and 1 hour after.

[Result]

Before 20 minutes:

1. only 2 partition upload the local to remote.

After 1 hour:

1. The remote storage size for one partition (on the broker with the code change) is much smaller than the other two.

2. The sizes of the local disks are similar.