[improve] [broker] Make the estimated entry size more accurate

[poorbarcode]

Motivation

Background

• B1: Broker assumes the entry size is 1 when no publishing after the topic was loaded up at the latest time. Otherwise, it uses the average entries size published during the lastest time-range of the topic loaded up.
  • Issue: the average entry size is not prectice for the catch-up read after a topic is reloaded.
• B2: The feature managedLedgerMaxReadsInFlightSize assumes the entry size is 10kb when it read nothing. Otherwise, it uses the average entries size of the entries read out before.

Issue
Regarding scenario B2, it may cause an error Time-out elapsed while acquiring enough permits on the memory limiter to read from ledger 33, {tenant}/{ns}/persistent/{topic}-partition-0, estimated read size 1937 bytes for 1 entries (check managedLedgerMaxReadsInFlightSizeInMB) when there is hundreds topic being read and the exact entry size is very little.

More accurate solutions

• P0: Use the exact size of the entries that we attempt to read.
• P1: Use the exact size of the ledger that the entries were stored at to estimate the entries' size.
• P2: Use the size of the topic that the entries were stored at to estimate the entries' size.

Modifications

• Rather than use the P2 solution, change the solution to P1, and it will never return an inaccurate value such as 1 or 10k

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x

[lhotari]

• The feature managedLedgerMaxReadsInFlightSize assumes the entry size is 10kb when it read nothing. Otherwise, it uses the average entries size of the entries read out before.

After #23901 changes this case would be very rare since it would apply only for the first read in case no entries were produced to the topic.

[poorbarcode]

@lhotari

The feature managedLedgerMaxReadsInFlightSize assumes the entry size is 10kb when it read nothing. Otherwise, it uses the average entries size of the entries read out before.

After #23901 changes this case would be very rare since it would apply only for the first read in case no entries were produced to the topic.

Agree, but for the following case, the current solution is better

• There are 50k entries in the ledger
  • 1 entry was read out, the average size estimated is less accurate than the current implementation.
• The topic was sent for 30 days, and the customer changed the schema of the topic.
  • total size/ total count is less accurate than the current implementation
• Additionally, the current solution will never get an inaccurate value such as 1 or 10k, because it always calculates the exact average size for the entries and it will read a unique entry if no entries to read.

[poorbarcode]

/pulsarbot rerun-failure-checks

[codelipenghui]

The change looks good, left a minor comments about this test change.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 81.25000% with 6 lines in your changes missing coverage. Please review.

Project coverage is 74.20%. Comparing base (bbc6224) to head (a46123f).
Report is 935 commits behind head on master.

Files with missing lines	Patch %	Lines
...che/bookkeeper/mledger/impl/ManagedCursorImpl.java	85.71%	2 Missing and 2 partials ⚠️
...keeper/mledger/impl/cache/RangeEntryCacheImpl.java	50.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23931      +/-   ##
============================================
+ Coverage     73.57%   74.20%   +0.63%     
+ Complexity    32624    32357     -267     
============================================
  Files          1877     1856      -21     
  Lines        139502   144162    +4660     
  Branches      15299    16415    +1116     
============================================
+ Hits         102638   106982    +4344     
+ Misses        28908    28765     -143     
- Partials       7956     8415     +459     

Flag	Coverage Δ
inttests	26.73% <62.50%> (+2.15%)	⬆️
systests	23.18% <71.87%> (-1.14%)	⬇️
unittests	73.74% <81.25%> (+0.89%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...che/bookkeeper/mledger/impl/ManagedLedgerImpl.java	82.06% <ø> (+1.40%)	⬆️
...keeper/mledger/impl/cache/RangeEntryCacheImpl.java	63.59% <50.00%> (+4.84%)	⬆️
...che/bookkeeper/mledger/impl/ManagedCursorImpl.java	79.40% <85.71%> (+0.10%)	⬆️

... and 1044 files with indirect coverage changes

[lhotari]

@poorbarcode Great work, Yubiao! I can backport and cherry-pick this to branch-3.3 and branch-3.0.