[improve][client] PIP-407 Add newMessage with schema and transactions

[omarkj]

Pulsar Client allows callers to create messages with a schema or a transaction, but not both.

This commit adds a new method in the producer that allows callers to create a message with both a schema and transaction.

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[dao-jun]

For PIP process, pls referring to https://github.com/apache/pulsar/tree/master/pip#pulsar-improvement-proposal-pip

[omarkj]

PIP PR filed here: #23950, will follow those guidelines by emailing out to the thread soon, etc.

[lhotari]

LGTM

[lhotari]

@BewareMyPower @dao-jun The PIP has been approved and merged, https://github.com/apache/pulsar/blob/master/pip/pip-407.md . PTAL and review

[lhotari]

I have a concern about backward compatibility with this interface change. Adding a new method without a default implementation will break source compatibility for any custom implementations of the Producer interface.
While it might be uncommon for users to implement this interface directly, it could happen in testing scenarios or custom proxy implementations. Java source compatibility rules require all implementors to provide all interface methods.

We could address the source compatibility issue by adding a default implementation:

Suggested change

	<V> TypedMessageBuilder<V> newMessage(Schema<V> schema,
	Transaction txn);
	default <V> TypedMessageBuilder<V> newMessage(Schema<V> schema, Transaction txn) {
	// to retain source compatibilty of Producer interface for implementors of the Producer interface
	throw new UnsupportedOperationException(
	String.format("Method not implemented in: %s", getClass().getName()));
	}

The benefit of making it source compatible is that we could also backport this change to the 4.0.x client without introducing breaking changes.

[BewareMyPower]

Do we really need to backport this change to 4.0.x client? LTS releases should not introduce such new APIs even if it does not break compatibility.

[BewareMyPower]

Once it's ported to 4.0.x, the forward compatibility will be broken. According to #15966, we should also guarantee the downgrade compatibility

[BewareMyPower]

Resolve this conversation. We can open another thread for the backport topic. For a new feature, adding a new non-default method should be accepted

[lhotari]

Do we really need to backport this change to 4.0.x client? LTS releases should not introduce such new APIs even if it does not break compatibility.

This is a feature which would benefit existing 4.0.x users. By making the solution compatible without introducing breaking changes to the interface, we could deliver this feature to 4.0.x users immediately. Obviously there's the question whether we need to do this or not.

[BewareMyPower]

I reviewed the PIP-175 and LTS releases recently again. PIP-175 mainly talked about the server side compatibility. Regarding the client side,

Note that this consideration is separate from the compatibility between clients and brokers, where we never break compatibility. The oldest available Pulsar client can still talk with the newest Pulsar broker, and vice versa, a new client, will be perfectly fine with an older broker (except the new features won't be working).

From the existing definitions, it's acceptable to cherry-pick this client-side feature for 4.0.x. It does not break any compatibility: with this new API, the latest 4.0.x client can still access an older broker (that supports transaction).

The only question is the source compatibility you mentioned by making it default. Could you show a concrete example for how could a non-default new method affect users?

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.

Project coverage is 74.21%. Comparing base (bbc6224) to head (c9b0771).
Report is 946 commits behind head on master.

Files with missing lines	Patch %	Lines
...va/org/apache/pulsar/client/impl/ProducerBase.java	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23942      +/-   ##
============================================
+ Coverage     73.57%   74.21%   +0.64%     
+ Complexity    32624    32392     -232     
============================================
  Files          1877     1861      -16     
  Lines        139502   144173    +4671     
  Branches      15299    16421    +1122     
============================================
+ Hits         102638   107001    +4363     
+ Misses        28908    28737     -171     
- Partials       7956     8435     +479     

Flag	Coverage Δ
inttests	26.74% <0.00%> (+2.15%)	⬆️
systests	23.11% <0.00%> (-1.22%)	⬇️
unittests	73.72% <66.66%> (+0.87%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...va/org/apache/pulsar/client/impl/ProducerBase.java	81.48% <66.66%> (+2.63%)	⬆️

... and 1052 files with indirect coverage changes