-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Commit messages by topic, partition, offset #160
base: main
Are you sure you want to change the base?
Conversation
public func scheduleCommit(topic: String, | ||
partition: KafkaPartition, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wondering if we have to expose both a topic and a partition argument here. Topic we should be able to get from our config and the consumer should always stick to the same topic. Partition is a bit more interesting due to rebalancing events.
NIT: Could you also fix the line breaks in this whole commit. We normally line break after the (
and before )
. This is matching what Xcode does.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A client can be configured to consume from multiple topics and partitions (see assign/1
) so I do not think there is a way around passing the topic in explicitly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right now a client can only be configured to consume from a single topic since we are not publicly exposing assign
. Following this reasoning we shouldn't expose topic
here either but take it from the config,
Motivation: In some cases, for instances when messages are handed out to other processes, it is not reasonable to hold on to the whole Kafka message to be able to commit it at a later time. It consumes much less memory to hold only onto the details needed to commit the message. Modifications: Add new public methods that allow users to commit messages if they have a reference to the topic, partition, and offset instead of only the whole message. Result: Developers no longer need to keep the whole message around to commit it, only the topic, partition, and offset.
5065315
to
6ed10e1
Compare
public func scheduleCommit(topic: String, | ||
partition: KafkaPartition, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right now a client can only be configured to consume from a single topic since we are not publicly exposing assign
. Following this reasoning we shouldn't expose topic
here either but take it from the config,
@omarkj thank you for your PR! There are some notes. Agree with @FranzBusch that indeed currently assign ( However, I would probably organise it as some dedicated struct, i.e.
For example: struct TopicPartitionOffset: Hashable, Equitable? {
public let topic: String
public let partition: KafkaPartition
public let offset: KafkaOffset
/// no public constructor (at least so far)
}
struct KafkaConsumerMessage {
var topicPartitionOffset: TopicPartitionOffset { .init(topic: self.topic, partition: self.partition, offset: self.offset) }
}
final class KafkaConsumer {
...
func commit(_ topicPartitionOffset) throws
} The usage would be the following:
|
Motivation:
In some cases, for instances when messages are handed out to other processes, it is not reasonable to hold on to the whole Kafka message to be able to commit it at a later time. It consumes much less memory to hold only onto the details needed to commit the message.
Modifications:
Add new public methods that allow users to commit messages if they have a reference to the topic, partition, and offset instead of only the whole message.
Result:
Developers no longer need to keep the whole message around to commit it, only the topic, partition, and offset.