Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update e2e to use apache/kafka image #3974

Open
busma13 opened this issue Feb 21, 2025 · 3 comments
Open

Update e2e to use apache/kafka image #3974

busma13 opened this issue Feb 21, 2025 · 3 comments
Assignees
@busma13
Labels
helmchart helm chart related pkg/scripts tests

Comments

@busma13
Copy link
Contributor

busma13 commented Feb 21, 2025 •
edited
Loading

We plan to publish a kafka helmchart to github to replace the chart in teraslice/e2e/helm/kafka. This chart will use the apache/kafka image instead of confluentinc/cp-kafka. The apache/kafka image uses KRaft out of the box, so we will need to modify scripts to allow for this in the e2e setup. We currently have to map from the kafka version to the cp-kafka version, so that needs to be removed as well.
@busma13 busma13 self-assigned this Feb 21, 2025
@busma13
Copy link
Contributor Author

busma13 commented Feb 21, 2025

I'm thinking it may be better to add some flag to let you select KRaft or zookeeper instead of removing all of the zookeeper code. I guess the same would apply to the version mapping. It would be a shame to remove all that code and then find out that Kraft isn't working for us.

@godber
Copy link
Member

godber commented Feb 21, 2025

I'm thinking it may be better to add some flag to let you select KRaft or zookeeper instead of removing all of the zookeeper code. I guess the same would apply to the version mapping. It would be a shame to remove all that code and then find out that Kraft isn't working for us.

That seems sensible. So, these are the two Kafka "modes":

  • apache kafka image running craft, no version mapping
  • cp-kafka running zookeeper, with version mapping

Right?

@busma13
Copy link
Contributor Author

busma13 commented Feb 21, 2025 •
edited
Loading

List of steps we need to take:

  • create terascope/charts repo
  • prepare branch of internal chart
    • use apache/kafka image
    • remove zookeeper
    • change config to use KRaft
    • remove anything specific to internal usecase
    • update documentation
    • fix chart tests to work without zookeeper
  • copy internal kafka chart to terascope/charts repo
  • publish a packaged chart
    • ???
  • update ts-scripts to use apache/kafka with KRaft
    • flag for zookeeper or KRaft
    • set correct image version based on kafka repository used
  • update e2e/helm to use terascope/charts/kafka
  • add encryption capability to kafka chart
  • add encryption capability to e2e tests
  • add encryption capability to ts-scripts
  • setup kafka chart CI pipelines

@busma13 busma13 mentioned this issue Mar 7, 2025
Merged
godber pushed a commit that referenced this issue Mar 7, 2025
@busma13
E2e apache kafka (#3989)
8b7841f 
This PR makes the following changes:
- add logic to use `KRaft` mode and `KAFKA_VERSION` or `zookeeper` mode
and `KAFKA_IMAGE_VERSION` depending on `KAFKA_DOCKER_IMAGE`
- all helmfile related tests will run `apache/kafka` image, using a copy
of the `terascope/helm-charts/kafka` v 1.0.0 chart

ref: #3974
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@busma13 busma13

Labels
helmchart helm chart related pkg/scripts tests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@godber @busma13