Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding kafka output support #33

Merged
merged 8 commits into from
Mar 23, 2022
Merged

Adding kafka output support #33

merged 8 commits into from
Mar 23, 2022

Conversation

P1llus
Copy link
Member

@P1llus P1llus commented Mar 22, 2022

Tested this with a local Kafka instance:

Start Kafka:
docker run -p 9092:9092 -d bashj79/kafka-kraft

Send data:

> ./stream log --addr 127.0.0.1:9092 -p kafka --kafka-topic test ./foo.txt
{"level":"debug","ts":"2022-03-22T15:23:29.439+0100","caller":"output/util.go:28","msg":"Connecting...","address":"127.0.0.1:9092"}
{"level":"info","ts":"2022-03-22T15:23:29.483+0100","caller":"output/util.go:41","msg":"Connected","address":"127.0.0.1:9092"}
{"level":"debug","ts":"2022-03-22T15:23:29.483+0100","caller":"command/log.go:81","msg":"Sending log line.","address":"127.0.0.1:9092","log":"./foo.txt","line_number":1}
{"level":"info","ts":"2022-03-22T15:23:29.525+0100","caller":"command/log.go:94","msg":"Log data sent.","address":"127.0.0.1:9092","log":"./foo.txt","total_bytes":19,"total_lines":1}

Double check that data is actually stored in topic:

bash-5.1# /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --max-messages 10
testmessage hey hey

While I could add in more options, currently it supports only Topic. We don't need things like groups etc, as we are not consuming but simply creating the topic and its data.

If you would like me to add more debugging output or similar, just let me know, but I feel that for stream this should be sufficient.

@elasticmachine
Copy link

elasticmachine commented Mar 22, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-03-23T16:45:39.000+0000

  • Duration: 6 min 43 sec

Test stats 🧪

Test Results
Failed 0
Passed 6
Skipped 0
Total 6

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@P1llus
Copy link
Member Author

P1llus commented Mar 22, 2022

Adding some tests as well now

@P1llus P1llus requested review from marc-gr and andrewkroh March 23, 2022 10:18
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please wait for the GH actions to run. Currently there are no checks on the last commit and I assume it's because of issues with GH infrastructure.

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a changelog entry.

@P1llus P1llus merged commit dcf09ff into elastic:main Mar 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants