Implement eland.DataFrame.to_json #661
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bartbroere
force-pushed
the
implement-to-json
branch
from
3d07310 to
e86d6f3
Compare
|
buildkite test this please |
pquentin
reviewed
Feb 8, 2024
pquentin
reviewed
Feb 8, 2024
Nice! I removed the two linting errors. Sorry about that. |
|
buildkite test this please |
|
buildkite test this please |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Dumping an Elastic index was the right solution for me at first. After a while csv did not offer all the guarantees I was looking for, since csv records for example can span multiple lines if any of the values contain line breaks.
For that reason, JSON lines was a more suitable format. Pandas'
DataFrame.to_jsoncan generate ".jsonl" files by supplinglines=True, orient='records'. This also lets us reuse the earlier solution that streams output from Elastic and appends it to a file, eliminating the need to be able to fit the entire dataset in memory.This pull request implements streaming an Elasticsearch index to
.jsonl, while falling back to runningto_pandas().to_json(...)if streaming is a bit harder to do.