Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

better chunking of the json output #6

Closed
diwu1989 opened this issue Jun 12, 2015 · 1 comment
Closed

better chunking of the json output #6

diwu1989 opened this issue Jun 12, 2015 · 1 comment

Comments

@diwu1989
Copy link
Contributor

I see that you have a json indent + line split hack to break the response into smaller sized chunks so that any single socket write operation doesn't overflow.

I think a better solution is to pick a reasonably safe chunk size, say 32 bytes at a time, and then just wrap the json output in a generator:

result = json.dumps(...)
def chunks(data, chunk_size=32):
    for i in xrange(0, len(data), chunk_size):
        yield data[i:i+chunk_size]
return chunks(result)
@jikamens
Copy link
Collaborator

There is something aesthetically pleasing, at least to me, about yielding the data line-by-line.

Your suggested change would consume less CPU and RAM (well, it would if you used a bigger chunk size than 32 bytes, which is much smaller than necessary ;-)), but in the vast majority of cases the difference would be so negligible as to be effectively meaningless.

Your suggested change would increase the line count and complexity of the app without increasing functionality.

I am hopeful the bug in wsgiref will eventually be fixed, at which point the workaround in Coal Mine can be removed. Given that, I'm not sure I see the value in making the workaround more complex as you propose.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants