memory leak with 0.12.0 #220
Comments
|
Wow, this is unfortunate. Does your application use SSL connections? |
|
Yes it does. |
|
UDP? 0.12.0 has a couple of UDP-related changes. If you use UDP please try to run 0.12.0rc1 too. The 0.12 release has a brand new SSL implementation, so the leak might be related to that. Ideally we need a script to reproduce it though. |
|
I am not using UDP. It is only HTTPS and called about 51 requests per second. I'll see what I can do to reproduce it. I have an API, mongo storage and a bunch of other things that need to be stripped. I might just create a tight loop with the two versions and record the memory. |
|
Alright. I was able to reproduce this pretty easily. Here is my script. import asyncio
import os
import aiohttp
import uvloop
from async_timeout import timeout
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
def headers():
return dict(authorization=f"Bearer {os.getenv('API_TOKEN')}")
async def fetch(url):
async with aiohttp.ClientSession(headers=headers()) as session:
async with timeout(8):
async with session.get(url) as response:
data = await response.json()
return response.status, data
for i in range(1000):
future = fetch("https://api.clashofclans.com/v1/clans/%23UGJPVJR")
code, response = asyncio.get_event_loop().run_until_complete(future)
print(i, code)I ran this with
I did Screenshot attached.
I have not investigated any further as to what could be causing this. |
|
Thanks for the reproducing! I'm checking into this. |
|
@fantix were you able to figure it out? |
|
@1st1 I'm still trying to find a clue, but I can reproduce the issue. |
|
Great, thank you! I'll try to find some time to help in the next couple of days. |
There were two issues: 1) at `connection_lost()` the timeout handle was never cancelled because getting a cdef property on a cdef class with `getattr` always return `None`, and 2) for now cancelling a uvloop timeout handle only clears references to call arguments but not the callback object itself, therefore the closure is now a part of the arguments to avoid SSLProtocol object stays in memory until the cancelled timeout handle actually get deallocated.
There were two issues: 1) at `connection_lost()` the timeout handle was never cancelled because getting a cdef property on a cdef class with `getattr` always return `None`, and 2) for now cancelling a uvloop timeout handle only clears references to call arguments but not the callback object itself, therefore the closure is now a part of the arguments to avoid SSLProtocol object stays in memory until the cancelled timeout handle actually get deallocated.
There were two issues: 1) at `connection_lost()` the timeout handle was never cancelled because getting a cdef property on a cdef class with `getattr` always return `None`, and 2) for now cancelling a uvloop timeout handle only clears references to call arguments but not the callback object itself, therefore the closure is now a part of the arguments to avoid SSLProtocol object stays in memory until the cancelled timeout handle actually get deallocated.
There were two issues: 1) at `connection_lost()` the timeout handle was never cancelled because getting a cdef property on a cdef class with `getattr` always return `None`, and 2) for now cancelling a uvloop timeout handle only clears references to call arguments but not the callback object itself, therefore the closure is now a part of the arguments to avoid SSLProtocol object stays in memory until the cancelled timeout handle actually get deallocated.
There were two issues: 1) at `connection_lost()` the timeout handle was never cancelled because getting a cdef property on a cdef class with `getattr` always return `None`, and 2) for now cancelling a uvloop timeout handle only clears references to call arguments but not the callback object itself, therefore the closure is now a part of the arguments to avoid SSLProtocol object stays in memory until the cancelled timeout handle actually get deallocated.
There were two issues: 1) at `connection_lost()` the timeout handle was never cancelled because getting a cdef property on a cdef class with `getattr` always return `None`, and 2) for now cancelling a uvloop timeout handle only clears references to call arguments but not the callback object itself, therefore the closure is now a part of the arguments to avoid SSLProtocol object stays in memory until the cancelled timeout handle actually get deallocated.
There were two issues: 1) at `connection_lost()` the timeout handle was never cancelled because getting a cdef property on a cdef class with `getattr` always return `None`, and 2) for now cancelling a uvloop timeout handle only clears references to call arguments but not the callback object itself, therefore the closure is now a part of the arguments to avoid SSLProtocol object stays in memory until the cancelled timeout handle actually get deallocated.
There were two issues: 1) at `connection_lost()` the timeout handle was never cancelled because getting a cdef property on a cdef class with `getattr` always return `None`, and 2) for now cancelling a uvloop timeout handle only clears references to call arguments but not the callback object itself, therefore the closure is now a part of the arguments to avoid SSLProtocol object stays in memory until the cancelled timeout handle actually get deallocated.
There were two issues: 1) at `connection_lost()` the timeout handle was never cancelled because getting a cdef property on a cdef class with `getattr` always return `None`, and 2) for now cancelling a uvloop timeout handle only clears references to call arguments but not the callback object itself, therefore the closure is now a part of the arguments to avoid SSLProtocol object stays in memory until the cancelled timeout handle actually get deallocated.
|
Thanks for the merge! @amir20 please feel free to let me know if it has further issues. Thanks! |
|
Nice. Would you be able to get a patch released? |
|
asap; I'm working on another fix right now. |
|
Please try the new uvloop 0.12.1; hopefully it fixes your case. |
|
I am testing right now. Will let it run for an hour and post an update. |
|
I think it works. After 30 minutes, its still at 135MB which is a good sign. Great job @1st1 :) |
PYTHONASYNCIODEBUGin env?: Have not tried yetRecently I upgraded from
0.11.3to0.12.0. I use uvloop to constantly pull data from an API and insert into a database. After running for a couple of minutes, I found that my memory usage jumped from ~100MB to ~450mb. Has anybody else run into this issue? I have a lot going on so reproducing it with a smaller gist would be hard right now. But reverting back to0.11.3immediately fixed the issue. So I am guessing something is related to the upgrade.The text was updated successfully, but these errors were encountered: