-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add rate limit handling #600
Comments
A better architecture is a proper queue/microservice with rate limiting, which flexibly adapts itself to rate limits, and exposes logs and exceptions to public. But first, I would like to see stats. |
Yes, this is something I have in my TODO. To better handle the amount of requests. Right now, we are doing check for all the projects in one go, which is not really nice. I don't understand the link you sent, this is pointing to tags sdispater/poetry project. How this has to do anything with stats? |
I change the title, because this is not only about GitHub. |
I think that the most basic part is mapping the rate limit and availability and then trying to fit into this frame, and most importantly: when the first rejection occurs (rate limit) the system should freeze the checks instead of keeping on probing for new versions marking everything as rate limited. |
@yarons |
I meant freezing the specific backend, not all of them 😆. |
I will add this to my TODO list and look at it later. For now it will work on per project basis. |
So I should be expecting lots of failures ahead :) |
I didn't noticed any ratelimit error with the current number of projects. So I don't think we will hit this anytime soon. |
Oh, last time I checked this, there wasn't any rate limit error. |
It now looks like we are hitting those, but the current PR should still help with this. |
There was one thing I didn't noticed before, how the actual ratelimit cap is handled by GitHub. So I need to change how ratelimit is detected. |
So in case of rate limit detection per backend it should stop querying once the first failed. The only problem is that there is no indication so if we're almost at the limit and I'm trying to add another service then when I'll save it and ask for querying I will hit the limit while working with the system so we shouldn't stop at the actual limit but maybe using only 90% of it and keeping some buffer for contributors work. |
This is probably good idea. To not waste whole rate limit at once. |
I created a new issue, that contains the discussed rate limit enhancements #665. Feel free to comment on it. |
This issue is already implemented by #678. Closing. |
In current state GitHub API is raising
RateLimit
exception, which is only logged by cronjob.It will be nice to reschedule the projects check when rate limit is reset. This could be handled by saving all the projects with GitHub backend after exception is raised and run check again only for them.
The text was updated successfully, but these errors were encountered: