Add rate limit handling

[Zlopez]

In current state GitHub API is raising RateLimit exception, which is only logged by cronjob.

It will be nice to reschedule the projects check when rate limit is reset. This could be handled by saving all the projects with GitHub backend after exception is raised and run check again only for them.

[abitrolly]

A better architecture is a proper queue/microservice with rate limiting, which flexibly adapts itself to rate limits, and exposes logs and exceptions to public.

But first, I would like to see stats.

[Zlopez]

Yes, this is something I have in my TODO. To better handle the amount of requests. Right now, we are doing check for all the projects in one go, which is not really nice.
I want to divide this load to the whole day. So it will check only a few projects every hour (maybe something more complex later).
The exposition of the logs and exceptions should be done using fedmsg like you proposed in #574
However, I'm not sure if there is any topic for this right now.

I don't understand the link you sent, this is pointing to tags sdispater/poetry project. How this has to do anything with stats?

[Zlopez]

I change the title, because this is not only about GitHub.

[yarons]

I think that the most basic part is mapping the rate limit and availability and then trying to fit into this frame, and most importantly: when the first rejection occurs (rate limit) the system should freeze the checks instead of keeping on probing for new versions marking everything as rate limited.

[Zlopez]

@yarons
This is not a bad idea, but we have plenty of backends and it's not good to freeze everything, because one backend reaches rate limit.
In PR #633 I have it implemented for every project separately. Each project is checked after specified time (currently 1 hour) and if rate limit is thrown it is rescheduled to time returned in rate limit exception.

[yarons]

I meant freezing the specific backend, not all of them 😆.

[Zlopez]

I will add this to my TODO list and look at it later. For now it will work on per project basis.

[yarons]

So I should be expecting lots of failures ahead :)

[Zlopez]

I didn't noticed any ratelimit error with the current number of projects. So I don't think we will hit this anytime soon.

[yarons]

Hmmmm sure?

[Zlopez]

Oh, last time I checked this, there wasn't any rate limit error.

[Zlopez]

It now looks like we are hitting those, but the current PR should still help with this.

[Zlopez]

There was one thing I didn't noticed before, how the actual ratelimit cap is handled by GitHub. So I need to change how ratelimit is detected.

[yarons]

So in case of rate limit detection per backend it should stop querying once the first failed.

The only problem is that there is no indication so if we're almost at the limit and I'm trying to add another service then when I'll save it and ask for querying I will hit the limit while working with the system so we shouldn't stop at the actual limit but maybe using only 90% of it and keeping some buffer for contributors work.

[Zlopez]

This is probably good idea. To not waste whole rate limit at once.

[Zlopez]

I created a new issue, that contains the discussed rate limit enhancements #665. Feel free to comment on it.

[Zlopez]

This issue is already implemented by #678. Closing.