[Enhancement][Deviantart?] Change HTTP 429 backoff from linear to exponential

[Scripter17]

So I just tried downloading a single file from DeviantArt, which tends to give a lot of 429's

The first 31 lines were, and I quote:

[deviantart][info] Requesting public access token
[deviantart][warning] API responded with 429 Too Many Requests. Using 1s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 2s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 3s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 4s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 5s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 6s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 7s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 8s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 9s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 10s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 11s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 12s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 13s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 14s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 15s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 16s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 17s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 18s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 19s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 20s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 21s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 22s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 23s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 24s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 25s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 26s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 27s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 28s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 29s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 28s delay.

Totaling of 493 seconds, or 8.2 minutes. Which, frankly, was more amusing than anything.

Ignoring how the last delay went down for some reason, this would've gone by significantly faster if the delay doubled each time instead of going up by 1 second.
If the delay went 1s, 2s, 4s, 8s, etc. it would've taken about 63 seconds. Possibly less if the Deviantart API docs on 429's (which also recommend an exponential backoff) are to be believed about an adaptive rate-limiting scheme

[Scripter17]

A quick test with changing line 1120 in deviantart.py from self.delay += 1 to self.delay = 1 if self.delay==0 else self.delay*2, it took 1 minute 33 seconds to download an image

This was the console output:

[deviantart][info] Requesting public access token
[deviantart][warning] API responded with 429 Too Many Requests. Using 1s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 2s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 4s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 8s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 16s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 32s delay.
[deviantart][warning] API responded with 429 Too Many Requests. Using 30s delay.
* .\gallery-dl\deviantart\[REDACTED]\[REDACTED].png

Not too sure why there was a 30s at the end, and line 1108 needs to be changed to self.delay //= 2, but if I can make these changes to all the extractors I can do a pull request

Edit: It appears the self.delay approach only applies to the Deviantart extractor. Is there something else I should be looking for in the other files or is this it?

[mikf]

It was exponential for the longest time, but got changed to linear in f6fd449. I've had more than enough experience with their rate limiting system where I can tell you that exponential does not work a advertised, at least not in the long run when downloading more than 100 files. You eventually end up with a very high exponent and incredibly long waiting times. Limiting the exponent to <10 helped, but waiting ~8min for each download is still too much.

Downloading a single file with default API credentials only got up to 5s delay just now, which would've been 2**4 == 16s with exponential backoff.

You are best off registering your own application and using the client-id and client-secret from that.

[Scripter17]

If anyone would know, it's you
Gonna close this now