Bad robot

[rmyorston]

release-monitoring.org is monitoring one of my projects. It uses the custom backend to download the project web page once an hour.

• Since the project is updated, on average, once a year this is far too frequent.
• There's no need to download the whole web page just to know an update has happened.

If you're going to consume my bandwidth you should at least make some effort to be efficient. It's only polite.

[Zlopez]

I'm not sure if we can do anything with this. Anitya couldn't possibly know when a new version will be released and it's not good if we report new version few days after release.
How could we parse the webpage and search for regex, if we don't download it. Some backends are using APIs or something other for checking, but the custom backend must be usable for anything, so it just downloads the page and do a regex check.

[Zlopez]

If there is other way, how to check your project and this could be applicable for similar project, it is possible to create a special backend for it that will be checking for new version using other method than just HTTPS GET.

[rmyorston]

Anitya couldn't possibly know when a new version will be released

No, even I don't know that. But the previous frequency of updates provides a clue.

it's not good if we report new version few days after release

For critical infrastructure, perhaps, but for many projects a slight delay in notification is neither here nor there.

How could we parse the webpage and search for regex, if we don't download it

You can't. But you only need to download it if it's changed. If-modified-since is your friend.

[Zlopez]

You can't. But you only need to download it if it's changed. If-modified-since is your friend.

Didn't know about this field in the header, thanks for letting me know. I will see, how I can use this in Anitya, when checking for new version.

[Zlopez]

Implementation details

There are few things that must be implemented in order to make this work:

1. We need to add If-modified-since to check header in https://github.com/release-monitoring/anitya/blob/master/anitya/lib/backends/__init__.py#L43 and https://github.com/release-monitoring/anitya/blob/master/anitya/lib/backends/__init__.py#L238
2. The If-modified-since needs to be filled with time value from latest retrieved version before doing actual check. We will need a method on Project model that will return us either the time or latest version. See https://github.com/release-monitoring/anitya/blob/master/anitya/db/models.py#L194
3. We need to handle HTTP 304 response correctly - No new version found