Expose pre_release and version filters via API

[heitbaum]

With the improvements delivered in #944 - would it be possible to extend the api for version checks to expose / filter this information now?

The 2 apis of interest are:

• GET /api/v2/versions/
  • https://release-monitoring.org/api/v2/versions/?project_id=12523
• GET /api/project/(distro)/(path: package_name)/
  • https://release-monitoring.org/api/project/LibreELEC/lame/
  • https://release-monitoring.org/api/project/LibreELEC/MarkupSafe

In the above examples:

• we have lame returning 398 for lame which we want blacklisted. So the latest version returned would be 3.100
• For MarkupSafe we want to filter for !pre_release which would return 1.1.1

It would be ok/good to use either the POST or GET mechanism as in the below api to read this data?
POST /api/v2/versions/

• pre_release_filter (string) – Filter for unstable versions.
• version_filter (string) – Filter for blacklisted versions.

With the code being something like

curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"distribution":"LibreELEC",”project”:”lame”,"version_filter":"398"}'  \
  https://release-monitoring.org/api/v2/versions/
  
curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"distribution":"LibreELEC",”project”:”MarkupSafe”,"pre_release":"false"}' \
  https://release-monitoring.org/api/v2/versions/

output would be some thing like: (with 1.1.1 in the version response)

{"version":"2.0.0a1","version_url":null,"versions":["2.0.0a1","1.1.1","1.1.0","1.0","0.23","0.22","0.21","0.20","0.19","0.18","0.17","0.16","0.15","0.14","0.13","0.12","0.11","0.9.3","0.9.2","0.9.1","0.9"]}

Ideally would like to use (distro)/(package_name) for api use, as it minimizes double calls to the API.

[Zlopez]

I'm not sure if I follow, but the POST /api/v2/versions/ already has both pre_release_filter and version_filter.

curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"distribution":"LibreELEC",”project”:”lame”,"version_filter":"398"}'  \
  https://release-monitoring.org/api/v2/versions/

This should work, but you need to first create the project and then distribution mapping, then you can call it like this:

curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"id":<id of lame project>}'  \
  https://release-monitoring.org/api/v2/versions/

It would be preferable way, if you want to use it multiple times. If the projects are already there (as they are), you could just edit them and then call them only with ids.

[heitbaum]

I'm not sure if I follow, but the POST /api/v2/versions/ already has both pre_release_filter and version_filter.

it does (but only to create or edit a project) not to filter output on the request for version.

This should work, but you need to first create the project and then distribution mapping, then you can call it like this:

Calling https://release-monitoring.org/api/v2/versions/ with the id# works, but as in the version 1 api - calling with the (distro)/(package_name) pair is more extensible and doesn’t require the distribution to add the hard coded ids into a reference table - we have mapped all the packages to projects 😀 https://release-monitoring.org/api/project/LibreELEC/lame/

my test results below.

You can still see 398 as the latest version, this is because as I understand the black_list mechanism (Anitya only blacklists on creating new version records)

I was able to convince Anitya to make it a pre-release version, in the GUI. https://release-monitoring.org/project/12523/ but what we really need to be able to compare our current package list against release-monitoring (and black_list the pre-release and dud versions)

As in the Versions table we don’t have access to the “Pre-release flag (or ideally a blacklist flag)” via the API.

Version	Retrieved on (UTC)	Pre-release
398	Date information unavailable	True
3.100	2019-01-16 11:05	False

curl --header "Content-Type: application/json" \
     --header "Authorization: Token ${Token}" \
     --request POST \
     --data '{"distribution":"LibreELEC","project":"lame","version_filter":"398"}'  \
     https://release-monitoring.org/api/v2/versions/

{"message": "Unknown arguments: distribution, project"}

curl --header "Content-Type: application/json" \
     --header "Authorization: Token ${Token}" \
     --request POST \
     --data '{"id":"12523"}'  \
     https://release-monitoring.org/api/v2/versions/

{"latest_version": "398", "found_versions": [], "versions": ["398", "3.100", "3.99.5", "3.99.4", "3.99.3", "3.99.2", "3.99.1", "3.99", "3.98.4", "3.98.2", "3.98b8", "3.98b7", "3.98b6", "3.98", "3.97b2", "3.97b", "3.97", "3.96.1", "3.96", "3.95.1", "3.95", "3.94b", "3.93.1", "3.93.0", "3.93", "3.92", "3.91", "3.90.1", "3.90", "3.70", "0.1"]}

[Zlopez]

I'm not sure if I follow, but the POST /api/v2/versions/ already has both pre_release_filter and version_filter.

it does (but only to create or edit a project) not to filter output on the request for version.

It was never meant to filter the output, it should only filter the retrieval of new versions on project.

This should work, but you need to first create the project and then distribution mapping, then you can call it like this:

Calling https://release-monitoring.org/api/v2/versions/ with the id# works, but as in the version 1 api - calling with the (distro)/(package_name) pair is more extensible and doesn’t require the distribution to add the hard coded ids into a reference table - we have mapped all the packages to projects https://release-monitoring.org/api/project/LibreELEC/lame/

You can use the name and homepage to be sure that you have the correct project, this combination is unique.

You can still see 398 as the latest version, this is because as I understand the black_list mechanism (Anitya only blacklists on creating new version records)
Someone with the admin rights need to delete it from the project first. It will be never retrieved, if it's blacklisted, but it will not be deleted if already existing. I did this for you.

I was able to convince Anitya to make it a pre-release version, in the GUI. https://release-monitoring.org/project/12523/ but what we really need to be able to compare our current package list against release-monitoring (and black_list the pre-release and dud versions)

As in the Versions table we don’t have access to the “Pre-release flag (or ideally a blacklist flag)” via the API.

I can add the same as in anitya.version.update.v2 message topic, the list of stable_versions, so you can look at this if you want only the stable releases and everything else in versions is considered unstable.

[heitbaum]

It was never meant to filter the output, it should only filter the retrieval of new versions on project.

understood, I tried using as a filter (until I worked out it was the interface for updating the backend) that I had used already to nap LibreELEC to the projects. Which as a side note has been really good and useful.

I understand why you chose to only allow version_filter To be used for new scans - but given some of the bad data (admittedly not much) it would be good to have the capability to blacklist some of the versions. I have shared what I see in the LibreELEC data as “bad” at the bottom of the comment.

You can use the name and homepage to be sure that you have the correct project, this combination is unique.

It would be great if we can use any of the uniq key sets to query the data.

I can add the same as in anitya.version.update.v2 message topic, the list of stable_versions, so you can look at this if you want only the stable releases and everything else in versions is considered unstable.

that would be great.

Package	latest version	release-monitoring
argtable2	2.13	13
lame	3.100	398
libusb-compat	0.1.7	1.0.18
vdr-plugin-restfulapi	0.2.6.5	2.6.0
wayland	1.18.0	1.20

[Zlopez]

I understand why you chose to only allow version_filter To be used for new scans - but given some of the bad data (admittedly not much) it would be good to have the capability to blacklist some of the versions. I have shared what I see in the LibreELEC data as “bad” at the bottom of the comment.

This is because for deletion of the existing versions you should have admin permissions.

It would be great if we can use any of the uniq key sets to query the data.

You can the combination of name and homepage is already supported in the API.

I can add the same as in anitya.version.update.v2 message topic, the list of stable_versions, so you can look at this if you want only the stable releases and everything else in versions is considered unstable.

that would be great.

Could you create a separate ticket for it? It will be lost in this one.

Package latest version release-monitoring
argtable2 2.13 13
lame 3.100 398
libusb-compat 0.1.7 1.0.18
vdr-plugin-restfulapi 0.2.6.5 2.6.0
wayland 1.18.0 1.20

argtable was incorrectly setup, fixed now
lame, already fixed
libusb-compat, bogus version deleted
vdr-plugin-restfulapi, it was wrong tag, so I added the filter and deleted it
wayland, this project is watching wayland-protocols releases on https://wayland.freedesktop.org/releases.html, which is really version 1.20

[heitbaum]

Thanks for fixing the data. Looking really good now.

Wayland cleanup

https://release-monitoring.org/project/10061/ - this is wayland
https://release-monitoring.org/project/13997/ - this is wayland protocols

if we update the regex to “Wayland-([\d.]+).tar.xz” I think this will fix wayland (and rebase it)

with regard to wayland-protocols people should be using 13997

[Zlopez]

I fixed the wayland project, don't know why it was watching wayland protocols and not wayland

[Zlopez]

I created the issue about exposing the stable versions in API #1014

Closing this one