WIP: Initial message schema for Anitya #570

jeremycline · 2018-08-01T20:50:39Z

This defines message schema using json-schema for a subset of messages
sent by Anitya.

Signed-off-by: Jeremy Cline [email protected]

Zlopez · 2018-10-11T11:34:38Z

As Fedora-messaging is released with version 1.0.0, this could be a good addition to the project.

@jeremycline
Do you need any help with this PR?

jeremycline · 2018-10-11T15:30:55Z

I'm going to take a bit of time today or tomorrow to clean it up, it'll be a good first app to move over

codecov-io · 2018-10-15T18:33:12Z

Codecov Report

Merging #570 into master will decrease coverage by 6.34%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #570      +/-   ##
==========================================
- Coverage   96.49%   90.14%   -6.35%     
==========================================
  Files          56       56              
  Lines        2909     2690     -219     
  Branches      392      352      -40     
==========================================
- Hits         2807     2425     -382     
- Misses         64      201     +137     
- Partials       38       64      +26

Impacted Files	Coverage Δ
anitya/ui.py	`77.62% <0%> (-20.86%)`	⬇️
anitya/lib/backends/drupal6.py	`75% <0%> (-18.55%)`	⬇️
anitya/lib/backends/drupal7.py	`75% <0%> (-18.55%)`	⬇️
anitya/lib/utilities.py	`80.1% <0%> (-14.99%)`	⬇️
anitya/lib/backends/pecl.py	`87.23% <0%> (-12.77%)`	⬇️
anitya/db/models.py	`87.8% <0%> (-12.2%)`	⬇️
anitya/lib/backends/bitbucket.py	`80.95% <0%> (-11.36%)`	⬇️
anitya/admin.py	`84.25% <0%> (-7.44%)`	⬇️
anitya/lib/backends/pear.py	`95.74% <0%> (-4.26%)`	⬇️
anitya/lib/backends/packagist.py	`90% <0%> (-2.6%)`	⬇️
... and 28 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fcbc0cc...02c7208. Read the comment docs.

Zlopez · 2018-10-16T07:26:09Z

I looked at the failing Travis job and it failed, because it took to long. The last test suite running was test_project.py. There must be something that caused infinite loop or something similar.

The docs test is failing because it doesn't know fedora-messaging.

ModuleNotFoundError: No module named 'fedora_messaging'

jeremycline · 2018-10-16T12:55:13Z

Yup, I'm not done with this yet, sorry for the noise

This defines message schema using json-schema for the messages Anitya sends. It does not, however, implement any Python API work with these messages. Signed-off-by: Jeremy Cline <[email protected]>

This is the bare minimum to publish messages using fedora_messaging. It includes support to use the fedmsg publisher as a config option in case things go wrong during deployment. This can be removed later. Signed-off-by: Jeremy Cline <[email protected]>

jeremycline · 2018-12-12T21:33:35Z

This depends on #635 which I think depended on #633. Tests are failing, but a lot of them have to do with the logging code that's about to be removed, so I don't want to spend time fixing them just to remove them.

Zlopez · 2018-12-13T08:54:29Z

Ok, I will than fix #635 first than.

Zlopez

It looks good I just have a few comments

Zlopez · 2018-12-14T09:53:28Z

anitya/lib/utilities.py

@@ -38,19 +40,29 @@

 def fedmsg_publish(*args, **kwargs):  # pragma: no cover


I think it will be good to rename this function to publish

This is a interim solution. The most reasonable refactor is likely to get rid of this function entirely. The reason is that different messages almost certainly have different failure scenarios and once you get rid of the fedmsg portion of this function, all it's doing is squashing exceptions.

Zlopez · 2018-12-14T09:56:52Z

anitya/tests/base.py

@@ -177,68 +180,66 @@ def create_distro(session):

 def create_project(session):
    """ Create some basic projects to work with. """
-    utilities.create_project(


I'm glad to see you are getting rid of the utilities functions. I want to remove most of them in the future.

Zlopez · 2018-12-14T10:00:15Z

anitya_schema/anitya_schema/project_messages.py

+        "type": "object",
+        "required": ["project", "message", "distro"],
+        "properties": {
+            "distro": {"type": "null"},


Why are we sending this in project related messages?

So these schema aren't what we should be sending, it's just what the current messages look like. Most of the message schema are crazy and make no sense. This is more about documenting the madness so we can fix it than it is about laying out a reasonable set of messages. Once we've done that you can come up with reasonable messages and a migration plan for consumers.

Zlopez · 2018-12-14T10:02:00Z

anitya_schema/anitya_schema/project_messages.py

+                "type": "object",
+                "properties": {
+                    "agent": {"type": "string"},
+                    "odd_change": {"type": "boolean"},


This is no longer happening. I removed this error when I introduced check for multiple versions.

That's not good :(

The message schema changed and it's possible consumers were relying on that field. If, for example, the-new-hotness expects this key to be present it will crash. The best path forward is likely to make sure that key remains the message getting sent out and is always false or something.

Oh, I didn't knew this could have impact like this. The change is only on master right now. It wasn't released yet, so I can add the field back.
It's hard to know what is required.

The odd_change is now always false.

Zlopez · 2018-12-14T10:02:57Z

anitya_schema/anitya_schema/tests/__init__.py

@@ -0,0 +1,20 @@
+# Copyright (C) 2018  Red Hat, Inc.


Fixtures are really nice solution for testing.

Zlopez · 2018-12-14T10:05:17Z

anitya_schema/anitya_schema/tests/test_schema.py

+    The fixtures are generated by downloading messages from datagrepper.
+    """
+
+    def test_valid(self):


I don't see any assert in this method.

Yep, that's expected, although I should probably add a comment to that effect. validate() raises an exception if the message is invalid which would cause the test to fail. The assertion is a basically there there isn't an exception.

Zlopez · 2018-12-14T10:07:01Z

anitya_schema/setup.py

+
+
+setup(
+    name="anitya_schema",


I expect that this should be the package I have to import in the-new-hotness.

Correct, it'll be published to PyPI/packaged separately from anitya so consumers can just pull in the schema

Zlopez · 2019-01-08T09:17:24Z

@jeremycline All the blocking PRs should be merged now, so you can continue on this PR when you have time.

Zlopez · 2019-01-30T15:53:10Z

I'm merging this and continue working where @jeremycline ended.

jeremycline force-pushed the message-schema branch from df9292c to 832e387 Compare August 1, 2018 20:52

jeremycline changed the title ~~Initial message schema for Anitya~~ WIP: Initial message schema for Anitya Aug 2, 2018

jeremycline force-pushed the message-schema branch 2 times, most recently from b0f4677 to d913085 Compare October 15, 2018 18:33

jeremycline force-pushed the message-schema branch from d913085 to 9722666 Compare October 15, 2018 21:29

jeremycline force-pushed the message-schema branch from 9722666 to 147f9d7 Compare October 16, 2018 21:19

jeremycline added 2 commits December 12, 2018 13:26

Initial message schema for Anitya

a8cc2f4

This defines message schema using json-schema for the messages Anitya sends. It does not, however, implement any Python API work with these messages. Signed-off-by: Jeremy Cline <[email protected]>

jeremycline force-pushed the message-schema branch from 147f9d7 to a893d6e Compare December 12, 2018 20:42

Zlopez reviewed Dec 14, 2018

View reviewed changes

Zlopez mentioned this pull request Dec 14, 2018

Return odd_change parameter #682

Merged

Merge branch 'master' into message-schema

02c7208

Zlopez merged commit b6c1568 into fedora-infra:master Jan 30, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Initial message schema for Anitya #570

WIP: Initial message schema for Anitya #570

jeremycline commented Aug 1, 2018

Zlopez commented Oct 11, 2018

jeremycline commented Oct 11, 2018

codecov-io commented Oct 15, 2018 •

edited

Loading

Zlopez commented Oct 16, 2018

jeremycline commented Oct 16, 2018

jeremycline commented Dec 12, 2018

Zlopez commented Dec 13, 2018

Zlopez left a comment

Zlopez Dec 14, 2018

jeremycline Dec 14, 2018

Zlopez Dec 14, 2018

Zlopez Dec 14, 2018

jeremycline Dec 14, 2018

Zlopez Dec 14, 2018

jeremycline Dec 14, 2018

Zlopez Dec 14, 2018

Zlopez Jan 23, 2019

Zlopez Dec 14, 2018

Zlopez Dec 14, 2018

jeremycline Dec 14, 2018

Zlopez Dec 14, 2018

jeremycline Dec 14, 2018

Zlopez commented Jan 8, 2019

Zlopez commented Jan 30, 2019

		@@ -38,19 +40,29 @@

		def fedmsg_publish(args, *kwargs): # pragma: no cover



		setup(
		name="anitya_schema",

WIP: Initial message schema for Anitya #570

WIP: Initial message schema for Anitya #570

Conversation

jeremycline commented Aug 1, 2018

Zlopez commented Oct 11, 2018

jeremycline commented Oct 11, 2018

codecov-io commented Oct 15, 2018 • edited Loading

Codecov Report

Zlopez commented Oct 16, 2018

jeremycline commented Oct 16, 2018

jeremycline commented Dec 12, 2018

Zlopez commented Dec 13, 2018

Zlopez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zlopez commented Jan 8, 2019

Zlopez commented Jan 30, 2019

codecov-io commented Oct 15, 2018 •

edited

Loading