Sometimes more changes are detected for one metric and one dateset

[jhutar]

Describe the bug

I do not know what triggered it, but this happened to me:

Another occurence:

I'm sure I have deleted all changes (across all metrics) at 2025-02-14 so this is new.

To Reproduce

Not sure, seen it here: .../test/254?fingerprint=%7B%22.metadata.env.JOB_NAME%22%3A%22periodic-ci-konflux-ci-e2e-tests-main-load-test-ci-daily-10u%22%7D

Version

Current on Prod

[lampajr]

Hi @jhutar, thanks for raising this issue!

I checked in the database and it looks like the change is duplicated multiple times:

   id    | dataset_id | variable_id 
---------+------------+-------------
 xxxxxx2 |     2yyyy4 |     5zzzzz4
 xxxxxx4 |     2yyyy4 |     5zzzzz4
 xxxxxx5 |     2yyyy4 |     5zzzzz4
 xxxxxx3 |     2yyyy4 |     5zzzzz4

My suspect is that for some reason they are not cleaned up when computing the change detection or I don't know, I am still trying to reproduce the same issue locally.

Out of curiosity, have you already tried recalculating the changes from Change Detection tab and see whether the duplications are wiped off?

[jhutar]

Hello. IIRC I have not ran it, but please feel free to try if it is handy or I can trigger it. Basically if it does not affect actual uploaded test results, feel free to do what is needed. I currently do not care too much about detected changes.

[lampajr]

Hello. IIRC I have not ran it, but please feel free to try if it is handy or I can trigger it. Basically if it does not affect actual uploaded test results, feel free to do what is needed. I currently do not care too much about detected changes.

Hi Jan, I re-triggered the recalculation for the last 6 months on your test and not it looks like there are no more duplicates:

Can you confirm that?

So I think that forcing the changes recalculation is a valid workaround for this issue but I'd like to keep it open to investigate what is causing the duplication to happen.

[jhutar]

Yep, looks perfectly fine now, thank you. FYI I deleted all the detected changes. We will see if some will appear once new run is uploaded in the morning. Thank you for looking into it!

[jhutar]

Hello. Few changes were detected and no duplicates, so that is fine.

But, please, do you know why e.g. in .measurements.scheduler_pending_pods_count.mean algorithm detected one change for 4 days old result (the other change it detected for today's result is understandable)?

https://horreum.corp.redhat.com/test/254?fingerprint=%7B%22.metadata.env.JOB_NAME%22%3A%22periodic-ci-konflux-ci-e2e-tests-main-load-test-ci-daily-10u%22%7D

[lampajr]

Hello. Few changes were detected and no duplicates, so that is fine.

That's good!

But, please, do you know why e.g. in .measurements.scheduler_pending_pods_count.mean algorithm detected one change for 4 days old result?

Not 100% sure but could be because yesterday I forced to to recompute the "change dection" for the past 6 months in order to cleanup the duplicates and re-generate those changes. That's my guess.

[jhutar]

Hmm, so 2025-02-20 I deleted all changes and today 2025-02-24 I see more changes back in for results from 2025-02-16 - 2025-02-21. I have deleted them except for changes detected in .results.measurements.KPI.errors so they can be investigated. See screenshots.

Is it possible to check what made the algorithm to trigger a change in older result?

It is possible that mine results uploading script is misbehaving, but it should be just uploading latest result, so I think that is fine.

[jhutar]

Hello. 2025-02-24 I deleted all changes except these in .results.measurements.KPI.errors graph. Today, besides 4 changes in latest result, 5 changes in older result is there again. Would you have an idea on why?

As usual, deleted all of them except for .results.measurements.KPI.errors, but recorded them in my screenshot docs.

[jhutar]

Anyway, maybe we can close this issue as duplicate changes did not appeared since then and try to solve this "changes in older results appearing with no obvious reason" issue separately?

[lampajr]

Anyway, maybe we can close this issue as duplicate changes did not appeared since then and try to solve this "changes in older results appearing with no obvious reason" issue separately?

Hi @jhutar, sorry I missed your previous comment. I will try to look at this as soon as I can.

Anyway, I agree I think we can close this one and create a new one for the older changes

[jhutar]

Original issue reported in this ticket did not appeard again, so closing it as there is nothing to investigate now. Thank you!

[jhutar]

For the other issue, created #2291

[jhutar]

Heh, again, double detected change appeared. See .measurements.scheduler_pending_pods_count.mean and 4 more graphs in Konflux loadtest changes. It waited for me to close this issue :-)

[lampajr]

Heh, again, double detected change appeared. See .measurements.scheduler_pending_pods_count.mean and 4 more graphs in Konflux loadtest changes. It waited for me to close this issue :-)

Ouch, it means there is something that is triggering the duplication to happen.. or some records are not properly cleaned up before getting recreated. We will need to investigate further 👀