Full baseline is never generated because since filter marks mutations as ignored due to not being unchanged compared to target

[rouke-broersma]

Describe the bug
The since filter marks mutations as ignored based on git information, this causes the full baseline for the baseline feature to never be generated making the baseline feature pretty much useless.

Expected behavior
If a baseline does not exist, a full mutation report should be generated even if there were no changes based on git information.

Additional context
This problem is caused by the splitting of the since and the baseline filter. These used to be one combined filter which meant that we could arrange things in such a way that the filtering was correct. Now that they are separated filters they run after each other. This makes sense, the baseline should first be determined and then the baseline information should be extended with information from git. However this should not happen in the case when a baseline does not exist and thus a fresh baseline should be generated. Our current filter setup does not allow for this scenario.

[rouke-broersma]

This is a problem because the baseline filter implicitly enables the since filter. We should remove this implicit enable. However we then lose the capability of determining that the baseline is incorrect due to undetected changes..

@richardwerkman @Stijn-Rutten what do you think

[Stijn-Rutten]

Removing the implicit enable would not fix anything. When the since feature is disabled we would place all the results from the baseline in the current test run. Then we would not edit any mutant results and thus none of the mutants covered by the baseline would change. Only new mutants would get a new result. All mutants covered by the old result could have wrong results. The most viable option I can think of right now is checking if a baseline is provided. If this is not the case we should disable both the baseline filter and the since filter. After we have done that we would run a complete test of all mutations. After that we can upload a fresh baseline.

[richardwerkman]

We could add a create-baseline command that stores the baseline in the desired storage provider after doing a complete run.

[rouke-broersma]

Wouldn't want to put the burden on the user when we can already detect that there is no baseline. This was supposed to be automatic (and used to be automatic).

[rouke-broersma]

The best solution IMO is to extract the actual git logic into a new class. Use this class both in the since filter and the baseline filter. Then we should not allow since and baseline to be enabled at the same time. This way both filters can share the existing logic but the baseline filter can choose not to enable the git filtering when there is no existing baseline.

[Stijn-Rutten]

@rouke-broersma would that mean either one is active at a given time? If baseline is enabled, since is disabled and vice versa?

[rouke-broersma]

Yes, at that point they cannot be enabled at the same time and there would also be no reason for the to be enabled at the same time. Since does since without the baseline stuff and baseline makes sure it takes git information into account to make sure the report is as correct as possible. Enabling both would then be pointless.

[richardwerkman]

Then it would make sense to turn them into commands like we wanted to. However that would introduce a breaking change

[rouke-broersma]

We can have them as both commands and options until the next major update and simply show a validation error if both are turned on for now.

[dupdob]

what is the status of this issue? Asking because it is old and marked as high priority ?
Shall the priority be revised?

[richardwerkman]

I guess the priority isn't that high as nobody outside of the stryker team noticed this (it seems). I stated on an implementation but got distracted and started on other work... Somebody else could pick this up if they want. Or I will continue after finishing the test detail feature

[richardwerkman]

Will be fixed by #1560