Implement mechanism of override and patch migration files

[weronikasosnowskaseqera]

Description

Closes #30

This PR adds functionality of override and patch files described in this issue https://github.com/seqeralabs/nf-tower-cloud/issues/3613.

The migration tool will scan migration files according to the given pattern like before. Besides collecting migrationEntries, It will also collect files with .patch or .override extensions in the file name and put Them in separate lists. In these collections, we will be looking for override or patch files during the applyMigration method.

Scenarios

• WHEN file.sql exists and was executed

  • AND file.override.sql exists -> nothing happens
  • AND file.patch.sql exists -> apply patch

• WHEN file.sql exists and was not executed

  • AND file.override.sql exists -> apply override
  • AND file.patch.sql exists -> apply file.sql and apply patch
  • AND file.patch.sql exists AND file.override.sql exists -> apply override

• WHEN file.sql doesn't exist but file.patch.sql exists OR file.override.sql exists -> nothing happens

[weronikasosnowskaseqera]

I'd like to have more opinions about this behavior mentioned above

WHEN file.sql exists and was not executed AND file.fixed.sql exists AND file.amended.sql exists -> apply amended and apply fix

I'm not sure if in this case we would prefer to execute only amended or both amended and fixed file.

[tcrespog]

I'm not sure if we should couple the amended/fixed logic to the migtool library. Originally, the proposal described at https://github.com/seqeralabs/nf-tower-cloud/issues/3613 suggested general-purpose changes that would allow to implement the logic on the side of the users of the library.

From the point of view of the tool/library, the idea was to:

1. Add the ability to check if a specific migration file has been applied.
2. Receive a set of migration files to run.

Therefore, with those capabilites in place, from the point of view of users of the tool, the amended/fixed logic could be applied by:

1. Getting all files to run and identifying those "broken" files with amended and fix companions.
2. Checking whether the broken migrations and companions were already applied with migtool.
3. Choose the .fix or .amended files depending on whether the broken file was already applied or not.
4. Send the specific subset of files to execute.

I'm not sure if in this case we would prefer to execute only amended or both amended and fixed file.

The idea in this case is to ignore the original (and .amended) and apply .fixed. In a nutshell:

• Original already applied: run .amended.
• Original not applied: run .fixed.

[weronikasosnowskaseqera]

After talking with @tcrespog decided to close this PR and prepare another one with less changes. We will handle fix and amend files on the platform side.

[pditommaso]

We will handle fix and amend files on the platform side.

Sorry to be late on this, but not sure that a good idea. This tools is used across different projects, tower, licman, groundswell thefore it has more sense to have this logic as a core feature

[weronikasosnowskaseqera]

@pditommaso okay, then we can reopen this PR. I tested It with the platform by running migration with all migration files and I added a few fixed, and amended files to test new functionality - all worked well.

When we have this PR merged I will open another PR with more e2e tests on the platform side and actual fixed and amended files described in this issue.

For visibility: I created similar PR for the platform, logic is the same. If we decided that we will place these changes here, in this repo - I will close that PR.

[tcrespog]

Worth adding adding a description about the .fixed/.amended files support in the README file.

[tcrespog]

I've just noticed a few things that I overlooked initially in the tests, but the rest is fine.

[weronikasosnowskaseqera]

@pditommaso Can you review this one, please?

[pditommaso]

Thanks. I'll check asap

[swampie]

@pditommaso can you please merge and start a release?

[weronikasosnowskaseqera]

I want to run It again with tower after the file pattern changes just in case. I will put a comment here when ready.

[pditommaso]

can you please merge and start a release?

we need fix before the current problem, I don't want to overlap things

[weronikasosnowskaseqera]

All right, I managed to run It locally with ./gradlew publishToMavenLocal on the branch with patch and override files https://github.com/seqeralabs/nf-tower-cloud/pull/6160. We need to release this PR to continue with the mentioned PR.

[pditommaso]

Ok