Feature Request: Automated testing when merging to main

[davesteps]

If we assume our currently deployed model is on the main branch. When we train a better model on a new branch and we decide to merge into main, we want to run some CI/CD tests that will confirm our branch is ok to merge. In particular we want to test that 1) all of the dvc tracked files have been pushed to the remote and 2) if you run dvc status it return something like ‘All data and pipeline up to date.’. We have set up some python tests to confirm this but it doesn’t seem the most robust. Could there be some CML function to do this in a more formal way?

FYI I asked this on he forum and was asked to open it as an issue.

Thanks.

[DavidGOrtega]

@davesteps thanks for the feature

[davesteps]

I think this would actually be covered by the dvc feature discussed here: iterative/dvc#5369

[dacbd]

Closing in favo[u]r of the DVC issue (iterative/dvc#5369), @davesteps if you want CML to revisit this feel free to reopen here :)

[casperdcl]

I assume the end goal would be something like:

cml send-github-check --status queued
pip install ...
cml send-github-check --status in_progress
dvc verify && cml send-github-check --conclusion success || cml send-github-check --conclusion failure