Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve metrics tests #3335

Open
1 of 27 tasks
vfdev-5 opened this issue Feb 27, 2025 · 1 comment
Open
1 of 27 tasks

Improve metrics tests #3335

vfdev-5 opened this issue Feb 27, 2025 · 1 comment

Comments

@vfdev-5
Copy link
Collaborator

vfdev-5 commented Feb 27, 2025

The idea is to add available_device pytest fixture into the appropriate metric tests. For example:

@pytest.mark.parametrize("n_times", range(5))
- def test_multiclass_input(n_times, test_data):
+ def test_multiclass_input(n_times, available_device, test_data):
    y_pred, y, num_classes, batch_size = test_data
-    cm = ConfusionMatrix(num_classes=num_classes)
+    cm = ConfusionMatrix(num_classes=num_classes, device=available_device)
    ...

This will generate 2 more tests and run test_multiclass_input on cpu, cuda if available, mps if available.

No need to add available_device fixture to the tests:

  1. checking if an error is raised, for example test_confusion_matrix.py::test_num_classes_wrong_input:
def test_num_classes_wrong_input():
    with pytest.raises(ValueError, match="Argument num_classes needs to be > 1"):
        ConfusionMatrix(num_classes=1)
  1. running distributed tests, e.g. tests inside TestDistributed or test_distrib_*.

Files to update:

  • test_average_precision.py
  • test_classification_report.py
  • test_average_precision.py
  • test_cohen_kappa.py
  • test_confusion_matrix.py
  • test_cosine_similarity.py
  • test_entropy.py
  • test_fbeta.py
  • test_frequency.py
  • test_hsic.py
  • test_js_divergence.py
  • test_kl_divergence.py
  • test_loss.py
  • test_maximum_mean_discrepancy.py
  • test_mean_absolute_error.py
  • test_mean_pairwise_distance.py
  • test_mean_squared_error.py
  • test_multilabel_confusion_matrix.py
  • test_mutual_information.py
  • test_precision_recall_curve.py
  • test_roc_auc.py
  • test_roc_curve.py
  • test_root_mean_squared_error.py
  • test_top_k_categorical_accuracy.py

Additional lot of metrics:

  • clustering / test_calinski_harabasz_score.py ... test_silhouette_score.py
  • nlp / test_bleu.py ...
  • regression / ...

Please, split the work into multiple PRs: one PR for updates in a single file.
This issue can't be assigned to a single person and can be tackled in collaboration.

For the questions/details, please ask here or on our discord.

Copy link

Hey 👋, I've just created a thread for this issue on PyTorch-Ignite Discord where you can quickly talk to the community on the topic.

🤖 This comment was automatically posted by Discuss on Discord

puhuk added a commit to puhuk/ignite that referenced this issue Feb 27, 2025
vfdev-5 added a commit that referenced this issue Feb 28, 2025
* Update test_average_precision.py to resolve issue #3335

* Update test_average_precision.py

---------

Co-authored-by: vfdev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant