Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GPU transform for FCMAE pre-training #196

Merged
merged 22 commits into from
Nov 8, 2024
Merged

GPU transform for FCMAE pre-training #196

merged 22 commits into from
Nov 8, 2024

Conversation

ziw-liu
Copy link
Collaborator

@ziw-liu ziw-liu commented Oct 23, 2024

Building on top of #196:
Doing non-trivial augmentations on large imaging volumes has become a bottleneck in training image translation models. By keeping the initial cropping and normalization on the CPU workers, while executing the more heavy transforms (especially the resampling ones) on the GPU, training can be significantly faster.

Current state of this PR

For 2D FCMAE, reaching the same validation loss is ~7x the speed compared to v0.2. Compare these logs:

/hpc/projects/comp.micro/virtual_staining/models/hek-a549-bj5a-20x/lightning_logs/tiny-2x2-fcmae-amp-hek-a549-bj5a-400ep
/hpc/projects/comp.micro/virtual_staining/models/fcmae-2d/test-logs/hek-a549-bj5a-gpu-aug-val-every-10

And for 3D FCMAE, the total voxel throughput is ~3x of v0.2, potentially limited by CPU-GPU transfer.

Caveat: since the transforms are now defined in the lightning module instead of the data module, they are the same for all the dataset. (fixed)

@ziw-liu ziw-liu changed the base branch from main to simple-cache October 23, 2024 17:02
@ziw-liu ziw-liu requested a review from edyoshikun October 23, 2024 17:02
@ziw-liu ziw-liu added enhancement New feature or request translation Image translation (VS) labels Oct 23, 2024
@ziw-liu ziw-liu marked this pull request as ready for review October 31, 2024 22:19
Copy link
Contributor

@edyoshikun edyoshikun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merging this so we can merge to base

@ziw-liu ziw-liu merged commit 949c445 into simple-cache Nov 8, 2024
4 checks passed
@ziw-liu ziw-liu deleted the gpu-transform branch November 8, 2024 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request translation Image translation (VS)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants