Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Memory usage regression in tasks using FlyteFiles in flytekit v.14.X #6188

Closed
2 tasks done
Sovietaced opened this issue Jan 23, 2025 · 0 comments
Closed
2 tasks done

[BUG] Memory usage regression in tasks using FlyteFiles in flytekit v.14.X #6188

Sovietaced opened this issue Jan 23, 2025 · 0 comments
Assignees
@wild-endeavor
Labels
bug Something isn't working flytekit FlyteKit Python related issue

Comments

@Sovietaced
Copy link
Contributor

Sovietaced commented Jan 23, 2025 •
edited
Loading

Describe the bug

After upgrading to Flyte v1.14.X we noticed that tasks that previously consumed ~1GB of memory started to consume much more memory (and lead to OOMs at 16GB). We've found this to happen on tasks with the same inputs/datasets.

Memory Usage on v1.13.X
Image

Memory Usage on v.14.X
Image

Expected behavior

Memory usage is similar to v1.13.X

Additional context to reproduce

Noticed with tasks using large amounts of FlyteFiles (~21k) sized between 2-3MB after the task returns.

Screenshots

No response

Are you sure this issue hasn't been raised already?

  • Yes

Have you read the Code of Conduct?

  • Yes
@Sovietaced Sovietaced added bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers labels Jan 23, 2025
@Sovietaced Sovietaced changed the title [BUG] Significant memory usage regression with FlyteFiles in flytekit v.14.X [BUG] Memory usage regression in tasks using FlyteFiles in flytekit v.14.X Jan 23, 2025
@wild-endeavor wild-endeavor mentioned this issue Jan 23, 2025
Merged
3 tasks
@wild-endeavor wild-endeavor added flytekit FlyteKit Python related issue and removed untriaged This issues has not yet been looked at by the Maintainers labels Jan 23, 2025
@wild-endeavor wild-endeavor self-assigned this Jan 23, 2025
@wild-endeavor wild-endeavor mentioned this issue Jan 24, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wild-endeavor wild-endeavor

Labels
bug Something isn't working flytekit FlyteKit Python related issue
Projects
Flyte Issues/PRs maintenance
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Sovietaced @wild-endeavor