Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python] Just add all the packages that transitively depend on torch for now #326

Merged
merged 1 commit into from
Jan 28, 2025

Conversation

blast-hardcheese
Copy link
Collaborator

@blast-hardcheese blast-hardcheese commented Jan 28, 2025

Why

Until uv's transitive-package-index-pinning is resolved, let's just proactively ensure we pin torch for anything that will transitively depend on it.

This list was generated using:

cat internal/backends/python/pkgs.json | jq -c --slurp 'map(select(.requires_dist // [] | contains(["torch"]))) | map(.name)'

What changed

Exhaustively populate torchOverrides.go

Test plan

Rough performance testing:

diff --git a/internal/backends/python/python.go b/internal/backends/python/python.go
index 8af0f13b..012b7e41 100644
--- a/internal/backends/python/python.go
+++ b/internal/backends/python/python.go
@@ -12,6 +12,7 @@ import (
 	"regexp"
 	"slices"
 	"strings"
+	"time"
 
 	"github.com/BurntSushi/toml"
 	"github.com/replit/upm/internal/api"
@@ -727,6 +728,28 @@ func makePythonPipBackend() api.LanguageBackend {
 
 // makePythonUvBackend returns a backend for invoking uv.
 func makePythonUvBackend() api.LanguageBackend {
+
+	before := time.Now()
+	i := 0
+	c := 0
+	seen := make(map[string]bool)
+	for i < 100000 {
+		pkgname := "zuko"
+		fst := pkgname[0:i % 4]
+		snd := pkgname[i % 4:len(pkgname)]
+		if _, ok := seen[fst]; !ok {
+			fmt.Println("Seen: ", fst, "+", snd)
+			seen[fst] = true
+		}
+		if slices.Contains(torchOverrides, string(fst + snd)) {
+			c += 1
+		}
+		i += 1
+	}
+	after := time.Now()
+
+	fmt.Println("100,000 runs:", after.Sub(before))
+
 	listUvSpecfile := func() map[api.PkgName]api.PkgSpec {
 		cfg, err := readPyproject()
 		if err != nil {

on a live repl in production yields acceptable performance results:

Seen:   + zuko
Seen:  z + uko
Seen:  zu + ko
Seen:  zuk + o
100,000 runs: 94.217053ms

@blast-hardcheese blast-hardcheese requested a review from a team as a code owner January 28, 2025 22:42
@blast-hardcheese blast-hardcheese requested review from ryantm and removed request for a team January 28, 2025 22:42
Copy link
Collaborator

@ryantm ryantm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ❤️ big static lists.

@blast-hardcheese blast-hardcheese enabled auto-merge (squash) January 28, 2025 22:51
@blast-hardcheese blast-hardcheese merged commit 01b04d8 into main Jan 28, 2025
5 checks passed
@blast-hardcheese blast-hardcheese deleted the dstewart/pytorch-packages branch January 28, 2025 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants