handling uploads?

[bewinsnw]

Looking over the code, I don't see endpoints for PUT requests, so I assume that's handled out of band. Thinking about our own setup (which would be in AWS), I can imagine running this purely as a proxy, with a "dumb" PEP-503 index built over s3 uploads for internal packages - just a lambda writing html index files - and rely on simple-repository's fancy ability to generate PEP-658 metadata on top?

How are you handling this yourselves (or have I completely misunderstood)

[pelson]

Looking over the code, I don't see endpoints for PUT requests, so I assume that's handled out of band

You've understood perfectly - we have a separate service for handling uploads. It does very little from a "simple-repository" perspective, and it handles a bunch of different internal auth mechanisms. For this reason, we didn't include it in our release. There may be scope for this to change in the future, as we are gradually adding more "smarts" such as rejecting releases with invalid (or undesirable) metadata, as well as having some ownership mechanisms such as is seen in PyPI.

Thinking about our own setup (which would be in AWS), I can imagine running this purely as a proxy with a "dumb" PEP-503 index built over s3 uploads for internal packages

Yes, this totally works. In fact, I think this was the setup that we tested at EuroPython last year with the Pyodide repository. There, they have some controlled way of managing packages in their index, and build a custom JSON view of the index (they don't have a HTML simple equivalent, e.g. https://pyodide-pypi-api.s3.amazonaws.com/simple/numpy/ points to a JSON doc (with the right headers)). We were able to get the simple-repository-browser to work out of the box on that index (actually, I think we found a bug with their index at the time, and they fixed it pretty quickly), and I doubt there will be any issue at all running a simple-repository-server which proxies the index (and metadata extraction on-the-fly).

[eddybl]

As you use GitLab at CERN, do you use the GitLab PyPI endpoints as part of your internal structure to feed packages to the repository?

[pelson]

We have discussed this option, but in the end we simply handle the endpoint ourselves. At the moment, Python package upload is rather trivial (though without a formal spec) - this may change with PEP-694 though.

From a practical perspective, I can certainly see that using GitLab in this way would be interesting. Getting simple-repository to perform well with many small repositories may be a bit of a challenge - I'd guess you either need to have a pre-defined mapping between project name and the repository from whence a distribution can be retrieved, or do some pre-fetching of the package lists. In principle, I guess it is also possible to have a single GitLab package repository which is open to many users - I don't know in practice how well that setup works though.

[eddybl]

Well, GitLab also provides group endpoints providing all the packages of the individual sub-groups and projects, but depending on your repository structure, you might still end up having to take care of many group repositories. But in this case I guess I am more considering to add our GitLab Group PyPI endpoint as an additional HTTP source. That should work, correct?

(so at that point it would not be a question about "uploading" packages anymore)

[pelson]

If it is following PEP-503 correctly, it will work. We are definitely interested to hear about how it goes though - especially since we haven't released our own upload service implementation, and given the prevalence of GitLab in scientific orgs.

[eddybl]

so just using the simple-repository-browser on the GitLab endpoints leads to:

packages can be found via the search, but any details page ( /project/<PACKAGE_NAME> ) just states, that the package cannot be found. I assume this comes down to the GitLab PyPI simple endpoint not being compliant to PEP-658 and/or PEP-691.

So as the simple-repsitory-server will add metadata, I wanted to try this route next, but there I am stuck that the current version of the simple-repsitory-server just does not work with the current version of simple-repository ( simple-repository/simple-repository-server#1 )