From dfe35622512019b36a9452b3d917debd78db7add Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Mon, 19 Feb 2024 13:00:03 -0800 Subject: [PATCH 1/5] src/doc/en/developer/packaging.rst: Explain that package-version.txt needs to be an acceptable version according to install-requires.txt --- src/doc/en/developer/packaging.rst | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/doc/en/developer/packaging.rst b/src/doc/en/developer/packaging.rst index ea391941c9c..8720ab755ba 100644 --- a/src/doc/en/developer/packaging.rst +++ b/src/doc/en/developer/packaging.rst @@ -1114,8 +1114,10 @@ obtains most of the necessary information by querying PyPI. The ``dependencies`` file may need editing (watch out for warnings regarding ``--no-deps`` that Sage issues during installation of the package!). + Also you may want to set lower and upper bounds for acceptable package versions -in the file ``install-requires.txt``. +in the file ``install-requires.txt``. (Make sure that the version in +``package-version.txt`` falls within this acceptable version range!) By default, when the package is available as a platform-independent wheel, the ``sage --package`` creates a wheel package. To create a normal package @@ -1152,6 +1154,12 @@ For Python packages available from PyPI, there is another shortcut:: Downloading tarball to ...matplotlib-3.3.1.tar.bz2 [...............................................................] +When preparing the update, check that any lower and upper bounds for +acceptable package versions that may be declared in the file +``install-requires.txt`` are still correct, and update them as needed. +The version in ``package-version.txt`` always needs to fall within the +version range! + If you pass the switch ``--commit``, the script will run ``git commit`` for you. From d7413b439963ed2f4bc5be0e148b0435279a6a43 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Tue, 20 Feb 2024 00:59:11 -0800 Subject: [PATCH 2/5] src/doc/en/developer/packaging.rst: Explain what is done with install-requires.txt --- src/doc/en/developer/packaging.rst | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/src/doc/en/developer/packaging.rst b/src/doc/en/developer/packaging.rst index 8720ab755ba..fb70576d061 100644 --- a/src/doc/en/developer/packaging.rst +++ b/src/doc/en/developer/packaging.rst @@ -603,9 +603,14 @@ For example, the ``scipy`` ``spkg-check.in`` file contains the line exec python3 spkg-check.py +The install-requires.txt file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + All normal Python packages and all wheel packages must have a file ``install-requires.txt``. If a Python package is available on PyPI, this file must contain the -name of the package as it is known to PyPI. Optionally, +name of the package as it is known to PyPI. + +Optionally, ``install-requires.txt`` can encode version constraints (such as lower and upper bounds). The constraints are in the format of the ``install_requires`` key of `setup.cfg @@ -613,6 +618,16 @@ and upper bounds). The constraints are in the format of the or `setup.py `_. +Sage uses these version constraints for two purposes: + +- As a source for generating the metadata of the Python + distribution packages in ``SAGE_ROOT/pkgs/``, see + :ref:`section_dependencies_distributions`. + +- When the experimental option ``configure --enable-system-site-packages`` is used, + then the ``configure`` script checks these constraints to determine whether + to accept an installation of this package in the system Python. + It is strongly recommended to include comments (starting with ``#``) in the file that explain why a particular lower or upper bound is warranted or why we wish to include or reject certain versions. From c6b56632227bd5b1e2dbc04800ef2472659066fa Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Fri, 23 Feb 2024 08:36:41 -0800 Subject: [PATCH 3/5] src/doc/en/developer/packaging.rst: More on package-version.txt for Python packages --- src/doc/en/developer/packaging.rst | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/src/doc/en/developer/packaging.rst b/src/doc/en/developer/packaging.rst index fb70576d061..541192fa60a 100644 --- a/src/doc/en/developer/packaging.rst +++ b/src/doc/en/developer/packaging.rst @@ -603,8 +603,8 @@ For example, the ``scipy`` ``spkg-check.in`` file contains the line exec python3 spkg-check.py -The install-requires.txt file -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Abstract requirements: The install-requires.txt file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All normal Python packages and all wheel packages must have a file ``install-requires.txt``. If a Python package is available on PyPI, this file must contain the @@ -660,6 +660,21 @@ Setting upper bounds to guard against incompatible future changes is a complex topic; see :trac:`33520`. +Concrete (pinned) requirements: The package-version.txt file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Like normal non-Python packages, all normal Python packages and all wheel packages +must have a file ``package-version.txt``. + +Sage uses this version for two purposes: + +- This is the version that the Sage distribution ships. + +- As a source for generating the ``requirements.txt`` files of + the Python distribution packages in ``SAGE_ROOT/pkgs/``, see + :ref:`section_dependencies_distributions`. + + .. _section-spkg-SPKG-txt: The SPKG.rst file From c3dfcbec929dc3e55292fd5b92f70fd9b59e7581 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Fri, 23 Feb 2024 08:44:26 -0800 Subject: [PATCH 4/5] src/doc/en/developer/packaging.rst: Link to pip User Guide --- src/doc/en/developer/packaging.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/doc/en/developer/packaging.rst b/src/doc/en/developer/packaging.rst index 541192fa60a..33d88ac99ed 100644 --- a/src/doc/en/developer/packaging.rst +++ b/src/doc/en/developer/packaging.rst @@ -674,6 +674,9 @@ Sage uses this version for two purposes: the Python distribution packages in ``SAGE_ROOT/pkgs/``, see :ref:`section_dependencies_distributions`. + For the use of the generated ``requirements.txt`` files, see + the `pip User Guide `_. + .. _section-spkg-SPKG-txt: From 8c5710d14f0f9a0748dc2b3c3e5474f1639fd598 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Tue, 27 Feb 2024 18:32:03 -0800 Subject: [PATCH 5/5] src/doc/en/developer/packaging.rst: Add more material on version pinning etc. from sage-devel thread 'allow standard packages to be pip packages' --- src/doc/en/developer/packaging.rst | 90 ++++++++++++++++++++++++++---- 1 file changed, 79 insertions(+), 11 deletions(-) diff --git a/src/doc/en/developer/packaging.rst b/src/doc/en/developer/packaging.rst index 33d88ac99ed..028725ad8ce 100644 --- a/src/doc/en/developer/packaging.rst +++ b/src/doc/en/developer/packaging.rst @@ -579,7 +579,7 @@ and most Python-based packages will also have ``$(PYTHON_TOOLCHAIN)`` as an order-only dependency, which will ensure that fundamental packages such as ``pip`` and ``setuptools`` are available at the time of building the package. -The best way to install a Python-based package is to use ``pip``, in which +The best way to install a ``normal`` Python-based package is to use ``pip``, in which case the ``spkg-install.in`` script template might just consist of .. CODE-BLOCK:: bash @@ -603,12 +603,15 @@ For example, the ``scipy`` ``spkg-check.in`` file contains the line exec python3 spkg-check.py -Abstract requirements: The install-requires.txt file -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Abstract requirements: The ``install-requires.txt`` file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -All normal Python packages and all wheel packages must have a file ``install-requires.txt``. -If a Python package is available on PyPI, this file must contain the -name of the package as it is known to PyPI. +All ``normal`` Python packages and all ``wheel`` packages must have a file +``install-requires.txt``. For ``pip`` packages, the file is optional; if +it is missing, the ``requirements.txt`` file is used instead. + +If a Python package is available on PyPI, the ``install-requires.txt`` file must +contain the name of the package as it is known to PyPI. Optionally, ``install-requires.txt`` can encode version constraints (such as lower @@ -660,11 +663,11 @@ Setting upper bounds to guard against incompatible future changes is a complex topic; see :trac:`33520`. -Concrete (pinned) requirements: The package-version.txt file -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Concrete (pinned) requirements of ``normal``, ``wheel``, ``script`` packages: The ``package-version.txt`` file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Like normal non-Python packages, all normal Python packages and all wheel packages -must have a file ``package-version.txt``. +Like ``normal`` non-Python packages, all ``normal`` Python packages and all ``wheel`` packages +must have a file ``package-version.txt``. For ``script`` Python packages, the file is optional. Sage uses this version for two purposes: @@ -678,6 +681,69 @@ Sage uses this version for two purposes: the `pip User Guide `_. +Concrete requirements of ``pip`` packages: The ``requirements.txt`` file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In contrast to ``normal``, ``wheel``, and ``script`` packages, the +``pip`` packages do not use a ``package-version.txt`` file. + +Instead, the concrete requirements are set in a ``requirements.txt`` +file, which is passed directly to ``pip`` at installation time. + +The ``requirements.txt`` file uses a very flexible format, defined +in the `pip User Guide +`_. +Through this format, the concrete requirements can either be +pinned to a specific version, or set acceptable version ranges, or be +entirely unconstrained. The format is even flexible enough to install +several distribution packages at the same time, and to conditionalize +on the operating system or Python version. + +Pinning a version has the potential benefit of stability, as it can +avoid retroactive breakage of the Sage distribution by new, +incompatible versions, and can also help achieve reproducibility +of computations. + +The cost is that updating the version requires +work by at least two Sage developers: One who prepares a PR and one +who reviews it. Moreover, when the package does not get the attention of +developers who upgrade it, there is the potential risk of missing out +on bugfixes made in newer versions, or missing out on features in +major new versions. + +Not pinning the version has the obvious potential benefit of always +being up to date, as ``pip`` contacts the index server (PyPI) to +obtain and install the package. (Note that ``normal`` and ``wheel`` +packages are always pinned and do not even have access to the index +server at the time of building and installing the package.) + +But this dynamism also brings a risk +of instability, either by the package itself being affected by bugs in +a new version, or by breaking compatibility with Sage. + +What policy is best for a package depends on various factors, +including the development velocity and quality control that the +upstream project uses, the interest by Sage developers in the package, +the depth of integration in Sage, whether it affects the mathematics, +etc. + + +Note about dependencies of ``pip`` packages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Dependencies of a ``pip`` package do not need to be available as packages +in the Sage distribution, as the package can pull some of its build-time and +run-time dependencies directly from PyPI. That's a mild convenience for developers, +and can be important if one wants to leave the version range wide open. + +However, if a dependency is also a package of the Sage distribution, +then we must declare this dependency. Otherwise, various errors +can occur when building or upgrading. When new versions of ``pip`` +packages add dependencies that happen to be Sage packages, there is a +separate source of instability. + + + .. _section-spkg-SPKG-txt: The SPKG.rst file @@ -1143,7 +1209,9 @@ For Python packages available from PyPI, you can use:: --type optional This automatically downloads the most recent version from PyPI and also -obtains most of the necessary information by querying PyPI. +obtains most of the necessary information by querying PyPI. In particular, +the ``SPKG.rst`` file is created as a copy of the package's README file. + The ``dependencies`` file may need editing (watch out for warnings regarding ``--no-deps`` that Sage issues during installation of the package!).