gh-91172: Create a workflow for verifying bundled pip and setuptools

[illia-v]

https://bugs.python.org/issue47016

[hugovk]

Perhaps this could be moved to a script (say, under Tools/scripts), so it can also be run locally?

[illia-v]

Good idea, but it looks like all scripts in Tools/scripts are written in Python.
This is a Bash script, and it may not work on some platforms.

Do you know how compatible with different platforms scripts located in Tools/scripts should be?

[hugovk]

Hmm, or perhaps the Misc directory? The devguide says:

Various tools with configuration files as found in the Misc directory

I don't know what compatibility is needed, but as this script is primarily intended for Ubuntu on the CI, I'd stick with that.

[illia-v]

Done. This is the latest run after I corrupted the pip wheel temporarily.

BTW, I added messages that GitHub has to show as annotations. But I could not see the annotations, maybe it does not show them for binary files…

[hugovk]

They show up on the Summary page, for example clicking "🏠 Summary" at the top left:

👍

[hugovk]

Thank you! 🕊️🇺🇦✌️

[AA-Turner]

@illia-v the below is my attempt at a Python rewrite:

"""Compares checksums for wheels in :mod:`ensurepip` against the Cheeseshop."""

import hashlib
import json
from pathlib import Path
import re
from urllib.request import urlopen

PACKAGE_NAMES = ("pip", "setuptools")
ROOT = Path(__file__).parent.parent / "Lib/ensurepip"
WHEEL_DIR = ROOT / "_bundled"
ENSURE_PIP_INIT_PY_TEXT = (ROOT / "__init__.py").read_text(encoding="utf-8")

# Contains names of successfully verified packages
verified_packages = {*()}

for package_name in PACKAGE_NAMES:
    # Find the package on disk
    package_path = next(WHEEL_DIR.glob(f"{package_name}*.whl"), None)
    if package_path is None:
        continue

    print(f"Verifying checksum for {package_path}.")

    # Find the version of the package used by ensurepip
    package_version_match = re.search(
        f'_{package_name.upper()}_VERSION = "([^"]+)',
        ENSURE_PIP_INIT_PY_TEXT
    )
    if package_version_match is None:
        continue
    package_version = package_version_match[1]

    # Get the SHA 256 digest from the Cheeseshop
    try:
        raw_text = urlopen(f"https://pypi.org/pypi/{package_name}/json").read()
    except (OSError, ValueError) as error:
        continue

    expected_digest = ""
    release_files = json.loads(raw_text)["releases"][package_version]
    for release_info in release_files:
        if package_path.name != release_info["filename"]:
            continue
        expected_digest = release_info["digests"].get("sha256", "")
    if expected_digest == "":
        continue

    # Compute the SHA 256 digest of the wheel on disk
    actual_digest = hashlib.sha256(package_path.read_bytes()).hexdigest()

    print(f"Expected digest: {expected_digest}")
    print(f"Actual digest:   {actual_digest}")

    # The messages are formatted to be parsed by GitHub Actions.
    # https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-a-notice-message
    if actual_digest == expected_digest:
        print(f"::notice file={package_path}::"
              f"Successfully verified the checksum of the {package_name} wheel.\n")
        verified_packages.add(package_name)
    else:
        print(f"::error file={package_path}::"
              f"Failed to verify the checksum of the {package_name} wheel.\n")

# If we verified all packages, the set of package names and the set of verified
# packages will be equal.
if {*PACKAGE_NAMES} == verified_packages:
    raise SystemExit(0)

# Otherwise we failed to verify all the packages.
raise SystemExit(1)

A

[illia-v]

@illia-v the below is my attempt at a Python rewrite:

@AA-Turner thanks! Could you please push it to this branch or create a pull request to it?

Also, what do you think about creating the {*()} and {*PACKAGE_NAMES} sets more explicitly using the set class? And printing an error near every continue?

BTW, isn't Warehouse the new name of what was called Cheese Shop?

[pradyunsg]

BTW, isn't Warehouse the new name of what was called Cheese Shop?

Well, Python Package Index used to be called Cheese Shop (based on a Monty Python sketch about a Cheese Shop that didn't have any cheese) -- the current codebase for PyPI is called Warehouse: https://github.com/pypa/warehouse/

[miss-islington]

Thanks @illia-v for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8, 3.9, 3.10, 3.11.
🐍🍒⛏🤖

[bedevere-bot]

GH-94121 is a backport of this pull request to the 3.11 branch.

[bedevere-bot]

GH-94123 is a backport of this pull request to the 3.9 branch.

[bedevere-bot]

GH-94124 is a backport of this pull request to the 3.8 branch.

[bedevere-bot]

GH-94122 is a backport of this pull request to the 3.10 branch.

[ambv]

I added my security branches. Rationale: if there is a security-related bump of pip and/or setuptools, Python 3.7 - 3.9 security releases will be made with that bump.

[miss-islington]

Thanks @illia-v for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

[bedevere-bot]

GH-94126 is a backport of this pull request to the 3.7 branch.