bpo-34632: Add importlib.metadata (GH-12547)

Add importlib.metadata module as forward port of the standalone importlib_metadata.

Doc/library/importlib.metadata.rst

@@ -0,0 +1,257 @@
+.. _using:
+
+==========================
+ Using importlib.metadata
+==========================
+
+.. note::
+   This functionality is provisional and may deviate from the usual
+   version semantics of the standard library.
+
+``importlib.metadata`` is a library that provides for access to installed
+package metadata.  Built in part on Python's import system, this library
+intends to replace similar functionality in the `entry point
+API`_ and `metadata API`_ of ``pkg_resources``.  Along with
+``importlib.resources`` in `Python 3.7
+and newer`_ (backported as `importlib_resources`_ for older versions of
+Python), this can eliminate the need to use the older and less efficient
+``pkg_resources`` package.
+
+By "installed package" we generally mean a third-party package installed into
+Python's ``site-packages`` directory via tools such as `pip
+<https://pypi.org/project/pip/>`_.  Specifically,
+it means a package with either a discoverable ``dist-info`` or ``egg-info``
+directory, and metadata defined by `PEP 566`_ or its older specifications.
+By default, package metadata can live on the file system or in zip archives on
+``sys.path``.  Through an extension mechanism, the metadata can live almost
+anywhere.
+
+
+Overview
+========
+
+Let's say you wanted to get the version string for a package you've installed
+using ``pip``.  We start by creating a virtual environment and installing
+something into it::
+
+.. highlight:: none
+
+    $ python3 -m venv example
+    $ source example/bin/activate
+    (example) $ pip install wheel
+
+You can get the version string for ``wheel`` by running the following::
+
+.. highlight:: none
+
+    (example) $ python
+    >>> from importlib.metadata import version  # doctest: +SKIP
+    >>> version('wheel')  # doctest: +SKIP
+    '0.32.3'
+
+You can also get the set of entry points keyed by group, such as
+``console_scripts``, ``distutils.commands`` and others.  Each group contains a
+sequence of :ref:`EntryPoint <entry-points>` objects.
+
+You can get the :ref:`metadata for a distribution <metadata>`::
+
+    >>> list(metadata('wheel'))  # doctest: +SKIP
+    ['Metadata-Version', 'Name', 'Version', 'Summary', 'Home-page', 'Author', 'Author-email', 'Maintainer', 'Maintainer-email', 'License', 'Project-URL', 'Project-URL', 'Project-URL', 'Keywords', 'Platform', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Requires-Python', 'Provides-Extra', 'Requires-Dist', 'Requires-Dist']
+
+You can also get a :ref:`distribution's version number <version>`, list its
+:ref:`constituent files <files>`, and get a list of the distribution's
+:ref:`requirements`.
+
+
+Functional API
+==============
+
+This package provides the following functionality via its public API.
+
+
+.. _entry-points:
+
+Entry points
+------------
+
+The ``entry_points()`` function returns a dictionary of all entry points,
+keyed by group.  Entry points are represented by ``EntryPoint`` instances;
+each ``EntryPoint`` has a ``.name``, ``.group``, and ``.value`` attributes and
+a ``.load()`` method to resolve the value.
+
+    >>> eps = entry_points()  # doctest: +SKIP
+    >>> list(eps)  # doctest: +SKIP
+    ['console_scripts', 'distutils.commands', 'distutils.setup_keywords', 'egg_info.writers', 'setuptools.installation']
+    >>> scripts = eps['console_scripts']  # doctest: +SKIP
+    >>> wheel = [ep for ep in scripts if ep.name == 'wheel'][0]  # doctest: +SKIP
+    >>> wheel  # doctest: +SKIP
+    EntryPoint(name='wheel', value='wheel.cli:main', group='console_scripts')
+    >>> main = wheel.load()  # doctest: +SKIP
+    >>> main  # doctest: +SKIP
+    <function main at 0x103528488>
+
+The ``group`` and ``name`` are arbitrary values defined by the package author
+and usually a client will wish to resolve all entry points for a particular
+group.  Read `the setuptools docs
+<https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins>`_
+for more information on entrypoints, their definition, and usage.
+
+
+.. _metadata:
+
+Distribution metadata
+---------------------
+
+Every distribution includes some metadata, which you can extract using the
+``metadata()`` function::
+
+    >>> wheel_metadata = metadata('wheel')  # doctest: +SKIP
+
+The keys of the returned data structure [#f1]_ name the metadata keywords, and
+their values are returned unparsed from the distribution metadata::
+
+    >>> wheel_metadata['Requires-Python']  # doctest: +SKIP
+    '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*'
+
+
+.. _version:
+
+Distribution versions
+---------------------
+
+The ``version()`` function is the quickest way to get a distribution's version
+number, as a string::
+
+    >>> version('wheel')  # doctest: +SKIP
+    '0.32.3'
+
+
+.. _files:
+
+Distribution files
+------------------
+
+You can also get the full set of files contained within a distribution.  The
+``files()`` function takes a distribution package name and returns all of the
+files installed by this distribution.  Each file object returned is a
+``PackagePath``, a `pathlib.Path`_ derived object with additional ``dist``,
+``size``, and ``hash`` properties as indicated by the metadata.  For example::
+
+    >>> util = [p for p in files('wheel') if 'util.py' in str(p)][0]  # doctest: +SKIP
+    >>> util  # doctest: +SKIP
+    PackagePath('wheel/util.py')
+    >>> util.size  # doctest: +SKIP
+    859
+    >>> util.dist  # doctest: +SKIP
+    <importlib.metadata._hooks.PathDistribution object at 0x101e0cef0>
+    >>> util.hash  # doctest: +SKIP
+    <FileHash mode: sha256 value: bYkw5oMccfazVCoYQwKkkemoVyMAFoR34mmKBx8R1NI>
+
+Once you have the file, you can also read its contents::
+
+    >>> print(util.read_text())  # doctest: +SKIP
+    import base64
+    import sys
+    ...
+    def as_bytes(s):
+        if isinstance(s, text_type):
+            return s.encode('utf-8')
+        return s
+
+
+.. _requirements:
+
+Distribution requirements
+-------------------------
+
+To get the full set of requirements for a distribution, use the ``requires()``
+function.  Note that this returns an iterator::
+
+    >>> list(requires('wheel'))  # doctest: +SKIP
+    ["pytest (>=3.0.0) ; extra == 'test'"]
+
+
+Distributions
+=============
+
+While the above API is the most common and convenient usage, you can get all
+of that information from the ``Distribution`` class.  A ``Distribution`` is an
+abstract object that represents the metadata for a Python package.  You can
+get the ``Distribution`` instance::
+
+    >>> from importlib.metadata import distribution  # doctest: +SKIP
+    >>> dist = distribution('wheel')  # doctest: +SKIP
+
+Thus, an alternative way to get the version number is through the
+``Distribution`` instance::
+
+    >>> dist.version  # doctest: +SKIP
+    '0.32.3'
+
+There are all kinds of additional metadata available on the ``Distribution``
+instance::
+
+    >>> d.metadata['Requires-Python']  # doctest: +SKIP
+    '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*'
+    >>> d.metadata['License']  # doctest: +SKIP
+    'MIT'
+
+The full set of available metadata is not described here.  See `PEP 566
+<https://www.python.org/dev/peps/pep-0566/>`_ for additional details.
+
+
+Extending the search algorithm
+==============================
+
+Because package metadata is not available through ``sys.path`` searches, or
+package loaders directly, the metadata for a package is found through import
+system `finders`_.  To find a distribution package's metadata,
+``importlib.metadata`` queries the list of `meta path finders`_ on
+`sys.meta_path`_.
+
+By default ``importlib.metadata`` installs a finder for distribution packages
+found on the file system.  This finder doesn't actually find any *packages*,
+but it can find the packages' metadata.
+
+The abstract class :py:class:`importlib.abc.MetaPathFinder` defines the
+interface expected of finders by Python's import system.
+``importlib.metadata`` extends this protocol by looking for an optional
+``find_distributions`` callable on the finders from
+``sys.meta_path``.  If the finder has this method, it must return
+an iterator over instances of the ``Distribution`` abstract class. This
+method must have the signature::
+
+    def find_distributions(name=None, path=None):
+        """Return an iterable of all Distribution instances capable of
+        loading the metadata for packages matching the name
+        (or all names if not supplied) along the paths in the list
+        of directories ``path`` (defaults to sys.path).
+        """
+
+What this means in practice is that to support finding distribution package
+metadata in locations other than the file system, you should derive from
+``Distribution`` and implement the ``load_metadata()`` method.  This takes a
+single argument which is the name of the package whose metadata is being
+found.  This instance of the ``Distribution`` base abstract class is what your
+finder's ``find_distributions()`` method should return.
+
+
+.. _`entry point API`: https://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points
+.. _`metadata API`: https://setuptools.readthedocs.io/en/latest/pkg_resources.html#metadata-api
+.. _`Python 3.7 and newer`: https://docs.python.org/3/library/importlib.html#module-importlib.resources
+.. _`importlib_resources`: https://importlib-resources.readthedocs.io/en/latest/index.html
+.. _`PEP 566`: https://www.python.org/dev/peps/pep-0566/
+.. _`finders`: https://docs.python.org/3/reference/import.html#finders-and-loaders
+.. _`meta path finders`: https://docs.python.org/3/glossary.html#term-meta-path-finder
+.. _`sys.meta_path`: https://docs.python.org/3/library/sys.html#sys.meta_path
+.. _`pathlib.Path`: https://docs.python.org/3/library/pathlib.html#pathlib.Path
+
+
+.. rubric:: Footnotes
+
+.. [#f1] Technically, the returned distribution metadata object is an
+         `email.message.Message
+         <https://docs.python.org/3/library/email.message.html#email.message.EmailMessage>`_
+         instance, but this is an implementation detail, and not part of the
+         stable API.  You should only use dictionary-like methods and syntax
+         to access the metadata contents.

Doc/library/modules.rst

@@ -17,3 +17,4 @@ The full list of modules described in this chapter is:
    modulefinder.rst
    runpy.rst
    importlib.rst
+   importlib.metadata.rst

Doc/tools/susp-ignored.csv

@@ -350,3 +350,6 @@ whatsnew/3.7,,::,error::BytesWarning
 whatsnew/changelog,,::,error::BytesWarning
 whatsnew/changelog,,::,default::BytesWarning
 whatsnew/changelog,,::,default::DeprecationWarning
+library/importlib.metadata,,.. highlight:,.. highlight:: none
+library/importlib.metadata,,:main,"EntryPoint(name='wheel', value='wheel.cli:main', group='console_scripts')"
+library/importlib.metadata,,`,of directories ``path`` (defaults to sys.path).

Lib/importlib/_bootstrap_external.py

@@ -1363,6 +1363,58 @@ class PathFinder:
             return None
         return spec.loader
 
+    search_template = r'(?:{pattern}(-.*)?\.(dist|egg)-info|EGG-INFO)'
+
+    @classmethod
+    def find_distributions(cls, name=None, path=None):
+        """
+        Find distributions.
+
+        Return an iterable of all Distribution instances capable of
+        loading the metadata for packages matching the ``name``
+        (or all names if not supplied) along the paths in the list
+        of directories ``path`` (defaults to sys.path).
+        """
+        import re
+        from importlib.metadata import PathDistribution
+        if path is None:
+            path = sys.path
+        pattern = '.*' if name is None else re.escape(name)
+        found = cls._search_paths(pattern, path)
+        return map(PathDistribution, found)
+
+    @classmethod
+    def _search_paths(cls, pattern, paths):
+        """Find metadata directories in paths heuristically."""
+        import itertools
+        return itertools.chain.from_iterable(
+            cls._search_path(path, pattern)
+            for path in map(cls._switch_path, paths)
+            )
+
+    @staticmethod
+    def _switch_path(path):
+        from contextlib import suppress
+        import zipfile
+        from pathlib import Path
+        with suppress(Exception):
+            return zipfile.Path(path)
+        return Path(path)
+
+    @classmethod
+    def _predicate(cls, pattern, root, item):
+        import re
+        return re.match(pattern, str(item.name), flags=re.IGNORECASE)
+
+    @classmethod
+    def _search_path(cls, root, pattern):
+        if not root.is_dir():
+            return ()
+        normalized = pattern.replace('-', '_')
+        matcher = cls.search_template.format(pattern=normalized)
+        return (item for item in root.iterdir()
+                if cls._predicate(matcher, root, item))
+
 
 class FileFinder:
 

Lib/importlib/metadata/__init__.py

@@ -0,0 +1,394 @@
+import io
+import re
+import abc
+import csv
+import sys
+import email
+import pathlib
+import operator
+import functools
+import itertools
+import collections
+
+from configparser import ConfigParser
+from contextlib import suppress
+from importlib import import_module
+from importlib.abc import MetaPathFinder
+from itertools import starmap
+
+
+__all__ = [
+    'Distribution',
+    'PackageNotFoundError',
+    'distribution',
+    'distributions',
+    'entry_points',
+    'files',
+    'metadata',
+    'requires',
+    'version',
+    ]
+
+
+class PackageNotFoundError(ModuleNotFoundError):
+    """The package was not found."""
+
+
+class EntryPoint(collections.namedtuple('EntryPointBase', 'name value group')):
+    """An entry point as defined by Python packaging conventions.
+
+    See `the packaging docs on entry points
+    <https://packaging.python.org/specifications/entry-points/>`_
+    for more information.
+    """
+
+    pattern = re.compile(
+        r'(?P<module>[\w.]+)\s*'
+        r'(:\s*(?P<attr>[\w.]+))?\s*'
+        r'(?P<extras>\[.*\])?\s*$'
+        )
+    """
+    A regular expression describing the syntax for an entry point,
+    which might look like:
+
+        - module
+        - package.module
+        - package.module:attribute
+        - package.module:object.attribute
+        - package.module:attr [extra1, extra2]
+
+    Other combinations are possible as well.
+
+    The expression is lenient about whitespace around the ':',
+    following the attr, and following any extras.
+    """
+
+    def load(self):
+        """Load the entry point from its definition. If only a module
+        is indicated by the value, return that module. Otherwise,
+        return the named object.
+        """
+        match = self.pattern.match(self.value)
+        module = import_module(match.group('module'))
+        attrs = filter(None, (match.group('attr') or '').split('.'))
+        return functools.reduce(getattr, attrs, module)
+
+    @property
+    def extras(self):
+        match = self.pattern.match(self.value)
+        return list(re.finditer(r'\w+', match.group('extras') or ''))
+
+    @classmethod
+    def _from_config(cls, config):
+        return [
+            cls(name, value, group)
+            for group in config.sections()
+            for name, value in config.items(group)
+            ]
+
+    @classmethod
+    def _from_text(cls, text):
+        config = ConfigParser()
+        try:
+            config.read_string(text)
+        except AttributeError:  # pragma: nocover
+            # Python 2 has no read_string
+            config.readfp(io.StringIO(text))
+        return EntryPoint._from_config(config)
+
+    def __iter__(self):
+        """
+        Supply iter so one may construct dicts of EntryPoints easily.
+        """
+        return iter((self.name, self))
+
+
+class PackagePath(pathlib.PurePosixPath):
+    """A reference to a path in a package"""
+
+    def read_text(self, encoding='utf-8'):
+        with self.locate().open(encoding=encoding) as stream:
+            return stream.read()
+
+    def read_binary(self):
+        with self.locate().open('rb') as stream:
+            return stream.read()
+
+    def locate(self):
+        """Return a path-like object for this path"""
+        return self.dist.locate_file(self)
+
+
+class FileHash:
+    def __init__(self, spec):
+        self.mode, _, self.value = spec.partition('=')
+
+    def __repr__(self):
+        return '<FileHash mode: {} value: {}>'.format(self.mode, self.value)
+
+
+class Distribution:
+    """A Python distribution package."""
+
+    @abc.abstractmethod
+    def read_text(self, filename):
+        """Attempt to load metadata file given by the name.
+
+        :param filename: The name of the file in the distribution info.
+        :return: The text if found, otherwise None.
+        """
+
+    @abc.abstractmethod
+    def locate_file(self, path):
+        """
+        Given a path to a file in this distribution, return a path
+        to it.
+        """
+
+    @classmethod
+    def from_name(cls, name):
+        """Return the Distribution for the given package name.
+
+        :param name: The name of the distribution package to search for.
+        :return: The Distribution instance (or subclass thereof) for the named
+            package, if found.
+        :raises PackageNotFoundError: When the named package's distribution
+            metadata cannot be found.
+        """
+        for resolver in cls._discover_resolvers():
+            dists = resolver(name)
+            dist = next(dists, None)
+            if dist is not None:
+                return dist
+        else:
+            raise PackageNotFoundError(name)
+
+    @classmethod
+    def discover(cls):
+        """Return an iterable of Distribution objects for all packages.
+
+        :return: Iterable of Distribution objects for all packages.
+        """
+        return itertools.chain.from_iterable(
+            resolver()
+            for resolver in cls._discover_resolvers()
+            )
+
+    @staticmethod
+    def _discover_resolvers():
+        """Search the meta_path for resolvers."""
+        declared = (
+            getattr(finder, 'find_distributions', None)
+            for finder in sys.meta_path
+            )
+        return filter(None, declared)
+
+    @property
+    def metadata(self):
+        """Return the parsed metadata for this Distribution.
+
+        The returned object will have keys that name the various bits of
+        metadata.  See PEP 566 for details.
+        """
+        text = (
+            self.read_text('METADATA')
+            or self.read_text('PKG-INFO')
+            # This last clause is here to support old egg-info files.  Its
+            # effect is to just end up using the PathDistribution's self._path
+            # (which points to the egg-info file) attribute unchanged.
+            or self.read_text('')
+            )
+        return email.message_from_string(text)
+
+    @property
+    def version(self):
+        """Return the 'Version' metadata for the distribution package."""
+        return self.metadata['Version']
+
+    @property
+    def entry_points(self):
+        return EntryPoint._from_text(self.read_text('entry_points.txt'))
+
+    @property
+    def files(self):
+        file_lines = self._read_files_distinfo() or self._read_files_egginfo()
+
+        def make_file(name, hash=None, size_str=None):
+            result = PackagePath(name)
+            result.hash = FileHash(hash) if hash else None
+            result.size = int(size_str) if size_str else None
+            result.dist = self
+            return result
+
+        return file_lines and starmap(make_file, csv.reader(file_lines))
+
+    def _read_files_distinfo(self):
+        """
+        Read the lines of RECORD
+        """
+        text = self.read_text('RECORD')
+        return text and text.splitlines()
+
+    def _read_files_egginfo(self):
+        """
+        SOURCES.txt might contain literal commas, so wrap each line
+        in quotes.
+        """
+        text = self.read_text('SOURCES.txt')
+        return text and map('"{}"'.format, text.splitlines())
+
+    @property
+    def requires(self):
+        """Generated requirements specified for this Distribution"""
+        return self._read_dist_info_reqs() or self._read_egg_info_reqs()
+
+    def _read_dist_info_reqs(self):
+        spec = self.metadata['Requires-Dist']
+        return spec and filter(None, spec.splitlines())
+
+    def _read_egg_info_reqs(self):
+        source = self.read_text('requires.txt')
+        return source and self._deps_from_requires_text(source)
+
+    @classmethod
+    def _deps_from_requires_text(cls, source):
+        section_pairs = cls._read_sections(source.splitlines())
+        sections = {
+            section: list(map(operator.itemgetter('line'), results))
+            for section, results in
+            itertools.groupby(section_pairs, operator.itemgetter('section'))
+            }
+        return cls._convert_egg_info_reqs_to_simple_reqs(sections)
+
+    @staticmethod
+    def _read_sections(lines):
+        section = None
+        for line in filter(None, lines):
+            section_match = re.match(r'\[(.*)\]$', line)
+            if section_match:
+                section = section_match.group(1)
+                continue
+            yield locals()
+
+    @staticmethod
+    def _convert_egg_info_reqs_to_simple_reqs(sections):
+        """
+        Historically, setuptools would solicit and store 'extra'
+        requirements, including those with environment markers,
+        in separate sections. More modern tools expect each
+        dependency to be defined separately, with any relevant
+        extras and environment markers attached directly to that
+        requirement. This method converts the former to the
+        latter. See _test_deps_from_requires_text for an example.
+        """
+        def make_condition(name):
+            return name and 'extra == "{name}"'.format(name=name)
+
+        def parse_condition(section):
+            section = section or ''
+            extra, sep, markers = section.partition(':')
+            if extra and markers:
+                markers = '({markers})'.format(markers=markers)
+            conditions = list(filter(None, [markers, make_condition(extra)]))
+            return '; ' + ' and '.join(conditions) if conditions else ''
+
+        for section, deps in sections.items():
+            for dep in deps:
+                yield dep + parse_condition(section)
+
+
+class DistributionFinder(MetaPathFinder):
+    """
+    A MetaPathFinder capable of discovering installed distributions.
+    """
+
+    @abc.abstractmethod
+    def find_distributions(self, name=None, path=None):
+        """
+        Find distributions.
+
+        Return an iterable of all Distribution instances capable of
+        loading the metadata for packages matching the ``name``
+        (or all names if not supplied) along the paths in the list
+        of directories ``path`` (defaults to sys.path).
+        """
+
+
+class PathDistribution(Distribution):
+    def __init__(self, path):
+        """Construct a distribution from a path to the metadata directory."""
+        self._path = path
+
+    def read_text(self, filename):
+        with suppress(FileNotFoundError, NotADirectoryError, KeyError):
+            return self._path.joinpath(filename).read_text(encoding='utf-8')
+    read_text.__doc__ = Distribution.read_text.__doc__
+
+    def locate_file(self, path):
+        return self._path.parent / path
+
+
+def distribution(package):
+    """Get the ``Distribution`` instance for the given package.
+
+    :param package: The name of the package as a string.
+    :return: A ``Distribution`` instance (or subclass thereof).
+    """
+    return Distribution.from_name(package)
+
+
+def distributions():
+    """Get all ``Distribution`` instances in the current environment.
+
+    :return: An iterable of ``Distribution`` instances.
+    """
+    return Distribution.discover()
+
+
+def metadata(package):
+    """Get the metadata for the package.
+
+    :param package: The name of the distribution package to query.
+    :return: An email.Message containing the parsed metadata.
+    """
+    return Distribution.from_name(package).metadata
+
+
+def version(package):
+    """Get the version string for the named package.
+
+    :param package: The name of the distribution package to query.
+    :return: The version string for the package as defined in the package's
+        "Version" metadata key.
+    """
+    return distribution(package).version
+
+
+def entry_points():
+    """Return EntryPoint objects for all installed packages.
+
+    :return: EntryPoint objects for all installed packages.
+    """
+    eps = itertools.chain.from_iterable(
+        dist.entry_points for dist in distributions())
+    by_group = operator.attrgetter('group')
+    ordered = sorted(eps, key=by_group)
+    grouped = itertools.groupby(ordered, by_group)
+    return {
+        group: tuple(eps)
+        for group, eps in grouped
+        }
+
+
+def files(package):
+    return distribution(package).files
+
+
+def requires(package):
+    """
+    Return a list of requirements for the indicated distribution.
+
+    :return: An iterator of requirements, suitable for
+    packaging.requirement.Requirement.
+    """
+    return distribution(package).requires

Lib/test/test_importlib/fixtures.py

@@ -0,0 +1,199 @@
+from __future__ import unicode_literals
+
+import os
+import sys
+import shutil
+import tempfile
+import textwrap
+import contextlib
+
+try:
+    from contextlib import ExitStack
+except ImportError:
+    from contextlib2 import ExitStack
+
+try:
+    import pathlib
+except ImportError:
+    import pathlib2 as pathlib
+
+
+__metaclass__ = type
+
+
+@contextlib.contextmanager
+def tempdir():
+    tmpdir = tempfile.mkdtemp()
+    try:
+        yield pathlib.Path(tmpdir)
+    finally:
+        shutil.rmtree(tmpdir)
+
+
+@contextlib.contextmanager
+def save_cwd():
+    orig = os.getcwd()
+    try:
+        yield
+    finally:
+        os.chdir(orig)
+
+
+@contextlib.contextmanager
+def tempdir_as_cwd():
+    with tempdir() as tmp:
+        with save_cwd():
+            os.chdir(str(tmp))
+            yield tmp
+
+
+class SiteDir:
+    def setUp(self):
+        self.fixtures = ExitStack()
+        self.addCleanup(self.fixtures.close)
+        self.site_dir = self.fixtures.enter_context(tempdir())
+
+
+class OnSysPath:
+    @staticmethod
+    @contextlib.contextmanager
+    def add_sys_path(dir):
+        sys.path[:0] = [str(dir)]
+        try:
+            yield
+        finally:
+            sys.path.remove(str(dir))
+
+    def setUp(self):
+        super(OnSysPath, self).setUp()
+        self.fixtures.enter_context(self.add_sys_path(self.site_dir))
+
+
+class DistInfoPkg(OnSysPath, SiteDir):
+    files = {
+        "distinfo_pkg-1.0.0.dist-info": {
+            "METADATA": """
+                Name: distinfo-pkg
+                Author: Steven Ma
+                Version: 1.0.0
+                Requires-Dist: wheel >= 1.0
+                Requires-Dist: pytest; extra == 'test'
+                """,
+            "RECORD": "mod.py,sha256=abc,20\n",
+            "entry_points.txt": """
+                [entries]
+                main = mod:main
+            """
+            },
+        "mod.py": """
+            def main():
+                print("hello world")
+            """,
+        }
+
+    def setUp(self):
+        super(DistInfoPkg, self).setUp()
+        build_files(DistInfoPkg.files, self.site_dir)
+
+
+class DistInfoPkgOffPath(SiteDir):
+    def setUp(self):
+        super(DistInfoPkgOffPath, self).setUp()
+        build_files(DistInfoPkg.files, self.site_dir)
+
+
+class EggInfoPkg(OnSysPath, SiteDir):
+    files = {
+        "egginfo_pkg.egg-info": {
+            "PKG-INFO": """
+                Name: egginfo-pkg
+                Author: Steven Ma
+                License: Unknown
+                Version: 1.0.0
+                Classifier: Intended Audience :: Developers
+                Classifier: Topic :: Software Development :: Libraries
+                """,
+            "SOURCES.txt": """
+                mod.py
+                egginfo_pkg.egg-info/top_level.txt
+            """,
+            "entry_points.txt": """
+                [entries]
+                main = mod:main
+            """,
+            "requires.txt": """
+                wheel >= 1.0; python_version >= "2.7"
+                [test]
+                pytest
+            """,
+            "top_level.txt": "mod\n"
+            },
+        "mod.py": """
+            def main():
+                print("hello world")
+            """,
+        }
+
+    def setUp(self):
+        super(EggInfoPkg, self).setUp()
+        build_files(EggInfoPkg.files, prefix=self.site_dir)
+
+
+class EggInfoFile(OnSysPath, SiteDir):
+    files = {
+        "egginfo_file.egg-info": """
+            Metadata-Version: 1.0
+            Name: egginfo_file
+            Version: 0.1
+            Summary: An example package
+            Home-page: www.example.com
+            Author: Eric Haffa-Vee
+            Author-email: eric@example.coms
+            License: UNKNOWN
+            Description: UNKNOWN
+            Platform: UNKNOWN
+            """,
+        }
+
+    def setUp(self):
+        super(EggInfoFile, self).setUp()
+        build_files(EggInfoFile.files, prefix=self.site_dir)
+
+
+def build_files(file_defs, prefix=pathlib.Path()):
+    """Build a set of files/directories, as described by the
+
+    file_defs dictionary.  Each key/value pair in the dictionary is
+    interpreted as a filename/contents pair.  If the contents value is a
+    dictionary, a directory is created, and the dictionary interpreted
+    as the files within it, recursively.
+
+    For example:
+
+    {"README.txt": "A README file",
+     "foo": {
+        "__init__.py": "",
+        "bar": {
+            "__init__.py": "",
+        },
+        "baz.py": "# Some code",
+     }
+    }
+    """
+    for name, contents in file_defs.items():
+        full_name = prefix / name
+        if isinstance(contents, dict):
+            full_name.mkdir()
+            build_files(contents, prefix=full_name)
+        else:
+            if isinstance(contents, bytes):
+                with full_name.open('wb') as f:
+                    f.write(contents)
+            else:
+                with full_name.open('w') as f:
+                    f.write(DALS(contents))
+
+
+def DALS(str):
+    "Dedent and left-strip"
+    return textwrap.dedent(str).lstrip()

Lib/test/test_importlib/test_main.py

@@ -0,0 +1,158 @@
+# coding: utf-8
+
+import re
+import textwrap
+import unittest
+import importlib.metadata
+
+from . import fixtures
+from importlib.metadata import (
+    Distribution, EntryPoint,
+    PackageNotFoundError, distributions,
+    entry_points, metadata, version,
+    )
+
+
+class BasicTests(fixtures.DistInfoPkg, unittest.TestCase):
+    version_pattern = r'\d+\.\d+(\.\d)?'
+
+    def test_retrieves_version_of_self(self):
+        dist = Distribution.from_name('distinfo-pkg')
+        assert isinstance(dist.version, str)
+        assert re.match(self.version_pattern, dist.version)
+
+    def test_for_name_does_not_exist(self):
+        with self.assertRaises(PackageNotFoundError):
+            Distribution.from_name('does-not-exist')
+
+    def test_new_style_classes(self):
+        self.assertIsInstance(Distribution, type)
+
+
+class ImportTests(fixtures.DistInfoPkg, unittest.TestCase):
+    def test_import_nonexistent_module(self):
+        # Ensure that the MetadataPathFinder does not crash an import of a
+        # non-existant module.
+        with self.assertRaises(ImportError):
+            importlib.import_module('does_not_exist')
+
+    def test_resolve(self):
+        entries = dict(entry_points()['entries'])
+        ep = entries['main']
+        self.assertEqual(ep.load().__name__, "main")
+
+    def test_resolve_without_attr(self):
+        ep = EntryPoint(
+            name='ep',
+            value='importlib.metadata',
+            group='grp',
+            )
+        assert ep.load() is importlib.metadata
+
+
+class NameNormalizationTests(
+        fixtures.OnSysPath, fixtures.SiteDir, unittest.TestCase):
+    @staticmethod
+    def pkg_with_dashes(site_dir):
+        """
+        Create minimal metadata for a package with dashes
+        in the name (and thus underscores in the filename).
+        """
+        metadata_dir = site_dir / 'my_pkg.dist-info'
+        metadata_dir.mkdir()
+        metadata = metadata_dir / 'METADATA'
+        with metadata.open('w') as strm:
+            strm.write('Version: 1.0\n')
+        return 'my-pkg'
+
+    def test_dashes_in_dist_name_found_as_underscores(self):
+        """
+        For a package with a dash in the name, the dist-info metadata
+        uses underscores in the name. Ensure the metadata loads.
+        """
+        pkg_name = self.pkg_with_dashes(self.site_dir)
+        assert version(pkg_name) == '1.0'
+
+    @staticmethod
+    def pkg_with_mixed_case(site_dir):
+        """
+        Create minimal metadata for a package with mixed case
+        in the name.
+        """
+        metadata_dir = site_dir / 'CherryPy.dist-info'
+        metadata_dir.mkdir()
+        metadata = metadata_dir / 'METADATA'
+        with metadata.open('w') as strm:
+            strm.write('Version: 1.0\n')
+        return 'CherryPy'
+
+    def test_dist_name_found_as_any_case(self):
+        """
+        Ensure the metadata loads when queried with any case.
+        """
+        pkg_name = self.pkg_with_mixed_case(self.site_dir)
+        assert version(pkg_name) == '1.0'
+        assert version(pkg_name.lower()) == '1.0'
+        assert version(pkg_name.upper()) == '1.0'
+
+
+class NonASCIITests(fixtures.OnSysPath, fixtures.SiteDir, unittest.TestCase):
+    @staticmethod
+    def pkg_with_non_ascii_description(site_dir):
+        """
+        Create minimal metadata for a package with non-ASCII in
+        the description.
+        """
+        metadata_dir = site_dir / 'portend.dist-info'
+        metadata_dir.mkdir()
+        metadata = metadata_dir / 'METADATA'
+        with metadata.open('w', encoding='utf-8') as fp:
+            fp.write('Description: pôrˈtend\n')
+        return 'portend'
+
+    @staticmethod
+    def pkg_with_non_ascii_description_egg_info(site_dir):
+        """
+        Create minimal metadata for an egg-info package with
+        non-ASCII in the description.
+        """
+        metadata_dir = site_dir / 'portend.dist-info'
+        metadata_dir.mkdir()
+        metadata = metadata_dir / 'METADATA'
+        with metadata.open('w', encoding='utf-8') as fp:
+            fp.write(textwrap.dedent("""
+                Name: portend
+
+                pôrˈtend
+                """).lstrip())
+        return 'portend'
+
+    def test_metadata_loads(self):
+        pkg_name = self.pkg_with_non_ascii_description(self.site_dir)
+        meta = metadata(pkg_name)
+        assert meta['Description'] == 'pôrˈtend'
+
+    def test_metadata_loads_egg_info(self):
+        pkg_name = self.pkg_with_non_ascii_description_egg_info(self.site_dir)
+        meta = metadata(pkg_name)
+        assert meta.get_payload() == 'pôrˈtend\n'
+
+
+class DiscoveryTests(fixtures.EggInfoPkg,
+                     fixtures.DistInfoPkg,
+                     unittest.TestCase):
+
+    def test_package_discovery(self):
+        dists = list(distributions())
+        assert all(
+            isinstance(dist, Distribution)
+            for dist in dists
+            )
+        assert any(
+            dist.metadata['Name'] == 'egginfo-pkg'
+            for dist in dists
+            )
+        assert any(
+            dist.metadata['Name'] == 'distinfo-pkg'
+            for dist in dists
+            )

Lib/test/test_importlib/test_metadata_api.py

@@ -0,0 +1,151 @@
+import re
+import textwrap
+import unittest
+import itertools
+
+from collections.abc import Iterator
+
+from . import fixtures
+from importlib.metadata import (
+    Distribution, PackageNotFoundError, distribution,
+    entry_points, files, metadata, requires, version,
+    )
+
+
+class APITests(
+        fixtures.EggInfoPkg,
+        fixtures.DistInfoPkg,
+        fixtures.EggInfoFile,
+        unittest.TestCase):
+
+    version_pattern = r'\d+\.\d+(\.\d)?'
+
+    def test_retrieves_version_of_self(self):
+        pkg_version = version('egginfo-pkg')
+        assert isinstance(pkg_version, str)
+        assert re.match(self.version_pattern, pkg_version)
+
+    def test_retrieves_version_of_distinfo_pkg(self):
+        pkg_version = version('distinfo-pkg')
+        assert isinstance(pkg_version, str)
+        assert re.match(self.version_pattern, pkg_version)
+
+    def test_for_name_does_not_exist(self):
+        with self.assertRaises(PackageNotFoundError):
+            distribution('does-not-exist')
+
+    def test_for_top_level(self):
+        self.assertEqual(
+            distribution('egginfo-pkg').read_text('top_level.txt').strip(),
+            'mod')
+
+    def test_read_text(self):
+        top_level = [
+            path for path in files('egginfo-pkg')
+            if path.name == 'top_level.txt'
+            ][0]
+        self.assertEqual(top_level.read_text(), 'mod\n')
+
+    def test_entry_points(self):
+        entries = dict(entry_points()['entries'])
+        ep = entries['main']
+        self.assertEqual(ep.value, 'mod:main')
+        self.assertEqual(ep.extras, [])
+
+    def test_metadata_for_this_package(self):
+        md = metadata('egginfo-pkg')
+        assert md['author'] == 'Steven Ma'
+        assert md['LICENSE'] == 'Unknown'
+        assert md['Name'] == 'egginfo-pkg'
+        classifiers = md.get_all('Classifier')
+        assert 'Topic :: Software Development :: Libraries' in classifiers
+
+    @staticmethod
+    def _test_files(files_iter):
+        assert isinstance(files_iter, Iterator), files_iter
+        files = list(files_iter)
+        root = files[0].root
+        for file in files:
+            assert file.root == root
+            assert not file.hash or file.hash.value
+            assert not file.hash or file.hash.mode == 'sha256'
+            assert not file.size or file.size >= 0
+            assert file.locate().exists()
+            assert isinstance(file.read_binary(), bytes)
+            if file.name.endswith('.py'):
+                file.read_text()
+
+    def test_file_hash_repr(self):
+        assertRegex = self.assertRegex
+
+        util = [
+            p for p in files('distinfo-pkg')
+            if p.name == 'mod.py'
+            ][0]
+        assertRegex(
+            repr(util.hash),
+            '<FileHash mode: sha256 value: .*>')
+
+    def test_files_dist_info(self):
+        self._test_files(files('distinfo-pkg'))
+
+    def test_files_egg_info(self):
+        self._test_files(files('egginfo-pkg'))
+
+    def test_version_egg_info_file(self):
+        self.assertEqual(version('egginfo-file'), '0.1')
+
+    def test_requires_egg_info_file(self):
+        requirements = requires('egginfo-file')
+        self.assertIsNone(requirements)
+
+    def test_requires(self):
+        deps = requires('egginfo-pkg')
+        assert any(
+            dep == 'wheel >= 1.0; python_version >= "2.7"'
+            for dep in deps
+            )
+
+    def test_requires_dist_info(self):
+        deps = list(requires('distinfo-pkg'))
+        assert deps and all(deps)
+
+    def test_more_complex_deps_requires_text(self):
+        requires = textwrap.dedent("""
+            dep1
+            dep2
+
+            [:python_version < "3"]
+            dep3
+
+            [extra1]
+            dep4
+
+            [extra2:python_version < "3"]
+            dep5
+            """)
+        deps = sorted(Distribution._deps_from_requires_text(requires))
+        expected = [
+            'dep1',
+            'dep2',
+            'dep3; python_version < "3"',
+            'dep4; extra == "extra1"',
+            'dep5; (python_version < "3") and extra == "extra2"',
+            ]
+        # It's important that the environment marker expression be
+        # wrapped in parentheses to avoid the following 'and' binding more
+        # tightly than some other part of the environment expression.
+
+        assert deps == expected
+
+
+class OffSysPathTests(fixtures.DistInfoPkgOffPath, unittest.TestCase):
+    def test_find_distributions_specified_path(self):
+        dists = itertools.chain.from_iterable(
+            resolver(path=[str(self.site_dir)])
+            for resolver in Distribution._discover_resolvers()
+            )
+        assert any(
+            dist.metadata['Name'] == 'distinfo-pkg'
+            for dist in dists
+            )

Lib/test/test_importlib/test_zip.py

@@ -0,0 +1,56 @@
+import sys
+import unittest
+
+from contextlib import ExitStack
+from importlib.metadata import distribution, entry_points, files, version
+from importlib.resources import path
+
+
+class TestZip(unittest.TestCase):
+    root = 'test.test_importlib.data'
+
+    def setUp(self):
+        # Find the path to the example-*.whl so we can add it to the front of
+        # sys.path, where we'll then try to find the metadata thereof.
+        self.resources = ExitStack()
+        self.addCleanup(self.resources.close)
+        wheel = self.resources.enter_context(
+            path(self.root, 'example-21.12-py3-none-any.whl'))
+        sys.path.insert(0, str(wheel))
+        self.resources.callback(sys.path.pop, 0)
+
+    def test_zip_version(self):
+        self.assertEqual(version('example'), '21.12')
+
+    def test_zip_entry_points(self):
+        scripts = dict(entry_points()['console_scripts'])
+        entry_point = scripts['example']
+        self.assertEqual(entry_point.value, 'example:main')
+
+    def test_missing_metadata(self):
+        self.assertIsNone(distribution('example').read_text('does not exist'))
+
+    def test_case_insensitive(self):
+        self.assertEqual(version('Example'), '21.12')
+
+    def test_files(self):
+        for file in files('example'):
+            path = str(file.dist.locate_file(file))
+            assert '.whl/' in path, path
+
+
+class TestEgg(TestZip):
+    def setUp(self):
+        # Find the path to the example-*.egg so we can add it to the front of
+        # sys.path, where we'll then try to find the metadata thereof.
+        self.resources = ExitStack()
+        self.addCleanup(self.resources.close)
+        egg = self.resources.enter_context(
+            path(self.root, 'example-21.12-py3.6.egg'))
+        sys.path.insert(0, str(egg))
+        self.resources.callback(sys.path.pop, 0)
+
+    def test_files(self):
+        for file in files('example'):
+            path = str(file.dist.locate_file(file))
+            assert '.egg/' in path, path

Misc/NEWS.d/next/Library/2019-05-11-02-30-45.bpo-34632.8MXa7T.rst

@@ -0,0 +1 @@
+Introduce the ``importlib.metadata`` module with (provisional) support for reading metadata from third-party packages.

Python.framework/Resources

@@ -0,0 +1 @@
+Versions/Current/Resources