merge with upstream 3.3 branch

Doc/library/collections.abc.rst

@@ -41,29 +41,35 @@ ABC                        Inherits from          Abstract Methods        Mixin
 :class:`Sized`                                    ``__len__``
 :class:`Callable`                                 ``__call__``
 
-:class:`Sequence`          :class:`Sized`,        ``__getitem__``         ``__contains__``, ``__iter__``, ``__reversed__``,
-                           :class:`Iterable`,                             ``index``, and ``count``
+:class:`Sequence`          :class:`Sized`,        ``__getitem__``,        ``__contains__``, ``__iter__``, ``__reversed__``,
+                           :class:`Iterable`,     ``__len__``             ``index``, and ``count``
                            :class:`Container`
 
-:class:`MutableSequence`   :class:`Sequence`      ``__setitem__``,        Inherited :class:`Sequence` methods and
-                                                  ``__delitem__``,        ``append``, ``reverse``, ``extend``, ``pop``,
-                                                  ``insert``              ``remove``, ``clear``, and ``__iadd__``
-
-:class:`Set`               :class:`Sized`,                                ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``,
-                           :class:`Iterable`,                             ``__gt__``, ``__ge__``, ``__and__``, ``__or__``,
-                           :class:`Container`                             ``__sub__``, ``__xor__``, and ``isdisjoint``
-
-:class:`MutableSet`        :class:`Set`           ``add``,                Inherited :class:`Set` methods and
-                                                  ``discard``             ``clear``, ``pop``, ``remove``, ``__ior__``,
-                                                                          ``__iand__``, ``__ixor__``, and ``__isub__``
-
-:class:`Mapping`           :class:`Sized`,        ``__getitem__``         ``__contains__``, ``keys``, ``items``, ``values``,
-                           :class:`Iterable`,                             ``get``, ``__eq__``, and ``__ne__``
-                           :class:`Container`
-
-:class:`MutableMapping`    :class:`Mapping`       ``__setitem__``,        Inherited :class:`Mapping` methods and
-                                                  ``__delitem__``         ``pop``, ``popitem``, ``clear``, ``update``,
-                                                                          and ``setdefault``
+:class:`MutableSequence`   :class:`Sequence`      ``__getitem__``,        Inherited :class:`Sequence` methods and
+                                                  ``__setitem__``,        ``append``, ``reverse``, ``extend``, ``pop``,
+                                                  ``__delitem__``,        ``remove``, and ``__iadd__``
+                                                  ``__len__``,
+                                                  ``insert``
+
+:class:`Set`               :class:`Sized`,        ``__contains__``,       ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``,
+                           :class:`Iterable`,     ``__iter__``,           ``__gt__``, ``__ge__``, ``__and__``, ``__or__``,
+                           :class:`Container`     ``__len__``             ``__sub__``, ``__xor__``, and ``isdisjoint``
+
+:class:`MutableSet`        :class:`Set`           ``__contains__``,       Inherited :class:`Set` methods and
+                                                  ``__iter__``,           ``clear``, ``pop``, ``remove``, ``__ior__``,
+                                                  ``__len__``,            ``__iand__``, ``__ixor__``, and ``__isub__``
+                                                  ``add``,
+                                                  ``discard``
+
+:class:`Mapping`           :class:`Sized`,        ``__getitem__``,        ``__contains__``, ``keys``, ``items``, ``values``,
+                           :class:`Iterable`,     ``__iter__``,           ``get``, ``__eq__``, and ``__ne__``
+                           :class:`Container`     ``__len__``
+
+:class:`MutableMapping`    :class:`Mapping`       ``__getitem__``,        Inherited :class:`Mapping` methods and
+                                                  ``__setitem__``,        ``pop``, ``popitem``, ``clear``, ``update``,
+                                                  ``__delitem__``,        and ``setdefault``
+                                                  ``__iter__``,
+                                                  ``__len__``
 
 
 :class:`MappingView`       :class:`Sized`                                 ``__len__``

Doc/library/stdtypes.rst

@@ -1627,6 +1627,8 @@ expression support in the :mod:`re` module).
    Return true if the string is a valid identifier according to the language
    definition, section :ref:`identifiers`.
 
+   Use :func:`keyword.iskeyword` to test for reserved identifiers such as
+   :keyword:`def` and :keyword:`class`.
 
 .. method:: str.islower()
 

Doc/library/subprocess.rst

@@ -356,7 +356,7 @@ are able to handle the less common cases not covered by the convenience
 functions.
 
 
-.. class:: Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, \
+.. class:: Popen(args, bufsize=-1, executable=None, stdin=None, stdout=None, \
                  stderr=None, preexec_fn=None, close_fds=True, shell=False, \
                  cwd=None, env=None, universal_newlines=False, \
                  startupinfo=None, creationflags=0, restore_signals=True, \
@@ -428,17 +428,20 @@ functions.
       untrusted input.  See the warning under :ref:`frequently-used-arguments`
       for details.
 
-   *bufsize*, if given, has the same meaning as the corresponding argument to the
-   built-in open() function: :const:`0` means unbuffered, :const:`1` means line
-   buffered, any other positive value means use a buffer of (approximately) that
-   size.  A negative *bufsize* means to use the system default, which usually means
-   fully buffered.  The default value for *bufsize* is :const:`0` (unbuffered).
+   *bufsize* will be supplied as the corresponding argument to the :meth:`io.open`
+   function when creating the stdin/stdout/stderr pipe file objects:
+   :const:`0` means unbuffered (read and write are one system call and can return short),
+   :const:`1` means line buffered, any other positive value means use a buffer of
+   approximately that size.  A negative bufsize (the default) means
+   the system default of io.DEFAULT_BUFFER_SIZE will be used.
 
-   .. note::
+   .. versionchanged:: 3.2.4, 3.3.1
 
-      If you experience performance issues, it is recommended that you try to
-      enable buffering by setting *bufsize* to either -1 or a large enough
-      positive value (such as 4096).
+      *bufsize* now defaults to -1 to enable buffering by default to match the
+      behavior that most code expects.  In 3.2.0 through 3.2.3 and 3.3.0 it
+      incorrectly defaulted to :const:`0` which was unbuffered and allowed
+      short reads.  This was unintentional and did not match the behavior of
+      Python 2 as most code expected.
 
    The *executable* argument specifies a replacement program to execute.   It
    is very seldom needed.  When ``shell=False``, *executable* replaces the

Lib/collections/abc.py

@@ -90,6 +90,7 @@ class Iterator(Iterable):
 
     @abstractmethod
     def __next__(self):
+        'Return the next item from the iterator. When exhausted, raise StopIteration'
         raise StopIteration
 
     def __iter__(self):
@@ -230,6 +231,7 @@ class Set(Sized, Iterable, Container):
         return self._from_iterable(value for value in other if value in self)
 
     def isdisjoint(self, other):
+        'Return True if two sets have a null intersection.'
         for value in other:
             if value in self:
                 return False
@@ -292,6 +294,16 @@ Set.register(frozenset)
 
 
 class MutableSet(Set):
+    """A mutable set is a finite, iterable container.
+
+    This class provides concrete generic implementations of all
+    methods except for __contains__, __iter__, __len__,
+    add(), and discard().
+
+    To override the comparisons (presumably for speed, as the
+    semantics are fixed), all you have to do is redefine __le__ and
+    then the other operations will automatically follow suit.
+    """
 
     __slots__ = ()
 
@@ -370,11 +382,20 @@ class Mapping(Sized, Iterable, Container):
 
     __slots__ = ()
 
+    """A Mapping is a generic container for associating key/value
+    pairs.
+
+    This class provides concrete generic implementations of all
+    methods except for __getitem__, __iter__, and __len__.
+
+    """
+
     @abstractmethod
     def __getitem__(self, key):
         raise KeyError
 
     def get(self, key, default=None):
+        'D.get(k[,d]) -> D[k] if k in D, else d.  d defaults to None.'
         try:
             return self[key]
         except KeyError:
@@ -389,12 +410,15 @@ class Mapping(Sized, Iterable, Container):
             return True
 
     def keys(self):
+        "D.keys() -> a set-like object providing a view on D's keys"
         return KeysView(self)
 
     def items(self):
+        "D.items() -> a set-like object providing a view on D's items"
         return ItemsView(self)
 
     def values(self):
+        "D.values() -> an object providing a view on D's values"
         return ValuesView(self)
 
     def __eq__(self, other):
@@ -477,6 +501,15 @@ class MutableMapping(Mapping):
 
     __slots__ = ()
 
+    """A MutableMapping is a generic container for associating
+    key/value pairs.
+
+    This class provides concrete generic implementations of all
+    methods except for __getitem__, __setitem__, __delitem__,
+    __iter__, and __len__.
+
+    """
+
     @abstractmethod
     def __setitem__(self, key, value):
         raise KeyError
@@ -488,6 +521,9 @@ class MutableMapping(Mapping):
     __marker = object()
 
     def pop(self, key, default=__marker):
+        '''D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
+          If key is not found, d is returned if given, otherwise KeyError is raised.
+        '''
         try:
             value = self[key]
         except KeyError:
@@ -499,6 +535,9 @@ class MutableMapping(Mapping):
             return value
 
     def popitem(self):
+        '''D.popitem() -> (k, v), remove and return some (key, value) pair
+           as a 2-tuple; but raise KeyError if D is empty.
+        '''
         try:
             key = next(iter(self))
         except StopIteration:
@@ -508,6 +547,7 @@ class MutableMapping(Mapping):
         return key, value
 
     def clear(self):
+        'D.clear() -> None.  Remove all items from D.'
         try:
             while True:
                 self.popitem()
@@ -515,6 +555,11 @@ class MutableMapping(Mapping):
             pass
 
     def update(*args, **kwds):
+        ''' D.update([E, ]**F) -> None.  Update D from mapping/iterable E and F.
+            If E present and has a .keys() method, does:     for k in E: D[k] = E[k]
+            If E present and lacks .keys() method, does:     for (k, v) in E: D[k] = v
+            In either case, this is followed by: for k, v in F.items(): D[k] = v
+        '''
         if len(args) > 2:
             raise TypeError("update() takes at most 2 positional "
                             "arguments ({} given)".format(len(args)))
@@ -536,6 +581,7 @@ class MutableMapping(Mapping):
             self[key] = value
 
     def setdefault(self, key, default=None):
+        'D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D'
         try:
             return self[key]
         except KeyError:
@@ -583,12 +629,16 @@ class Sequence(Sized, Iterable, Container):
             yield self[i]
 
     def index(self, value):
+        '''S.index(value) -> integer -- return first index of value.
+           Raises ValueError if the value is not present.
+        '''
         for i, v in enumerate(self):
             if v == value:
                 return i
         raise ValueError
 
     def count(self, value):
+        'S.count(value) -> integer -- return number of occurrences of value'
         return sum(1 for v in self if v == value)
 
 Sequence.register(tuple)
@@ -613,6 +663,13 @@ class MutableSequence(Sequence):
 
     __slots__ = ()
 
+    """All the operations on a read-only sequence.
+
+    Concrete subclasses must provide __new__ or __init__,
+    __getitem__, __setitem__, __delitem__, __len__, and insert().
+
+    """
+
     @abstractmethod
     def __setitem__(self, index, value):
         raise IndexError
@@ -623,12 +680,15 @@ class MutableSequence(Sequence):
 
     @abstractmethod
     def insert(self, index, value):
+        'S.insert(index, value) -- insert value before index'
         raise IndexError
 
     def append(self, value):
+        'S.append(value) -- append value to the end of the sequence'
         self.insert(len(self), value)
 
     def clear(self):
+        'S.clear() -> None -- remove all items from S'
         try:
             while True:
                 self.pop()
@@ -636,20 +696,28 @@ class MutableSequence(Sequence):
             pass
 
     def reverse(self):
+        'S.reverse() -- reverse *IN PLACE*'
         n = len(self)
         for i in range(n//2):
             self[i], self[n-i-1] = self[n-i-1], self[i]
 
     def extend(self, values):
+        'S.extend(iterable) -- extend sequence by appending elements from the iterable'
         for v in values:
             self.append(v)
 
     def pop(self, index=-1):
+        '''S.pop([index]) -> item -- remove and return item at index (default last).
+           Raise IndexError if list is empty or index is out of range.
+        '''
         v = self[index]
         del self[index]
         return v
 
     def remove(self, value):
+        '''S.remove(value) -- remove first occurrence of value.
+           Raise ValueError if the value is not present.
+        '''
         del self[self.index(value)]
 
     def __iadd__(self, values):

Lib/subprocess.py

@@ -25,7 +25,7 @@ Using the subprocess module
 ===========================
 This module defines one class called Popen:
 
-class Popen(args, bufsize=0, executable=None,
+class Popen(args, bufsize=-1, executable=None,
             stdin=None, stdout=None, stderr=None,
             preexec_fn=None, close_fds=True, shell=False,
             cwd=None, env=None, universal_newlines=False,
@@ -56,12 +56,12 @@ not all MS Windows applications interpret the command line the same
 way: The list2cmdline is designed for applications using the same
 rules as the MS C runtime.
 
-bufsize, if given, has the same meaning as the corresponding argument
-to the built-in open() function: 0 means unbuffered, 1 means line
-buffered, any other positive value means use a buffer of
-(approximately) that size.  A negative bufsize means to use the system
-default, which usually means fully buffered.  The default value for
-bufsize is 0 (unbuffered).
+bufsize will be supplied as the corresponding argument to the io.open()
+function when creating the stdin/stdout/stderr pipe file objects:
+0 means unbuffered (read & write are one system call and can return short),
+1 means line buffered, any other positive value means use a buffer of
+approximately that size.  A negative bufsize, the default, means the system
+default of io.DEFAULT_BUFFER_SIZE will be used.
 
 stdin, stdout and stderr specify the executed programs' standard
 input, standard output and standard error file handles, respectively.
@@ -711,7 +711,7 @@ _PLATFORM_DEFAULT_CLOSE_FDS = object()
 
 
 class Popen(object):
-    def __init__(self, args, bufsize=0, executable=None,
+    def __init__(self, args, bufsize=-1, executable=None,
                  stdin=None, stdout=None, stderr=None,
                  preexec_fn=None, close_fds=_PLATFORM_DEFAULT_CLOSE_FDS,
                  shell=False, cwd=None, env=None, universal_newlines=False,
@@ -725,7 +725,7 @@ class Popen(object):
         self._input = None
         self._communication_started = False
         if bufsize is None:
-            bufsize = 0  # Restore default
+            bufsize = -1  # Restore default
         if not isinstance(bufsize, int):
             raise TypeError("bufsize must be an integer")
 

Lib/test/test_queue.py

@@ -46,6 +46,9 @@ class _TriggerThread(threading.Thread):
 
 class BlockingTestMixin:
 
+    def tearDown(self):
+        self.t = None
+
     def do_blocking_test(self, block_func, block_args, trigger_func, trigger_args):
         self.t = _TriggerThread(trigger_func, trigger_args)
         self.t.start()
@@ -260,7 +263,7 @@ class FailingQueue(queue.Queue):
             raise FailingQueueException("You Lose")
         return queue.Queue._get(self)
 
-class FailingQueueTest(unittest.TestCase, BlockingTestMixin):
+class FailingQueueTest(BlockingTestMixin, unittest.TestCase):
 
     def failing_queue_test(self, q):
         if q.qsize():

Lib/test/test_subprocess.py

@@ -82,6 +82,34 @@ class PopenExecuteChildRaises(subprocess.Popen):
 
 class ProcessTestCase(BaseTestCase):
 
+    def test_io_buffered_by_default(self):
+        p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"],
+                             stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+                             stderr=subprocess.PIPE)
+        try:
+            self.assertIsInstance(p.stdin, io.BufferedIOBase)
+            self.assertIsInstance(p.stdout, io.BufferedIOBase)
+            self.assertIsInstance(p.stderr, io.BufferedIOBase)
+        finally:
+            p.stdin.close()
+            p.stdout.close()
+            p.stderr.close()
+            p.wait()
+
+    def test_io_unbuffered_works(self):
+        p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"],
+                             stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+                             stderr=subprocess.PIPE, bufsize=0)
+        try:
+            self.assertIsInstance(p.stdin, io.RawIOBase)
+            self.assertIsInstance(p.stdout, io.RawIOBase)
+            self.assertIsInstance(p.stderr, io.RawIOBase)
+        finally:
+            p.stdin.close()
+            p.stdout.close()
+            p.stderr.close()
+            p.wait()
+
     def test_call_seq(self):
         # call() function with sequence argument
         rc = subprocess.call([sys.executable, "-c",

Lib/unittest/mock.py

@@ -948,9 +948,6 @@ class Mock(CallableMixin, NonCallableMock):
       the next value from the iterable. If any of the members of the iterable
       are exceptions they will be raised instead of returned.
 
-      If `side_effect` is an iterable then each call to the mock will return
-      the next value from the iterable.
-
     * `return_value`: The value returned when the mock is called. By default
       this is a new Mock (created on first access). See the
       `return_value` attribute.

Lib/webbrowser.py

@@ -468,9 +468,13 @@ def register_X_browsers():
     if "KDE_FULL_SESSION" in os.environ and _iscommand("kfmclient"):
         register("kfmclient", Konqueror, Konqueror("kfmclient"))
 
+    if _iscommand("x-www-browser"):
+        register("x-www-browser", None, BackgroundBrowser("x-www-browser"))
+
     # The Mozilla/Netscape browsers
     for browser in ("mozilla-firefox", "firefox",
                     "mozilla-firebird", "firebird",
+                    "iceweasel", "iceape",
                     "seamonkey", "mozilla", "netscape"):
         if _iscommand(browser):
             register(browser, None, Mozilla(browser))
@@ -513,6 +517,8 @@ if os.environ.get("DISPLAY"):
 
 # Also try console browsers
 if os.environ.get("TERM"):
+    if _iscommand("www-browser"):
+        register("www-browser", None, GenericBrowser("www-browser"))
     # The Links/elinks browsers <http://artax.karlin.mff.cuni.cz/~mikulas/links/>
     if _iscommand("links"):
         register("links", None, GenericBrowser("links"))

Misc/NEWS

@@ -196,6 +196,14 @@ Core and Builtins
 Library
 -------
 
+- Issue #17536: Add to webbrowser's browser list: www-browser, x-www-browser,
+  iceweasel, iceape.
+
+- Issue #17488: Change the subprocess.Popen bufsize parameter default value
+  from unbuffered (0) to buffering (-1) to match the behavior existing code
+  expects and match the behavior of the subprocess module in Python 2 to avoid
+  introducing hard to track down bugs.
+
 - Issue #17521: Corrected non-enabling of logger following two calls to
   fileConfig().
 
@@ -803,6 +811,8 @@ Tests
 Build
 -----
 
+- Issue #17425: Build with openssl 1.0.1d on Windows.
+
 - Issue #16754: Fix the incorrect shared library extension on linux. Introduce
   two makefile macros SHLIB_SUFFIX and EXT_SUFFIX. SO now has the value of
   SHLIB_SUFFIX again (as in 2.x and 3.1). The SO macro is removed in 3.4.

Objects/unicodeobject.c

@@ -11493,7 +11493,10 @@ PyDoc_STRVAR(isidentifier__doc__,
              "S.isidentifier() -> bool\n\
 \n\
 Return True if S is a valid identifier according\n\
-to the language definition.");
+to the language definition.\n\
+\n\
+Use keyword.iskeyword() to test for reserved identifiers\n\
+such as \"def\" and \"class\".\n");
 
 static PyObject*
 unicode_isidentifier(PyObject *self)

PC/VC6/readme.txt

@@ -153,9 +153,9 @@ _ssl
 
     Unpack into the "dist" directory, retaining the folder name from
     the archive - for example, the latest stable OpenSSL will install as
-        dist/openssl-1.0.1c
+        dist/openssl-1.0.1d
 
-    You need to use version 1.0.1c of OpenSSL.
+    You need to use version 1.0.1d of OpenSSL.
 
     You can install the NASM assembler from
         http://www.nasm.us/

PC/VS8.0/pyproject.vsprops

@@ -58,7 +58,7 @@
 	/>
 	<UserMacro
 		Name="opensslDir"
-		Value="$(externalsDir)\openssl-1.0.1c"
+		Value="$(externalsDir)\openssl-1.0.1d"
 	/>
 	<UserMacro
 		Name="tcltkDir"

PC/VS9.0/pyproject.vsprops

@@ -62,7 +62,7 @@
 	/>
 	<UserMacro
 		Name="opensslDir"
-		Value="$(externalsDir)\openssl-1.0.1c"
+		Value="$(externalsDir)\openssl-1.0.1d"
 	/>
 	<UserMacro
 		Name="tcltkDir"

PC/bdist_wininst/extract.c

@@ -127,7 +127,7 @@ char *map_new_file(DWORD flags, char *filename,
 
     CloseHandle(hFile);
 
-    if (hFileMapping == INVALID_HANDLE_VALUE) {
+    if (hFileMapping == NULL) {
         if (notify)
             notify(SYSTEM_ERROR,
                    "CreateFileMapping (%s)", filename);

PC/bdist_wininst/install.c

@@ -1019,7 +1019,7 @@ static char *MapExistingFile(char *pathname, DWORD *psize)
                                       NULL, PAGE_READONLY, 0, 0, NULL);
     CloseHandle(hFile);
 
-    if (hFileMapping == INVALID_HANDLE_VALUE)
+    if (hFileMapping == NULL)
         return NULL;
 
     data = MapViewOfFile(hFileMapping,

PCbuild/pyproject.props

@@ -20,7 +20,7 @@
     <sqlite3Dir>$(externalsDir)\sqlite-3.7.12</sqlite3Dir>
     <bz2Dir>$(externalsDir)\bzip2-1.0.6</bz2Dir>
     <lzmaDir>$(externalsDir)\xz-5.0.3</lzmaDir>
-    <opensslDir>$(externalsDir)\openssl-1.0.1c</opensslDir>
+    <opensslDir>$(externalsDir)\openssl-1.0.1d</opensslDir>
     <tcltkDir>$(externalsDir)\tcltk</tcltkDir>
     <tcltk64Dir>$(externalsDir)\tcltk64</tcltk64Dir>
     <tcltkLib>$(tcltkDir)\lib\tcl85.lib;$(tcltkDir)\lib\tk85.lib</tcltkLib>

PCbuild/readme.txt

@@ -142,7 +142,7 @@ _ssl
 
     Get the source code through
 
-    svn export http://svn.python.org/projects/external/openssl-1.0.1c
+    svn export http://svn.python.org/projects/external/openssl-1.0.1d
 
     ** NOTE: if you use the Tools\buildbot\external(-amd64).bat approach for
     obtaining external sources then you don't need to manually get the source

Python/dynload_win.c

@@ -235,7 +235,7 @@ dl_funcptr _PyImport_GetDynLoadWindows(const char *shortname,
                            SUBLANG_DEFAULT),
                            /* Default language */
                 theInfo, /* the buffer */
-                sizeof(theInfo), /* the buffer size */
+                sizeof(theInfo) / sizeof(wchar_t), /* size in wchars */
                 NULL); /* no additional format args. */
 
             /* Problem: could not get the error message.

Tools/buildbot/external-common.bat

@@ -14,7 +14,7 @@ cd ..
 @rem if exist tk8.4.16 rd /s/q tk8.4.16
 @rem if exist tk-8.4.18.1 rd /s/q tk-8.4.18.1
 @rem if exist db-4.4.20 rd /s/q db-4.4.20
-@rem if exist openssl-1.0.1c rd /s/q openssl-1.0.1c
+@rem if exist openssl-1.0.1d rd /s/q openssl-1.0.1d
 @rem if exist sqlite-3.7.12 rd /s/q sqlite-3.7.12    
 
 @rem bzip
@@ -24,9 +24,9 @@ if not exist bzip2-1.0.6 (
 )
 
 @rem OpenSSL
-if not exist openssl-1.0.1c (
-    rd /s/q openssl-1.0.0j
-    svn export http://svn.python.org/projects/external/openssl-1.0.1c
+if not exist openssl-1.0.1d (
+    rd /s/q openssl-1.0.1c
+    svn export http://svn.python.org/projects/external/openssl-1.0.1d
 )
 
 @rem tcl/tk