Issue #5319: New Py_FinalizeEx() API to exit with status 120 on failure

Doc/c-api/init.rst

@@ -25,7 +25,7 @@ Initializing and finalizing the interpreter
       triple: module; search; path
       single: PySys_SetArgv()
       single: PySys_SetArgvEx()
-      single: Py_Finalize()
+      single: Py_FinalizeEx()
 
    Initialize the Python interpreter.  In an application embedding  Python, this
    should be called before using any other Python/C API functions; with the
@@ -34,7 +34,7 @@ Initializing and finalizing the interpreter
    modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`.  It also initializes
    the module search path (``sys.path``). It does not set ``sys.argv``; use
    :c:func:`PySys_SetArgvEx` for that.  This is a no-op when called for a second time
-   (without calling :c:func:`Py_Finalize` first).  There is no return value; it is a
+   (without calling :c:func:`Py_FinalizeEx` first).  There is no return value; it is a
    fatal error if the initialization fails.
 
 
@@ -48,19 +48,20 @@ Initializing and finalizing the interpreter
 .. c:function:: int Py_IsInitialized()
 
    Return true (nonzero) when the Python interpreter has been initialized, false
-   (zero) if not.  After :c:func:`Py_Finalize` is called, this returns false until
+   (zero) if not.  After :c:func:`Py_FinalizeEx` is called, this returns false until
    :c:func:`Py_Initialize` is called again.
 
 
-.. c:function:: void Py_Finalize()
+.. c:function:: int Py_FinalizeEx()
 
    Undo all initializations made by :c:func:`Py_Initialize` and subsequent use of
    Python/C API functions, and destroy all sub-interpreters (see
    :c:func:`Py_NewInterpreter` below) that were created and not yet destroyed since
    the last call to :c:func:`Py_Initialize`.  Ideally, this frees all memory
    allocated by the Python interpreter.  This is a no-op when called for a second
-   time (without calling :c:func:`Py_Initialize` again first).  There is no return
-   value; errors during finalization are ignored.
+   time (without calling :c:func:`Py_Initialize` again first).  Normally the
+   return value is 0.  If there were errors during finalization
+   (flushing buffered data), -1 is returned.
 
    This function is provided for a number of reasons.  An embedding application
    might want to restart Python without having to restart the application itself.
@@ -79,7 +80,15 @@ Initializing and finalizing the interpreter
    freed.  Some memory allocated by extension modules may not be freed.  Some
    extensions may not work properly if their initialization routine is called more
    than once; this can happen if an application calls :c:func:`Py_Initialize` and
-   :c:func:`Py_Finalize` more than once.
+   :c:func:`Py_FinalizeEx` more than once.
+
+   .. versionadded:: 3.6
+
+
+.. c:function:: void Py_Finalize()
+
+   This is a backwards-compatible version of :c:func:`Py_FinalizeEx` that
+   disregards the return value.
 
 
 Process-wide parameters
@@ -107,7 +116,7 @@ Process-wide parameters
    Note that :data:`sys.stderr` always uses the "backslashreplace" error
    handler, regardless of this (or any other) setting.
 
-   If :c:func:`Py_Finalize` is called, this function will need to be called
+   If :c:func:`Py_FinalizeEx` is called, this function will need to be called
    again in order to affect subsequent calls to :c:func:`Py_Initialize`.
 
    Returns 0 if successful, a nonzero value on error (e.g. calling after the
@@ -918,7 +927,7 @@ using the following functions:
    entry.)
 
    .. index::
-      single: Py_Finalize()
+      single: Py_FinalizeEx()
       single: Py_Initialize()
 
    Extension modules are shared between (sub-)interpreters as follows: the first
@@ -928,7 +937,7 @@ using the following functions:
    and filled with the contents of this copy; the extension's ``init`` function is
    not called.  Note that this is different from what happens when an extension is
    imported after the interpreter has been completely re-initialized by calling
-   :c:func:`Py_Finalize` and :c:func:`Py_Initialize`; in that case, the extension's
+   :c:func:`Py_FinalizeEx` and :c:func:`Py_Initialize`; in that case, the extension's
    ``initmodule`` function *is* called again.
 
    .. index:: single: close() (in module os)
@@ -936,14 +945,14 @@ using the following functions:
 
 .. c:function:: void Py_EndInterpreter(PyThreadState *tstate)
 
-   .. index:: single: Py_Finalize()
+   .. index:: single: Py_FinalizeEx()
 
    Destroy the (sub-)interpreter represented by the given thread state. The given
    thread state must be the current thread state.  See the discussion of thread
    states below.  When the call returns, the current thread state is *NULL*.  All
    thread states associated with this interpreter are destroyed.  (The global
    interpreter lock must be held before calling this function and is still held
-   when it returns.)  :c:func:`Py_Finalize` will destroy all sub-interpreters that
+   when it returns.)  :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
    haven't been explicitly destroyed at that point.
 
 

Doc/c-api/intro.rst

@@ -578,9 +578,9 @@ Sometimes, it is desirable to "uninitialize" Python.  For instance,  the
 application may want to start over (make another call to
 :c:func:`Py_Initialize`) or the application is simply done with its  use of
 Python and wants to free memory allocated by Python.  This can be accomplished
-by calling :c:func:`Py_Finalize`.  The function :c:func:`Py_IsInitialized` returns
+by calling :c:func:`Py_FinalizeEx`.  The function :c:func:`Py_IsInitialized` returns
 true if Python is currently in the initialized state.  More information about
-these functions is given in a later chapter. Notice that :c:func:`Py_Finalize`
+these functions is given in a later chapter. Notice that :c:func:`Py_FinalizeEx`
 does *not* free all memory allocated by the Python interpreter, e.g. memory
 allocated by extension modules currently cannot be released.
 

Doc/c-api/sys.rst

@@ -212,20 +212,24 @@ Process Control
 .. c:function:: void Py_Exit(int status)
 
    .. index::
-      single: Py_Finalize()
+      single: Py_FinalizeEx()
       single: exit()
 
-   Exit the current process.  This calls :c:func:`Py_Finalize` and then calls the
-   standard C library function ``exit(status)``.
+   Exit the current process.  This calls :c:func:`Py_FinalizeEx` and then calls the
+   standard C library function ``exit(status)``.  If :c:func:`Py_FinalizeEx`
+   indicates an error, the exit status is set to 120.
+
+   .. versionchanged:: 3.6
+      Errors from finalization no longer ignored.
 
 
 .. c:function:: int Py_AtExit(void (*func) ())
 
    .. index::
-      single: Py_Finalize()
+      single: Py_FinalizeEx()
       single: cleanup functions
 
-   Register a cleanup function to be called by :c:func:`Py_Finalize`.  The cleanup
+   Register a cleanup function to be called by :c:func:`Py_FinalizeEx`.  The cleanup
    function will be called with no arguments and should return no value.  At most
    32 cleanup functions can be registered.  When the registration is successful,
    :c:func:`Py_AtExit` returns ``0``; on failure, it returns ``-1``.  The cleanup

Doc/extending/embedding.rst

@@ -67,7 +67,9 @@ perform some operation on a file. ::
        Py_Initialize();
        PyRun_SimpleString("from time import time,ctime\n"
                           "print('Today is', ctime(time()))\n");
-       Py_Finalize();
+       if (Py_FinalizeEx() < 0) {
+           exit(120);
+       }
        PyMem_RawFree(program);
        return 0;
    }
@@ -76,7 +78,7 @@ The :c:func:`Py_SetProgramName` function should be called before
 :c:func:`Py_Initialize` to inform the interpreter about paths to Python run-time
 libraries.  Next, the Python interpreter is initialized with
 :c:func:`Py_Initialize`, followed by the execution of a hard-coded Python script
-that prints the date and time.  Afterwards, the :c:func:`Py_Finalize` call shuts
+that prints the date and time.  Afterwards, the :c:func:`Py_FinalizeEx` call shuts
 the interpreter down, followed by the end of the program.  In a real program,
 you may want to get the Python script from another source, perhaps a text-editor
 routine, a file, or a database.  Getting the Python code from a file can better

Doc/includes/run-func.c

@@ -63,6 +63,8 @@ main(int argc, char *argv[])
         fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
         return 1;
     }
-    Py_Finalize();
+    if (Py_FinalizeEx() < 0) {
+        return 120;
+    }
     return 0;
 }

Doc/library/sys.rst

@@ -255,7 +255,7 @@ always available.
    (defaulting to zero), or another type of object.  If it is an integer, zero
    is considered "successful termination" and any nonzero value is considered
    "abnormal termination" by shells and the like.  Most systems require it to be
-   in the range 0-127, and produce undefined results otherwise.  Some systems
+   in the range 0--127, and produce undefined results otherwise.  Some systems
    have a convention for assigning specific meanings to specific exit codes, but
    these are generally underdeveloped; Unix programs generally use 2 for command
    line syntax errors and 1 for all other kind of errors.  If another type of
@@ -268,6 +268,11 @@ always available.
    the process when called from the main thread, and the exception is not
    intercepted.
 
+   .. versionchanged:: 3.6
+      If an error occurs in the cleanup after the Python interpreter
+      has caught :exc:`SystemExit` (such as an error flushing buffered data
+      in the standard streams), the exit status is changed to 120.
+
 
 .. data:: flags
 

Doc/whatsnew/3.6.rst

@@ -171,7 +171,8 @@ Optimizations
 Build and C API Changes
 =======================
 
-* None yet.
+* New :c:func:`Py_FinalizeEx` API which indicates if flushing buffered data
+  failed (:issue:`5319`).
 
 
 Deprecated
@@ -247,4 +248,5 @@ Changes in the Python API
 Changes in the C API
 --------------------
 
-* None yet.
+* :c:func:`Py_Exit` (and the main interpreter) now override the exit status
+  with 120 if flushing buffered data failed.  See :issue:`5319`.

Include/pylifecycle.h

@@ -27,6 +27,7 @@ PyAPI_FUNC(void) Py_InitializeEx(int);
 PyAPI_FUNC(void) _Py_InitializeEx_Private(int, int);
 #endif
 PyAPI_FUNC(void) Py_Finalize(void);
+PyAPI_FUNC(int) Py_FinalizeEx(void);
 PyAPI_FUNC(int) Py_IsInitialized(void);
 PyAPI_FUNC(PyThreadState *) Py_NewInterpreter(void);
 PyAPI_FUNC(void) Py_EndInterpreter(PyThreadState *);

Lib/test/test_cmd_line.py

@@ -348,8 +348,9 @@ class CmdLineTest(unittest.TestCase):
             test.support.SuppressCrashReport().__enter__()
             sys.stdout.write('x')
             os.close(sys.stdout.fileno())"""
-        rc, out, err = assert_python_ok('-c', code)
+        rc, out, err = assert_python_failure('-c', code)
         self.assertEqual(b'', out)
+        self.assertEqual(120, rc)
         self.assertRegex(err.decode('ascii', 'ignore'),
                          'Exception ignored in.*\nOSError: .*')
 

Misc/NEWS

@@ -10,6 +10,9 @@ Release date: XXXX-XX-XX
 Core and Builtins
 -----------------
 
+- Issue #5319: New Py_FinalizeEx() API allowing Python to set an exit status
+  of 120 on failure to flush buffered streams.
+
 - Issue #25485: telnetlib.Telnet is now a context manager.
 
 - Issue #24097: Fixed crash in object.__reduce__() if slot name is freed inside

Misc/SpecialBuilds.txt

@@ -65,9 +65,9 @@ sys.getobjects(max[, type])
     simply by virtue of being in the list.
 
 envvar PYTHONDUMPREFS
-    If this envvar exists, Py_Finalize() arranges to print a list of all
+    If this envvar exists, Py_FinalizeEx() arranges to print a list of all
     still-live heap objects.  This is printed twice, in different formats,
-    before and after Py_Finalize has cleaned up everything it can clean up.  The
+    before and after Py_FinalizeEx has cleaned up everything it can clean up.  The
     first output block produces the repr() of each object so is more
     informative; however, a lot of stuff destined to die is still alive then.
     The second output block is much harder to work with (repr() can't be invoked
@@ -144,7 +144,7 @@ Special gimmicks:
 
 envvar PYTHONMALLOCSTATS
     If this envvar exists, a report of pymalloc summary statistics is printed to
-    stderr whenever a new arena is allocated, and also by Py_Finalize().
+    stderr whenever a new arena is allocated, and also by Py_FinalizeEx().
 
 Changed in 2.5:  The number of extra bytes allocated is 4*sizeof(size_t).
 Before it was 16 on all boxes, reflecting that Python couldn't make use of
@@ -179,7 +179,7 @@ Each type object grows three new members:
      */
     int tp_maxalloc;
 
-Allocation and deallocation code keeps these counts up to date.  Py_Finalize()
+Allocation and deallocation code keeps these counts up to date.  Py_FinalizeEx()
 displays a summary of the info returned by sys.getcounts() (see below), along
 with assorted other special allocation counts (like the number of tuple
 allocations satisfied by a tuple free-list, the number of 1-character strings

Modules/main.c

@@ -654,7 +654,7 @@ Py_Main(int argc, wchar_t **argv)
             Py_SetProgramName(wbuf);
 
             /* Don't free wbuf, the argument to Py_SetProgramName
-             * must remain valid until the Py_Finalize is called.
+             * must remain valid until Py_FinalizeEx is called.
              */
         } else {
             Py_SetProgramName(argv[0]);
@@ -785,7 +785,11 @@ Py_Main(int argc, wchar_t **argv)
         sts = PyRun_AnyFileFlags(stdin, "<stdin>", &cf) != 0;
     }
 
-    Py_Finalize();
+    if (Py_FinalizeEx() < 0) {
+        /* Value unlikely to be confused with a non-error exit status or
+        other special meaning */
+        sts = 120;
+    }
 
 #ifdef __INSURE__
     /* Insure++ is a memory analysis tool that aids in discovering

PC/bdist_wininst/install.c

@@ -709,7 +709,7 @@ static int prepare_script_environment(HINSTANCE hPython)
  * 1 if the Python-dll does not export the functions we need
  * 2 if no install-script is specified in pathname
  * 3 if the install-script file could not be opened
- * the return value of PyRun_SimpleString() otherwise,
+ * the return value of PyRun_SimpleString() or Py_FinalizeEx() otherwise,
  * which is 0 if everything is ok, -1 if an exception had occurred
  * in the install-script.
  */
@@ -722,7 +722,7 @@ do_run_installscript(HINSTANCE hPython, char *pathname, int argc, char **argv)
     DECLPROC(hPython, void, Py_Initialize, (void));
     DECLPROC(hPython, int, PySys_SetArgv, (int, wchar_t **));
     DECLPROC(hPython, int, PyRun_SimpleString, (char *));
-    DECLPROC(hPython, void, Py_Finalize, (void));
+    DECLPROC(hPython, int, Py_FinalizeEx, (void));
     DECLPROC(hPython, PyObject *, Py_BuildValue, (char *, ...));
     DECLPROC(hPython, PyObject *, PyCFunction_New,
              (PyMethodDef *, PyObject *));
@@ -730,7 +730,7 @@ do_run_installscript(HINSTANCE hPython, char *pathname, int argc, char **argv)
     DECLPROC(hPython, PyObject *, PyErr_Format, (PyObject *, char *));
 
     if (!Py_Initialize || !PySys_SetArgv
-        || !PyRun_SimpleString || !Py_Finalize)
+        || !PyRun_SimpleString || !Py_FinalizeEx)
         return 1;
 
     if (!Py_BuildValue || !PyArg_ParseTuple || !PyErr_Format)
@@ -777,7 +777,9 @@ do_run_installscript(HINSTANCE hPython, char *pathname, int argc, char **argv)
             }
         }
     }
-    Py_Finalize();
+    if (Py_FinalizeEx() < 0) {
+        result = -1;
+    }
 
     close(fh);
     return result;
@@ -839,11 +841,11 @@ static int do_run_simple_script(HINSTANCE hPython, char *script)
     int rc;
     DECLPROC(hPython, void, Py_Initialize, (void));
     DECLPROC(hPython, void, Py_SetProgramName, (wchar_t *));
-    DECLPROC(hPython, void, Py_Finalize, (void));
+    DECLPROC(hPython, int, Py_FinalizeEx, (void));
     DECLPROC(hPython, int, PyRun_SimpleString, (char *));
     DECLPROC(hPython, void, PyErr_Print, (void));
 
-    if (!Py_Initialize || !Py_SetProgramName || !Py_Finalize ||
+    if (!Py_Initialize || !Py_SetProgramName || !Py_FinalizeEx ||
         !PyRun_SimpleString || !PyErr_Print)
         return -1;
 
@@ -853,7 +855,9 @@ static int do_run_simple_script(HINSTANCE hPython, char *script)
     rc = PyRun_SimpleString(script);
     if (rc)
         PyErr_Print();
-    Py_Finalize();
+    if (Py_FinalizeEx() < 0) {
+        rc = -1;
+    }
     return rc;
 }
 

PC/python3.def

@@ -648,6 +648,7 @@ EXPORTS
   Py_FatalError=python36.Py_FatalError
   Py_FileSystemDefaultEncoding=python36.Py_FileSystemDefaultEncoding DATA
   Py_Finalize=python36.Py_Finalize
+  Py_FinalizeEx=python36.Py_FinalizeEx
   Py_GetBuildInfo=python36.Py_GetBuildInfo
   Py_GetCompiler=python36.Py_GetCompiler
   Py_GetCopyright=python36.Py_GetCopyright

Python/frozenmain.c

@@ -99,7 +99,9 @@ Py_FrozenMain(int argc, char **argv)
 #ifdef MS_WINDOWS
     PyWinFreeze_ExeTerm();
 #endif
-    Py_Finalize();
+    if (Py_FinalizeEx() < 0) {
+        sts = 120;
+    }
 
 error:
     PyMem_RawFree(argv_copy);

Python/pylifecycle.c

@@ -154,8 +154,8 @@ Py_SetStandardStreamEncoding(const char *encoding, const char *errors)
     return 0;
 }
 
-/* Global initializations.  Can be undone by Py_Finalize().  Don't
-   call this twice without an intervening Py_Finalize() call.  When
+/* Global initializations.  Can be undone by Py_FinalizeEx().  Don't
+   call this twice without an intervening Py_FinalizeEx() call.  When
    initializations fail, a fatal error is issued and the function does
    not return.  On return, the first thread and interpreter state have
    been created.
@@ -327,11 +327,11 @@ _Py_InitializeEx_Private(int install_sigs, int install_importlib)
     (void) PyThreadState_Swap(tstate);
 
 #ifdef WITH_THREAD
-    /* We can't call _PyEval_FiniThreads() in Py_Finalize because
+    /* We can't call _PyEval_FiniThreads() in Py_FinalizeEx because
        destroying the GIL might fail when it is being referenced from
        another running thread (see issue #9901).
        Instead we destroy the previously created GIL here, which ensures
-       that we can call Py_Initialize / Py_Finalize multiple times. */
+       that we can call Py_Initialize / Py_FinalizeEx multiple times. */
     _PyEval_FiniThreads();
 
     /* Auto-thread-state API */
@@ -477,28 +477,35 @@ file_is_closed(PyObject *fobj)
     return r > 0;
 }
 
-static void
+static int
 flush_std_files(void)
 {
     PyObject *fout = _PySys_GetObjectId(&PyId_stdout);
     PyObject *ferr = _PySys_GetObjectId(&PyId_stderr);
     PyObject *tmp;
+    int status = 0;
 
     if (fout != NULL && fout != Py_None && !file_is_closed(fout)) {
         tmp = _PyObject_CallMethodId(fout, &PyId_flush, "");
-        if (tmp == NULL)
+        if (tmp == NULL) {
             PyErr_WriteUnraisable(fout);
+            status = -1;
+        }
         else
             Py_DECREF(tmp);
     }
 
     if (ferr != NULL && ferr != Py_None && !file_is_closed(ferr)) {
         tmp = _PyObject_CallMethodId(ferr, &PyId_flush, "");
-        if (tmp == NULL)
+        if (tmp == NULL) {
             PyErr_Clear();
+            status = -1;
+        }
         else
             Py_DECREF(tmp);
     }
+
+    return status;
 }
 
 /* Undo the effect of Py_Initialize().
@@ -515,14 +522,15 @@ flush_std_files(void)
 
 */
 
-void
-Py_Finalize(void)
+int
+Py_FinalizeEx(void)
 {
     PyInterpreterState *interp;
     PyThreadState *tstate;
+    int status = 0;
 
     if (!initialized)
-        return;
+        return status;
 
     wait_for_thread_shutdown();
 
@@ -547,7 +555,9 @@ Py_Finalize(void)
     initialized = 0;
 
     /* Flush sys.stdout and sys.stderr */
-    flush_std_files();
+    if (flush_std_files() < 0) {
+        status = -1;
+    }
 
     /* Disable signal handling */
     PyOS_FiniInterrupts();
@@ -576,7 +586,9 @@ Py_Finalize(void)
     PyImport_Cleanup();
 
     /* Flush sys.stdout and sys.stderr (again, in case more was printed) */
-    flush_std_files();
+    if (flush_std_files() < 0) {
+        status = -1;
+    }
 
     /* Collect final garbage.  This disposes of cycles created by
      * class definitions, for example.
@@ -696,6 +708,13 @@ Py_Finalize(void)
 #endif
 
     call_ll_exitfuncs();
+    return status;
+}
+
+void
+Py_Finalize(void)
+{
+    Py_FinalizeEx();
 }
 
 /* Create and initialize a new interpreter and thread, and return the
@@ -803,7 +822,7 @@ handle_error:
    frames, and that it is its interpreter's only remaining thread.
    It is a fatal error to violate these constraints.
 
-   (Py_Finalize() doesn't have these constraints -- it zaps
+   (Py_FinalizeEx() doesn't have these constraints -- it zaps
    everything, regardless.)
 
    Locking: as above.
@@ -1016,7 +1035,8 @@ create_stdio(PyObject* io,
         mode = "rb";
     buf = _PyObject_CallMethodId(io, &PyId_open, "isiOOOi",
                                  fd, mode, buffering,
-                                 Py_None, Py_None, Py_None, 0);
+                                 Py_None, Py_None, /* encoding, errors */
+                                 Py_None, 0); /* newline, closefd */
     if (buf == NULL)
         goto error;
 
@@ -1450,7 +1470,9 @@ call_ll_exitfuncs(void)
 void
 Py_Exit(int sts)
 {
-    Py_Finalize();
+    if (Py_FinalizeEx() < 0) {
+        sts = 120;
+    }
 
     exit(sts);
 }

Python/pystate.c

@@ -686,7 +686,7 @@ PyThreadState_IsCurrent(PyThreadState *tstate)
 }
 
 /* Internal initialization/finalization functions called by
-   Py_Initialize/Py_Finalize
+   Py_Initialize/Py_FinalizeEx
 */
 void
 _PyGILState_Init(PyInterpreterState *i, PyThreadState *t)

Tools/scripts/combinerefs.py

@@ -6,7 +6,7 @@ combinerefs path
 A helper for analyzing PYTHONDUMPREFS output.
 
 When the PYTHONDUMPREFS envar is set in a debug build, at Python shutdown
-time Py_Finalize() prints the list of all live objects twice:  first it
+time Py_FinalizeEx() prints the list of all live objects twice:  first it
 prints the repr() of each object while the interpreter is still fully intact.
 After cleaning up everything it can, it prints all remaining live objects
 again, but the second time just prints their addresses, refcounts, and type
@@ -41,7 +41,7 @@ CAUTION:  If object is a container type, it may not actually contain all the
 objects shown in the repr:  the repr was captured from the first output block,
 and some of the containees may have been released since then.  For example,
 it's common for the line showing the dict of interned strings to display
-strings that no longer exist at the end of Py_Finalize; this can be recognized
+strings that no longer exist at the end of Py_FinalizeEx; this can be recognized
 (albeit painfully) because such containees don't have a line of their own.
 
 The objects are listed in allocation order, with most-recently allocated