The multiprocessing.resource_tracker replaces the multiprocessing.semaphore_tracker module. Other than semaphores, resource_tracker also tracks shared_memory segments. Patch by Pierre Glaser.
* Added tests for shared_memory submodule.
* Added tests for ShareableList.
* Fix bug in allocationn size during creation of empty ShareableList illuminated by existing test run on Linux.
* Initial set of docs for shared_memory module.
* Added docs for ShareableList, added doctree entry for shared_memory submodule, name refactoring for greater clarity.
* Added examples to SharedMemoryManager docs, for ease of documentation switched away from exclusively registered functions to some explicit methods on SharedMemoryManager.
* Wording tweaks to docs.
* Fix test failures on Windows.
* Added tests around SharedMemoryManager.
* Documentation tweaks.
* Fix inappropriate test on Windows.
* Further documentation tweaks.
* Fix bare exception.
* Removed __copyright__.
* Fixed typo in doc, removed comment.
* Updated SharedMemoryManager preliminary tests to reflect change of not supporting all registered functions on SyncManager.
* Added Sphinx doctest run controls.
* CloseHandle should be in a finally block in case MapViewOfFile fails.
* Missed opportunity to use with statement.
* Switch to self.addCleanup to spare long try/finally blocks and save one indentation, change to use decorator to skip test instead.
* Simplify the posixshmem extension module.
Provide shm_open() and shm_unlink() functions. Move other
functionality into the shared_memory.py module.
* Added to doc around size parameter of SharedMemory.
* Changed PosixSharedMemory.size to use os.fstat.
* Change SharedMemory.buf to a read-only property as well as NamedSharedMemory.size.
* Marked as provisional per PEP411 in docstring.
* Changed SharedMemoryTracker to be private.
* Removed registered Proxy Objects from SharedMemoryManager.
* Removed shareable_wrap().
* Removed shareable_wrap() and dangling references to it.
* For consistency added __reduce__ to key classes.
* Fix for potential race condition on Windows for O_CREX.
* Remove unused imports.
* Update access to kernel32 on Windows per feedback from eryksun.
* Moved kernel32 calls to _winapi.
* Removed ShareableList.copy as redundant.
* Changes to _winapi use from eryksun feedback.
* Adopt simpler SharedMemory API, collapsing PosixSharedMemory and WindowsNamedSharedMemory into one.
* Fix missing docstring on class, add test for ignoring size when attaching.
* Moved SharedMemoryManager to managers module, tweak to fragile test.
* Tweak to exception in OpenFileMapping suggested by eryksun.
* Mark a few dangling bits as private as suggested by Giampaolo.
Changes in this commit:
1. Use a _strong_ reference between the Pool and associated iterators
2. Rework PR #8450 to eliminate a cycle in the Pool.
There is no test in this commit because any test that automatically tests this behaviour needs to eliminate the pool before joining the pool to check that the pool object is garbaged collected/does not hang. But doing this will potentially leak threads and processes (see https://bugs.python.org/issue35413).
Replace time.time() with time.monotonic() in tests to measure time
delta.
test_zipfile64: display progress every minute (60 secs) rather than
every 5 minutes (5*60 seconds).
Join 3 pools in these tests:
* test.test_multiprocessing_spawn.WithProcessesTestPool.test_context
* test.test_multiprocessing_spawn.WithProcessesTestPool.test_traceback
Fix WithThreadsTestPool.test_wrapped_exception()
of test_multiprocessing_fork: join the pool.
WithThreadsTestPool.test_del_pool() is now also decorated
with @support.reap_threads.
Tests involving sending signals to the semaphore_tracker will not fail anymore due to
the fact that running the test suite with -Werror propagates warnings as errors.
Fix a missing assertion when the semaphore_tracker is expected to die.
Fix a reference issue inside multiprocessing.Pool that caused the pool to remain alive if it was deleted without being closed or terminated explicitly.
Fail `test_semaphore_tracker_sigint` if no warnings are expected and one is received.
Fix race condition when the child receives SIGINT before it can register signal handlers for it.
The race condition occurs when the parent calls
`_semaphore_tracker.ensure_running()` (which in turn spawns the
semaphore_tracker using `_posixsubprocess.fork_exec`), the child
registers the signal handlers and the parent tries to kill the child.
What seem to happen is that in some slow systems, the parent sends the
signal to kill the child before the child protects against the signal.
Multiprocessing test_timeout() now accepts a delta of 100 ms instead
of just 50 ms, since the test failed with 135.8 ms instead of the
expected 200 ms.
Fix test_forkserver_sigkill() of test_multiprocessing_forkserver:
give more time to the first child process to complete, double the
sleep in the parent process.
Reduce also the child process sleep from 1000 ms to 500 ms, to not change
the total duration of the test.
When hunting memory leaks using -R 3:3, test_imap_unordered() of
test_multiprocessing leaks randomly a few memory blocks. It is a
false alarm: when testing using -R 3:20 for example, no leak is
detected.
Modify test_imap_unordered() to be closer to test_imap():
* Only test 10 numbers instead of 1000: it's a pool of 4 processes, so
10 is enough to test at least one number per process
* Use chunksize=100 instead of chunksize=53 to mimick test_imap()
test_mymanager_context() now also accepts -SIGTERM as an expected
exitcode for the manager process. The process is killed with SIGTERM
if it takes longer than 1 second to stop.
Tolerate a different of 50 ms, instead of just 30 ms, in
test_timeout() of multiprocessing tests. This change should fix such
test failure on Windows:
FAIL: test_timeout (test.test_multiprocessing_spawn.WithProcessesTestQueue)
Traceback (most recent call last):
File "lib\test\_test_multiprocessing.py", line 753, in test_timeout
self.assertGreaterEqual(delta, 0.170)
AssertionError: 0.16138982772827148 not greater than or equal to 0.17
* Add support.MS_WINDOWS: True if Python is running on Microsoft Windows.
* Add support.MACOS: True if Python is running on Apple macOS.
* Replace support.is_android with support.ANDROID
* Replace support.is_jython with support.JYTHON
* Cleanup code to initialize unix_shell
In some conditions the standard streams will be None or closed in the child process (for example if using "pythonw" instead of "python" on Windows). Avoid failing with a non-0 exit code in those conditions.
Report and initial patch by poxthegreat.
Fix socket(fileno=fd) by auto-detecting the socket's family, type,
and proto from the file descriptor. The auto-detection can be overruled
by passing in family, type, and proto explicitly.
Without the fix, all socket except for TCP/IP over IPv4 are basically broken:
>>> s = socket.create_connection(('www.python.org', 443))
>>> s
<socket.socket fd=3, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('2003:58:bc4a:3b00:56ee:75ff:fe47:ca7b', 59730, 0, 0), raddr=('2a04:4e42:1b::223', 443, 0, 0)>
>>> socket.socket(fileno=s.fileno())
<socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('2003:58:bc4a:3b00::%2550471192', 59730, 0, 2550471192), raddr=('2a04:4e42:1b:0:700c:e70b:ff7f:0%2550471192', 443, 0, 2550471192)>
Signed-off-by: Christian Heimes <christian@python.org>
Fix deadlocks in :class:`concurrent.futures.ProcessPoolExecutor` when task arguments or results cause pickling or unpickling errors.
This should make sure that calls to the :class:`ProcessPoolExecutor` API always eventually return.
* Fix multiple typos in code comments
* Add spacing in comments (test_logging.py, test_math.py)
* Fix spaces at the beginning of comments in test_logging.py
kB (*kilo* byte) unit means 1000 bytes, whereas KiB ("kibibyte")
means 1024 bytes. KB was misused: replace kB or KB with KiB when
appropriate.
Same change for MB and GB which become MiB and GiB.
Change the output of Tools/iobench/iobench.py.
Round also the size of the documentation from 5.5 MB to 5 MiB.
* bpo-31310: multiprocessing's semaphore tracker should be launched again if crashed
* Avoid mucking with process state in test.
Add a warning if the semaphore process died, as semaphores may then be leaked.
* Add NEWS entry
* bpo-31308: If multiprocessing's forkserver dies, launch it again when necessary.
* Fix test on Windows
* Add NEWS entry
* Adopt a different approach: ignore SIGINT and SIGTERM, as in semaphore tracker.
* Fix comment
* Make sure the test doesn't muck with process state
* Also test previously-started processes
* Update 2017-08-30-17-59-36.bpo-31308.KbexyC.rst
* Avoid masking SIGTERM in forkserver. It's not necessary and causes a race condition in test_many_processes.
Pierre Glaser
Thomas Moreau
Giampaolo Rodola
Pablo Galindo
Davin Potts
Rémi Lapeyre
Victor Stinner
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి)
Julien Palard
Zackery Spytz
tzickel
Serhiy Storchaka
Derek B. Kim
Ville Skyttä
Antoine Pitrou
Christian Heimes
Victor Stinner
Mike