Python Sensing Δ 8th of September 2014 Ω 11:37 AM

ΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞ
~ Multi-processing techniques in Python ~ MultiProcessing Examples ~ test_multiprocessing.py Challenges	~ Reading queues asynchronously in Python’s multiprocessing module ~ sense for Ξ ~ sense for Ξ	~ sense for Ξ ~ sense for Ξ ~ sense for Ξ
~ sense for Ξ ~ sense for Ξ ~ sense for Ξ	~ sense for Ξ ~ sense for Ξ ~ sense for Ξ	~ sense for Ξ ~ sense for Ξ ~ sense for Ξ
~ sense for Ξ ~ sense for Ξ ~ sense for Ξ	~ sense for Ξ ~ sense for Ξ ~ sense for Ξ	~ sense for Ξ ~ sense for Ξ ~ sense for Ξ
~ sense for Ξ ~ sense for Ξ ~ sense for Ξ	~ sense for Ξ ~ sense for Ξ ~ sense for Ξ	ξ ξ ξ
«Software Sensing	Θ	Θ
ΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞΞ

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~ Multi-processing techniques in Python

»Multi-processing techniques in Python

Scaling Python: Threads vs. Processes
ξ In the ongoing discussion of performance and scaling issues with Python, one persistent theme is the Global Interpreter Lock (GIL).
ξ While the GIL has the advantage of simplifying the implementation of CPython internals and extension modules,
ξ it prevents users from achieving true multi-threaded parallelism
ξ by limiting the interpreter to executing byte-codes in one thread at a time on a single processor.
ξ Threads which block on I/O or use extension modules written in another language
ξ can release the GIL to allow other threads to take over control, of course.
ξ But if an application is written entirely in Python,
ξ only a limited number of statements will be executed before one thread is suspended and another is started.

Eliminating the GIL
ξ has been on the wish lists of many Python developers for a long time
ξ a set of patches existed for Python 1.5 that eliminated the GIL entirely,
ξ replacing it with a whole set of individual locks for the mutable data structures (dictionaries, lists, etc.) that had been protected by the GIL.
ξ The result was an interpreter that ran at roughly half the normal speed,
ξ a side-effect of acquiring and releasing the individual locks used to replace the GIL.

The GIL issue is unique to the C implementation of the interpreter
ξ The Java implementation of Python, Jython, supports true threading by taking advantage of the underlying JVM.
ξ The IronPython port, running on Microsoft's CLR, also has better threading.
ξ On the other hand, those platforms are always playing catch-up with new language or library features,
ξ so if you're hot to use the latest and greatest, like I am, the C reference-implementation is still your best option.

Dropping the GIL from the C implementation remains a low priority for a variety of reasons
ξ the scope of the changes involved is beyond the level of anything the current developers are interested in tackling
ξ recently, Guido has said he would entertain patches contributed by the Python community to remove the GIL,
ξ as long as performance of single-threaded applications was not adversely affected

Requests pops up
ξ comp.lang.python
ξ the Python-related mailing lists to rewrite the interpreter so the lock can be removed

Use processes instead of threads
ξ that response does have some merit
ξ extension modules become more complicated without the safety of the GIL
ξ processes typically have fewer inherent deadlocking issues than threads
ξ they can be distributed between the CPUs on a host, and even more importantly,
ξ an application that uses multiple processes is not limited by the size of a single server, as a multi-threaded application would be

GIL is still present in Python 3.0
ξ it seems unlikely that it will be removed from a future version any time soon
ξ this may disappoint some people, but it is not the end of the world
ξ there are, after all, strategies for working with multiple processes to scale large applications
ξ techniques using low-level, operating system-specific, libraries for process management
ξ are as passe as using compiled languages for CGI programming
ξ don't have time for this low-level stuff any more, and neither do you
ξ there are some modern alternatives

select: ~[Σ] ~[Δ]!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~ MultiProcessing Examples

MultiProcessing Examples

»http://bugs.python.org/issue3350
Python 2.6b1+ (trunk:64899:64900, Jul 13 2008, 13:33:02)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing.util, pickle

»http://bugs.python.org/issue3589
The package level imports from the new multiprocessing package exhibit
some very strange behaviour because they are actually functions
pretending to be classes:

Python 2.6b1+ (trunk:64945, Jul 14 2008, 20:00:46)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing as mp
>>> isinstance(mp.Lock(), mp.Lock)
Traceback (most recent call last):
File "", line 1, in
TypeError: isinstance() arg 2 must be a class, type, or tuple of classes
and types
>>> mp.Lock.__name__
'Lock'
>>> mp.Lock.__module__
'multiprocessing'
>>> mp.Lock().__class__.__name__
'Lock'
>>> mp.Lock().__class__.__module__
'multiprocessing.synchronize'

The delayed import functions in multiprocessing.__init__ look like a
serious misfeature to me. I'd be inclined to replace them with "from
.synchronize import *" and "from .process import *" (leaving anything
which isn't covered by those two imports to be retrieved directly from
the relevant mp submodule)

select: ~[Σ] ~[Ω] ~[Δ]!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~ test_multiprocessing.py Challenges

test_multiprocessing.py Challenges

»http://mail.python.org/pipermail/new-bugs-announce/2008-June/002061.html
If I run test_multiprocessing.py manually from the _build_ directory it
works fine.

But if I use the locally installed Python 3 (soft linked to python3 in
my PATH), on both systems I get this:

: python3 -V
Python 3.0b1
: python3 Lib/test/test_multiprocessing.py
Traceback (most recent call last):
File "Lib/test/test_multiprocessing.py", line 19, in
import multiprocessing.dummy
ImportError: No module named multiprocessing.dummy

Now given that the tests all pass it seemed to me that the problem was
with the build rather than with the module itself, and that led to the
solution, which for both systems is the same simple command:

cp -R ~/download/Python-3.0b1/Lib/multiprocessing
~/opt/python30b1/lib/python3.0

So it seems that although the shared library is copied correctly from
the build directory, what has been forgotten is to copy the
multiprocessing directory itself.

select: ~[Σ] ~[Ω] ~[Δ]!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~ Reading queues asynchronously in Python’s multiprocessing module

»Reading queues asynchronously in Python’s multiprocessing module

Reading queues asynchronously in Python’s multiprocessing module
ξ Python 2.6 will add a multiprocessing module that supports fork/join semantics and management of process pools.
ξ However even in this multiprocessor environment it is still necesary to do some asynchronous.
ξ The main way that processes communicate in this environment are through shared queues
ξ implemented as socket-pairs or unix pipes under the hood
ξ but we need a method for the UI to be notified about updates to any queues
ξ it is listening on without blocking for example in a UI
ξ Luckily it is possible to get at underlying file-descriptor,
ξ which can then be managed with select() or some abstraction thereof.
ξ Unfortunately there's no official support for this
ξ but we can still poke around inside the implementation class to get at the details.
ξ The code below shows how to do this with GTK.
ξ This just spawns off a sub-process that counts up and sends the current number to the UI via a queue, which then displays the number:

PYTHON:
1. from __future__ import print_function
2. import gtk, gobject, time, multiprocessing, Queue
3.
4.
5. class UI:
6.
7. def destroy(self, widget, data=None):
8. self.outq.put("quit")
9. gtk.main_quit()
10.
11. __init__(self):
12. self.window = gtk.Window(gtk.WINDOW_TOPLEVEL)
13. self.window.connect("destroy", self.destroy)
14. self.window.set_border_width(10)
15.
16.
self.label = gtk.Label("Not Set!")
17. self.window.add(self.label)
18.
19. self.label.show()
20. self.window.show()
21.

22. update(self, fd, cond):
23. val = self.inq.get()
24. self.label.set_text("Value is %s" % val)
25. return True
26.
27. main(self, outq, inq):
28. self.outq = outq
29. self.inq = inq
30.
31. # This is the tricky bit, we get the socket FD of the
32. # underlying pipe from the queue object. We hope that the
33. # internal reader variable doesn't change.
34. fd = inq._reader.fileno()
35.
36. # This tells GTK to watch this FD and notify us when data is
37. # available. We use newer API, gdk_input_read is deprecated
38. gobject.io_add_watch(fd, gobject.IO_IN, self.update)
39. gtk.main()
40.
41.
42.def counter(inq, outq, interval):
43. count = 1
44. again = True
45. while again:
46. print("Putting %s on queue" % count)
47. outq.put(count)
48. count += 1
49.
50. try:
51. print("Reading from queue...")
52. ret = inq.get(True, interval)
53. if ret == "quit":
54. print("Got quit message")
55. again = False
56. except Queue.Empty:
57. # Timed-out, carry on
58. pass
59.
60.
61.if __name__ == "__main__":
62. q1 = multiprocessing.Queue()
63. q2 = multiprocessing.Queue()
64.
65. proc = multiprocessing.Process(target=counter, args=(q1, q2, 1,))
66. proc.start()
67.
68. ui = UI()
69. ui.main(q1, q2)
70.
71. proc.join()

select: ~[Σ] ~[Ω] ~[Δ]!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~

select: ~[Σ] ~[Ω] ~[Δ]!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~

select: ~[Σ] ~[Ω]!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Small & Smart Inc reserves rights to change this document without any notice
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~