Answering this question about nested pools for multiprocessing:
https://stackoverflow.com/a/40852258/6522112
The code I proposed was not responding the way I wanted. Which is to say if ThreadingPool(8) and ProcessingPool(3), I expected the number of processes to be 24 but I only got 3! So I moved on using this https://stackoverflow.com/a/8963618/6522112 which allows nested process pool by setting the daemon flag to False. This works perfectly but:
- Was it normal to only get 3 processes instead of 24?
- Am I doing it correctly with this hack?
- The final aim would be to do this using several nodes on a cluster. Does this prevent it?
- I am not sure about the type of
mapto consider but it seems that only theimapcombination is doing the job. Any insight?
Here is the modified code that uses pathos from @MikeMcKerns instead of simple multiprocessing:
import pathos
import multiprocess
class NoDaemonProcess(multiprocess.Process):
"""NoDaemonProcess class.
Inherit from :class:`multiprocessing.Process`.
``daemon`` attribute always returns False.
"""
def _get_daemon(self):
return False
def _set_daemon(self, value):
pass
daemon = property(_get_daemon, _set_daemon)
class NestedPool(pathos.multiprocessing.Pool):
"""NestedPool class.
Inherit from :class:`pathos.multiprocessing.Pool`.
Enable nested process pool.
"""
Process = NoDaemonProcess
Care to be taken if you want to use this, you have to terminate the inner pools otherwise you will end up with zombies:
import NestedPool
def triple(x):
return 3*x
def refork(x):
pool = NestedPool(3)
result = pool.imap(triple, range(5))
result = list(result)
pool.terminate()
return result
pool = NestedPool(8)
result = pool.imap(refork, range(3))