Multiprocessing Python Pool

Question

When is it necessary to call .join() and .close() on a Pool in the case below? Reading the docs, it looks like it is for waiting for the processes to finish. For instance, if I do something like this:

while True:
    pool = Pool(processes=4)
    results = []
    for x in range(1000):
        result = pool.apply_async(f, (x,))
        results.append(result)

    for result in results:
        result.get(timeout=1)
    print "finished"

Do I still need to wait for the other process to finish with join() and close()? As I assume, that since I am iterating over all async results and waiting (blocking) for them to finish, by the time I get to print finished, all processes will have exited already?

Is this correct?

Also when do the processes start working on a function? I noticed that there are 4 processes running in parallel with ps -elf. Do the processes only start to work on the function after result.get() is called in this case?

Answer 1

close()

Prevents any more tasks from being submitted to the pool. Once all the tasks have been completed the worker processes will exit.

join()

Wait for all processes to properly terminate

Good link to start with Proper way to use multiprocessor.Pool in a nested loop

As soon as you call pool.apply_async the process will start working on the function, it'll return a result object