Python Multiprocessing Pool Hang or Freeze on Exception

A multiprocessing.Pool that hangs indefinitely rather than raising an exception when a worker function fails is a well-known and common pattern, particularly with certain Pool methods — the exception did occur, but depending on which method you used to submit work, it may not propagate back to the main process the way you'd expect from normal, single-threaded exception handling.

The Problem

A script using a multiprocessing pool simply hangs, never completing and never producing an error, when one of the worker processes encounters an exception:

from multiprocessing import Pool

def risky_task(x):
    if x == 3:
        raise ValueError("something went wrong")
    return x * 2

with Pool(4) as pool:
    results = pool.map(risky_task, range(10))
    # hangs indefinitely if x == 3 is processed, no visible error

Why It Happens

An exception raised inside a worker process is captured by the multiprocessing machinery and stored, ready to be re-raised in the main process — but it's only actually re-raised at the point where the main process retrieves that specific worker's result. Depending on which Pool method you used and how you're consuming results, this retrieval step might never actually happen in a way that surfaces the error, leading to an apparent hang instead of a clear exception. Common contributing factors:

• Using pool.map(), which does correctly propagate exceptions, but only once the entire batch attempts to complete — if other tasks are still running or the pool's internal bookkeeping gets into an inconsistent state due to the exception, this can occasionally look like a hang before the exception is actually raised, especially with a large task count
• Using apply_async() without ever calling .get() on the returned result object — the exception is stored and waiting, but if your code never calls .get(), it's simply never actually retrieved or raised, and the program can appear to hang or silently continue without any indication a worker failed
• A worker process crashing entirely (not just raising a Python exception, but a genuine process-level crash, like a segmentation fault from a native extension) can leave the pool in a state where it's waiting indefinitely for a worker that will never respond again
• Deadlock-prone patterns within the worker function itself (like a worker attempting its own nested multiprocessing operations without proper handling) that hang the worker process itself, independent of the pool's own exception handling

The Fix

Ensure your worker function itself handles exceptions explicitly and returns a clear, structured result indicating success or failure, rather than letting an unhandled exception propagate up in a way that depends on correctly using .get() to surface it:

def risky_task(x):
    try:
        if x == 3:
            raise ValueError("something went wrong")
        return {'success': True, 'value': x * 2}
    except Exception as e:
        return {'success': False, 'error': str(e)}

with Pool(4) as pool:
    results = pool.map(risky_task, range(10))
    for r in results:
        if not r['success']:
            print(f"Task failed: {r['error']}")

This pattern guarantees every task completes and returns a value (never raising unhandled inside the worker), sidestepping the entire class of "did the exception actually propagate correctly" concern, since failures are represented as normal return values rather than relying on exception propagation across the process boundary at all.

If using apply_async() directly, always call .get() on every result object, ideally with an explicit timeout, ensuring you actually retrieve (and therefore properly surface) any exception a worker might have raised:

result = pool.apply_async(risky_task, (3,))
try:
    value = result.get(timeout=10)
except Exception as e:
    print(f"Task raised: {e}")

The explicit timeout parameter here is also valuable independently — if a worker genuinely hangs (rather than raising a clean exception), .get(timeout=...) raises a multiprocessing.TimeoutError after the specified duration rather than blocking your main process indefinitely, converting a silent, indefinite hang into a clear, actionable timeout error you can handle explicitly.

For genuine worker process crashes (not just Python exceptions, but actual process termination), monitor pool health and consider implementing your own timeout/retry wrapper around the entire pool operation, since a crashed worker process can leave the pool's internal state in a way that standard exception handling within the worker function itself can't address, since the worker never gets the chance to return any value — clean or otherwise — at all.

Still Not Working?

If tasks genuinely hang (not crash, not raise) even with try/except wrapping in place, the hang is likely inside the worker function's own logic itself, independent of multiprocessing's exception handling entirely — add logging at the very start and end of the worker function, and at key points within it, to identify exactly which specific line or operation the worker is stuck on, since a genuine internal hang (an infinite loop, a blocking I/O call with no timeout) needs to be fixed at that specific point in the worker's own logic, not addressed through any change to how the pool itself handles exceptions or results.