Python

Python Multiprocessing Not Using Multiple Cores

When Python code using the multiprocessing module doesn't seem to actually use multiple CPU cores — monitoring shows only one core busy despite spawning several processes — the cause is usually a setup mistake that accidentally defeats true parallelism, not a fundamental limitation of multiprocessing itself.

The Problem

You use multiprocessing.Pool or Process expecting work to be distributed across multiple CPU cores, but monitoring tools show only one core under load, and the total execution time isn't meaningfully faster than running the same work sequentially.

Why It Happens

Common causes of multiprocessing failing to actually parallelize work across cores:

  • The workload itself is I/O-bound rather than CPU-bound (waiting on network requests or disk access), in which case multiple cores don't help much regardless of how correctly multiprocessing is set up — the bottleneck isn't CPU time in the first place
  • Using multiprocessing.dummy (which is actually a thread pool, not a process pool) by mistake, meaning the code is still limited by Python's Global Interpreter Lock (GIL) for CPU-bound work, since threads share a single interpreter lock while true processes don't
  • The chunk size or task granularity is too fine, so the overhead of distributing tiny units of work across processes outweighs any parallelism benefit, making it look like nothing is happening in parallel even though processes technically are being used
  • On some platforms (notably Windows, and macOS with certain Python versions), multiprocessing has stricter requirements around how the entry point is structured, and code that doesn't follow this can silently fail to actually spawn separate processes correctly

The Fix

First, confirm you're actually using real processes, not threads. Genuine multiprocessing:

from multiprocessing import Pool

def cpu_intensive_task(n):
    return sum(i * i for i in range(n))

if __name__ == '__main__':
    with Pool(processes=4) as pool:
        results = pool.map(cpu_intensive_task, [10_000_000] * 4)

Note the if __name__ == '__main__': guard — on Windows especially, this is required, not optional, since the multiprocessing module re-imports your script in each new process, and without this guard, that re-import can trigger infinite recursive process spawning or fail to work correctly at all.

Confirm your workload is genuinely CPU-bound before expecting multiprocessing to help. If the work is mostly waiting on network I/O or disk access, asyncio or threading (which handle I/O-bound concurrency well despite the GIL, since the GIL is released during I/O waits) are usually a better fit than multiprocessing's heavier process-based overhead:

# I/O-bound work: asyncio or threading, not multiprocessing
# CPU-bound work: multiprocessing (or, in some cases, thread-based
# parallelism with libraries that release the GIL, like NumPy)

Increase task granularity if you're distributing many very small units of work. Batching work into larger chunks per process reduces the relative overhead of inter-process communication:

with Pool(processes=4) as pool:
    results = pool.map(cpu_intensive_task, large_input_list, chunksize=100)

Verify processes are actually being created and used by checking system-level monitoring during execution, rather than relying on assumptions:

import os
print(f"Running in process {os.getpid()}")

Adding this inside your worker function and checking the logged output confirms multiple distinct process IDs are genuinely being used, which directly validates whether the parallelism is real rather than just configured.

Still Not Working?

If genuine separate processes are confirmed (different PIDs logged) but performance still isn't improving proportionally to the number of cores used, check whether the workload has a serial bottleneck that limits speedup regardless of core count — for example, all processes writing to the same shared resource (a single file, a database with limited connections) that serializes access even though the actual computation is happening in parallel. In that case, the parallel computation itself may be working correctly, but a shared resource contention point elsewhere is masking the benefit.

You might also like