padne.parallel

Process/thread fan-out for embarrassingly parallel stages.

The Parallel class bundles one parallelism budget (Config.jobs) with the lazily-created worker pool that enforces it, and provides two order-preserving map primitives:

thread_map(fn, items) for work that releases the GIL (C++ kernels).

Shares memory with the caller, nothing is pickled.

process_map(fn, items) for GIL-bound pure-Python work. fn must be

importable top-level; items and results must be cheap to pickle.

Both degrade to a plain serial loop in the caller’s thread when jobs == 1, so –jobs 1 gives clean tracebacks and a working pdb.

Application code uses the module-level functions, which delegate to a process-wide singleton instance (the same pattern as random.Random vs the random module functions). Tests can instantiate their own Parallel, which also works as a context manager that shuts its pool down on exit.

Worker processes are started with the “spawn” method; fork is not safe here since the parent typically has pcbnew (and in GUI mode Qt) loaded. The spawn pool is created lazily on first use and kept warm until shutdown(). Spawned workers do not inherit import hooks, so any typeguard import hooks active in the parent (as under pytest) are reinstalled in each worker.

Stage timings recorded via context.stage_timer inside process workers are shipped back and merged into the parent’s active timing session, so parallelized stages stay visible in the timing summary. Their perf times sum across workers and may exceed wall clock time for the enclosing stage.

Attributes

log
T
R

Classes

Config	Parallelism configuration. The default budget is the number of
Parallel	One parallelism budget plus the worker pool that enforces it.

Functions

config(→ Config)
configure(→ None)	Replace the module-default instance with one using the given budget.
thread_map(→ list[R])
process_map(→ list[R])
shutdown(→ None)

Module Contents

padne.parallel.log

padne.parallel.T

padne.parallel.R

class padne.parallel.Config

Parallelism configuration. The default budget is the number of available CPU cores.

jobs: int

class padne.parallel.Parallel(config: Config | None = None)

One parallelism budget plus the worker pool that enforces it.

The configuration is fixed at construction; to change the budget, create a new instance (the module-level configure() does exactly that for the shared singleton). Application code should normally go through the module-level functions; separate instances exist for tests and other scoped uses. Usable as a context manager: the pool is shut down on exit.

property config: Config

property pool_is_live: bool

shutdown() → None

Tear down the warm process pool. The next process_map call recreates it with the current configuration.

thread_map(fn: Callable[[T], R], items: Iterable[T]) → list[R]

Apply fn to every item in threads, returning results in input order. Only worthwhile when fn spends its time outside the GIL.

process_map(fn: Callable[[T], R], items: Iterable[T]) → list[R]

Apply fn to every item in worker processes, returning results in input order.

fn must be defined at the top level of an importable module; items and results must be picklable. The first worker exception propagates to the caller unchanged.

padne.parallel.config() → Config

padne.parallel.configure(jobs: int) → None

Replace the module-default instance with one using the given budget.

Raises RuntimeError if the current default has a live pool; parallelism must be configured before any parallel work runs (or after shutdown()).

padne.parallel.thread_map(fn: Callable[[T], R], items: Iterable[T]) → list[R]

padne.parallel.process_map(fn: Callable[[T], R], items: Iterable[T]) → list[R]

padne.parallel.shutdown() → None