jztools.parallelization.multiprocessing.outsourced_iterable#

Classes

ProcessOutsourcedIterable(source[, ...])

Takes an iterable and runs it in another process, transferring data between processes using a queue.
class jztools.parallelization.multiprocessing.outsourced_iterable.ProcessOutsourcedIterable(source: Iterable, initializer: Callable[[Iterable], None] | None = None, finalizer: Callable[[Iterable], None] | None = None, filter: Callable[[Any], bool] | None = None, mapper: Callable[[Any], Any] | None = None, post_filter: Callable[[Any], bool] | None = None, queue_size: int = 2, name=None)#

Bases: OutsourcedIterable

Takes an iterable and runs it in another process, transferring data between processes using a queue.

Parameters:

  • source – The iterable object to run in the remotely process. Depending on the multiprocessing process start method (see this discussion), the object might need to support pickling.

  • initializer – Called at the start of the worker process. See Process initializer / finalizer.

  • finalizer – Called in the worker process at the end of normal or abnormal (because of an exception) execution. See Process initializer / finalizer.

Process initializer / finalizer

Both initializer and finalizer run in the remote worker process and take the process’s copy of source as their only input parameter. Both callables must be module-level functions (including static and class methods of non-dynamically defined classes).

Parameters:

  • source – The iterable object to outsource.

  • initializer – Called at the start of the worker process, with source as an argument.

  • finalizer – Called in the worker process at the end of normal or abnormal (because of an exception) execution, with source as an argument.

  • filter – Items will be placed in the queue only if this returns True for that item.

  • mapper – Applied to each item before placing it in the queue.

  • post_filter – Items will be placed in the queue only if this returns True for the mapped item.

  • queue_size – The size of the queue to pre-compute.

  • name – The name given to the spawned work unit (e.g., thread or process).

Queue(maxsize=0)#

Returns a queue object

Event()#

Returns an event object

Spawner#

alias of Process