jztools.shared_memory#

The following operations produce xndarray objects:

Copying an xndarray – the copy will have new shared memory.

Slicing an xndarray – the slice will reference the original shared memory.

The following operations do not:

Operators that are not in-place, e.g., new_arr = 1 + x_arr.

Classes

`XSharedMemoryManager`(args, *kwargs)	SharedMemoryManager that can be used as an argument when creating multiprocessing processes.
`xndarray`(sh_mem, manager, shape, dtype, ...)	Class derived from `numpy.ndarray` that uses shared memory.

Exceptions

InvalidAsXndarray(obj)

exception jztools.shared_memory.InvalidAsXndarray(obj)#: Bases: Exception

class jztools.shared_memory.XSharedMemoryManager(*args, **kwargs)#

Bases: SharedMemoryManager

SharedMemoryManager that can be used as an argument when creating multiprocessing processes.

Allocated shared memory can be viewed using ls /dev/shm -lth

from_ndarray(arr, **kwargs)#: Cretes an xndarray from the input np.ndarray.

empty(*args, **kwargs)#: Creates an empty array in shared memory.

class jztools.shared_memory.xndarray(sh_mem: str | SharedMemory | None, manager: XSharedMemoryManager, shape, dtype, offset, strides, scope: str | None = 'local', **kwargs)#

Bases: ndarray

Class derived from numpy.ndarray that uses shared memory. Implements a pickling interface that pickles a reference to the shared memory instead of the actual buffer contents and is compatible with multiprocessing. This inclues support for array slicing - sliced arrays will be sent as shared memory arrays. When indexing produces a new copy of memory (e.g., when using non-uniform indexing such as arr[list([0,10])]), __getitem__() returns a numpy.ndarray

Parameters:

sh_mem – The shared memory with the contents of this array.
scope – Specifies when the shared memory will be released. Valid values are 'local', 'remote', or None. See Memory management.

Memory management

Shared memory needs to be released in order to avoid memory leaks that can persist after the end of the program.

The approached used herein relies on XSharedMemoryManager – a context manager derived from multiprocessing’s SharedMemoryManager that further supports being pickled and initializing xndarray instances. All shared memory created within a given XSharedMemoryManager context will be released upon exiting the context.

Within the context, xndarrays will be released based on their scope attribute, which can take one of the following values:

'local'

The shared memory will be released when the xndarray goes out of scope in the process where it was declared.
When pickling, this will be converted to None so that child processes do not release the memory.

Warning

An array created in child process should not use scope='local', as the created array will be garbage-collected in the child process. Attempting to access it in the parent process can result in undefined behavior, including raising a FileNotFoundError exception, or accessing the memory before it is garbage-collected.

'remote'

When the xndarray is pickled, this value will be pickled as 'local' .
The shared memory will hence be released when the unpickled array goes out of scope in the process that unpickled it, unless that process changes the value of the xndarray.scope attribute.

None

The shared memory will not be automatically released – it is the user’s responsibility to release the shared memory using method release(). Otherwise, the shared memory will be released by the memory manager, but will result in a memory leak in the meantime.

Examples

from jztools.shared_memory import XSharedMemoryManager, xndarray
from concurrent.futures import ProcessPoolExecutor
import numpy as np

with XSharedMemoryManager() as smm:

  # Fill an xndarry with random data
  N = int(1e6)
  rng = np.random.default_rng(0)
  xarr = smm.empty(N)
  rng.random(N, out=xarr)

  # Take the mean of both halves in different processes
  futures = []
  with ProcessPoolExecutor() as pool:
    futures.append(pool.submit(np.mean, xarr[:N//2]))
    futures.append(pool.submit(np.mean, xarr[N//2:]))

  results = [x.result().item() for x in futures]
  print(results)

[0.5001123274748334, 0.5002061854525354]

scope: str | None = None#: Can be one of 'local', 'remote' or None. See Memory management.

property shared_memory_manager#: Inherited from the source xndarray.

is_shared()#: Returns True if the xndarray points to data in shared memory. If False, the xndarray is no different than a regular numpy.ndarray

property shared_memory#: Inherited from the source xndarray.

release()#

Unlinks the underlying shared memory block. The array should not be accessed after it is released. In most situations, calling this method should not be necessary, as the scope based mechanism deals with releasing the underlying shared memory.

For example, if the array was initialized with the scope='local' keyword arg (the default), this method is called as part of the array’s __del__.

classmethod from_ndarray(manager: XSharedMemoryManager, arr: ndarray, **kwargs) → xndarray#

Creates a new array in newly-allocated shared memory and copies the data in arr to it.

Keyword arguments are passed empty().

as_ndarray() → ndarray#: Returns a copy of the array in non-shared memory.

classmethod empty(manager, shape, dtype=<class 'float'>, order='C', scope='local') → xndarray#: Creates an uninitialized array in newly-allocated shared memory.

jztools.shared_memory

Contents

jztools.shared_memory#