nest_joblib package

nest_joblib.apply(*, set_default: bool = True, auto_register: bool = True) → None[source]

Apply nest_joblib.

The following joblib specification of not doing nested-parallelism may be inefficient in an environment with sufficient memory.

joblib/_parallel_backends.py:

>>> def get_nested_backend(self):
>>> """Backend instance to be used by nested Parallel calls.
>>>
>>> By default a thread-based backend is used for the first level of
>>> nesting. Beyond, switch to sequential backend to avoid spawning too
>>> many threads on the host.
>>> """
>>> nesting_level = getattr(self, 'nesting_level', 0) + 1
>>> if nesting_level > 1:
>>>     return SequentialBackend(nesting_level=nesting_level), None
>>> else:
>>>     return ThreadingBackend(nesting_level=nesting_level), None

After calling this function, when joblib.parallel.register_parallel_backend(name, backend) is called, a subclass of backend with modified get_nested_backend is dynamically generated and registered with the name f”nested-{name}”.

Parameters:

set_default (bool, optional) – Whether to set DEFAULT_BACKEND to nested-loky, by default True
auto_register (bool, optional) – Whether to automatically register nested parallel backend. Useful when registering external parallel backend like ray AFTER calling this function. If False, nested version of already registered parallel backends will still be registered. If error occurs, setting this to False may help, by default True

Examples

Using nested-loky (nested version of loky) as default parallel backend: >>> import nest_joblib >>> nest_joblib.apply()

Using nested-dask (nested version of another parallel backend dask) as default parallel backend: >>> from joblib import parallel_backend >>> from nest_joblib import apply >>> >>> apply() >>> parallel_backend(“nested-dask”)

Using nested-ray (nested version of external parallel backend ray) as default parallel backend: >>> from joblib import parallel_backend >>> from nest_joblib import apply >>> from ray.util.joblib import register_ray >>> >>> apply() >>> register_ray() >>> parallel_backend(“nested-ray”)