Welcome to nest-joblib documentation!
nest-joblib
Patch joblib to allow nested parallelism.
Installation
Install this via pip (or your favourite package manager):
pip install nest-joblib
Usage
from nest_joblib import apply
apply()
With the above code, LokyBackend supports nested-parallelism.
Advanced Usage
The following joblib specification of not doing nested-parallelism may be inefficient in an environment with sufficient memory.
joblib/_parallel_backends.py
:
def get_nested_backend(self):
"""Backend instance to be used by nested Parallel calls.
By default a thread-based backend is used for the first level of
nesting. Beyond, switch to sequential backend to avoid spawning too
many threads on the host.
"""
nesting_level = getattr(self, 'nesting_level', 0) + 1
if nesting_level > 1:
return SequentialBackend(nesting_level=nesting_level), None
else:
return ThreadingBackend(nesting_level=nesting_level), None
After calling nest_joblib.apply()
, when joblib.parallel.register_parallel_backend(name, backend)
is called, a subclass of backend
with modified get_nested_backend
is dynamically generated and registered with the name f"nested-{name}"
.
from joblib.parallel import parallel_backend
from nest_joblib import apply
from ray.util.joblib import register_ray
# use LokyBackend
apply()
# use DaskDistributedBackend
apply()
parallel_backend("nested-dask")
# use RayBackend
apply()
register_ray()
parallel_backend("nested-ray")
# use custom backend
from joblib.parallel import LokyBackend
apply()
class MyBackend(LokyBackend):
pass
register_parallel_backend("custom", MyBackend)
parallel_backend("nested-custom")
Contributors ✨
Thanks goes to these wonderful people (emoji key):
34j 💻 🤔 📖 |
This project follows the all-contributors specification. Contributions of any kind welcome!