parallelmanager

The parallel tools for Qurry. (qurry.tools.parallelmanager)

qurry.tools.parallelmanager.CPU_COUNT = 16

The number of CPUs available for multiprocessing. This number is guaranteed to be at least 1.

qurry.tools.parallelmanager.CPU_COUNT_UNSAFE = 16

The number of CPUs available for multiprocessing. But this number may be None in some cases.

qurry.tools.parallelmanager.DEFAULT_POOL_SIZE = 16

The default number of workers for multiprocessing. This number is guaranteed to be at least 1.

class qurry.tools.parallelmanager.ParallelManager(workers_num: int | None = 16, bar_format: str = 'qurry-full', bar_ascii: str = '4squares', **pool_kwargs)

Process manager for multiprocessing.

map(func: Callable[[T_tgt], T_map], arg_list: Iterable[T_tgt], start_method: Literal['spawn', 'fork', 'forkserver'] = 'spawn') → list[T_map]

This function is a wrapper for map from multiprocessing.

Parameters:

• func (Callable[[Iterable[T_tgt]], T_map]) – Function to be mapped.

• arg_list (Iterable[Iterable[T_tgt]]) – Arguments to be mapped.

• start_method (Optional[Literal["spawn", "fork", "forkserver"]], optional) – Start method for multiprocessing. Defaults to “spawn”.

Returns:

Results.

Return type:

list[T_map]

process_map(func: Callable[[...], T_map], args_list: Iterable[Iterable[Any]], bar_format: str = 'qurry-full', bar_ascii: str = '4squares', **kwargs) → list[T_map]

Call process_map from tqdm. This function is a wrapper for process_map from tqdm. But, it won’t use pool_kwargs for they are different implementations with multiprocessing.Pool in this class.

Parameters:

• func (Callable[[Any], T_map]) – Function to be mapped.

• args (Iterable[Any]) – Arguments to be mapped.

• bar_format (str, optional) – Progress bar format. Defaults to “qurry-full”.

• bar_ascii (str, optional) – Progress bar ascii. Defaults to “4squares”.

• **kwargs – Other arguments.

Returns:

Results.

Return type:

list[T_map]

starmap(func: Callable[[...], T_map], args_list: Iterable, start_method: Literal['spawn', 'fork', 'forkserver'] = 'spawn') → list[T_map]

This function is a wrapper for starmap from multiprocessing.

Parameters:

• func (Callable[[Iterable[T_tgt]], T_map]) – Function to be mapped.

• args_list (Iterable[Iterable[T_tgt]]) – Arguments to be mapped.

• start_method (Optional[Literal["spawn", "fork", "forkserver"]], optional) – Start method for multiprocessing. Defaults to “spawn”.

Returns:

Results.

Return type:

tqdm.tqdm[T_map]

qurry.tools.parallelmanager.workers_distribution(workers_num: int | None = None, default: int = 16) → int

Distribute the workers number.

Parameters:

• workers_num (Optional[int], optional) – Desired workers number. Defaults to None.

• default (int, optional) – Default workers number. Defaults to DEFAULT_POOL_SIZE.

Returns:

Workers number.

Return type:

int