chunk

Chunk distribution for multiprocess. (qurry.qurrium.utils.chunk)

This module provides functions to calculate chunk size and distribute tasks

qurry.qurrium.utils.chunk.very_easy_chunk_distribution(respect_memory_array: list[tuple[str, int]], num_process: int = 16, max_chunk_size: int = 64) → tuple[int, list[tuple[str, int]], list[list[str]]][source]

Distribute the chunk for multiprocess. The chunk distribution is based on the number of CPU cores.

Parameters:

  • respect_memory_array (list[tuple[str, int]]) – The array of respect memory. Each element is a tuple of (id, memory). The id is the ID of the experiment, and the memory is the memory usage. The array is sorted by the memory usage.

  • num_process (int, optional) – The number of processes. Defaults to DEFAULT_POOL_SIZE.

  • max_chunk_size (int, optional) – The maximum chunk size. Defaults to CPU_COUNT * 4.

Returns:

The chunk distribution is a list of tuples of (id, memory).

Return type:

tuple[int, list[tuple[str, int]], list[list[str]]]

qurry.qurrium.utils.chunk.very_easy_chunk_size(tasks_num: int, num_process: int = 16, max_chunk_size: int = 64) → int[source]

Calculate the chunk size for multiprocess.

Parameters:

  • tasks_num (int) – The number of tasks.

  • num_process (int, optional) – The number of processes. Defaults to DEFAULT_POOL_SIZE.

  • max_chunk_size (int, optional) – The maximum chunk size. Defaults to CPU_COUNT * 4.

Returns:

The chunk size.

Return type:

int