num_can_process

bigmultipipe.num_can_process(num_to_process=None, num_processes=None, mem_available=None, mem_frac=0.8, process_size=None, error_if_zero=True)[source]

Calculates maximum number of processes that can run simultaneously

Parameters
num_to_processint or None, optional

Total number of items to process. This number is returned if it is less than the maximum possible simultaneous processes. If None, not used in calculation. Default is None

num_processesnumber or None, optional

Maximum number of parallel processes. If None or 0, set to the number of physical (not logical) cores available using psutil.cpu_count(logical=False). If less than 1, the fraction of maximum cores to use. Default is None

mem_availableint or None. optional

Amount of memory available in bytes for the total set of processes. If None, mem_frac parameter is used. Default is None

mem_fracfloat, optional

Maximum fraction of current memory available that total set of processes is allowed to occupy. Current memory available is queried Default is 0.8

process_sizeint, or None, optional

Maximum process size in bytes of an individual process. If None, processes are assumed to be small enough to all fit in memory at once. Default is None

error_if_zerobool, optional

If True, throw an EnvironmentError error if return value would be zero. Useful for catching case when there is not enough memory for even one process. Set to False if subprocess can handle subdividing task. Default is True

Returns
num_can_processint

Maximum number of processes that can run simultaneously given input parameters