num_can_process¶
- bigmultipipe.num_can_process(num_to_process=None, num_processes=None, mem_available=None, mem_frac=0.8, process_size=None, error_if_zero=True)[source]¶
Calculates maximum number of processes that can run simultaneously
- Parameters
- num_to_processint or
None
, optional Total number of items to process. This number is returned if it is less than the maximum possible simultaneous processes. If
None
, not used in calculation. Default isNone
- num_processesnumber or
None
, optional Maximum number of parallel processes. If
None
or 0, set to the number of physical (not logical) cores available usingpsutil.cpu_count(logical=False)
. If less than 1, the fraction of maximum cores to use. Default isNone
- mem_availableint or None. optional
Amount of memory available in bytes for the total set of processes. If
None
,mem_frac
parameter is used. Default isNone
- mem_fracfloat, optional
Maximum fraction of current memory available that total set of processes is allowed to occupy. Current memory available is queried Default is
0.8
- process_sizeint, or None, optional
Maximum process size in bytes of an individual process. If None, processes are assumed to be small enough to all fit in memory at once. Default is
None
- error_if_zerobool, optional
If True, throw an
EnvironmentError
error if return value would be zero. Useful for catching case when there is not enough memory for even one process. Set toFalse
if subprocess can handle subdividing task. Default isTrue
- num_to_processint or
- Returns
- num_can_processint
Maximum number of processes that can run simultaneously given input parameters