API Reference

bigmultipipe Module

Provides tools for parallel pipeline processing of large data structures



Assures x is type list.

bmp_cleanup(data[, bmp_meta, add])

Enables select BigMultiPipe metadata to be deleted after all BigMultiPipe post-processing routines have completed.

cached_pout(pipe_code[, poutname, ...])

Write/read BigMultiPipe.pipeline() metadata output ("pout") to/from a file using pickle

multi_logging(level, meta, message)

Implements logging on a per-process basis in BigMultiPipe pipeline post-processing routines

multi_proc(func[, element_type])

Returns a function which applies func to each element in a (possibly nested) list.

no_outfile(data, **kwargs)

bigmultipipe post-processing routine that stops pipeline processing and returns accumulated metadata without writing any output files.

num_can_process([num_to_process, ...])

Calculates maximum number of processes that can run simultaneously

outname_creator(in_name[, meta, outdir, ...])

Convenience function to create Bigmultipipe output filename

prune_pout(pout, in_names)

Removes entries marked for deletion in a BigMultiPipe.pipeline() output


BigMultiPipe([num_processes, mem_available, ...])

Base class for memory- and processing power-optimized pipelines

NestablePool(*args, **kwargs)

For Python = 3.9(+?) enable child processes of Pool to run their own sub-processes.


Used for bigmultipipe.NestablePool

NoDaemonPool37([processes, initializer, ...])

For Python <=3.7, enable child processes of Pool to run their own sub-processes.

NoDaemonProcess([group, target, name, args, ...])

Used for bigmultipipe.NestablePool

NoDaemonProcess37([group, target, name, ...])

Used for bigmultipipe.NoDaemonPool37

WorkerWithKwargs(function, **kwargs)

Class to hold static kwargs for use with, e.g., multiprocessing.pool.Pool.map()