API Reference¶

bigmultipipe Module¶

Provides tools for parallel pipeline processing of large data structures

`assure_list`(x)	Assures x is type `list`.
`bmp_cleanup`(data[, bmp_meta, add])	Enables select `BigMultiPipe` metadata to be deleted after all `BigMultiPipe` post-processing routines have completed.
`cached_pout`(pipe_code[, poutname, ...])	Write/read `BigMultiPipe.pipeline()` metadata output ("pout") to/from a file using `pickle`
`multi_logging`(level, meta, message)	Implements logging on a per-process basis in `BigMultiPipe` pipeline post-processing routines
`multi_proc`(func[, element_type])	Returns a function which applies func to each element in a (possibly nested) list.
`no_outfile`(data, **kwargs)	`bigmultipipe` post-processing routine that stops pipeline processing and returns accumulated metadata without writing any output files.
`num_can_process`([num_to_process, ...])	Calculates maximum number of processes that can run simultaneously
`outname_creator`(in_name[, meta, outdir, ...])	Convenience function to create `Bigmultipipe` output filename
`prune_pout`(pout, in_names)	Removes entries marked for deletion in a `BigMultiPipe.pipeline()` output

`BigMultiPipe`([num_processes, mem_available, ...])	Base class for memory- and processing power-optimized pipelines
`NestablePool`(args, *kwargs)	For Python = 3.9(+?) enable child processes of `Pool` to run their own sub-processes.
`NoDaemonContext`()	Used for `bigmultipipe.NestablePool`
`NoDaemonPool37`([processes, initializer, ...])	For Python <=3.7, enable child processes of `Pool` to run their own sub-processes.
`NoDaemonProcess`([group, target, name, args, ...])	Used for `bigmultipipe.NestablePool`
`NoDaemonProcess37`([group, target, name, ...])	Used for `bigmultipipe.NoDaemonPool37`
`WorkerWithKwargs`(function, **kwargs)	Class to hold static kwargs for use with, e.g., `multiprocessing.pool.Pool.map()`