multi_proc

bigmultipipe.multi_proc(func, element_type=None, **kwargs)

Returns a function which applies func to each element in a (possibly nested) list.

Parameters

funcfunction

function to apply to items in list. Function must have at least one argument (see **kwargs)

element_typetype, optional

type of element to which func will be applied. If specified, allows nested lists of files to be processed

**kwargspassed to func

Returns

function

Notes

This function is useful when the list of bigmultipipe in_names is not just a list of single files, but a list of possibly nested lists and certain pre- and post-processing routines are best applied to the individual elements of those lists (e.g. individual files)

It is syntactically correct to use multi_proc to define a function on-the-fly, e.g.:

>>> newfunc = multi_proc(plus1)

however that function becomes a member of locals and cannot be pickled for passing in a multiprocessing environment. The form in Examples avoids the pickling error. See https://stackoverflow.com/questions/52265120/python-multiprocessing-pool-attributeerror

Examples

>>> def plus1(data, **kwargs):
>>>     return data + 1
>>>
>>> def multi_plus1(data, **kwargs):
>>>     return multi_proc(plus1)(data)