h5flow.modules

h5flow.modules.find_class(classname, directory)

Search the specified directory for a file containing a python implementation with the specified class name

Parameters

• classname – class name to look for

• directory – directory to search for *.py files describing the class

Returns

class object of matching desired class (if found), or None (if not found)

h5flow.modules.get_class(classname, path=None)

Look in current directory, ./h5flow_modules/, and h5flow/modules/ for the specified class. Raises a RuntimeError if class can’t be found in any of those directories. Optionally, a specific python path can be provided and the class will be loaded directly from that module (faster)

Parameters

• classname – class name to search for

• path – python path to module that class can be accessed from

Returns

class object of desired class

class h5flow.modules.h5_flow_dataset_loop_generator.H5FlowDatasetLoopGenerator(**params)

Bases: H5FlowGenerator

Default dataset looping generator

First copies input file to output file. Then slices up the dataset defined by dset_name into chunk_size chunks, separated by MPI rank.

For some example use cases, the default configuration declaration:

flow:
    source: <group name>/<dataset group name>
    stages: [...]

will auto chunk the dataset given by <group name>/<dataset group name>. But the manual chunk size specification:

flow:
    source: input
    stages: [...]

input:
    classname: H5FlowDatasetLoopGenerator
    dset_name: <group name>/<dataset group name>
    params:
        chunk_size: <num_rows, opt>

will chunk the same dataset, but into chunks of <num_rows>.

class_version = '0.0.0'

copy(f0, f1, block=True)

init()

Prepare output file to be written to, called once before initializing the flow stages.

next()

Generate a new slice into the source dataset in the data file. To end loop, return an empty slice (H5FlowGenerator.EMPTY).

Returns

<slice> into self.dset_name data

setup_slices()

Initialize slices for loop

class h5flow.modules.h5_flow_test_stage.H5FlowTestStage(**params)

Bases: H5FlowStage

A test stage that does nothing except load data from the requires list and print arguments passed along with method.

Example config:

test:
    classname: H5FlowTestStage
    requires:
        - 'example/dataset'

init(source_name)

Called once before starting the loop. Used to create datasets and set file meta-data

Returns

None

run(source_name, source_slice, cache)

Called once per source_slice provided by the h5flow generator.

Parameters

• source_name – path to the source dataset group

• source_slice – a 1D slice into the source dataset

• cache – pre-loaded data from requires list

Returns

None