8.3.1. Hdf5 module and classes

8.3.1.1. Hdf5 backends

The H5Backend is a wrapper around three hdf5 python packages: pytables, h5py and h5pyd. It allows seamless integration of any of these with PyMoDAQ features.

class pymodaq_data.h5modules.backends.H5Backend(backend='tables')

Attributes:

filename

h5file

is_swmr_active

Return True if SWMR mode is currently active on the file.

is_swmr_capable

Return True if the current backend supports SWMR mode.

is_swmr_compatible

Return True if the open file was created with SWMR support.

Methods

add_group(group_name, group_type, where[, ...])	Add a node in the h5 file tree of the group type :type group_name: (str) a custom name for this group :param group_name: :type group_name: (str) a custom name for this group :param group_type: one of the possible values of GroupType, should be enforced by higher level modules not here :type group_type: Union[GroupType, str] :type where: (str or node) parent node where to create the new group :param where: :type where: (str or node) parent node where to create the new group :type metadata: (dict) extra metadata to be saved with this new group node :param metadata: :type metadata: (dict) extra metadata to be saved with this new group node
close_file()	Flush data and close the h5file
create_earray(where, name, dtype[, ...])	create enlargeable arrays from data with a given shape and of a given type.
create_vlarray(where, name, dtype[, title])	create variable data length and type and enlargeable 1D arrays
define_compression(compression, compression_opts)	Define cmpression library and level of compression :type compression: (str) either gzip and zlib are supported here as they are compatible :param compression: but zlib is used by pytables while gzip is used by h5py :type compression: (str) either gzip and zlib are supported here as they are compatible :param compression_opts (int): :type compression_opts (int): 0 to 9  0: None, 9: maximum compression
enable_swmr()	Activate SWMR mode on the open h5py file.
finalize_swmr([keep_open])	End SWMR by closing the file, reopening in 'a' mode, and reconciling deferred attrs.
get_children(where)	Get a dict containing all children node hanging from where with their name as keys and types among Node, CARRAY, EARRAY, VLARRAY or StringARRAY
get_node_name(node)	return node name :param node (str or node instance): :param see h5py and pytables documentation on nodes:
get_node_path(node)	return node path :param node (str or node instance): :param see h5py and pytables documentation on nodes:
get_set_group(where, name[, title])	Retrieve or create (if absent) a node group Get attributed to the class attribute current_group
is_node_in_group(where, name)	Check if a given node with name is in the group defined by where (comparison on lower case strings) :type where: (str or node) :param where: path or parent node instance :type where: (str or node) :type name: (str) :param name: group node name :type name: (str)
reconcile_swmr_attrs()	Walk all EARRAY/VLARRAY nodes and update attrs['shape'] from actual data.
set_backend(backend)	Switch the active backend, closing any open file first.

create_carray
flush
get_attr
get_group_by_title
get_node
get_parent_node
has_attr
isopen
open_file
read
root
save_file_as
set_attr
walk_groups
walk_nodes

add_group(group_name, group_type, where, title='', metadata=None)

Add a node in the h5 file tree of the group type :type group_name: (str) a custom name for this group :param group_name: :type group_name: (str) a custom name for this group :param group_type: one of the possible values of GroupType, should be enforced by higher level modules not here :type group_type: Union[GroupType, str] :type where: (str or node) parent node where to create the new group :param where: :type where: (str or node) parent node where to create the new group :type metadata: (dict) extra metadata to be saved with this new group node :param metadata: :type metadata: (dict) extra metadata to be saved with this new group node

Returns:

(node)

Return type:

GROUP

close_file()

Flush data and close the h5file

create_earray(where, name, dtype, data_shape=None, title='')

create enlargeable arrays from data with a given shape and of a given type. The array is enlargeable along the first dimension

create_vlarray(where, name, dtype, title='')

create variable data length and type and enlargeable 1D arrays

Parameters:

• where ((str) group location in the file where to create the array node)

• name ((str) name of the array)

• dtype ((dtype) numpy dtype style, for particular case of strings, use dtype=``’string’``)

• title ((str) node title attribute (written in capitals))

Return type:

array

define_compression(compression, compression_opts)

Define cmpression library and level of compression :type compression: (str) either gzip and zlib are supported here as they are compatible :param compression: but zlib is used by pytables while gzip is used by h5py :type compression: (str) either gzip and zlib are supported here as they are compatible :param compression_opts (int): :type compression_opts (int): 0 to 9  0: None, 9: maximum compression

enable_swmr()

Activate SWMR mode on the open h5py file.

Must be called after all groups/datasets have been created. Raises RuntimeError if backend is not h5py or file was not opened with swmr_mode=True. Idempotent: does nothing if already enabled.

finalize_swmr(keep_open=False)

End SWMR by closing the file, reopening in ‘a’ mode, and reconciling deferred attrs.

After SWMR mode, attrs[‘shape’] on EARRAY/VLARRAY nodes may be stale. This method closes the file (ending SWMR), reopens it in append mode, and updates all deferred attributes.

Parameters:

keep_open (bool) – If True, leaves the file open in ‘a’ mode after reconciling. If False (default), closes the file after reconciling.

get_children(where)

Get a dict containing all children node hanging from where with their name as keys and types among Node, CARRAY, EARRAY, VLARRAY or StringARRAY

Parameters:

instance) (where (str or node) – see h5py and pytables documentation on nodes, and Node objects of this module

Returns:

dict

Return type:

keys are children node names, values are the children nodes

See also

GROUP.children_name()

get_node_name(node)

return node name :param node (str or node instance): :param see h5py and pytables documentation on nodes:

Returns:

str

Return type:

name of the node

get_node_path(node)

return node path :param node (str or node instance): :param see h5py and pytables documentation on nodes:

Returns:

str

Return type:

full path of the node

get_set_group(where, name, title='', **kwargs)

Retrieve or create (if absent) a node group Get attributed to the class attribute current_group

Parameters:

• where (str or node) – path or parent node instance

• name (str) – group node name

• title (str) – node title

• Arguments (Keyword) – any other metadata related to this node (for example: origin)

Returns:

group

Return type:

group node

is_node_in_group(where, name)

Check if a given node with name is in the group defined by where (comparison on lower case strings) :type where: (str or node) :param where: path or parent node instance :type where: (str or node) :type name: (str) :param name: group node name :type name: (str)

Returns:

True if node exists, False otherwise

Return type:

bool

reconcile_swmr_attrs()

Walk all EARRAY/VLARRAY nodes and update attrs[‘shape’] from actual data.

Called after SWMR is ended (file closed and reopened in ‘a’ mode) to fix deferred attribute writes that were skipped during SWMR.

set_backend(backend)

Switch the active backend, closing any open file first.

Updates both self.backend (the name string) and self.h5_library (the imported module), which both need to be consistent for file operations.

Parameters:

backend (str) – One of 'tables', 'h5py', or 'h5pyd'.

property is_swmr_active

Return True if SWMR mode is currently active on the file.

property is_swmr_capable

Return True if the current backend supports SWMR mode.

property is_swmr_compatible

Return True if the open file was created with SWMR support.

8.3.1.2. Low Level saving

H5SaverLowLevel is the base saving class providing file creation and node management. H5SaverBase and H5Saver build on it and integrate with the PyMoDAQ Framework. H5SaverBase adds a ParameterManager with settings for backend selection, SWMR options, file paths and compression. H5Saver inherits H5SaverBase and adds Qt signals (new_file_sig, file_changed_sig) and a file browser dialog. To save and load data, one should use higher level objects, see High Level saving/loading.

Created the 15/11/2022

@author: Sebastien Weber

class pymodaq_data.h5modules.saving.H5SaverLowLevel(save_type='scan', backend=None)

Object containing basic methods in order to structure and interact with a h5file compatible with the h5browser

See also

H5Browser

h5_file

object used to save all datas and metadas

Type:

pytables hdf5 file

h5_file_path

The file path

Type:

str or Path

classmethod from_file(path, save_type='scan', new_file=False, metadata=None)

Create and initialise an H5SaverLowLevel from a file path.

Convenience factory that combines the constructor and init_file() call into a single expression. :param path: Full path to the HDF5 file. :type path: Union[Path, str] :param save_type: Type label stored in the file (default 'scan'). :type save_type: SaveType :param new_file: If True a new file is created, otherwise the existing file is

opened for appending.

Parameters:

metadata (dict) – Extra attributes written to the raw-data group on creation.

Returns:

A fully initialised instance ready for reading/writing.

Return type:

H5SaverLowLevel

add_act_group(where, title='', settings_as_xml='', metadata=None)

Add a new group of type detector .. seealso:: add_incremental_group

add_array(where, name, data_type, array_to_save=None, data_shape=None, array_type=None, fill_value=None, data_dimension=None, scan_shape=(), add_scan_dim=False, enlargeable=False, title='', metadata={})

save data arrays on the hdf5 file together with metadata :param where: node where to save the array :type where: Union[GROUP, str] :param name: name of the array in the hdf5 file :type name: str :param data_type: mandatory so that the h5Browser can interpret correctly the array :type data_type: DataType :param data_shape: the shape of the array to save, mandatory if array_to_save is None :type data_shape: tuple :param data_dimension: The data’s dimension :type data_dimension: DataDim :param scan_shape: the shape of the scan dimensions :type scan_shape: tuple :param title: the title attribute of the array node :type title: str :param array_to_save: data to be saved in the array. If None, array_type and data_shape should be specified in order to init

correctly the memory

Parameters:

• array_type (dtype) – eg np.float, np.int32 …

• fill_value (float or int) – value to be used to fill the array if array_to_save is None

• enlargeable (bool) – if False, data are saved as a CARRAY, otherwise as a EARRAY (for ragged data, see add_string_array)

• metadata (dict) – dictionnary whose keys will be saved as the array attributes

• add_scan_dim (if True, the scan axes dimension (scan_shape iterable) is prepended to the array shape on the hdf5) – In that case, the array is usually initialized as zero and further populated

Return type:

array (CARRAY or EARRAY)

See also

add_data, add_string_array

add_ch_group(where, title='', settings_as_xml='', metadata=None)

Add a new group of type channel .. seealso:: add_incremental_group

add_data_group(where, data_dim, title='', settings_as_xml='', metadata=None, group_name=None)

Creates a group node at given location in the tree

Parameters:

• where (group node) – where to create data group

• data_dim (DataDim) – the dimensionality of the data group

• title (str, optional) – a title for this node, will be saved as metadata

• settings_as_xml (str, optional) – XML string created from a Parameter object to be saved as metadata

• metadata (dict, optional) – will be saved as a new metadata attribute with name: key and value: dict value

• group_name (str) – the name of the group to create if None, the name of the DataDim enum is used (default)

Returns:

group

Return type:

group node

See also

add_group()

add_det_group(where, title='', settings_as_xml='', metadata=None)

Add a new group of type detector .. seealso:: add_incremental_group

add_generic_group(where='/RawData', title='', settings_as_xml='', metadata=None, group_type=GroupType.scan)

Add a new group of type given by the input argument group_type

At creation adds the attributes description to be used elsewhere

See also

add_incremental_group

add_incremental_group(group_type, where, title='', settings_as_xml='', metadata=None)

Add a node in the h5 file tree of the group type with an increment in the given name :param group_type: one of the possible values of group_types :type group_type: Union[str, GroupType, Enum] :type where: str or node :param where: parent node where to create the new group :type where: str or node :type title: str :param title: node title :type title: str :type settings_as_xml: str :param settings_as_xml: XML string containing Parameter representation :type settings_as_xml: str :type metadata: dict :param metadata: extra metadata to be saved with this new group node :type metadata: dict

Returns:

node

Return type:

newly created group node

add_move_group(where, title='', settings_as_xml='', metadata=None)

Add a new group of type actuator .. seealso:: add_incremental_group

add_scan_group(where='/RawData', title='', settings_as_xml='', metadata=None)

Add a new group of type scan

deprecated, use add_generic_group with a group type as GroupType.scan

get_groups(where, group_type)

Get all groups hanging from a Group and of a certain type

get_node_from_attribute_match(where, attr_name, attr_value)

Get a Node starting from a given node (Group) matching a pair of node attribute name and value

get_node_from_title(where, title)

Get a Node starting from a given node (Group) matching the given title

get_set_group(where, name, title='', **kwargs)

Get the group located at where if it exists otherwise creates it

This also set the _current_group property

get_set_logger(where=None)

Retrieve or create (if absent) a logger enlargeable array to store logs Get attributed to the class attribute logger_array :param where: location within the tree where to save or retrieve the array :type where: Node

Returns:

enlargeable array accepting strings as elements

Return type:

VLARRAY

init_file(file_name, raw_group_name='RawData', new_file=False, metadata=None, swmr_mode=False)

Initializes a new h5 file,

Should have an extension with h5 in it.

Parameters:

• file_name (Path) – a complete Path pointing to a h5 file

• raw_group_name (str) – Base node name

• new_file (bool) – If True create a new file, otherwise append to a potential existing one

• metadata (dict) – A dictionary to be saved as attributes

• swmr_mode (bool) – If True, prepare the file for SWMR (h5py backend only)

Returns:

True if new file has been created, False otherwise

Return type:

bool

set_swmr_flush_interval(interval)

Set how often to flush data for SWMR readers.

Parameters:

interval (int) – 0 = flush only at end, N = every N writes

tick_flush()

Increment the write counter and flush if the interval is reached.

No-op if SWMR is not active or flush interval is 0. To be called by data savers after each logical data write.

They both inherit from the ParameterManager MixIn class that deals with Parameter and ParameterTree, see saving_settings_fig.

8.3.1.3. SWMR utilities

When using the h5py backend with SWMR mode enabled, the following utility functions help readers access the file while a scan is in progress.

pymodaq_data.h5modules.open_h5_file_for_reading(filepath, swmr='auto', locking=None)

Open an HDF5 file for reading, automatically handling SWMR mode.

This utility function handles the complexity of opening HDF5 files that may or may not be currently being written with SWMR mode.

Parameters:

• filepath (str or Path) – Path to the HDF5 file

• swmr (bool or 'auto', optional) –

  • ‘auto’ (default): Try to detect if SWMR is needed

  • True: Force SWMR reader mode

  • False: Open normally without SWMR

• locking (bool or None, optional) – File locking mode. None uses h5py default. Set to False on Windows if you get locking errors.

Returns:

(h5py.File, is_swmr_active) - The file handle and whether SWMR is active

Return type:

tuple

Examples

>>> # Open a file being written by PyMoDAQ scan
>>> f, is_swmr = open_h5_file_for_reading("scan_data.h5")
>>> if is_swmr:
...     ds = f['path/to/data']
...     ds.id.refresh()  # Call refresh to see latest data
>>> f.close()

pymodaq_data.h5modules.is_file_swmr_active(filepath)

Check if an HDF5 file is currently being written with SWMR mode.

Parameters:

filepath (str or Path) – Path to the HDF5 file

Returns:

True if the file has swmr_active=True attribute, False otherwise

Return type:

bool

Utilities for reading HDF5 files written with SWMR (Single Writer Multiple Reader) mode.

8.3.1.3.1. Typical usage

Open the file once, collect dataset references, then poll in a loop:

from pymodaq_data.h5modules import open_h5_file_for_reading from pymodaq_data.h5modules.swmr import collect_datasets, refresh_cached

f, is_swmr = open_h5_file_for_reading(“scan.h5”) cache = collect_datasets(f[“RawData”]) # dict[str, h5py.Dataset]

while acquiring:

refresh_cached(cache) data = cache[“/RawData/CH000/Data0D/Data00/data”][:]

pymodaq_data.h5modules.swmr.collect_datasets(group)

Walk group recursively and return a mapping of absolute path → dataset.

The returned dict can be passed to refresh_cached() on every poll cycle instead of re-walking the tree each time.

Parameters:

group (Group) – Any h5py.Group (or h5py.File, which is also a group).

Returns:

{"/absolute/path": h5py.Dataset, ...} for every dataset found under group.

Return type:

Dict[str, Dataset]

Examples

>>> f, _ = open_h5_file_for_reading("scan.h5")
>>> cache = collect_datasets(f["RawData"])
>>> cache.keys()
dict_keys(['/RawData/CH000/Data0D/Data00/data', ...])

pymodaq_data.h5modules.swmr.refresh_cached(cache)

Refresh every dataset in a pre-built cache dict.

This is the fast path for polling loops: call collect_datasets() once to build cache, then call this function on each iteration.

Parameters:

cache (Dict[str, Dataset]) – A {path: h5py.Dataset} dict as returned by collect_datasets().

Return type:

None

Examples

>>> cache = collect_datasets(f["RawData"])
>>> while acquiring:
...     refresh_cached(cache)
...     latest_row = cache["/RawData/CH000/Data0D/Data00/data"][-1]

pymodaq_data.h5modules.swmr.refresh_datasets(group)

Refresh every dataset under group so that SWMR readers see the latest data written by the writer process.

This is a convenience wrapper for one-shot use. For polling loops prefer collect_datasets() + refresh_cached() to avoid re-walking the tree on every iteration.

Parameters:

group (Group) – Any h5py.Group (or h5py.File).

Return type:

None

Notes

refresh() is a metadata/chunk-index call; it does not read the actual data. The data is only transferred when you access dataset elements (ds[:], ds[-1], etc.).

8.3.1.4. High Level saving/loading

Each PyMoDAQ’s data type: Axis, DataWithAxes, DataToExport (see What is PyMoDAQ’s Data?) is associated with its saver/loader counterpart. These objects ensures that all metadata necessary for an exact regeneration of the data is being saved at the correct location in the hdf5 file hierarchy. The AxisSaverLoader, DataSaverLoader, DataToExportSaver all derive from an abstract class: DataManagement allowing the manipulation of the nodes and making sure the data type is defined.

8.3.1.4.1. Base data class saver/loader

Created the 21/11/2022

@author: Sebastien Weber

class pymodaq_data.h5modules.data_saving.AxisSaverLoader(h5saver)

Specialized Object to save and load Axis object to and from a h5file

Parameters:

h5saver (H5SaverLowLevel)

data_type

The enum for this type of data, here ‘axis’

Type:

DataType

add_axis(where, axis, enlargeable=False)

Write Axis info at a given position within a h5 file

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• axis (Axis) – the Axis object to add as a node in the h5file

• enlargeable (bool) – Specify if the underlying array will be enlargebale

get_axes(where)

Return a list of Axis objects from the Axis Nodes hanging from (or among) a given Node

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself

Returns:

List[Axis]

Return type:

List[Axis]

load_axis(where)

create an Axis object from the data and metadata at a given node if of data_type: ‘axis

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself

Return type:

Axis

class pymodaq_data.h5modules.data_saving.DataManagement(*args, **kwargs)

Base abstract class to be used for all specialized object saving and loading data to/from a h5file

data_type

The enum for this type of data, here abstract and should be redefined

Type:

DataType

get_last_node_name(where)

Get the last node name among the ones already saved

Parameters:

where (Union[str, Node]) – the path of a given node or the node itself

Returns:

str

Return type:

Optional[str]

property raw_group: Node

Get the base RawGroup where raw data should be saved

class pymodaq_data.h5modules.data_saving.DataSaverLoader(h5saver, new_file=False, metadata=None, save_type=SaveType.custom)

Specialized Object to save and load DataWithAxes object to and from a h5file

Parameters:

h5saver (Union[H5SaverLowLevel, Path])

data_type

The enum for this type of data, here ‘data’

Type:

DataType

add_data(where, data, save_axes=True, **kwargs)

Adds Array nodes to a given location adding eventually axes as others nodes and metadata

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data (DataWithAxes)

• save_axes (bool)

get_axes(where)

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself

Return type:

List[Axis]

get_data_arrays(where, with_bkg=False, load_all=False)

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• with_bkg (bool) – If True try to load background node and return the array with background subtraction

• load_all (bool) – If True load all similar nodes hanging from a parent

Return type:

List[ndarray]

isopen()

Get the opened status of the underlying hdf5 file

Return type:

bool

load_data(where, with_bkg=False, load_all=False)

Return a DataWithAxes object from the Data and Axis Nodes hanging from (or among) a given Node

Does not include navigation axes stored elsewhere in the h5file. The node path is stored in the DatWithAxis using the attribute path

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• with_bkg (bool) – If True try to load background node and return the data with background subtraction

• load_all (bool) – If True, will load all data hanging from the same parent node

See also

load_data

Return type:

DataWithAxes

class pymodaq_data.h5modules.data_saving.DataToExportSaver(h5saver, save_type=SaveType.custom, new_file=False, metadata=None)

Object used to save DataToExport object into a h5file following the PyMoDAQ convention

Parameters:

h5saver (Union[H5SaverLowLevel, Path, str])

static channel_formatter(ind)

All DataWithAxes included in the DataToExport will be saved into a channel group indexed and formatted as below

add_data(where, data, settings_as_xml='', **kwargs)

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data (DataToExport)

• settings_as_xml (str) – The settings parameter as an XML string

• Arguments (Keyword) – all extra metadata to be saved in the group node where data will be saved

isopen()

Get the opened status of the underlying hdf5 file

Return type:

bool

8.3.1.4.2. Specific data class saver/loader

Some more dedicated objects are derived from the objects above. They allow to add background data, Extended arrays (arrays that will be populated after creation, for instance for a scan) and Enlargeable arrays (whose final length is not known at the moment of creation, for instance when logging or continuously saving)

Created the 21/11/2022

@author: Sebastien Weber

class pymodaq_data.h5modules.data_saving.BkgSaver(h5saver)

Specialized Object to save and load DataWithAxes background object to and from a h5file

Parameters:

hsaver (H5SaverLowLevel)

data_type

The enum for this type of data, here ‘bkg’

Type:

DataType

class pymodaq_data.h5modules.data_saving.DataEnlargeableSaver(h5saver, enl_axis_names=('nav axis',), enl_axis_units=('',))

Specialized Object to save and load enlargeable DataWithAxes saved object to and from a h5file

Particular case of DataND with a single nav_indexes parameter will be appended as chunks of signal data

Parameters:

h5saver (Union[H5SaverLowLevel, Path])

data_type

The enum for this type of data, here ‘data_enlargeable’

Type:

DataType

Notes

To be used to save data from a timed logger (DAQViewer continuous saving or DAQLogger extension) or from an adaptive scan where the final shape is unknown or other module that need this feature

add_data(where, data, axis_values=None, **kwargs)

Append data to an enlargeable array node

Data of dim (0, 1 or 2) will be just appended to the enlargeable array.

Uniform DataND with one navigation axis of length (Lnav) will be considered as a collection of Lnav signal data of dim (0, 1 or 2) and will therefore be appended as Lnav signal data

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data (DataWithAxes)

• axis_values (Iterable[float]) – the new spread axis values added to the data if None the axes are not added to the h5 file

class pymodaq_data.h5modules.data_saving.DataExtendedSaver(h5saver, extended_shape, fill_value=None)

Specialized Object to save and load DataWithAxes saved object to and from a h5file in extended arrays

Parameters:

• h5saver (H5SaverLowLevel)

• extended_shape (Tuple[int]) – the extra shape compared to the data the h5array will have

data_type

The enum for this type of data, here ‘data’

Type:

DataType

add_data(where, data, indexes, distribution=DataDistribution.uniform)

Adds given DataWithAxes at a location within the initialized h5 array

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data (DataWithAxes)

• indexes (List[int]) – indexes where to save data in the init h5array (should have the same length as extended_shape and with values coherent with this shape

class pymodaq_data.h5modules.data_saving.DataToExportEnlargeableSaver(h5saver, enl_axis_names=None, enl_axis_units=None, axis_name='nav axis', axis_units='')

Generic object to save DataToExport objects in an enlargeable h5 array

The next enlarged value should be specified in the add_data method

Parameters:

• h5saver (Union[Path, H5SaverLowLevel])

• enl_axis_names (Iterable[str]) – The names of the enlargeable axis, default [‘nav_axis’]

• enl_axis_units (Iterable[str]) – The names of the enlargeable axis, default [‘’]

• axis_name (str) – the name of the enlarged axis array

• axis_units (str) – the units of the enlarged axis array

add_data(where, data, axis_values=None, axis_value=None, settings_as_xml='', **kwargs)

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data (DataToExport) – The data to be saved into an enlargeable array

• axis_values (List[Union[float, ndarray]]) – The next value (or values) of the enlarged axis

• axis_value (Union[float, ndarray]) – The next value (or values) of the enlarged axis

• settings_as_xml (str) – The settings parameter as an XML string

• Arguments (Keyword) – all extra metadata to be saved in the group node where data will be saved

class pymodaq_data.h5modules.data_saving.DataToExportExtendedSaver(h5saver, extended_shape, fill_value=None)

Object to save DataToExport at given indexes within arrays including extended shape

Mostly used for data generated from the DAQScan

Parameters:

• h5saver (H5SaverLowLevel)

• extended_shape (Tuple[int]) – the extra shape compared to the data the h5array will have

add_data(where, data, indexes, distribution=DataDistribution.uniform, settings_as_xml='', **kwargs)

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data (DataToExport)

• indexes (Iterable[int]) – indexes where to save data in the init h5array (should have the same length as extended_shape and with values coherent with this shape

• settings_as_xml (str) – The settings parameter as an XML string

• Arguments (Keyword) – all extra metadata to be saved in the group node where data will be saved

add_nav_axes(where, axes)

Used to add navigation axes related to the extended array

Notes

For instance the scan axes in the DAQScan

class pymodaq_data.h5modules.data_saving.DataToExportTimedSaver(h5saver)

Specialized DataToExportEnlargeableSaver to save data as a function of a time axis

Only one element ca be added at a time, the time axis value are enlarged using the data to be added timestamp

Notes

This object is made for continuous saving mode of DAQViewer and logging to h5file for DAQLogger

8.3.1.5. Specialized loading

Data saved from a DAQ_Scan will naturally include navigation axes shared between many different DataWithAxes (as many as detectors/channels/ROIs). They are therefore saved at the root of the scan node and cannot be retrieved using the standard data loader. Hence this DataLoader object.

class pymodaq_data.h5modules.data_saving.DataLoader(h5saver)

Specialized Object to load DataWithAxes object from a h5file

On the contrary to DataSaverLoader, does include navigation axes stored elsewhere in the h5file (for instance if saved from the DAQ_Scan)

Parameters:

h5saver (Union[H5SaverLowLevel, Path])

Attributes:

h5saver

raw_group

Get the base RawGroup where raw data should be saved

Methods

get_nav_group(where)
get_node(where[, name])	Convenience method to get node
load_data(where[, with_bkg, load_all])	Load data from a node (or channel node)
load_data_from_name_origin(name[, origin, ...])	Load data from a node if this data as the given name and origin
walk_nodes([where])	Return a Node generator iterating over the h5file content

close_file
load_all

get_nav_group(where)

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself

Return type:

Optional[Node]

Returns:

• GROUP (returns the group named SPECIAL_GROUP_NAMES[``’nav_axes’``] holding all NavAxis for)

• those data

See also

SPECIAL_GROUP_NAMES

get_node(where, name=None)

Convenience method to get node

Return type:

Node

load_data(where, with_bkg=False, load_all=False)

Load data from a node (or channel node)

Loaded data contains also nav_axes if any and with optional background subtraction

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• with_bkg (bool) – If True will attempt to substract a background data node before loading

• load_all (bool) – If True, will load all data hanging from the same parent node

Return type:

DataWithAxes

load_data_from_name_origin(name, origin='', where=None, with_bkg=False, load_all=True)

Load data from a node if this data as the given name and origin

Parameters:

• name (str) – The name of the data (stored in the title attribute)

• origin (str) – The origin of the data

• where (Union[GROUP, Node, str]) – If specified start to look for matching Dwa at where Node

• with_bkg (bool) – If True load with bkg substraction if any

• load_all (bool) – if True load all channels of the parent node

Return type:

DataWithAxes

walk_nodes(where='/')

Return a Node generator iterating over the h5file content

property raw_group: Node

Get the base RawGroup where raw data should be saved

Convenience method

8.3.1.6. Browsing Data

Using the H5Backend it is possible to write scripts to easily access a hdf5 file content. However, PyMoDAQ includes a dedicated hdf5 viewer understanding dedicated metadata and therefore displaying nicely the content of the file, see Data Browsing: the H5Browser module. Two objects can be used to browse data: H5BrowserUtil and H5Browser. H5BrowserUtil gives you methods to quickly (in a script) get info and data from your file while the H5Browser adds a UI to interact with the hdf5 file.

8.3.1.7. Module savers