8.3.1. Hdf5 module and classes

8.3.1.1. Hdf5 backends

The H5Backend is a wrapper around three hdf5 python packages: pytables, h5py and h5pyd. It allows seamless integration of any of these with PyMoDAQ features.

class pymodaq_data.h5modules.backends.H5Backend(backend='tables')

Attributes:

filename

h5file

Methods

add_group(group_name, group_type, where[, ...])	Add a node in the h5 file tree of the group type :type group_name: (str) a custom name for this group :param group_name: :type group_name: (str) a custom name for this group :param group_type: one of the possible values of GroupType, should be enforced by higher level modules not here :type group_type: Union[GroupType, str] :type where: (str or node) parent node where to create the new group :param where: :type where: (str or node) parent node where to create the new group :type metadata: (dict) extra metadata to be saved with this new group node :param metadata: :type metadata: (dict) extra metadata to be saved with this new group node
close_file()	Flush data and close the h5file
create_earray(where, name, dtype[, ...])	create enlargeable arrays from data with a given shape and of a given type.
create_vlarray(where, name, dtype[, title])	create variable data length and type and enlargeable 1D arrays
define_compression(compression, compression_opts)	Define cmpression library and level of compression :type compression: (str) either gzip and zlib are supported here as they are compatible :param compression: but zlib is used by pytables while gzip is used by h5py :type compression: (str) either gzip and zlib are supported here as they are compatible :param compression_opts (int): :type compression_opts (int): 0 to 9  0: None, 9: maximum compression
get_children(where)	Get a dict containing all children node hanging from where with their name as keys and types among Node, CARRAY, EARRAY, VLARRAY or StringARRAY
get_node_name(node)	return node name :param node (str or node instance): :param see h5py and pytables documentation on nodes:
get_node_path(node)	return node path :param node (str or node instance): :param see h5py and pytables documentation on nodes:
get_set_group(where, name[, title])	Retrieve or create (if absent) a node group Get attributed to the class attribute current_group
is_node_in_group(where, name)	Check if a given node with name is in the group defined by where (comparison on lower case strings) :type where: (str or node) :param where: path or parent node instance :type where: (str or node) :type name: (str) :param name: group node name :type name: (str)

create_carray
flush
get_attr
get_group_by_title
get_node
get_parent_node
has_attr
isopen
open_file
read
root
save_file_as
set_attr
walk_groups
walk_nodes

add_group(group_name, group_type, where, title='', metadata={})

Add a node in the h5 file tree of the group type :type group_name: (str) a custom name for this group :param group_name: :type group_name: (str) a custom name for this group :param group_type: one of the possible values of GroupType, should be enforced by higher level modules not here :type group_type: Union[GroupType, str] :type where: (str or node) parent node where to create the new group :param where: :type where: (str or node) parent node where to create the new group :type metadata: (dict) extra metadata to be saved with this new group node :param metadata: :type metadata: (dict) extra metadata to be saved with this new group node

Returns:

(node)

Return type:

GROUP

close_file()

Flush data and close the h5file

create_earray(where, name, dtype, data_shape=None, title='')

create enlargeable arrays from data with a given shape and of a given type. The array is enlargeable along the first dimension

create_vlarray(where, name, dtype, title='')

create variable data length and type and enlargeable 1D arrays

Parameters:

• where ((str) group location in the file where to create the array node)

• name ((str) name of the array)

• dtype ((dtype) numpy dtype style, for particular case of strings, use dtype=``’string’``)

• title ((str) node title attribute (written in capitals))

Return type:

array

define_compression(compression, compression_opts)

Define cmpression library and level of compression :type compression: (str) either gzip and zlib are supported here as they are compatible :param compression: but zlib is used by pytables while gzip is used by h5py :type compression: (str) either gzip and zlib are supported here as they are compatible :param compression_opts (int): :type compression_opts (int): 0 to 9  0: None, 9: maximum compression

get_children(where)

Get a dict containing all children node hanging from where with their name as keys and types among Node, CARRAY, EARRAY, VLARRAY or StringARRAY

Parameters:

instance) (where (str or node) – see h5py and pytables documentation on nodes, and Node objects of this module

Returns:

dict

Return type:

keys are children node names, values are the children nodes

See also

GROUP.children_name()

get_node_name(node)

return node name :param node (str or node instance): :param see h5py and pytables documentation on nodes:

Returns:

str

Return type:

name of the node

get_node_path(node)

return node path :param node (str or node instance): :param see h5py and pytables documentation on nodes:

Returns:

str

Return type:

full path of the node

get_set_group(where, name, title='', **kwargs)

Retrieve or create (if absent) a node group Get attributed to the class attribute current_group

Parameters:

• where (str or node) – path or parent node instance

• name (str) – group node name

• title (str) – node title

• Arguments (Keyword) – any other metadata related to this node (for example: origin)

Returns:

group

Return type:

group node

is_node_in_group(where, name)

Check if a given node with name is in the group defined by where (comparison on lower case strings) :type where: (str or node) :param where: path or parent node instance :type where: (str or node) :type name: (str) :param name: group node name :type name: (str)

Returns:

True if node exists, False otherwise

Return type:

bool

8.3.1.2. Low Level saving

H5SaverBase and H5Saver classes are a help to save data in a hierachical hdf5 binary file through the H5Backend object and allowing integration in the PyMoDAQ Framework. These objects allows the creation of a file, of the various nodes necessary to save PyMoDAQ’s data. The saving functionalities are divided in two objects: H5SaverBase and H5Saver. H5SaverBase contains everything needed for saving, while H5Saver, inheriting H5SaverBase, add Qt functionality such as emitted signals. However, these are not specific of PyMoDAQ’s data types. To save and load data, one should use higher level objects, see High Level saving/loading.

Created the 15/11/2022

@author: Sebastien Weber

They both inherits from the ParameterManager MixIn class that deals with Parameter and ParameterTree, see saving_settings_fig.

8.3.1.3. High Level saving/loading

Each PyMoDAQ’s data type: Axis, DataWithAxes, DataToExport (see What is PyMoDAQ’s Data?) is associated with its saver/loader counterpart. These objects ensures that all metadata necessary for an exact regeneration of the data is being saved at the correct location in the hdf5 file hierarchy. The AxisSaverLoader, DataSaverLoader, DataToExportSaver all derive from an abstract class: DataManagement allowing the manipulation of the nodes and making sure the data type is defined.

8.3.1.3.1. Base data class saver/loader

Created the 21/11/2022

@author: Sebastien Weber

class pymodaq_data.h5modules.data_saving.AxisSaverLoader(h5saver)

Specialized Object to save and load Axis object to and from a h5file

Parameters:

h5saver (H5SaverLowLevel)

data_type

The enum for this type of data, here ‘axis’

Type:

DataType

add_axis(where, axis, enlargeable=False)

Write Axis info at a given position within a h5 file

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• axis (Axis) – the Axis object to add as a node in the h5file

• enlargeable (bool) – Specify if the underlying array will be enlargebale

get_axes(where)

Return a list of Axis objects from the Axis Nodes hanging from (or among) a given Node

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself

Returns:

List[Axis]

Return type:

List[Axis]

load_axis(where)

create an Axis object from the data and metadata at a given node if of data_type: ‘axis

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself

Return type:

Axis

class pymodaq_data.h5modules.data_saving.DataManagement(*args, **kwargs)

Base abstract class to be used for all specialized object saving and loading data to/from a h5file

data_type

The enum for this type of data, here abstract and should be redefined

Type:

DataType

get_last_node_name(where)

Get the last node name among the ones already saved

Parameters:

where (Union[str, Node]) – the path of a given node or the node itself

Returns:

str

Return type:

Optional[str]

class pymodaq_data.h5modules.data_saving.DataSaverLoader(h5saver)

Specialized Object to save and load DataWithAxes object to and from a h5file

Parameters:

h5saver (Union[H5SaverLowLevel, Path])

data_type

The enum for this type of data, here ‘data’

Type:

DataType

add_data(where, data, save_axes=True, **kwargs)

Adds Array nodes to a given location adding eventually axes as others nodes and metadata

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data (DataWithAxes)

• save_axes (bool)

get_axes(where)

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself

Return type:

List[Axis]

get_data_arrays(where, with_bkg=False, load_all=False)

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• with_bkg (bool) – If True try to load background node and return the array with background subtraction

• load_all (bool) – If True load all similar nodes hanging from a parent

Return type:

List[ndarray]

isopen()

Get the opened status of the underlying hdf5 file

Return type:

bool

load_data(where, with_bkg=False, load_all=False)

Return a DataWithAxes object from the Data and Axis Nodes hanging from (or among) a given Node

Does not include navigation axes stored elsewhere in the h5file. The node path is stored in the DatWithAxis using the attribute path

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• with_bkg (bool) – If True try to load background node and return the data with background subtraction

• load_all (bool) – If True, will load all data hanging from the same parent node

See also

load_data

Return type:

DataWithAxes

class pymodaq_data.h5modules.data_saving.DataToExportSaver(h5saver, save_type=SaveType.scan)

Object used to save DataToExport object into a h5file following the PyMoDAQ convention

Parameters:

h5saver (Union[H5SaverLowLevel, Path, str])

static channel_formatter(ind)

All DataWithAxes included in the DataToExport will be saved into a channel group indexed and formatted as below

add_data(where, data, settings_as_xml='', **kwargs)

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data (DataToExport)

• settings_as_xml (str) – The settings parameter as an XML string

• Arguments (Keyword) – all extra metadata to be saved in the group node where data will be saved

isopen()

Get the opened status of the underlying hdf5 file

Return type:

bool

8.3.1.3.2. Specific data class saver/loader

Some more dedicated objects are derived from the objects above. They allow to add background data, Extended arrays (arrays that will be populated after creation, for instance for a scan) and Enlargeable arrays (whose final length is not known at the moment of creation, for instance when logging or continuously saving)

Created the 21/11/2022

@author: Sebastien Weber

class pymodaq_data.h5modules.data_saving.BkgSaver(h5saver)

Specialized Object to save and load DataWithAxes background object to and from a h5file

Parameters:

hsaver (H5SaverLowLevel)

data_type

The enum for this type of data, here ‘bkg’

Type:

DataType

class pymodaq_data.h5modules.data_saving.DataEnlargeableSaver(h5saver, enl_axis_names=('nav axis',), enl_axis_units=('',))

Specialized Object to save and load enlargeable DataWithAxes saved object to and from a h5file

Particular case of DataND with a single nav_indexes parameter will be appended as chunks of signal data

Parameters:

h5saver (Union[H5SaverLowLevel, Path])

data_type

The enum for this type of data, here ‘data_enlargeable’

Type:

DataType

Notes

To be used to save data from a timed logger (DAQViewer continuous saving or DAQLogger extension) or from an adaptive scan where the final shape is unknown or other module that need this feature

add_data(where, data, axis_values=None, **kwargs)

Append data to an enlargeable array node

Data of dim (0, 1 or 2) will be just appended to the enlargeable array.

Uniform DataND with one navigation axis of length (Lnav) will be considered as a collection of Lnav signal data of dim (0, 1 or 2) and will therefore be appended as Lnav signal data

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data (DataWithAxes)

• axis_values (Iterable[float]) – the new spread axis values added to the data if None the axes are not added to the h5 file

class pymodaq_data.h5modules.data_saving.DataExtendedSaver(h5saver, extended_shape)

Specialized Object to save and load DataWithAxes saved object to and from a h5file in extended arrays

Parameters:

• h5saver (H5SaverLowLevel)

• extended_shape (Tuple[int]) – the extra shape compared to the data the h5array will have

data_type

The enum for this type of data, here ‘data’

Type:

DataType

add_data(where, data, indexes, distribution=DataDistribution.uniform)

Adds given DataWithAxes at a location within the initialized h5 array

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data (DataWithAxes)

• indexes (List[int]) – indexes where to save data in the init h5array (should have the same length as extended_shape and with values coherent with this shape

class pymodaq_data.h5modules.data_saving.DataToExportEnlargeableSaver(h5saver, enl_axis_names=None, enl_axis_units=None, axis_name='nav axis', axis_units='')

Generic object to save DataToExport objects in an enlargeable h5 array

The next enlarged value should be specified in the add_data method

Parameters:

• h5saver (H5SaverLowLevel)

• enl_axis_names (Iterable[str]) – The names of the enlargeable axis, default [‘nav_axis’]

• enl_axis_units (Iterable[str]) – The names of the enlargeable axis, default [‘’]

• axis_name (str) – the name of the enlarged axis array

• axis_units (str) – the units of the enlarged axis array

add_data(where, data, axis_values=None, axis_value=None, settings_as_xml='', **kwargs)

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data (DataToExport) – The data to be saved into an enlargeable array

• axis_values (List[Union[float, ndarray]]) – The next value (or values) of the enlarged axis

• axis_value (Union[float, ndarray]) – The next value (or values) of the enlarged axis

• settings_as_xml (str) – The settings parameter as an XML string

• Arguments (Keyword) – all extra metadata to be saved in the group node where data will be saved

class pymodaq_data.h5modules.data_saving.DataToExportExtendedSaver(h5saver, extended_shape)

Object to save DataToExport at given indexes within arrays including extended shape

Mostly used for data generated from the DAQScan

Parameters:

• h5saver (H5SaverLowLevel)

• extended_shape (Tuple[int]) – the extra shape compared to the data the h5array will have

add_data(where, data, indexes, distribution=DataDistribution.uniform, settings_as_xml='', **kwargs)

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data (DataToExport)

• indexes (Iterable[int]) – indexes where to save data in the init h5array (should have the same length as extended_shape and with values coherent with this shape

• settings_as_xml (str) – The settings parameter as an XML string

• Arguments (Keyword) – all extra metadata to be saved in the group node where data will be saved

add_nav_axes(where, axes)

Used to add navigation axes related to the extended array

Notes

For instance the scan axes in the DAQScan

class pymodaq_data.h5modules.data_saving.DataToExportTimedSaver(h5saver)

Specialized DataToExportEnlargeableSaver to save data as a function of a time axis

Only one element ca be added at a time, the time axis value are enlarged using the data to be added timestamp

Notes

This object is made for continuous saving mode of DAQViewer and logging to h5file for DAQLogger

8.3.1.4. Specialized loading

Data saved from a DAQ_Scan will naturally include navigation axes shared between many different DataWithAxes (as many as detectors/channels/ROIs). They are therefore saved at the root of the scan node and cannot be retrieved using the standard data loader. Hence this DataLoader object.

class pymodaq_data.h5modules.data_saving.DataLoader(h5saver)

Specialized Object to load DataWithAxes object from a h5file

On the contrary to DataSaverLoader, does include navigation axes stored elsewhere in the h5file (for instance if saved from the DAQ_Scan)

Parameters:

h5saver (Union[H5SaverLowLevel, Path])

Attributes:

h5saver

Methods

get_nav_group(where)
get_node(where[, name])	Convenience method to get node
load_data(where[, with_bkg, load_all])	Load data from a node (or channel node)
walk_nodes([where])	Return a Node generator iterating over the h5file content

close_file
load_all

get_nav_group(where)

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself

Return type:

Optional[Node]

Returns:

• GROUP (returns the group named SPECIAL_GROUP_NAMES[``’nav_axes’``] holding all NavAxis for)

• those data

See also

SPECIAL_GROUP_NAMES

get_node(where, name=None)

Convenience method to get node

Return type:

Node

load_data(where, with_bkg=False, load_all=False)

Load data from a node (or channel node)

Loaded data contains also nav_axes if any and with optional background subtraction

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• with_bkg (bool) – If True will attempt to substract a background data node before loading

• load_all (bool) – If True, will load all data hanging from the same parent node

Return type:

DataWithAxes

walk_nodes(where='/')

Return a Node generator iterating over the h5file content

8.3.1.5. Browsing Data

Using the H5Backend it is possible to write scripts to easily access a hdf5 file content. However, PyMoDAQ includes a dedicated hdf5 viewer understanding dedicated metadata and therefore displaying nicely the content of the file, see Data Browsing: the H5Browser module. Two objects can be used to browse data: H5BrowserUtil and H5Browser. H5BrowserUtil gives you methods to quickly (in a script) get info and data from your file while the H5Browser adds a UI to interact with the hdf5 file.

Created the 15/11/2022

@author: Sebastien Weber

class pymodaq_gui.h5modules.browsing.H5Browser(parent, h5file=None, h5file_path=None, backend='tables')

UI used to explore h5 files, plot and export subdatas

Parameters:

• parent (QMainWindow) – either a QWidget or a QMainWindow

• h5file (h5file instance) – exact type depends on the backend

• h5file_path (str or Path) – if specified load the corresponding file, otherwise open a select file dialog

• backend (str) – either ‘tables, ‘h5py’ or ‘h5pyd’

See also

H5Backend, H5Backend

add_comments(status, comment='')

Add comments to a node

Parameters:

• status (bool)

• comment (str) – The comment to be added in a comment attribute to the current node path

See also

current_node_path

check_version()

Check version of PyMoDAQ to assert if file is compatible or not with the current version of the Browser

export_data()

Opens a dialog to export data

See also

H5BrowserUtil.export_data

get_tree_node_path()

Get the node path of the currently selected node in the UI

populate_tree()

Init the ui-tree and store data into calling the h5_tree_to_Qtree convertor method

See also

h5tree_to_QTree, update_status

quit_fun()

show_h5_data(item, with_bkg=False, plot_all=False)

Parameters:

• item

• with_bkg

• plot_all

class pymodaq_gui.h5modules.browsing.H5BrowserUtil(backend='tables')

Utility object to interact and get info and data from a hdf5 file

Inherits H5Backend and all its functionalities

Parameters:

backend (str) – The used hdf5 backend: either tables, h5py or h5pyd

export_data(node_path='/', filesavename='datafile.h5', filter=None)

Initialize the correct exporter and export the node

get_h5_attributes(node_path)

get_h5file_scans(where='/')

Get the list of the scan nodes in the file

Parameters:

where (str) – the path in the file

Returns:

dict with keys: scan_name, path (within the file) and data (the live scan png image)

Return type:

list of dict

8.3.1.6. Module savers

Created the 23/11/2022

@author: Sebastien Weber

class pymodaq.utils.h5modules.module_saving.ActuatorSaver(module)

Implementation of the ModuleSaver class dedicated to DAQ_Move modules

Parameters:

• h5saver

• module (DAQ_Move)

class pymodaq.utils.h5modules.module_saving.DetectorEnlargeableSaver(module, enl_axis_names=None, enl_axis_units=None)

Implementation of the ModuleSaver class dedicated to DAQ_Viewer modules in order to save enlargeable data

Parameters:

module (DAQ_Viewer)

class pymodaq.utils.h5modules.module_saving.DetectorExtendedSaver(module, extended_shape)

Implementation of the ModuleSaver class dedicated to DAQ_Viewer modules in order to save enlargeable data

Parameters:

module (DAQ_Viewer)

class pymodaq.utils.h5modules.module_saving.DetectorSaver(module)

Implementation of the ModuleSaver class dedicated to DAQ_Viewer modules

Parameters:

module (DAQ_Viewer)

add_bkg(where, data_bkg)

Adds a DataToExport as a background node in the h5file

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• data_bkg (DataToExport) – The data to be saved as background

class pymodaq.utils.h5modules.module_saving.LoggerSaver(*args, **kwargs)

Implementation of the ModuleSaver class dedicated to H5Logger module

H5Logger is the special logger to h5file of the DAQ_Logger extension

Parameters:

• h5saver

• module

add_data(dte)

Add data to it’s corresponding control module

The name of the control module is the DataToExport name attribute

class pymodaq.utils.h5modules.module_saving.ModuleSaver(*args, **kwargs)

Abstract base class to save info and data from main modules (DAQScan, DAQViewer, DAQMove, …)

flush()

Flush the underlying file

get_last_node(where=None)

Get the last node corresponding to this particular Module instance

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• new (bool) – if True force the creation of a new indexed node of this class type if False return the last node (or create one if None)

Returns:

GROUP

Return type:

the Node associated with this module which should be a GROUP node

get_set_node(where=None, name=None)

Get or create the node corresponding to this particular Module instance

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• new (bool) – if True force the creation of a new indexed node of this class type if False return the last node (or create one if None)

Returns:

GROUP

Return type:

GROUP

class pymodaq.utils.h5modules.module_saving.ScanSaver(*args, **kwargs)

Implementation of the ModuleSaver class dedicated to DAQScan module

Parameters:

• h5saver

• module

get_set_node(where=None, new=False)

Get the last group scan node

Get the last Scan Group or create one get the last Scan Group if: * there is one already created * new is False

Parameters:

• where (Union[Node, str]) – the path of a given node or the node itself

• new (bool)

Returns:

GROUP

Return type:

GROUP