8.3.1. Hdf5 module and classes

8.3.1.1. Hdf5 backends

The H5Backend is a wrapper around three hdf5 python packages: pytables, h5py and h5pyd. It allows seamless integration of any of these with PyMoDAQ features.

class pymodaq_data.h5modules.backends.H5Backend(backend='tables')[source]

Attributes:

filename
h5file
is_swmr_active: Return True if SWMR mode is currently active on the file.
is_swmr_capable: Return True if the current backend supports SWMR mode.
is_swmr_compatible: Return True if the open file was created with SWMR support.
swmr_mode

Methods

`add_group`(group_name, group_type, where[, ...])	Add a node in the h5 file tree of the group type :type group_name: `str` :param group_name: a custom name for this group :type group_name: `str` :param group_type: one of the possible values of GroupType, should be enforced by higher level modules not here :type group_type: `Union`[`GroupType`, `str`] :type where: `str` or `Node` :param where: parent node where to create the new group :type where: `str` or `Node` :type metadata: `dict` :param metadata: extra metadata to be saved with this new group node :type metadata: `dict`
`close_file`()	Flush data and close the h5file
`create_earray`(where, name, dtype[, ...])	create enlargeable arrays from data with a given shape and of a given type.
`create_vlarray`(where, name, dtype[, title])	create variable data length and type and enlargeable 1D arrays
`define_compression`(compression, compression_opts)	Define cmpression library and level of compression :type compression: `(str) either gzip` and `zlib are supported here as they are compatible` :param compression: but zlib is used by pytables while gzip is used by h5py :type compression: `(str) either gzip` and `zlib are supported here as they are compatible` :param compression_opts (int): :type compression_opts (int): `0` to `9 0`: `None`, `9`: `maximum compression`
`enable_swmr`()	Activate SWMR mode on the open h5py file.
`finalize_swmr`([keep_open])	End SWMR by closing the file, reopening in 'a' mode, and reconciling deferred attrs.
`get_children`(where)	Get a dict containing all children node hanging from where with their name as keys and types among Node, CARRAY, EARRAY, VLARRAY or StringARRAY
`get_node_name`(node)	return node name
`get_node_path`(node)	return node path
`get_set_group`(where, name[, title])	Retrieve or create (if absent) a node group
`is_node_in_group`(where, name)	Check if a given node with name is in the group defined by where (comparison on lower case strings)
`reconcile_swmr_attrs`()	Walk all EARRAY/VLARRAY nodes and update attrs['shape'] from actual data.
`set_backend`(backend)	Switch the active backend, closing any open file first.
`walk_nodes`(where[, depth, only_groups])	Node Generator recursively iterating in the tree starting from where down to the specified depth (counted from where).

create_carray
flush
get_attr
get_group_by_title
get_node
get_parent_node
has_attr
isopen
open_file
read
root
save_file_as
set_attr
walk_groups

add_group(group_name, group_type, where, title='', metadata=None)[source]

Add a node in the h5 file tree of the group type :type group_name: str :param group_name: a custom name for this group :type group_name: str :param group_type: one of the possible values of GroupType, should be enforced by higher level modules not here :type group_type: Union[GroupType, str] :type where: str or Node :param where: parent node where to create the new group :type where: str or Node :type metadata: dict :param metadata: extra metadata to be saved with this new group node :type metadata: dict

Returns:: newly created group node
Return type:: GROUP

close_file()[source]: Flush data and close the h5file

create_earray(where, name, dtype, data_shape=None, title='')[source]: create enlargeable arrays from data with a given shape and of a given type. The array is enlargeable along the first dimension

create_vlarray(where, name, dtype, title='')[source]

create variable data length and type and enlargeable 1D arrays

Parameters:

where (str) – group location in the file where to create the array node
name (str) – name of the array
dtype (dtype) – numpy dtype style, for particular case of strings, use dtype=’string’
title (str) – node title attribute (written in capitals)

Return type:

VLARRAY

define_compression(compression, compression_opts)[source]: Define cmpression library and level of compression :type compression: (str) either gzip and zlib are supported here as they are compatible :param compression: but zlib is used by pytables while gzip is used by h5py :type compression: (str) either gzip and zlib are supported here as they are compatible :param compression_opts (int): :type compression_opts (int): 0 to 9 0: None, 9: maximum compression

enable_swmr()[source]

Activate SWMR mode on the open h5py file.

Must be called after all groups/datasets have been created. Raises RuntimeError if backend is not h5py or file was not opened with swmr_mode=True. Idempotent: does nothing if already enabled.

finalize_swmr(keep_open=False)[source]

End SWMR by closing the file, reopening in ‘a’ mode, and reconciling deferred attrs.

After SWMR mode, attrs[‘shape’] on EARRAY/VLARRAY nodes may be stale. This method closes the file (ending SWMR), reopens it in append mode, and updates all deferred attributes.

Parameters:: keep_open (bool) – If True, leaves the file open in ‘a’ mode after reconciling. If False (default), closes the file after reconciling.

get_children(where)[source]

Get a dict containing all children node hanging from where with their name as keys and types among Node, CARRAY, EARRAY, VLARRAY or StringARRAY

Parameters:: where (str or Node) – see h5py and pytables documentation on nodes, and Node objects of this module
Returns:: keys are children node names, values are the children nodes
Return type:: dict[str, Node]

See also

GROUP.children_name()

get_node_name(node)[source]

return node name

Parameters:: node (str or Node) – see h5py and pytables documentation on nodes
Returns:: name of the node
Return type:: str

get_node_path(node)[source]

return node path

Parameters:: node (str or Node) – see h5py and pytables documentation on nodes
Returns:: full path of the node
Return type:: str

get_set_group(where, name, title='', **kwargs)[source]

Retrieve or create (if absent) a node group

Get attributed to the class attribute current_group

Parameters:

where (str or Node) – path or parent node instance
name (str) – group node name
title (str) – node title
kwargs (dict) – any other metadata related to this node (for example: origin)

Returns:

the group node

Return type:

GROUP

is_node_in_group(where, name)[source]

Check if a given node with name is in the group defined by where (comparison on lower case strings)

Parameters:

where (str or Node) – path or parent node instance
name (str) – group node name

Returns:

True if node exists, False otherwise

Return type:

bool

reconcile_swmr_attrs()[source]

Walk all EARRAY/VLARRAY nodes and update attrs[‘shape’] from actual data.

Called after SWMR is ended (file closed and reopened in ‘a’ mode) to fix deferred attribute writes that were skipped during SWMR.

set_backend(backend)[source]

Switch the active backend, closing any open file first.

Updates both self.backend (the name string) and self.h5_library (the imported module), which both need to be consistent for file operations.

Parameters:: backend (str) – One of 'tables', 'h5py', or 'h5pyd'.

walk_nodes(where, depth=None, only_groups=False)[source]

Node Generator recursively iterating in the tree starting from where down to the specified depth (counted from where).

Parameters:

where (str | Node) – The starting node of the iteration
depth (int) – The depth of the iteration in the h5 tree
only_groups (bool) – if False (default) return all nodes otherwise retruns only GROUP instances

Yields:

Node

property is_swmr_active: Return True if SWMR mode is currently active on the file.

property is_swmr_capable: Return True if the current backend supports SWMR mode.

property is_swmr_compatible: Return True if the open file was created with SWMR support.

8.3.1.2. Low Level saving

H5SaverLowLevel is the base saving class providing file creation and node management. H5SaverBase and H5Saver build on it and integrate with the PyMoDAQ Framework. H5SaverBase adds a ParameterManager with settings for backend selection, SWMR options, file paths and compression. H5Saver inherits H5SaverBase and adds Qt signals (new_file_sig, file_changed_sig) and a file browser dialog. To save and load data, one should use higher level objects, see High Level saving/loading.

Created the 15/11/2022

@author: Sebastien Weber

class pymodaq_data.h5modules.saving.H5SaverLowLevel(save_type='scan', backend=None)[source]

Object containing basic methods in order to structure and interact with a h5file compatible with the h5browser

See also

H5Browser

Variables:

h5_file (pytables hdf5 file) – object used to save all datas and metadas
h5_file_path (str or Path) – The file path

classmethod from_file(path, save_type='scan', new_file=False, metadata=None, swmr_mode=False, mode='a')[source]

Create and initialise an H5SaverLowLevel from a file path.

Convenience factory that combines the constructor and init_file() call into a single expression.

Parameters:

path (Union[Path, str]) – Full path to the HDF5 file.
save_type (SaveType) – Type label stored in the file (default 'scan').
new_file (bool) – If True a new file is created, otherwise the existing file is opened for appending.
metadata (dict) – Extra attributes written to the raw-data group on creation.
swmr_mode (bool) – If True, prepare the file for SWMR (h5py backend only)
mode (str) – valid if new_file is False. Then could be ‘r’ for readonly or ‘a’ to append data

Returns:

A fully initialised instance ready for reading/writing.

Return type:

H5SaverLowLevel

add_act_group(where, title='', settings_as_xml='', metadata=None)[source]: Add a new group of type detector .. seealso:: add_incremental_group

add_array(where, name, data_type, array_to_save=None, data_shape=None, array_type=None, fill_value=None, data_dimension=None, scan_shape=(), add_scan_dim=False, enlargeable=False, title='', metadata={})[source]

save data arrays on the hdf5 file together with metadata

Parameters:

where (Union[GROUP, str]) – node where to save the array
name (str) – name of the array in the hdf5 file
data_type (DataType) – mandatory so that the h5Browser can interpret correctly the array
data_shape (tuple) – the shape of the array to save, mandatory if array_to_save is None
data_dimension (DataDim) – The data’s dimension
scan_shape (tuple) – the shape of the scan dimensions
title (str) – the title attribute of the array node
array_to_save (ndarray) – data to be saved in the array. If None, array_type and data_shape should be specified in order to init correctly the memory
array_type (dtype) – eg np.float, np.int32 …
fill_value (float or int) – value to be used to fill the array if array_to_save is None
enlargeable (bool) – if False, data are saved as a CARRAY, otherwise as a EARRAY (for ragged data, see add_string_array)
metadata (dict) – dictionnary whose keys will be saved as the array attributes
add_scan_dim (bool) – if True, the scan axes dimension (scan_shape iterable) is prepended to the array shape on the hdf5 In that case, the array is usually initialized as zero and further populated

Return type:

CARRAY or EARRAY

See also

add_data, add_string_array

add_ch_group(where, title='', settings_as_xml='', metadata=None)[source]: Add a new group of type channel .. seealso:: add_incremental_group

add_data_group(where, data_dim, title='', settings_as_xml='', metadata=None, group_name=None)[source]

Creates a group node at given location in the tree

Parameters:

where (group node) – where to create data group
data_dim (DataDim) – the dimensionality of the data group
title (str, optional) – a title for this node, will be saved as metadata
settings_as_xml (str, optional) – XML string created from a Parameter object to be saved as metadata
metadata (dict, optional) – will be saved as a new metadata attribute with name: key and value: dict value
group_name (str) – the name of the group to create if None, the name of the DataDim enum is used (default)

Returns:

group

Return type:

group node

See also

add_group()

add_det_group(where, title='', settings_as_xml='', metadata=None)[source]: Add a new group of type detector .. seealso:: add_incremental_group

add_generic_group(where='/RawData', title='', settings_as_xml='', metadata=None, group_type=GroupType.scan)[source]

Add a new group of type given by the input argument group_type

At creation adds the attributes description to be used elsewhere

See also

add_incremental_group

add_incremental_group(group_type, where, title='', settings_as_xml='', metadata=None)[source]

Add a node in the h5 file tree of the group type with an increment in the given name :param group_type: one of the possible values of group_types :type group_type: Union[str, GroupType, Enum] :type where: str or node :param where: parent node where to create the new group :type where: str or node :type title: str :param title: node title :type title: str :type settings_as_xml: str :param settings_as_xml: XML string containing Parameter representation :type settings_as_xml: str :type metadata: dict :param metadata: extra metadata to be saved with this new group node :type metadata: dict

Returns:: node
Return type:: newly created group node

add_move_group(where, title='', settings_as_xml='', metadata=None)[source]: Add a new group of type actuator .. seealso:: add_incremental_group

add_scan_group(where='/RawData', title='', settings_as_xml='', metadata=None)[source]

Add a new group of type scan

deprecated, use add_generic_group with a group type as GroupType.scan

get_groups(where, group_type)[source]: Get all groups hanging from a Group and of a certain type

get_node_from_attribute_match(where, attr_name, attr_value)[source]: Get a Node starting from a given node (Group) matching a pair of node attribute name and value

get_node_from_title(where, title)[source]: Get a Node starting from a given node (Group) matching the given title

get_set_group(where, name, title='', **kwargs)[source]

Get the group located at where if it exists otherwise creates it

This also set the _current_group property

get_set_logger(where=None)[source]

Retrieve or create (if absent) a logger enlargeable array to store logs Get attributed to the class attribute logger_array :param where: location within the tree where to save or retrieve the array :type where: Node

Returns:: enlargeable array accepting strings as elements
Return type:: VLARRAY

init_file(file_name, raw_group_name='RawData', new_file=False, metadata=None, swmr_mode=False, mode='a')[source]

Initializes a new h5 file,

Should have an extension with h5 in it.

Parameters:

file_name (Path) – a complete Path pointing to a h5 file
raw_group_name (str) – Base node name
new_file (bool) – If True create a new file, otherwise append to a potential existing one
metadata (dict) – A dictionary to be saved as attributes
swmr_mode (bool) – If True, prepare the file for SWMR (h5py backend only)
mode (str) – valid if new_file is False. Then could be ‘r’ for readonly or ‘a’ to append data

Returns:

True if new file has been created, False otherwise

Return type:

bool

set_swmr_flush_interval(interval)[source]

Set how often to flush data for SWMR readers.

Parameters:: interval (int) – 0 = flush only at end, N = every N writes

tick_flush()[source]

Increment the write counter and flush if the interval is reached.

No-op if SWMR is not active or flush interval is 0. To be called by data savers after each logical data write.

Created the 15/11/2022

@author: Sebastien Weber

class pymodaq_gui.h5modules.saving.H5Saver(*args, **kwargs)[source]

status_sig: Signal: emits a signal of type Threadcommand in order to senf log information to a main UI
new_file_sig: Signal: emitted to let the program know when the user pressed the new file button on the UI
file_changed_sig: Signal: emits a str (file path) whenever the active h5 file changes (browse, new, reopen)

open_file_dialog_result(file_path)[source]: Open the h5 file the user just picked via the ‘current_h5_file’ browsepath’s browse button (not named open_file: that name is already the inherited low-level H5Backend.open_file(fullpathname, mode, …)).

class pymodaq_gui.h5modules.saving.H5SaverBase(save_type='scan', backend=None)[source]

Object containing all methods in order to save datas in a hdf5 file with a hierarchy compatible with the H5Browser. The saving parameters are contained within a Parameter object: self.settings that can be displayed on a UI using the widget self.settings_tree. At the creation of a new file, a node group named Raw_data and represented by the attribute raw_group is created and set with a metadata attribute:

‘type’ given by the save_type class parameter

The root group of the file is then set with a few metadata:

‘pymodaq_version’ the current pymodaq version, e.g. 1.6.2
‘pymodaq_data_version’ the current pymodaq_data version, e.g. 0.0.1
‘file’ the file name
‘date’ the current date
‘time’ the current time

All data will then be saved under this node in various groups

See also

H5Browser

Parameters:

h5_file (pytables hdf5 file) – object used to save all datas and metadas
h5_file_path (str or Path) – Signal signal represented by a float. Is emitted each time the hardware reached the target position within the epsilon precision (see comon_parameters variable)
save_type (str) – an element of the enum module attribute SaveType * ‘scan’ is used for DAQScan module and should be used for similar application * ‘detector’ is used for DAQ_Viewer module and should be used for similar application * ‘custom’ should be used for customized applications

Variables:

settings (Parameter) – Parameter instance (pyqtgraph) containing all settings (could be represented using the settings_tree widget)
settings_tree (ParameterTree) – Widget representing as a Tree structure, all the settings defined in the class preamble variable params

classmethod find_part_in_path_and_subpath(base_dir, part='', create=False, increment=True)[source]

Find path from part time.

Parameters	Type	Description
base_dir	Path object	The directory to browse
part	string	The date of the directory to find/create
create	boolean	Indicate the creation flag of the directory

Returns:: found path from part
Return type:: Path object

classmethod set_current_scan_path(base_dir, base_name='Scan', update_h5=False, next_scan_index=0, create_scan_folder=False, create_dataset_folder=True, curr_date=None, ind_dataset=None)[source]

Parameters:

base_dir
base_name
update_h5
next_scan_index
create_scan_folder
create_dataset_folder

get_last_scan()[source]

Gets the last scan node within the h5_file and under the raw_group

Returns:: scan_group
Return type:: pytables group or None

get_scan_index()[source]: return the scan group index in the “scan templating”: Scan000, Scan001 as an integer

init_file(update_h5=False, custom_naming=False, addhoc_file_path=None, metadata={}, mode='a')[source]

Initializes a new h5 file.

Could set the h5_file attributes as:

a file with a name following a template if custom_naming is False and addhoc_file_path is None
a file within a name set using a file dialog popup if custom_naming is True
a file with a custom name if addhoc_file_path is a Path object or a path string

Parameters:

update_h5 (bool) – create a new h5 file with name specified by other parameters if false try to open an existing file and will append new data to it or just read it
custom_naming (bool) – if True, a selection file dialog opens to set a new file name
addhoc_file_path (Path or str) – supplied name by the user for the new file
metadata (dict) – dictionnary with pair of key, value that should be saved as attributes of the root group
mode (str) – valid if update_h5 is False. Then could be ‘r’ for readonly or ‘a’ to append data

Returns:

True if new file has been created, False otherwise

Return type:

bool

load_file(base_path=None, file_path=None)[source]

Opens a file dialog to select a h5file saved on disk to be used

Parameters:

base_path
file_path

See also

init_file()

update_file_paths(update_h5=False)[source]

Parameters:

update_h5 (bool) – if True, will increment the file name and eventually the current scan index if False, get the current scan index in the h5 file

Returns:

scan_path (Path)
current_filename (str)
dataset_path (Path)

They both inherit from the ParameterManager MixIn class that deals with Parameter and ParameterTree.

8.3.1.3. SWMR utilities

When using the h5py backend with SWMR mode enabled, the following utility functions help readers access the file while a scan is in progress.

pymodaq_data.h5modules.open_h5_file_for_reading(filepath, swmr='auto', locking=None)[source]

Open an HDF5 file for reading, automatically handling SWMR mode.

This utility function handles the complexity of opening HDF5 files that may or may not be currently being written with SWMR mode.

Parameters:

filepath (str or Path) – Path to the HDF5 file
swmr (bool or 'auto', optional) –
- ‘auto’ (default): Try to detect if SWMR is needed
- True: Force SWMR reader mode
- False: Open normally without SWMR
locking (bool or None, optional) – File locking mode. None uses h5py default. Set to False on Windows if you get locking errors.

Returns:

(h5py.File, is_swmr_active) - The file handle and whether SWMR is active

Return type:

tuple

Examples

>>> # Open a file being written by PyMoDAQ scan
>>> f, is_swmr = open_h5_file_for_reading("scan_data.h5")
>>> if is_swmr:
...     ds = f['path/to/data']
...     ds.id.refresh()  # Call refresh to see latest data
>>> f.close()

pymodaq_data.h5modules.is_file_swmr_active(filepath)[source]

Check if an HDF5 file is currently being written with SWMR mode.

Parameters:: filepath (str or Path) – Path to the HDF5 file
Returns:: True if the file has swmr_active=True attribute, False otherwise
Return type:: bool

Utilities for reading HDF5 files written with SWMR (Single Writer Multiple Reader) mode.

8.3.1.3.1. Typical usage

Open the file once, collect dataset references, then poll in a loop:

from pymodaq_data.h5modules import open_h5_file_for_reading from pymodaq_data.h5modules.swmr import collect_datasets, refresh_cached

f, is_swmr = open_h5_file_for_reading(“scan.h5”) cache = collect_datasets(f[“RawData”]) # dict[str, h5py.Dataset]

while acquiring:
refresh_cached(cache) data = cache[“/RawData/CH000/Data0D/Data00/data”][:]

pymodaq_data.h5modules.swmr.collect_datasets(group)[source]

Walk group recursively and return a mapping of absolute path → dataset.

The returned dict can be passed to refresh_cached() on every poll cycle instead of re-walking the tree each time.

Parameters:: group (Group) – Any h5py.Group (or h5py.File, which is also a group).
Returns:: {"/absolute/path": h5py.Dataset, ...} for every dataset found under group.
Return type:: Dict[str, Dataset]

Examples

>>> f, _ = open_h5_file_for_reading("scan.h5")
>>> cache = collect_datasets(f["RawData"])
>>> cache.keys()
dict_keys(['/RawData/CH000/Data0D/Data00/data', ...])

pymodaq_data.h5modules.swmr.refresh_cached(cache)[source]

Refresh every dataset in a pre-built cache dict.

This is the fast path for polling loops: call collect_datasets() once to build cache, then call this function on each iteration.

Parameters:: cache (Dict[str, Dataset]) – A {path: h5py.Dataset} dict as returned by collect_datasets().
Return type:: None

Examples

>>> cache = collect_datasets(f["RawData"])
>>> while acquiring:
...     refresh_cached(cache)
...     latest_row = cache["/RawData/CH000/Data0D/Data00/data"][-1]

pymodaq_data.h5modules.swmr.refresh_datasets(group)[source]

Refresh every dataset under group so that SWMR readers see the latest data written by the writer process.

This is a convenience wrapper for one-shot use. For polling loops prefer collect_datasets() + refresh_cached() to avoid re-walking the tree on every iteration.

Parameters:: group (Group) – Any h5py.Group (or h5py.File).
Return type:: None

Notes

refresh() is a metadata/chunk-index call; it does not read the actual data. The data is only transferred when you access dataset elements (ds[:], ds[-1], etc.).

8.3.1.4. High Level saving/loading

Each PyMoDAQ’s data type: Axis, DataWithAxes, DataToExport (see What is PyMoDAQ’s Data?) is associated with its saver/loader counterpart. These objects ensures that all metadata necessary for an exact regeneration of the data is being saved at the correct location in the hdf5 file hierarchy. The AxisSaverLoader, DataSaverLoader, DataToExportSaver all derive from an abstract class: DataManagement allowing the manipulation of the nodes and making sure the data type is defined.

8.3.1.4.1. Base data class saver/loader

Created the 21/11/2022

@author: Sebastien Weber

class pymodaq_data.h5modules.data_saving.AxisSaverLoader(h5saver)[source]

Specialized Object to save and load Axis object to and from a h5file

Parameters:: h5saver (H5SaverLowLevel)
Variables:: data_type (DataType) – The enum for this type of data, here ‘axis’

add_axis(where, axis, enlargeable=False)[source]

Write Axis info at a given position within a h5 file

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself
axis (Axis) – the Axis object to add as a node in the h5file
enlargeable (bool) – Specify if the underlying array will be enlargebale

get_axes(where)[source]

Return a list of Axis objects from the Axis Nodes hanging from (or among) a given Node

Parameters:: where (Union[Node, str]) – the path of a given node or the node itself
Returns:: List[Axis]
Return type:: List[Axis]

load_axis(where)[source]

create an Axis object from the data and metadata at a given node if of data_type: ‘axis

Parameters:: where (Union[Node, str]) – the path of a given node or the node itself
Return type:: Axis

class pymodaq_data.h5modules.data_saving.DataManagement(*args, **kwargs)[source]

Base abstract class to be used for all specialized object saving and loading data to/from a h5file

Variables:: data_type (DataType) – The enum for this type of data, here abstract and should be redefined

get_last_node_name(where)[source]

Get the last node name among the ones already saved

Parameters:: where (Union[str, Node]) – the path of a given node or the node itself
Returns:: str
Return type:: Optional[str]

property raw_group: Node: Get the base RawGroup where raw data should be saved

class pymodaq_data.h5modules.data_saving.DataSaverLoader(h5saver, new_file=False, metadata=None, save_type=SaveType.custom, swmr_mode=False)[source]

Specialized Object to save and load DataWithAxes object to and from a h5file

Parameters:: h5saver (Union[H5SaverLowLevel, Path])
Variables:: data_type (DataType) – The enum for this type of data, here ‘data’

add_data(where, data, save_axes=True, **kwargs)[source]

Adds Array nodes to a given location adding eventually axes as others nodes and metadata

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself
data (DataWithAxes)
save_axes (bool)

get_axes(where)[source]

Parameters:: where (Union[Node, str]) – the path of a given node or the node itself
Return type:: List[Axis]

get_data_arrays(where, with_bkg=False, load_all=False)[source]

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself
with_bkg (bool) – If True try to load background node and return the array with background subtraction
load_all (bool) – If True load all similar nodes hanging from a parent

Return type:

List[ndarray]

isopen()[source]

Get the opened status of the underlying hdf5 file

Return type:: bool

load_data(where, with_bkg=False, load_all=False)[source]

Return a DataWithAxes object from the Data and Axis Nodes hanging from (or among) a given Node

Does not include navigation axes stored elsewhere in the h5file. The node path is stored in the DatWithAxis using the attribute path

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself
with_bkg (bool) – If True try to load background node and return the data with background subtraction
load_all (bool) – If True, will load all data hanging from the same parent node

See also

load_data

Return type:: DataWithAxes

class pymodaq_data.h5modules.data_saving.DataToExportSaver(h5saver, save_type=SaveType.custom, new_file=False, metadata=None)[source]

Object used to save DataToExport object into a h5file following the PyMoDAQ convention

Parameters:: h5saver (Union[H5SaverLowLevel, Path, str])

static channel_formatter(ind)[source]: All DataWithAxes included in the DataToExport will be saved into a channel group indexed and formatted as below

add_data(where, data, settings_as_xml='', **kwargs)[source]

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself
data (DataToExport)
settings_as_xml (str) – The settings parameter as an XML string
Arguments (Keyword) – all extra metadata to be saved in the group node where data will be saved

isopen()[source]

Get the opened status of the underlying hdf5 file

Return type:: bool

8.3.1.4.2. Specific data class saver/loader

Some more dedicated objects are derived from the objects above. They allow to add background data, Extended arrays (arrays that will be populated after creation, for instance for a scan) and Enlargeable arrays (whose final length is not known at the moment of creation, for instance when logging or continuously saving)

Created the 21/11/2022

@author: Sebastien Weber

class pymodaq_data.h5modules.data_saving.BkgSaver(h5saver)[source]

Specialized Object to save and load DataWithAxes background object to and from a h5file

Parameters:: hsaver (H5SaverLowLevel)
Variables:: data_type (DataType) – The enum for this type of data, here ‘bkg’

class pymodaq_data.h5modules.data_saving.DataEnlargeableSaver(h5saver, enl_axis_names=('nav axis',), enl_axis_units=('',))[source]

Specialized Object to save and load enlargeable DataWithAxes saved object to and from a h5file

Particular case of DataND with a single nav_indexes parameter will be appended as chunks of signal data

Parameters:: h5saver (Union[H5SaverLowLevel, Path])
Variables:: data_type (DataType) – The enum for this type of data, here ‘data_enlargeable’

Notes

To be used to save data from a timed logger (DAQViewer continuous saving or DAQLogger extension) or from an adaptive scan where the final shape is unknown or other module that need this feature

add_data(where, data, axis_values=None, **kwargs)[source]

Append data to an enlargeable array node

Data of dim (0, 1 or 2) will be just appended to the enlargeable array.

Uniform DataND with one navigation axis of length (Lnav) will be considered as a collection of Lnav signal data of dim (0, 1 or 2) and will therefore be appended as Lnav signal data

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself
data (DataWithAxes)
axis_values (Iterable[float]) – the new spread axis values added to the data if None the axes are not added to the h5 file

class pymodaq_data.h5modules.data_saving.DataExtendedSaver(h5saver, extended_shape, fill_value=None)[source]

Specialized Object to save and load DataWithAxes saved object to and from a h5file in extended arrays

Parameters:

h5saver (H5SaverLowLevel)
extended_shape (Iterable[int]) – the extra shape compared to the data the h5array will have

Variables:

data_type (DataType) – The enum for this type of data, here ‘data’

add_data(where, data, indexes, distribution=DataDistribution.uniform)[source]

Adds given DataWithAxes at a location within the initialized h5 array

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself
data (DataWithAxes)
indexes (Iterable[int]) – indexes where to save data in the init h5array (should have the same length as extended_shape and with values coherent with this shape

class pymodaq_data.h5modules.data_saving.DataToExportEnlargeableSaver(h5saver, enl_axis_names=None, enl_axis_units=None, axis_name='nav axis', axis_units='')[source]

Generic object to save DataToExport objects in an enlargeable h5 array

The next enlarged value should be specified in the add_data method

Parameters:

h5saver (Union[Path, H5SaverLowLevel])
enl_axis_names (Iterable[str]) – The names of the enlargeable axis, default [‘nav_axis’]
enl_axis_units (Iterable[str]) – The names of the enlargeable axis, default [‘’]
axis_name (str) – the name of the enlarged axis array
axis_units (str) – the units of the enlarged axis array

add_data(where, data, axis_values=None, axis_value=None, settings_as_xml='', **kwargs)[source]

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself
data (DataToExport) – The data to be saved into an enlargeable array
axis_values (List[Union[float, ndarray]]) – The next value (or values) of the enlarged axis
axis_value (Union[float, ndarray]) – The next value (or values) of the enlarged axis
settings_as_xml (str) – The settings parameter as an XML string
Arguments (Keyword) – all extra metadata to be saved in the group node where data will be saved

class pymodaq_data.h5modules.data_saving.DataToExportExtendedSaver(h5saver, extended_shape, fill_value=None)[source]

Object to save DataToExport at given indexes within arrays including extended shape

Mostly used for data generated from the DAQScan

Parameters:

h5saver (H5SaverLowLevel)
extended_shape (Iterable[int]) – the extra shape compared to the data the h5array will have

add_data(where, data, indexes, distribution=DataDistribution.uniform, settings_as_xml='', **kwargs)[source]

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself
data (DataToExport)
indexes (Iterable[int]) – indexes where to save data in the init h5array (should have the same length as extended_shape and with values coherent with this shape
settings_as_xml (str) – The settings parameter as an XML string
Arguments (Keyword) – all extra metadata to be saved in the group node where data will be saved

add_nav_axes(where, axes)[source]

Used to add navigation axes related to the extended array

Notes

For instance the scan axes in the DAQScan

class pymodaq_data.h5modules.data_saving.DataToExportTimedSaver(h5saver)[source]

Specialized DataToExportEnlargeableSaver to save data as a function of a time axis

Only one element ca be added at a time, the time axis value are enlarged using the data to be added timestamp

Notes

This object is made for continuous saving mode of DAQViewer and logging to h5file for DAQLogger

8.3.1.5. Specialized loading

Data saved from a DAQ_Scan will naturally include navigation axes shared between many different DataWithAxes (as many as detectors/channels/ROIs). They are therefore saved at the root of the scan node and cannot be retrieved using the standard data loader. Hence this DataLoader object.

class pymodaq_data.h5modules.data_saving.DataLoader(h5saver, swmr_mode=False)[source]

Specialized Object to load DataWithAxes / DataToExport objects from a h5file

On the contrary to DataSaverLoader, does include navigation axes stored elsewhere in the h5file (for instance if saved from the DAQ_Scan)

Parameters:

h5saver (Union[H5SaverLowLevel, Path])

Attributes:

h5saver
raw_group: Get the base RawGroup where raw data should be saved

Methods

`get_nav_group`(where)
`get_node`(where[, name])	Convenience method to get node
`load_data`(where[, with_bkg, load_all])	Load data from a node (or channel node)
`load_data_from_name_origin`(name[, origin, ...])	Load data from a node if this data as the given name and origin
`walk_nodes`([where, depth, only_groups])	Return a Node generator iterating over the h5file content

close_file
load_all

get_nav_group(where)[source]

Parameters:: where (Union[Node, str]) – the path of a given node or the node itself
Returns:: the group named SPECIAL_GROUP_NAMES['nav_axes'] holding all NavAxis for those data
Return type:: Optional[Node]

See also

SPECIAL_GROUP_NAMES

get_node(where, name=None)[source]

Convenience method to get node

Return type:: Node

load_data(where, with_bkg=False, load_all=False)[source]

Load data from a node (or channel node)

Loaded data contains also nav_axes if any and with optional background subtraction

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself
with_bkg (bool) – If True will attempt to substract a background data node before loading
load_all (bool) – If True, will load all data hanging from the same parent node

Return type:

DataWithAxes

load_data_from_name_origin(name, origin='', where=None, with_bkg=False, load_all=True)[source]

Load data from a node if this data as the given name and origin

Parameters:

name (str) – The name of the data (stored in the title attribute)
origin (str) – The origin of the data
where (Union[GROUP, Node, str]) – If specified start to look for matching Dwa at where Node
with_bkg (bool) – If True load with bkg substraction if any
load_all (bool) – if True load all channels of the parent node

Return type:

DataWithAxes

walk_nodes(where='/', depth=None, only_groups=False)[source]

Return a Node generator iterating over the h5file content

Return type:: Iterable[Node]

property raw_group: Node

Get the base RawGroup where raw data should be saved

Convenience method

8.3.1.6. Browsing Data

Using the H5Backend it is possible to write scripts to easily access a hdf5 file content. However, PyMoDAQ includes a dedicated hdf5 viewer understanding dedicated metadata and therefore displaying nicely the content of the file, see Data Browsing: the H5Browser module. Two objects can be used to browse data: H5BrowserUtil and H5Browser. H5BrowserUtil gives you methods to quickly (in a script) get info and data from your file while the H5Browser adds a UI to interact with the hdf5 file.

Created the 15/11/2022

@author: Sebastien Weber

class pymodaq_gui.h5modules.browsing.H5Browser(parent, h5file=None, h5file_path=None, backend='h5py', swmr=False)[source]

App used to explore h5 files, plot and export subdatas

Parameters:

parent (QMainWindow) – either a QWidget or a QMainWindow
h5file (h5file instance) – exact type depends on the backend
h5file_path (str or Path) – if specified load the corresponding file, otherwise open a select file dialog
backend (str) – either ‘tables, ‘h5py’ or ‘h5pyd’
swmr (bool) – if True, open the file in SWMR reading mode (h5py backend only)

See also

H5Backend, H5Backend

add_comments(status, comment='')[source]

Add comments to a node

Parameters:

status (bool)
comment (str) – The comment to be added in a comment attribute to the current node path

See also

current_node_path

check_version()[source]: Check version of PyMoDAQ to assert if file is compatible or not with the current version of the Browser

export_data()[source]: Opens a dialog to export data

See also

H5BrowserUtil.export_data

get_tree_node_path()[source]: Get the node path of the currently selected node in the UI

populate_tree()[source]: Init the ui-tree and store data into calling the h5_tree_to_Qtree convertor method

See also

h5tree_to_QTree, update_status

quit_fun()[source]

refresh_file()[source]

Refresh the file view to show newly written data.

In SWMR reader mode the h5py datasets are refreshed in-place so that new data written by the SWMR writer becomes visible. In non-SWMR mode the file is closed and reopened to pick up external changes.

The tree is then repopulated and expanded.

show_h5_data(item, with_bkg=False, plot_all=False)[source]

Parameters:

item
with_bkg
plot_all

class pymodaq_gui.h5modules.browsing.H5BrowserUtil(backend='tables')[source]

Utility object to interact and get info and data from a hdf5 file

Inherits H5SaverLowLevel and all its functionalities

Parameters:: backend (str) – The used hdf5 backend: either tables, h5py or h5pyd

export_data(node_path='/', filesavename='datafile.h5', filter=None)[source]: Initialize the correct exporter and export the node

get_h5_attributes(node_path)[source]

get_h5file_scans(where='/')[source]

Get the list of the scan nodes in the file

Parameters:: where (str) – the path in the file
Returns:: dict with keys: scan_name, path (within the file) and data (the live scan png image)
Return type:: list of dict

8.3.1.7. Module savers

Created the 23/11/2022

@author: Sebastien Weber

class pymodaq.utils.h5modules.module_saving.ActuatorSaver(module)[source]

Implementation of the ModuleSaver class dedicated to DAQ_Move modules

Parameters:

h5saver
module (DAQ_Move)

class pymodaq.utils.h5modules.module_saving.DetectorEnlargeableSaver(module, enl_axis_names=None, enl_axis_units=None)[source]

Implementation of the ModuleSaver class dedicated to DAQ_Viewer modules in order to save enlargeable data

Parameters:: module (DAQ_Viewer)

class pymodaq.utils.h5modules.module_saving.DetectorExtendedSaver(module, extended_shape)[source]

Implementation of the ModuleSaver class dedicated to DAQ_Viewer modules in order to save enlargeable data

Parameters:: module (DAQ_Viewer)

class pymodaq.utils.h5modules.module_saving.DetectorSaver(module)[source]

Implementation of the ModuleSaver class dedicated to DAQ_Viewer modules

Parameters:: module (DAQ_Viewer)

add_bkg(where, data_bkg)[source]

Adds a DataToExport as a background node in the h5file

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself
data_bkg (DataToExport) – The data to be saved as background

filter_data(dte)[source]

Filter Data to be saved depending on first the presence of the extra_attribute: do_save and then on the H5Saver settings

Return type:: DataToExport

filter_data_wrt_settings(dwa)[source]

Check if this DataWithAxes should be saved depending on the H5Saver settings

Return True if it should be saved

Return type:: bool

class pymodaq.utils.h5modules.module_saving.LoggerSaver(*args, **kwargs)[source]

Implementation of the ModuleSaver class dedicated to Logger module

H5Logger is the special logger to h5file of the DAQ_Logger extension

Parameters:

h5saver
module

add_data(dte, **kwargs)[source]

Add data to it’s corresponding control module

The name of the control module is the DataToExport name attribute

class pymodaq.utils.h5modules.module_saving.ModuleSaver(*args, **kwargs)[source]

Abstract base class to save info and data from main modules (DAQScan, DAQViewer, DAQMove, …)

flush()[source]: Flush the underlying file

get_last_node(where=None)[source]

Get the last node corresponding to this particular Module instance

Parameters:: where (Union[Node, str]) – the path of a given node or the node itself
Returns:: GROUP
Return type:: the Node associated with this module which should be a GROUP node

get_set_node(where=None, name=None)[source]

Get or create the node corresponding to this particular Module instance

Parameters:

where (Union[Node, str]) – the path of a given node or the node itself
new (bool) – if True force the creation of a new indexed node of this class type if False return the last node (or create one if None)

Returns:

GROUP

Return type:

GROUP

abstractmethod update_after_h5changed()[source]: Propagate the h5saver to eventual child modules

class pymodaq.utils.h5modules.module_saving.ScanSaver(*args, **kwargs)[source]

Implementation of the ModuleSaver class dedicated to DAQScan module

Parameters:

h5saver
module

update_after_h5changed()[source]: To be updated depending on the actual Saver you want to use