8.3.1. Hdf5 module and classes
8.3.1.1. Hdf5 backends
The H5Backend is a wrapper around three hdf5 python packages: pytables, h5py and h5pyd. It allows seamless integration of any of these with PyMoDAQ features.
- class pymodaq.utils.h5modules.backends.H5Backend(backend='tables')[source]
- Attributes
- filename
- h5file
Methods
add_group
(group_name, group_type, where[, ...])Add a node in the h5 file tree of the group type :param group_name: :type group_name: (str) a custom name for this group :param group_type: one of the possible values of GroupType :type group_type: str or GroupType enum :param where: :type where: (str or node) parent node where to create the new group :param metadata: :type metadata: (dict) extra metadata to be saved with this new group node
Flush data and close the h5file
create_earray
(where, name, dtype[, ...])create enlargeable arrays from data with a given shape and of a given type.
create_vlarray
(where, name, dtype[, title])create variable data length and type and enlargeable 1D arrays
define_compression
(compression, compression_opts)Define cmpression library and level of compression :param compression: but zlib is used by pytables while gzip is used by h5py :type compression: (str) either gzip and zlib are supported here as they are compatible :param compression_opts (int): :type compression_opts (int): 0 to 9 0: None, 9: maximum compression
get_children
(where)Get a dict containing all children node hanging from where with their name as keys and types among Node, CARRAY, EARRAY, VLARRAY or StringARRAY
get_node_name
(node)return node name :param node (str or node instance): :param see h5py and pytables documentation on nodes:
get_node_path
(node)return node path :param node (str or node instance): :param see h5py and pytables documentation on nodes:
get_set_group
(where, name[, title])Retrieve or create (if absent) a node group Get attributed to the class attribute
current_group
is_node_in_group
(where, name)Check if a given node with name is in the group defined by where (comparison on lower case strings) :param where: path or parent node instance :type where: (str or node) :param name: group node name :type name: (str)
create_carray
flush
get_attr
get_group_by_title
get_node
get_parent_node
has_attr
isopen
open_file
read
root
save_file_as
set_attr
walk_groups
walk_nodes
- add_group(group_name, group_type: GroupType, where, title='', metadata={}) GROUP [source]
Add a node in the h5 file tree of the group type :param group_name: :type group_name: (str) a custom name for this group :param group_type: one of the possible values of GroupType :type group_type: str or GroupType enum :param where: :type where: (str or node) parent node where to create the new group :param metadata: :type metadata: (dict) extra metadata to be saved with this new group node
- Returns
(node)
- Return type
newly created group node
- create_earray(where, name, dtype, data_shape=None, title='')[source]
create enlargeable arrays from data with a given shape and of a given type. The array is enlargeable along the first dimension
- create_vlarray(where, name, dtype, title='')[source]
create variable data length and type and enlargeable 1D arrays
- define_compression(compression, compression_opts)[source]
Define cmpression library and level of compression :param compression: but zlib is used by pytables while gzip is used by h5py :type compression: (str) either gzip and zlib are supported here as they are compatible :param compression_opts (int): :type compression_opts (int): 0 to 9 0: None, 9: maximum compression
- get_children(where)[source]
Get a dict containing all children node hanging from where with their name as keys and types among Node, CARRAY, EARRAY, VLARRAY or StringARRAY
- Parameters
instance) (where (str or node) – see h5py and pytables documentation on nodes, and Node objects of this module
- Returns
dict
- Return type
keys are children node names, values are the children nodes
See also
GROUP.children_name()
- get_node_name(node)[source]
return node name :param node (str or node instance): :param see h5py and pytables documentation on nodes:
- Returns
str
- Return type
name of the node
- get_node_path(node)[source]
return node path :param node (str or node instance): :param see h5py and pytables documentation on nodes:
- Returns
str
- Return type
full path of the node
- get_set_group(where, name, title='')[source]
Retrieve or create (if absent) a node group Get attributed to the class attribute
current_group
- is_node_in_group(where, name)[source]
Check if a given node with name is in the group defined by where (comparison on lower case strings) :param where: path or parent node instance :type where: (str or node) :param name: group node name :type name: (str)
- Returns
True if node exists, False otherwise
- Return type
8.3.1.2. Low Level saving
H5SaverBase
and H5Saver
classes are a help to save data in a hierachical hdf5 binary file through the H5Backend
object and allowing integration in the PyMoDAQ Framework. These objects allows the creation of a file, of the various
nodes necessary to save PyMoDAQ’s data. The saving functionalities are divided in two objects: H5SaverBase and H5Saver
. H5SaverBase contains everything
needed for saving, while H5Saver
, inheriting H5SaverBase, add Qt functionality such as emitted signals. However,
these are not specific of PyMoDAQ’s data types. To save and load data, one should use higher level objects, see
High Level saving/loading.
Created the 15/11/2022
@author: Sebastien Weber
- class pymodaq.utils.h5modules.saving.H5Saver(*args, **kwargs)[source]
- status_sig: Signal
emits a signal of type Threadcommand in order to senf log information to a main UI
- new_file_sig: Signal
emits a boolean signal to let the program know when the user pressed the new file button on the UI
- class pymodaq.utils.h5modules.saving.H5SaverBase(save_type='scan', backend='tables')[source]
Object containing all methods in order to save datas in a hdf5 file with a hierarchy compatible with the H5Browser. The saving parameters are contained within a Parameter object: self.settings that can be displayed on a UI using the widget self.settings_tree. At the creation of a new file, a node group named Raw_datas and represented by the attribute
raw_group
is created and set with a metadata attribute:‘type’ given by the save_type class parameter
The root group of the file is then set with a few metadata:
‘pymodaq_version’ the current pymodaq version, e.g. 1.6.2
‘file’ the file name
‘date’ the current date
‘time’ the current time
All datas will then be saved under this node in various groups
See also
H5Browser
- Parameters
h5_file (pytables hdf5 file) – object used to save all datas and metadas
h5_file_path (str or Path) – Signal signal represented by a float. Is emitted each time the hardware reached the target position within the epsilon precision (see comon_parameters variable)
save_type (str) – an element of the enum module attribute SaveType * ‘scan’ is used for DAQScan module and should be used for similar application * ‘detector’ is used for DAQ_Viewer module and should be used for similar application * ‘custom’ should be used for customized applications
- settings
Parameter instance (pyqtgraph) containing all settings (could be represented using the settings_tree widget)
- Type
Parameter
- settings_tree
Widget representing as a Tree structure, all the settings defined in the class preamble variable
params
- Type
ParameterTree
- classmethod find_part_in_path_and_subpath(base_dir, part='', create=False, increment=True)[source]
Find path from part time.
Parameters
Type
Description
base_dir
Path object
The directory to browse
part
string
The date of the directory to find/create
create
boolean
Indicate the creation flag of the directory
- Returns
found path from part
- Return type
Path object
- get_last_scan()[source]
Gets the last scan node within the h5_file and under the raw_group
- Returns
scan_group
- Return type
pytables group or None
- get_scan_index()[source]
return the scan group index in the “scan templating”: Scan000, Scan001 as an integer
- init_file(update_h5=False, custom_naming=False, addhoc_file_path=None, metadata={})[source]
Initializes a new h5 file. Could set the h5_file attributes as:
a file with a name following a template if
custom_naming
isFalse
andaddhoc_file_path
isNone
a file within a name set using a file dialog popup if
custom_naming
isTrue
a file with a custom name if
addhoc_file_path
is aPath
object or a path string
- Parameters
update_h5 (bool) – create a new h5 file with name specified by other parameters if false try to open an existing file and will append new data to it
custom_naming (bool) – if True, a selection file dialog opens to set a new file name
addhoc_file_path (Path or str) – supplied name by the user for the new file
metadata (dict) – dictionnary with pair of key, value that should be saved as attributes of the root group
- Returns
update_h5 – True if new file has been created, False otherwise
- Return type
- load_file(base_path=None, file_path=None)[source]
Opens a file dialog to select a h5file saved on disk to be used
- Parameters
base_path –
file_path –
See also
- classmethod set_current_scan_path(base_dir, base_name='Scan', update_h5=False, next_scan_index=0, create_scan_folder=False, create_dataset_folder=True, curr_date=None, ind_dataset=None)[source]
- Parameters
base_dir –
base_name –
update_h5 –
next_scan_index –
create_scan_folder –
create_dataset_folder –
- update_file_paths(update_h5=False)[source]
- Parameters
update_h5 (bool) – if True, will increment the file name and eventually the current scan index if False, get the current scan index in the h5 file
- Returns
scan_path (Path)
current_filename (str)
dataset_path (Path)
- value_changed(param)[source]
Non-mandatory method to be subclassed for actions to perform (methods to call) when one of the param’s value in self._settings is changed
- Parameters
param (Parameter) – the parameter whose value just changed
Examples
>>> if param.name() == 'do_something': >>> if param.value(): >>> print('Do something') >>> self.settings.child('main_settings', 'something_done').setValue(False)
They both inherits from the ParameterManager
MixIn class that deals with Parameter and ParameterTree,
see saving_settings_fig
.
8.3.1.3. High Level saving/loading
Each PyMoDAQ’s data type: Axis
, DataWithAxes
, DataToExport
(see What is PyMoDAQ’s Data?) is associated
with its saver/loader
counterpart. These objects ensures that all metadata necessary for an exact regeneration of the data is being saved at
the correct location in the hdf5 file hierarchy. The AxisSaverLoader
, DataSaverLoader
, DataToExportSaver
all derive from an abstract class: DataManagement
allowing the manipulation of the nodes and making sure the data type
is defined.
8.3.1.3.1. Base data class saver/loader
Created the 21/11/2022
@author: Sebastien Weber
- class pymodaq.utils.h5modules.data_saving.AxisSaverLoader(*args, **kwargs)[source]
Specialized Object to save and load Axis object to and from a h5file
- Parameters
h5saver (H5Saver) –
- data_type
The enum for this type of data, here ‘axis’
- Type
DataType
- add_axis(where: Union[Node, str], axis: Axis, enlargeable=False)[source]
Write Axis info at a given position within a h5 file
- class pymodaq.utils.h5modules.data_saving.DataManagement(*args, **kwargs)[source]
Base abstract class to be used for all specialized object saving and loading data to/from a h5file
- data_type
The enum for this type of data, here abstract and should be redefined
- Type
DataType
- class pymodaq.utils.h5modules.data_saving.DataSaverLoader(*args, **kwargs)[source]
Specialized Object to save and load DataWithAxes object to and from a h5file
- Parameters
h5saver (H5Saver) –
- data_type
The enum for this type of data, here ‘data’
- Type
DataType
- add_data(where: Union[Node, str], data: DataWithAxes, save_axes=True, **kwargs)[source]
Adds Array nodes to a given location adding eventually axes as others nodes and metadata
- get_axes(where: Union[Node, str]) List[Axis] [source]
- Parameters
where (Union[Node, str]) – the path of a given node or the node itself
- load_data(where, with_bkg=False, load_all=False) DataWithAxes [source]
Return a DataWithAxes object from the Data and Axis Nodes hanging from (or among) a given Node
Does not include navigation axes stored elsewhere in the h5file. The node path is stored in the DatWithAxis using the attribute path
- Parameters
See also
- class pymodaq.utils.h5modules.data_saving.DataToExportSaver(h5saver: H5Saver)[source]
Object used to save DataToExport object into a h5file following the PyMoDAQ convention
- Parameters
h5saver (H5Saver) –
- add_data(where: Union[Node, str], data: DataToExport, settings_as_xml='', metadata={})[source]
- Parameters
where (Union[Node, str]) – the path of a given node or the node itself
data (DataToExport) –
settings_as_xml (str) – The settings parameter as an XML string
metadata (dict) – all extra metadata to be saved in the group node where data will be saved
8.3.1.3.2. Specific data class saver/loader
Some more dedicated objects are derived from the objects above. They allow to add background data, Extended arrays (arrays that will be populated after creation, for instance for a scan) and Enlargeable arrays (whose final length is not known at the moment of creation, for instance when logging or continuously saving)
Created the 21/11/2022
@author: Sebastien Weber
- class pymodaq.utils.h5modules.data_saving.BkgSaver(*args, **kwargs)[source]
Specialized Object to save and load DataWithAxes background object to and from a h5file
- Parameters
hsaver (H5Saver) –
- data_type
The enum for this type of data, here ‘data’
- Type
DataType
- class pymodaq.utils.h5modules.data_saving.DataEnlargeableSaver(*args, **kwargs)[source]
Specialized Object to save and load enlargeable DataWithAxes saved object to and from a h5file
Particular case of DataND will a single nav_indexes parameter will be appended as chunks of signal data
- Parameters
h5saver (H5Saver) –
- data_type
The enum for this type of data, here ‘data_enlargeable’
- Type
DataType
Notes
To be used to save data from a timed logger (DAQViewer continuous saving or DAQLogger extension) or from an adaptive scan where the final shape is unknown or other module that need this feature
- add_data(where: Union[Node, str], data: DataWithAxes)[source]
Append data to an enlargeable array node
Data of dim (0, 1 or 2) will be just appended to the enlargeable array. DataND with one navigation axis of length (Lnav) will be considered as a collection of Lnav signal data of dim (0, 1 or 2) and will therefore be appended as Lnav signal data
- Parameters
where (Union[Node, str]) – the path of a given node or the node itself
data (DataWithAxes) –
- class pymodaq.utils.h5modules.data_saving.DataExtendedSaver(*args, **kwargs)[source]
Specialized Object to save and load DataWithAxes saved object to and from a h5file in extended arrays
- Parameters
- data_type
The enum for this type of data, here ‘data’
- Type
DataType
- class pymodaq.utils.h5modules.data_saving.DataToExportEnlargeableSaver(h5saver: H5Saver, axis_name: str = 'nav axis', axis_units: str = '')[source]
Generic object to save DataToExport objects in an enlargeable h5 array
The next enlarged value should be specified in the add_data method
- Parameters
- add_data(where: Union[Node, str], data: DataToExport, axis_value: Union[float, ndarray], settings_as_xml='', metadata={})[source]
- Parameters
where (Union[Node, str]) – the path of a given node or the node itself
data (DataToExport) – The data to be saved into an enlargeable array
axis_value (float or np.ndarray) – The next value (or values) of the enlarged axis
settings_as_xml (str) – The settings parameter as an XML string
metadata (dict) – all extra metadata to be saved in the group node where data will be saved
- class pymodaq.utils.h5modules.data_saving.DataToExportExtendedSaver(h5saver: H5Saver, extended_shape: Tuple[int])[source]
Object to save DataToExport at given indexes within arrays including extended shape
Mostly used for data generated from the DAQScan
- Parameters
- add_data(where: Union[Node, str], data: DataToExport, indexes: List[int], distribution=DataDistribution.uniform, settings_as_xml='', metadata={})[source]
- Parameters
where (Union[Node, str]) – the path of a given node or the node itself
data (DataToExport) –
indexes (List[int]) – indexes where to save data in the init h5array (should have the same length as extended_shape and with values coherent with this shape
settings_as_xml (str) – The settings parameter as an XML string
metadata (dict) – all extra metadata to be saved in the group node where data will be saved
Used to add navigation axes related to the extended array
Notes
For instance the scan axes in the DAQScan
- class pymodaq.utils.h5modules.data_saving.DataToExportTimedSaver(h5saver: H5Saver)[source]
Specialized DataToExportEnlargeableSaver to save data as a function of a time axis
Only one element ca be added at a time, the time axis value are enlarged using the data to be added timestamp
Notes
This object is made for continuous saving mode of DAQViewer and logging to h5file for DAQLogger
- add_data(where: Union[Node, str], data: DataToExport, settings_as_xml='', metadata={})[source]
- Parameters
where (Union[Node, str]) – the path of a given node or the node itself
data (DataToExport) – The data to be saved into an enlargeable array
axis_value (float or np.ndarray) – The next value (or values) of the enlarged axis
settings_as_xml (str) – The settings parameter as an XML string
metadata (dict) – all extra metadata to be saved in the group node where data will be saved
8.3.1.4. Specialized loading
Data saved from a DAQ_Scan
will naturally include navigation axes shared between many different DataWithAxes
(as many as detectors/channels/ROIs). They are therefore saved at the root of the scan node and cannot be retrieved
using the standard data loader. Hence this DataLoader
object.
- class pymodaq.utils.h5modules.data_saving.DataLoader(h5saver: H5Saver)[source]
Specialized Object to load DataWithAxes object from a h5file
On the contrary to DataSaverLoader, does include navigation axes stored elsewhere in the h5file (for instance if saved from the DAQ_Scan)
- Parameters
h5saver (H5Saver) –
- Attributes
- h5saver
Methods
get_nav_group
(where)- param where
the path of a given node or the node itself
get_node
(where[, name])Convenience method to get node
load_data
(where[, with_bkg, load_all])Load data from a node (or channel node)
load_all
- Parameters
where (Union[Node, str]) – the path of a given node or the node itself
- Returns
GROUP
- Return type
returns the group named SPECIAL_GROUP_NAMES[‘nav_axes’] holding all NavAxis for those data
See also
SPECIAL_GROUP_NAMES
- get_node(where: Union[Node, str], name: Optional[str] = None) Node [source]
Convenience method to get node
8.3.1.5. Browsing Data
Using the H5Backend it is possible to write scripts to easily access a hdf5 file content. However, PyMoDAQ includes a dedicated hdf5 viewer understanding dedicated metadata and therefore displaying nicely the content of the file, see H5Browser. Two objects can be used to browse data: H5BrowserUtil and H5Browser. H5BrowserUtil gives you methods to quickly (in a script) get info and data from your file while the H5Browser adds a UI to interact with the hdf5 file.
Created the 15/11/2022
@author: Sebastien Weber
- class pymodaq.utils.h5modules.browsing.H5Browser(parent: QMainWindow, h5file=None, h5file_path=None, backend='tables')[source]
UI used to explore h5 files, plot and export subdatas
- Parameters
See also
H5Backend
,H5Backend
- add_comments(status: bool, comment='')[source]
Add comments to a node
- Parameters
See also
current_node_path
- check_version()[source]
Check version of PyMoDAQ to assert if file is compatible or not with the current version of the Browser
- populate_tree()[source]
- Init the ui-tree and store data into calling the h5_tree_to_Qtree convertor method
See also
h5tree_to_QTree
,update_status
- setup_actions()[source]
Method where to create actions to be subclassed. Mandatory
Examples
>>> self.add_action('Quit', 'close2', "Quit program") >>> self.add_action('Grab', 'camera', "Grab from camera", checkable=True) >>> self.add_action('Load', 'Open', "Load target file (.h5, .png, .jpg) or data from camera", checkable=False) >>> self.add_action('Save', 'SaveAs', "Save current data", checkable=False)
See also
ActionManager.add_action
- class pymodaq.utils.h5modules.browsing.H5BrowserUtil(backend='tables')[source]
Utility object to interact and get info and data from a hdf5 file
Inherits H5Backend and all its functionalities
- Parameters
backend (str) – The used hdf5 backend: either tables, h5py or h5pyd
8.3.1.6. Module savers
Created the 23/11/2022
@author: Sebastien Weber
- class pymodaq.utils.h5modules.module_saving.ActuatorSaver(*args, **kwargs)[source]
Implementation of the ModuleSaver class dedicated to DAQ_Move modules
- Parameters
h5saver –
module –
- class pymodaq.utils.h5modules.module_saving.DetectorEnlargeableSaver(*args, **kwargs)[source]
Implementation of the ModuleSaver class dedicated to DAQ_Viewer modules in order to save enlargeable data
- Parameters
module –
- class pymodaq.utils.h5modules.module_saving.DetectorExtendedSaver(*args, **kwargs)[source]
Implementation of the ModuleSaver class dedicated to DAQ_Viewer modules in order to save enlargeable data
- Parameters
module –
- class pymodaq.utils.h5modules.module_saving.DetectorSaver(*args, **kwargs)[source]
Implementation of the ModuleSaver class dedicated to DAQ_Viewer modules
- Parameters
module –
- add_bkg(where: Union[Node, str], data_bkg: DataToExport)[source]
Adds a DataToExport as a background node in the h5file
- Parameters
where (Union[Node, str]) – the path of a given node or the node itself
data_bkg (DataToExport) – The data to be saved as background
- class pymodaq.utils.h5modules.module_saving.LoggerSaver(*args, **kwargs)[source]
Implementation of the ModuleSaver class dedicated to H5Logger module
H5Logger is the special logger to h5file of the DAQ_Logger extension
- Parameters
h5saver –
module –
- class pymodaq.utils.h5modules.module_saving.ModuleSaver(*args, **kwargs)[source]
Abstract base class to save info and data from main modules (DAQScan, DAQViewer, DAQMove, …)
- get_last_node(where: Optional[Union[Node, str]] = None)[source]
Get the last node corresponding to this particular Module instance
- Parameters
- Returns
GROUP
- Return type
the Node associated with this module which should be a GROUP node
- class pymodaq.utils.h5modules.module_saving.ScanSaver(*args, **kwargs)[source]
Implementation of the ModuleSaver class dedicated to DAQScan module
- Parameters
h5saver –
module –