(to be included in the data management section)

2.7. Saving datas

Datas saved using PyMoDAQ, either the DAQ_Scan or the DAQ_Viewer modules, use a binary format known as hdf5. This format was originally developped to save big volume of datas from large instruments. Its structure is hierarchical (a bit as folder trees) and one can add metadata to all entries in the tree. For instance, the data type, shape but also some more complex info such as all the settings related to a module or a plugin. This gives a unique file containing both datas and metadata. For these data and metadata to be saved and interpreted correctly by the H5Browser, a utility module is used: H5Saver within the DAQ_Viewer module and the DAQ_Scan module. This should also be used when building a custom application.

2.7.1. From DAQ_Viewer

2.7.1.1. Single Datas

Datas saved directly from a DAQ_Viewer (for instance the one on Fig. 2.18) will be recorded in a h5file whose structure will be represented like Fig. 2.61 using PyMoDAQ’s h5 browser (see H5Browser). The various type of data exported by the viewer are saved in a dedicated entry (Data0D, …) and all channels are recorded within as subentries. Metadata from the detector settings, the measurement module and the activated ROIs are also recorded.

single_data

Fig. 2.61 h5 file content containing saved data from a DAQ_Viewer detector (see Fig. 2.18)

2.7.1.2. Continuous Saving

When the continuous saving parameter is set, new parameters are appearing on the DAQ_Viewer panel (see Fig. 2.62).

  • Base path: indicates where the data will be saved. If it doesn’t exist the module will try to create it

  • Base name: indicates the base name from which the save file will derive

  • Current Path: readonly, complete path of the saved file

  • Do Save: Initialize the file and logging can start. A new file is created if clicked again.

  • Compression options: data can be compressed before saving, using one of the proposed library and the given value of compression [0-9], see pytables documentation.

continuous

Fig. 2.62 Continuous Saving options

The saved file will follow this general structure:

D:\Data\2018\20181220\Data_20181220_16_58_48.h5

With a base path (D:\Data in this case) followed by a subfolder year, a subfolder day and a filename formed from a base name followed by the date of the day and the time at which you started to log data. Fig. 2.63 displays the tree structure of such a file, with

continuous

Fig. 2.63 Continuous Saving options

2.7.2. From DAQ_Scan

DAQ_Scan module will save your data in datasets. Each dataset is a unique h5 file and may contain multiple scans. The idea behind this is to have a unique file for a set of related data (the dataset) together with all the meta information: logger data, module parameters (settings, ROI…) even png screenshots of the various panels.

Fig. 2.64 displays the content of a typical dataset file containing various scans and how each data and metadata is used by the H5Browser to display the info to the user.

h5 browser

Fig. 2.64 h5 browser and arrows to explain how each data or metadata is being displayed

In order to save correctly your datas in custom applications, a utility module is to be used: pymodaq.utils.h5saver, it will save scans and datas following the rules displayed on figure_dataset_layout below:

File layout

2.7.3. Exploring datas

h5file saved using PyMoDAQ can be explored using the H5Browser utility module. Both datas and metadatas can be explored.