8.4.6. Data Management
|
Enum for dimensionality representation of data |
|
Enum for source of data |
|
Enum for distribution of data |
|
Object holding info and data about physical axis of some data |
|
Base object to store homogeneous data and metadata generated by pymodaq's objects. |
|
Data object with Axis objects corresponding to underlying data nd-arrays |
|
Specialized DataWithAxes set with source as 'raw'. |
|
Specialized DataWithAxes set with source as 'calculated'. |
|
Specialized DataWithAxes set with source as 'calculated'. |
|
Object to store all raw and calculated DataWithAxes data for later exporting, saving, sending signal... |
|
Specialized DataWithAxes set with source as 'raw'. |
|
Specialized DataWithAxes set with source as 'raw'. |
8.4.6.1. Axes
Created the 28/10/2022
@author: Sebastien Weber
- class pymodaq_data.data.Axis(label='', units='', data=None, index=0, scaling=None, offset=None, size=None, spread_order=0)[source]
Object holding info and data about physical axis of some data
In case the axis’s data is linear, store the info as a scale and offset else store the data
- Parameters:
label (
str) – The label of the axis, for instance ‘time’ for a temporal axisunits (
str) – The units of the data in the object, for instance ‘s’ for secondsdata (
ndarray) – A 1D ndarray holding the data of the axisindex (
int) – an integer representing the index of the Data object this axis is related toscaling (
float) – The scaling to apply to a linspace version in order to obtain the proper scalingoffset (
float) – The offset to apply to a linspace/scaled version in order to obtain the proper axissize (
int) – The size of the axis array (to be specified if data is None)spread_order (
int) – An integer needed in the case where data has a spread DataDistribution. It refers to the index along the data’s spread_index dimension
Examples
>>> axis = Axis('myaxis', units='seconds', data=np.array([1,2,3,4,5]), index=0)
- static deserialize(bytes_str)[source]
Convert bytes into an Axis object
Convert the first bytes into an Axis reading first information about the Axis
- static serialize(axis)[source]
Convert an Axis object into a bytes message together with the info to convert it back
- Parameters:
axis (
Axis)- Returns:
bytes
- Return type:
the total bytes messagetoserialize the Axis
Notes
The bytes sequence is constructed as:
serialize the axis label
serialize the axis units
serialize the axis array
serialize the axis
serialize the axis spread_order
- create_linear_data(nsteps)[source]
replace the axis data with a linear version using scaling and offset
- find_index(threshold)[source]
find the index of the threshold value within the axis
- Return type:
- flip()[source]
flip the direction of the axis
- force_units(units)[source]
Change immediately the units to whatever else. Use this with care!
- get_data()[source]
Convenience method to obtain the axis data (usually None because scaling and offset are used)
- Return type:
- get_data_at(indexes)[source]
Get data at specified indexes
- get_quantity()[source]
Convenience method to obtain the numerical data as a quantity array
- Return type:
Quantity
- get_scale_offset_from_data(data=None)[source]
Get the scaling and offset from the axis’s data
If data is not None, extract the scaling and offset
- Parameters:
data (
ndarray)
- property axis_width: Quantity
get the width of the axis in axis unit
That is the difference between the max value and the min value of the axis
- property data
get/set the data of Axis
- Type:
np.ndarray
8.4.6.2. DataObjects
Created the 28/10/2022
@author: Sebastien Weber
- class pymodaq_data.data.DataBase(name, source=None, dim=None, distribution=DataDistribution.uniform, data=None, labels=None, origin='', units='', **kwargs)[source]
Base object to store homogeneous data and metadata generated by pymodaq’s objects.
To be inherited for real data
- Parameters:
name (
str) – the identifier of these datasource (
DataSource) – Enum specifying if data are raw or processed (for instance from roi)dim (
DataDim) – The identifier of the data typedistribution (
DataDistribution) – The distribution type of the data: uniform if distributed on a regular grid or spread if on specific unordered pointsdata (
List[ndarray]) – The data the object is storing. In case of Quantities, the object units attribute will be forced to the unit of this quantity, ignoring the units argument.origin (
str) – An identifier of the element where the data originated, for instance the DAQ_Viewer’s name. Used when appending DataToExport in DAQ_Scan to disintricate from which origin data comes from when scanning multiple detectors.units (
str) – A unit string identifier as specified in the UnitRegistry of the pint modulekwargs (
named parameters) – All other parameters are stored dynamically using the name/value pair. The name of these extra parameters are added into the extra_attributes attribute
- Variables:
name (
str) – the identifier of these datasource (
DataSourceorstr) – Enum specifying if data are raw or processed (for instance from roi)distribution (
DataDistributionorstr) – The distribution type of the data: uniform if distributed on a regular grid or spread if on specific unordered pointsdata (
listofndarray) – The data the object is storingorigin (
str) – An identifier of the element where the data originated, for instance the DAQ_Viewer’s name. Used when appending DataToExport in DAQ_Scan to disintricate from which origin data comes from when scanning multiple detectors.shape (
Tuple[int]) – The shape of the underlying datasize (
int) – The size of the ndarrays stored in the objectlength (
int) – The number of ndarrays stored in the objectextra_attributes (
List[str]) – list of string giving identifiers of the attributes added dynamically at the initialization (for instance to save extra metadata using the DataSaverLoader
See also
DataWithAxes,DataFromPlugins,DataRaw,DataSaverLoaderExamples
>>> import numpy as np >>> from pymodaq.utils.data import DataBase, DataSource, DataDim, DataDistribution >>> data = DataBase('mydata', source=DataSource['raw'], dim=DataDim['Data1D'], distribution=DataDistribution.uniform, data=[np.array([1.,2.,3.]), np.array([4.,5.,6.])], labels=['channel1', 'channel2'], origin='docutils code') >>> data.dim <DataDim.Data1D: 1> >>> data.source <DataSource.raw: 0> >>> data.shape (3,) >>> data.length 2 >>> data.size 3
- static get_origin_name_from_full_name(full_name)[source]
Standardize the obtention of the origin and name from the full name expression
Origin is always the first bit before the first ‘/’ character while the name will be the remaining characters
- abs()[source]
Take the absolute value of itself
- angle()[source]
Take the phase value of itself
- append(data)[source]
Append data content if the underlying arrays have the same shape and compatible units
- as_dte(name='mydte')[source]
Convenience method to wrap the DataWithAxes object into a DataToExport
- Return type:
- average(other, weight)[source]
Compute the weighted average between self and other DataBase
- equal_to(other, epsilon)[source]
Check if two data object are equal within epsilon
- Return type:
- fliplr()[source]
Reverse the order of elements along axis 1 (left/right)
- flipud()[source]
Reverse the order of elements along axis 0 (up/down)
- force_units(units)[source]
Change immediately the units to whatever else. Use this with care!
- get_data_index(index=0)[source]
Get the data by its index in the list, same as self[index]
- Return type:
- get_dim_from_data(data)[source]
Get the dimensionality DataDim from data
- get_full_name()[source]
Get the data ful name including the origin attribute into the returned value
- Returns:
str
- Return type:
Examples
d0 = DataBase(name=’datafromdet0’, origin=’det0’)
- imag()[source]
Take the imaginary part of itself
- pop(index)[source]
Returns a copy of self but with data taken at the specified index
- Return type:
- real()[source]
Take the real part of itself
- set_dim(dim)[source]
Addhoc modification of dim independantly of the real data shape, should be used with extra care
- split_as_dte(name='mydte')[source]
Convenience method to split each ndarray into a DataWithAxes object
- Return type:
- stack_as_array(axis=0, dtype=None)[source]
Stack all data arrays in a single numpy array
- Parameters:
- Return type:
See also
np.stack()
- to_dict()[source]
Get the data arrays into dictionary whose keys are the labels
- units_as(units, inplace=True, context=None, **context_kwargs)[source]
Set the object units to the new one (if possible)
- unwrap()[source]
unwrap the underlying array (should be angles otherwise meaningless)
- value(units=None)[source]
Returns the underlying float value (of the first elt in the data list) if this data holds only a float otherwise returns a mean of the underlying data
- values(units=None)[source]
Returns the underlying float value (for each data array in the data list) if this data holds only a float otherwise returns a mean of the underlying data
- property averaged: bool
Get/Set a boolean depending if self is the result of an average operation
See also
- property data: List[ndarray]
get/set (and check) the data the object is storing
- Type:
List[np.ndarray]
- property dim
the enum representing the dimensionality of the stored data
- Type:
- property distribution
the enum representing the distribution of the stored data
- Type:
- property length
The length of data. This is the length of the list containing the nd-arrays
- property n_averaged: int
Get/set the number of averaging that resulted in this data
- property quantities: list[Quantity]
Get the arrays as pint quantities (with units)
- property shape
The shape of the nd-arrays
- property size
The size of the nd-arrays
- property source
the enum representing the source of the data
- Type:
- class pymodaq_data.data.DataCalculated(name, dim=None, distribution=DataDistribution.uniform, data=None, labels=None, origin='', units='', axes=[], nav_indexes=(), errors=None, **kwargs)[source]
Specialized DataWithAxes set with source as ‘calculated’. To be used for processed/calculated data
- class pymodaq_data.data.DataFromRoi(name, dim=None, distribution=DataDistribution.uniform, data=None, labels=None, origin='', units='', axes=[], nav_indexes=(), errors=None, **kwargs)[source]
Specialized DataWithAxes set with source as ‘calculated’. To be used for processed data from region of interest
- class pymodaq_data.data.DataRaw(name, dim=None, distribution=DataDistribution.uniform, data=None, labels=None, origin='', units='', axes=[], nav_indexes=(), errors=None, **kwargs)[source]
Specialized DataWithAxes set with source as ‘raw’. To be used for raw data
- class pymodaq_data.data.DataWithAxes(name, source=None, dim=None, distribution=DataDistribution.uniform, data=None, labels=None, origin='', units='', axes=[], nav_indexes=(), errors=None, **kwargs)[source]
Data object with Axis objects corresponding to underlying data nd-arrays
- Parameters:
axes (
List[Axis]) – the list of Axis object for proper plotting, calibration …nav_indexes (
Tuple[int]) – highlight which Axis in axes is Signal or Navigation axis depending on the content: For instance, nav_indexes = (2,), means that the axis with index 2 in a at least 3D ndarray data is the first navigation axis For instance, nav_indexes = (3,2), means that the axis with index 3 in a at least 4D ndarray data is the first navigation axis while the axis with index 2 is the second navigation Axis. Axes with index 0 and 1 are signal axes of 2D ndarray dataerrors (
Iterable[ndarray]) – The list should match the length of the data attribute while the ndarrays should match the data ndarray
- classmethod deserialize(bytes_str)[source]
Convert bytes into a DataWithAxes object
Convert the first bytes into a DataWithAxes reading first information about the underlying data
- Return type:
- Returns:
DataWithAxes (
the decoded DataWithAxes)bytes (
the remaining bytes string if any)
- classmethod from_xarray(ds)[source]
Construct a DataWithAxes from an xarray Dataset (or DataArray).
- Parameters:
ds (
xr.Datasetorxr.DataArray)- Return type:
- Raises:
ImportError – If xarray is not installed.
- static serialize(dwa)[source]
Convert a DataWithAxes into a bytes string
- Parameters:
dwa (
DataWithAxes)- Returns:
bytes
- Return type:
the total bytes messagetoserialize the DataWithAxes
Notes
The bytes sequence is constructed as:
serialize the timestamp: float
serialize the name
serialize the source enum as a string
serialize the dim enum as a string
serialize the distribution enum as a string
serialize the list of numpy arrays
serialize the list of labels
serialize the origin
serialize the nav_index tuple as a list of int
serialize the list of axis
serialize the errors attributes (None or list(np.ndarray))
serialize the list of names of extra attributes
serialize the extra attributes
- axes_limits(axes_indexes=None)[source]
Get the limits of specified axes (all if axes_indexes is None)
- check_axes_linear(axes=None)[source]
Check if any axis may be non linear
Should trigger a spread like distribution except id dim is Data1D, in which cas, it doesn’t matter
- Return type:
- create_missing_axes()[source]
Check if given the data shape, some axes are missing to properly define the data (especially for plotting)
- deepcopy_with_new_data(data=None, remove_axes_index=None, source=DataSource.calculated, keep_dim=False, errors=None)[source]
deepcopy without copying the initial data (saving memory)
The new data, may have some axes stripped as specified in remove_axes_index
- Parameters:
remove_axes_index (
Union[int,List[int]]) – indexes of the axis to be removedsource (
DataSource)keep_dim (
bool) – if False (the default) will calculate the new dim based on the data shape else keep the same (be aware it could lead to issues)errors (
List[ndarray]) – The new errors corresponding to the new data
- Return type:
- errors_as_dwa()[source]
Get a dwa from self replacing the data content with the error attribute (if not None)
New in 4.2.0
- find_peaks(height=None, threshold=None, **kwargs)[source]
Apply the scipy find_peaks method to 1D data
- Parameters:
height (
numberorndarrayorsequence, optional)threshold (
numberorndarrayorsequence, optional)kwargs (
dict) – extra named parameters applied to the find_peaks scipy method
- Return type:
See also
find_peaks()
- fit(function, initial_guess, data_index=None, axis_index=0, **kwargs)[source]
Apply 1D curve fitting using the scipy optimization package
- Parameters:
function (
Callable) – a callable to be used for the fitinitial_guess (
Iterable) – The initial parameters for the fitdata_index (
int) – The index of the data over which to do the fit, if None apply the fit to allaxis_index (
int) – the axis index to use for the fit (if multiple) but there should be only onekwargs (
dict) – extra named parameters applied to the curve_fit scipy method
- Return type:
See also
curve_fit()
- ft(axis=0, axis_label=None, axis_units=None, labels=None)[source]
Process the Fourier Transform of the data on the specified axis and returns the new data
- Parameters:
- Return type:
See also
ft(),fft()
- get_axis_from_label(label)[source]
Get the axis referred by a given label
- get_data_as_dwa(index=0)[source]
Get the underlying data selected from the list at index, returned as a DataWithAxes
- Return type:
- get_dim_from_data_axes()[source]
Get the dimensionality DataDim from data taking into account nav indexes
- Return type:
- get_error(index)[source]
Get a particular error ndarray at the given index in the list
new in 4.2.0
- get_nav_axes_with_data()[source]
Get the data’s navigation axes making sure there is data in the data field
- ift(axis=0, axis_label=None, axis_units=None, labels=None)[source]
Process the inverse Fourier Transform of the data on the specified axis and returns the new data
- Parameters:
- Return type:
See also
ift(),ifft()
- interp(new_axis_data, **kwargs)[source]
Performs linear interpolation for 1D data only.
For more complex ones, see
scipy.interpolate()- Parameters:
- Return type:
See also
interp(),interpolate()
- mean(axis=0)[source]
Process the mean of the data on the specified axis and returns the new data
- Parameters:
axis (
int)- Return type:
- moment()[source]
returns the two first moments of the data over the axis
only valid for Data1D data
- Return type:
- Returns:
DataCalculated (
containing the momentoforder 0 (mean))DataCalculated (
containing the momentoforder 1 (std))
- pad(pad_width, **kwargs)[source]
Pad the data arrays using the numpy pad function
The accepted pad_witdh type is the same than the numpy pad function
see numpy.pad method for the signature and possible named arguments
- plot(plotter_backend='matplotlib', *args, viewer=None, **kwargs)[source]
Call a plotter factory and its plot method over the actual data
- rot90(k=1, axes=(0, 1))[source]
Rotate an array by 90 degrees in the plane specified by axes.
Valid only for 2D data
- sort_data(axis_index=0, spread_index=0, inplace=False)[source]
Sort data along a given axis, default is 0
- Parameters:
- Return type:
- sum(axis=0)[source]
Process the sum of the data on the specified axis and returns the new data
- Parameters:
axis (
int)- Return type:
- to_xarray()[source]
Convert this DataWithAxes to an xarray.Dataset.
Each array in self.data becomes a data variable (keyed by its label). Each Axis becomes a coordinate on the corresponding dimension. Error arrays (if present) are stored as
<label>_errordata variables.- Return type:
xr.Dataset- Raises:
ImportError – If xarray is not installed.
- transpose()[source]
replace the data by their transposed version
Valid only for 2D data
- property axes
convenience property to fetch attribute from axis_manager
- property errors
Get/Set the errors bar values as a list of np.ndarray
new in 4.2.0
- property n_axes
Get the number of axes (even if not specified)
- property nav_indexes
convenience property to fetch attribute from axis_manager
- property sig_indexes
convenience property to fetch attribute from axis_manager
- class pymodaq.utils.data.DataActuator(*args, **kwargs)[source]
Specialized DataWithAxes set with source as ‘raw’. To be used for raw data generated by actuator plugins
8.4.6.3. Data Characteristics
Created the 28/10/2022
@author: Sebastien Weber
- class pymodaq_data.data.DataDim(*values)[source]
Enum for dimensionality representation of data
- class pymodaq_data.data.DataDistribution(*values)[source]
Enum for distribution of data
- class pymodaq_data.data.DataSource(*values)[source]
Enum for source of data
8.4.6.4. Union of Data
When exporting multiple set of Data objects, one should use a DataToExport
Created the 28/10/2022
@author: Sebastien Weber
- class pymodaq_data.data.DataToExport(name, data=[], **kwargs)[source]
Object to store all raw and calculated DataWithAxes data for later exporting, saving, sending signal…
Includes methods to retrieve data from dim, source… Stored data have a unique identifier their name. If some data is appended with an existing name, it will replace the existing data. So if you want to append data that has the same name
- Parameters:
name (
str) – The identifier of the exporting objectdata (
List[DataWithAxes]) – All the raw and calculated data to be exported
- Variables:
name
timestamp
data
- classmethod deserialize(bytes_str)[source]
Convert bytes into a DataToExport object
Convert the first bytes into a DataToExport reading first information about the underlying data
- Return type:
- Returns:
DataToExport (
the decoded DataToExport)bytes (
the remaining bytes if any)
- classmethod from_xarray(dt, name=None)[source]
Construct a DataToExport from an xarray.DataTree or a dict of Datasets.
- Parameters:
dt (
xr.DataTreeordict[str,xr.Dataset])name (
str) – Override the name; if None, read fromdt.attrs['pymodaq_name'].
- Return type:
- Raises:
ImportError – If xarray is not installed.
- static serialize(dte)[source]
Convert a DataToExport into a bytes string
- Parameters:
dte (
DataToExport)- Returns:
bytes
- Return type:
the total bytes messagetoserialize the DataToExport
Notes
The bytes sequence is constructed as:
serialize the string type: ‘DataToExport’
serialize the timestamp: float
serialize the name
serialize the list of DataWithAxes
- affect_name_to_origin_if_none()[source]
Affect self.name to all DataWithAxes children’s attribute origin if this origin is not defined
- average(other, weight)[source]
Compute the weighted average between self and other DataToExport and attributes it to self
- Parameters:
other (
DataToExport)weight (
int) – The weight the ‘other_data’ holds with respect to self
- Return type:
- get_data_from_Naxes(Naxes, deepcopy=False)[source]
Get the data matching the given number of axes
- Parameters:
Naxes (
int) – Number of axes in the DataWithAxes objects- Returns:
DataToExport
- Return type:
- get_data_from_attribute(attribute, attribute_value, deepcopy=False, sort_by_name=False)[source]
Get the data matching a given attribute value
- Parameters:
- Returns:
DataToExport
- Return type:
- get_data_from_dim(dim, deepcopy=False, sort_by_name=False)[source]
Get the data matching the given DataDim
- Returns:
DataToExport
- Return type:
- get_data_from_dims(dims, deepcopy=False, sort_by_name=False)[source]
Get the data matching the given DataDim
- Returns:
DataToExport
- Return type:
- get_data_from_full_name(full_name, deepcopy=False)[source]
Get the DataWithAxes with matching full name
- Return type:
- get_data_from_missing_attribute(attribute, deepcopy=False)[source]
Get the data matching a given attribute value
- Parameters:
- Returns:
DataToExport
- Return type:
- get_data_from_name(name)[source]
Get the data matching the given name
- Return type:
- get_data_from_name_origin(name, origin='')[source]
Get the data matching the given name and the given origin
- Return type:
- get_data_from_sig_axes(Naxes, deepcopy=False)[source]
Get the data matching the given number of signal axes
- Parameters:
Naxes (
int) – Number of signal axes in the DataWithAxes objects- Returns:
DataToExport
- Return type:
- get_data_from_source(source, deepcopy=False, sort_by_name=False)[source]
Get the data matching the given DataSource
- Returns:
DataToExport
- Return type:
- get_data_with_naxes_lower_than(n_axes=2, deepcopy=False)[source]
Get the data with n axes lower than the given number
- Parameters:
Naxes (
int) – Number of axes in the DataWithAxes objects- Returns:
DataToExport
- Return type:
- get_full_names(dim=None)[source]
Get the ful names including the origin attribute into the returned value, eventually filtered by dim
- Parameters:
dim (
DataDim)- Returns:
list of str
- Return type:
the namesof the (filtered) DataWithAxes data constructed as :origin/name
Examples
d0 = DataWithAxes(name=’datafromdet0’, origin=’det0’)
- get_names(dim=None)[source]
Get the names of the stored DataWithAxes, eventually filtered by dim
- get_origins(dim=None)[source]
Get the origins of the underlying data into the returned value, eventually filtered by dim
- Parameters:
dim (
DataDim)- Returns:
list of str
- Return type:
the originsofthe (filtered) DataWithAxes data
Examples
d0 = DataWithAxes(name=’datafromdet0’, origin=’det0’)
- index(data)[source]
Here use a comparison to assert data is equal to one element in the list
But the __eq__ method is not checking the name while it is the main issue for elt finding Hence here I’m doing both checks
- index_from_name_origin(name, origin='')[source]
Get the index of a given DataWithAxes within the list of data
- Return type:
- merge_as_dwa(dim, name=None)[source]
attempt to merge filtered dwa into one
Only possible if all filtered dwa and underlying data have same shape
- plot(plotter_backend='matplotlib', *args, **kwargs)[source]
Call a plotter factory and its plot method over the actual data
- pop(index)[source]
return and remove the DataWithAxes referred by its index
- Parameters:
index (
int) – index as returned by self.index_from_name_origin
See also
- Return type:
- remove(dwa)[source]
Use the DataWithAxes object comparison __eq__ to retrieve the elt to remove
- Parameters:
dwa (
DataWithAxes) – The da to remove from the list
- to_xarray()[source]
Convert this DataToExport to an xarray.DataTree.
The root node carries
pymodaq_namein its attrs. Each DataWithAxes becomes a child node whose dataset is produced byDataWithAxes.to_xarray().- Return type:
xr.DataTree- Raises:
ImportError – If xarray is not installed.
- property data: List[DataWithAxes]
get the data contained in the object
- Type:
List[DataWithAxes]