6.4.7. Data Management¶
|
Object holding info and data about physical axis of some data |
|
Base object to store homogeneous data and metadata generated by pymodaq's objects. |
|
Specialized DataWithAxes set with source as 'raw'. |
|
Specialized DataWithAxes set with source as 'calculated'. |
|
Specialized DataWithAxes set with source as 'raw'. |
|
Specialized DataWithAxes set with source as 'calculated'.To be used for processed data from region of interest |
Created the 28/10/2022
@author: Sebastien Weber
- pymodaq.utils.data.warning¶
alias of
DataSizeWarning
- class pymodaq.utils.data.AxesManagerSpread(*args, **kwargs)[source]¶
For this particular data category, some explanation is needed, see example below:
Examples
One take images data (20x30) as a function of 2 parameters, say xaxis and yaxis non-linearly spaced on a regular grid.
data.shape = (150, 20, 30) data.nav_indexes = (0,)
The first dimension (150) corresponds to the navigation (there are 150 non uniform data points taken) The second and third could correspond to signal data, here an image of size (20x30) so: * nav_indexes is (0, ) * sig_indexes are (1, 2)
xaxis = Axis(name=xaxis, index=0, data…) length 150 yaxis = Axis(name=yaxis, index=0, data…) length 150
In fact from such a data shape the number of navigation axes in unknown . In our example, they are 2. To somehow keep track of some ordering in these navigation axes, one adds an attribute to the Axis object: the spread_order xaxis = Axis(name=xaxis, index=0, spread_order=0, data…) length 150 yaxis = Axis(name=yaxis, index=0, spread_order=1, data…) length 150
Methods
Get data shape from axes
get_axis_from_index
(index[, create])in spread mode, different nav axes have the same index (but not the same spread_order integer value) so may return multiple axis
get_shape_from_index
(index)Get the data shape at the given index
- compute_shape_from_axes()[source]¶
Get data shape from axes
First get the nav length from one of the navigation axes Then check for signal axes
- class pymodaq.utils.data.Axis(label: str = '', units: str = '', data: Optional[ndarray] = None, index: int = 0, scaling=None, offset=None, spread_order: Optional[int] = None)[source]¶
Object holding info and data about physical axis of some data
In case the axis’s data is linear, store the info as a scale and offset else store the data
- Parameters
label (str) – The label of the axis, for instance ‘time’ for a temporal axis
units (str) – The units of the data in the object, for instance ‘s’ for seconds
data (ndarray) – A 1D ndarray holding the data of the axis
index (int) – an integer representing the index of the Data object this axis is related to
scaling (float) – The scaling to apply to a linspace version in order to obtain the proper scaling
offset (float) – The offset to apply to a linspace/scaled version in order to obtain the proper axis
spread_order (int) – An integer needed in the case where data has a spread DataDistribution. It refers to the index along the data’s spread_index dimension
- Attributes
Methods
create_linear_data
(nsteps)replace the axis data with a linear version using scaling and offset
find_index
(threshold)find the index of the threshold value within the axis
get_data
()Convenience method to obtain the axis data (usually None because scaling and offset are used)
get_scale_offset_from_data
([data])Get the scaling and offset from the axis's data
copy
create_simple_linear_data
find_indexes
is_axis_linear
max
mean
min
- create_linear_data(nsteps: int)[source]¶
replace the axis data with a linear version using scaling and offset
- get_data() ndarray [source]¶
Convenience method to obtain the axis data (usually None because scaling and offset are used)
- get_scale_offset_from_data(data: Optional[ndarray] = None)[source]¶
Get the scaling and offset from the axis’s data
If data is not None, extract the scaling and offset
- Parameters
data (ndarray) –
- property data¶
get/set the data of Axis
- Type
np.ndarray
- class pymodaq.utils.data.DataBase(name: str, source: Optional[DataSource] = None, dim: Optional[DataDim] = None, distribution: DataDistribution = DataDistribution.uniform, data: Optional[List[ndarray]] = None, labels: List[str] = [], origin: Optional[str] = None, **kwargs)[source]¶
Base object to store homogeneous data and metadata generated by pymodaq’s objects. To be inherited for real data
- Parameters
name (str) – the identifier of these data
source (DataSource or str) – Enum specifying if data are raw or processed (for instance from roi)
distribution (DataDistribution or str) – The distribution type of the data: uniform if distributed on a regular grid or spread if on specific unordered points
data (list of ndarray) – The data the object is storing
origin (str) – An identifier of the element where the data originated, for instance the DAQ_Viewer’s name. Used when appending DataToExport in DAQ_Scan to disintricate from which origin data comes from when scanning multiple detectors.
kwargs (named parameters) – All other parameters are stored dynamically using the name/value pair. The name of these extra parameters are added into the extra_attributes attribute
- source¶
Enum specifying if data are raw or processed (for instance from roi)
- Type
DataSource or str
- distribution¶
The distribution type of the data: uniform if distributed on a regular grid or spread if on specific unordered points
- Type
- origin¶
An identifier of the element where the data originated, for instance the DAQ_Viewer’s name. Used when appending DataToExport in DAQ_Scan to disintricate from which origin data comes from when scanning multiple detectors.
- Type
- extra_attributes¶
list of string giving identifiers of the attributes added dynamically at the initialization (for instance to save extra metadata using the DataSaverLoader
- Type
List[str]
See also
DataWithAxes
,DataFromPlugins
,DataRaw
,DataSaverLoader
- Attributes
data
List[np.ndarray]: get/set (and check) the data the object is storing
dim
DataDim: the enum representing the dimensionality of the stored data
distribution
DataDistribution: the enum representing the distribution of the stored data
- labels
length
The length of data.
shape
The shape of the nd-arrays
size
The size of the nd-arrays
source
DataSource: the enum representing the source of the data
Methods
average
(other, weight)Compute the weighted average between self and other DataBase and attributes it to self
get_data_index
([index])Get the data by its index in the list
get_dim_from_data
(data)Get the dimensionality DataDim from data
Get the data ful name including the origin attribute into the returned value
set_dim
(dim)Addhoc modification of dim independantly of the real data shape, should be used with extra care
append
check_shape_from_data
- average(other: DataBase, weight: int) DataBase [source]¶
Compute the weighted average between self and other DataBase and attributes it to self
- get_full_name() str [source]¶
Get the data ful name including the origin attribute into the returned value
- Returns
str
- Return type
the name of the ataWithAxes data constructed as : origin/name
Examples
d0 = DataBase(name=’datafromdet0’, origin=’det0’)
- set_dim(dim: Union[DataDim, str])[source]¶
Addhoc modification of dim independantly of the real data shape, should be used with extra care
- property data: List[ndarray]¶
get/set (and check) the data the object is storing
- Type
List[np.ndarray]
- property distribution¶
the enum representing the distribution of the stored data
- Type
- property length¶
The length of data. This is the length of the list containing the nd-arrays
- property shape¶
The shape of the nd-arrays
- property size¶
The size of the nd-arrays
- property source¶
the enum representing the source of the data
- Type
- class pymodaq.utils.data.DataCalculated(*args, axes=[], **kwargs)[source]¶
Specialized DataWithAxes set with source as ‘calculated’. To be used for processed/calculated data
- class pymodaq.utils.data.DataFromPlugins(*args, **kwargs)[source]¶
Specialized DataWithAxes set with source as ‘raw’. To be used for raw data generated by plugins
- class pymodaq.utils.data.DataFromRoi(*args, axes=[], **kwargs)[source]¶
Specialized DataWithAxes set with source as ‘calculated’.To be used for processed data from region of interest
- class pymodaq.utils.data.DataLowLevel(name: str)[source]¶
Abstract object for all Data Object
- Parameters
name (str) – the identifier of the data
- Attributes
- property timestamp¶
Get/Set the timestamp of when the object has been created
- class pymodaq.utils.data.DataRaw(*args, **kwargs)[source]¶
Specialized DataWithAxes set with source as ‘raw’. To be used for raw data
- class pymodaq.utils.data.DataScan(name: str, data: List[DataWithAxes] = [], **kwargs)[source]¶
Specialized DataToExport.To be used for data to be saved
- class pymodaq.utils.data.DataToExport(name: str, data: List[DataWithAxes] = [], **kwargs)[source]¶
Object to store all raw and calculated DataWithAxes data for later exporting, saving, sending signal…
Includes methods to retrieve data from dim, source… Stored data have a unique identifier their name. If some data is appended with an existing name, it will replace the existing data. So if you want to append data that has the same name
- Parameters
name (str) – The identifier of the exporting object
data (list of DataWithAxes) – All the raw and calculated data to be exported
- name¶
- timestamp¶
- data¶
- Attributes
data
List[DataWithAxes]: get the data contained in the object
Methods
Affect self.name to all DataWithAxes children's attribute origin if this origin is not defined
append
Dispatch methods based on type signature
average
(other, weight)Compute the weighted average between self and other DataToExport and attributes it to self
get_data_from_Naxes
(Naxes[, deepcopy])Get the data matching the given number of axes
get_data_from_dim
(dim[, deepcopy])Get the data matching the given DataDim
get_data_from_dims
(dims[, deepcopy])Get the data matching the given DataDim
get_data_from_full_name
(full_name[, deepcopy])Get the DataWithAxes with matching full name
get_data_from_name
(name)Get the data matching the given name
get_data_from_name_origin
(name[, origin])Get the data matching the given name and the given origin
get_data_from_source
(source[, deepcopy])Get the data matching the given DataSource
get_full_names
([dim])Get the ful names including the origin attribute into the returned value, eventually filtered by dim
get_names
([dim])Get the names of the stored DataWithAxes, eventually filtered by dim
index_from_name_origin
(name[, origin])Get the index of a given DataWithAxes within the list of data
pop
(index)return and remove the DataWithAxes referred by its index
get_data_from_full_names
get_dim_presents
index
remove
- affect_name_to_origin_if_none()[source]¶
Affect self.name to all DataWithAxes children’s attribute origin if this origin is not defined
- average(other: DataToExport, weight: int) DataToExport [source]¶
Compute the weighted average between self and other DataToExport and attributes it to self
- Parameters
other (DataToExport) –
weight (int) – The weight the ‘other_data’ holds with respect to self
- get_data_from_Naxes(Naxes: int, deepcopy: bool = False) DataToExport [source]¶
Get the data matching the given number of axes
- Returns
DataToExport
- Return type
filtered with data matching the number of axes
- get_data_from_dim(dim: DataDim, deepcopy=False) DataToExport [source]¶
Get the data matching the given DataDim
- Returns
DataToExport
- Return type
filtered with data matching the dimensionality
- get_data_from_dims(dims: List[DataDim], deepcopy=False) DataToExport [source]¶
Get the data matching the given DataDim
- Returns
DataToExport
- Return type
filtered with data matching the dimensionality
- get_data_from_full_name(full_name: str, deepcopy=False) DataWithAxes [source]¶
Get the DataWithAxes with matching full name
- get_data_from_name(name: str) List[DataWithAxes] [source]¶
Get the data matching the given name
- get_data_from_name_origin(name: str, origin: Optional[str] = None) DataWithAxes [source]¶
Get the data matching the given name and the given origin
- get_data_from_source(source: DataSource, deepcopy=False) DataToExport [source]¶
Get the data matching the given DataSource
- Returns
DataToExport
- Return type
filtered with data matching the dimensionality
- get_full_names(dim: Optional[DataDim] = None)[source]¶
Get the ful names including the origin attribute into the returned value, eventually filtered by dim
- Parameters
- Returns
list of str
- Return type
the names of the (filtered) DataWithAxes data constructed as : origin/name
Examples
d0 = DataWithAxes(name=’datafromdet0’, origin=’det0’)
- get_names(dim: Optional[DataDim] = None)[source]¶
Get the names of the stored DataWithAxes, eventually filtered by dim
- index_from_name_origin(name: str, origin: Optional[str] = None) List[DataWithAxes] [source]¶
Get the index of a given DataWithAxes within the list of data
- pop(index: int) DataWithAxes [source]¶
return and remove the DataWithAxes referred by its index
- Parameters
index (int) – index as returned by self.index_from_name_origin
See also
- property data: List[DataWithAxes]¶
get the data contained in the object
- Type
List[DataWithAxes]
- class pymodaq.utils.data.DataWithAxes(*args, axes: List[Axis] = [], nav_indexes: Tuple[int] = (), **kwargs)[source]¶
Data object with Axis objects corresponding to underlying data nd-arrays
- Parameters
axes (list of Axis) – the list of Axis object for proper plotting, calibration …
nav_indexes (tuple of int) – highlight which Axis in axes is Signal or Navigation axis depending on the content: For instance, nav_indexes = (2,), means that the axis with index 2 in a at least 3D ndarray data is the first navigation axis For instance, nav_indexes = (3,2), means that the axis with index 3 in a at least 4D ndarray data is the first navigation axis while the axis with index 2 is the second navigation Axis. Axes with index 0 and 1 are signal axes of 2D ndarray data
- Attributes
axes
convenience property to fetch attribute from axis_manager
nav_indexes
convenience property to fetch attribute from axis_manager
sig_indexes
convenience property to fetch attribute from axis_manager
Methods
Check if given the data shape, some axes are missing to properly define the data (especially for plotting)
deepcopy_with_new_data
([data, ...])deepcopy without copying the initial data (saving memory)
Get all present different axis indexes
Get the dimensionality DataDim from data taking into account nav indexes
Get the data's navigation axes making sure there is data in the data field
mean
([axis])Process the mean of the data on the specified axis and returns the new data
sort_data
([nav_axis])Sort spread data along a given navigation axis, default is 0
replace the data by their transposed version
deepcopy
get_axis_from_index
get_data_dimension
get_nav_axes
set_axes_manager
- create_missing_axes()[source]¶
Check if given the data shape, some axes are missing to properly define the data (especially for plotting)
- deepcopy_with_new_data(data: Optional[List[ndarray]] = None, remove_axes_index: Optional[List[int]] = None, source: DataSource = 'calculated') DataWithAxes [source]¶
deepcopy without copying the initial data (saving memory)
The new data, may have some axes stripped as specified in remove_axes_index
- get_dim_from_data_axes() DataDim [source]¶
Get the dimensionality DataDim from data taking into account nav indexes
Get the data’s navigation axes making sure there is data in the data field
- mean(axis: int = 0) DataWithAxes [source]¶
Process the mean of the data on the specified axis and returns the new data
- Parameters
axis (int) –
- Return type
- property axes¶
convenience property to fetch attribute from axis_manager
convenience property to fetch attribute from axis_manager
- property sig_indexes¶
convenience property to fetch attribute from axis_manager