6.4.7. Data Management

Axis([label, units, data, index, scaling, ...])

Object holding info and data about physical axis of some data

DataBase(name[, source, dim, distribution, ...])

Base object to store homogeneous data and metadata generated by pymodaq's objects.

DataRaw(*args, **kwargs)

Specialized DataWithAxes set with source as 'raw'.

DataCalculated(*args[, axes])

Specialized DataWithAxes set with source as 'calculated'.

DataFromPlugins(*args, **kwargs)

Specialized DataWithAxes set with source as 'raw'.

DataFromRoi(*args[, axes])

Specialized DataWithAxes set with source as 'calculated'.To be used for processed data from region of interest

Created the 28/10/2022

@author: Sebastien Weber

exception pymodaq.utils.data.DataDimWarning[source]
exception pymodaq.utils.data.DataIndexWarning[source]
exception pymodaq.utils.data.DataLengthError[source]
exception pymodaq.utils.data.DataShapeError[source]
exception pymodaq.utils.data.DataSizeWarning[source]
exception pymodaq.utils.data.DataTypeWarning[source]
pymodaq.utils.data.warning

alias of DataSizeWarning

class pymodaq.utils.data.AxesManagerSpread(*args, **kwargs)[source]

For this particular data category, some explanation is needed, see example below:

Examples

One take images data (20x30) as a function of 2 parameters, say xaxis and yaxis non-linearly spaced on a regular grid.

data.shape = (150, 20, 30) data.nav_indexes = (0,)

The first dimension (150) corresponds to the navigation (there are 150 non uniform data points taken) The second and third could correspond to signal data, here an image of size (20x30) so: * nav_indexes is (0, ) * sig_indexes are (1, 2)

xaxis = Axis(name=xaxis, index=0, data…) length 150 yaxis = Axis(name=yaxis, index=0, data…) length 150

In fact from such a data shape the number of navigation axes in unknown . In our example, they are 2. To somehow keep track of some ordering in these navigation axes, one adds an attribute to the Axis object: the spread_order xaxis = Axis(name=xaxis, index=0, spread_order=0, data…) length 150 yaxis = Axis(name=yaxis, index=0, spread_order=1, data…) length 150

Methods

compute_shape_from_axes()

Get data shape from axes

get_axis_from_index(index[, create])

in spread mode, different nav axes have the same index (but not the same spread_order integer value) so may return multiple axis

get_shape_from_index(index)

Get the data shape at the given index

compute_shape_from_axes()[source]

Get data shape from axes

First get the nav length from one of the navigation axes Then check for signal axes

get_axis_from_index(index: int, create: bool = False) List[Axis][source]

in spread mode, different nav axes have the same index (but not the same spread_order integer value) so may return multiple axis

No possible “linear” creation in this mode

get_shape_from_index(index: int) int[source]

Get the data shape at the given index

class pymodaq.utils.data.Axis(label: str = '', units: str = '', data: Optional[ndarray] = None, index: int = 0, scaling=None, offset=None, spread_order: Optional[int] = None)[source]

Object holding info and data about physical axis of some data

In case the axis’s data is linear, store the info as a scale and offset else store the data

Parameters
  • label (str) – The label of the axis, for instance ‘time’ for a temporal axis

  • units (str) – The units of the data in the object, for instance ‘s’ for seconds

  • data (ndarray) – A 1D ndarray holding the data of the axis

  • index (int) – an integer representing the index of the Data object this axis is related to

  • scaling (float) – The scaling to apply to a linspace version in order to obtain the proper scaling

  • offset (float) – The offset to apply to a linspace/scaled version in order to obtain the proper axis

  • spread_order (int) – An integer needed in the case where data has a spread DataDistribution. It refers to the index along the data’s spread_index dimension

Attributes
data

np.ndarray: get/set the data of Axis

index

int: get/set the index this axis corresponds to in a DataWithAxis object

label

str: get/set the label of this axis

offset
scaling
size

int: get/set the size/length of the 1D ndarray

units

str: get/set the units for this axis

Methods

create_linear_data(nsteps)

replace the axis data with a linear version using scaling and offset

find_index(threshold)

find the index of the threshold value within the axis

get_data()

Convenience method to obtain the axis data (usually None because scaling and offset are used)

get_scale_offset_from_data([data])

Get the scaling and offset from the axis's data

copy

create_simple_linear_data

find_indexes

is_axis_linear

max

mean

min

create_linear_data(nsteps: int)[source]

replace the axis data with a linear version using scaling and offset

find_index(threshold: float) int[source]

find the index of the threshold value within the axis

get_data() ndarray[source]

Convenience method to obtain the axis data (usually None because scaling and offset are used)

get_scale_offset_from_data(data: Optional[ndarray] = None)[source]

Get the scaling and offset from the axis’s data

If data is not None, extract the scaling and offset

Parameters

data (ndarray) –

property data

get/set the data of Axis

Type

np.ndarray

property index: int

get/set the index this axis corresponds to in a DataWithAxis object

Type

int

property label: str

get/set the label of this axis

Type

str

property size: int

get/set the size/length of the 1D ndarray

Type

int

property units: str

get/set the units for this axis

Type

str

class pymodaq.utils.data.DataBase(name: str, source: Optional[DataSource] = None, dim: Optional[DataDim] = None, distribution: DataDistribution = DataDistribution.uniform, data: Optional[List[ndarray]] = None, labels: List[str] = [], origin: Optional[str] = None, **kwargs)[source]

Base object to store homogeneous data and metadata generated by pymodaq’s objects. To be inherited for real data

Parameters
  • name (str) – the identifier of these data

  • source (DataSource or str) – Enum specifying if data are raw or processed (for instance from roi)

  • dim (DataDim or str) – The identifier of the data type

  • distribution (DataDistribution or str) – The distribution type of the data: uniform if distributed on a regular grid or spread if on specific unordered points

  • data (list of ndarray) – The data the object is storing

  • labels (list of str) – The labels of the data nd-arrays

  • origin (str) – An identifier of the element where the data originated, for instance the DAQ_Viewer’s name. Used when appending DataToExport in DAQ_Scan to disintricate from which origin data comes from when scanning multiple detectors.

  • kwargs (named parameters) – All other parameters are stored dynamically using the name/value pair. The name of these extra parameters are added into the extra_attributes attribute

name

the identifier of these data

Type

str

source

Enum specifying if data are raw or processed (for instance from roi)

Type

DataSource or str

dim

The identifier of the data type

Type

DataDim or str

distribution

The distribution type of the data: uniform if distributed on a regular grid or spread if on specific unordered points

Type

DataDistribution or str

data

The data the object is storing

Type

list of ndarray

labels

The labels of the data nd-arrays

Type

list of str

origin

An identifier of the element where the data originated, for instance the DAQ_Viewer’s name. Used when appending DataToExport in DAQ_Scan to disintricate from which origin data comes from when scanning multiple detectors.

Type

str

shape

The shape of the underlying data

Type

Tuple[int]

size

The size of the ndarrays stored in the object

Type

int

length

The number of ndarrays stored in the object

Type

int

extra_attributes

list of string giving identifiers of the attributes added dynamically at the initialization (for instance to save extra metadata using the DataSaverLoader

Type

List[str]

See also

DataWithAxes, DataFromPlugins, DataRaw, DataSaverLoader

Attributes
data

List[np.ndarray]: get/set (and check) the data the object is storing

dim

DataDim: the enum representing the dimensionality of the stored data

distribution

DataDistribution: the enum representing the distribution of the stored data

labels
length

The length of data.

shape

The shape of the nd-arrays

size

The size of the nd-arrays

source

DataSource: the enum representing the source of the data

Methods

average(other, weight)

Compute the weighted average between self and other DataBase and attributes it to self

get_data_index([index])

Get the data by its index in the list

get_dim_from_data(data)

Get the dimensionality DataDim from data

get_full_name()

Get the data ful name including the origin attribute into the returned value

set_dim(dim)

Addhoc modification of dim independantly of the real data shape, should be used with extra care

append

check_shape_from_data

average(other: DataBase, weight: int) DataBase[source]

Compute the weighted average between self and other DataBase and attributes it to self

Parameters
  • other_data (DataBase) –

  • weight (int) – The weight the ‘other’ holds with respect to self

get_data_index(index: int = 0)[source]

Get the data by its index in the list

get_dim_from_data(data: List[ndarray])[source]

Get the dimensionality DataDim from data

get_full_name() str[source]

Get the data ful name including the origin attribute into the returned value

Returns

str

Return type

the name of the ataWithAxes data constructed as : origin/name

Examples

d0 = DataBase(name=’datafromdet0’, origin=’det0’)

set_dim(dim: Union[DataDim, str])[source]

Addhoc modification of dim independantly of the real data shape, should be used with extra care

property data: List[ndarray]

get/set (and check) the data the object is storing

Type

List[np.ndarray]

property dim

the enum representing the dimensionality of the stored data

Type

DataDim

property distribution

the enum representing the distribution of the stored data

Type

DataDistribution

property length

The length of data. This is the length of the list containing the nd-arrays

property shape

The shape of the nd-arrays

property size

The size of the nd-arrays

property source

the enum representing the source of the data

Type

DataSource

class pymodaq.utils.data.DataCalculated(*args, axes=[], **kwargs)[source]

Specialized DataWithAxes set with source as ‘calculated’. To be used for processed/calculated data

class pymodaq.utils.data.DataDim(value)[source]

Enum for dimensionality representation of data

class pymodaq.utils.data.DataDistribution(value)[source]

Enum for distribution of data

class pymodaq.utils.data.DataFromPlugins(*args, **kwargs)[source]

Specialized DataWithAxes set with source as ‘raw’. To be used for raw data generated by plugins

class pymodaq.utils.data.DataFromRoi(*args, axes=[], **kwargs)[source]

Specialized DataWithAxes set with source as ‘calculated’.To be used for processed data from region of interest

class pymodaq.utils.data.DataLowLevel(name: str)[source]

Abstract object for all Data Object

Parameters

name (str) – the identifier of the data

name
Type

str

timestamp

Time in seconds since epoch. See method time.time()

Type

float

Attributes
name

str: the identifier of the data

timestamp

Get/Set the timestamp of when the object has been created

property name

the identifier of the data

Type

str

property timestamp

Get/Set the timestamp of when the object has been created

class pymodaq.utils.data.DataRaw(*args, **kwargs)[source]

Specialized DataWithAxes set with source as ‘raw’. To be used for raw data

class pymodaq.utils.data.DataScan(name: str, data: List[DataWithAxes] = [], **kwargs)[source]

Specialized DataToExport.To be used for data to be saved

class pymodaq.utils.data.DataSource(value)[source]

Enum for source of data

class pymodaq.utils.data.DataToExport(name: str, data: List[DataWithAxes] = [], **kwargs)[source]

Object to store all raw and calculated DataWithAxes data for later exporting, saving, sending signal…

Includes methods to retrieve data from dim, source… Stored data have a unique identifier their name. If some data is appended with an existing name, it will replace the existing data. So if you want to append data that has the same name

Parameters
  • name (str) – The identifier of the exporting object

  • data (list of DataWithAxes) – All the raw and calculated data to be exported

name
timestamp
data
Attributes
data

List[DataWithAxes]: get the data contained in the object

Methods

affect_name_to_origin_if_none()

Affect self.name to all DataWithAxes children's attribute origin if this origin is not defined

append

Dispatch methods based on type signature

average(other, weight)

Compute the weighted average between self and other DataToExport and attributes it to self

get_data_from_Naxes(Naxes[, deepcopy])

Get the data matching the given number of axes

get_data_from_dim(dim[, deepcopy])

Get the data matching the given DataDim

get_data_from_dims(dims[, deepcopy])

Get the data matching the given DataDim

get_data_from_full_name(full_name[, deepcopy])

Get the DataWithAxes with matching full name

get_data_from_name(name)

Get the data matching the given name

get_data_from_name_origin(name[, origin])

Get the data matching the given name and the given origin

get_data_from_source(source[, deepcopy])

Get the data matching the given DataSource

get_full_names([dim])

Get the ful names including the origin attribute into the returned value, eventually filtered by dim

get_names([dim])

Get the names of the stored DataWithAxes, eventually filtered by dim

index_from_name_origin(name[, origin])

Get the index of a given DataWithAxes within the list of data

pop(index)

return and remove the DataWithAxes referred by its index

get_data_from_full_names

get_dim_presents

index

remove

affect_name_to_origin_if_none()[source]

Affect self.name to all DataWithAxes children’s attribute origin if this origin is not defined

average(other: DataToExport, weight: int) DataToExport[source]

Compute the weighted average between self and other DataToExport and attributes it to self

Parameters
  • other (DataToExport) –

  • weight (int) – The weight the ‘other_data’ holds with respect to self

get_data_from_Naxes(Naxes: int, deepcopy: bool = False) DataToExport[source]

Get the data matching the given number of axes

Returns

DataToExport

Return type

filtered with data matching the number of axes

get_data_from_dim(dim: DataDim, deepcopy=False) DataToExport[source]

Get the data matching the given DataDim

Returns

DataToExport

Return type

filtered with data matching the dimensionality

get_data_from_dims(dims: List[DataDim], deepcopy=False) DataToExport[source]

Get the data matching the given DataDim

Returns

DataToExport

Return type

filtered with data matching the dimensionality

get_data_from_full_name(full_name: str, deepcopy=False) DataWithAxes[source]

Get the DataWithAxes with matching full name

get_data_from_name(name: str) List[DataWithAxes][source]

Get the data matching the given name

get_data_from_name_origin(name: str, origin: Optional[str] = None) DataWithAxes[source]

Get the data matching the given name and the given origin

get_data_from_source(source: DataSource, deepcopy=False) DataToExport[source]

Get the data matching the given DataSource

Returns

DataToExport

Return type

filtered with data matching the dimensionality

get_full_names(dim: Optional[DataDim] = None)[source]

Get the ful names including the origin attribute into the returned value, eventually filtered by dim

Parameters

dim (DataDim or str) –

Returns

list of str

Return type

the names of the (filtered) DataWithAxes data constructed as : origin/name

Examples

d0 = DataWithAxes(name=’datafromdet0’, origin=’det0’)

get_names(dim: Optional[DataDim] = None)[source]

Get the names of the stored DataWithAxes, eventually filtered by dim

Parameters

dim (DataDim or str) –

Returns

list of str

Return type

the names of the (filtered) DataWithAxes data

index_from_name_origin(name: str, origin: Optional[str] = None) List[DataWithAxes][source]

Get the index of a given DataWithAxes within the list of data

pop(index: int) DataWithAxes[source]

return and remove the DataWithAxes referred by its index

Parameters

index (int) – index as returned by self.index_from_name_origin

property data: List[DataWithAxes]

get the data contained in the object

Type

List[DataWithAxes]

class pymodaq.utils.data.DataWithAxes(*args, axes: List[Axis] = [], nav_indexes: Tuple[int] = (), **kwargs)[source]

Data object with Axis objects corresponding to underlying data nd-arrays

Parameters
  • axes (list of Axis) – the list of Axis object for proper plotting, calibration …

  • nav_indexes (tuple of int) – highlight which Axis in axes is Signal or Navigation axis depending on the content: For instance, nav_indexes = (2,), means that the axis with index 2 in a at least 3D ndarray data is the first navigation axis For instance, nav_indexes = (3,2), means that the axis with index 3 in a at least 4D ndarray data is the first navigation axis while the axis with index 2 is the second navigation Axis. Axes with index 0 and 1 are signal axes of 2D ndarray data

Attributes
axes

convenience property to fetch attribute from axis_manager

nav_indexes

convenience property to fetch attribute from axis_manager

sig_indexes

convenience property to fetch attribute from axis_manager

Methods

create_missing_axes()

Check if given the data shape, some axes are missing to properly define the data (especially for plotting)

deepcopy_with_new_data([data, ...])

deepcopy without copying the initial data (saving memory)

get_axis_indexes()

Get all present different axis indexes

get_dim_from_data_axes()

Get the dimensionality DataDim from data taking into account nav indexes

get_nav_axes_with_data()

Get the data's navigation axes making sure there is data in the data field

mean([axis])

Process the mean of the data on the specified axis and returns the new data

sort_data([nav_axis])

Sort spread data along a given navigation axis, default is 0

transpose()

replace the data by their transposed version

deepcopy

get_axis_from_index

get_data_dimension

get_nav_axes

set_axes_manager

create_missing_axes()[source]

Check if given the data shape, some axes are missing to properly define the data (especially for plotting)

deepcopy_with_new_data(data: Optional[List[ndarray]] = None, remove_axes_index: Optional[List[int]] = None, source: DataSource = 'calculated') DataWithAxes[source]

deepcopy without copying the initial data (saving memory)

The new data, may have some axes stripped as specified in remove_axes_index

get_axis_indexes() List[int][source]

Get all present different axis indexes

get_dim_from_data_axes() DataDim[source]

Get the dimensionality DataDim from data taking into account nav indexes

get_nav_axes_with_data() List[Axis][source]

Get the data’s navigation axes making sure there is data in the data field

mean(axis: int = 0) DataWithAxes[source]

Process the mean of the data on the specified axis and returns the new data

Parameters

axis (int) –

Return type

DataWithAxes

sort_data(nav_axis: int = 0)[source]

Sort spread data along a given navigation axis, default is 0

transpose()[source]

replace the data by their transposed version

Valid only for 2D data

property axes

convenience property to fetch attribute from axis_manager

property nav_indexes

convenience property to fetch attribute from axis_manager

property sig_indexes

convenience property to fetch attribute from axis_manager

class pymodaq.utils.data.NavAxis(*args, **kwargs)[source]