8.1.5. Storage

Provide imports for storage sub-package.

class junifer.storage.BaseFeatureStorage(uri, storage_types, single_output=True)

Abstract base class for feature storage.

For every interface that is required, one needs to provide a concrete implementation of this abstract class.

Parameters:

uristr or pathlib.Path

The path to the storage.

storage_typesstr or list of str

The available storage types for the class.

single_outputbool, optional

Whether to have single output (default True).

abstract collect()

Collect data.

get_valid_inputs()

Get valid storage types for input.

Returns:

list of str

The list of storage types that can be used as input for this ” “storage.

abstract list_features()

List the features in the storage.

Returns:

dict

List of features in the storage. The keys are the feature names to be used in read_features() and the values are the metadata of each feature.

abstract read_df(feature_name=None, feature_md5=None)

Read feature into a pandas DataFrame.

Parameters:

feature_namestr, optional

Name of the feature to read (default None).

feature_md5str, optional

MD5 hash of the feature to read (default None).

Returns:

pandas.DataFrame

The features as a dataframe.

store(kind, **kwargs)

Store extracted features data.

Parameters:

kind{“matrix”, “timeseries”, “table”}

The storage kind.

**kwargs

The keyword arguments.

Raises:

ValueError

If kind is invalid.

store_matrix(meta_md5, element, data, col_names=None, row_names=None, matrix_kind='full', diagonal=True)

Store matrix.

Parameters:

meta_md5str

The metadata MD5 hash.

elementdict

The element as a dictionary.

datanumpy.ndarray

The matrix data to store.

col_nameslist or tuple of str, optional

The column names (default None).

row_namesstr, optional

The column name to use in case number of rows greater than 1. If None and number of rows greater than 1, then the name will be “index” (default None).

matrix_kindstr, optional

The kind of matrix:

• triu : store upper triangular only

• tril : store lower triangular

• full : full matrix

(default “full”).

diagonalbool, optional

Whether to store the diagonal. If matrix_kind is “full”, setting this to False will raise an error (default True).

abstract store_metadata(meta_md5, element, meta)

Store metadata.

Parameters:

meta_md5str

The metadata MD5 hash.

elementdict

The element as a dictionary.

metadict

The metadata as a dictionary.

store_table(meta_md5, element, data, columns=None, rows_col_name=None)

Store table.

Parameters:

meta_md5str

The metadata MD5 hash.

elementdict

The element as a dictionary.

datanumpy.ndarray or list

The table data to store.

columnslist or tuple of str, optional

The columns (default None).

rows_col_namestr, optional

The column name to use in case number of rows greater than 1. If None and number of rows greater than 1, then the name will be “index” (default None).

store_timeseries(meta_md5, element, data, columns=None)

Implement timeseries storing.

Parameters:

meta_md5str

The metadata MD5 hash.

elementdict

The element as a dictionary.

datanumpy.ndarray

The timeseries data to store.

columnslist or tuple of str, optional

The column labels (default None).

validate(input_)

Validate the input to the pipeline step.

Parameters:

input_list of str

The input to the pipeline step.

Raises:

ValueError

If the input_ is invalid.

class junifer.storage.PandasBaseFeatureStorage(uri, single_output=True, **kwargs)

Abstract base class for feature storage via pandas.

For every interface that is required, one needs to provide a concrete implementation of this abstract class.

Parameters:

uristr or pathlib.Path

The path to the storage.

single_outputbool, optional

Whether to have single output (default True).

**kwargs

Keyword arguments passed to superclass.

See also

BaseFeatureStorage

The base class for feature storage.

static element_to_index(element, n_rows=1, rows_col_name=None)

Convert the element metadata to index.

Parameters:

elementdict

The element as a dictionary.

n_rowsint, optional

Number of rows to create (default 1).

rows_col_name: str, optional

The column name to use in case n_rows > 1. If None and n_rows > 1, the name will be “idx” (default None).

Returns:

pandas.MultiIndex

The index of the dataframe to store.

Raises:

ValueError

If meta does not contain the key “element”.

get_valid_inputs()

Get valid storage types for input.

Returns:

list of str

The list of storage types that can be used as input for this ” “storage.

store_df(meta_md5, element, df)

Implement pandas DataFrame storing.

Parameters:

dfpandas.DataFrame or pandas.Series

The pandas DataFrame or Series to store.

metadict

The metadata as a dictionary.

Raises:

ValueError

If the dataframe index has items that are not in the index generated from the metadata.

store_table(meta_md5, element, data, columns=None, rows_col_name=None)

Implement table storing.

Parameters:

meta_md5str

The metadata MD5 hash.

elementdict

The element as a dictionary.

datanumpy.ndarray or List

The table data to store.

columnslist or tuple of str, optional

The columns (default None).

rows_col_namestr, optional

The column name to use in case number of rows greater than 1. If None and number of rows greater than 1, then the name will be “index” (default None).

store_timeseries(meta_md5, element, data, columns=None)

Implement timeseries storing.

Parameters:

meta_md5str

The metadata MD5 hash.

elementdict

The element as a dictionary.

datanumpy.ndarray

The timeseries data to store.

columnslist or tuple of str, optional

The column labels (default None).

class junifer.storage.SQLiteFeatureStorage(uri, single_output=True, upsert='update', **kwargs)

Concrete implementation for feature storage via SQLite.

Parameters:

uristr or pathlib.Path

The path to the file to be used.

single_outputbool, optional

If False, will create one file per element. The name of the file will be prefixed with the respective element. If True, will create only one file as specified in the uri and store all the elements in the same file. This behaviour is only suitable for non-parallel executions. SQLite does not support concurrency (default True).

upsert{“ignore”, “update”}, optional

Upsert mode. If “ignore” is used, the existing elements are ignored. If “update”, the existing elements are updated (default “update”).

**kwargsdict

The keyword arguments passed to the superclass.

See also

PandasBaseFeatureStorage

The base class for Pandas-based feature storage.

collect()

Implement data collection.

Raises:

NotImplementedError

If single_output is True.

get_engine(element=None)

Get engine.

Parameters:

metadict, optional

The metadata as dictionary (default None).

Returns:

sqlalchemy.engine.Engine

The sqlalchemy engine.

list_features()

List the features in the storage.

Returns:

dict

List of features in the storage. The keys are the feature names to be used in read_features() and the values are the metadata of each feature.

read_df(feature_name=None, feature_md5=None)

Implement feature reading into a pandas DataFrame.

Either one of feature_name or feature_md5 needs to be specified.

Parameters:

feature_namestr, optional

Name of the feature to read (default None).

feature_md5str, optional

MD5 hash of the feature to read (default None).

Returns:

pandas.DataFrame

The features as a dataframe.

Raises:

ValueError

If parameter values are invalid or feature is not found or multiple features are found.

store_df(meta_md5, element, df)

Implement pandas DataFrame storing.

Parameters:

dfpandas.DataFrame or pandas.Series

The pandas DataFrame or Series to store.

metadict

The metadata as a dictionary.

Raises:

ValueError

If the dataframe index has items that are not in the index generated from the metadata.

store_matrix(meta_md5, element, data, col_names=None, row_names=None, matrix_kind='full', diagonal=True)

Implement matrix storing.

Parameters:

meta_md5str

The metadata MD5 hash.

elementdict

The element as a dictionary.

datanumpy.ndarray

The matrix data to store.

metadict

The metadata as a dictionary.

col_nameslist or tuple of str, optional

The column names (default None).

row_namesstr, optional

The column name to use in case number of rows greater than 1. If None and number of rows greater than 1, then the name will be “index” (default None).

matrix_kindstr, optional

The kind of matrix:

• triu : store upper triangular only

• tril : store lower triangular

• full : full matrix

(default “full”).

diagonalbool, optional

Whether to store the diagonal. If matrix_kind is “full”, setting this to False will raise an error (default True).

store_metadata(meta_md5, element, meta)

Implement metadata storing in the storage.

Parameters:

meta_md5str

The metadata MD5 hash.

elementdict

The element as a dictionary.

metadict

The metadata as a dictionary.