13. Implementation details¶

This section describes some of the assumptions and technical details of junifer, for developers. See the docstrings of the mentioned functions and classes for more details.

13.1. Elements¶

The keys of the elements of a PatternDataGrabber (PatternDataGrabber.get_element_keys()) are the replacements used by the patterns (including the nested ones) of the data types to grab, in the order of replacements.
PatternDataGrabber.get_elements() finds the elements of each pattern in the files (translating the Unix-like path expansion directives of the patterns to a regular expression) and joins the elements of the patterns on their common keys (_join_elements), so an element is available if it has the files of all the data types, regardless of their order.
The values of the replacements to grab (PatternDataGrabber.get_replacement_values()) are given by replacements (as a dictionary) or by the fields linked to the replacements in _REPLACEMENT_FIELDS, but not both. They filter the elements found in the files, and grabbing an element with other values raises an error (_check_element, called by BaseDataGrabber.__getitem__).
BaseDataGrabber.select_elements() selects the elements to run: the selectors must have the same number of values. If they have a value for each element key, they are the elements (the dataset is not searched), otherwise they select the elements of the dataset that have their values.
MultipleDataGrabber joins the elements of its DataGrabbers on their common keys, and grabs an element with each DataGrabber using its keys. Each data type is grabbed by one DataGrabber, except for the nested data types grabbed by a DataGrabber without the main file of the data type, which replace the ones of the other DataGrabber.

13.2. Metadata¶

The meta of each data type in the data object is built by the steps of the pipeline. Each step adds an entry with its class and parameters (UpdateMetaMixin.update_meta) and its dependencies:

Key	Added by	Content	In the MD5
`datagrabber`	DataGrabber	The class and parameters of the DataGrabber, e.g., `types`, `patterns` and `replacements`. For the DataLad-based ones, also the dataset ID and commit, and whether the dataset has local changes (`datalad_id`, `datalad_commit_id` and `datalad_dirty`), so different versions of the dataset give different hashes. For `MultipleDataGrabber`, also `datagrabbers` with the entries of its DataGrabbers.	Yes, except `datadir` for the DataLad-based DataGrabbers (which is removed from the stored metadata too)
`element`	DataGrabber	The element being processed (see below).	No
`_element_keys`	DataGrabber	The keys of the element that the data depends on (see below).	Yes
`datareader`	DataReader	The class and parameters of the DataReader.	Yes
`preprocess`	Preprocessors	The list of the classes and parameters of the preprocessors that processed the data, in order.	Yes
`type`	Marker	The data type, e.g., `BOLD`.	Yes
`marker`	Marker	The class and parameters of the marker, with `name` as the name of the marker followed by the name of the feature, e.g., `mean_mean`.	Yes
`dependencies`	All the steps	The names of the dependencies of the steps (their `_DEPENDENCIES`). When storing, they are replaced by their versions, with the version of `junifer`, e.g., `{'junifer': '0.0.8', 'nilearn': '0.14.1'}`.	Yes (the versions)
`name`	Storage (`process_meta`)	The name of the feature: the data type and the marker name, e.g., `BOLD_mean_mean`.	Yes

The DataGrabber adds two keys about the element to the meta of each data type (BaseDataGrabber.__getitem__):

element: the element being processed, with all its keys. It is the same for all the data types of the element.
_element_keys: the keys of the element that the data of the data type depends on (BaseDataGrabber.get_type_element_keys()). For a PatternDataGrabber, the replacements in the patterns of the data type.

For example, with BOLD data for each subject and task, and VBM_GM data for each subject, the element (sub-01, rest) gives:

{'BOLD': {'meta': {'_element_keys': ['subject', 'task'],
                   'element': {'subject': 'sub-01', 'task': 'rest'},
                   ...},
          ...},
 'VBM_GM': {'meta': {'_element_keys': ['subject'],
                     'element': {'subject': 'sub-01', 'task': 'rest'},
                     ...},
            ...}}

When storing the output of a marker, process_meta (in junifer.storage.utils):

Removes element from the metadata and returns the element of the data, with only the keys in _element_keys (all the keys of element if it is not set).
Computes the MD5 hash of the metadata, which identifies the feature. It includes _element_keys and the versions of the dependencies of the steps and of junifer, but not element: the data of all the elements of a feature have the same hash, and the features computed with different versions of junifer (or of the dependencies) have different hashes. This is by design, for reproducibility: the differences between features with the same hash can only come from the data, not from different versions of the algorithms. A feature computed again with another version is a new feature, with the same name, so it must be read by its MD5.

13.3. Storage¶

The data is stored for the element of the data (the element with only the keys in _element_keys), so the data that does not depend on a key (e.g., the VBM_GM data for the tasks of a subject) is stored once.
BaseFeatureStorage.store() passes the element being processed to the methods storing the metadata and the data (store_metadata, store_vector, etc.) as processed_element, besides the element of the data (element). The storages that write a file for each element (with single_output=False) name it after the element being processed, e.g., element_sub-01_rest_out.hdf5, so that the processes running different elements in parallel never write to the same file. The storages without files (e.g., databases) can ignore it, but their methods must accept it.
The data already stored for an element is not stored again (the storages skip the existing elements, SQLiteFeatureStorage with upsert="ignore" by default).
collect only collects the files of single elements (one element for each feature, as written for each element with single_output=False), and raises an error otherwise. The data of an element can be in the files of several elements being processed (e.g., the VBM_GM data of sub-01 in the files of each of its tasks), and it is only collected once.
HDF5FeatureStorage.collect() works in two passes: the first one reads the metadata and the elements of each file, to know which files to collect for each feature (one for each element), and the second one collects the data of each feature from these files, in chunks. Each file is read once in the first pass, and once for each of its features in the second one.