5.2. Running Jobs¶

Once we have the code-less configuration file, we can use the command line interface to extract the features. This is achieved in a two-step process: run and collect.

The run command is used to extract the features from each element in the dataset. However, depending on the storage interface, this may create one file per subject. The collect command is then used to collect all of the individual results into a single file.

Assuming that we have a configuration file named config.yaml, the following commands will extract the features:

junifer run config.yaml

The run command accepts the following additional arguments:

--help: Show a help message.
--verbose: Set the verbosity level. Options are warning, info, debug.
--element: The element to run. If not specified, all elements will be run. This parameter can be specified multiple times to run multiple elements. If the element requires several parameters, they can be specified by separating them with ,. It also accepts a file (e.g., elements.txt) containing complete or partial element(s) (see Complete and partial elements).

5.2.1. Example of running two elements:¶

junifer run config.yaml --element sub-01 --element sub-02

You can also specify the elements via a text file like so:

junifer run config.yaml --element elements.txt

And the corresponding elements.txt would be like so:

sub-01
sub-02

5.2.2. Example of elements with multiple parameters and verbose output:¶

junifer run --verbose info config.yaml --element sub-01,ses-01

You can also specify the elements via a text file like so:

junifer run --verbose info config.yaml --element elements.txt

And the corresponding elements.txt would be like so:

sub-01,ses-01

In case you wanted to run for all possible sessions (e.g., ses-01, ses-02, ses-03) but only for sub-01, you could also do:

junifer run --verbose info config.yaml --element sub-01

or,

junifer run --verbose info config.yaml --element elements.txt

and then the elements.txt would be like so:

sub-01

The elements can also be set in the YAML file, with the elements key. They are used by run, queue and list-elements when --element is not given (--element has precedence over the YAML file):

elements:
  - [sub-01, ses-01]
  - [sub-02, ses-01]

5.2.3. Complete and partial elements¶

An element has a value for each of its keys, e.g., subject and session. The keys depend on the data types that the DataGrabber grabs: an element only has the keys that the data types need. For example, a DataGrabber with resting-state BOLD (one per subject and session) and VBM data (one per subject) has the keys subject and session when grabbing the BOLD data, but only subject when grabbing only the VBM data, so that the VBM data of a subject is not processed once for each session. The list-elements command lists the elements with their values, in the order of the keys:

junifer list-elements config.yaml

The elements given with --element can be:

Complete, with a value for each key, in the order of the keys, e.g., sub-01,ses-01. Only these elements are run, without searching the dataset for its elements. This is the case for the jobs created by queue, which run one element each.
Partial, with fewer values, e.g., sub-01. They run all the elements of the dataset that have these values for any key, e.g., all the sessions of sub-01. This requires searching the dataset for its elements.

All the elements given in a command must have the same number of values, so they are either all complete or all partial. A partial element that does not match any element of the dataset is reported before running any element, while a complete element without data fails when its data is grabbed.

5.3. Collecting Results¶

Once the run command has been executed, the results are stored in the output directory. However, depending on the storage interface, this may create one file per subject. The collect command is then used to collect all of the individual results into a single file.

Assuming that we have a configuration file named config.yaml, the following commands will collect the results:

junifer collect config.yaml

The collect command accepts the following additional arguments:

--help: Show a help message.
--verbose: Set the verbosity level. Options are warning, info, debug.

5.4. Analysing Results¶

After collect-ing the results into a single file, we can analyse them as we wish. We would need to do this programmatically so feel free to choose your Python interpreter of choice or use it via a Python script.

First we load the storage like so:

from junifer.storage import HDF5FeatureStorage

# If the features were stored with the legacy SQLiteFeatureStorage,
# import and use that class instead
storage = HDF5FeatureStorage("<path/to/your/collected/file>")

The best way to start analysing would be to list all the extracted features like so:

# This would output a dictionary with MD5 checksum of the features as keys
# and metadata of the features as values
storage.list_features()

{'eb85b61eefba61f13d712d425264697b': {'datagrabber': {'class': 'SPMAuditoryTestingDataGrabber',
                                                      'types': ['BOLD', 'T1w']},
                                       'dependencies': {'scikit-learn': '1.3.0', 'nilearn': '0.9.2'},
                                       'datareader': {'class': 'DefaultDataReader'},
                                       'type': 'BOLD',
                                       'marker': {'class': 'FunctionalConnectivityParcels',
                                                  'parcellation': 'Schaefer100x7',
                                                  'agg_method': 'mean',
                                                  'agg_method_params': None,
                                                  'cor_method': 'covariance',
                                                  'cor_method_params': {'empirical': False},
                                                  'masks': None,
                                                  'name': 'schaefer_100x7_fc_parcels'},
                                       '_element_keys': ['subject'],
                                       'name': 'BOLD_schaefer_100x7_fc_parcels'}}

Once we have this, we can retrieve a single feature like so:

feature_dict = storage.read("<name-key-from-feature-metadata>")

to get the stored dictionary or,

feature_df = storage.read_df("<name-key-from-feature-metadata>")

to get it as a pandas.DataFrame.

If there are features with duplicate name s, then we would need to use the MD5 checksum and pass it like so:

feature_df = storage.read_df(feature_md5="<md5-hash-of-feature>")

We can now manipulate the dictionary or the pandas.DataFrame as we wish or use the DataFrame directly with julearn if desired.

5.4.1. On-the-fly transforms¶

junifer supports performing some computationally cheap operations (like computing brain connectivity or BrainPrint post-analysis) directly on the storage objects. To make sure everything works, install junifer like so:

pip install junifer[onthefly]

or if installed via conda, everything should be there and no further action is required.

Computing brain connectivity via bctpy ¶

You can compute for example, the degree of each node considering an undirected graph via bctpy like so:

from junifer.onthefly import read_transform

transformed_df = read_transform(
    storage,
    # md5 hash can also be passed via `feature_md5`
    feature_name="<name-key-from-feature-metadata>",
    # format is `package_function`
    transform="bctpy_degrees_und",
)

Post-analysis of BrainPrint results¶

To perform post-analysis on BrainPrint results, start by importing:

from junifer.onthefly import brainprint

Normalising:¶

Surface normalisation can be done by:

surface_normalised_df = brainprint.normalize(
    storage,
    {
        "areas": {
            "feature_name": "<name-key-from-areas-feature>",
            # if md5 hash is passed, the above should be None
            "feature_md5": None,
        },
        "eigenvalues": {
            "feature_name": "<name-key-from-eigenvalues-feature>",
            "feature_md5": None,
        },
    },
    kind="surface",
)

and volume normalisation can be done by:

volume_normalised_df = brainprint.normalize(
    storage,
    {
        "volumes": {
            "feature_name": "<name-key-from-volumes-feature>",
            "feature_md5": None,
        },
        "eigenvalues": {
            "feature_name": "<name-key-from-eigenvalues-feature>",
            "feature_md5": None,
        },
    },
    kind="volume",
)

Re-weighting:¶

To perform re-weighting, run like so:

reweighted_df = brainprint.reweight(
    storage,
    # md5 hash can also be passed via `feature_md5`
    feature_name="<name-key-from-eigenvalues-feature>",
)