.. include:: ../links.inc .. _running: Running Jobs ============ Once we have the :ref:`code-less configuration file `, we can use the command line interface to extract the features. This is achieved in a two-step process: ``run`` and ``collect``. The ``run`` command is used to extract the features from each element in the dataset. However, depending on the storage interface, this may create one file per subject. The ``collect`` command is then used to collect all of the individual results into a single file. Assuming that we have a configuration file named ``config.yaml``, the following commands will extract the features: .. code-block:: bash junifer run config.yaml The ``run`` command accepts the following additional arguments: * ``--help``: Show a help message. * ``--verbose``: Set the verbosity level. Options are ``warning``, ``info``, ``debug``. * ``--element``: The *element* to run. If not specified, all elements will be run. This parameter can be specified multiple times to run multiple elements. If the *element* requires several parameters, they can be specified by separating them with ``,``. It also accepts a file (e.g., ``elements.txt``) containing complete or partial element(s) (see :ref:`running_elements`). Example of running two elements: -------------------------------- .. code-block:: bash junifer run config.yaml --element sub-01 --element sub-02 You can also specify the elements via a text file like so: .. code-block:: bash junifer run config.yaml --element elements.txt And the corresponding ``elements.txt`` would be like so: .. code-block:: text sub-01 sub-02 Example of elements with multiple parameters and verbose output: ---------------------------------------------------------------- .. code-block:: bash junifer run --verbose info config.yaml --element sub-01,ses-01 You can also specify the elements via a text file like so: .. code-block:: bash junifer run --verbose info config.yaml --element elements.txt And the corresponding ``elements.txt`` would be like so: .. code-block:: text sub-01,ses-01 In case you wanted to run for all possible sessions (e.g., ``ses-01``, ``ses-02``, ``ses-03``) but only for ``sub-01``, you could also do: .. code-block:: bash junifer run --verbose info config.yaml --element sub-01 or, .. code-block:: bash junifer run --verbose info config.yaml --element elements.txt and then the ``elements.txt`` would be like so: .. code-block:: text sub-01 The elements can also be set in the YAML file, with the ``elements`` key. They are used by ``run``, ``queue`` and ``list-elements`` when ``--element`` is not given (``--element`` has precedence over the YAML file): .. code-block:: yaml elements: - [sub-01, ses-01] - [sub-02, ses-01] .. _running_elements: Complete and partial elements ----------------------------- An *element* has a value for each of its *keys*, e.g., ``subject`` and ``session``. The keys depend on the :ref:`data types ` that the DataGrabber grabs: an element only has the keys that the data types need. For example, a DataGrabber with resting-state BOLD (one per subject and session) and VBM data (one per subject) has the keys ``subject`` and ``session`` when grabbing the BOLD data, but only ``subject`` when grabbing only the VBM data, so that the VBM data of a subject is not processed once for each session. The ``list-elements`` command lists the elements with their values, in the order of the keys: .. code-block:: bash junifer list-elements config.yaml The elements given with ``--element`` can be: * *Complete*, with a value for each key, in the order of the keys, e.g., ``sub-01,ses-01``. Only these elements are run, without searching the dataset for its elements. This is the case for the jobs created by :ref:`queue `, which run one element each. * *Partial*, with fewer values, e.g., ``sub-01``. They run all the elements of the dataset that have these values for any key, e.g., all the sessions of ``sub-01``. This requires searching the dataset for its elements. All the elements given in a command must have the same number of values, so they are either all complete or all partial. A partial element that does not match any element of the dataset is reported before running any element, while a complete element without data fails when its data is grabbed. .. _collect: Collecting Results ================== Once the ``run`` command has been executed, the results are stored in the output directory. However, depending on the storage interface, this may create one file per subject. The ``collect`` command is then used to collect all of the individual results into a single file. Assuming that we have a configuration file named ``config.yaml``, the following commands will collect the results: .. code-block:: bash junifer collect config.yaml The ``collect`` command accepts the following additional arguments: * ``--help``: Show a help message. * ``--verbose``: Set the verbosity level. Options are ``warning``, ``info``, ``debug``. .. _analysing_extracted_features: Analysing Results ================= After ``collect``-ing the results into a single file, we can analyse them as we wish. We would need to do this programmatically so feel free to choose your Python interpreter of choice or use it via a Python script. First we load the storage like so: .. code-block:: python from junifer.storage import HDF5FeatureStorage # If the features were stored with the legacy SQLiteFeatureStorage, # import and use that class instead storage = HDF5FeatureStorage("") The best way to start analysing would be to list all the extracted features like so: .. code-block:: python # This would output a dictionary with MD5 checksum of the features as keys # and metadata of the features as values storage.list_features() .. code-block:: {'eb85b61eefba61f13d712d425264697b': {'datagrabber': {'class': 'SPMAuditoryTestingDataGrabber', 'types': ['BOLD', 'T1w']}, 'dependencies': {'scikit-learn': '1.3.0', 'nilearn': '0.9.2'}, 'datareader': {'class': 'DefaultDataReader'}, 'type': 'BOLD', 'marker': {'class': 'FunctionalConnectivityParcels', 'parcellation': 'Schaefer100x7', 'agg_method': 'mean', 'agg_method_params': None, 'cor_method': 'covariance', 'cor_method_params': {'empirical': False}, 'masks': None, 'name': 'schaefer_100x7_fc_parcels'}, '_element_keys': ['subject'], 'name': 'BOLD_schaefer_100x7_fc_parcels'}} Once we have this, we can retrieve a single feature like so: .. code-block:: python feature_dict = storage.read("") to get the stored dictionary or, .. code-block:: python feature_df = storage.read_df("") to get it as a :class:`pandas.DataFrame`. If there are features with duplicate ``name`` s, then we would need to use the MD5 checksum and pass it like so: .. code-block:: python feature_df = storage.read_df(feature_md5="") We can now manipulate the dictionary or the :class:`pandas.DataFrame` as we wish or use the DataFrame directly with `julearn`_ if desired. On-the-fly transforms --------------------- ``junifer`` supports performing some computationally cheap operations (like computing brain connectivity or BrainPrint post-analysis) directly on the storage objects. To make sure everything works, install ``junifer`` like so: .. code-block:: bash pip install junifer[onthefly] or if installed via ``conda``, everything should be there and no further action is required. Computing brain connectivity via `bctpy `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can compute for example, the degree of each node considering an undirected graph via `bctpy`_ like so: .. code-block:: python from junifer.onthefly import read_transform transformed_df = read_transform( storage, # md5 hash can also be passed via `feature_md5` feature_name="", # format is `package_function` transform="bctpy_degrees_und", ) Post-analysis of BrainPrint results ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To perform post-analysis on BrainPrint results, start by importing: .. code-block:: python from junifer.onthefly import brainprint Normalising: ~~~~~~~~~~~~ Surface normalisation can be done by: .. code-block:: python surface_normalised_df = brainprint.normalize( storage, { "areas": { "feature_name": "", # if md5 hash is passed, the above should be None "feature_md5": None, }, "eigenvalues": { "feature_name": "", "feature_md5": None, }, }, kind="surface", ) and volume normalisation can be done by: .. code-block:: python volume_normalised_df = brainprint.normalize( storage, { "volumes": { "feature_name": "", "feature_md5": None, }, "eigenvalues": { "feature_name": "", "feature_md5": None, }, }, kind="volume", ) Re-weighting: ~~~~~~~~~~~~~ To perform re-weighting, run like so: .. code-block:: python reweighted_df = brainprint.reweight( storage, # md5 hash can also be passed via `feature_md5` feature_name="", )