4.3. Data Grabber¶

4.3.1. Description¶

The DataGrabber is an object that can provide an interface to datasets you want to work with in junifer. Every concrete implementation of a DataGrabber is aware of a particular dataset’s structure and thus allows you to fetch specific elements of interest from the dataset. It adds the path key to each data type in the Data object.

DataGrabbers are intended to be used as context managers. When used within a context, a DataGrabber takes care of any pre and post steps for interacting with the dataset, for example, downloading and cleaning up. As the interface is consistent, you always use the same procedure to interact with the DataGrabber.

For example, a concrete implementation of DataladDataGrabber can provide junifer with data from a Datalad dataset. Of course, DataGrabbers are not only meant to work with Datalad datasets but any dataset.

If you are interested in using already provided DataGrabbers, please go to Built-in Pipeline Components. And, if you want to implement your own DataGrabber, you need to provide concrete implementations of abstract base classes already provided.

4.3.2. Elements¶

A DataGrabber gives the data of a dataset in small units called elements. An element is the smallest unit of data that can be processed: it has one file for each data type (e.g., one T1w image and one BOLD image). Each element is identified by the values of its keys, e.g., the element ("sub-01", "rest") has the value sub-01 for the key subject and rest for the key task.

To see how this works, we will use this small dataset with two subjects, a T1w image for each subject and a BOLD image for each subject and task (sub-02 has no movie task):

sub-01/anat/sub-01_T1w.nii.gz
sub-01/func/sub-01_task-rest_bold.nii.gz
sub-01/func/sub-01_task-movie_bold.nii.gz
sub-02/anat/sub-02_T1w.nii.gz
sub-02/func/sub-02_task-rest_bold.nii.gz

A PatternDataGrabber describes the files of each data type with a pattern, where the parts that change between the files (the replacements) are written between braces:

datagrabber:
  kind: PatternDataGrabber
  datadir: /data/example
  types:
    - T1w
    - BOLD
  patterns:
    T1w:
      pattern: "{subject}/anat/{subject}_T1w.nii.gz"
      space: native
    BOLD:
      pattern: "{subject}/func/{subject}_task-{task}_bold.nii.gz"
      space: native
  replacements:
    - subject
    - task

Which data types are grabbed¶

The DataGrabber finds the elements by searching the dataset for the files that match the patterns. The keys of the elements are the replacements in the patterns of the data types to grab (types), and an element is only available if it has a file for each of these data types:

`types`	Keys	Elements
`T1w`	`subject`	`sub-01`, `sub-02`
`BOLD`	`subject`, `task`	`(sub-01, movie)`, `(sub-01, rest)`, `(sub-02, rest)`
`T1w`, `BOLD`	`subject`, `task`	`(sub-01, movie)`, `(sub-01, rest)`, `(sub-02, rest)`

With only T1w, the elements do not have the task key, as the T1w image does not depend on the task: each T1w image is processed once. With BOLD, the elements need the task, so the T1w image of sub-01 is part of two elements, (sub-01, movie) and (sub-01, rest).

Which values are grabbed¶

By default, all the values found in the dataset are grabbed. They can be restricted with replacements, giving the values to grab for each replacement (or null to grab all of them). For example, to only grab the rest task:

replacements:
  subject: null
  task:
    - rest

The built-in DataGrabbers have parameters for this instead, e.g., tasks in DataladAOMICPIOP1. The values only restrict the elements of the data types that use the replacement:

`types`	Values to grab	Elements
`BOLD`	`task`: `rest`	`(sub-01, rest)`, `(sub-02, rest)`
`BOLD`	`subject`: `sub-02`	`(sub-02, rest)`
`T1w`	`task`: `rest`	`sub-01`, `sub-02` (the `T1w` image does not depend on the task)

The elements with other values are not listed, and asking for them (e.g., (sub-01, movie) when only rest is grabbed) raises an error. See Restricting the values of the replacements for more details, and Advanced: Using Unix-like path expansion directives for patterns with *, e.g., to match files whose names change with the task.

Which elements are processed¶

The list-elements command lists the elements of the dataset. By default, the run and queue commands process all of them, but the --element option can select some of them, with the values of the keys separated by ,. With BOLD:

`--element`	Elements processed
`sub-01,rest`	`(sub-01, rest)`: a complete element (a value for each key, in order), the dataset is not searched
`sub-01`	`(sub-01, movie)`, `(sub-01, rest)`: a partial element selects all the elements with its values
`rest`	`(sub-01, rest)`, `(sub-02, rest)`
`sub-03`	None: an error is raised, as no element matches it

See Complete and partial elements for more details.

4.3.3. Combining datasets¶

The MultipleDataGrabber combines several DataGrabbers, e.g., to use data from different datasets. For example, we can take the BOLD and T1w images from a dataset with the fMRIPrep derivatives, the VBM_GM images from a dataset with the CAT12 derivatives, and the confounds of the BOLD images from a third dataset:

datagrabber:
  kind: MultipleDataGrabber
  datagrabbers:
    # BOLD and T1w from the fMRIPrep derivatives
    - kind: DataladAOMICPIOP1
      types:
        - BOLD
        - T1w
      tasks:
        - restingstate
    # VBM from the CAT12 derivatives
    - kind: PatternDataladDataGrabber
      uri: https://example.org/aomic-piop1-cat12
      types:
        - VBM_GM
      patterns:
        VBM_GM:
          pattern: "{subject}/mri/mwp1{subject}_T1w.nii"
          space: MNI152NLin2009cAsym
      replacements:
        - subject
    # Only the confounds of the BOLD images
    - kind: PatternDataladDataGrabber
      uri: https://example.org/aomic-piop1-confounds
      types:
        - BOLD
      patterns:
        BOLD:
          confounds:
            pattern: "{subject}/func/{subject}_task-{task}_acq-*_desc-confounds.tsv"
            format: fmriprep
      replacements:
        - subject
        - task
      partial_pattern_ok: true

The DataGrabbers are combined with these rules:

Each data type is grabbed by one DataGrabber, which is chosen with the types of each DataGrabber. Here, the first one grabs BOLD and T1w, and the second one VBM_GM. If the first one also grabbed VBM_GM (e.g., without setting its types), an error would be raised, asking to remove it from the types of one of them.
A DataGrabber that only grabs the nested data types of a data type (without the main file of the data type) replaces these nested data types of the other DataGrabber. Here, the third one only grabs the confounds of BOLD, so they replace the ones of the first DataGrabber, regardless of the order of the DataGrabbers.
The elements are the ones available in all the DataGrabbers. The DataGrabbers can have different keys: the element keys are the ones of the first DataGrabber, followed by the other keys of the other DataGrabbers, and each DataGrabber grabs an element with its own keys. The keys with the same name must have the same values in all the datasets (e.g., the same subject names).

For the element (sub-0001, restingstate):

DataGrabber	Keys	Grabs
First (fMRIPrep)	`subject`, `task`	`(sub-0001, restingstate)`: the `BOLD` image (with its mask) and the `T1w` image
Second (CAT12)	`subject`	`sub-0001`: the `VBM_GM` image
Third (confounds)	`subject`, `task`	`(sub-0001, restingstate)`: the confounds of the `BOLD` image

4.3.4. Base Classes¶

In this section, we showcase different abstract and concrete base classes you might want to use to implement your own DataGrabber.

Name	Description
`BaseDataGrabber`	The abstract base class providing you an interface to implement your own DataGrabber. You should try to avoid using this directly and instead use `PatternDataGrabber` or `DataladDataGrabber`. To build your own custom low-level DataGrabber, you need to override the `get_element_keys`, `get_elements` and `get_item` methods, and most of the time you should also override other existing methods like `__enter__` and `__exit__`.
`PatternDataGrabber`	It implements functionality to help you define the pattern of the dataset you want to get. For example, you know that T1w images are found in a directory following the pattern: `{subject}/anat/{subject}_T1w.nii.gz` inside of the dataset. Now you can provide this to the `PatternDataGrabber` and it will be able to get the file.
`DataladDataGrabber`	It implements functionality to deal with Datalad datasets. Specifically, the `__enter__` and `__exit__` methods take care of cloning and removing the Datalad dataset.
`PatternDataladDataGrabber`	It is a combination of `PatternDataGrabber` and `DataladDataGrabber`. This is probably the class you are looking for when using Datalad datasets.
`MultipleDataGrabber`	It combines several DataGrabbers, e.g., to use data from different datasets (see Combining datasets).