14. What’s new

14.1. Junifer 0.0.4 (2024-04-05)

14.1.1. Bugfixes

• Make copying of assets in with block of YAML, relative to YAML and not to current working directory by Fede Raimondo and Synchon Mandal (#224)

• Adapt junifer queue to properly use HTCondor >=10.4.0 condor_submit_dag by Fede Raimondo and Synchon Mandal (#233)

• Use 1 instead of 0 for successful FSL commands in _check_fsl() by Synchon Mandal (#272)

• Store native warped parcellations, coordinates and masks in element-scoped tempdirs for the pipeline to work by Synchon Mandal (#274)

• Change interpolation scheme for parcel and mask native space transformation to nearest neighbour by Synchon Mandal (#276)

• Bypass FSL std2imgcoord stdin bug and use recommended piped input for coordinates transformation by Synchon Mandal (#278)

• Add -std to FSL std2imgcoord for coordinates transformation by Synchon Mandal (#280)

• Replace FSL std2imgcoord with img2imgcoord as the former is incorrect for coordinates transformation to other template spaces. (#281)

• Propagate ReHo and fALFF maps, for aggregation to convert to other template spaces when required by Synchon Mandal (#282)

• Allow junifer.pipeline.WorkDirManager to accept str via the workdir parameter by Synchon Mandal (#283)

• Avoid warping mask preprocessed with fMRIPrepConfoundRemover and used by markers with mask="inherit" in subject-native template space by Fede Raimondo and Synchon Mandal (#284)

• Pass down input path if input space is “native” for ReHoEstimator and ALFFEstimator, else use respective compute maps by Fede Raimondo and Synchon Mandal (#286)

• Fix HTCondorAdapter’s script generation to use double quotes instead of single quotes for HTCondor’s VARS by Synchon Mandal (#312)

• Fix element access for DMCC13Benchmark DataGrabber by Synchon Mandal (#314)

• Force str dtype when parsing elements from file by Synchon Mandal (#322)

14.1.2. API Changes

• Rename Power coordinates to Power2011 by Synchon Mandal (#245)

• Add feature_md5 argument to read_transform() by Synchon Mandal (#248)

• Add native_t1w parameter to DataladAOMICID1000, DataladAOMICPIOP1, DataladAOMICPIOP2, enabling fetching of T1w data in subject-native space by Synchon Mandal (#252)

• Modify preprocessor to preprocessors in run() and preprocessing to preprocessors in MarkerCollection to accept multiple preprocessors by Synchon Mandal (#263)

• Add space parameter to register_coordinates(), register_parcellation() and register_mask() and return space from load_coordinates(), load_parcellation() and load_mask() by Synchon Mandal and Fede Raimondo (#268)

• Add template_type parameter to get_template() by Synchon Mandal (#299)

• Change BasePreprocessor.preprocess() return values to preprocessed target data and “helper” data types as a dictionary by Synchon Mandal (#310)

• Add a positional argument using for Markers and Preprocessors having implementation-based variations, in particular ReHoParcels, ReHoSpheres, ALFFParcels, ALFFSpheres and BOLDWarper by Synchon Mandal (#311)

• Change all probseg_ types to VBM_ types by Fede Raimondo (#320)

• Change the subject and session patterns for DataladAOMICID1000, DataladAOMICPIOP1, DataladAOMICPIOP2 and DMCC13Benchmark so that they are consistent with their own "participants.tsv" file by Fede Raimondo (#325)

14.1.3. Improved Documentation

• Add Zenodo badge in README and improve general documentation by Synchon Mandal (#247)

• Rename extMDN to extDMN (extended default mode network) and fix listing for eMDN (extended multiple-demand network) by Synchon Mandal (#251)

• Fixed typo in code example for adding masks from “register_custom_mask” to “register_mask” by Tobias Muganga (#291)

• Rename Misc section to Miscellaneous in docs/whats_new.rst by Synchon Mandal (#300)

• Improve documentation by adding information about space transformation and writing custom Preprocessors by Synchon Mandal (#317)

14.1.4. Enhancements

• Support element(s) to be specified via text file for --element option of junifer run by Synchon Mandal (#182)

• Support element-scoped directory and temporary directories for junifer.pipeline.WorkDirManager by Synchon Mandal (#258)

• Improve element directory cleanup via junifer.pipeline.WorkDirManager.cleanup_elementdir method by Synchon Mandal (#259)

• Improve BasePreprocessor for easy subclassing and adapt fMRIPrepConfoundRemover to it by Synchon Mandal (#260)

• Add space information to existing datagrabbers, masks, parcellations and coordinates by Synchon Mandal and Fede Raimondo (#268)

• Add mode as an aggregation function option in get_aggfunc_by_name() by Synchon Mandal (#287)

• Adapt BOLDWarper to use FSL or ANTs depending on warp file extension by Synchon Mandal (#293)

• Rewrite compute_brain_mask() to allow variable template fetching via templateflow, according to target data by Synchon Mandal (#299)

• Replace requests with httpx for fetching parcellations by Synchon Mandal (#300)

• Allow BOLDWarper to warp BOLD data to other MNI spaces by Synchon Mandal (#302)

• Add support for local junifer queue via GNU Parallel by Synchon Mandal (#306)

• Improve PatternDataGrabber and PatternDataladDataGrabber’s patterns to enable space, format, mask_item and other metadata description handling via YAML by Synchon Mandal (#308)

• Improve BasePreprocessor by revamping BasePreprocessor.preprocess() and BasePreprocessor._fit_transform to handle “helper” data types better and make the pipeline explicit where data is being altered by Synchon Mandal (#310)

• Improve external dependency handling for PipelineStepMixin-derived objects having implementation-based variations by Synchon Mandal (#311)

14.1.5. Features

• Introduce complexity markers: HurstExponent, MultiscaleEntropyAUC, PermEntropy, RangeEntropy, RangeEntropyAUC and SampleEntropy by Amir Omidvarnia (#145)

• Add junifer reset to reset storage and jobs directory by Synchon Mandal (#240)

• Add support for Power2013 coordinates by Synchon Mandal (#245)

• Support venv as environment kind for queueing jobs by Synchon Mandal (#249)

• Add support for AutobiographicalMemory coordinates by Synchon Mandal (#250)

• Add support for subject-native space by Synchon Mandal and Fede Raimondo (#252)

• Introduce junifer.pipeline.WorkDirManager singleton class to manage working and temporary directories across pipeline by Synchon Mandal (#254)

• Introduce get_parcellation() to fetch parcellation tailored for the data by Synchon Mandal (#264)

• Introduce get_coordinates() to fetch coordinates tailored for the data by Synchon Mandal (#265)

• Introduce junifer.preprocess.fsl.apply_warper._ApplyWarper to wrap FSL’s applywarp by Synchon Mandal (#266)

• Introduce BOLDWarper for warping BOLD data via FSL’s applywarp by Synchon Mandal (#267)

• Introduce DMCC13Benchmark to access DMCC13benchmark dataset by Synchon Mandal (#271)

• Add Brainnetome 246 parcellation to junifer.data by Synchon Mandal (#275)

• Introduce junifer.preprocess.ants.ants_apply_transforms_warper._AntsApplyTransformsWarper to wrap ANTs’ antsApplyTransforms by Synchon Mandal (#293)

• Introduce run_ext_cmd() to take care of the boilerplate code for running external commands from FSL, ANTs and others by Synchon Mandal (#295)

• Introduce get_xfm() to fetch transformation files for moving between template spaces by Synchon Mandal (#297)

• Introduce get_template() to fetch template space image tailored to a target data by Synchon Mandal (#298)

• Add support for on-the-fly template space transformation in get_parcellation() and get_mask() to allow parcellation and mask in different template spaces to work with a DataGrabber’s data in a specified template space. (#299)

• Introduce SpaceWarper for warping T1w, BOLD, VBM_GM, VBM_WM, fALFF, GCOR and LCOR data to other spaces by Synchon Mandal (#301)

• Introduce QueueContextAdapter as an abstract base class for job queueing and HTCondorAdapter as its implementation for HTCondor by Synchon Mandal (#309)

14.1.6. Miscellaneous

• Update dependencies requirements by Fede Raimondo (#253)

• Pin ruff to 0.1.0 as the lowest version and update pre-commit config by Synchon Mandal (#261)

• Add support for accessing FSL via Docker wrapper along with flirt, applywarp and std2imgcoord commands by Synchon Mandal (#262)

• Improve documentation, packaging and code style by Synchon Mandal (#269)

• Add support for Python 3.12 and make Python 3.11 the base for code coverage and CI checks by Synchon Mandal (#270)

• Add support for accessing ANTs via Docker wrapper along with antsApplyTransforms and antsApplyTransformsToPoints by Synchon Mandal (#277)

• Make the external tool wrappers output to stderr instead of stdout by Synchon Mandal (#279)

• Add support for accessing FSL img2imgcoord via Docker wrapper command by Synchon Mandal (#281)

• Make BOLDWarper tool-agnostic by moving it from junifer.preprocess.fsl to junifer.preprocess by Synchon Mandal (#288)

• Add support for accessing ANTs’ ResampleImage via Docker wrapper by Synchon Mandal (#293)

• Update pyproject.toml and add FAIR shield in README.md by Synchon Mandal (#294)

• Update dependency listing in pyproject.toml, add .github/dependabot.yml to auto-update GitHub Actions, add ANTs and FSL installation in CI and improve general code style by Synchon Mandal (#300)

14.1.7. Deprecations and Removals

• Deprecate BOLDWarper and mark for removal in v0.0.4 by Synchon Mandal (#301)

14.2. Junifer 0.0.3 (2023-07-21)

14.2.1. Bugfixes

• Enable YAML 1.2 support and allow multiline strings in YAML which would not work earlier by Synchon Mandal (#223)

• Handle datalad.IncompleteResultsError exception for partial clone in DataladDataGrabber by Synchon Mandal (#235)

14.2.2. API Changes

• Expose types parameter for DataladAOMICID1000, DataladAOMICPIOP1, DataladAOMICPIOP2 and JuselessUCLA by Synchon Mandal (#132)

• Rename junifer.testing.datagrabbers.SPMAuditoryTestingDatagrabber to SPMAuditoryTestingDataGrabber and junifer.testing.datagrabbers.OasisVBMTestingDatagrabber to OasisVBMTestingDataGrabber by Synchon Mandal (#222)

• Add BaseFeatureStorage.read() method for storage-like objects by Synchon Mandal (#236)

14.2.3. Improved Documentation

• Improve general prose, formatting and code blocks in docs and set line length for .rst files to 80 by Synchon Mandal (#220)

• Update julearn example under examples by Synchon Mandal (#242)

14.2.4. Enhancements

• Change validation of types against patterns to allow a subset of patterns’s types to be used for DataGrabber data fetch by Synchon Mandal (#132)

• Rename instances of “Datagrabber” to “DataGrabber” especially in junifer.testing to be consistent by Synchon Mandal (#222)

• Use ruamel.yaml instead of pyyaml as YAML I/O library by Synchon Mandal (#223)

• Adopt DataGrabber consistently throughout codebase to match with the documentation by Synchon Mandal (#226)

• Adopt DataReader consistently throughout codebase to match with the documentation by Synchon Mandal (#227)

• Enable stdout and stderr capture for AFNI commands by Synchon Mandal (#234)

• Adapt HDF5FeatureStorage.read_df() due to addition of HDF5FeatureStorage.read() by Synchon Mandal (#236)

14.2.5. Features

• Add AICHA v1 and AICHA v2 parcellations to junifer.data by Synchon Mandal (#173)

• Add Shen 2013, Shen 2015 and Shen 2019 parcellations to junifer.data by Synchon Mandal (#184)

• Add Yan 2023 parcellation to junifer.data by Synchon Mandal (#225)

• Introduce onthefly sub-module and read_transform() for quick transform operations on stored data by Synchon Mandal (#237)

14.2.6. Miscellaneous

• Consistent docstrings for pytest fixtures used in the test suite by Synchon Mandal (#228)

• Adopt ruff as the only linter for the codebase by Synchon Mandal (#229)

• Improve codespell support by fixing typos in documentation by Synchon Mandal (#230)

• Adopt pre-commit for adding and managing git pre-commit hooks by Synchon Mandal (#232)

• Improve docstrings and code style and parametrize remaining tests for junifer.data.parcellations by Synchon Mandal (#238)

• Bump numpy version constraint to >=1.24,<1.26 by Synchon Mandal (#241)

14.3. Junifer 0.0.2 (2023-03-31)

14.3.1. Bugfixes

• Fix a bug in which relative storage URIs will be computed relative to the CWD and not to the location of the YAML file by Fede Raimondo. (#127)

• Fix junifer run to respect preprocess step specified in the pipeline by Synchon Mandal (#159)

• Fix a bug in which only REST1 and REST2 tasks could be accessed in DataladHCP1200 and HCP1200 datagrabbers by Fede Raimondo (#183)

• Fix a bug in which fitting a marker (e.g. SphereAggregation) on a specific type (e.g.: BOLD) will fail if another non-supported type (e.g.: BOLD_confounds) is present in the data object by Fede Raimondo (#185)

• Fix ALFFParcels, ALFFSpheres, ReHoSpheres and ReHoParcels pass the extra_input parameter by Fede Raimondo (#187)

• Fix several markers that did not properly handle the extra_input parameter by Fede Raimondo (#189)

• Fix a bug in which relative paths in the YAML with directive would be computed relative to the current working directory of the process instead of the location of the YAML file by Fede Raimondo. (#191)

• Fix an issue with datalad cache and locks in which the user-specific configuration might create a conflict in high throughput systems by Fede Raimondo (#192)

• Fix a bug in which ParcelAggregation could yield duplicated column names if two or more parcels were used and label names were not unique by Fede Raimondo (#194)

• Fix a bug in which count() will not be correctly applied across an axis by Fede Raimondo (#195)

• Fix an issue with datalad cache and locks in which the overridden settings in Junifer were not propagated to subprocesses, resulting in using the default settings by Fede Raimondo (#199)

• Fix a bug in which get_mask() fails for FunctionalConnectivityBase class, because of missing extra_input parameter by Leonard Sasse (#200)

• Fix the output of RSSETSMarker to be 2D by Synchon Mandal (#215)

14.3.2. API Changes

• Add confounds_format parameter to PatternDataGrabber constructor for improved handling of confounds specified via BOLD_confounds data type by Synchon Mandal (#158)

• Rename store_table() to store_vector() for storage-like objects and adapt marker-like objects to use "vector" in place of "table" for storage. Also, improve the logic of storing vectors by Synchon Mandal (#181)

• Add ica_fix parameter to DataladHCP1200 and HCP1200 datagrabbers to allow for selecting data processed with ICA+FIX. Default value is False which changes behaviour since 0.0.1 release. By Fede Raimondo (#183)

• Add pre_run parameter to _queue_condor by Fede Raimondo (#188)

• Expose allow_overlap parameter in SphereAggregation and related markers by Fede Raimondo (#190)

• Rename AmplitudeLowFrequencyFluctuationParcels and AmplitudeLowFrequencyFluctuationSpheres to ALFFParcels and ALFFSpheres by Synchon Mandal (#216)

14.3.3. Improved Documentation

• Add more documentation on registering parcellations, coordinates, and masks by Leonard Sasse (#166)

• Migrate and document changelog creation and maintenance via towncrier by Synchon Mandal (#203)

• Add copy button to documentation code blocks by Synchon Mandal (#205)

• Add sections starting and help in the documentation by Fede Raimondo (#210)

• Shorten Sphinx references across code and docs, and add black shield in README by Synchon Mandal (#218)

14.3.4. Enhancements

• Organize functional connectivity markers in junifer.markers.functional_connectivity by Synchon Mandal (#107)

• Change HCP datagrabber tests to decrease CI running time by Fede Raimondo (#155)

• Allow PartlyCloudyTestingDataGrabber to be accessible via import junifer.testing.registry by Synchon Mandal (#160)

• Update docstrings and fix typo in log message by Synchon Mandal (#165)

• Add fMRIPrep brain masks to the datagrabber patterns for all datagrabbers in the aomic sub-package by Leonard Sasse (#177)

• Improved logging output for preprocessing, collecting and pipeline building from YAML by Fede Raimondo (#185)

• Allow for empty spheres in JuniferNiftiSpheresMasker, that will result in NaNs. Also, modify the behaviour of the collect parameter in HTCondor queue function to run a collect job even if some of the previous jobs fail. This is useful to collect the results of a pipeline even if some of the jobs fail by Fede Raimondo (#190)

• Allow for empty parcels in ParcelAggregation, that will result in NaNs by Fede Raimondo (#194)

• Improve metadata and data I/O for HDF5FeatureStorage by Synchon Mandal (#196)

• Force datalad to be non-interactive on queued jobs by Fede Raimondo (#201)

• Add missing abstractmethod decorators for get_valid_inputs methods of BaseMarker and BasePreprocessor by Synchon Mandal (#214)

14.3.5. Features

• Add EdgeCentricFCParcels and EdgeCentricFCSpheres by Leonard Sasse (#64)

• Expose a merge_parcellations() function to merge a list of parcellations by Leonard Sasse (#146)

• Add support for HDF5 feature storage via HDF5FeatureStorage by Synchon Mandal (#147)

• Add TemporalSNRParcels and TemporalSNRSpheres by Leonard Sasse (#163)

• Add support for Power coordinates by Synchon Mandal (#167)

• Add support for Dosenbach coordinates by Synchon Mandal (#168)

• Add support for nilearn computed masks (compute_epi_mask, compute_brain_mask, compute_background_mask, fetch_icbm152_brain_gm_mask) by Fede Raimondo (#175)

• Add aggregation function count() that returns the number of elements in a given axis. This allows to count the number of voxels per sphere/parcel when used as method in markers by Fede Raimondo (#190)

14.4. Junifer 0.0.1 (2022-12-20)

14.4.1. Bugfixes

• Fix a bug in which a PatternDataGrabber would now work with relative datadir paths (reported by Leonard Sasse, fixed by Fede Raimondo) (#96, #98)

• Fix a bug in which DataladAOMICPIOP2 datagrabber did not use user input to constrain elements based on tasks by Leonard Sasse (#105)

• Fix a bug in which a datalad dataset could remove a user-cloned dataset by Fede Raimondo (#53)

• Fix a bug in which CLI command would not work using elements with more than one field by Fede Raimondo

• Fix a bug in which the generated DAG for HTCondor will not work by Fede Raimondo (#143)

14.4.2. API Changes

• Change the single_output default parameter in storage classes from False to True by Fede Raimondo (#134)

14.4.3. Improved Documentation

• Add an example how to use junifer and julearn in one pipeline to extract features and do machine learning by Leonard Sasse, Nicolas Nieto, and Sami Hamdan (#40)

• Add documentation for the datagrabbers by Leonard Sasse, Nicolas Nieto, and Sami Hamdan

• Change documentation template to furo. Fix references and standardize documentation by Fede Raimondo (#114)

14.4.4. Enhancements

• Add comments to DataladDataGrabber datagrabber and change to use datalad-clone instead of datalad-install by Benjamin Poldrack (#55)

• Upgrade storage interface for storage-like objects by Synchon Mandal (#84)

• Add missing type annotations by Synchon Mandal (#74)

• Refactor markers on attribute and get_valid_inputs to verify that the marker can be computed on the input data types by Fede Raimondo

• Add test for DataladHCP1200 datagrabber by Synchon Mandal (#93)

• Refactor DataladAOMICID1000 slightly by Leonard Sasse (#94)

• Rename “atlas” to “parcellation” by Fede Raimondo (#116)

• Refactor the BaseDataGrabber class to allow for easier subclassing by Fede Raimondo (#123)

• Allow custom aggregation method for SphereAggregation by Synchon Mandal (#102)

• Add support for “masks” by Fede Raimondo (#79)

• Allow ParcelAggregation to apply multiple parcellations at once by Fede Raimondo (#131)

• Refactor PipelineStepMixin to improve its implementation and validation for pipeline steps by Synchon Mandal (#152)

14.4.5. Features

• Implement SPMAuditoryTestingDataGrabber datagrabber by Fede Raimondo (#52)

• Implement matrix storage in SQliteFeatureStorage by Fede Raimondo (#42)

• Implement FunctionalConnectivityParcels marker for functional connectivity using a parcellation by Amir Omidvarnia and Kaustubh R. Patil (#41)

• Implement register_coordinates(), list_coordinates() and load_coordinates() by Fede Raimondo (#11)

• Add DataladAOMICID1000 datagrabber for AOMIC ID1000 dataset including tests and creation of mock dataset for testing by Vera Komeyer and Xuan Li (#60)

• Add support to access other input in the data object in the compute method by Fede Raimondo

• Implement RSSETSMarker marker by Leonard Sasse, Nicolas Nieto and Sami Hamdan (#51)

• Implement SphereAggregation marker by Fede Raimondo (#83)

• Implement DataladAOMICPIOP1 and DataladAOMICPIOP2 datagrabbers for AOMIC PIOP1 and PIOP2 datasets respectively by Leonard Sasse (#94)

• Implement JuselessDataladCamCANVBM datagrabber by Leonard Sasse (#99)

• Implement JuselessDataladIXIVBM CAT output datagrabber for juseless by Leonard Sasse (#48)

• Add junifer wtf to report environment details by Synchon Mandal (#33)

• Add junifer selftest to report environment details by Synchon Mandal (#9)

• Implement JuselessDataladAOMICID1000VBM datagrabber for accessing AOMIC ID1000 VBM from juseless by Felix Hoffstaedter and Synchon Mandal (#57)

• Add fMRIPrepConfoundRemover by Fede Raimondo and Leonard Sasse (#111)

• Implement CrossParcellationFC marker by Leonard Sasse and Kaustubh R. Patil (#85)

• Add JuselessUCLA datagrabber for the UCLA dataset available on juseless by Leonard Sasse (#118)

• Introduce a singleton decorator for marker computations by Synchon Mandal (#151)

• Implement ReHoParcels and ReHoSpheres markers by Synchon Mandal (#36)

• Implement ALFFParcels and ALFFSpheres markers by Fede Raimondo (#35)

14.4.6. Miscellaneous

• Create the repository based on the mockup by Fede Raimondo