Note

Go to the end to download the full example code

Tuning Multiple Hyperparameters Grids#

This example uses the fmri dataset, performs simple binary classification using a Support Vector Machine classifier while tuning multiple hyperparameters grids at the same time.

References#

Waskom, M.L., Frank, M.C., Wagner, A.D. (2016). Adaptive engagement of cognitive control in context-dependent decision-making. Cerebral Cortex.

# Authors: Federico Raimondo <f.raimondo@fz-juelich.de>
# License: AGPL

import numpy as np
from seaborn import load_dataset

from julearn import run_cross_validation
from julearn.utils import configure_logging
from julearn.pipeline import PipelineCreator

Set the logging level to info to see extra information.

configure_logging(level="INFO")

/home/runner/work/julearn/julearn/julearn/utils/logging.py:66: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  vstring = str(getattr(module, "__version__", None))
2024-10-17 14:15:44,360 - julearn - INFO - ===== Lib Versions =====
2024-10-17 14:15:44,360 - julearn - INFO - numpy: 1.26.4
2024-10-17 14:15:44,360 - julearn - INFO - scipy: 1.14.1
2024-10-17 14:15:44,360 - julearn - INFO - sklearn: 1.5.2
2024-10-17 14:15:44,360 - julearn - INFO - pandas: 2.2.3
2024-10-17 14:15:44,360 - julearn - INFO - julearn: 0.3.4
2024-10-17 14:15:44,360 - julearn - INFO - ========================

Set the random seed to always have the same example.

np.random.seed(42)

Load the dataset.

df_fmri = load_dataset("fmri")
df_fmri.head()
subject timepoint event region signal
0 s13 18 stim parietal -0.017552
1 s5 14 stim parietal -0.080883
2 s12 18 stim parietal -0.081033
3 s11 18 stim parietal -0.046134
4 s10 18 stim parietal -0.037970


Set the dataframe in the right format.

df_fmri = df_fmri.pivot(
    index=["subject", "timepoint", "event"], columns="region", values="signal"
)

df_fmri = df_fmri.reset_index()
df_fmri.head()
region subject timepoint event frontal parietal
0 s0 0 cue 0.007766 -0.006899
1 s0 0 stim -0.021452 -0.039327
2 s0 1 cue 0.016440 0.000300
3 s0 1 stim -0.021054 -0.035735
4 s0 2 cue 0.024296 0.033220


Lets do a first attempt and use a linear SVM with the default parameters.

X = ["frontal", "parietal"]
y = "event"

creator = PipelineCreator(problem_type="classification")
creator.add("zscore")
creator.add("svm", kernel="linear")

scores = run_cross_validation(X=X, y=y, data=df_fmri, model=creator)

print(scores["test_score"].mean())

2024-10-17 14:15:44,370 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-10-17 14:15:44,370 - julearn - INFO - Step added
2024-10-17 14:15:44,370 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-10-17 14:15:44,370 - julearn - INFO - Setting hyperparameter kernel = linear
2024-10-17 14:15:44,370 - julearn - INFO - Step added
2024-10-17 14:15:44,370 - julearn - INFO - ==== Input Data ====
2024-10-17 14:15:44,370 - julearn - INFO - Using dataframe as input
2024-10-17 14:15:44,370 - julearn - INFO -      Features: ['frontal', 'parietal']
2024-10-17 14:15:44,370 - julearn - INFO -      Target: event
2024-10-17 14:15:44,370 - julearn - INFO -      Expanded features: ['frontal', 'parietal']
2024-10-17 14:15:44,370 - julearn - INFO -      X_types:{}
2024-10-17 14:15:44,370 - julearn - WARNING - The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/prepare.py:509: RuntimeWarning: The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
  warn_with_log(
2024-10-17 14:15:44,371 - julearn - INFO - ====================
2024-10-17 14:15:44,371 - julearn - INFO -
2024-10-17 14:15:44,372 - julearn - INFO - = Model Parameters =
2024-10-17 14:15:44,372 - julearn - INFO - ====================
2024-10-17 14:15:44,372 - julearn - INFO -
2024-10-17 14:15:44,372 - julearn - INFO - = Data Information =
2024-10-17 14:15:44,372 - julearn - INFO -      Problem type: classification
2024-10-17 14:15:44,372 - julearn - INFO -      Number of samples: 532
2024-10-17 14:15:44,372 - julearn - INFO -      Number of features: 2
2024-10-17 14:15:44,372 - julearn - INFO - ====================
2024-10-17 14:15:44,372 - julearn - INFO -
2024-10-17 14:15:44,372 - julearn - INFO -      Number of classes: 2
2024-10-17 14:15:44,372 - julearn - INFO -      Target type: object
2024-10-17 14:15:44,373 - julearn - INFO -      Class distributions: event
cue     266
stim    266
Name: count, dtype: int64
2024-10-17 14:15:44,373 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2024-10-17 14:15:44,373 - julearn - INFO - Binary classification problem detected.
0.5939164168576971

Now let’s tune a bit this SVM. We will use a grid search to tune the regularization parameter C and the kernel. We will also tune the gamma. But since the gamma is only used for the rbf kernel, we will use a different grid for the "rbf" kernel.

To specify two different sets of parameters for the same step, we can explicitly specify the name of the step. This is done by passing the name parameter to the add method.

creator = PipelineCreator(problem_type="classification")
creator.add("zscore")
creator.add("svm", kernel="linear", C=[0.01, 0.1], name="svm")
creator.add(
    "svm",
    kernel="rbf",
    C=[0.01, 0.1],
    gamma=["scale", "auto", 1e-2, 1e-3],
    name="svm",
)

search_params = {
    "kind": "grid",
    "cv": 2,  # to speed up the example
}

scores, estimator = run_cross_validation(
    X=X,
    y=y,
    data=df_fmri,
    model=creator,
    search_params=search_params,
    return_estimator="final",
)

print(scores["test_score"].mean())

2024-10-17 14:15:44,431 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-10-17 14:15:44,431 - julearn - INFO - Step added
2024-10-17 14:15:44,431 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-10-17 14:15:44,431 - julearn - INFO - Setting hyperparameter kernel = linear
2024-10-17 14:15:44,431 - julearn - INFO - Tuning hyperparameter C = [0.01, 0.1]
2024-10-17 14:15:44,431 - julearn - INFO - Step added
2024-10-17 14:15:44,431 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-10-17 14:15:44,431 - julearn - INFO - Setting hyperparameter kernel = rbf
2024-10-17 14:15:44,431 - julearn - INFO - Tuning hyperparameter C = [0.01, 0.1]
2024-10-17 14:15:44,431 - julearn - INFO - Tuning hyperparameter gamma = ['scale', 'auto', 0.01, 0.001]
2024-10-17 14:15:44,432 - julearn - INFO - Step added
2024-10-17 14:15:44,432 - julearn - INFO - ==== Input Data ====
2024-10-17 14:15:44,432 - julearn - INFO - Using dataframe as input
2024-10-17 14:15:44,432 - julearn - INFO -      Features: ['frontal', 'parietal']
2024-10-17 14:15:44,432 - julearn - INFO -      Target: event
2024-10-17 14:15:44,432 - julearn - INFO -      Expanded features: ['frontal', 'parietal']
2024-10-17 14:15:44,432 - julearn - INFO -      X_types:{}
2024-10-17 14:15:44,432 - julearn - WARNING - The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/prepare.py:509: RuntimeWarning: The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
  warn_with_log(
2024-10-17 14:15:44,433 - julearn - INFO - ====================
2024-10-17 14:15:44,433 - julearn - INFO -
2024-10-17 14:15:44,433 - julearn - INFO - = Model Parameters =
2024-10-17 14:15:44,433 - julearn - INFO - Tuning hyperparameters using grid
2024-10-17 14:15:44,433 - julearn - INFO - Hyperparameters:
2024-10-17 14:15:44,433 - julearn - INFO -      svm__C: [0.01, 0.1]
2024-10-17 14:15:44,434 - julearn - INFO - Using inner CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2024-10-17 14:15:44,434 - julearn - INFO - Search Parameters:
2024-10-17 14:15:44,434 - julearn - INFO -      cv: KFold(n_splits=2, random_state=None, shuffle=False)
2024-10-17 14:15:44,434 - julearn - INFO - ====================
2024-10-17 14:15:44,434 - julearn - INFO -
2024-10-17 14:15:44,434 - julearn - INFO - = Model Parameters =
2024-10-17 14:15:44,434 - julearn - INFO - Tuning hyperparameters using grid
2024-10-17 14:15:44,434 - julearn - INFO - Hyperparameters:
2024-10-17 14:15:44,434 - julearn - INFO -      svm__C: [0.01, 0.1]
2024-10-17 14:15:44,435 - julearn - INFO -      svm__gamma: ['scale', 'auto', 0.01, 0.001]
2024-10-17 14:15:44,435 - julearn - INFO - Using inner CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2024-10-17 14:15:44,435 - julearn - INFO - Search Parameters:
2024-10-17 14:15:44,435 - julearn - INFO -      cv: KFold(n_splits=2, random_state=None, shuffle=False)
2024-10-17 14:15:44,435 - julearn - INFO - ====================
2024-10-17 14:15:44,435 - julearn - INFO -
2024-10-17 14:15:44,435 - julearn - INFO - = Model Parameters =
2024-10-17 14:15:44,435 - julearn - INFO - Tuning hyperparameters using grid
2024-10-17 14:15:44,435 - julearn - INFO - Hyperparameters list:
2024-10-17 14:15:44,435 - julearn - INFO -      Set 0
2024-10-17 14:15:44,435 - julearn - INFO -              svm__C: [0.01, 0.1]
2024-10-17 14:15:44,435 - julearn - INFO -              set_column_types: [SetColumnTypes(X_types={})]
2024-10-17 14:15:44,436 - julearn - INFO -              svm: [SVC(kernel='linear')]
2024-10-17 14:15:44,436 - julearn - INFO -      Set 1
2024-10-17 14:15:44,436 - julearn - INFO -              svm__C: [0.01, 0.1]
2024-10-17 14:15:44,436 - julearn - INFO -              svm__gamma: ['scale', 'auto', 0.01, 0.001]
2024-10-17 14:15:44,436 - julearn - INFO -              set_column_types: [SetColumnTypes(X_types={})]
2024-10-17 14:15:44,436 - julearn - INFO -              svm: [SVC()]
2024-10-17 14:15:44,436 - julearn - INFO - Using inner CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2024-10-17 14:15:44,436 - julearn - INFO - Search Parameters:
2024-10-17 14:15:44,436 - julearn - INFO -      cv: KFold(n_splits=2, random_state=None, shuffle=False)
2024-10-17 14:15:44,436 - julearn - INFO - ====================
2024-10-17 14:15:44,436 - julearn - INFO -
2024-10-17 14:15:44,436 - julearn - INFO - = Data Information =
2024-10-17 14:15:44,436 - julearn - INFO -      Problem type: classification
2024-10-17 14:15:44,436 - julearn - INFO -      Number of samples: 532
2024-10-17 14:15:44,437 - julearn - INFO -      Number of features: 2
2024-10-17 14:15:44,437 - julearn - INFO - ====================
2024-10-17 14:15:44,437 - julearn - INFO -
2024-10-17 14:15:44,437 - julearn - INFO -      Number of classes: 2
2024-10-17 14:15:44,437 - julearn - INFO -      Target type: object
2024-10-17 14:15:44,437 - julearn - INFO -      Class distributions: event
cue     266
stim    266
Name: count, dtype: int64
2024-10-17 14:15:44,438 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False) (incl. final model)
2024-10-17 14:15:44,438 - julearn - INFO - Binary classification problem detected.
0.7087109857168048

It seems that we might have found a better model, but which one is it?

print(estimator.best_params_)
print(estimator.best_estimator_["svm"]._gamma)

{'set_column_types': SetColumnTypes(X_types={}), 'svm': SVC(), 'svm__C': 0.1, 'svm__gamma': 'scale'}
0.5

Total running time of the script: (0 minutes 1.331 seconds)

Gallery generated by Sphinx-Gallery