Note
Go to the end to download the full example code
Tuning Hyperparameters#
This example uses the ‘fmri’ dataset, performs simple binary classification using a Support Vector Machine classifier and analyse the model.
References#
Waskom, M.L., Frank, M.C., Wagner, A.D. (2016). Adaptive engagement of cognitive control in context-dependent decision-making. Cerebral Cortex.
# Authors: Federico Raimondo <f.raimondo@fz-juelich.de>
#
# License: AGPL
import numpy as np
from seaborn import load_dataset
from julearn import run_cross_validation
from julearn.utils import configure_logging
from julearn.pipeline import PipelineCreator
Set the logging level to info to see extra information
configure_logging(level="INFO")
2023-07-19 12:42:04,412 - julearn - INFO - ===== Lib Versions =====
2023-07-19 12:42:04,412 - julearn - INFO - numpy: 1.25.1
2023-07-19 12:42:04,412 - julearn - INFO - scipy: 1.11.1
2023-07-19 12:42:04,412 - julearn - INFO - sklearn: 1.3.0
2023-07-19 12:42:04,412 - julearn - INFO - pandas: 2.0.3
2023-07-19 12:42:04,412 - julearn - INFO - julearn: 0.3.1.dev1
2023-07-19 12:42:04,412 - julearn - INFO - ========================
Set the random seed to always have the same example
np.random.seed(42)
Load the dataset
df_fmri = load_dataset("fmri")
print(df_fmri.head())
subject timepoint event region signal
0 s13 18 stim parietal -0.017552
1 s5 14 stim parietal -0.080883
2 s12 18 stim parietal -0.081033
3 s11 18 stim parietal -0.046134
4 s10 18 stim parietal -0.037970
Set the dataframe in the right format
df_fmri = df_fmri.pivot(
index=["subject", "timepoint", "event"], columns="region", values="signal"
)
df_fmri = df_fmri.reset_index()
print(df_fmri.head())
X = ["frontal", "parietal"]
y = "event"
region subject timepoint event frontal parietal
0 s0 0 cue 0.007766 -0.006899
1 s0 0 stim -0.021452 -0.039327
2 s0 1 cue 0.016440 0.000300
3 s0 1 stim -0.021054 -0.035735
4 s0 2 cue 0.024296 0.033220
Lets do a first attempt and use a linear SVM with the default parameters.
2023-07-19 12:42:04,430 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:42:04,430 - julearn - INFO - Step added
2023-07-19 12:42:04,430 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:42:04,430 - julearn - INFO - Setting hyperparameter kernel = linear
2023-07-19 12:42:04,430 - julearn - INFO - Step added
2023-07-19 12:42:04,430 - julearn - INFO - ==== Input Data ====
2023-07-19 12:42:04,430 - julearn - INFO - Using dataframe as input
2023-07-19 12:42:04,430 - julearn - INFO - Features: ['frontal', 'parietal']
2023-07-19 12:42:04,430 - julearn - INFO - Target: event
2023-07-19 12:42:04,430 - julearn - INFO - Expanded features: ['frontal', 'parietal']
2023-07-19 12:42:04,430 - julearn - INFO - X_types:{}
2023-07-19 12:42:04,431 - julearn - WARNING - The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/utils/logging.py:238: RuntimeWarning: The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
warn(msg, category=category)
2023-07-19 12:42:04,431 - julearn - INFO - ====================
2023-07-19 12:42:04,431 - julearn - INFO -
2023-07-19 12:42:04,432 - julearn - INFO - = Model Parameters =
2023-07-19 12:42:04,432 - julearn - INFO - ====================
2023-07-19 12:42:04,432 - julearn - INFO -
2023-07-19 12:42:04,432 - julearn - INFO - = Data Information =
2023-07-19 12:42:04,432 - julearn - INFO - Problem type: classification
2023-07-19 12:42:04,432 - julearn - INFO - Number of samples: 532
2023-07-19 12:42:04,432 - julearn - INFO - Number of features: 2
2023-07-19 12:42:04,433 - julearn - INFO - ====================
2023-07-19 12:42:04,433 - julearn - INFO -
2023-07-19 12:42:04,433 - julearn - INFO - Number of classes: 2
2023-07-19 12:42:04,433 - julearn - INFO - Target type: object
2023-07-19 12:42:04,433 - julearn - INFO - Class distributions: event
cue 266
stim 266
Name: count, dtype: int64
2023-07-19 12:42:04,434 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2023-07-19 12:42:04,434 - julearn - INFO - Binary classification problem detected.
0.5939164168576971
The score is not so good. Lets try to see if there is an optimal regularization parameter (C) for the linear SVM. We will use a grid search to find the best C.
creator = PipelineCreator(problem_type="classification")
creator.add("zscore")
creator.add("svm", kernel="linear", C=[0.01, 0.1])
search_params = {
"kind": "grid",
"cv": 2, # to speed up the example
}
scores, estimator = run_cross_validation(
X=X,
y=y,
data=df_fmri,
model=creator,
search_params=search_params,
return_estimator="final",
)
print(scores["test_score"].mean())
2023-07-19 12:42:04,506 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:42:04,506 - julearn - INFO - Step added
2023-07-19 12:42:04,506 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:42:04,506 - julearn - INFO - Setting hyperparameter kernel = linear
2023-07-19 12:42:04,506 - julearn - INFO - Tuning hyperparameter C = [0.01, 0.1]
2023-07-19 12:42:04,507 - julearn - INFO - Step added
2023-07-19 12:42:04,507 - julearn - INFO - ==== Input Data ====
2023-07-19 12:42:04,507 - julearn - INFO - Using dataframe as input
2023-07-19 12:42:04,507 - julearn - INFO - Features: ['frontal', 'parietal']
2023-07-19 12:42:04,507 - julearn - INFO - Target: event
2023-07-19 12:42:04,507 - julearn - INFO - Expanded features: ['frontal', 'parietal']
2023-07-19 12:42:04,507 - julearn - INFO - X_types:{}
2023-07-19 12:42:04,507 - julearn - WARNING - The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/utils/logging.py:238: RuntimeWarning: The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
warn(msg, category=category)
2023-07-19 12:42:04,508 - julearn - INFO - ====================
2023-07-19 12:42:04,508 - julearn - INFO -
2023-07-19 12:42:04,509 - julearn - INFO - = Model Parameters =
2023-07-19 12:42:04,509 - julearn - INFO - Tuning hyperparameters using grid
2023-07-19 12:42:04,509 - julearn - INFO - Hyperparameters:
2023-07-19 12:42:04,509 - julearn - INFO - svm__C: [0.01, 0.1]
2023-07-19 12:42:04,509 - julearn - INFO - Using inner CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2023-07-19 12:42:04,509 - julearn - INFO - Search Parameters:
2023-07-19 12:42:04,509 - julearn - INFO - cv: KFold(n_splits=2, random_state=None, shuffle=False)
2023-07-19 12:42:04,509 - julearn - INFO - ====================
2023-07-19 12:42:04,509 - julearn - INFO -
2023-07-19 12:42:04,509 - julearn - INFO - = Data Information =
2023-07-19 12:42:04,509 - julearn - INFO - Problem type: classification
2023-07-19 12:42:04,509 - julearn - INFO - Number of samples: 532
2023-07-19 12:42:04,509 - julearn - INFO - Number of features: 2
2023-07-19 12:42:04,509 - julearn - INFO - ====================
2023-07-19 12:42:04,510 - julearn - INFO -
2023-07-19 12:42:04,510 - julearn - INFO - Number of classes: 2
2023-07-19 12:42:04,510 - julearn - INFO - Target type: object
2023-07-19 12:42:04,510 - julearn - INFO - Class distributions: event
cue 266
stim 266
Name: count, dtype: int64
2023-07-19 12:42:04,511 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2023-07-19 12:42:04,511 - julearn - INFO - Binary classification problem detected.
0.588308940222183
This did not change much, lets explore other kernels too.
creator = PipelineCreator(problem_type="classification")
creator.add("zscore")
creator.add("svm", kernel=["linear", "rbf", "poly"], C=[0.01, 0.1])
scores, estimator = run_cross_validation(
X=X,
y=y,
data=df_fmri,
model=creator,
search_params=search_params,
return_estimator="final",
)
print(scores["test_score"].mean())
2023-07-19 12:42:04,870 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:42:04,870 - julearn - INFO - Step added
2023-07-19 12:42:04,870 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:42:04,870 - julearn - INFO - Tuning hyperparameter kernel = ['linear', 'rbf', 'poly']
2023-07-19 12:42:04,870 - julearn - INFO - Tuning hyperparameter C = [0.01, 0.1]
2023-07-19 12:42:04,870 - julearn - INFO - Step added
2023-07-19 12:42:04,870 - julearn - INFO - ==== Input Data ====
2023-07-19 12:42:04,871 - julearn - INFO - Using dataframe as input
2023-07-19 12:42:04,871 - julearn - INFO - Features: ['frontal', 'parietal']
2023-07-19 12:42:04,871 - julearn - INFO - Target: event
2023-07-19 12:42:04,871 - julearn - INFO - Expanded features: ['frontal', 'parietal']
2023-07-19 12:42:04,871 - julearn - INFO - X_types:{}
2023-07-19 12:42:04,871 - julearn - WARNING - The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/utils/logging.py:238: RuntimeWarning: The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
warn(msg, category=category)
2023-07-19 12:42:04,872 - julearn - INFO - ====================
2023-07-19 12:42:04,872 - julearn - INFO -
2023-07-19 12:42:04,872 - julearn - INFO - = Model Parameters =
2023-07-19 12:42:04,872 - julearn - INFO - Tuning hyperparameters using grid
2023-07-19 12:42:04,873 - julearn - INFO - Hyperparameters:
2023-07-19 12:42:04,873 - julearn - INFO - svm__kernel: ['linear', 'rbf', 'poly']
2023-07-19 12:42:04,873 - julearn - INFO - svm__C: [0.01, 0.1]
2023-07-19 12:42:04,873 - julearn - INFO - Using inner CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2023-07-19 12:42:04,873 - julearn - INFO - Search Parameters:
2023-07-19 12:42:04,873 - julearn - INFO - cv: KFold(n_splits=2, random_state=None, shuffle=False)
2023-07-19 12:42:04,873 - julearn - INFO - ====================
2023-07-19 12:42:04,873 - julearn - INFO -
2023-07-19 12:42:04,873 - julearn - INFO - = Data Information =
2023-07-19 12:42:04,873 - julearn - INFO - Problem type: classification
2023-07-19 12:42:04,873 - julearn - INFO - Number of samples: 532
2023-07-19 12:42:04,873 - julearn - INFO - Number of features: 2
2023-07-19 12:42:04,873 - julearn - INFO - ====================
2023-07-19 12:42:04,873 - julearn - INFO -
2023-07-19 12:42:04,874 - julearn - INFO - Number of classes: 2
2023-07-19 12:42:04,874 - julearn - INFO - Target type: object
2023-07-19 12:42:04,874 - julearn - INFO - Class distributions: event
cue 266
stim 266
Name: count, dtype: int64
2023-07-19 12:42:04,875 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2023-07-19 12:42:04,875 - julearn - INFO - Binary classification problem detected.
0.7087109857168048
It seems that we might have found a better model, but which one is it?
print(estimator.best_params_)
{'svm__C': 0.1, 'svm__kernel': 'rbf'}
Now that we know that a RBF kernel is better, lest test different gamma parameters.
creator = PipelineCreator(problem_type="classification")
creator.add("zscore")
creator.add(
"svm", kernel="rbf", C=[0.01, 0.1], gamma=[1e-2, 1e-3]
)
scores, estimator = run_cross_validation(
X=X,
y=y,
data=df_fmri,
model=creator,
search_params=search_params,
return_estimator="final",
)
print(scores["test_score"].mean())
print(estimator.best_params_)
2023-07-19 12:42:05,805 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:42:05,805 - julearn - INFO - Step added
2023-07-19 12:42:05,805 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:42:05,805 - julearn - INFO - Setting hyperparameter kernel = rbf
2023-07-19 12:42:05,805 - julearn - INFO - Tuning hyperparameter C = [0.01, 0.1]
2023-07-19 12:42:05,805 - julearn - INFO - Tuning hyperparameter gamma = [0.01, 0.001]
2023-07-19 12:42:05,806 - julearn - INFO - Step added
2023-07-19 12:42:05,806 - julearn - INFO - ==== Input Data ====
2023-07-19 12:42:05,806 - julearn - INFO - Using dataframe as input
2023-07-19 12:42:05,806 - julearn - INFO - Features: ['frontal', 'parietal']
2023-07-19 12:42:05,806 - julearn - INFO - Target: event
2023-07-19 12:42:05,806 - julearn - INFO - Expanded features: ['frontal', 'parietal']
2023-07-19 12:42:05,806 - julearn - INFO - X_types:{}
2023-07-19 12:42:05,806 - julearn - WARNING - The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/utils/logging.py:238: RuntimeWarning: The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
warn(msg, category=category)
2023-07-19 12:42:05,807 - julearn - INFO - ====================
2023-07-19 12:42:05,807 - julearn - INFO -
2023-07-19 12:42:05,808 - julearn - INFO - = Model Parameters =
2023-07-19 12:42:05,808 - julearn - INFO - Tuning hyperparameters using grid
2023-07-19 12:42:05,808 - julearn - INFO - Hyperparameters:
2023-07-19 12:42:05,808 - julearn - INFO - svm__C: [0.01, 0.1]
2023-07-19 12:42:05,808 - julearn - INFO - svm__gamma: [0.01, 0.001]
2023-07-19 12:42:05,808 - julearn - INFO - Using inner CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2023-07-19 12:42:05,808 - julearn - INFO - Search Parameters:
2023-07-19 12:42:05,808 - julearn - INFO - cv: KFold(n_splits=2, random_state=None, shuffle=False)
2023-07-19 12:42:05,808 - julearn - INFO - ====================
2023-07-19 12:42:05,808 - julearn - INFO -
2023-07-19 12:42:05,808 - julearn - INFO - = Data Information =
2023-07-19 12:42:05,808 - julearn - INFO - Problem type: classification
2023-07-19 12:42:05,808 - julearn - INFO - Number of samples: 532
2023-07-19 12:42:05,808 - julearn - INFO - Number of features: 2
2023-07-19 12:42:05,808 - julearn - INFO - ====================
2023-07-19 12:42:05,809 - julearn - INFO -
2023-07-19 12:42:05,809 - julearn - INFO - Number of classes: 2
2023-07-19 12:42:05,809 - julearn - INFO - Target type: object
2023-07-19 12:42:05,809 - julearn - INFO - Class distributions: event
cue 266
stim 266
Name: count, dtype: int64
2023-07-19 12:42:05,810 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2023-07-19 12:42:05,810 - julearn - INFO - Binary classification problem detected.
0.5188855581026275
{'svm__C': 0.01, 'svm__gamma': 0.001}
It seems that without tuning the gamma parameter we had a better accuracy. Let’s add the default value and see what happens.
creator = PipelineCreator(problem_type="classification")
creator.add("zscore")
creator.add(
"svm", kernel="rbf", C=[0.01, 0.1], gamma=[1e-2, 1e-3, "scale"]
)
X = ["frontal", "parietal"]
y = "event"
search_params = {"cv": 2}
scores, estimator = run_cross_validation(
X=X,
y=y,
data=df_fmri,
model=creator,
return_estimator="final",
search_params=search_params,
)
print(scores["test_score"].mean())
print(estimator.best_params_)
2023-07-19 12:42:06,517 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:42:06,518 - julearn - INFO - Step added
2023-07-19 12:42:06,518 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:42:06,518 - julearn - INFO - Setting hyperparameter kernel = rbf
2023-07-19 12:42:06,518 - julearn - INFO - Tuning hyperparameter C = [0.01, 0.1]
2023-07-19 12:42:06,518 - julearn - INFO - Tuning hyperparameter gamma = [0.01, 0.001, 'scale']
2023-07-19 12:42:06,518 - julearn - INFO - Step added
2023-07-19 12:42:06,518 - julearn - INFO - ==== Input Data ====
2023-07-19 12:42:06,518 - julearn - INFO - Using dataframe as input
2023-07-19 12:42:06,518 - julearn - INFO - Features: ['frontal', 'parietal']
2023-07-19 12:42:06,518 - julearn - INFO - Target: event
2023-07-19 12:42:06,518 - julearn - INFO - Expanded features: ['frontal', 'parietal']
2023-07-19 12:42:06,518 - julearn - INFO - X_types:{}
2023-07-19 12:42:06,518 - julearn - WARNING - The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/utils/logging.py:238: RuntimeWarning: The following columns are not defined in X_types: ['frontal', 'parietal']. They will be treated as continuous.
warn(msg, category=category)
2023-07-19 12:42:06,519 - julearn - INFO - ====================
2023-07-19 12:42:06,519 - julearn - INFO -
2023-07-19 12:42:06,520 - julearn - INFO - = Model Parameters =
2023-07-19 12:42:06,520 - julearn - INFO - Tuning hyperparameters using grid
2023-07-19 12:42:06,520 - julearn - INFO - Hyperparameters:
2023-07-19 12:42:06,520 - julearn - INFO - svm__C: [0.01, 0.1]
2023-07-19 12:42:06,520 - julearn - INFO - svm__gamma: [0.01, 0.001, 'scale']
2023-07-19 12:42:06,520 - julearn - INFO - Using inner CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2023-07-19 12:42:06,520 - julearn - INFO - Search Parameters:
2023-07-19 12:42:06,521 - julearn - INFO - cv: KFold(n_splits=2, random_state=None, shuffle=False)
2023-07-19 12:42:06,521 - julearn - INFO - ====================
2023-07-19 12:42:06,521 - julearn - INFO -
2023-07-19 12:42:06,521 - julearn - INFO - = Data Information =
2023-07-19 12:42:06,521 - julearn - INFO - Problem type: classification
2023-07-19 12:42:06,521 - julearn - INFO - Number of samples: 532
2023-07-19 12:42:06,521 - julearn - INFO - Number of features: 2
2023-07-19 12:42:06,521 - julearn - INFO - ====================
2023-07-19 12:42:06,521 - julearn - INFO -
2023-07-19 12:42:06,521 - julearn - INFO - Number of classes: 2
2023-07-19 12:42:06,521 - julearn - INFO - Target type: object
2023-07-19 12:42:06,522 - julearn - INFO - Class distributions: event
cue 266
stim 266
Name: count, dtype: int64
2023-07-19 12:42:06,522 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2023-07-19 12:42:06,522 - julearn - INFO - Binary classification problem detected.
0.7087109857168048
{'svm__C': 0.1, 'svm__gamma': 'scale'}
print(estimator.best_estimator_["svm"]._gamma)
0.5
Total running time of the script: ( 0 minutes 3.109 seconds)