Note
Go to the end to download the full example code
Tuning Hyperparameters
This example uses the ‘fmri’ dataset, performs simple binary classification using a Support Vector Machine classifier and analyse the model.
References
Waskom, M.L., Frank, M.C., Wagner, A.D. (2016). Adaptive engagement of cognitive control in context-dependent decision-making. Cerebral Cortex.
# Authors: Federico Raimondo <f.raimondo@fz-juelich.de>
#
# License: AGPL
import numpy as np
from seaborn import load_dataset
from julearn import run_cross_validation
from julearn.utils import configure_logging
Set the logging level to info to see extra information
configure_logging(level='INFO')
2023-04-06 09:51:29,002 - julearn - INFO - ===== Lib Versions =====
2023-04-06 09:51:29,003 - julearn - INFO - numpy: 1.23.5
2023-04-06 09:51:29,003 - julearn - INFO - scipy: 1.10.1
2023-04-06 09:51:29,003 - julearn - INFO - sklearn: 1.0.2
2023-04-06 09:51:29,003 - julearn - INFO - pandas: 1.4.4
2023-04-06 09:51:29,003 - julearn - INFO - julearn: 0.3.1.dev2
2023-04-06 09:51:29,003 - julearn - INFO - ========================
Set the random seed to always have the same example
np.random.seed(42)
Load the dataset
df_fmri = load_dataset('fmri')
print(df_fmri.head())
subject timepoint event region signal
0 s13 18 stim parietal -0.017552
1 s5 14 stim parietal -0.080883
2 s12 18 stim parietal -0.081033
3 s11 18 stim parietal -0.046134
4 s10 18 stim parietal -0.037970
Set the dataframe in the right format
df_fmri = df_fmri.pivot(
index=['subject', 'timepoint', 'event'],
columns='region',
values='signal')
df_fmri = df_fmri.reset_index()
print(df_fmri.head())
region subject timepoint event frontal parietal
0 s0 0 cue 0.007766 -0.006899
1 s0 0 stim -0.021452 -0.039327
2 s0 1 cue 0.016440 0.000300
3 s0 1 stim -0.021054 -0.035735
4 s0 2 cue 0.024296 0.033220
Lets do a first attempt and use a linear SVM with the default parameters.
model_params = {'svm__kernel': 'linear'}
X = ['frontal', 'parietal']
y = 'event'
scores = run_cross_validation(
X=X, y=y, data=df_fmri, model='svm', preprocess_X='zscore',
model_params=model_params)
print(scores['test_score'].mean())
2023-04-06 09:51:29,029 - julearn - INFO - Using default CV
2023-04-06 09:51:29,029 - julearn - INFO - ==== Input Data ====
2023-04-06 09:51:29,029 - julearn - INFO - Using dataframe as input
2023-04-06 09:51:29,029 - julearn - INFO - Features: ['frontal', 'parietal']
2023-04-06 09:51:29,029 - julearn - INFO - Target: event
2023-04-06 09:51:29,029 - julearn - INFO - Expanded X: ['frontal', 'parietal']
2023-04-06 09:51:29,030 - julearn - INFO - Expanded Confounds: []
2023-04-06 09:51:29,031 - julearn - INFO - ====================
2023-04-06 09:51:29,031 - julearn - INFO -
2023-04-06 09:51:29,031 - julearn - INFO - ====== Model ======
2023-04-06 09:51:29,031 - julearn - INFO - Obtaining model by name: svm
2023-04-06 09:51:29,031 - julearn - INFO - ===================
2023-04-06 09:51:29,031 - julearn - INFO -
2023-04-06 09:51:29,031 - julearn - INFO - = Model Parameters =
2023-04-06 09:51:29,031 - julearn - INFO - Setting hyperparameter svm__kernel = linear
2023-04-06 09:51:29,032 - julearn - INFO - ====================
2023-04-06 09:51:29,032 - julearn - INFO -
2023-04-06 09:51:29,033 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds
0.5765508728619291
The score is not so good. Lets try to see if there is an optimal regularization parameter (C) for the linear SVM.
model_params = {
'svm__kernel': 'linear',
'svm__C': [0.01, 0.1],
'cv': 2} # CV=2 too speed up the example
X = ['frontal', 'parietal']
y = 'event'
scores, estimator = run_cross_validation(
X=X, y=y, data=df_fmri, model='svm', preprocess_X='zscore',
model_params=model_params, return_estimator='final')
print(scores['test_score'].mean())
2023-04-06 09:51:29,784 - julearn - INFO - Using default CV
2023-04-06 09:51:29,784 - julearn - INFO - ==== Input Data ====
2023-04-06 09:51:29,784 - julearn - INFO - Using dataframe as input
2023-04-06 09:51:29,784 - julearn - INFO - Features: ['frontal', 'parietal']
2023-04-06 09:51:29,784 - julearn - INFO - Target: event
2023-04-06 09:51:29,784 - julearn - INFO - Expanded X: ['frontal', 'parietal']
2023-04-06 09:51:29,784 - julearn - INFO - Expanded Confounds: []
2023-04-06 09:51:29,786 - julearn - INFO - ====================
2023-04-06 09:51:29,786 - julearn - INFO -
2023-04-06 09:51:29,786 - julearn - INFO - ====== Model ======
2023-04-06 09:51:29,786 - julearn - INFO - Obtaining model by name: svm
2023-04-06 09:51:29,786 - julearn - INFO - ===================
2023-04-06 09:51:29,786 - julearn - INFO -
2023-04-06 09:51:29,786 - julearn - INFO - = Model Parameters =
2023-04-06 09:51:29,786 - julearn - INFO - Setting hyperparameter svm__kernel = linear
2023-04-06 09:51:29,787 - julearn - WARNING - `cv` should not be directly provided in the`model_params` anymore. This functionality willbe removed in the next version of julearn.Please use `cv` inside of `search_params` instead
2023-04-06 09:51:29,787 - julearn - INFO - Tunning hyperparameters using grid
2023-04-06 09:51:29,787 - julearn - INFO - Hyperparameters:
2023-04-06 09:51:29,788 - julearn - INFO - svm__C: [0.01, 0.1]
2023-04-06 09:51:29,788 - julearn - INFO - Using scikit-learn CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2023-04-06 09:51:29,788 - julearn - INFO - Search Parameters:
2023-04-06 09:51:29,788 - julearn - INFO - cv: KFold(n_splits=2, random_state=None, shuffle=False)
2023-04-06 09:51:29,788 - julearn - INFO - scoring: None
2023-04-06 09:51:29,788 - julearn - INFO - ====================
2023-04-06 09:51:29,788 - julearn - INFO -
2023-04-06 09:51:29,788 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds
0.575591606418621
This did not change much, lets explore other kernels too.
model_params = {
'svm__kernel': ['linear', 'rbf', 'poly'],
'svm__C': [0.01, 0.1],
'cv': 2} # CV=2 too speed up the example
X = ['frontal', 'parietal']
y = 'event'
scores, estimator = run_cross_validation(
X=X, y=y, data=df_fmri, model='svm', preprocess_X='zscore',
model_params=model_params, return_estimator='final')
print(scores['test_score'].mean())
2023-04-06 09:51:33,319 - julearn - INFO - Using default CV
2023-04-06 09:51:33,319 - julearn - INFO - ==== Input Data ====
2023-04-06 09:51:33,320 - julearn - INFO - Using dataframe as input
2023-04-06 09:51:33,320 - julearn - INFO - Features: ['frontal', 'parietal']
2023-04-06 09:51:33,320 - julearn - INFO - Target: event
2023-04-06 09:51:33,320 - julearn - INFO - Expanded X: ['frontal', 'parietal']
2023-04-06 09:51:33,320 - julearn - INFO - Expanded Confounds: []
2023-04-06 09:51:33,321 - julearn - INFO - ====================
2023-04-06 09:51:33,321 - julearn - INFO -
2023-04-06 09:51:33,321 - julearn - INFO - ====== Model ======
2023-04-06 09:51:33,321 - julearn - INFO - Obtaining model by name: svm
2023-04-06 09:51:33,321 - julearn - INFO - ===================
2023-04-06 09:51:33,321 - julearn - INFO -
2023-04-06 09:51:33,322 - julearn - INFO - = Model Parameters =
2023-04-06 09:51:33,322 - julearn - WARNING - `cv` should not be directly provided in the`model_params` anymore. This functionality willbe removed in the next version of julearn.Please use `cv` inside of `search_params` instead
2023-04-06 09:51:33,322 - julearn - INFO - Tunning hyperparameters using grid
2023-04-06 09:51:33,322 - julearn - INFO - Hyperparameters:
2023-04-06 09:51:33,322 - julearn - INFO - svm__kernel: ['linear', 'rbf', 'poly']
2023-04-06 09:51:33,322 - julearn - INFO - svm__C: [0.01, 0.1]
2023-04-06 09:51:33,322 - julearn - INFO - Using scikit-learn CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2023-04-06 09:51:33,322 - julearn - INFO - Search Parameters:
2023-04-06 09:51:33,323 - julearn - INFO - cv: KFold(n_splits=2, random_state=None, shuffle=False)
2023-04-06 09:51:33,323 - julearn - INFO - scoring: None
2023-04-06 09:51:33,323 - julearn - INFO - ====================
2023-04-06 09:51:33,323 - julearn - INFO -
2023-04-06 09:51:33,323 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds
0.7116487391994357
It seems that we might have found a better model, but which one is it?
print(estimator.best_params_)
{'svm__C': 0.1, 'svm__kernel': 'rbf'}
Now that we know that a RBF kernel is better, lest test different gamma parameters.
model_params = {
'svm__kernel': 'rbf',
'svm__C': [0.01, 0.1],
'svm__gamma': [1e-2, 1e-3],
'cv': 2} # CV=2 too speed up the example
X = ['frontal', 'parietal']
y = 'event'
scores, estimator = run_cross_validation(
X=X, y=y, data=df_fmri, model='svm', preprocess_X='zscore',
model_params=model_params, return_estimator='final')
print(scores['test_score'].mean())
print(estimator.best_params_)
2023-04-06 09:51:42,395 - julearn - INFO - Using default CV
2023-04-06 09:51:42,395 - julearn - INFO - ==== Input Data ====
2023-04-06 09:51:42,395 - julearn - INFO - Using dataframe as input
2023-04-06 09:51:42,395 - julearn - INFO - Features: ['frontal', 'parietal']
2023-04-06 09:51:42,395 - julearn - INFO - Target: event
2023-04-06 09:51:42,395 - julearn - INFO - Expanded X: ['frontal', 'parietal']
2023-04-06 09:51:42,395 - julearn - INFO - Expanded Confounds: []
2023-04-06 09:51:42,396 - julearn - INFO - ====================
2023-04-06 09:51:42,396 - julearn - INFO -
2023-04-06 09:51:42,396 - julearn - INFO - ====== Model ======
2023-04-06 09:51:42,396 - julearn - INFO - Obtaining model by name: svm
2023-04-06 09:51:42,396 - julearn - INFO - ===================
2023-04-06 09:51:42,397 - julearn - INFO -
2023-04-06 09:51:42,397 - julearn - INFO - = Model Parameters =
2023-04-06 09:51:42,397 - julearn - INFO - Setting hyperparameter svm__kernel = rbf
2023-04-06 09:51:42,398 - julearn - WARNING - `cv` should not be directly provided in the`model_params` anymore. This functionality willbe removed in the next version of julearn.Please use `cv` inside of `search_params` instead
2023-04-06 09:51:42,398 - julearn - INFO - Tunning hyperparameters using grid
2023-04-06 09:51:42,398 - julearn - INFO - Hyperparameters:
2023-04-06 09:51:42,398 - julearn - INFO - svm__C: [0.01, 0.1]
2023-04-06 09:51:42,398 - julearn - INFO - svm__gamma: [0.01, 0.001]
2023-04-06 09:51:42,398 - julearn - INFO - Using scikit-learn CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2023-04-06 09:51:42,398 - julearn - INFO - Search Parameters:
2023-04-06 09:51:42,399 - julearn - INFO - cv: KFold(n_splits=2, random_state=None, shuffle=False)
2023-04-06 09:51:42,399 - julearn - INFO - scoring: None
2023-04-06 09:51:42,399 - julearn - INFO - ====================
2023-04-06 09:51:42,399 - julearn - INFO -
2023-04-06 09:51:42,399 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds
0.47479104214424267
{'svm__C': 0.01, 'svm__gamma': 0.001}
It seems that without tuning the gamma parameter we had a better accuracy. Let’s add the default value and see what happens.
model_params = {
'svm__kernel': 'rbf',
'svm__C': [0.01, 0.1],
'svm__gamma': [1e-2, 1e-3, 'scale'],
'cv': 2} # CV=2 too speed up the example
X = ['frontal', 'parietal']
y = 'event'
scores, estimator = run_cross_validation(
X=X, y=y, data=df_fmri, model='svm', preprocess_X='zscore',
model_params=model_params, return_estimator='final')
print(scores['test_score'].mean())
print(estimator.best_params_)
2023-04-06 09:51:49,024 - julearn - INFO - Using default CV
2023-04-06 09:51:49,025 - julearn - INFO - ==== Input Data ====
2023-04-06 09:51:49,025 - julearn - INFO - Using dataframe as input
2023-04-06 09:51:49,025 - julearn - INFO - Features: ['frontal', 'parietal']
2023-04-06 09:51:49,025 - julearn - INFO - Target: event
2023-04-06 09:51:49,025 - julearn - INFO - Expanded X: ['frontal', 'parietal']
2023-04-06 09:51:49,025 - julearn - INFO - Expanded Confounds: []
2023-04-06 09:51:49,026 - julearn - INFO - ====================
2023-04-06 09:51:49,026 - julearn - INFO -
2023-04-06 09:51:49,027 - julearn - INFO - ====== Model ======
2023-04-06 09:51:49,027 - julearn - INFO - Obtaining model by name: svm
2023-04-06 09:51:49,027 - julearn - INFO - ===================
2023-04-06 09:51:49,027 - julearn - INFO -
2023-04-06 09:51:49,027 - julearn - INFO - = Model Parameters =
2023-04-06 09:51:49,027 - julearn - INFO - Setting hyperparameter svm__kernel = rbf
2023-04-06 09:51:49,028 - julearn - WARNING - `cv` should not be directly provided in the`model_params` anymore. This functionality willbe removed in the next version of julearn.Please use `cv` inside of `search_params` instead
2023-04-06 09:51:49,028 - julearn - INFO - Tunning hyperparameters using grid
2023-04-06 09:51:49,028 - julearn - INFO - Hyperparameters:
2023-04-06 09:51:49,028 - julearn - INFO - svm__C: [0.01, 0.1]
2023-04-06 09:51:49,028 - julearn - INFO - svm__gamma: [0.01, 0.001, 'scale']
2023-04-06 09:51:49,028 - julearn - INFO - Using scikit-learn CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2023-04-06 09:51:49,028 - julearn - INFO - Search Parameters:
2023-04-06 09:51:49,029 - julearn - INFO - cv: KFold(n_splits=2, random_state=None, shuffle=False)
2023-04-06 09:51:49,029 - julearn - INFO - scoring: None
2023-04-06 09:51:49,029 - julearn - INFO - ====================
2023-04-06 09:51:49,029 - julearn - INFO -
2023-04-06 09:51:49,029 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds
0.7074977958032092
{'svm__C': 0.1, 'svm__gamma': 'scale'}
So what was the best gamma in the end?
print(estimator.best_estimator_['svm']._gamma)
0.5
Total running time of the script: ( 0 minutes 29.374 seconds)