Note
Click here to download the full example code
Tuning Hyperparameters¶
This example uses the ‘fmri’ dataset, performs simple binary classification using a Support Vector Machine classifier and analyse the model.
References¶
Waskom, M.L., Frank, M.C., Wagner, A.D. (2016). Adaptive engagement of cognitive control in context-dependent decision-making. Cerebral Cortex.
# Authors: Federico Raimondo <f.raimondo@fz-juelich.de>
#
# License: AGPL
import numpy as np
from seaborn import load_dataset
from julearn import run_cross_validation
from julearn.utils import configure_logging
Set the logging level to info to see extra information
configure_logging(level='INFO')
Out:
2021-01-28 20:08:11,437 - julearn - INFO - ===== Lib Versions =====
2021-01-28 20:08:11,437 - julearn - INFO - numpy: 1.19.5
2021-01-28 20:08:11,437 - julearn - INFO - scipy: 1.6.0
2021-01-28 20:08:11,437 - julearn - INFO - sklearn: 0.24.1
2021-01-28 20:08:11,438 - julearn - INFO - pandas: 1.2.1
2021-01-28 20:08:11,438 - julearn - INFO - julearn: 0.2.5.dev19+g9c15c5f
2021-01-28 20:08:11,438 - julearn - INFO - ========================
Set the random seed to always have the same example
np.random.seed(42)
Load the dataset
df_fmri = load_dataset('fmri')
print(df_fmri.head())
Out:
subject timepoint event region signal
0 s13 18 stim parietal -0.017552
1 s5 14 stim parietal -0.080883
2 s12 18 stim parietal -0.081033
3 s11 18 stim parietal -0.046134
4 s10 18 stim parietal -0.037970
Set the dataframe in the right format
df_fmri = df_fmri.pivot(
index=['subject', 'timepoint', 'event'],
columns='region',
values='signal')
df_fmri = df_fmri.reset_index()
print(df_fmri.head())
Out:
region subject timepoint event frontal parietal
0 s0 0 cue 0.007766 -0.006899
1 s0 0 stim -0.021452 -0.039327
2 s0 1 cue 0.016440 0.000300
3 s0 1 stim -0.021054 -0.035735
4 s0 2 cue 0.024296 0.033220
Lets do a first attempt and use a linear SVM with the default parameters.
model_params = {'svm__kernel': 'linear'}
X = ['frontal', 'parietal']
y = 'event'
scores = run_cross_validation(
X=X, y=y, data=df_fmri, model='svm', preprocess_X='zscore',
model_params=model_params)
print(scores['test_score'].mean())
Out:
2021-01-28 20:08:11,461 - julearn - INFO - Using default CV
2021-01-28 20:08:11,461 - julearn - INFO - ==== Input Data ====
2021-01-28 20:08:11,461 - julearn - INFO - Using dataframe as input
2021-01-28 20:08:11,461 - julearn - INFO - Features: ['frontal', 'parietal']
2021-01-28 20:08:11,461 - julearn - INFO - Target: event
2021-01-28 20:08:11,461 - julearn - INFO - Expanded X: ['frontal', 'parietal']
2021-01-28 20:08:11,461 - julearn - INFO - Expanded Confounds: []
2021-01-28 20:08:11,462 - julearn - INFO - ====================
2021-01-28 20:08:11,462 - julearn - INFO -
2021-01-28 20:08:11,463 - julearn - INFO - ====== Model ======
2021-01-28 20:08:11,463 - julearn - INFO - Obtaining model by name: svm
2021-01-28 20:08:11,463 - julearn - INFO - ===================
2021-01-28 20:08:11,463 - julearn - INFO -
2021-01-28 20:08:11,463 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds
2021-01-28 20:08:11,463 - julearn - INFO - = Model Parameters =
2021-01-28 20:08:11,463 - julearn - INFO - Setting hyperparameter svm__kernel = linear
2021-01-28 20:08:11,464 - julearn - INFO - ====================
2021-01-28 20:08:11,464 - julearn - INFO -
0.5765508728619291
The score is not so good. Lets try to see if there is an optimal regularization parameter (C) for the linear SVM.
model_params = {
'svm__kernel': 'linear',
'svm__C': [0.01, 0.1],
'cv': 2} # CV=2 too speed up the example
X = ['frontal', 'parietal']
y = 'event'
scores, estimator = run_cross_validation(
X=X, y=y, data=df_fmri, model='svm', preprocess_X='zscore',
model_params=model_params, return_estimator='final')
print(scores['test_score'].mean())
Out:
2021-01-28 20:08:12,100 - julearn - INFO - Using default CV
2021-01-28 20:08:12,100 - julearn - INFO - ==== Input Data ====
2021-01-28 20:08:12,100 - julearn - INFO - Using dataframe as input
2021-01-28 20:08:12,100 - julearn - INFO - Features: ['frontal', 'parietal']
2021-01-28 20:08:12,100 - julearn - INFO - Target: event
2021-01-28 20:08:12,100 - julearn - INFO - Expanded X: ['frontal', 'parietal']
2021-01-28 20:08:12,101 - julearn - INFO - Expanded Confounds: []
2021-01-28 20:08:12,101 - julearn - INFO - ====================
2021-01-28 20:08:12,101 - julearn - INFO -
2021-01-28 20:08:12,101 - julearn - INFO - ====== Model ======
2021-01-28 20:08:12,102 - julearn - INFO - Obtaining model by name: svm
2021-01-28 20:08:12,102 - julearn - INFO - ===================
2021-01-28 20:08:12,102 - julearn - INFO -
2021-01-28 20:08:12,102 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds
2021-01-28 20:08:12,102 - julearn - INFO - = Model Parameters =
2021-01-28 20:08:12,102 - julearn - INFO - Setting hyperparameter svm__kernel = linear
2021-01-28 20:08:12,103 - julearn - INFO - Tunning hyperparameters using {search}
2021-01-28 20:08:12,103 - julearn - INFO - Hyperparameters:
2021-01-28 20:08:12,103 - julearn - INFO - svm__C: [0.01, 0.1]
2021-01-28 20:08:12,103 - julearn - INFO - Using scikit-learn CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2021-01-28 20:08:12,103 - julearn - INFO - Search Parameters:
2021-01-28 20:08:12,103 - julearn - INFO - cv: KFold(n_splits=2, random_state=None, shuffle=False)
2021-01-28 20:08:12,103 - julearn - INFO - scoring: None
2021-01-28 20:08:12,103 - julearn - INFO - ====================
2021-01-28 20:08:12,103 - julearn - INFO -
0.575591606418621
This did not change much, lets explore other kernels too.
model_params = {
'svm__kernel': ['linear', 'rbf', 'poly'],
'svm__C': [0.01, 0.1],
'cv': 2} # CV=2 too speed up the example
X = ['frontal', 'parietal']
y = 'event'
scores, estimator = run_cross_validation(
X=X, y=y, data=df_fmri, model='svm', preprocess_X='zscore',
model_params=model_params, return_estimator='final')
print(scores['test_score'].mean())
Out:
2021-01-28 20:08:15,183 - julearn - INFO - Using default CV
2021-01-28 20:08:15,183 - julearn - INFO - ==== Input Data ====
2021-01-28 20:08:15,183 - julearn - INFO - Using dataframe as input
2021-01-28 20:08:15,183 - julearn - INFO - Features: ['frontal', 'parietal']
2021-01-28 20:08:15,183 - julearn - INFO - Target: event
2021-01-28 20:08:15,183 - julearn - INFO - Expanded X: ['frontal', 'parietal']
2021-01-28 20:08:15,183 - julearn - INFO - Expanded Confounds: []
2021-01-28 20:08:15,184 - julearn - INFO - ====================
2021-01-28 20:08:15,184 - julearn - INFO -
2021-01-28 20:08:15,184 - julearn - INFO - ====== Model ======
2021-01-28 20:08:15,184 - julearn - INFO - Obtaining model by name: svm
2021-01-28 20:08:15,184 - julearn - INFO - ===================
2021-01-28 20:08:15,184 - julearn - INFO -
2021-01-28 20:08:15,184 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds
2021-01-28 20:08:15,185 - julearn - INFO - = Model Parameters =
2021-01-28 20:08:15,185 - julearn - INFO - Tunning hyperparameters using {search}
2021-01-28 20:08:15,185 - julearn - INFO - Hyperparameters:
2021-01-28 20:08:15,185 - julearn - INFO - svm__kernel: ['linear', 'rbf', 'poly']
2021-01-28 20:08:15,185 - julearn - INFO - svm__C: [0.01, 0.1]
2021-01-28 20:08:15,185 - julearn - INFO - Using scikit-learn CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2021-01-28 20:08:15,185 - julearn - INFO - Search Parameters:
2021-01-28 20:08:15,185 - julearn - INFO - cv: KFold(n_splits=2, random_state=None, shuffle=False)
2021-01-28 20:08:15,185 - julearn - INFO - scoring: None
2021-01-28 20:08:15,186 - julearn - INFO - ====================
2021-01-28 20:08:15,186 - julearn - INFO -
0.7116487391994357
It seems that we might have found a better model, but which one is it?
print(estimator.best_params_)
Out:
{'svm__C': 0.1, 'svm__kernel': 'rbf'}
Now that we know that a RBF kernel is better, lest test different gamma parameters.
model_params = {
'svm__kernel': 'rbf',
'svm__C': [0.01, 0.1],
'svm__gamma': [1e-2, 1e-3],
'cv': 2} # CV=2 too speed up the example
X = ['frontal', 'parietal']
y = 'event'
scores, estimator = run_cross_validation(
X=X, y=y, data=df_fmri, model='svm', preprocess_X='zscore',
model_params=model_params, return_estimator='final')
print(scores['test_score'].mean())
print(estimator.best_params_)
Out:
2021-01-28 20:08:23,474 - julearn - INFO - Using default CV
2021-01-28 20:08:23,475 - julearn - INFO - ==== Input Data ====
2021-01-28 20:08:23,475 - julearn - INFO - Using dataframe as input
2021-01-28 20:08:23,475 - julearn - INFO - Features: ['frontal', 'parietal']
2021-01-28 20:08:23,475 - julearn - INFO - Target: event
2021-01-28 20:08:23,475 - julearn - INFO - Expanded X: ['frontal', 'parietal']
2021-01-28 20:08:23,475 - julearn - INFO - Expanded Confounds: []
2021-01-28 20:08:23,476 - julearn - INFO - ====================
2021-01-28 20:08:23,476 - julearn - INFO -
2021-01-28 20:08:23,476 - julearn - INFO - ====== Model ======
2021-01-28 20:08:23,476 - julearn - INFO - Obtaining model by name: svm
2021-01-28 20:08:23,476 - julearn - INFO - ===================
2021-01-28 20:08:23,476 - julearn - INFO -
2021-01-28 20:08:23,476 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds
2021-01-28 20:08:23,476 - julearn - INFO - = Model Parameters =
2021-01-28 20:08:23,477 - julearn - INFO - Setting hyperparameter svm__kernel = rbf
2021-01-28 20:08:23,477 - julearn - INFO - Tunning hyperparameters using {search}
2021-01-28 20:08:23,477 - julearn - INFO - Hyperparameters:
2021-01-28 20:08:23,477 - julearn - INFO - svm__C: [0.01, 0.1]
2021-01-28 20:08:23,477 - julearn - INFO - svm__gamma: [0.01, 0.001]
2021-01-28 20:08:23,478 - julearn - INFO - Using scikit-learn CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2021-01-28 20:08:23,478 - julearn - INFO - Search Parameters:
2021-01-28 20:08:23,478 - julearn - INFO - cv: KFold(n_splits=2, random_state=None, shuffle=False)
2021-01-28 20:08:23,478 - julearn - INFO - scoring: None
2021-01-28 20:08:23,478 - julearn - INFO - ====================
2021-01-28 20:08:23,478 - julearn - INFO -
0.47479104214424267
{'svm__C': 0.01, 'svm__gamma': 0.001}
It seems that without tuning the gamma parameter we had a better accuracy. Let’s add the default value and see what happens.
model_params = {
'svm__kernel': 'rbf',
'svm__C': [0.01, 0.1],
'svm__gamma': [1e-2, 1e-3, 'scale'],
'cv': 2} # CV=2 too speed up the example
X = ['frontal', 'parietal']
y = 'event'
scores, estimator = run_cross_validation(
X=X, y=y, data=df_fmri, model='svm', preprocess_X='zscore',
model_params=model_params, return_estimator='final')
print(scores['test_score'].mean())
print(estimator.best_params_)
Out:
2021-01-28 20:08:29,895 - julearn - INFO - Using default CV
2021-01-28 20:08:29,895 - julearn - INFO - ==== Input Data ====
2021-01-28 20:08:29,895 - julearn - INFO - Using dataframe as input
2021-01-28 20:08:29,895 - julearn - INFO - Features: ['frontal', 'parietal']
2021-01-28 20:08:29,895 - julearn - INFO - Target: event
2021-01-28 20:08:29,895 - julearn - INFO - Expanded X: ['frontal', 'parietal']
2021-01-28 20:08:29,896 - julearn - INFO - Expanded Confounds: []
2021-01-28 20:08:29,896 - julearn - INFO - ====================
2021-01-28 20:08:29,897 - julearn - INFO -
2021-01-28 20:08:29,897 - julearn - INFO - ====== Model ======
2021-01-28 20:08:29,897 - julearn - INFO - Obtaining model by name: svm
2021-01-28 20:08:29,897 - julearn - INFO - ===================
2021-01-28 20:08:29,897 - julearn - INFO -
2021-01-28 20:08:29,897 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds
2021-01-28 20:08:29,897 - julearn - INFO - = Model Parameters =
2021-01-28 20:08:29,897 - julearn - INFO - Setting hyperparameter svm__kernel = rbf
2021-01-28 20:08:29,898 - julearn - INFO - Tunning hyperparameters using {search}
2021-01-28 20:08:29,898 - julearn - INFO - Hyperparameters:
2021-01-28 20:08:29,898 - julearn - INFO - svm__C: [0.01, 0.1]
2021-01-28 20:08:29,898 - julearn - INFO - svm__gamma: [0.01, 0.001, 'scale']
2021-01-28 20:08:29,898 - julearn - INFO - Using scikit-learn CV scheme KFold(n_splits=2, random_state=None, shuffle=False)
2021-01-28 20:08:29,898 - julearn - INFO - Search Parameters:
2021-01-28 20:08:29,898 - julearn - INFO - cv: KFold(n_splits=2, random_state=None, shuffle=False)
2021-01-28 20:08:29,898 - julearn - INFO - scoring: None
2021-01-28 20:08:29,899 - julearn - INFO - ====================
2021-01-28 20:08:29,899 - julearn - INFO -
0.7074977958032092
{'svm__C': 0.1, 'svm__gamma': 'scale'}
So what was the best gamma in the end?
print(estimator.best_estimator_['svm']._gamma)
Out:
0.5
Total running time of the script: ( 0 minutes 27.576 seconds)