Custom Scoring Function for Regression

This example uses the ‘diabetes’ data from sklearn datasets and performs a regression analysis using a Ridge Regression model. As scorers, it uses scikit-learn, julearn and a custom metric defined by the user.

# Authors: Shammi More <s.more@fz-juelich.de>
#          Federico Raimondo <f.raimondo@fz-juelich.de>
#
# License: AGPL

import pandas as pd
import scipy
from sklearn.datasets import load_diabetes

from sklearn.metrics import make_scorer
from julearn.scoring import register_scorer

from julearn import run_cross_validation
from julearn.utils import configure_logging

Set the logging level to info to see extra information

configure_logging(level='INFO')

Out:

2022-07-21 09:55:04,058 - julearn - INFO - ===== Lib Versions =====
2022-07-21 09:55:04,058 - julearn - INFO - numpy: 1.23.1
2022-07-21 09:55:04,058 - julearn - INFO - scipy: 1.8.1
2022-07-21 09:55:04,058 - julearn - INFO - sklearn: 1.0.2
2022-07-21 09:55:04,058 - julearn - INFO - pandas: 1.4.3
2022-07-21 09:55:04,058 - julearn - INFO - julearn: 0.2.5
2022-07-21 09:55:04,058 - julearn - INFO - ========================

load the diabetes data from sklearn as a pandas dataframe

features, target = load_diabetes(return_X_y=True, as_frame=True)

Dataset contains ten variables age, sex, body mass index, average blood pressure, and six blood serum measurements (s1-s6) diabetes patients and a quantitative measure of disease progression one year after baseline which is the target we are interested in predicting.

print('Features: \n', features.head())  # type: ignore
print('Target: \n', target.describe())  # type: ignore

Out:

Features:
         age       sex       bmi  ...        s4        s5        s6
0  0.038076  0.050680  0.061696  ... -0.002592  0.019908 -0.017646
1 -0.001882 -0.044642 -0.051474  ... -0.039493 -0.068330 -0.092204
2  0.085299  0.050680  0.044451  ... -0.002592  0.002864 -0.025930
3 -0.089063 -0.044642 -0.011595  ...  0.034309  0.022692 -0.009362
4  0.005383 -0.044642 -0.036385  ... -0.002592 -0.031991 -0.046641

[5 rows x 10 columns]
Target:
 count    442.000000
mean     152.133484
std       77.093005
min       25.000000
25%       87.000000
50%      140.500000
75%      211.500000
max      346.000000
Name: target, dtype: float64

Let’s combine features and target together in one dataframe and define X and y

data_diabetes = pd.concat([features, target], axis=1)  # type: ignore

X = ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
y = 'target'

Train a ridge regression model on train dataset and use mean absolute error for scoring

scores, model = run_cross_validation(
    X=X, y=y, data=data_diabetes, preprocess_X='zscore',
    problem_type='regression', model='ridge', return_estimator='final',
    scoring='neg_mean_absolute_error')

Out:

2022-07-21 09:55:04,076 - julearn - INFO - Using default CV
2022-07-21 09:55:04,076 - julearn - INFO - ==== Input Data ====
2022-07-21 09:55:04,077 - julearn - INFO - Using dataframe as input
2022-07-21 09:55:04,077 - julearn - INFO - Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2022-07-21 09:55:04,077 - julearn - INFO - Target: target
2022-07-21 09:55:04,077 - julearn - INFO - Expanded X: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2022-07-21 09:55:04,077 - julearn - INFO - Expanded Confounds: []
2022-07-21 09:55:04,078 - julearn - INFO - ====================
2022-07-21 09:55:04,078 - julearn - INFO -
2022-07-21 09:55:04,078 - julearn - INFO - ====== Model ======
2022-07-21 09:55:04,078 - julearn - INFO - Obtaining model by name: ridge
2022-07-21 09:55:04,078 - julearn - INFO - ===================
2022-07-21 09:55:04,078 - julearn - INFO -
2022-07-21 09:55:04,078 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds

The scores dataframe has all the values for each CV split.

print(scores.head())

Out:

   fit_time  score_time  test_score  repeat  fold
0  0.008516    0.005568  -44.386924       0     0
1  0.008201    0.005484  -45.094063       0     1
2  0.008235    0.005721  -43.188016       0     2
3  0.008671    0.005671  -41.591935       0     3
4  0.008196    0.005460  -49.226121       0     4

Mean value of mean absolute error across CV

print(scores['test_score'].mean() * -1)  # type: ignore

Out:

44.56947555620945

Now do the same thing, but use mean absolute error and Pearson product-moment correlation coefficient (squared) as scoring functions

scores, model = run_cross_validation(
    X=X, y=y, data=data_diabetes, preprocess_X='zscore',
    problem_type='regression', model='ridge', return_estimator='final',
    scoring=['neg_mean_absolute_error', 'r2_corr'])

Out:

2022-07-21 09:55:04,457 - julearn - INFO - Using default CV
2022-07-21 09:55:04,457 - julearn - INFO - ==== Input Data ====
2022-07-21 09:55:04,457 - julearn - INFO - Using dataframe as input
2022-07-21 09:55:04,457 - julearn - INFO - Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2022-07-21 09:55:04,457 - julearn - INFO - Target: target
2022-07-21 09:55:04,458 - julearn - INFO - Expanded X: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2022-07-21 09:55:04,458 - julearn - INFO - Expanded Confounds: []
2022-07-21 09:55:04,458 - julearn - INFO - ====================
2022-07-21 09:55:04,459 - julearn - INFO -
2022-07-21 09:55:04,459 - julearn - INFO - ====== Model ======
2022-07-21 09:55:04,459 - julearn - INFO - Obtaining model by name: ridge
2022-07-21 09:55:04,459 - julearn - INFO - ===================
2022-07-21 09:55:04,459 - julearn - INFO -
2022-07-21 09:55:04,459 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds

Now the scores dataframe has all the values for each CV split, but two scores unders the column names ‘test_neg_mean_absolute_error’ and ‘test_r2_corr’.

print(scores[['test_neg_mean_absolute_error', 'test_r2_corr']].mean())

Out:

test_neg_mean_absolute_error   -44.257386
test_r2_corr                     0.502022
dtype: float64

If we want to define a custom scoring metric, we need to define a function that takes the predicted and the actual values as input and returns a value. In this case, we want to compute Pearson correlation coefficient (r).

def pearson_scorer(y_true, y_pred):
    return scipy.stats.pearsonr(  # type: ignore
        y_true.squeeze(), y_pred.squeeze())[0]

Before using it, we need to convert it to a sklearn scorer and register it with julearn.

register_scorer(scorer_name='pearsonr', scorer=make_scorer(pearson_scorer))

Out:

2022-07-21 09:55:04,969 - julearn - INFO - registering scorer named pearsonr

Now we can use it as another scoring metric.

scores, model = run_cross_validation(
    X=X, y=y, data=data_diabetes, preprocess_X='zscore',
    problem_type='regression', model='ridge', return_estimator='final',
    scoring=['neg_mean_absolute_error', 'r2_corr', 'pearsonr'])

Out:

2022-07-21 09:55:04,970 - julearn - INFO - Using default CV
2022-07-21 09:55:04,970 - julearn - INFO - ==== Input Data ====
2022-07-21 09:55:04,970 - julearn - INFO - Using dataframe as input
2022-07-21 09:55:04,970 - julearn - INFO - Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2022-07-21 09:55:04,970 - julearn - INFO - Target: target
2022-07-21 09:55:04,970 - julearn - INFO - Expanded X: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2022-07-21 09:55:04,971 - julearn - INFO - Expanded Confounds: []
2022-07-21 09:55:04,971 - julearn - INFO - ====================
2022-07-21 09:55:04,971 - julearn - INFO -
2022-07-21 09:55:04,971 - julearn - INFO - ====== Model ======
2022-07-21 09:55:04,971 - julearn - INFO - Obtaining model by name: ridge
2022-07-21 09:55:04,971 - julearn - INFO - ===================
2022-07-21 09:55:04,971 - julearn - INFO -
2022-07-21 09:55:04,972 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds