Custom Scoring Function for Regression

This example uses the ‘diabetes’ data from sklearn datasets and performs a regression analysis using a Ridge Regression model. As scorers, it uses scikit-learn, julearn and a custom metric defined by the user.

# Authors: Shammi More <s.more@fz-juelich.de>
#          Federico Raimondo <f.raimondo@fz-juelich.de>
#
# License: AGPL

import pandas as pd
import scipy
from sklearn.datasets import load_diabetes

from sklearn.metrics import make_scorer
from julearn.scoring import register_scorer

from julearn import run_cross_validation
from julearn.utils import configure_logging

Set the logging level to info to see extra information

configure_logging(level='INFO')

2023-04-06 09:51:26,375 - julearn - INFO - ===== Lib Versions =====
2023-04-06 09:51:26,375 - julearn - INFO - numpy: 1.23.5
2023-04-06 09:51:26,375 - julearn - INFO - scipy: 1.10.1
2023-04-06 09:51:26,375 - julearn - INFO - sklearn: 1.0.2
2023-04-06 09:51:26,375 - julearn - INFO - pandas: 1.4.4
2023-04-06 09:51:26,375 - julearn - INFO - julearn: 0.3.1.dev2
2023-04-06 09:51:26,375 - julearn - INFO - ========================

load the diabetes data from sklearn as a pandas dataframe

features, target = load_diabetes(return_X_y=True, as_frame=True)

Dataset contains ten variables age, sex, body mass index, average blood pressure, and six blood serum measurements (s1-s6) diabetes patients and a quantitative measure of disease progression one year after baseline which is the target we are interested in predicting.

print('Features: \n', features.head())  # type: ignore
print('Target: \n', target.describe())  # type: ignore

Features:
         age       sex       bmi  ...        s4        s5        s6
0  0.038076  0.050680  0.061696  ... -0.002592  0.019908 -0.017646
1 -0.001882 -0.044642 -0.051474  ... -0.039493 -0.068330 -0.092204
2  0.085299  0.050680  0.044451  ... -0.002592  0.002864 -0.025930
3 -0.089063 -0.044642 -0.011595  ...  0.034309  0.022692 -0.009362
4  0.005383 -0.044642 -0.036385  ... -0.002592 -0.031991 -0.046641

[5 rows x 10 columns]
Target:
 count    442.000000
mean     152.133484
std       77.093005
min       25.000000
25%       87.000000
50%      140.500000
75%      211.500000
max      346.000000
Name: target, dtype: float64

Let’s combine features and target together in one dataframe and define X and y

data_diabetes = pd.concat([features, target], axis=1)  # type: ignore

X = ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
y = 'target'

Train a ridge regression model on train dataset and use mean absolute error for scoring

scores, model = run_cross_validation(
    X=X, y=y, data=data_diabetes, preprocess_X='zscore',
    problem_type='regression', model='ridge', return_estimator='final',
    scoring='neg_mean_absolute_error')

2023-04-06 09:51:26,402 - julearn - INFO - Using default CV
2023-04-06 09:51:26,402 - julearn - INFO - ==== Input Data ====
2023-04-06 09:51:26,402 - julearn - INFO - Using dataframe as input
2023-04-06 09:51:26,402 - julearn - INFO - Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2023-04-06 09:51:26,403 - julearn - INFO - Target: target
2023-04-06 09:51:26,403 - julearn - INFO - Expanded X: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2023-04-06 09:51:26,403 - julearn - INFO - Expanded Confounds: []
2023-04-06 09:51:26,404 - julearn - INFO - ====================
2023-04-06 09:51:26,404 - julearn - INFO -
2023-04-06 09:51:26,404 - julearn - INFO - ====== Model ======
2023-04-06 09:51:26,404 - julearn - INFO - Obtaining model by name: ridge
2023-04-06 09:51:26,404 - julearn - INFO - ===================
2023-04-06 09:51:26,404 - julearn - INFO -
2023-04-06 09:51:26,405 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds

The scores dataframe has all the values for each CV split.

print(scores.head())

   fit_time  score_time  test_score  repeat  fold
0  0.013237    0.010358  -44.386924       0     0
1  0.013348    0.008491  -45.094063       0     1
2  0.013541    0.008924  -43.188016       0     2
3  0.013484    0.008634  -41.591935       0     3
4  0.013350    0.008951  -49.226121       0     4

Mean value of mean absolute error across CV

print(scores['test_score'].mean() * -1)  # type: ignore

44.56947555620943

Now do the same thing, but use mean absolute error and Pearson product-moment correlation coefficient (squared) as scoring functions

scores, model = run_cross_validation(
    X=X, y=y, data=data_diabetes, preprocess_X='zscore',
    problem_type='regression', model='ridge', return_estimator='final',
    scoring=['neg_mean_absolute_error', 'r2_corr'])

2023-04-06 09:51:27,020 - julearn - INFO - Using default CV
2023-04-06 09:51:27,020 - julearn - INFO - ==== Input Data ====
2023-04-06 09:51:27,020 - julearn - INFO - Using dataframe as input
2023-04-06 09:51:27,020 - julearn - INFO - Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2023-04-06 09:51:27,020 - julearn - INFO - Target: target
2023-04-06 09:51:27,021 - julearn - INFO - Expanded X: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2023-04-06 09:51:27,021 - julearn - INFO - Expanded Confounds: []
2023-04-06 09:51:27,022 - julearn - INFO - ====================
2023-04-06 09:51:27,022 - julearn - INFO -
2023-04-06 09:51:27,022 - julearn - INFO - ====== Model ======
2023-04-06 09:51:27,022 - julearn - INFO - Obtaining model by name: ridge
2023-04-06 09:51:27,022 - julearn - INFO - ===================
2023-04-06 09:51:27,022 - julearn - INFO -
2023-04-06 09:51:27,022 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds

Now the scores dataframe has all the values for each CV split, but two scores unders the column names ‘test_neg_mean_absolute_error’ and ‘test_r2_corr’.

print(scores[['test_neg_mean_absolute_error', 'test_r2_corr']].mean())

test_neg_mean_absolute_error   -44.257386
test_r2_corr                     0.502022
dtype: float64

If we want to define a custom scoring metric, we need to define a function that takes the predicted and the actual values as input and returns a value. In this case, we want to compute Pearson correlation coefficient (r).

def pearson_scorer(y_true, y_pred):
    return scipy.stats.pearsonr(  # type: ignore
        y_true.squeeze(), y_pred.squeeze())[0]

Before using it, we need to convert it to a sklearn scorer and register it with julearn.

register_scorer(scorer_name='pearsonr', scorer=make_scorer(pearson_scorer))

2023-04-06 09:51:27,830 - julearn - INFO - registering scorer named pearsonr

Now we can use it as another scoring metric.

scores, model = run_cross_validation(
    X=X, y=y, data=data_diabetes, preprocess_X='zscore',
    problem_type='regression', model='ridge', return_estimator='final',
    scoring=['neg_mean_absolute_error', 'r2_corr', 'pearsonr'])

2023-04-06 09:51:27,830 - julearn - INFO - Using default CV
2023-04-06 09:51:27,830 - julearn - INFO - ==== Input Data ====
2023-04-06 09:51:27,831 - julearn - INFO - Using dataframe as input
2023-04-06 09:51:27,831 - julearn - INFO - Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2023-04-06 09:51:27,831 - julearn - INFO - Target: target
2023-04-06 09:51:27,831 - julearn - INFO - Expanded X: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2023-04-06 09:51:27,831 - julearn - INFO - Expanded Confounds: []
2023-04-06 09:51:27,832 - julearn - INFO - ====================
2023-04-06 09:51:27,832 - julearn - INFO -
2023-04-06 09:51:27,832 - julearn - INFO - ====== Model ======
2023-04-06 09:51:27,832 - julearn - INFO - Obtaining model by name: ridge
2023-04-06 09:51:27,832 - julearn - INFO - ===================
2023-04-06 09:51:27,832 - julearn - INFO -
2023-04-06 09:51:27,833 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds