⚠ Old version — You are viewing an old version of julearn (v0.2.7). Switch to the latest stable release (v0.3.5).

Custom Scoring Function for Regression

This example uses the ‘diabetes’ data from sklearn datasets and performs a regression analysis using a Ridge Regression model. As scorers, it uses scikit-learn, julearn and a custom metric defined by the user.

# Authors: Shammi More <s.more@fz-juelich.de>
#          Federico Raimondo <f.raimondo@fz-juelich.de>
#
# License: AGPL

import pandas as pd
import scipy
from sklearn.datasets import load_diabetes

from sklearn.metrics import make_scorer
from julearn.scoring import register_scorer

from julearn import run_cross_validation
from julearn.utils import configure_logging

Set the logging level to info to see extra information

configure_logging(level='INFO')

2026-05-29 20:46:21,209 - julearn - INFO - ===== Lib Versions =====
2026-05-29 20:46:21,210 - julearn - INFO - numpy: 1.23.5
2026-05-29 20:46:21,210 - julearn - INFO - scipy: 1.10.1
2026-05-29 20:46:21,210 - julearn - INFO - sklearn: 1.0.2
2026-05-29 20:46:21,210 - julearn - INFO - pandas: 2.0.3
2026-05-29 20:46:21,210 - julearn - INFO - julearn: 0.3.0.dev0
2026-05-29 20:46:21,210 - julearn - INFO - ========================

load the diabetes data from sklearn as a pandas dataframe

features, target = load_diabetes(return_X_y=True, as_frame=True)

Dataset contains ten variables age, sex, body mass index, average blood pressure, and six blood serum measurements (s1-s6) diabetes patients and a quantitative measure of disease progression one year after baseline which is the target we are interested in predicting.

print('Features: \n', features.head())  # type: ignore
print('Target: \n', target.describe())  # type: ignore

Features:
         age       sex       bmi        bp        s1        s2        s3        s4        s5        s6
0  0.038076  0.050680  0.061696  0.021872 -0.044223 -0.034821 -0.043401 -0.002592  0.019908 -0.017646
1 -0.001882 -0.044642 -0.051474 -0.026328 -0.008449 -0.019163  0.074412 -0.039493 -0.068330 -0.092204
2  0.085299  0.050680  0.044451 -0.005671 -0.045599 -0.034194 -0.032356 -0.002592  0.002864 -0.025930
3 -0.089063 -0.044642 -0.011595 -0.036656  0.012191  0.024991 -0.036038  0.034309  0.022692 -0.009362
4  0.005383 -0.044642 -0.036385  0.021872  0.003935  0.015596  0.008142 -0.002592 -0.031991 -0.046641
Target:
 count    442.000000
mean     152.133484
std       77.093005
min       25.000000
25%       87.000000
50%      140.500000
75%      211.500000
max      346.000000
Name: target, dtype: float64

Let’s combine features and target together in one dataframe and define X and y

data_diabetes = pd.concat([features, target], axis=1)  # type: ignore

X = ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
y = 'target'

Train a ridge regression model on train dataset and use mean absolute error for scoring

scores, model = run_cross_validation(
    X=X, y=y, data=data_diabetes, preprocess_X='zscore',
    problem_type='regression', model='ridge', return_estimator='final',
    scoring='neg_mean_absolute_error')

2026-05-29 20:46:21,240 - julearn - INFO - Using default CV
2026-05-29 20:46:21,240 - julearn - INFO - ==== Input Data ====
2026-05-29 20:46:21,240 - julearn - INFO - Using dataframe as input
2026-05-29 20:46:21,240 - julearn - INFO - Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2026-05-29 20:46:21,240 - julearn - INFO - Target: target
2026-05-29 20:46:21,241 - julearn - INFO - Expanded X: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2026-05-29 20:46:21,241 - julearn - INFO - Expanded Confounds: []
2026-05-29 20:46:21,242 - julearn - INFO - ====================
2026-05-29 20:46:21,242 - julearn - INFO -
2026-05-29 20:46:21,243 - julearn - INFO - ====== Model ======
2026-05-29 20:46:21,243 - julearn - INFO - Obtaining model by name: ridge
2026-05-29 20:46:21,243 - julearn - INFO - ===================
2026-05-29 20:46:21,243 - julearn - INFO -
2026-05-29 20:46:21,243 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds

The scores dataframe has all the values for each CV split.

print(scores.head())

   fit_time  score_time  test_score  repeat  fold
0  0.012274    0.010971  -44.386924       0     0
1  0.014085    0.005766  -45.094063       0     1
2  0.012964    0.007606  -43.188016       0     2
3  0.009178    0.007386  -41.591935       0     3
4  0.009075    0.007330  -49.226121       0     4

Mean value of mean absolute error across CV

print(scores['test_score'].mean() * -1)  # type: ignore

44.56947555620943

Now do the same thing, but use mean absolute error and Pearson product-moment correlation coefficient (squared) as scoring functions

scores, model = run_cross_validation(
    X=X, y=y, data=data_diabetes, preprocess_X='zscore',
    problem_type='regression', model='ridge', return_estimator='final',
    scoring=['neg_mean_absolute_error', 'r2_corr'])

2026-05-29 20:46:21,794 - julearn - INFO - Using default CV
2026-05-29 20:46:21,795 - julearn - INFO - ==== Input Data ====
2026-05-29 20:46:21,795 - julearn - INFO - Using dataframe as input
2026-05-29 20:46:21,795 - julearn - INFO - Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2026-05-29 20:46:21,795 - julearn - INFO - Target: target
2026-05-29 20:46:21,796 - julearn - INFO - Expanded X: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2026-05-29 20:46:21,796 - julearn - INFO - Expanded Confounds: []
2026-05-29 20:46:21,797 - julearn - INFO - ====================
2026-05-29 20:46:21,798 - julearn - INFO -
2026-05-29 20:46:21,798 - julearn - INFO - ====== Model ======
2026-05-29 20:46:21,798 - julearn - INFO - Obtaining model by name: ridge
2026-05-29 20:46:21,798 - julearn - INFO - ===================
2026-05-29 20:46:21,798 - julearn - INFO -
2026-05-29 20:46:21,798 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds

Now the scores dataframe has all the values for each CV split, but two scores unders the column names ‘test_neg_mean_absolute_error’ and ‘test_r2_corr’.

print(scores[['test_neg_mean_absolute_error', 'test_r2_corr']].mean())

test_neg_mean_absolute_error   -44.257386
test_r2_corr                     0.502022
dtype: float64

If we want to define a custom scoring metric, we need to define a function that takes the predicted and the actual values as input and returns a value. In this case, we want to compute Pearson correlation coefficient (r).

def pearson_scorer(y_true, y_pred):
    return scipy.stats.pearsonr(  # type: ignore
        y_true.squeeze(), y_pred.squeeze())[0]

Before using it, we need to convert it to a sklearn scorer and register it with julearn.

register_scorer(scorer_name='pearsonr', scorer=make_scorer(pearson_scorer))

2026-05-29 20:46:22,523 - julearn - INFO - registering scorer named pearsonr

Now we can use it as another scoring metric.

scores, model = run_cross_validation(
    X=X, y=y, data=data_diabetes, preprocess_X='zscore',
    problem_type='regression', model='ridge', return_estimator='final',
    scoring=['neg_mean_absolute_error', 'r2_corr', 'pearsonr'])

2026-05-29 20:46:22,523 - julearn - INFO - Using default CV
2026-05-29 20:46:22,524 - julearn - INFO - ==== Input Data ====
2026-05-29 20:46:22,524 - julearn - INFO - Using dataframe as input
2026-05-29 20:46:22,524 - julearn - INFO - Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2026-05-29 20:46:22,524 - julearn - INFO - Target: target
2026-05-29 20:46:22,524 - julearn - INFO - Expanded X: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2026-05-29 20:46:22,525 - julearn - INFO - Expanded Confounds: []
2026-05-29 20:46:22,526 - julearn - INFO - ====================
2026-05-29 20:46:22,526 - julearn - INFO -
2026-05-29 20:46:22,526 - julearn - INFO - ====== Model ======
2026-05-29 20:46:22,526 - julearn - INFO - Obtaining model by name: ridge
2026-05-29 20:46:22,526 - julearn - INFO - ===================
2026-05-29 20:46:22,527 - julearn - INFO -
2026-05-29 20:46:22,527 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds