Note

Go to the end to download the full example code

Custom Scoring Function for Regression#

This example uses the diabetes data from sklearn datasets and performs a regression analysis using a Ridge Regression model. As scorers, it uses scikit-learn, julearn and a custom metric defined by the user.

# Authors: Shammi More <s.more@fz-juelich.de>
#          Federico Raimondo <f.raimondo@fz-juelich.de>
# License: AGPL

import pandas as pd
import scipy
from sklearn.datasets import load_diabetes

from sklearn.metrics import make_scorer
from julearn.scoring import register_scorer

from julearn import run_cross_validation
from julearn.utils import configure_logging

Set the logging level to info to see extra information.

configure_logging(level="INFO")

2024-05-16 08:52:54,316 - julearn - INFO - ===== Lib Versions =====
2024-05-16 08:52:54,316 - julearn - INFO - numpy: 1.26.4
2024-05-16 08:52:54,316 - julearn - INFO - scipy: 1.13.0
2024-05-16 08:52:54,316 - julearn - INFO - sklearn: 1.4.2
2024-05-16 08:52:54,316 - julearn - INFO - pandas: 2.1.4
2024-05-16 08:52:54,316 - julearn - INFO - julearn: 0.3.3
2024-05-16 08:52:54,316 - julearn - INFO - ========================

load the diabetes data from sklearn as a pandas.DataFrame.

features, target = load_diabetes(return_X_y=True, as_frame=True)

Dataset contains ten variables age, sex, body mass index, average blood pressure, and six blood serum measurements (s1-s6) diabetes patients and a quantitative measure of disease progression one year after baseline which is the target we are interested in predicting.

print("Features: \n", features.head())
print("Target: \n", target.describe())

Features:
         age       sex       bmi  ...        s4        s5        s6
0  0.038076  0.050680  0.061696  ... -0.002592  0.019907 -0.017646
1 -0.001882 -0.044642 -0.051474  ... -0.039493 -0.068332 -0.092204
2  0.085299  0.050680  0.044451  ... -0.002592  0.002861 -0.025930
3 -0.089063 -0.044642 -0.011595  ...  0.034309  0.022688 -0.009362
4  0.005383 -0.044642 -0.036385  ... -0.002592 -0.031988 -0.046641

[5 rows x 10 columns]
Target:
 count    442.000000
mean     152.133484
std       77.093005
min       25.000000
25%       87.000000
50%      140.500000
75%      211.500000
max      346.000000
Name: target, dtype: float64

Let’s combine features and target together in one dataframe and define X and y.

data_diabetes = pd.concat([features, target], axis=1)  # type: ignore

X = ["age", "sex", "bmi", "bp", "s1", "s2", "s3", "s4", "s5", "s6"]
y = "target"

Train a ridge regression model on train dataset and use mean absolute error for scoring.

scores, model = run_cross_validation(
    X=X,
    y=y,
    data=data_diabetes,
    preprocess="zscore",
    problem_type="regression",
    model="ridge",
    return_estimator="final",
    scoring="neg_mean_absolute_error",
)

2024-05-16 08:52:54,331 - julearn - INFO - ==== Input Data ====
2024-05-16 08:52:54,331 - julearn - INFO - Using dataframe as input
2024-05-16 08:52:54,331 - julearn - INFO -      Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2024-05-16 08:52:54,331 - julearn - INFO -      Target: target
2024-05-16 08:52:54,332 - julearn - INFO -      Expanded features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2024-05-16 08:52:54,332 - julearn - INFO -      X_types:{}
2024-05-16 08:52:54,332 - julearn - WARNING - The following columns are not defined in X_types: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/prepare.py:505: RuntimeWarning: The following columns are not defined in X_types: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']. They will be treated as continuous.
  warn_with_log(
2024-05-16 08:52:54,332 - julearn - INFO - ====================
2024-05-16 08:52:54,332 - julearn - INFO -
2024-05-16 08:52:54,333 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-05-16 08:52:54,333 - julearn - INFO - Step added
2024-05-16 08:52:54,333 - julearn - INFO - Adding step ridge that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-05-16 08:52:54,333 - julearn - INFO - Step added
2024-05-16 08:52:54,333 - julearn - INFO - = Model Parameters =
2024-05-16 08:52:54,333 - julearn - INFO - ====================
2024-05-16 08:52:54,333 - julearn - INFO -
2024-05-16 08:52:54,333 - julearn - INFO - = Data Information =
2024-05-16 08:52:54,334 - julearn - INFO -      Problem type: regression
2024-05-16 08:52:54,334 - julearn - INFO -      Number of samples: 442
2024-05-16 08:52:54,334 - julearn - INFO -      Number of features: 10
2024-05-16 08:52:54,334 - julearn - INFO - ====================
2024-05-16 08:52:54,334 - julearn - INFO -
2024-05-16 08:52:54,334 - julearn - INFO -      Target type: float64
2024-05-16 08:52:54,334 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2024-05-16 08:52:54,372 - julearn - INFO - Fitting final model

The scores dataframe has all the values for each CV split.

scores.head()
fit_time score_time test_score n_train n_test repeat fold cv_mdsum
0 0.004858 0.002419 -43.104359 353 89 0 0 b10eef89b4192178d482d7a1587a248a
1 0.004534 0.002306 -44.861364 353 89 0 1 b10eef89b4192178d482d7a1587a248a
2 0.004487 0.002314 -47.981407 354 88 0 2 b10eef89b4192178d482d7a1587a248a
3 0.004463 0.002302 -42.956254 354 88 0 3 b10eef89b4192178d482d7a1587a248a
4 0.004431 0.002301 -42.419886 354 88 0 4 b10eef89b4192178d482d7a1587a248a


Mean value of mean absolute error across CV.

print(scores["test_score"].mean() * -1)

44.264653948271885

Now do the same thing, but use mean absolute error and Pearson product-moment correlation coefficient (squared) as scoring functions.

scores, model = run_cross_validation(
    X=X,
    y=y,
    data=data_diabetes,
    preprocess="zscore",
    problem_type="regression",
    model="ridge",
    return_estimator="final",
    scoring=["neg_mean_absolute_error", "r2_corr"],
)

2024-05-16 08:52:54,379 - julearn - INFO - ==== Input Data ====
2024-05-16 08:52:54,379 - julearn - INFO - Using dataframe as input
2024-05-16 08:52:54,379 - julearn - INFO -      Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2024-05-16 08:52:54,380 - julearn - INFO -      Target: target
2024-05-16 08:52:54,380 - julearn - INFO -      Expanded features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2024-05-16 08:52:54,380 - julearn - INFO -      X_types:{}
2024-05-16 08:52:54,380 - julearn - WARNING - The following columns are not defined in X_types: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/prepare.py:505: RuntimeWarning: The following columns are not defined in X_types: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']. They will be treated as continuous.
  warn_with_log(
2024-05-16 08:52:54,380 - julearn - INFO - ====================
2024-05-16 08:52:54,381 - julearn - INFO -
2024-05-16 08:52:54,381 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-05-16 08:52:54,381 - julearn - INFO - Step added
2024-05-16 08:52:54,381 - julearn - INFO - Adding step ridge that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-05-16 08:52:54,381 - julearn - INFO - Step added
2024-05-16 08:52:54,381 - julearn - INFO - = Model Parameters =
2024-05-16 08:52:54,381 - julearn - INFO - ====================
2024-05-16 08:52:54,382 - julearn - INFO -
2024-05-16 08:52:54,382 - julearn - INFO - = Data Information =
2024-05-16 08:52:54,382 - julearn - INFO -      Problem type: regression
2024-05-16 08:52:54,382 - julearn - INFO -      Number of samples: 442
2024-05-16 08:52:54,382 - julearn - INFO -      Number of features: 10
2024-05-16 08:52:54,382 - julearn - INFO - ====================
2024-05-16 08:52:54,382 - julearn - INFO -
2024-05-16 08:52:54,382 - julearn - INFO -      Target type: float64
2024-05-16 08:52:54,382 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2024-05-16 08:52:54,424 - julearn - INFO - Fitting final model

Now the scores dataframe has all the values for each CV split, but two scores unders the column names "test_neg_mean_absolute_error" and "test_r2_corr".

print(scores[["test_neg_mean_absolute_error", "test_r2_corr"]].mean())

test_neg_mean_absolute_error   -44.264654
test_r2_corr                     0.486498
dtype: float64

If we want to define a custom scoring metric, we need to define a function that takes the predicted and the actual values as input and returns a value. In this case, we want to compute Pearson correlation coefficient (r).

def pearson_scorer(y_true, y_pred):
    return scipy.stats.pearsonr(y_true.squeeze(), y_pred.squeeze())[0]

Before using it, we need to convert it to a sklearn scorer and register it with julearn.

register_scorer(scorer_name="pearsonr", scorer=make_scorer(pearson_scorer))

2024-05-16 08:52:54,431 - julearn - INFO - registering scorer named pearsonr

Now we can use it as another scoring metric.

scores, model = run_cross_validation(
    X=X,
    y=y,
    data=data_diabetes,
    preprocess="zscore",
    problem_type="regression",
    model="ridge",
    return_estimator="final",
    scoring=["neg_mean_absolute_error", "r2_corr", "pearsonr"],
)

2024-05-16 08:52:54,431 - julearn - INFO - ==== Input Data ====
2024-05-16 08:52:54,431 - julearn - INFO - Using dataframe as input
2024-05-16 08:52:54,431 - julearn - INFO -      Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2024-05-16 08:52:54,432 - julearn - INFO -      Target: target
2024-05-16 08:52:54,432 - julearn - INFO -      Expanded features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2024-05-16 08:52:54,432 - julearn - INFO -      X_types:{}
2024-05-16 08:52:54,432 - julearn - WARNING - The following columns are not defined in X_types: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/prepare.py:505: RuntimeWarning: The following columns are not defined in X_types: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']. They will be treated as continuous.
  warn_with_log(
2024-05-16 08:52:54,433 - julearn - INFO - ====================
2024-05-16 08:52:54,433 - julearn - INFO -
2024-05-16 08:52:54,433 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-05-16 08:52:54,433 - julearn - INFO - Step added
2024-05-16 08:52:54,434 - julearn - INFO - Adding step ridge that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-05-16 08:52:54,434 - julearn - INFO - Step added
2024-05-16 08:52:54,434 - julearn - INFO - = Model Parameters =
2024-05-16 08:52:54,434 - julearn - INFO - ====================
2024-05-16 08:52:54,435 - julearn - INFO -
2024-05-16 08:52:54,435 - julearn - INFO - = Data Information =
2024-05-16 08:52:54,435 - julearn - INFO -      Problem type: regression
2024-05-16 08:52:54,435 - julearn - INFO -      Number of samples: 442
2024-05-16 08:52:54,435 - julearn - INFO -      Number of features: 10
2024-05-16 08:52:54,435 - julearn - INFO - ====================
2024-05-16 08:52:54,435 - julearn - INFO -
2024-05-16 08:52:54,435 - julearn - INFO -      Target type: float64
2024-05-16 08:52:54,435 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2024-05-16 08:52:54,487 - julearn - INFO - Fitting final model

Total running time of the script: (0 minutes 0.181 seconds)

Gallery generated by Sphinx-Gallery