Return Confounds in Confound Removal

In most cases confound removal is a simple operation. You regress out the confound from the features and only continue working with these new confound removed features. This is also the default setting for julearn’s remove_confound step. But sometimes you want to work with the confound even after removing it from the features. In this example, we will discuss the options you have.

# Authors: Sami Hamdan <s.hamdan@fz-juelich.de>
#
# License: AGPL
from sklearn.datasets import load_diabetes  # to load data
from julearn.transformers import ChangeColumnTypes
from julearn import run_cross_validation
import warnings

# load in the data
df_features, target = load_diabetes(return_X_y=True, as_frame=True)

First, we can have a look at our features. You can see it includes Age, BMI, average blood pressure (bp) and 6 other measures from s1 to s6 Furthermore, it includes sex which will be considered as a confound in this example.

print('Features: ', df_features.head())
Features:          age       sex       bmi  ...        s4        s5        s6
0  0.038076  0.050680  0.061696  ... -0.002592  0.019908 -0.017646
1 -0.001882 -0.044642 -0.051474  ... -0.039493 -0.068330 -0.092204
2  0.085299  0.050680  0.044451  ... -0.002592  0.002864 -0.025930
3 -0.089063 -0.044642 -0.011595  ...  0.034309  0.022692 -0.009362
4  0.005383 -0.044642 -0.036385  ... -0.002592 -0.031991 -0.046641

[5 rows x 10 columns]

Second, we can have a look at the target

print('Target: ', target.describe())
Target:  count    442.000000
mean     152.133484
std       77.093005
min       25.000000
25%       87.000000
50%      140.500000
75%      211.500000
max      346.000000
Name: target, dtype: float64

Now, we can put both into one DataFrame:

data = df_features.copy()
data['target'] = target

In the following we will explore different settings of confound removal using Julearns pipeline functionalities.

Confound Removal Typical Use Case

Here, we want to deconfound the features and not include the confound as a feature into our last model. Afterwards, we will transform our features with a pca and run a linear regression.

feature_names = list(df_features.drop(columns='sex').columns)
with warnings.catch_warnings():
    warnings.simplefilter("ignore", lineno=443)
    scores, model = run_cross_validation(
        X=feature_names, y='target', data=data,
        confounds='sex', model='linreg', problem_type='regression',
        preprocess_X=['remove_confound', 'pca'],
        return_estimator='final')
2023-04-06 09:51:58,496 - julearn - INFO - Using default CV
2023-04-06 09:51:58,496 - julearn - INFO - ==== Input Data ====
2023-04-06 09:51:58,496 - julearn - INFO - Using dataframe as input
2023-04-06 09:51:58,496 - julearn - INFO - Features: ['age', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2023-04-06 09:51:58,496 - julearn - INFO - Target: target
2023-04-06 09:51:58,496 - julearn - INFO - Confounds: sex
2023-04-06 09:51:58,497 - julearn - INFO - Expanded X: ['age', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2023-04-06 09:51:58,497 - julearn - INFO - Expanded Confounds: ['sex']
2023-04-06 09:51:58,498 - julearn - INFO - ====================
2023-04-06 09:51:58,498 - julearn - INFO -
2023-04-06 09:51:58,498 - julearn - INFO - ====== Model ======
2023-04-06 09:51:58,498 - julearn - INFO - Obtaining model by name: linreg
2023-04-06 09:51:58,498 - julearn - INFO - ===================
2023-04-06 09:51:58,498 - julearn - INFO -
2023-04-06 09:51:58,498 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds

We can use the preprocess method of the .ExtendedDataFramePipeline to inspect the transformations/preprocessing steps of the returned estimator. By providing a step name to the until argument of the preprocess method we return the transformed X and y up to the provided step (inclusive). This output consists of a tuple containing the transformed X and y,

X_deconfounded, _ = model.preprocess(
    df_features, target, until='remove_confound')
print(X_deconfounded.head())

# As you can see the confound `sex` was dropped
# and only the confound removed features are used in the following pca.
# But what if you want to keep the confound after removal for
# other transformations.
#
# For example, let's assume that you want to do a pca on the confound removed
# feature, but want to keep the confound for the actual modelling step.
# Let us have a closer look to the confound remover in order to understand
# how we could achieve such a task:
#
# .. autoclass:: julearn.transformers.DataFrameConfoundRemover
        age       bmi        bp  ...        s4        s5        s6
0  0.029271  0.057228  0.009658  ... -0.019424  0.012311 -0.028194
1  0.005874 -0.047538 -0.015569  ... -0.024667 -0.061637 -0.082913
2  0.076494  0.039983 -0.017885  ... -0.019424 -0.004734 -0.036479
3 -0.081307 -0.007659 -0.025897  ...  0.049135  0.029385 -0.000071
4  0.013139 -0.032449  0.032632  ...  0.012234 -0.025299 -0.037349

[5 rows x 9 columns]

Above, you can see that we can set the keep_confounds argument to True. This will keep the confounds after confound removal. Here, is an example of how this can look like:

scores, model = run_cross_validation(
    X=feature_names, y='target', data=data,
    confounds='sex', model='linreg', problem_type='regression',
    preprocess_X=['remove_confound', 'pca'],
    model_params=dict(remove_confound__keep_confounds=True),
    return_estimator='final')
2023-04-06 09:52:00,045 - julearn - INFO - Using default CV
2023-04-06 09:52:00,045 - julearn - INFO - ==== Input Data ====
2023-04-06 09:52:00,045 - julearn - INFO - Using dataframe as input
2023-04-06 09:52:00,045 - julearn - INFO - Features: ['age', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2023-04-06 09:52:00,045 - julearn - INFO - Target: target
2023-04-06 09:52:00,045 - julearn - INFO - Confounds: sex
2023-04-06 09:52:00,046 - julearn - INFO - Expanded X: ['age', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2023-04-06 09:52:00,046 - julearn - INFO - Expanded Confounds: ['sex']
2023-04-06 09:52:00,047 - julearn - INFO - ====================
2023-04-06 09:52:00,047 - julearn - INFO -
2023-04-06 09:52:00,047 - julearn - INFO - ====== Model ======
2023-04-06 09:52:00,047 - julearn - INFO - Obtaining model by name: linreg
2023-04-06 09:52:00,047 - julearn - INFO - ===================
2023-04-06 09:52:00,047 - julearn - INFO -
2023-04-06 09:52:00,047 - julearn - INFO - = Model Parameters =
2023-04-06 09:52:00,047 - julearn - INFO - Setting hyperparameter remove_confound__keep_confounds = True
2023-04-06 09:52:00,049 - julearn - INFO - ====================
2023-04-06 09:52:00,049 - julearn - INFO -
2023-04-06 09:52:00,049 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds

As you can see this will keep the confound

X_deconfounded, _ = model.preprocess(
    df_features, target, until='remove_confound')
print(X_deconfounded.head())
        age       sex       bmi  ...        s4        s5        s6
0  0.029271  0.050680  0.057228  ... -0.019424  0.012311 -0.028194
1  0.005874 -0.044642 -0.047538  ... -0.024667 -0.061637 -0.082913
2  0.076494  0.050680  0.039983  ... -0.019424 -0.004734 -0.036479
3 -0.081307 -0.044642 -0.007659  ...  0.049135  0.029385 -0.000071
4  0.013139 -0.044642 -0.032449  ...  0.012234 -0.025299 -0.037349

[5 rows x 10 columns]

Even after the pca the confound will still be present. This is the case because by default transformers only transform continuous features (including features without a specified type) and ignore confounds and categorical variables.

X_transformed, _ = model.preprocess(df_features, target)
print(X_transformed.head())

# This means that the resulting Linear Regression will use the deconfounded
# features together with the confound to predict the target.
   pca_component_0  pca_component_1  ...  pca_component_7  pca_component_8
0        -0.014050         0.075715  ...        -0.008604        -0.002330
1        -0.099883        -0.062830  ...         0.024022         0.002074
2        -0.029014         0.053253  ...        -0.001197        -0.002579
3         0.035164        -0.001321  ...        -0.006567        -0.003546
4        -0.003952        -0.025446  ...         0.002095        -0.000516

[5 rows x 9 columns]

Lastly, you can also use the confound as a normal feature after confound removal. To do so you can either add the confound(s) to the which return the same columns or you can use the .ChangeColumnTypes to change the returned confounds to a continuous variable like this:

scores, model = run_cross_validation(
    X=feature_names, y='target', data=data,
    confounds='sex', model='linreg', problem_type='regression',
    preprocess_X=['remove_confound',
                  ChangeColumnTypes('.*confound', 'continuous'),
                  'pca'],
    preprocess_confounds='zscore',
    model_params=dict(remove_confound__keep_confounds=True),
    return_estimator='final'
)
2023-04-06 09:52:01,687 - julearn - INFO - Using default CV
2023-04-06 09:52:01,688 - julearn - INFO - ==== Input Data ====
2023-04-06 09:52:01,688 - julearn - INFO - Using dataframe as input
2023-04-06 09:52:01,688 - julearn - INFO - Features: ['age', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2023-04-06 09:52:01,688 - julearn - INFO - Target: target
2023-04-06 09:52:01,688 - julearn - INFO - Confounds: sex
2023-04-06 09:52:01,688 - julearn - INFO - Expanded X: ['age', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
2023-04-06 09:52:01,688 - julearn - INFO - Expanded Confounds: ['sex']
2023-04-06 09:52:01,689 - julearn - INFO - ====================
2023-04-06 09:52:01,689 - julearn - INFO -
2023-04-06 09:52:01,690 - julearn - INFO - ====== Model ======
2023-04-06 09:52:01,690 - julearn - INFO - Obtaining model by name: linreg
2023-04-06 09:52:01,690 - julearn - INFO - ===================
2023-04-06 09:52:01,690 - julearn - INFO -
2023-04-06 09:52:01,690 - julearn - INFO - = Model Parameters =
2023-04-06 09:52:01,690 - julearn - INFO - Setting hyperparameter remove_confound__keep_confounds = True
2023-04-06 09:52:01,692 - julearn - INFO - ====================
2023-04-06 09:52:01,692 - julearn - INFO -
2023-04-06 09:52:01,692 - julearn - INFO - CV interpreted as RepeatedKFold with 5 repetitions of 5 folds

As you can see this will keep the confound and change its type to a continuous variable.

X_deconfounded, _ = model.preprocess(
    df_features, target, until='changecolumntypes',
    return_trans_column_type=True)
print(X_deconfounded.head())
        age       bmi        bp  ...        s5        s6  sex__:type:__continuous
0  0.029271  0.057228  0.009658  ...  0.012311 -0.028194                 1.065488
1  0.005874 -0.047538 -0.015569  ... -0.061637 -0.082913                -0.938537
2  0.076494  0.039983 -0.017885  ... -0.004734 -0.036479                 1.065488
3 -0.081307 -0.007659 -0.025897  ...  0.029385 -0.000071                -0.938537
4  0.013139 -0.032449  0.032632  ... -0.025299 -0.037349                -0.938537

[5 rows x 10 columns]

Because the confound is treated as a normal continuous feature after removal it will be transformed in the pca as well

X_transformed, _ = model.preprocess(df_features, target)
print(X_transformed.head())
   pca_component_0  pca_component_1  ...  pca_component_8  pca_component_9
0         1.065488        -0.014050  ...        -0.008604        -0.002330
1        -0.938537        -0.099883  ...         0.024022         0.002074
2         1.065488        -0.029014  ...        -0.001197        -0.002579
3        -0.938537         0.035164  ...        -0.006567        -0.003546
4        -0.938537        -0.003952  ...         0.002095        -0.000516

[5 rows x 10 columns]

Total running time of the script: ( 0 minutes 5.381 seconds)

Gallery generated by Sphinx-Gallery