Preprocessing with variance threshold, zscore and PCA#

This example uses the make_regression function to create a simple dataset, performs a simple regression after the preprocessing of the features including removal of low variance features, feature normalization for only two features using zscore and feature reduction using PCA. We will check the features after each preprocessing step.

# Authors: Shammi More <s.more@fz-juelich.de>
#          Leonard Sasse <l.sasse@fz-juelich.de>
# License: AGPL

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from sklearn.datasets import make_regression

from julearn import run_cross_validation
from julearn.inspect import preprocess
from julearn.pipeline import PipelineCreator
from julearn.utils import configure_logging

Set the logging level to info to see extra information.

configure_logging(level="INFO")
2024-01-23 10:53:25,568 - julearn - INFO - ===== Lib Versions =====
2024-01-23 10:53:25,568 - julearn - INFO - numpy: 1.26.3
2024-01-23 10:53:25,568 - julearn - INFO - scipy: 1.12.0
2024-01-23 10:53:25,568 - julearn - INFO - sklearn: 1.3.2
2024-01-23 10:53:25,568 - julearn - INFO - pandas: 2.1.4
2024-01-23 10:53:25,569 - julearn - INFO - julearn: 0.3.1
2024-01-23 10:53:25,569 - julearn - INFO - ========================

Create a dataset using sklearn make_regression.

df = pd.DataFrame()
X, y = [f"Feature {x}" for x in range(1, 5)], "y"
df[X], df[y] = make_regression(
    n_samples=100, n_features=4, n_informative=3, noise=0.3, random_state=0
)

# We only want to zscore the first two features, so let's get their names.
first_two = X[:2]

# We can define a dictionary, in which the 'key' defines the names of our
# different 'types' of 'X'. The 'value' determine, which features belong to
# this type.
X_types = {"X_to_zscore": first_two}

Let’s look at the summary statistics of the raw features.

print("Summary Statistics of the raw features : \n", df.describe())
Summary Statistics of the raw features :
         Feature 1   Feature 2   Feature 3   Feature 4           y
count  100.000000  100.000000  100.000000  100.000000  100.000000
mean    -0.111790    0.141618    0.023682   -0.167718  -14.686170
std      0.977322    0.972988    1.087946    0.895011   79.741431
min     -2.069985   -2.772593   -2.552990   -2.659172 -204.293317
25%     -0.751257   -0.484100   -0.759419   -0.688891  -64.724008
50%     -0.206729    0.194442   -0.028152   -0.174160  -16.789944
75%      0.421885    0.724302    0.772156    0.408321   30.392804
max      1.943621    2.256723    2.383145    2.259309  216.221085

We will preprocess all features using variance thresholding. We will only zscore the first two features, and then perform PCA using all features. We will zscore the target and then train a random forest model. Since we use the PipelineCreator object we have to explicitly declare which X_types each preprocessing step should be applied to. If we do not declare the type in the add method using the apply_to keyword argument, the step will default to "continuous" or to another type that can be declared in the constructor of the PipelineCreator. To transform the target we could set apply_to="target", which is a special type, that cannot be user-defined. Please note also that if a step is added to transform the target, you also have to explicitly add the model that is to be used in the regression to the PipelineCreator.

# Define model parameters and preprocessing steps first
# The hyperparameters for each step can be added as a keyword argument and
# should be either one parameter or an iterable of multiple parameters for a
# search.

# Setting the threshold for variance to 0.15, number of PCA components to 2
# and number of trees for random forest to 200.

# By setting "apply_to=*", we can apply the preprocessing step to all features.
pipeline_creator = PipelineCreator(problem_type="regression")

pipeline_creator.add("select_variance", apply_to="*", threshold=0.15)
pipeline_creator.add("zscore", apply_to="X_to_zscore")
pipeline_creator.add("pca", apply_to="*", n_components=2)
pipeline_creator.add("rf", apply_to="*", n_estimators=200)

# Because we have already added the model to the pipeline creator, we can
# simply drop in the ``pipeline_creator`` as a model. If we did not add a model
# here, we could add the ``pipeline_creator`` using the keyword argument
# ``preprocess`` and hand over a model separately.

scores, model = run_cross_validation(
    X=X,
    y=y,
    X_types=X_types,
    data=df,
    model=pipeline_creator,
    scoring=["r2", "neg_mean_absolute_error"],
    return_estimator="final",
    seed=200,
)

# We can use the final estimator to inspect the transformed features at a
# specific step of the pipeline. Since the PCA was the last step added to the
# pipeline, we can simply get the model up to this step by indexing as follows:

X_after_pca = model[:-1].transform(df[X])

print("X after PCA:")
print("=" * 79)
print(X_after_pca)

# We can select pipelines up to earlier steps by indexing previous elements
# in the final estimator. For example, to inspect features after the zscoring
# step:

X_after_zscore = model[:-2].transform(df[X])
print("X after zscore:")
print("=" * 79)
print(X_after_zscore)

# However, to make this less confusing you can also simply use the high-level
# function ``preprocess`` to explicitly refer to a pipeline step by name:

X_after_pca = preprocess(model, X=X, data=df, until="pca")
X_after_zscore = preprocess(model, X=X, data=df, until="zscore")

# Let's plot scatter plots for raw features and the PCA components.
fig, axes = plt.subplots(1, 2, figsize=(12, 6))
sns.scatterplot(x=X[0], y=X[1], data=df, ax=axes[0])
axes[0].set_title("Raw features")
sns.scatterplot(x="pca__pca0", y="pca__pca1", data=X_after_pca, ax=axes[1])
axes[1].set_title("PCA components")
Raw features, PCA components
2024-01-23 10:53:25,583 - julearn - INFO - Adding step select_variance that applies to ColumnTypes<types={'*'}; pattern=.*>
2024-01-23 10:53:25,583 - julearn - INFO - Setting hyperparameter threshold = 0.15
2024-01-23 10:53:25,583 - julearn - INFO - Step added
2024-01-23 10:53:25,583 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'X_to_zscore'}; pattern=(?:__:type:__X_to_zscore)>
2024-01-23 10:53:25,583 - julearn - INFO - Step added
2024-01-23 10:53:25,584 - julearn - INFO - Adding step pca that applies to ColumnTypes<types={'*'}; pattern=.*>
2024-01-23 10:53:25,584 - julearn - INFO - Setting hyperparameter n_components = 2
2024-01-23 10:53:25,584 - julearn - INFO - Step added
2024-01-23 10:53:25,584 - julearn - INFO - Adding step rf that applies to ColumnTypes<types={'*'}; pattern=.*>
2024-01-23 10:53:25,584 - julearn - INFO - Setting hyperparameter n_estimators = 200
2024-01-23 10:53:25,584 - julearn - INFO - Step added
2024-01-23 10:53:25,584 - julearn - INFO - Setting random seed to 200
2024-01-23 10:53:25,584 - julearn - INFO - ==== Input Data ====
2024-01-23 10:53:25,584 - julearn - INFO - Using dataframe as input
2024-01-23 10:53:25,584 - julearn - INFO -      Features: ['Feature 1', 'Feature 2', 'Feature 3', 'Feature 4']
2024-01-23 10:53:25,584 - julearn - INFO -      Target: y
2024-01-23 10:53:25,585 - julearn - INFO -      Expanded features: ['Feature 1', 'Feature 2', 'Feature 3', 'Feature 4']
2024-01-23 10:53:25,585 - julearn - INFO -      X_types:{'X_to_zscore': ['Feature 1', 'Feature 2']}
2024-01-23 10:53:25,585 - julearn - WARNING - The following columns are not defined in X_types: ['Feature 3', 'Feature 4']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/prepare.py:503: RuntimeWarning: The following columns are not defined in X_types: ['Feature 3', 'Feature 4']. They will be treated as continuous.
  warn_with_log(
2024-01-23 10:53:25,586 - julearn - INFO - ====================
2024-01-23 10:53:25,586 - julearn - INFO -
2024-01-23 10:53:25,587 - julearn - INFO - = Model Parameters =
2024-01-23 10:53:25,587 - julearn - INFO - ====================
2024-01-23 10:53:25,587 - julearn - INFO -
2024-01-23 10:53:25,587 - julearn - INFO - = Data Information =
2024-01-23 10:53:25,587 - julearn - INFO -      Problem type: regression
2024-01-23 10:53:25,587 - julearn - INFO -      Number of samples: 100
2024-01-23 10:53:25,587 - julearn - INFO -      Number of features: 4
2024-01-23 10:53:25,587 - julearn - INFO - ====================
2024-01-23 10:53:25,587 - julearn - INFO -
2024-01-23 10:53:25,587 - julearn - INFO -      Target type: float64
2024-01-23 10:53:25,588 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
X after PCA:
===============================================================================
    pca__pca0  pca__pca1
0    1.725696  -1.582295
1    0.314458  -0.942935
2    0.977336   0.463217
3    0.525139  -0.219493
4   -1.618559   0.908592
..        ...        ...
95  -0.395105   0.743192
96  -0.328111  -0.670912
97  -0.573900  -0.401304
98   0.918034   0.137435
99   0.894566   0.357729

[100 rows x 2 columns]
X after zscore:
===============================================================================
    Feature 1__:type:__X_to_zscore  ...  Feature 4__:type:__continuous
0                        -2.013728  ...                       0.426259
1                        -0.364686  ...                      -0.944446
2                         0.118838  ...                       0.931848
3                        -0.004437  ...                      -2.030684
4                         0.309092  ...                       0.523891
..                             ...  ...                            ...
95                       -0.206189  ...                      -0.157667
96                       -1.099154  ...                      -0.028182
97                       -0.335538  ...                      -1.252795
98                       -0.049139  ...                      -0.133702
99                        0.240085  ...                       0.443863

[100 rows x 4 columns]

Text(0.5, 1.0, 'PCA components')

Let’s look at the summary statistics of the zscored features. We see here that the mean of all the features is zero and standard deviation is one.

print(
    "Summary Statistics of the zscored features : \n",
    X_after_zscore.describe(),
)
Summary Statistics of the zscored features :
           Feature 1     Feature 2   Feature 3   Feature 4
count  1.000000e+02  1.000000e+02  100.000000  100.000000
mean  -5.689893e-17  1.387779e-17    0.023682   -0.167718
std    1.005038e+00  1.005038e+00    1.087946    0.895011
min   -2.013728e+00 -3.010202e+00   -2.552990   -2.659172
25%   -6.576015e-01 -6.463286e-01   -0.759419   -0.688891
50%   -9.763131e-02  5.456486e-02   -0.028152   -0.174160
75%    5.488094e-01  6.018781e-01    0.772156    0.408321
max    2.113700e+00  2.184776e+00    2.383145    2.259309

Total running time of the script: (0 minutes 1.328 seconds)

Gallery generated by Sphinx-Gallery