Simple Binary Classification

This example uses the iris dataset and performs a simple binary classification using a Support Vector Machine classifier.

# Authors: Federico Raimondo <f.raimondo@fz-juelich.de>
#
# License: AGPL

from seaborn import load_dataset
from julearn import run_cross_validation
from julearn.utils import configure_logging

Set the logging level to info to see extra information

configure_logging(level="INFO")

2024-04-29 11:45:22,921 - julearn - INFO - ===== Lib Versions =====
2024-04-29 11:45:22,921 - julearn - INFO - numpy: 1.26.4
2024-04-29 11:45:22,921 - julearn - INFO - scipy: 1.13.0
2024-04-29 11:45:22,921 - julearn - INFO - sklearn: 1.4.2
2024-04-29 11:45:22,921 - julearn - INFO - pandas: 2.1.4
2024-04-29 11:45:22,921 - julearn - INFO - julearn: 0.3.2.dev57
2024-04-29 11:45:22,921 - julearn - INFO - ========================

df_iris = load_dataset("iris")

The dataset has three kind of species. We will keep two to perform a binary classification.

df_iris = df_iris[df_iris["species"].isin(["versicolor", "virginica"])]

As features, we will use the sepal length, width and petal length. We will try to predict the species.

X = ["sepal_length", "sepal_width", "petal_length"]
y = "species"
scores = run_cross_validation(
    X=X,
    y=y,
    data=df_iris,
    model="svm",
    problem_type="classification",
    preprocess="zscore",
)

print(scores["test_score"])

2024-04-29 11:45:22,992 - julearn - INFO - ==== Input Data ====
2024-04-29 11:45:22,992 - julearn - INFO - Using dataframe as input
2024-04-29 11:45:22,992 - julearn - INFO -      Features: ['sepal_length', 'sepal_width', 'petal_length']
2024-04-29 11:45:22,992 - julearn - INFO -      Target: species
2024-04-29 11:45:22,992 - julearn - INFO -      Expanded features: ['sepal_length', 'sepal_width', 'petal_length']
2024-04-29 11:45:22,992 - julearn - INFO -      X_types:{}
2024-04-29 11:45:22,992 - julearn - WARNING - The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/prepare.py:507: RuntimeWarning: The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous.
  warn_with_log(
2024-04-29 11:45:22,993 - julearn - INFO - ====================
2024-04-29 11:45:22,993 - julearn - INFO -
2024-04-29 11:45:22,993 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-04-29 11:45:22,993 - julearn - INFO - Step added
2024-04-29 11:45:22,994 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-04-29 11:45:22,994 - julearn - INFO - Step added
2024-04-29 11:45:22,995 - julearn - INFO - = Model Parameters =
2024-04-29 11:45:22,995 - julearn - INFO - ====================
2024-04-29 11:45:22,995 - julearn - INFO -
2024-04-29 11:45:22,995 - julearn - INFO - = Data Information =
2024-04-29 11:45:22,995 - julearn - INFO -      Problem type: classification
2024-04-29 11:45:22,995 - julearn - INFO -      Number of samples: 100
2024-04-29 11:45:22,995 - julearn - INFO -      Number of features: 3
2024-04-29 11:45:22,995 - julearn - INFO - ====================
2024-04-29 11:45:22,995 - julearn - INFO -
2024-04-29 11:45:22,995 - julearn - INFO -      Number of classes: 2
2024-04-29 11:45:22,995 - julearn - INFO -      Target type: object
2024-04-29 11:45:22,996 - julearn - INFO -      Class distributions: species
versicolor    50
virginica     50
Name: count, dtype: int64
2024-04-29 11:45:22,996 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2024-04-29 11:45:22,996 - julearn - INFO - Binary classification problem detected.
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:73: FutureWarning: `fit_params` is deprecated and will be removed in version 1.6. Pass parameters via `params` instead.
  warnings.warn(
0    0.90
1    0.75
2    0.95
3    0.70
4    0.90
Name: test_score, dtype: float64

Additionally, we can choose to assess the performance of the model using different scoring functions.

For example, we might have an unbalanced dataset:

df_unbalanced = df_iris[20:]  # drop the first 20 versicolor samples
print(df_unbalanced["species"].value_counts())

species
virginica     50
versicolor    30
Name: count, dtype: int64

If we compute the accuracy, we might not account for this imbalance. A more suitable metric is the balanced_accuracy. More information in scikit-learn: balanced_accuracy_score().

We will also set the random seed so we always split the data in the same way.

scores = run_cross_validation(
    X=X,
    y=y,
    data=df_unbalanced,
    model="svm",
    seed=42,
    preprocess="zscore",
    problem_type="classification",
    scoring=["accuracy", "balanced_accuracy"],
)

print(scores["test_accuracy"].mean())
print(scores["test_balanced_accuracy"].mean())

2024-04-29 11:45:23,037 - julearn - INFO - Setting random seed to 42
2024-04-29 11:45:23,037 - julearn - INFO - ==== Input Data ====
2024-04-29 11:45:23,038 - julearn - INFO - Using dataframe as input
2024-04-29 11:45:23,038 - julearn - INFO -      Features: ['sepal_length', 'sepal_width', 'petal_length']
2024-04-29 11:45:23,038 - julearn - INFO -      Target: species
2024-04-29 11:45:23,038 - julearn - INFO -      Expanded features: ['sepal_length', 'sepal_width', 'petal_length']
2024-04-29 11:45:23,038 - julearn - INFO -      X_types:{}
2024-04-29 11:45:23,038 - julearn - WARNING - The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/prepare.py:507: RuntimeWarning: The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous.
  warn_with_log(
2024-04-29 11:45:23,038 - julearn - INFO - ====================
2024-04-29 11:45:23,038 - julearn - INFO -
2024-04-29 11:45:23,039 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-04-29 11:45:23,039 - julearn - INFO - Step added
2024-04-29 11:45:23,039 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-04-29 11:45:23,039 - julearn - INFO - Step added
2024-04-29 11:45:23,039 - julearn - INFO - = Model Parameters =
2024-04-29 11:45:23,039 - julearn - INFO - ====================
2024-04-29 11:45:23,039 - julearn - INFO -
2024-04-29 11:45:23,039 - julearn - INFO - = Data Information =
2024-04-29 11:45:23,039 - julearn - INFO -      Problem type: classification
2024-04-29 11:45:23,039 - julearn - INFO -      Number of samples: 80
2024-04-29 11:45:23,040 - julearn - INFO -      Number of features: 3
2024-04-29 11:45:23,040 - julearn - INFO - ====================
2024-04-29 11:45:23,040 - julearn - INFO -
2024-04-29 11:45:23,040 - julearn - INFO -      Number of classes: 2
2024-04-29 11:45:23,040 - julearn - INFO -      Target type: object
2024-04-29 11:45:23,040 - julearn - INFO -      Class distributions: species
virginica     50
versicolor    30
Name: count, dtype: int64
2024-04-29 11:45:23,040 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2024-04-29 11:45:23,040 - julearn - INFO - Binary classification problem detected.
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:73: FutureWarning: `fit_params` is deprecated and will be removed in version 1.6. Pass parameters via `params` instead.
  warnings.warn(
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sklearn/metrics/_classification.py:2458: UserWarning: y_pred contains classes not in y_true
  warnings.warn("y_pred contains classes not in y_true")
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sklearn/metrics/_classification.py:2458: UserWarning: y_pred contains classes not in y_true
  warnings.warn("y_pred contains classes not in y_true")
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sklearn/metrics/_classification.py:2458: UserWarning: y_pred contains classes not in y_true
  warnings.warn("y_pred contains classes not in y_true")
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sklearn/metrics/_classification.py:2458: UserWarning: y_pred contains classes not in y_true
  warnings.warn("y_pred contains classes not in y_true")
0.8625
0.8678571428571429

Other kind of metrics allows us to evaluate how good our model is to detect specific targets. Suppose we want to create a model that correctly identifies the versicolor samples.

Now we might want to evaluate the precision score, or the ratio of true positives (tp) over all positives (true and false positives). More information in scikit-learn: precision_score().

For this metric to work, we need to define which are our positive values. In this example, we are interested in detecting versicolor.

precision_scores = run_cross_validation(
    X=X,
    y=y,
    data=df_unbalanced,
    model="svm",
    preprocess="zscore",
    problem_type="classification",
    seed=42,
    scoring="precision",
    pos_labels="versicolor",
)
print(precision_scores["test_score"].mean())

2024-04-29 11:45:23,086 - julearn - INFO - Setting random seed to 42
2024-04-29 11:45:23,086 - julearn - INFO - ==== Input Data ====
2024-04-29 11:45:23,086 - julearn - INFO - Using dataframe as input
2024-04-29 11:45:23,086 - julearn - INFO -      Features: ['sepal_length', 'sepal_width', 'petal_length']
2024-04-29 11:45:23,086 - julearn - INFO -      Target: species
2024-04-29 11:45:23,086 - julearn - INFO -      Expanded features: ['sepal_length', 'sepal_width', 'petal_length']
2024-04-29 11:45:23,086 - julearn - INFO -      X_types:{}
2024-04-29 11:45:23,086 - julearn - WARNING - The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/prepare.py:507: RuntimeWarning: The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous.
  warn_with_log(
2024-04-29 11:45:23,087 - julearn - INFO - Setting the following as positive labels ['versicolor']
2024-04-29 11:45:23,087 - julearn - INFO - ====================
2024-04-29 11:45:23,087 - julearn - INFO -
2024-04-29 11:45:23,088 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-04-29 11:45:23,088 - julearn - INFO - Step added
2024-04-29 11:45:23,088 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2024-04-29 11:45:23,088 - julearn - INFO - Step added
2024-04-29 11:45:23,088 - julearn - INFO - = Model Parameters =
2024-04-29 11:45:23,088 - julearn - INFO - ====================
2024-04-29 11:45:23,088 - julearn - INFO -
2024-04-29 11:45:23,088 - julearn - INFO - = Data Information =
2024-04-29 11:45:23,089 - julearn - INFO -      Problem type: classification
2024-04-29 11:45:23,089 - julearn - INFO -      Number of samples: 80
2024-04-29 11:45:23,089 - julearn - INFO -      Number of features: 3
2024-04-29 11:45:23,089 - julearn - INFO - ====================
2024-04-29 11:45:23,089 - julearn - INFO -
2024-04-29 11:45:23,089 - julearn - INFO -      Number of classes: 2
2024-04-29 11:45:23,089 - julearn - INFO -      Target type: int64
2024-04-29 11:45:23,089 - julearn - INFO -      Class distributions: species
0    50
1    30
Name: count, dtype: int64
2024-04-29 11:45:23,090 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2024-04-29 11:45:23,090 - julearn - INFO - Binary classification problem detected.
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:73: FutureWarning: `fit_params` is deprecated and will be removed in version 1.6. Pass parameters via `params` instead.
  warnings.warn(
0.4