Note

Go to the end to download the full example code

Simple Binary Classification#

This example uses the ‘iris’ dataset and performs a simple binary classification using a Support Vector Machine classifier.

# Authors: Federico Raimondo <f.raimondo@fz-juelich.de>
#
# License: AGPL
from seaborn import load_dataset
from julearn import run_cross_validation
from julearn.utils import configure_logging

Set the logging level to info to see extra information

configure_logging(level="INFO")

2023-07-19 12:41:44,542 - julearn - INFO - ===== Lib Versions =====
2023-07-19 12:41:44,542 - julearn - INFO - numpy: 1.25.1
2023-07-19 12:41:44,542 - julearn - INFO - scipy: 1.11.1
2023-07-19 12:41:44,542 - julearn - INFO - sklearn: 1.3.0
2023-07-19 12:41:44,542 - julearn - INFO - pandas: 2.0.3
2023-07-19 12:41:44,542 - julearn - INFO - julearn: 0.3.1.dev1
2023-07-19 12:41:44,542 - julearn - INFO - ========================

df_iris = load_dataset("iris")

The dataset has three kind of species. We will keep two to perform a binary classification.

df_iris = df_iris[df_iris["species"].isin(["versicolor", "virginica"])]

As features, we will use the sepal length, width and petal length. We will try to predict the species.

X = ["sepal_length", "sepal_width", "petal_length"]
y = "species"
scores = run_cross_validation(
    X=X,
    y=y,
    data=df_iris,
    model="svm",
    problem_type="classification",
    preprocess="zscore",
)

print(scores["test_score"])

2023-07-19 12:41:44,545 - julearn - INFO - ==== Input Data ====
2023-07-19 12:41:44,545 - julearn - INFO - Using dataframe as input
2023-07-19 12:41:44,545 - julearn - INFO -      Features: ['sepal_length', 'sepal_width', 'petal_length']
2023-07-19 12:41:44,545 - julearn - INFO -      Target: species
2023-07-19 12:41:44,546 - julearn - INFO -      Expanded features: ['sepal_length', 'sepal_width', 'petal_length']
2023-07-19 12:41:44,546 - julearn - INFO -      X_types:{}
2023-07-19 12:41:44,546 - julearn - WARNING - The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/utils/logging.py:238: RuntimeWarning: The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous.
  warn(msg, category=category)
2023-07-19 12:41:44,547 - julearn - INFO - ====================
2023-07-19 12:41:44,547 - julearn - INFO -
2023-07-19 12:41:44,547 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:41:44,547 - julearn - INFO - Step added
2023-07-19 12:41:44,547 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:41:44,547 - julearn - INFO - Step added
2023-07-19 12:41:44,548 - julearn - INFO - = Model Parameters =
2023-07-19 12:41:44,548 - julearn - INFO - ====================
2023-07-19 12:41:44,548 - julearn - INFO -
2023-07-19 12:41:44,548 - julearn - INFO - = Data Information =
2023-07-19 12:41:44,549 - julearn - INFO -      Problem type: classification
2023-07-19 12:41:44,549 - julearn - INFO -      Number of samples: 100
2023-07-19 12:41:44,549 - julearn - INFO -      Number of features: 3
2023-07-19 12:41:44,549 - julearn - INFO - ====================
2023-07-19 12:41:44,549 - julearn - INFO -
2023-07-19 12:41:44,549 - julearn - INFO -      Number of classes: 2
2023-07-19 12:41:44,549 - julearn - INFO -      Target type: object
2023-07-19 12:41:44,550 - julearn - INFO -      Class distributions: species
versicolor    50
virginica     50
Name: count, dtype: int64
2023-07-19 12:41:44,550 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2023-07-19 12:41:44,550 - julearn - INFO - Binary classification problem detected.
0    0.90
1    0.75
2    0.95
3    0.70
4    0.90
Name: test_score, dtype: float64

Additionally, we can choose to assess the performance of the model using different scoring functions.

For example, we might have an unbalanced dataset:

df_unbalanced = df_iris[20:]  # drop the first 20 versicolor samples
print(df_unbalanced["species"].value_counts())

species
virginica     50
versicolor    30
Name: count, dtype: int64

If we compute the accuracy, we might not account for this imbalance. A more suitable metric is the balanced_accuracy. More information in scikit-learn: balanced_accuracy_score().

We will also set the random seed so we always split the data in the same way.

scores = run_cross_validation(
    X=X,
    y=y,
    data=df_unbalanced,
    model="svm",
    seed=42,
    preprocess="zscore",
    problem_type="classification",
    scoring=["accuracy", "balanced_accuracy"],
)

print(scores["test_accuracy"].mean())
print(scores["test_balanced_accuracy"].mean())

2023-07-19 12:41:44,602 - julearn - INFO - Setting random seed to 42
2023-07-19 12:41:44,602 - julearn - INFO - ==== Input Data ====
2023-07-19 12:41:44,602 - julearn - INFO - Using dataframe as input
2023-07-19 12:41:44,602 - julearn - INFO -      Features: ['sepal_length', 'sepal_width', 'petal_length']
2023-07-19 12:41:44,602 - julearn - INFO -      Target: species
2023-07-19 12:41:44,602 - julearn - INFO -      Expanded features: ['sepal_length', 'sepal_width', 'petal_length']
2023-07-19 12:41:44,602 - julearn - INFO -      X_types:{}
2023-07-19 12:41:44,602 - julearn - WARNING - The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/utils/logging.py:238: RuntimeWarning: The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous.
  warn(msg, category=category)
2023-07-19 12:41:44,603 - julearn - INFO - ====================
2023-07-19 12:41:44,603 - julearn - INFO -
2023-07-19 12:41:44,603 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:41:44,603 - julearn - INFO - Step added
2023-07-19 12:41:44,603 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:41:44,603 - julearn - INFO - Step added
2023-07-19 12:41:44,604 - julearn - INFO - = Model Parameters =
2023-07-19 12:41:44,604 - julearn - INFO - ====================
2023-07-19 12:41:44,604 - julearn - INFO -
2023-07-19 12:41:44,604 - julearn - INFO - = Data Information =
2023-07-19 12:41:44,604 - julearn - INFO -      Problem type: classification
2023-07-19 12:41:44,604 - julearn - INFO -      Number of samples: 80
2023-07-19 12:41:44,604 - julearn - INFO -      Number of features: 3
2023-07-19 12:41:44,604 - julearn - INFO - ====================
2023-07-19 12:41:44,604 - julearn - INFO -
2023-07-19 12:41:44,604 - julearn - INFO -      Number of classes: 2
2023-07-19 12:41:44,605 - julearn - INFO -      Target type: object
2023-07-19 12:41:44,605 - julearn - INFO -      Class distributions: species
virginica     50
versicolor    30
Name: count, dtype: int64
2023-07-19 12:41:44,605 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2023-07-19 12:41:44,606 - julearn - INFO - Binary classification problem detected.
/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/sklearn/metrics/_classification.py:2394: UserWarning: y_pred contains classes not in y_true
  warnings.warn("y_pred contains classes not in y_true")
/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/sklearn/metrics/_classification.py:2394: UserWarning: y_pred contains classes not in y_true
  warnings.warn("y_pred contains classes not in y_true")
/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/sklearn/metrics/_classification.py:2394: UserWarning: y_pred contains classes not in y_true
  warnings.warn("y_pred contains classes not in y_true")
/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/sklearn/metrics/_classification.py:2394: UserWarning: y_pred contains classes not in y_true
  warnings.warn("y_pred contains classes not in y_true")
0.8625
0.8678571428571429

Other kind of metrics allows us to evaluate how good our model is to detect specific targets. Suppose we want to create a model that correctly identifies the versicolor samples.

Now we might want to evaluate the precision score, or the ratio of true positives (tp) over all positives (true and false positives). More information in scikit-learn: precision_score().

For this metric to work, we need to define which are our positive values. In this example, we are interested in detecting versicolor.

precision_scores = run_cross_validation(
    X=X,
    y=y,
    data=df_unbalanced,
    model="svm",
    preprocess="zscore",
    problem_type="classification",
    seed=42,
    scoring="precision",
    pos_labels="versicolor",
)
print(precision_scores["test_score"].mean())

2023-07-19 12:41:44,660 - julearn - INFO - Setting random seed to 42
2023-07-19 12:41:44,661 - julearn - INFO - ==== Input Data ====
2023-07-19 12:41:44,661 - julearn - INFO - Using dataframe as input
2023-07-19 12:41:44,661 - julearn - INFO -      Features: ['sepal_length', 'sepal_width', 'petal_length']
2023-07-19 12:41:44,661 - julearn - INFO -      Target: species
2023-07-19 12:41:44,661 - julearn - INFO -      Expanded features: ['sepal_length', 'sepal_width', 'petal_length']
2023-07-19 12:41:44,661 - julearn - INFO -      X_types:{}
2023-07-19 12:41:44,661 - julearn - WARNING - The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous.
/home/runner/work/julearn/julearn/julearn/utils/logging.py:238: RuntimeWarning: The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous.
  warn(msg, category=category)
2023-07-19 12:41:44,662 - julearn - INFO - Setting the following as positive labels ['versicolor']
2023-07-19 12:41:44,662 - julearn - INFO - ====================
2023-07-19 12:41:44,662 - julearn - INFO -
2023-07-19 12:41:44,662 - julearn - INFO - Adding step zscore that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:41:44,662 - julearn - INFO - Step added
2023-07-19 12:41:44,663 - julearn - INFO - Adding step svm that applies to ColumnTypes<types={'continuous'}; pattern=(?:__:type:__continuous)>
2023-07-19 12:41:44,663 - julearn - INFO - Step added
2023-07-19 12:41:44,663 - julearn - INFO - = Model Parameters =
2023-07-19 12:41:44,663 - julearn - INFO - ====================
2023-07-19 12:41:44,663 - julearn - INFO -
2023-07-19 12:41:44,663 - julearn - INFO - = Data Information =
2023-07-19 12:41:44,663 - julearn - INFO -      Problem type: classification
2023-07-19 12:41:44,663 - julearn - INFO -      Number of samples: 80
2023-07-19 12:41:44,663 - julearn - INFO -      Number of features: 3
2023-07-19 12:41:44,664 - julearn - INFO - ====================
2023-07-19 12:41:44,664 - julearn - INFO -
2023-07-19 12:41:44,664 - julearn - INFO -      Number of classes: 2
2023-07-19 12:41:44,664 - julearn - INFO -      Target type: int64
2023-07-19 12:41:44,664 - julearn - INFO -      Class distributions: species
0    50
1    30
Name: count, dtype: int64
2023-07-19 12:41:44,665 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False)
2023-07-19 12:41:44,665 - julearn - INFO - Binary classification problem detected.
0.4

Total running time of the script: ( 0 minutes 0.182 seconds)

Gallery generated by Sphinx-Gallery