.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/01_model_comparison/plot_simple_model_comparison.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_01_model_comparison_plot_simple_model_comparison.py: Simple Model Comparison ======================= This example uses the ``iris`` dataset and performs binary classifications using different models. At the end, it compares the performance of the models using different scoring functions and performs a statistical test to assess whether the difference in performance is significant. .. include:: ../../links.inc .. GENERATED FROM PYTHON SOURCE LINES 12-21 .. code-block:: Python # Authors: Federico Raimondo # License: AGPL from seaborn import load_dataset from sklearn.model_selection import RepeatedStratifiedKFold from julearn import run_cross_validation from julearn.utils import configure_logging from julearn.stats.corrected_ttest import corrected_ttest .. GENERATED FROM PYTHON SOURCE LINES 22-23 Set the logging level to info to see extra information. .. GENERATED FROM PYTHON SOURCE LINES 23-25 .. code-block:: Python configure_logging(level="INFO") .. rst-class:: sphx-glr-script-out .. code-block:: none 2026-01-16 10:53:56,515 - julearn - INFO - ===== Lib Versions ===== 2026-01-16 10:53:56,515 - julearn - INFO - numpy: 1.26.4 2026-01-16 10:53:56,515 - julearn - INFO - scipy: 1.17.0 2026-01-16 10:53:56,515 - julearn - INFO - sklearn: 1.7.2 2026-01-16 10:53:56,515 - julearn - INFO - pandas: 2.3.3 2026-01-16 10:53:56,515 - julearn - INFO - julearn: 0.3.5.dev123 2026-01-16 10:53:56,515 - julearn - INFO - ======================== .. GENERATED FROM PYTHON SOURCE LINES 26-28 .. code-block:: Python df_iris = load_dataset("iris") .. GENERATED FROM PYTHON SOURCE LINES 29-31 The dataset has three kind of species. We will keep two to perform a binary classification. .. GENERATED FROM PYTHON SOURCE LINES 31-33 .. code-block:: Python df_iris = df_iris[df_iris["species"].isin(["versicolor", "virginica"])] .. GENERATED FROM PYTHON SOURCE LINES 34-36 As features, we will use the sepal length, width and petal length. We will try to predict the species. .. GENERATED FROM PYTHON SOURCE LINES 36-50 .. code-block:: Python X = ["sepal_length", "sepal_width", "petal_length"] y = "species" scores = run_cross_validation( X=X, y=y, data=df_iris, model="svm", problem_type="classification", preprocess="zscore", ) print(scores["test_score"]) .. rst-class:: sphx-glr-script-out .. code-block:: none 2026-01-16 10:53:56,518 - julearn - INFO - ==== Input Data ==== 2026-01-16 10:53:56,518 - julearn - INFO - Using dataframe as input 2026-01-16 10:53:56,518 - julearn - INFO - Features: ['sepal_length', 'sepal_width', 'petal_length'] 2026-01-16 10:53:56,518 - julearn - INFO - Target: species 2026-01-16 10:53:56,518 - julearn - INFO - Expanded features: ['sepal_length', 'sepal_width', 'petal_length'] 2026-01-16 10:53:56,519 - julearn - INFO - X_types:{} 2026-01-16 10:53:56,519 - julearn - WARNING - The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous. /home/runner/work/julearn/julearn/julearn/prepare.py:576: RuntimeWarning: The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous. warn_with_log( 2026-01-16 10:53:56,519 - julearn - INFO - ==================== 2026-01-16 10:53:56,520 - julearn - INFO - 2026-01-16 10:53:56,520 - julearn - INFO - Adding step zscore that applies to ColumnTypes 2026-01-16 10:53:56,520 - julearn - INFO - Step added 2026-01-16 10:53:56,520 - julearn - INFO - Adding step svm that applies to ColumnTypes 2026-01-16 10:53:56,520 - julearn - INFO - Step added 2026-01-16 10:53:56,521 - julearn - INFO - = Model Parameters = 2026-01-16 10:53:56,521 - julearn - INFO - ==================== 2026-01-16 10:53:56,521 - julearn - INFO - 2026-01-16 10:53:56,521 - julearn - INFO - = Data Information = 2026-01-16 10:53:56,521 - julearn - INFO - Problem type: classification 2026-01-16 10:53:56,521 - julearn - INFO - Number of samples: 100 2026-01-16 10:53:56,521 - julearn - INFO - Number of features: 3 2026-01-16 10:53:56,521 - julearn - INFO - ==================== 2026-01-16 10:53:56,521 - julearn - INFO - 2026-01-16 10:53:56,522 - julearn - INFO - Number of classes: 2 2026-01-16 10:53:56,522 - julearn - INFO - Target type: object 2026-01-16 10:53:56,522 - julearn - INFO - Class distributions: species versicolor 50 virginica 50 Name: count, dtype: int64 2026-01-16 10:53:56,523 - julearn - INFO - Using outer CV scheme KFold(n_splits=5, random_state=None, shuffle=False) 2026-01-16 10:53:56,523 - julearn - INFO - Binary classification problem detected. 0 0.90 1 0.75 2 0.95 3 0.70 4 0.90 Name: test_score, dtype: float64 .. GENERATED FROM PYTHON SOURCE LINES 51-55 Additionally, we can choose to assess the performance of the model using different scoring functions. For example, we might have an unbalanced dataset: .. GENERATED FROM PYTHON SOURCE LINES 55-59 .. code-block:: Python df_unbalanced = df_iris[20:] # drop the first 20 versicolor samples print(df_unbalanced["species"].value_counts()) .. rst-class:: sphx-glr-script-out .. code-block:: none species virginica 50 versicolor 30 Name: count, dtype: int64 .. GENERATED FROM PYTHON SOURCE LINES 60-61 So we will choose to use the `balanced_accuracy` and `roc_auc` metrics. .. GENERATED FROM PYTHON SOURCE LINES 61-64 .. code-block:: Python scoring = ["balanced_accuracy", "roc_auc"] .. GENERATED FROM PYTHON SOURCE LINES 65-67 Since we are comparing the performance of different models, we will need to use the same random seed to split the data in the same way. .. GENERATED FROM PYTHON SOURCE LINES 67-70 .. code-block:: Python cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=5, random_state=42) .. GENERATED FROM PYTHON SOURCE LINES 71-72 First we will use a default SVM model. .. GENERATED FROM PYTHON SOURCE LINES 72-85 .. code-block:: Python scores1 = run_cross_validation( X=X, y=y, data=df_unbalanced, model="svm", preprocess="zscore", problem_type="classification", scoring=scoring, cv=cv, ) scores1["model"] = "svm" .. rst-class:: sphx-glr-script-out .. code-block:: none 2026-01-16 10:53:56,576 - julearn - INFO - ==== Input Data ==== 2026-01-16 10:53:56,576 - julearn - INFO - Using dataframe as input 2026-01-16 10:53:56,576 - julearn - INFO - Features: ['sepal_length', 'sepal_width', 'petal_length'] 2026-01-16 10:53:56,576 - julearn - INFO - Target: species 2026-01-16 10:53:56,576 - julearn - INFO - Expanded features: ['sepal_length', 'sepal_width', 'petal_length'] 2026-01-16 10:53:56,576 - julearn - INFO - X_types:{} 2026-01-16 10:53:56,576 - julearn - WARNING - The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous. /home/runner/work/julearn/julearn/julearn/prepare.py:576: RuntimeWarning: The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous. warn_with_log( 2026-01-16 10:53:56,577 - julearn - INFO - ==================== 2026-01-16 10:53:56,577 - julearn - INFO - 2026-01-16 10:53:56,577 - julearn - INFO - Adding step zscore that applies to ColumnTypes 2026-01-16 10:53:56,577 - julearn - INFO - Step added 2026-01-16 10:53:56,578 - julearn - INFO - Adding step svm that applies to ColumnTypes 2026-01-16 10:53:56,578 - julearn - INFO - Step added 2026-01-16 10:53:56,578 - julearn - INFO - = Model Parameters = 2026-01-16 10:53:56,578 - julearn - INFO - ==================== 2026-01-16 10:53:56,578 - julearn - INFO - 2026-01-16 10:53:56,579 - julearn - INFO - = Data Information = 2026-01-16 10:53:56,579 - julearn - INFO - Problem type: classification 2026-01-16 10:53:56,579 - julearn - INFO - Number of samples: 80 2026-01-16 10:53:56,579 - julearn - INFO - Number of features: 3 2026-01-16 10:53:56,579 - julearn - INFO - ==================== 2026-01-16 10:53:56,579 - julearn - INFO - 2026-01-16 10:53:56,579 - julearn - INFO - Number of classes: 2 2026-01-16 10:53:56,579 - julearn - INFO - Target type: object 2026-01-16 10:53:56,580 - julearn - INFO - Class distributions: species virginica 50 versicolor 30 Name: count, dtype: int64 2026-01-16 10:53:56,580 - julearn - INFO - Using outer CV scheme RepeatedStratifiedKFold(n_repeats=5, n_splits=5, random_state=42) 2026-01-16 10:53:56,580 - julearn - INFO - Binary classification problem detected. .. GENERATED FROM PYTHON SOURCE LINES 86-87 Second we will use a default Random Forest model. .. GENERATED FROM PYTHON SOURCE LINES 87-100 .. code-block:: Python scores2 = run_cross_validation( X=X, y=y, data=df_unbalanced, model="rf", preprocess="zscore", problem_type="classification", scoring=scoring, cv=cv, ) scores2["model"] = "rf" .. rst-class:: sphx-glr-script-out .. code-block:: none 2026-01-16 10:53:56,937 - julearn - INFO - ==== Input Data ==== 2026-01-16 10:53:56,938 - julearn - INFO - Using dataframe as input 2026-01-16 10:53:56,938 - julearn - INFO - Features: ['sepal_length', 'sepal_width', 'petal_length'] 2026-01-16 10:53:56,938 - julearn - INFO - Target: species 2026-01-16 10:53:56,938 - julearn - INFO - Expanded features: ['sepal_length', 'sepal_width', 'petal_length'] 2026-01-16 10:53:56,938 - julearn - INFO - X_types:{} 2026-01-16 10:53:56,938 - julearn - WARNING - The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous. /home/runner/work/julearn/julearn/julearn/prepare.py:576: RuntimeWarning: The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous. warn_with_log( 2026-01-16 10:53:56,939 - julearn - INFO - ==================== 2026-01-16 10:53:56,939 - julearn - INFO - 2026-01-16 10:53:56,939 - julearn - INFO - Adding step zscore that applies to ColumnTypes 2026-01-16 10:53:56,939 - julearn - INFO - Step added 2026-01-16 10:53:56,939 - julearn - INFO - Adding step rf that applies to ColumnTypes 2026-01-16 10:53:56,939 - julearn - INFO - Step added 2026-01-16 10:53:56,940 - julearn - INFO - = Model Parameters = 2026-01-16 10:53:56,940 - julearn - INFO - ==================== 2026-01-16 10:53:56,940 - julearn - INFO - 2026-01-16 10:53:56,940 - julearn - INFO - = Data Information = 2026-01-16 10:53:56,940 - julearn - INFO - Problem type: classification 2026-01-16 10:53:56,940 - julearn - INFO - Number of samples: 80 2026-01-16 10:53:56,941 - julearn - INFO - Number of features: 3 2026-01-16 10:53:56,941 - julearn - INFO - ==================== 2026-01-16 10:53:56,941 - julearn - INFO - 2026-01-16 10:53:56,941 - julearn - INFO - Number of classes: 2 2026-01-16 10:53:56,941 - julearn - INFO - Target type: object 2026-01-16 10:53:56,942 - julearn - INFO - Class distributions: species virginica 50 versicolor 30 Name: count, dtype: int64 2026-01-16 10:53:56,942 - julearn - INFO - Using outer CV scheme RepeatedStratifiedKFold(n_repeats=5, n_splits=5, random_state=42) 2026-01-16 10:53:56,942 - julearn - INFO - Binary classification problem detected. .. GENERATED FROM PYTHON SOURCE LINES 101-102 The third model will be a SVM with a linear kernel. .. GENERATED FROM PYTHON SOURCE LINES 102-116 .. code-block:: Python scores3 = run_cross_validation( X=X, y=y, data=df_unbalanced, model="svm", model_params={"svm__kernel": "linear"}, preprocess="zscore", problem_type="classification", scoring=scoring, cv=cv, ) scores3["model"] = "svm_linear" .. rst-class:: sphx-glr-script-out .. code-block:: none 2026-01-16 10:54:00,015 - julearn - INFO - ==== Input Data ==== 2026-01-16 10:54:00,015 - julearn - INFO - Using dataframe as input 2026-01-16 10:54:00,015 - julearn - INFO - Features: ['sepal_length', 'sepal_width', 'petal_length'] 2026-01-16 10:54:00,015 - julearn - INFO - Target: species 2026-01-16 10:54:00,015 - julearn - INFO - Expanded features: ['sepal_length', 'sepal_width', 'petal_length'] 2026-01-16 10:54:00,015 - julearn - INFO - X_types:{} 2026-01-16 10:54:00,015 - julearn - WARNING - The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous. /home/runner/work/julearn/julearn/julearn/prepare.py:576: RuntimeWarning: The following columns are not defined in X_types: ['sepal_length', 'sepal_width', 'petal_length']. They will be treated as continuous. warn_with_log( 2026-01-16 10:54:00,016 - julearn - INFO - ==================== 2026-01-16 10:54:00,016 - julearn - INFO - 2026-01-16 10:54:00,016 - julearn - INFO - Adding step zscore that applies to ColumnTypes 2026-01-16 10:54:00,017 - julearn - INFO - Step added 2026-01-16 10:54:00,017 - julearn - INFO - Adding step svm that applies to ColumnTypes 2026-01-16 10:54:00,017 - julearn - INFO - Setting hyperparameter kernel = linear 2026-01-16 10:54:00,017 - julearn - INFO - Step added 2026-01-16 10:54:00,018 - julearn - INFO - = Model Parameters = 2026-01-16 10:54:00,018 - julearn - INFO - ==================== 2026-01-16 10:54:00,018 - julearn - INFO - 2026-01-16 10:54:00,018 - julearn - INFO - = Data Information = 2026-01-16 10:54:00,018 - julearn - INFO - Problem type: classification 2026-01-16 10:54:00,018 - julearn - INFO - Number of samples: 80 2026-01-16 10:54:00,018 - julearn - INFO - Number of features: 3 2026-01-16 10:54:00,018 - julearn - INFO - ==================== 2026-01-16 10:54:00,018 - julearn - INFO - 2026-01-16 10:54:00,019 - julearn - INFO - Number of classes: 2 2026-01-16 10:54:00,019 - julearn - INFO - Target type: object 2026-01-16 10:54:00,019 - julearn - INFO - Class distributions: species virginica 50 versicolor 30 Name: count, dtype: int64 2026-01-16 10:54:00,020 - julearn - INFO - Using outer CV scheme RepeatedStratifiedKFold(n_repeats=5, n_splits=5, random_state=42) 2026-01-16 10:54:00,020 - julearn - INFO - Binary classification problem detected. .. GENERATED FROM PYTHON SOURCE LINES 117-118 We can now compare the performance of the models using corrected statistics. .. GENERATED FROM PYTHON SOURCE LINES 118-122 .. code-block:: Python stats_df = corrected_ttest(scores1, scores2, scores3) print(stats_df) .. rst-class:: sphx-glr-script-out .. code-block:: none metric t-stat ... model_2 p-val-corrected 0 test_balanced_accuracy -0.175075 ... rf 1.000000 2 test_balanced_accuracy -1.062567 ... svm_linear 0.895662 4 test_balanced_accuracy -1.151390 ... svm_linear 0.782741 1 test_roc_auc 1.108944 ... rf 0.835331 3 test_roc_auc -1.236153 ... svm_linear 0.685092 5 test_roc_auc -1.669010 ... svm_linear 0.324331 [6 rows x 6 columns] .. GENERATED FROM PYTHON SOURCE LINES 123-126 .. rst-class:: hidden This block is hidden in the documentation. This files are used to generate the plots in the documentation. (not working for now) .. GENERATED FROM PYTHON SOURCE LINES 126-128 .. code-block:: Python :dedent: 1 .. GENERATED FROM PYTHON SOURCE LINES 137-139 We can also plot the performance of the models using the ``julearn`` Score Viewer. .. GENERATED FROM PYTHON SOURCE LINES 139-148 .. code-block:: Python from julearn.viz import plot_scores panel = plot_scores(scores1, scores2, scores3) # panel.show() # uncomment the previous line show the plot # read the documentation for more information # https://panel.holoviz.org/getting_started/build_app.html#deploying-panels .. GENERATED FROM PYTHON SOURCE LINES 149-167 This is how the plot looks like. .. note:: The plot is interactive. You can zoom in and out, and hover over. However, buttons will not work in this documentation. .. bokeh-plot:: :source-position: none from julearn.viz import plot_scores from bokeh.io import output_notebook, show import pandas as pd output_notebook() scores1 = pd.read_csv("/tmp/scores1.csv") scores2 = pd.read_csv("/tmp/scores2.csv") scores3 = pd.read_csv("/tmp/scores3.csv") panel = plot_scores(scores1, scores2, scores3, width=600) show(panel.get_root()) .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 3.994 seconds) .. _sphx_glr_download_auto_examples_01_model_comparison_plot_simple_model_comparison.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_simple_model_comparison.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_simple_model_comparison.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_simple_model_comparison.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_