In [1]:
from sklearn_benchmarks.reporting.hpo import HPOReporting
from sklearn_benchmarks.utils import default_run_dir, default_report_config
from pathlib import Path
import pandas as pd
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

Gradient boosting libraries: randomized hyperparameter optimization (HPO) benchmarks

In [2]:
run_dir = default_run_dir()
report_config = default_report_config("gradient_boosting")
In [3]:
# Parameters
report_config = {
    "estimators": [
        {
            "name": "sklearn_HistGradientBoostingClassifier",
            "lib": "sklearn",
            "legend": "scikit-learn",
            "color": "blue",
        },
        {"name": "lightgbm", "lib": "lightgbm", "color": "red"},
        {"name": "xgboost", "lib": "xgboost", "color": "green"},
        {
            "name": "catboost_lossguide",
            "lib": "catboost",
            "legend": "catboost lossguide",
            "color": "purple",
        },
        {
            "name": "catboost_symmetric",
            "lib": "catboost",
            "legend": "catboost symmetric",
            "color": "orange",
        },
    ],
    "config": "config.yml",
}
run_dir = "./results/local/20220316T125621/"
In [4]:
run_dir = Path(run_dir)
In [5]:
reporting_hpo = HPOReporting(**report_config, run_dir=run_dir)
reporting_hpo.make_report()

Here, we assume there is no perfect match of hyperparameters between librairies. We perform a randomized hyperparameters search with a fixed time budget for each library.

Smoothed HPO Curves

We boostrap 10 000 times the hyperparameters optimization data points represented on the plot above. Then we compute the average cumulated time to reach a specific validation score by taking the mean across the bootstrapped samples. The shaded areas represent the quartiles of the bootstrap samples. The fastest libraries are therefore the closest to the upper left corner. The specification of the HP grid can be found in the configuration file.

Speedup barplot

The speed-up threshold is found by taking 95% of the best score of the worst performing library.

/home/runner/work/sklearn_benchmarks/sklearn_benchmarks/sklearn_benchmarks/reporting/hpo.py:480: FutureWarning:

The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

/home/runner/work/sklearn_benchmarks/sklearn_benchmarks/sklearn_benchmarks/reporting/hpo.py:480: FutureWarning:

The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

/home/runner/work/sklearn_benchmarks/sklearn_benchmarks/sklearn_benchmarks/reporting/hpo.py:480: FutureWarning:

The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

/home/runner/work/sklearn_benchmarks/sklearn_benchmarks/sklearn_benchmarks/reporting/hpo.py:480: FutureWarning:

The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

/home/runner/work/sklearn_benchmarks/sklearn_benchmarks/sklearn_benchmarks/reporting/hpo.py:480: FutureWarning:

The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

Speedup curves

Benchmark environment information

System
python 3.8.12 | packaged by conda-forge | (default, J...
executable /usr/share/miniconda/envs/sklbench/bin/python
machine Linux-5.11.0-1028-azure-x86_64-with-glibc2.10

Dependencies
pip 22.0.4
setuptools 60.9.3
sklearn 1.0.2
numpy 1.22.3
scipy 1.8.0
Cython None
pandas 1.4.1
matplotlib 3.5.1
joblib 1.1.0
threadpoolctl 3.1.0

Threadpool

user_api internal_api prefix filepath version threading_layer architecture num_threads
0 blas openblas libopenblas /usr/share/miniconda/envs/sklbench/lib/libopen... 0.3.18 pthreads SkylakeX 2
1 openmp openmp libgomp /usr/share/miniconda/envs/sklbench/lib/libgomp... None NaN NaN 2

Cpu_count
cpu_count 2
physical_cpu_count 2