In [1]:
from sklearn_benchmarks.reporting.hp_match import HpMatchReporting
from sklearn_benchmarks.utils import default_results_directory
from pathlib import Path
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

ONNX Runtime vs. scikit-learn¶

In [2]:
results_dir = default_results_directory()
In [3]:
# Parameters
results_dir = "./results/local/20220315T132808/"
In [4]:
results_dir = Path(results_dir)
In [5]:
reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True, results_dir=results_dir)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speed-up time scikit-learn / time onnx. For instance, a speed-up of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier (brute force) ¶

onnx (1.11.0) vs. scikit-learn (1.0.2)

Speedup barplots ¶

All estimators share the following parameters: algorithm=brute.

Raw results ¶

predict

function n_samples_train n_samples n_features mean_duration_sklearn std_duration_sklearn iteration_throughput latency n_jobs n_neighbors accuracy_score_sklearn mean_duration_onnx std_duration_onnx accuracy_score_onnx speedup std_speedup sklearn_profiling onnx_profiling
0 predict 100000 1000 100 2.471 0.234 0.000 0.002 -1 1 0.676 15.118 0.008 0.676 0.163 0.163 Download Download
1 predict 100000 1 100 0.023 0.003 0.000 0.023 -1 1 0.000 0.292 0.004 0.000 0.079 0.079 Download Download
2 predict 100000 1000 100 3.125 0.066 0.000 0.003 -1 5 0.743 15.057 0.036 0.743 0.208 0.208 Download Download
3 predict 100000 1 100 0.024 0.003 0.000 0.024 -1 5 1.000 0.294 0.002 1.000 0.083 0.083 Download Download
4 predict 100000 1000 100 1.939 0.006 0.000 0.002 1 100 0.846 15.218 0.064 0.846 0.127 0.127 Download Download
5 predict 100000 1 100 0.021 0.000 0.000 0.021 1 100 1.000 0.297 0.009 1.000 0.069 0.069 Download Download
6 predict 100000 1000 100 2.830 0.049 0.000 0.003 -1 100 0.846 15.203 0.038 0.846 0.186 0.186 Download Download
7 predict 100000 1 100 0.025 0.004 0.000 0.025 -1 100 1.000 0.293 0.002 1.000 0.086 0.086 Download Download
8 predict 100000 1000 100 1.927 0.006 0.000 0.002 1 5 0.743 15.268 0.092 0.743 0.126 0.126 Download Download
9 predict 100000 1 100 0.021 0.000 0.000 0.021 1 5 1.000 0.295 0.002 1.000 0.070 0.070 Download Download
10 predict 100000 1000 100 1.227 0.004 0.001 0.001 1 1 0.676 15.275 0.020 0.676 0.080 0.080 Download Download
11 predict 100000 1 100 0.023 0.006 0.000 0.023 1 1 0.000 0.295 0.002 0.000 0.077 0.077 Download Download
12 predict 100000 1000 2 1.901 0.022 0.000 0.002 -1 1 0.845 3.713 0.013 0.845 0.512 0.512 Download Download
13 predict 100000 1 2 0.006 0.004 0.000 0.006 -1 1 1.000 0.214 0.002 1.000 0.030 0.030 Download Download
14 predict 100000 1000 2 2.698 0.038 0.000 0.003 -1 5 0.883 3.702 0.012 0.883 0.729 0.729 Download Download
15 predict 100000 1 2 0.005 0.000 0.000 0.005 -1 5 1.000 0.213 0.002 1.000 0.025 0.025 Download Download
16 predict 100000 1000 2 1.905 0.011 0.000 0.002 1 100 0.887 3.727 0.021 0.887 0.511 0.511 Download Download
17 predict 100000 1 2 0.003 0.000 0.000 0.003 1 100 1.000 0.214 0.002 1.000 0.012 0.012 Download Download
18 predict 100000 1000 2 2.660 0.025 0.000 0.003 -1 100 0.887 3.740 0.007 0.887 0.711 0.711 Download Download
19 predict 100000 1 2 0.007 0.004 0.000 0.007 -1 100 1.000 0.214 0.002 1.000 0.031 0.031 Download Download
20 predict 100000 1000 2 1.887 0.006 0.000 0.002 1 5 0.883 3.697 0.009 0.883 0.510 0.510 Download Download
21 predict 100000 1 2 0.003 0.000 0.000 0.003 1 5 1.000 0.214 0.002 1.000 0.013 0.013 Download Download
22 predict 100000 1000 2 1.125 0.007 0.000 0.001 1 1 0.845 3.753 0.016 0.845 0.300 0.300 Download Download
23 predict 100000 1 2 0.002 0.000 0.000 0.002 1 1 1.000 0.213 0.001 1.000 0.009 0.009 Download Download

Profiling traces can be visualized using Perfetto UI.

HistGradientBoostingClassifier ¶

onnx (1.11.0) vs. scikit-learn (1.0.2)

Speedup barplots ¶

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

Raw results ¶

predict

function n_samples_train n_samples n_features mean_duration_sklearn std_duration_sklearn iteration_throughput latency accuracy_score_sklearn mean_duration_onnx std_duration_onnx accuracy_score_onnx speedup std_speedup sklearn_profiling onnx_profiling
0 predict 100000 1000 100 0.115 0.002 0.007 0.0 0.795 0.52 0.006 0.795 0.222 0.222 Download Download

Profiling traces can be visualized using Perfetto UI.

Benchmark environment information¶

System¶
python 3.8.12 | packaged by conda-forge | (default, Jan 30 2022, 23:42:07) [GCC 9.4.0]
executable /usr/share/miniconda/envs/sklbench/bin/python
machine Linux-5.11.0-1028-azure-x86_64-with-glibc2.10

Dependencies¶

version
pip 22.0.4
setuptools 60.9.3
sklearn 1.0.2
numpy 1.22.3
scipy 1.8.0
Cython None
pandas 1.4.1
matplotlib 3.5.1
joblib 1.1.0
threadpoolctl 3.1.0

Threadpool¶

user_api internal_api prefix filepath version threading_layer architecture num_threads
0 blas openblas libopenblas /usr/share/miniconda/envs/sklbench/lib/libopenblasp-r0.3.18.so 0.3.18 pthreads SkylakeX 2
1 openmp openmp libgomp /usr/share/miniconda/envs/sklbench/lib/libgomp.so.1.0.0 None NaN NaN 2

CPU count¶
cpu_count 2
physical_cpu_count 2