Benchmarking - comparing estimator performance#
benchmarking modules allows you to easily orchestrate benchmarking experiments in which you want to compare the performance of one or more algorithms over one or more datasets and benchmark configurations. This module is still in development.
Benchmarking as an endevour in general is very easy to get wrong, giving false conclusions about estimator performance - see this 2022 research from Princeton for numerous examples of such mistakes in peer reviewed academic papers as evidence of this.
benchmarking module is designed to provide benchmarking functionality while enforcing best practices and structure to help users avoid making mistakes (such as data leakage, etc.) which invalidate their results. The
benchmarking module is designed for easy usage in mind, as such it interfaces directly with
aeon objects and classes. Previously developed estimator should be usable as they are without alterations.
We also include tools for comparing your results to published work and for testing and visualising relative performance of algorithms. See
This notebook demonstrates usage of the
import warnings warnings.filterwarnings("ignore") from aeon.benchmarking.forecasting import ForecastingBenchmark from aeon.datasets import load_airline from aeon.forecasting.model_selection import ExpandingWindowSplitter from aeon.forecasting.naive import NaiveForecaster from aeon.performance_metrics.forecasting import MeanSquaredPercentageError
Instantiate an instance of a benchmark class#
In this example we are comparing forecasting estimators.
benchmark = ForecastingBenchmark()
Add competing estimators#
We add different competing estimators to the benchmark instance. All added estimators will be automatically ran through each added benchmark tasks, and their results compiled.
benchmark.add_estimator( estimator=NaiveForecaster(strategy="mean", sp=12), estimator_id="NaiveForecaster-mean-v1", ) benchmark.add_estimator( estimator=NaiveForecaster(strategy="last", sp=12), estimator_id="NaiveForecaster-last-v1", )
Add benchmarking tasks#
These are the prediction/validation tasks over which every estimator will be tested and their results compiled.
The exact arguments for a benchmarking task depend on the whether the objective is forecasting, classification, etc., but generally they are similar. The following are the required arguments for defining a forecasting benchmark task.
Specify cross-validation split regime(s)#
Define cross-validation split regimes, using standard
cv_splitter = ExpandingWindowSplitter( initial_window=24, step_length=12, fh=12, )
Specify performance metric(s)#
Define performance metrics on which to compare estimators, using standard
scorers = [MeanSquaredPercentageError()]
Specify dataset loaders#
Define dataset loaders, which are callables (functions) which should return a dataset. Generally this is a callable which returns a dataframe containing the entire dataset. One can use the
aeon defined datasets, or define their own. Something as simple as the following example will suffice:
def my_dataset_loader(): return pd.read_csv("path/to/data.csv")
The datasets will be loaded when running the benchmarking tasks, ran through the cross-validation regime(s) and subsequently the estimators will be tested over the dataset splits.
dataset_loaders = [load_airline]
Add tasks to the benchmark instance#
Use the previously defined objects to add tasks to the benchmark instance. Optionally use loops etc. to easily setup multiple benchmark tasks reusing arguments.
for dataset_loader in dataset_loaders: benchmark.add_task( dataset_loader, cv_splitter, scorers, )
Run all task-estimator combinations and store results#
run won’t rerun tasks it already has results for, so adding a new estimator and running
run again will only run tasks for that new estimator.
results_df = benchmark.run("./forecasting_results.csv") results_df.T