MeanRelativeAbsoluteError

class MeanRelativeAbsoluteError(multioutput='uniform_average', multilevel='uniform_average')

Mean relative absolute error (MRAE).

In relative error metrics, relative errors are first calculated by scaling (dividing) the individual forecast errors by the error calculated using a benchmark method at the same index position. If the error of the benchmark method is zero then a large value is returned.

MRAE applies mean absolute error (MAE) to the resulting relative errors.

Parameters:

multioutput{‘raw_values’, ‘uniform_average’} or array-like of shape (n_outputs,), default=’uniform_average’

Defines how to aggregate metric for multivariate (multioutput) data. If array-like, values used as weights to average the errors. If ‘raw_values’, returns a full set of errors in case of multioutput input. If ‘uniform_average’, errors of all outputs are averaged with uniform weight.

See also

MedianRelativeAbsoluteError

GeometricMeanRelativeAbsoluteError

GeometricMeanRelativeSquaredError

References

Hyndman, R. J and Koehler, A. B. (2006). “Another look at measures of forecast accuracy”, International Journal of Forecasting, Volume 22, Issue 4.

Examples

>>> import numpy as np
>>> from aeon.performance_metrics.forecasting import MeanRelativeAbsoluteError
>>> y_true = np.array([3, -0.5, 2, 7, 2])
>>> y_pred = np.array([2.5, 0.0, 2, 8, 1.25])
>>> y_pred_benchmark = y_pred*1.1
>>> mrae = MeanRelativeAbsoluteError()
>>> mrae(y_true, y_pred, y_pred_benchmark=y_pred_benchmark)
0.9511111111111111
>>> y_true = np.array([[0.5, 1], [-1, 1], [7, -6]])
>>> y_pred = np.array([[0, 2], [-1, 2], [8, -5]])
>>> y_pred_benchmark = y_pred*1.1
>>> mrae(y_true, y_pred, y_pred_benchmark=y_pred_benchmark)
0.8703703703703702
>>> mrae = MeanRelativeAbsoluteError(multioutput='raw_values')
>>> mrae(y_true, y_pred, y_pred_benchmark=y_pred_benchmark)
array([0.51851852, 1.22222222])
>>> mrae = MeanRelativeAbsoluteError(multioutput=[0.3, 0.7])
>>> mrae(y_true, y_pred, y_pred_benchmark=y_pred_benchmark)
1.0111111111111108

Methods

__call__(y_true, y_pred, **kwargs)	Calculate metric value using underlying metric function.
clone()	Obtain a clone of the object with same hyper-parameters.
clone_tags(estimator[, tag_names])	Clone/mirror tags from another estimator as dynamic override.
create_test_instance([parameter_set])	Construct Estimator instance if possible.
create_test_instances_and_names([parameter_set])	Create list of all test instances and a list of names for them.
evaluate(y_true, y_pred, **kwargs)	Evaluate the desired metric on given inputs.
evaluate_by_index(y_true, y_pred, **kwargs)	Return the metric evaluated at each time point.
func(y_pred[, horizon_weight, multioutput])	Mean relative absolute error (MRAE).
get_class_tag(tag_name[, tag_value_default])	Get tag value from estimator class (only class tags).
get_class_tags()	Get class tags from estimator class and all its parent classes.
get_param_defaults()	Get parameter defaults for the object.
get_param_names()	Get parameter names for the object.
get_params([deep])	Get parameters for this estimator.
get_tag(tag_name[, tag_value_default, ...])	Get tag value from estimator class.
get_tags()	Get tags from estimator class.
get_test_params([parameter_set])	Return testing parameter settings for the estimator.
is_composite()	Check if the object is composite.
load_from_path(serial)	Load object from file location.
load_from_serial(serial)	Load object from serialized memory container.
reset()	Reset the object to a clean post-init state.
save([path])	Save serialized self to bytes-like object or to (.zip) file.
set_params(**params)	Set the parameters of this object.
set_tags(**tag_dict)	Set dynamic tags to given values.

func(y_pred, horizon_weight=None, multioutput='uniform_average', **kwargs)

Mean relative absolute error (MRAE).

In relative error metrics, relative errors are first calculated by scaling (dividing) the individual forecast errors by the error calculated using a benchmark method at the same index position. If the error of the benchmark method is zero then a large value is returned.

MRAE applies mean absolute error (MAE) to the resulting relative errors.

Parameters:

y_truepd.Series, pd.DataFrame or np.array of shape (fh,) or (fh, n_outputs) where fh is the forecasting horizon

Ground truth (correct) target values.

y_predpd.Series, pd.DataFrame or np.array of shape (fh,) or (fh, n_outputs) where fh is the forecasting horizon

Forecasted values.

y_pred_benchmarkpd.Series, pd.DataFrame or np.array of shape (fh,) or (fh, n_outputs) where fh is the forecasting horizon, default=None

Forecasted values from benchmark method.

horizon_weightarray-like of shape (fh,), default=None

Forecast horizon weights.

multioutput{‘raw_values’, ‘uniform_average’} or array-like of shape (n_outputs,), default=’uniform_average’

Defines how to aggregate metric for multivariate (multioutput) data. If array-like, values used as weights to average the errors. If ‘raw_values’, returns a full set of errors in case of multioutput input. If ‘uniform_average’, errors of all outputs are averaged with uniform weight.

Returns:

lossfloat

MRAE loss. If multioutput is ‘raw_values’, then MRAE is returned for each output separately. If multioutput is ‘uniform_average’ or an ndarray of weights, then the weighted average MRAE of all output errors is returned.

See also

median_relative_absolute_error

geometric_mean_relative_absolute_error

geometric_mean_relative_squared_error

References

Hyndman, R. J and Koehler, A. B. (2006). “Another look at measures of forecast accuracy”, International Journal of Forecasting, Volume 22, Issue 4.

Examples

>>> from aeon.performance_metrics.forecasting import mean_relative_absolute_error
>>> y_true = np.array([3, -0.5, 2, 7, 2])
>>> y_pred = np.array([2.5, 0.0, 2, 8, 1.25])
>>> y_pred_benchmark = y_pred*1.1
>>> mean_relative_absolute_error(y_true, y_pred,     y_pred_benchmark=y_pred_benchmark)
0.9511111111111111
>>> y_true = np.array([[0.5, 1], [-1, 1], [7, -6]])
>>> y_pred = np.array([[0, 2], [-1, 2], [8, -5]])
>>> y_pred_benchmark = y_pred*1.1
>>> mean_relative_absolute_error(y_true, y_pred,     y_pred_benchmark=y_pred_benchmark)
0.8703703703703702
>>> mean_relative_absolute_error(y_true, y_pred,     y_pred_benchmark=y_pred_benchmark, multioutput='raw_values')
array([0.51851852, 1.22222222])
>>> mean_relative_absolute_error(y_true, y_pred,     y_pred_benchmark=y_pred_benchmark, multioutput=[0.3, 0.7])
1.0111111111111108

__call__(y_true, y_pred, **kwargs)

Calculate metric value using underlying metric function.

Parameters:

y_truetime series in aeon compatible data container format

Ground truth (correct) target values y can be in one of the following formats: Series scitype: pd.Series, pd.DataFrame, or np.ndarray (1D or 2D) Panel scitype: pd.DataFrame with 2-level row MultiIndex,

3D np.ndarray, list of Series pd.DataFrame, or nested pd.DataFrame

Hierarchical scitype: pd.DataFrame with 3 or more level row MultiIndex

y_pred :time series in aeon compatible data container format

Forecasted values to evaluate must be of same format as y_true, same indices and columns if indexed

Returns:

lossfloat, np.ndarray, or pd.DataFrame

Calculated metric, averaged or by variable. float if self.multioutput=”uniform_average” or array-like

and self.multilevel=”uniform_average” or “uniform_average_time” value is metric averaged over variables and levels (see class docstring)

np.ndarray of shape (y_true.columns,) if self.multioutput=”raw_values”

and self.multilevel=”uniform_average” or “uniform_average_time” i-th entry is metric calculated for i-th variable

pd.DataFrame if self.multilevel=raw.values

of shape (n_levels, ) if self.multioutput = “uniform_average” or array of shape (n_levels, y_true.columns) if self.multioutput=”raw_values” metric is applied per level, row averaging (yes/no) as in multioutput

clone()

Obtain a clone of the object with same hyper-parameters.

A clone is a different object without shared references, in post-init state. This function is equivalent to returning sklearn.clone of self. Equal in value to type(self)(**self.get_params(deep=False)).

Returns:

instance of type(self), clone of self (see above)

clone_tags(estimator, tag_names=None)

Clone/mirror tags from another estimator as dynamic override.

Parameters:

estimatorobject

Estimator inheriting from :class:BaseEstimator.

tag_namesstr or list of str, default = None

Names of tags to clone. If None then all tags in estimator are used as tag_names.

Returns:

Self

Reference to self.

Notes

Changes object state by setting tag values in tag_set from estimator as dynamic tags in self.

classmethod create_test_instance(parameter_set='default')

Construct Estimator instance if possible.

Parameters:

parameter_setstr, default=”default”

Name of the set of test parameters to return, for use in tests. If no special parameters are defined for a value, will return “default” set.

Returns:

instanceinstance of the class with default parameters.

Notes

get_test_params can return dict or list of dict. This function takes first or single dict that get_test_params returns, and constructs the object with that.

classmethod create_test_instances_and_names(parameter_set='default')

Create list of all test instances and a list of names for them.

Parameters:

parameter_setstr, default=”default”

Name of the set of test parameters to return, for use in tests. If no special parameters are defined for a value, will return “default” set.

Returns:

objslist of instances of cls

i-th instance is cls(**cls.get_test_params()[i]).

nameslist of str, same length as objs

i-th element is name of i-th instance of obj in tests convention is {cls.__name__}-{i} if more than one instance otherwise {cls.__name__}.

parameter_setstr, default=”default”

Name of the set of test parameters to return, for use in tests. If no special parameters are defined for a value, will return “default” set.

evaluate(y_true, y_pred, **kwargs)

Evaluate the desired metric on given inputs.

Parameters:

y_truetime series in aeon compatible data container format

Ground truth (correct) target values y can be in one of the following formats: Series scitype: pd.Series, pd.DataFrame, or np.ndarray (1D or 2D) Panel scitype: pd.DataFrame with 2-level row MultiIndex,

3D np.ndarray, list of Series pd.DataFrame, or nested pd.DataFrame

Hierarchical scitype: pd.DataFrame with 3 or more level row MultiIndex

y_pred :time series in aeon compatible data container format

Forecasted values to evaluate must be of same format as y_true, same indices and columns if indexed

Returns:

lossfloat, np.ndarray, or pd.DataFrame

Calculated metric, averaged or by variable. float if self.multioutput=”uniform_average” or array-like

and self.multilevel=”uniform_average” or “uniform_average_time” value is metric averaged over variables and levels (see class docstring)

np.ndarray of shape (y_true.columns,) if self.multioutput=”raw_values”

and self.multilevel=”uniform_average” or “uniform_average_time” i-th entry is metric calculated for i-th variable

pd.DataFrame if self.multilevel=raw.values

of shape (n_levels, ) if self.multioutput = “uniform_average” or array of shape (n_levels, y_true.columns) if self.multioutput=”raw_values” metric is applied per level, row averaging (yes/no) as in multioutput

evaluate_by_index(y_true, y_pred, **kwargs)

Return the metric evaluated at each time point.

Parameters:

y_truetime series in aeon compatible data container format

Ground truth (correct) target values y can be in one of the following formats: Series scitype: pd.Series, pd.DataFrame, or np.ndarray (1D or 2D) Panel scitype: pd.DataFrame with 2-level row MultiIndex,

3D np.ndarray, list of Series pd.DataFrame, or nested pd.DataFrame

Hierarchical scitype: pd.DataFrame with 3 or more level row MultiIndex

y_pred :time series in aeon compatible data container format

Forecasted values to evaluate must be of same format as y_true, same indices and columns if indexed

Returns:

losspd.Series or pd.DataFrame

Calculated metric, by time point (default=jackknife pseudo-values). pd.Series if self.multioutput=”uniform_average” or array-like

index is equal to index of y_true entry at index i is metric at time i, averaged over variables

pd.DataFrame if self.multioutput=”raw_values”

index and columns equal to those of y_true i,j-th entry is metric at time i, at variable j

classmethod get_class_tag(tag_name, tag_value_default=None)

Get tag value from estimator class (only class tags).

Parameters:

tag_namestr

Name of tag value.

tag_value_defaultany type

Default/fallback value if tag is not found.

Returns:

tag_value

Value of the tag_name tag in self. If not found, returns tag_value_default.

See also

get_tag

Get a single tag from an object.

get_tags

Get all tags from an object.

get_class_tag

Get a single tag from a class.

Examples

>>> from aeon.classification import DummyClassifier
>>> DummyClassifier.get_class_tag("capability:multivariate")
True

classmethod get_class_tags()

Get class tags from estimator class and all its parent classes.

Returns:

collected_tagsdict

Dictionary of tag name : tag value pairs. Collected from _tags class attribute via nested inheritance. NOT overridden by dynamic tags set by set_tags or mirror_tags.

classmethod get_param_defaults()

Get parameter defaults for the object.

Returns:

default_dict: dict with str keys

keys are all parameters of cls that have a default defined in __init__ values are the defaults, as defined in __init__.

classmethod get_param_names()

Get parameter names for the object.

Returns:

param_names: list of str, alphabetically sorted list of parameter names of cls

get_params(deep=True)

Get parameters for this estimator.

Parameters:

deepbool, default=True

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns:

paramsdict

Parameter names mapped to their values.

get_tag(tag_name, tag_value_default=None, raise_error=True)

Get tag value from estimator class.

Uses dynamic tag overrides.

Parameters:

tag_namestr

Name of tag to be retrieved.

tag_value_defaultany type, default=None

Default/fallback value if tag is not found.

raise_errorbool

Whether a ValueError is raised when the tag is not found.

Returns:

tag_value

Value of the tag_name tag in self. If not found, returns an error if raise_error is True, otherwise it returns tag_value_default.

Raises:

ValueError if raise_error is True i.e. if tag_name is not in self.get_tags(

).keys()

See also

get_tags

Get all tags from an object.

get_clas_tags

Get all tags from a class.

get_class_tag

Get a single tag from a class.

Examples

>>> from aeon.classification import DummyClassifier
>>> d = DummyClassifier()
>>> d.get_tag("capability:multivariate")
True

get_tags()

Get tags from estimator class.

Includes the dynamic tag overrides.

Returns:

dict

Dictionary of tag name : tag value pairs. Collected from _tags class attribute via nested inheritance and then any overrides and new tags from _tags_dynamic object attribute.

See also

get_tag

Get a single tag from an object.

get_clas_tags

Get all tags from a class.

get_class_tag

Get a single tag from a class.

Examples

>>> from aeon.classification import DummyClassifier
>>> d = DummyClassifier()
>>> tags = d.get_tags()

classmethod get_test_params(parameter_set='default')

Return testing parameter settings for the estimator.

Parameters:

parameter_setstr, default=”default”

Name of the set of test parameters to return, for use in tests. If no special parameters are defined for a value, will return “default” set.

Returns:

paramsdict or list of dict, default = {}

Parameters to create testing instances of the class. Each dict are parameters to construct an “interesting” test instance, i.e., MyClass(**params) or MyClass(**params[i]) creates a valid test instance. create_test_instance uses the first (or only) dictionary in params.

is_composite()

Check if the object is composite.

A composite object is an object which contains objects, as parameters. Called on an instance, since this may differ by instance.

Returns:

composite: bool

Whether self contains a parameter which is BaseObject.

classmethod load_from_path(serial)

Load object from file location.

Parameters:

serialobject

Result of ZipFile(path).open(“object).

Returns:

deserialized self resulting in output at path, of cls.save(path)

classmethod load_from_serial(serial)

Load object from serialized memory container.

Parameters:

serialobject

First element of output of cls.save(None).

Returns:

deserialized self resulting in output serial, of cls.save(None).

reset()

Reset the object to a clean post-init state.

Equivalent to sklearn.clone but overwrites self. After self.reset() call, self is equal in value to type(self)(**self.get_params(deep=False))

Detail behaviour: removes any object attributes, except:

hyper-parameters = arguments of __init__ object attributes containing double-underscores, i.e., the string “__”

runs __init__ with current values of hyper-parameters (result of get_params)

Not affected by the reset are: object attributes containing double-underscores class and object methods, class attributes

save(path=None)

Save serialized self to bytes-like object or to (.zip) file.

Behaviour: if path is None, returns an in-memory serialized self if path is a file location, stores self at that location as a zip file

saved files are zip files with following contents: _metadata - contains class of self, i.e., type(self) _obj - serialized self. This class uses the default serialization (pickle).

Parameters:

pathNone or file location (str or Path).

if None, self is saved to an in-memory object if file location, self is saved to that file location. If:

path=”estimator” then a zip file estimator.zip will be made at cwd. path=”/home/stored/estimator” then a zip file estimator.zip will be stored in /home/stored/.

Returns:

if path is None - in-memory serialized self

if path is file location - ZipFile with reference to the file.

set_params(**params)

Set the parameters of this object.

The method works on simple estimators as well as on nested objects. The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters:

**paramsdict

BaseObject parameters

Returns:

selfreference to self (after parameters have been set)

set_tags(**tag_dict)

Set dynamic tags to given values.

Parameters:

**tag_dictdict

Dictionary of tag name : tag value pairs.

Returns:

Self

Reference to self.

Notes

Changes object state by settting tag values in tag_dict as dynamic tags in self.