A20 - A20 Index

The A20 Index [15] is an empirical evaluation metric that quantifies the proportion of predictions falling within a ±20% deviation from the experimental (actual) values.

A higher A20 score indicates better predictive accuracy, demonstrating that a larger percentage of the model’s predictions are practically acceptable within this 20% tolerance margin.

\[\begin{split}\text{A20}(y, \hat{y}) = \frac{1}{N} \sum_{i=1}^{N} \begin{cases} 1, & \text{if } \frac{|y_i - \hat{y}_i|}{|y_i|} \leq 0.2 \\ 0, & \text{otherwise} \end{cases}\end{split}\]

Description

Advantages:

  • High interpretability: Easily understood by non-technical stakeholders (e.g., “85% of predictions are within a 20% error margin”).

  • Outlier resilience: Extreme deviations do not disproportionately skew the metric; they are simply counted as outside the tolerance threshold (score = 0).

Disadvantages:

  • Rigid threshold (Cliff effect): A prediction with a 20.1% error is penalized exactly the same as a prediction with a 200% error. It treats “near-misses” and “massive failures” equally.

  • Zero-target vulnerability: Because the formula divides by the actual value (\(y_i\)), the calculation will become undefined (division by zero) if the ground truth data contains absolute zeros.

Properties

  • Best possible score: 1.0 (Higher is better; 100% of samples fall within the ±20% tolerance zone).

  • Range: [0.0, 1.0]

Example Usage

from numpy import array
from permetrics.regression import RegressionMetric

## 1. For 1-D array (Single-output)
y_true = array([3, -0.5, 2, 7])
y_pred = array([2.5, 0.0, 2, 8])

evaluator = RegressionMetric(y_true, y_pred)
# Calculate A20 Index
print("A20 Index: ", evaluator.A20())

## 2. For > 1-D array (Multi-output)
y_true = array([[0.5, 1], [-1, 1], [7, -6]])
y_pred = array([[0, 2], [-1, 2], [8, -5]])

evaluator = RegressionMetric(y_true, y_pred)
# Return an array of scores for each column
print("A20 Index (Multi-output): ", evaluator.A20(multi_output="raw_values"))