Installation

  • Install the current PyPI release:

    $ pip install permetrics==2.0.0

  • Install directly from source code:

    $ git clone https://github.com/thieu1995/permetrics.git
$ cd permetrics
$ python setup.py install

  • In case, you want to install the development version from Github:

    $ pip install git+https://github.com/thieu1995/permetrics

After installation, you can import Permetrics as any other Python module:

$ python
>>> import permetrics
>>> permetrics.__version__

Let’s go through some examples.

Examples

There are several ways you can use a performance metrics in this library. However, the most used are these two ways: functional-based and object-oriented based programming. We will go through detail of how to use 3 main type of metrics (regression, classification, and clustering) with these two methods.

Regression Metrics

Functional Style

  • This is a traditional way to call a specific metric you want to use. Everytime you want to use a metric, you need to pass y_true and y_pred

## 1. Import packages, classes
## 2. Create object
## 3. From object call function and use

import numpy as np
from permetrics import RegressionMetric

y_true = np.array([3, -0.5, 2, 7, 5, 6])
y_pred = np.array([2.5, 0.0, 2, 8, 5, 6])

evaluator = RegressionMetric()

## 3.1 Call specific function inside object, each function has 2 names like below
rmse_1 = evaluator.RMSE(y_true, y_pred)
rmse_2 = evaluator.root_mean_squared_error(y_true, y_pred)
print(f"RMSE: {rmse_1}, {rmse_2}")

mse = evaluator.MSE(y_true, y_pred)
mae = evaluator.MAE(y_true, y_pred)
print(f"MSE: {mse}, MAE: {mae}")

Object-Oriented Style

  • This is modern and better way to use metrics. You only need to pass y_true, y_pred one time when creating metric object.

  • After that, you can get the value of any metrics without passing y_true, y_pred

## 1. Import packages, classes
## 2. Create object
## 3. From object call function and use

import numpy as np
from permetrics import RegressionMetric

y_true = np.array([3, -0.5, 2, 7, 5, 6])
y_pred = np.array([2.5, 0.0, 2, 8, 5, 6])

evaluator = RegressionMetric(y_true, y_pred)

## Get the result of any function you want to
rmse = evaluator.RMSE()
mse = evaluator.MSE()
mae = evaluator.MAE()

print(f"RMSE: {rmse}, MSE: {mse}, MAE: {mae}")

Multiple Metrics Style

  • To reduce coding time when using multiple metrics. There are few ways to do it with Permetrics by using OOP style

import numpy as np
from permetrics import RegressionMetric

y_true = np.array([3, -0.5, 2, 7, 5, 6])
y_pred = np.array([2.5, 0.0, 2, 8, 5, 6])

evaluator = RegressionMetric(y_true, y_pred)

## Define list of metrics you want to use
list_metrics = ["RMSE", "MAE", "MAPE", "NSE"]

## 1. Get list metrics by using loop
list_results = []
for metric in list_metrics:
        list_results.append( evaluator.get_metric_by_name(metric) )
print(list_results)


## 2. Get list metrics by using function
dict_result_2 = evaluator.get_metrics_by_list_names(list_metrics)
print(dict_result_2)


## 3. Get list metrics by using function and parameters
dict_metrics = {
    "RMSE": None,
    "MAE": None,
    "MAPE": None,
    "NSE": None,
}
dict_result_3 = evaluator.get_metrics_by_dict(dict_metrics)
print(dict_result_3)

Multiple Outputs for Multiple Metrics

  • The Scikit-learn library is limited with multi-output metrics, but Permetrics can produce multi-output for all of metrics

import numpy as np
from permetrics import RegressionMetric

## This y_true and y_pred have 4 columns, 4 outputs
y_true = np.array([ [3, -0.5, 2, 7],
                    [5, 6, -0.3, 9],
                    [-11, 23, 8, 3.9] ])

y_pred = np.array([ [2.5, 0.0, 2, 8],
                    [5.2, 5.4, 0, 9.1],
                    [-10, 23, 8.2, 4] ])

evaluator = RegressionMetric(y_true, y_pred)

## 1. By default, all metrics can automatically return the multi-output results
# rmse = evaluator.RMSE()
# print(rmse)

## 2. If you want to take mean of all outputs, can set the parameter: multi-output = "mean"
# rmse_2 = evaluator.RMSE(multi_output="mean")
# print(rmse_2)

## 3. If you want a specific metric has more important than other, you can set weight for each output.
# rmse_3 = evaluator.RMSE(multi_output=[0.5, 0.05, 0.1, 0.35])
# print(rmse_3)


## Get multiple metrics with multi-output or single-output by parameters


## 1. Get list metrics by using list_names
list_metrics = ["RMSE", "MAE", "MSE"]
list_paras = [
    {"multi_output": "mean"},
    {"multi_output": [0.5, 0.2, 0.1, 0.2]},
    {"multi_output": "raw_values"}
]
dict_result_1 = evaluator.get_metrics_by_list_names(list_metrics, list_paras)
print(dict_result_1)


## 2. Get list metrics by using dict_metrics
dict_metrics = {
    "RMSE": {"multi_output": "mean"},
    "MAE": {"multi_output": "raw_values"},
    "MSE": {"multi_output": [0.5, 0.2, 0.1, 0.2]},
}
dict_result_2 = evaluator.get_metrics_by_dict(dict_metrics)
print(dict_result_2)

Classification Metrics

Functional Style

  • This is a traditional way to call a specific metric you want to use. Everytime you want to use a metric, you need to pass y_true and y_pred

## 1. Import packages, classes
## 2. Create object
## 3. From object call function and use

import numpy as np
from permetrics import ClassificationMetric

y_true = [0, 1, 0, 0, 1, 0]
y_pred = [0, 1, 0, 0, 0, 1]

evaluator = ClassificationMetric()

## 3.1 Call specific function inside object, each function has 2 names like below
ps1 = evaluator.precision_score(y_true, y_pred)
ps2 = evaluator.PS(y_true, y_pred)
ps3 = evaluator.PS(y_true, y_pred)
print(f"Precision: {ps1}, {ps2}, {ps3}")

recall = evaluator.recall_score(y_true, y_pred)
accuracy = evaluator.accuracy_score(y_true, y_pred)
print(f"recall: {recall}, accuracy: {accuracy}")

Object-Oriented Style

  • This is modern and better way to use metrics. You only need to pass y_true, y_pred one time when creating metric object.

  • After that, you can get the value of any metrics without passing y_true, y_pred

import numpy as np
from permetrics import ClassificationMetric

y_true = [0, 1, 0, 0, 1, 0]
y_pred = [0, 1, 0, 0, 0, 1]

evaluator = ClassificationMetric(y_true, y_pred)

## Get the result of any function you want to

hamming_score = evaluator.hamming_score()
mcc = evaluator.matthews_correlation_coefficient()
specificity = evaluator.specificity_score()
print(f"HL: {hamming_score}, MCC: {mcc}, specificity: {specificity}")

Clustering Metrics

Note that, this type of metrics is kinda differ from regression and classification. There are two type of clustering metrics include internal and external metrics, each serving a different purpose:

  1. Internal Metrics:

    • Objective: Internal metrics evaluate the quality of clusters based on the data itself without relying on external information or ground truth labels.

    • Example Metrics:

      • Silhouette Score: Measures how well-separated clusters are.

      • Davies-Bouldin Score: Computes the compactness and separation of clusters.

      • Inertia (within-cluster sum of squares): Measures how far points within a cluster are from the cluster’s centroid.

  2. External Metrics:

    • Objective: External metrics assess the quality of clusters by comparing them to some external criterion, often ground truth labels or known groupings.

    • Example Metrics:

      • Adjusted Rand Index (ARI): Measures the similarity between true and predicted clusters, adjusted for chance.

      • Normalized Mutual Information Index (NMII): Measures the mutual information between true and predicted clusters, normalized.

      • Fowlkes-Mallows Index: Computes the geometric mean of precision and recall between true and predicted clusters.

While internal metrics provide insights into the structure of the data within the clusters, external metrics help evaluate clustering performance against a known or expected structure, such as labeled data. The choice between internal and external metrics depends on the availability of ground truth information and the specific goals of the clustering analysis.

To clearly distinguish between internal and external clustering metrics, we use specific suffixes in their function names. Using the suffix index can indicate internal clustering metrics, while using the suffix score can indicate external clustering metrics. This naming convention makes it easier for users to differentiate between the two types of metrics and facilitates their usage.

By following this convention, users can easily identify whether a metric is designed for evaluating the quality of clusters within a dataset (internal) or for comparing clusters to external reference labels or ground truth (external). This distinction is important because internal and external metrics serve different purposes and have different interpretations.

Functional Style

  • External clustering metrics

import numpy as np
from permetrics import ClusteringMetric

y_true = [0, 1, 0, 0, 1, 0]
y_pred = [0, 1, 0, 0, 0, 1]

evaluator = ClusteringMetric()

ps1 = evaluator.mutual_info_score(y_true, y_pred)
ps2 = evaluator.MIS(y_true, y_pred)
print(f"Mutual Information score: {ps1}, {ps2}")

homogeneity = evaluator.homogeneity_score(y_true, y_pred)
completeness  = evaluator.CS(y_true, y_pred)
print(f"Homogeneity: {homogeneity}, Completeness : {completeness}")

  • Internal clustering metrics

import numpy as np
from permetrics import ClusteringMetric
from sklearn.datasets import make_blobs

# generate sample data
X, y_true = make_blobs(n_samples=300, centers=4, cluster_std=0.60, random_state=0)
y_pred = np.random.randint(0, 4, size=300)

evaluator = ClusteringMetric()

evaluator.BHI(X=X, y_pred=y_pred)
evaluator.BRI(X=X, y_pred=y_pred)

Object-Oriented Style

  • External clustering metrics

import numpy as np
from permetrics import ClusteringMetric

y_true = [0, 1, 0, 0, 1, 0]
y_pred = [0, 1, 0, 0, 0, 1]

evaluator = ClusteringMetric(y_true, y_pred)

## Get the result of any function you want to
x1 = evaluator.kulczynski_score()
x2 = evaluator.mc_nemar_score()
x3 = evaluator.rogers_tanimoto_score()
print(f"Kulczynski: {x1}, Mc Nemar: {x2}, Rogers Tanimoto: {x3}")

  • Internal clustering metrics

import numpy as np
from permetrics import ClusteringMetric
from sklearn.datasets import make_blobs

# generate sample data
X, y_true = make_blobs(n_samples=300, centers=4, cluster_std=0.60, random_state=0)
y_pred = np.random.randint(0, 4, size=300)

evaluator = ClusteringMetric(X=X, y_pred=y_pred)

evaluator.BHI()
evaluator.CHI()
evaluator.DBI()

## Or
print(evaluator.get_metrics_by_list_names(["BHI", "CHI", "XBI", "BRI", "DBI", "DRI", "DI", "KDI", "LDRI", "LSRI", "SI"]))