recbole.evaluator.metrics

Suppose there is a set of \(n\) items to be ranked. Given a user \(u\) in the user set \(U\), we use \(\hat R(u)\) to represent a ranked list of items that a model produces, and \(R(u)\) to represent a ground-truth set of items that user \(u\) has interacted with. For top-k recommendation, only top-ranked items are important to consider. Therefore, in top-k evaluation scenarios, we truncate the recommendation list with a length \(K\). Besides, in loss-based metrics, \(S\) represents the set of user(u)-item(i) pairs, \(\hat r_{u i}\) represents the score predicted by the model, \({r}_{u i}\) represents the ground-truth labels.

class recbole.evaluator.metrics.AUC(config)

Bases: recbole.evaluator.base_metric.LossMetric

AUC (also known as Area Under Curve) is used to evaluate the two-class model, referring to the area under the ROC curve.

Note

This metric does not calculate group-based AUC which considers the AUC scores averaged across users. It is also not limited to k. Instead, it calculates the scores on the entire prediction results regardless the users. We call the interface in scikit-learn, and code calculates the metric using the variation of following formula.

\[\mathrm {AUC} = \frac {{{M} \times {(N+1)} - \frac{M \times (M+1)}{2}} - \sum\limits_{i=1}^{M} rank_{i}} {{M} \times {(N - M)}}\]

\(M\) denotes the number of positive items. \(N\) denotes the total number of user-item interactions. \(rank_i\) denotes the descending rank of the i-th positive item.

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

metric_info(preds, trues)

Calculate the value of the metric.

Parameters

• preds (numpy.ndarray) – the scores predicted by model, a one-dimensional vector.

• trues (numpy.ndarray) – the label of items, which has the same shape as preds.

Returns

The value of the metric.

Return type

float

class recbole.evaluator.metrics.AveragePopularity(config)

Bases: recbole.evaluator.base_metric.AbstractMetric

AveragePopularity computes the average popularity of recommended items.

For further details, please refer to the paper and paper.

\[\mathrm{AveragePopularity@K}=\frac{1}{|U|} \sum_{u \in U } \frac{\sum_{i \in R_{u}} \phi(i)}{|R_{u}|}\]

\(\phi(i)\) is the number of interaction of item i in training data.

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

get_pop(item_matrix, item_count)

Convert the matrix of item id to the matrix of item popularity using a dict:{id,count}.

Parameters

• item_matrix (numpy.ndarray) – matrix of items recommended to users.

• item_count (dict) – the number of interaction of items in training data.

Returns

the popularity of items in the recommended list.

Return type

numpy.ndarray

metric_info(values)

metric_need = ['rec.items', 'data.count_items']

metric_type = 1

smaller = True

topk_result(metric, value)

Match the metric value to the k and put them in dictionary form

Parameters

• metric (str) – the name of calculated metric.

• value (numpy.ndarray) – metrics for each user, including values from metric@1 to metric@max(self.topk).

Returns

metric values required in the configuration.

Return type

dict

used_info(dataobject)

Get the matrix of recommendation items and the popularity of items in training data

class recbole.evaluator.metrics.GAUC(config)

Bases: recbole.evaluator.base_metric.AbstractMetric

GAUC (also known as Grouped Area Under Curve) is used to evaluate the two-class model, referring to the area under the ROC curve grouped by user. We weighted the index of each user \(u\) by the number of positive samples of users to get the final result.

For further details, please refer to the paper

Note

It calculates the AUC score of each user, and finally obtains GAUC by weighting the user AUC. It is also not limited to k. Due to our padding for scores_tensor with -np.inf, the padding value will influence the ranks of origin items. Therefore, we use descending sort here and make an identity transformation to the formula of AUC, which is shown in auc_ function. For readability, we didn’t do simplification in the code.

\[\begin{split}\begin{align*} \mathrm {AUC(u)} &= \frac {{{|R(u)|} \times {(n+1)} - \frac{|R(u)| \times (|R(u)|+1)}{2}} - \sum\limits_{i=1}^{|R(u)|} rank_{i}} {{|R(u)|} \times {(n - |R(u)|)}} \\ \mathrm{GAUC} &= \frac{1}{\sum_{u \in U} |R(u)|}\sum_{u \in U} |R(u)| \cdot(\mathrm {AUC(u)}) \end{align*}\end{split}\]

\(rank_i\) is the descending rank of the i-th items in \(R(u)\).

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

metric_info(pos_rank_sum, user_len_list, pos_len_list)

Get the value of GAUC metric.

Parameters

• pos_rank_sum (numpy.ndarray) – sum of descending rankings for positive items of each users.

• user_len_list (numpy.ndarray) – the number of predicted items for users.

• pos_len_list (numpy.ndarray) – the number of positive items for users.

Returns

The value of the GAUC.

Return type

float

metric_need = ['rec.meanrank']

metric_type = 1

class recbole.evaluator.metrics.GiniIndex(config)

Bases: recbole.evaluator.base_metric.AbstractMetric

GiniIndex presents the diversity of the recommendation items. It is used to measure the inequality of a distribution.

For further details, please refer to the paper.

\[\mathrm {GiniIndex@K}=\left(\frac{\sum_{i=1}^{|I|}(2 i-|I|-1) P{(i)}}{|I| \sum_{i=1}^{|I|} P{(i)}}\right)\]

\(P{(i)}\) represents the number of times all items appearing in the recommended list, which is indexed in non-decreasing order (P_{(i)} leq P_{(i+1)}).

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

get_gini(item_matrix, num_items)

Get gini index through the top-k recommendation list.

Parameters

• item_matrix (numpy.ndarray) – matrix of items recommended to users.

• num_items (int) – the total number of items.

Returns

the gini index.

Return type

float

metric_need = ['rec.items', 'data.num_items']

metric_type = 1

smaller = True

used_info(dataobject)

Get the matrix of recommendation items and number of items in total item set

class recbole.evaluator.metrics.Hit(config)

Bases: recbole.evaluator.base_metric.TopkMetric

HR (also known as truncated Hit-Ratio) is a way of calculating how many ‘hits’ you have in an n-sized list of ranked items. If there is at least one item that falls in the ground-truth set, we call it a hit.

\[\mathrm {HR@K} = \frac{1}{|U|}\sum_{u \in U} \delta(\hat{R}(u) \cap R(u) \neq \emptyset),\]

\(\delta(·)\) is an indicator function. \(\delta(b)\) = 1 if \(b\) is true and 0 otherwise. \(\emptyset\) denotes the empty set.

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

metric_info(pos_index)

Calculate the value of the metric.

Parameters

• pos_index (numpy.ndarray) – a bool matrix, shape of n_users * max(topk). The item with the (j+1)-th highest score of i-th user is positive if pos_index[i][j] == True and negative otherwise.

• pos_len (numpy.ndarray) – a vector representing the number of positive items per user, shape of (n_users,).

Returns

metrics for each user, including values from metric@1 to metric@max(self.topk).

Return type

numpy.ndarray

class recbole.evaluator.metrics.ItemCoverage(config)

Bases: recbole.evaluator.base_metric.AbstractMetric

ItemCoverage computes the coverage of recommended items over all items.

For further details, please refer to the paper and paper.

\[\mathrm{Coverage@K}=\frac{\left| \bigcup_{u \in U} \hat{R}(u) \right|}{|I|}\]

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

get_coverage(item_matrix, num_items)

Get the coverage of recommended items over all items

Parameters

• item_matrix (numpy.ndarray) – matrix of items recommended to users.

• num_items (int) – the total number of items.

Returns

the coverage metric.

Return type

float

metric_need = ['rec.items', 'data.num_items']

metric_type = 1

used_info(dataobject)

Get the matrix of recommendation items and number of items in total item set

class recbole.evaluator.metrics.LogLoss(config)

Bases: recbole.evaluator.base_metric.LossMetric

Logloss (also known as logistic loss or cross-entropy loss) is used to evaluate the probabilistic output of the two-class classifier.

\[LogLoss = \frac{1}{|S|} \sum_{(u,i) \in S}(-((r_{u i} \ \log{\hat{r}_{u i}}) + {(1 - r_{u i})}\ \log{(1 - \hat{r}_{u i})}))\]

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

metric_info(preds, trues)

Calculate the value of the metric.

Parameters

• preds (numpy.ndarray) – the scores predicted by model, a one-dimensional vector.

• trues (numpy.ndarray) – the label of items, which has the same shape as preds.

Returns

The value of the metric.

Return type

float

smaller = True

class recbole.evaluator.metrics.MAE(config)

Bases: recbole.evaluator.base_metric.LossMetric

MAE (also known as Mean Absolute Error regression loss) is used to evaluate the difference between the score predicted by the model and the actual behavior of the user.

\[\mathrm{MAE}=\frac{1}{|{S}|} \sum_{(u, i) \in {S}}\left|\hat{r}_{u i}-r_{u i}\right|\]

\(|S|\) represents the number of pairs in \(S\).

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

metric_info(preds, trues)

Calculate the value of the metric.

Parameters

• preds (numpy.ndarray) – the scores predicted by model, a one-dimensional vector.

• trues (numpy.ndarray) – the label of items, which has the same shape as preds.

Returns

The value of the metric.

Return type

float

smaller = True

class recbole.evaluator.metrics.MAP(config)

Bases: recbole.evaluator.base_metric.TopkMetric

MAP (also known as Mean Average Precision) is meant to calculate average precision for the relevant items.

Note

In this case the normalization factor used is \(\frac{1}{min(|\hat R(u)|, K)}\), which prevents your AP score from being unfairly suppressed when your number of recommendations couldn’t possibly capture all the correct ones.

\[\mathrm{MAP@K} = \frac{1}{|U|}\sum_{u \in U} (\frac{1}{min(|\hat R(u)|, K)} \sum_{j=1}^{|\hat{R}(u)|} I\left(\hat{R}_{j}(u) \in R(u)\right) \cdot Precision@j)\]

\(\hat{R}_{j}(u)\) is the j-th item in the recommendation list of hat R (u)).

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

metric_info(pos_index, pos_len)

Calculate the value of the metric.

Parameters

• pos_index (numpy.ndarray) – a bool matrix, shape of n_users * max(topk). The item with the (j+1)-th highest score of i-th user is positive if pos_index[i][j] == True and negative otherwise.

• pos_len (numpy.ndarray) – a vector representing the number of positive items per user, shape of (n_users,).

Returns

metrics for each user, including values from metric@1 to metric@max(self.topk).

Return type

numpy.ndarray

class recbole.evaluator.metrics.MRR(config)

Bases: recbole.evaluator.base_metric.TopkMetric

The MRR (also known as Mean Reciprocal Rank) computes the reciprocal rank of the first relevant item found by an algorithm.

\[\mathrm {MRR@K} = \frac{1}{|U|}\sum_{u \in U} \frac{1}{\operatorname{rank}_{u}^{*}}\]

\({rank}_{u}^{*}\) is the rank position of the first relevant item found by an algorithm for a user \(u\).

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

metric_info(pos_index)

Calculate the value of the metric.

Parameters

• pos_index (numpy.ndarray) – a bool matrix, shape of n_users * max(topk). The item with the (j+1)-th highest score of i-th user is positive if pos_index[i][j] == True and negative otherwise.

• pos_len (numpy.ndarray) – a vector representing the number of positive items per user, shape of (n_users,).

Returns

metrics for each user, including values from metric@1 to metric@max(self.topk).

Return type

numpy.ndarray

class recbole.evaluator.metrics.NDCG(config)

Bases: recbole.evaluator.base_metric.TopkMetric

NDCG (also known as normalized discounted cumulative gain) is a measure of ranking quality, where positions are discounted logarithmically. It accounts for the position of the hit by assigning higher scores to hits at top ranks.

\[\mathrm {NDCG@K} = \frac{1}{|U|}\sum_{u \in U} (\frac{1}{\sum_{i=1}^{\min (|R(u)|, K)} \frac{1}{\log _{2}(i+1)}} \sum_{i=1}^{K} \delta(i \in R(u)) \frac{1}{\log _{2}(i+1)})\]

\(\delta(·)\) is an indicator function.

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

metric_info(pos_index, pos_len)

Calculate the value of the metric.

Parameters

• pos_index (numpy.ndarray) – a bool matrix, shape of n_users * max(topk). The item with the (j+1)-th highest score of i-th user is positive if pos_index[i][j] == True and negative otherwise.

• pos_len (numpy.ndarray) – a vector representing the number of positive items per user, shape of (n_users,).

Returns

metrics for each user, including values from metric@1 to metric@max(self.topk).

Return type

numpy.ndarray

class recbole.evaluator.metrics.Precision(config)

Bases: recbole.evaluator.base_metric.TopkMetric

Precision (also called positive predictive value) is a measure for computing the fraction of relevant items out of all the recommended items. We average the metric for each user \(u\) get the final result.

\[\mathrm {Precision@K} = \frac{1}{|U|}\sum_{u \in U} \frac{|\hat{R}(u) \cap R(u)|}{|\hat {R}(u)|}\]

\(|\hat R(u)|\) represents the item count of \(\hat R(u)\).

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

metric_info(pos_index)

Calculate the value of the metric.

Parameters

• pos_index (numpy.ndarray) – a bool matrix, shape of n_users * max(topk). The item with the (j+1)-th highest score of i-th user is positive if pos_index[i][j] == True and negative otherwise.

• pos_len (numpy.ndarray) – a vector representing the number of positive items per user, shape of (n_users,).

Returns

metrics for each user, including values from metric@1 to metric@max(self.topk).

Return type

numpy.ndarray

class recbole.evaluator.metrics.RMSE(config)

Bases: recbole.evaluator.base_metric.LossMetric

RMSE (also known as Root Mean Squared Error) is another error metric like MAE.

\[\mathrm{RMSE} = \sqrt{\frac{1}{|{S}|} \sum_{(u, i) \in {S}}(\hat{r}_{u i}-r_{u i})^{2}}\]

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

metric_info(preds, trues)

Calculate the value of the metric.

Parameters

• preds (numpy.ndarray) – the scores predicted by model, a one-dimensional vector.

• trues (numpy.ndarray) – the label of items, which has the same shape as preds.

Returns

The value of the metric.

Return type

float

smaller = True

class recbole.evaluator.metrics.Recall(config)

Bases: recbole.evaluator.base_metric.TopkMetric

Recall is a measure for computing the fraction of relevant items out of all relevant items.

\[\mathrm {Recall@K} = \frac{1}{|U|}\sum_{u \in U} \frac{|\hat{R}(u) \cap R(u)|}{|R(u)|}\]

\(|R(u)|\) represents the item count of \(R(u)\).

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

metric_info(pos_index, pos_len)

Calculate the value of the metric.

Parameters

• pos_index (numpy.ndarray) – a bool matrix, shape of n_users * max(topk). The item with the (j+1)-th highest score of i-th user is positive if pos_index[i][j] == True and negative otherwise.

• pos_len (numpy.ndarray) – a vector representing the number of positive items per user, shape of (n_users,).

Returns

metrics for each user, including values from metric@1 to metric@max(self.topk).

Return type

numpy.ndarray

class recbole.evaluator.metrics.ShannonEntropy(config)

Bases: recbole.evaluator.base_metric.AbstractMetric

ShannonEntropy presents the diversity of the recommendation items. It is the entropy over items’ distribution.

For further details, please refer to the paper and paper

\[\mathrm {ShannonEntropy@K}=-\sum_{i=1}^{|I|} p(i) \log p(i)\]

\(p(i)\) is the probability of recommending item i which is the number of item i in recommended list over all items.

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

get_entropy(item_matrix)

Get shannon entropy through the top-k recommendation list.

Parameters

item_matrix (numpy.ndarray) – matrix of items recommended to users.

Returns

the shannon entropy.

Return type

float

metric_need = ['rec.items']

metric_type = 1

used_info(dataobject)

Get the matrix of recommendation items.

class recbole.evaluator.metrics.TailPercentage(config)

Bases: recbole.evaluator.base_metric.AbstractMetric

TailPercentage computes the percentage of long-tail items in recommendation items.

For further details, please refer to the paper.

\[\mathrm {TailPercentage@K}=\frac{1}{|U|} \sum_{u \in U} \frac{\sum_{i \in R_{u}} {\delta(i \in T)}}{|R_{u}|}\]

\(\delta(·)\) is an indicator function. \(T\) is the set of long-tail items, which is a portion of items that appear in training data seldomly.

Note

If you want to use this metric, please set the parameter ‘tail_ratio’ in the config which can be an integer or a float in (0,1]. Otherwise it will default to 0.1.

calculate_metric(dataobject)

Get the dictionary of a metric.

Parameters

dataobject (DataStruct) – it contains all the information needed to calculate metrics.

Returns

such as {'metric@10': 3153, 'metric@20': 0.3824}

Return type

dict

get_tail(item_matrix, count_items)

Get long-tail percentage through the top-k recommendation list.

Parameters

• item_matrix (numpy.ndarray) – matrix of items recommended to users.

• count_items (dict) – the number of interaction of items in training data.

Returns

long-tail percentage.

Return type

float

metric_info(values)

metric_need = ['rec.items', 'data.count_items']

metric_type = 1

topk_result(metric, value)

Match the metric value to the k and put them in dictionary form.

Parameters

• metric (str) – the name of calculated metric.

• value (numpy.ndarray) – metrics for each user, including values from metric@1 to metric@max(self.topk).

Returns

metric values required in the configuration.

Return type

dict

used_info(dataobject)

Get the matrix of recommendation items and number of items in total item set.