recbole.evaluator.metrics¶
Suppose there is a set of \(n\) items to be ranked. Given a user \(u\) in the user set \(U\), we use \(\hat R(u)\) to represent a ranked list of items that a model produces, and \(R(u)\) to represent a ground-truth set of items that user \(u\) has interacted with. For top-k recommendation, only top-ranked items are important to consider. Therefore, in top-k evaluation scenarios, we truncate the recommendation list with a length \(K\). Besides, in loss-based metrics, \(S\) represents the set of user(u)-item(i) pairs, \(\hat r_{u i}\) represents the score predicted by the model, \({r}_{u i}\) represents the ground-truth labels.
- class recbole.evaluator.metrics.AUC(config)[source]¶
Bases:
recbole.evaluator.base_metric.LossMetric
AUC (also known as Area Under Curve) is used to evaluate the two-class model, referring to the area under the ROC curve.
Note
This metric does not calculate group-based AUC which considers the AUC scores averaged across users. It is also not limited to k. Instead, it calculates the scores on the entire prediction results regardless the users. We call the interface in scikit-learn, and code calculates the metric using the variation of following formula.
\[\mathrm {AUC} = \frac {{{M} \times {(N+1)} - \frac{M \times (M+1)}{2}} - \sum\limits_{i=1}^{M} rank_{i}} {{M} \times {(N - M)}}\]\(M\) denotes the number of positive items. \(N\) denotes the total number of user-item interactions. \(rank_i\) denotes the descending rank of the i-th positive item.
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- class recbole.evaluator.metrics.AveragePopularity(config)[source]¶
Bases:
recbole.evaluator.base_metric.AbstractMetric
AveragePopularity computes the average popularity of recommended items.
For further details, please refer to the paper and paper.
\[\mathrm{AveragePopularity@K}=\frac{1}{|U|} \sum_{u \in U } \frac{\sum_{i \in R_{u}} \phi(i)}{|R_{u}|}\]\(\phi(i)\) is the number of interaction of item i in training data.
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- get_pop(item_matrix, item_count)[source]¶
Convert the matrix of item id to the matrix of item popularity using a dict:{id,count}.
- Parameters
item_matrix (numpy.ndarray) – matrix of items recommended to users.
item_count (dict) – the number of interaction of items in training data.
- Returns
the popularity of items in the recommended list.
- Return type
numpy.ndarray
- metric_need = ['rec.items', 'data.count_items']¶
- metric_type = 1¶
- smaller = True¶
- topk_result(metric, value)[source]¶
Match the metric value to the k and put them in dictionary form
- Parameters
metric (str) – the name of calculated metric.
value (numpy.ndarray) – metrics for each user, including values from metric@1 to metric@max(self.topk).
- Returns
metric values required in the configuration.
- Return type
dict
- class recbole.evaluator.metrics.GAUC(config)[source]¶
Bases:
recbole.evaluator.base_metric.AbstractMetric
GAUC (also known as Grouped Area Under Curve) is used to evaluate the two-class model, referring to the area under the ROC curve grouped by user. We weighted the index of each user \(u\) by the number of positive samples of users to get the final result.
For further details, please refer to the paper
Note
It calculates the AUC score of each user, and finally obtains GAUC by weighting the user AUC. It is also not limited to k. Due to our padding for scores_tensor with -np.inf, the padding value will influence the ranks of origin items. Therefore, we use descending sort here and make an identity transformation to the formula of AUC, which is shown in auc_ function. For readability, we didn’t do simplification in the code.
\[\begin{split}\begin{align*} \mathrm {AUC(u)} &= \frac {{{|R(u)|} \times {(n+1)} - \frac{|R(u)| \times (|R(u)|+1)}{2}} - \sum\limits_{i=1}^{|R(u)|} rank_{i}} {{|R(u)|} \times {(n - |R(u)|)}} \\ \mathrm{GAUC} &= \frac{1}{\sum_{u \in U} |R(u)|}\sum_{u \in U} |R(u)| \cdot(\mathrm {AUC(u)}) \end{align*}\end{split}\]\(rank_i\) is the descending rank of the i-th items in \(R(u)\).
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- metric_info(pos_rank_sum, user_len_list, pos_len_list)[source]¶
Get the value of GAUC metric.
- Parameters
pos_rank_sum (numpy.ndarray) – sum of descending rankings for positive items of each users.
user_len_list (numpy.ndarray) – the number of predicted items for users.
pos_len_list (numpy.ndarray) – the number of positive items for users.
- Returns
The value of the GAUC.
- Return type
float
- metric_need = ['rec.meanrank']¶
- metric_type = 1¶
- class recbole.evaluator.metrics.GiniIndex(config)[source]¶
Bases:
recbole.evaluator.base_metric.AbstractMetric
GiniIndex presents the diversity of the recommendation items. It is used to measure the inequality of a distribution.
For further details, please refer to the paper.
\[\mathrm {GiniIndex@K}=\left(\frac{\sum_{i=1}^{|I|}(2 i-|I|-1) P{(i)}}{|I| \sum_{i=1}^{|I|} P{(i)}}\right)\]\(P{(i)}\) represents the number of times all items appearing in the recommended list, which is indexed in non-decreasing order (P_{(i)} leq P_{(i+1)}).
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- get_gini(item_matrix, num_items)[source]¶
Get gini index through the top-k recommendation list.
- Parameters
item_matrix (numpy.ndarray) – matrix of items recommended to users.
num_items (int) – the total number of items.
- Returns
the gini index.
- Return type
float
- metric_need = ['rec.items', 'data.num_items']¶
- metric_type = 1¶
- smaller = True¶
- class recbole.evaluator.metrics.Hit(config)[source]¶
Bases:
recbole.evaluator.base_metric.TopkMetric
HR (also known as truncated Hit-Ratio) is a way of calculating how many ‘hits’ you have in an n-sized list of ranked items. If there is at least one item that falls in the ground-truth set, we call it a hit.
\[\mathrm {HR@K} = \frac{1}{|U|}\sum_{u \in U} \delta(\hat{R}(u) \cap R(u) \neq \emptyset),\]\(\delta(·)\) is an indicator function. \(\delta(b)\) = 1 if \(b\) is true and 0 otherwise. \(\emptyset\) denotes the empty set.
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- metric_info(pos_index)[source]¶
Calculate the value of the metric.
- Parameters
pos_index (numpy.ndarray) – a bool matrix, shape of
n_users * max(topk)
. The item with the (j+1)-th highest score of i-th user is positive ifpos_index[i][j] == True
and negative otherwise.pos_len (numpy.ndarray) – a vector representing the number of positive items per user, shape of
(n_users,)
.
- Returns
metrics for each user, including values from metric@1 to metric@max(self.topk).
- Return type
numpy.ndarray
- class recbole.evaluator.metrics.ItemCoverage(config)[source]¶
Bases:
recbole.evaluator.base_metric.AbstractMetric
ItemCoverage computes the coverage of recommended items over all items.
For further details, please refer to the paper and paper.
\[\mathrm{Coverage@K}=\frac{\left| \bigcup_{u \in U} \hat{R}(u) \right|}{|I|}\]- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- get_coverage(item_matrix, num_items)[source]¶
Get the coverage of recommended items over all items
- Parameters
item_matrix (numpy.ndarray) – matrix of items recommended to users.
num_items (int) – the total number of items.
- Returns
the coverage metric.
- Return type
float
- metric_need = ['rec.items', 'data.num_items']¶
- metric_type = 1¶
- class recbole.evaluator.metrics.LogLoss(config)[source]¶
Bases:
recbole.evaluator.base_metric.LossMetric
Logloss (also known as logistic loss or cross-entropy loss) is used to evaluate the probabilistic output of the two-class classifier.
\[LogLoss = \frac{1}{|S|} \sum_{(u,i) \in S}(-((r_{u i} \ \log{\hat{r}_{u i}}) + {(1 - r_{u i})}\ \log{(1 - \hat{r}_{u i})}))\]- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- metric_info(preds, trues)[source]¶
Calculate the value of the metric.
- Parameters
preds (numpy.ndarray) – the scores predicted by model, a one-dimensional vector.
trues (numpy.ndarray) – the label of items, which has the same shape as
preds
.
- Returns
The value of the metric.
- Return type
float
- smaller = True¶
- class recbole.evaluator.metrics.MAE(config)[source]¶
Bases:
recbole.evaluator.base_metric.LossMetric
MAE (also known as Mean Absolute Error regression loss) is used to evaluate the difference between the score predicted by the model and the actual behavior of the user.
\[\mathrm{MAE}=\frac{1}{|{S}|} \sum_{(u, i) \in {S}}\left|\hat{r}_{u i}-r_{u i}\right|\]\(|S|\) represents the number of pairs in \(S\).
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- metric_info(preds, trues)[source]¶
Calculate the value of the metric.
- Parameters
preds (numpy.ndarray) – the scores predicted by model, a one-dimensional vector.
trues (numpy.ndarray) – the label of items, which has the same shape as
preds
.
- Returns
The value of the metric.
- Return type
float
- smaller = True¶
- class recbole.evaluator.metrics.MAP(config)[source]¶
Bases:
recbole.evaluator.base_metric.TopkMetric
MAP (also known as Mean Average Precision) is meant to calculate average precision for the relevant items.
Note
In this case the normalization factor used is \(\frac{1}{min(|\hat R(u)|, K)}\), which prevents your AP score from being unfairly suppressed when your number of recommendations couldn’t possibly capture all the correct ones.
\[\mathrm{MAP@K} = \frac{1}{|U|}\sum_{u \in U} (\frac{1}{min(|\hat R(u)|, K)} \sum_{j=1}^{|\hat{R}(u)|} I\left(\hat{R}_{j}(u) \in R(u)\right) \cdot Precision@j)\]\(\hat{R}_{j}(u)\) is the j-th item in the recommendation list of hat R (u)).
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- metric_info(pos_index, pos_len)[source]¶
Calculate the value of the metric.
- Parameters
pos_index (numpy.ndarray) – a bool matrix, shape of
n_users * max(topk)
. The item with the (j+1)-th highest score of i-th user is positive ifpos_index[i][j] == True
and negative otherwise.pos_len (numpy.ndarray) – a vector representing the number of positive items per user, shape of
(n_users,)
.
- Returns
metrics for each user, including values from metric@1 to metric@max(self.topk).
- Return type
numpy.ndarray
- class recbole.evaluator.metrics.MRR(config)[source]¶
Bases:
recbole.evaluator.base_metric.TopkMetric
The MRR (also known as Mean Reciprocal Rank) computes the reciprocal rank of the first relevant item found by an algorithm.
\[\mathrm {MRR@K} = \frac{1}{|U|}\sum_{u \in U} \frac{1}{\operatorname{rank}_{u}^{*}}\]\({rank}_{u}^{*}\) is the rank position of the first relevant item found by an algorithm for a user \(u\).
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- metric_info(pos_index)[source]¶
Calculate the value of the metric.
- Parameters
pos_index (numpy.ndarray) – a bool matrix, shape of
n_users * max(topk)
. The item with the (j+1)-th highest score of i-th user is positive ifpos_index[i][j] == True
and negative otherwise.pos_len (numpy.ndarray) – a vector representing the number of positive items per user, shape of
(n_users,)
.
- Returns
metrics for each user, including values from metric@1 to metric@max(self.topk).
- Return type
numpy.ndarray
- class recbole.evaluator.metrics.NDCG(config)[source]¶
Bases:
recbole.evaluator.base_metric.TopkMetric
NDCG (also known as normalized discounted cumulative gain) is a measure of ranking quality, where positions are discounted logarithmically. It accounts for the position of the hit by assigning higher scores to hits at top ranks.
\[\mathrm {NDCG@K} = \frac{1}{|U|}\sum_{u \in U} (\frac{1}{\sum_{i=1}^{\min (|R(u)|, K)} \frac{1}{\log _{2}(i+1)}} \sum_{i=1}^{K} \delta(i \in R(u)) \frac{1}{\log _{2}(i+1)})\]\(\delta(·)\) is an indicator function.
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- metric_info(pos_index, pos_len)[source]¶
Calculate the value of the metric.
- Parameters
pos_index (numpy.ndarray) – a bool matrix, shape of
n_users * max(topk)
. The item with the (j+1)-th highest score of i-th user is positive ifpos_index[i][j] == True
and negative otherwise.pos_len (numpy.ndarray) – a vector representing the number of positive items per user, shape of
(n_users,)
.
- Returns
metrics for each user, including values from metric@1 to metric@max(self.topk).
- Return type
numpy.ndarray
- class recbole.evaluator.metrics.Precision(config)[source]¶
Bases:
recbole.evaluator.base_metric.TopkMetric
Precision (also called positive predictive value) is a measure for computing the fraction of relevant items out of all the recommended items. We average the metric for each user \(u\) get the final result.
\[\mathrm {Precision@K} = \frac{1}{|U|}\sum_{u \in U} \frac{|\hat{R}(u) \cap R(u)|}{|\hat {R}(u)|}\]\(|\hat R(u)|\) represents the item count of \(\hat R(u)\).
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- metric_info(pos_index)[source]¶
Calculate the value of the metric.
- Parameters
pos_index (numpy.ndarray) – a bool matrix, shape of
n_users * max(topk)
. The item with the (j+1)-th highest score of i-th user is positive ifpos_index[i][j] == True
and negative otherwise.pos_len (numpy.ndarray) – a vector representing the number of positive items per user, shape of
(n_users,)
.
- Returns
metrics for each user, including values from metric@1 to metric@max(self.topk).
- Return type
numpy.ndarray
- class recbole.evaluator.metrics.RMSE(config)[source]¶
Bases:
recbole.evaluator.base_metric.LossMetric
RMSE (also known as Root Mean Squared Error) is another error metric like MAE.
\[\mathrm{RMSE} = \sqrt{\frac{1}{|{S}|} \sum_{(u, i) \in {S}}(\hat{r}_{u i}-r_{u i})^{2}}\]- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- metric_info(preds, trues)[source]¶
Calculate the value of the metric.
- Parameters
preds (numpy.ndarray) – the scores predicted by model, a one-dimensional vector.
trues (numpy.ndarray) – the label of items, which has the same shape as
preds
.
- Returns
The value of the metric.
- Return type
float
- smaller = True¶
- class recbole.evaluator.metrics.Recall(config)[source]¶
Bases:
recbole.evaluator.base_metric.TopkMetric
Recall is a measure for computing the fraction of relevant items out of all relevant items.
\[\mathrm {Recall@K} = \frac{1}{|U|}\sum_{u \in U} \frac{|\hat{R}(u) \cap R(u)|}{|R(u)|}\]\(|R(u)|\) represents the item count of \(R(u)\).
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- metric_info(pos_index, pos_len)[source]¶
Calculate the value of the metric.
- Parameters
pos_index (numpy.ndarray) – a bool matrix, shape of
n_users * max(topk)
. The item with the (j+1)-th highest score of i-th user is positive ifpos_index[i][j] == True
and negative otherwise.pos_len (numpy.ndarray) – a vector representing the number of positive items per user, shape of
(n_users,)
.
- Returns
metrics for each user, including values from metric@1 to metric@max(self.topk).
- Return type
numpy.ndarray
- class recbole.evaluator.metrics.ShannonEntropy(config)[source]¶
Bases:
recbole.evaluator.base_metric.AbstractMetric
ShannonEntropy presents the diversity of the recommendation items. It is the entropy over items’ distribution.
For further details, please refer to the paper and paper
\[\mathrm {ShannonEntropy@K}=-\sum_{i=1}^{|I|} p(i) \log p(i)\]\(p(i)\) is the probability of recommending item i which is the number of item i in recommended list over all items.
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- get_entropy(item_matrix)[source]¶
Get shannon entropy through the top-k recommendation list.
- Parameters
item_matrix (numpy.ndarray) – matrix of items recommended to users.
- Returns
the shannon entropy.
- Return type
float
- metric_need = ['rec.items']¶
- metric_type = 1¶
- class recbole.evaluator.metrics.TailPercentage(config)[source]¶
Bases:
recbole.evaluator.base_metric.AbstractMetric
TailPercentage computes the percentage of long-tail items in recommendation items.
For further details, please refer to the paper.
\[\mathrm {TailPercentage@K}=\frac{1}{|U|} \sum_{u \in U} \frac{\sum_{i \in R_{u}} {\delta(i \in T)}}{|R_{u}|}\]\(\delta(·)\) is an indicator function. \(T\) is the set of long-tail items, which is a portion of items that appear in training data seldomly.
Note
If you want to use this metric, please set the parameter ‘tail_ratio’ in the config which can be an integer or a float in (0,1]. Otherwise it will default to 0.1.
- calculate_metric(dataobject)[source]¶
Get the dictionary of a metric.
- Parameters
dataobject (DataStruct) – it contains all the information needed to calculate metrics.
- Returns
such as
{'metric@10': 3153, 'metric@20': 0.3824}
- Return type
dict
- get_tail(item_matrix, count_items)[source]¶
Get long-tail percentage through the top-k recommendation list.
- Parameters
item_matrix (numpy.ndarray) – matrix of items recommended to users.
count_items (dict) – the number of interaction of items in training data.
- Returns
long-tail percentage.
- Return type
float
- metric_need = ['rec.items', 'data.count_items']¶
- metric_type = 1¶
- topk_result(metric, value)[source]¶
Match the metric value to the k and put them in dictionary form.
- Parameters
metric (str) – the name of calculated metric.
value (numpy.ndarray) – metrics for each user, including values from metric@1 to metric@max(self.topk).
- Returns
metric values required in the configuration.
- Return type
dict