Overview of evaluation module
The function of evaluation module is to implement commonly used evaluation protocols for recommender systems. Since different models can be compared under the same evaluation module, RecBole is useful to standardize the evaluation of recommender systems.
Evaluation protocols and settings
In the following table, we summarize the evaluation settings that RecBole supports, where the first four rows correspond to the dataset splitting methods and the last two rows correspond to the ranking mechanisms.
| Notation | Explanation |
|---|---|
| RO_RS | Random Ordering + Ratio-based Splitting |
| TO_LS | Temporal Ordering + Leave-one-out Splitting |
| RO_LS | Random Ordering + Leave-one-out Splitting |
| TO_RS | Temporal Ordering + Ratio-based Splitting |
| full | full ranking with all item candidates |
| uniN | sample-based ranking: each positive item is paired with N sampled negative items |
In RecBole, the evaluation module consists of two major classes of Ranking-based Evaluator and Value-based Evaluator. Both classes can work with the above evaluation settings. For `t\op`-`k` item recommendation, the implemented evaluation settings cover the various settings of our earlier work in (Zhao et al., 2020), where we have studied the influence of different evaluation protocols on the performance comparison of models.
Ranking-based Evaluator
Ranking-based Evaluator is mainly used for the evaluation of ranking tasks. At present, RecBole supports
the
evaluation
metrics such as HR (hit ratio), NDCG, MRR, recall, MAP and precision.
These metrics are often computed in a truncated version with a length of `k`, such as
Recall@5 , NDCG@10
For more information about the `t\op`-`k` metrics, see also: metrics
Value-based Evaluator
Value-based Evaluator is mainly used for rating prediction and click-through rate prediction. Currently, it supports the evaluation metrics such as AUC, logloss, MAE and RMSE.
For more information about the Loss metrics, see also: metrics
Acceleration Strategy for `t\op`-`k` Evaluation
Computing `t\op`-`k` evaluation metrics is usually time consuming. The basic reason lies in that one need to exhaustively estimate the score for each user-item pair. Since the methods of score estimation varies across different models, it is not easy to optimize the entire evaluation procedure in a general way. Therefore, we mainly focus on the step of selecting and generating `t\op`-`k` items given the ranking scores.
The following illustration is the proposed acceleration strategy for `t\op`-`k` item evaluation. Here, `u0, · · · , u5` indicate six users; black, blue and grey boxes denote training items, test items and other candidate items, respectively. The entries in `A` correspond to the indices of the identified `k` items by a recommendation algorithm. The entries in `B` correspond to indicate the existence of an item in the test set (blue box in the Figure). The entries in `C` correspond to indicate whether the corresponding position of `A` matrix is predicted correctly.
Reassigning the score matrix
- reshaping: Given `n` users and `m` items for consideration, when performing full ranking, we can obtain a `n \times m` matrix `D` consisting of the confidence scores from a model over the entire item set. When performing sample-based ranking, we create an `n \times m` matrix `D`, initializing all elements to negative infinity. Then, we fill matrix `D` with the confidence scores over sampled items.
- filling: When performing full ranking with all item candidates, we provide an option to mask the score of training items. If the user choose to mask, the matrix obtained in the above step cannot be directly used for `t\op`-`k` prediction. Our solution is to set the scores of training items to negative infinity, and perform the full ranking over the entire item set without removing training items.
Get the `t\op`-`k` items
We utilize the GPU-version topk() function provided by PyTorch to find the `t\op`-`k` items with the highest scores for users. The GPU-version topk() function has been specially optimized based on CUDA, which is very efficient in our case. With the topk() function, we can obtain a result matrix `A` of size `n\times k`, which records the original index of the selected `t\op`-`k` items.
Calculate the intersection of `t\op`-`k` items and ground truth items
- indexing: We further generate a binary matrix `B` of size `n \times m` to indicate the existence of an item in the test set. Next, we use each row of matrix `A` to index the same row in matrix `B` and obtain a binary matrix `C` of size `n \times K` indicating whether the corresponding entry is a right recommendation or not, which can be implemented efficiently through gather() function provided by PyTorch.
Calculate `t\op`-`k` metrics
Considering that users may want to calculate the results of one metric under different positions (i.e., `k` ) at the same time, we implement the parallel computation of evaluation metrics. The major procedure is given as follows:
- If a user requires to compute the evaluation metrics at several positions, we will maintain a `k` list, and get the `k` items matrix with size `n \times \max (k)`
- In calculation, we use the function provided by NumPy such as
np.cumsum()instead ofnp.sum()to calculate the metrics. Different from the traditional calculation method, our metrics function will returns a list instead of a value, which records the `t\op`-`k` result from `1` to `\max(k)`. - We replace matrix operation with
forstatements. Because we store the `t\op`-`k` items as matrix, we can make full use of the rich matrix functions provided by NumPy to speed-up our calculations.