Customize Metrics

Here we present how to develop a new metric, and apply it into RecBole. Users can implement their customized metrics and combine the metrics with others for personal evaluation.

Here, it only takes three steps to incorporate a new metric and we introduce them step by step.

Create a New Metric Class

Then, we create a new class in the file ~recbole.evaluator.metrics and define the parameter in __init__():

from recbole.evaluator.base_metric import AbstractMetric
class MyMetric(AbstractMetric):
    def __init__(self, config):

Set properties of the metric

After that, we set the properties of metrics:

Set metric_need

It is a list that contains one or multiple string that corresponding to needed input of metrics. For now, we support 9 inputs for metrics including both recommendation results and information of dataset which are listed below.




K recommended items for each user


Boolean matrix indicating the existence of a recommended item in the test set and number of positive items for each user


Mean ranking of positive items for each user


Pure output score


Number of item in dataset


Number of user in dataset


Interaction number of each item in training data


Interaction number of each user in training data


Pure label field of input data (Usually used with rec.score together)

Set metric_type

It indicates whether the scores required by metric are grouped by user, range in EvaluatorType.RANKING (for grouped ones) and EvaluatorType.VALUE (for non-grouped ones). In current RecBole, all the “grouped” metrics are ranking-based and all the “non-grouped” metrics are value-based. To keep with our paper, we adopted the more formal terms: RANKING and VALUE.

Set smaller

It indicates whether the smaller metric value represents better performance, range in True and False, default to False.


If we want to add a ranking-based metric named YourMetric which needs the recommended items and the total item number, and the smaller YourMetric indicates better model performance, the code is shown below:

from recbole.evaluator.base_metric import AbstractMetric
from recbole.utils import EvaluatorType
class MyMetric(AbstractMetric):
    metric_type = EvaluatorType.RANKING
    metric_need = ['rec.items', 'data.num_items']
    smaller = True

    def __init__(self, config):

Implement calculate_metric(self, dataobject)

All the computational process is defined in this function. The args is a packaged data object that contains all the result above. We can treat it as a dict and get data from it by rec_items = dataobject.get('rec.items') . The returned value should be a dict with key of metric name and value of final result. Note that the metric name should be lowercase.

Example code:

def calculate_metric(self, dataobject):
    """Get the dictionary of a metric.

        dataobject(DataStruct): it contains all the information needed to calculate metrics.

        dict: such as ``{'mymetric@10': 3153, 'mymetric@20': 0.3824}``
    rec_items = dataobject.get('rec.items')
    # Add the logic of your metric here.

    return result_dict