A Novel Classification Framework for Evaluating Individual and Aggregate Diversity in Top-N Recommendations

Jennifer Moody, David H. Glass

Research output: Contribution to journalArticle

7 Citations (Scopus)

Abstract

The primary goal of a recommender system is to generate high quality user-centred recommendations. However, the traditional evaluation methods and metrics were developed before researchers understood all the factors that increase user satisfaction. This study is an introduction to a novel user and item classification framework. It is proposed that this framework should be used during user-centred evaluation of recommender systems and the need for this framework is justified through experiments. User profiles are constructed and matched against other users’ profiles to formulate neighbourhoods and generate top-N recommendations.The recommendations are evaluated to measure the success of the process. In conjunction with the framework, a new diversity metric is presented and explained. The accuracy, coverage, and diversity of top-N recommendations is illustrated and discussed for groups of users. It is found that in contradiction to common assumptions, not all users suffer as expected from the data sparsity problem. In fact, the group of users that receive the most accurate recommendations do not belong to the least sparse area of the dataset.
LanguageEnglish
Pages42:1-42:21
JournalACM Transactions on Intelligent Systems and Technology
Volume7
Issue number3
DOIs
Publication statusPublished - 1 Apr 2016

Fingerprint

Recommender systems
Recommendations
User Profile
Recommender Systems
Metric
User Satisfaction
Experiments
Evaluation Method
Sparsity
Framework
Coverage
Evaluation
Experiment

Keywords

  • Recommender systems
  • recommendation accuracy
  • recommendation
  • quality
  • performance evaluation metrics
  • recommendation diversity
  • collaborative filtering

Cite this

@article{49d8ebae5ca9464ba6871c3a9a9b7063,
title = "A Novel Classification Framework for Evaluating Individual and Aggregate Diversity in Top-N Recommendations",
abstract = "The primary goal of a recommender system is to generate high quality user-centred recommendations. However, the traditional evaluation methods and metrics were developed before researchers understood all the factors that increase user satisfaction. This study is an introduction to a novel user and item classification framework. It is proposed that this framework should be used during user-centred evaluation of recommender systems and the need for this framework is justified through experiments. User profiles are constructed and matched against other users’ profiles to formulate neighbourhoods and generate top-N recommendations.The recommendations are evaluated to measure the success of the process. In conjunction with the framework, a new diversity metric is presented and explained. The accuracy, coverage, and diversity of top-N recommendations is illustrated and discussed for groups of users. It is found that in contradiction to common assumptions, not all users suffer as expected from the data sparsity problem. In fact, the group of users that receive the most accurate recommendations do not belong to the least sparse area of the dataset.",
keywords = "Recommender systems, recommendation accuracy, recommendation, quality, performance evaluation metrics, recommendation diversity, collaborative filtering",
author = "Jennifer Moody and Glass, {David H.}",
year = "2016",
month = "4",
day = "1",
doi = "10.1145/2700491",
language = "English",
volume = "7",
pages = "42:1--42:21",
journal = "ACM Transactions on Intelligent Systems and Technology",
issn = "2157-6904",
number = "3",

}

TY - JOUR

T1 - A Novel Classification Framework for Evaluating Individual and Aggregate Diversity in Top-N Recommendations

AU - Moody, Jennifer

AU - Glass, David H.

PY - 2016/4/1

Y1 - 2016/4/1

N2 - The primary goal of a recommender system is to generate high quality user-centred recommendations. However, the traditional evaluation methods and metrics were developed before researchers understood all the factors that increase user satisfaction. This study is an introduction to a novel user and item classification framework. It is proposed that this framework should be used during user-centred evaluation of recommender systems and the need for this framework is justified through experiments. User profiles are constructed and matched against other users’ profiles to formulate neighbourhoods and generate top-N recommendations.The recommendations are evaluated to measure the success of the process. In conjunction with the framework, a new diversity metric is presented and explained. The accuracy, coverage, and diversity of top-N recommendations is illustrated and discussed for groups of users. It is found that in contradiction to common assumptions, not all users suffer as expected from the data sparsity problem. In fact, the group of users that receive the most accurate recommendations do not belong to the least sparse area of the dataset.

AB - The primary goal of a recommender system is to generate high quality user-centred recommendations. However, the traditional evaluation methods and metrics were developed before researchers understood all the factors that increase user satisfaction. This study is an introduction to a novel user and item classification framework. It is proposed that this framework should be used during user-centred evaluation of recommender systems and the need for this framework is justified through experiments. User profiles are constructed and matched against other users’ profiles to formulate neighbourhoods and generate top-N recommendations.The recommendations are evaluated to measure the success of the process. In conjunction with the framework, a new diversity metric is presented and explained. The accuracy, coverage, and diversity of top-N recommendations is illustrated and discussed for groups of users. It is found that in contradiction to common assumptions, not all users suffer as expected from the data sparsity problem. In fact, the group of users that receive the most accurate recommendations do not belong to the least sparse area of the dataset.

KW - Recommender systems

KW - recommendation accuracy

KW - recommendation

KW - quality

KW - performance evaluation metrics

KW - recommendation diversity

KW - collaborative filtering

U2 - 10.1145/2700491

DO - 10.1145/2700491

M3 - Article

VL - 7

SP - 42:1-42:21

JO - ACM Transactions on Intelligent Systems and Technology

T2 - ACM Transactions on Intelligent Systems and Technology

JF - ACM Transactions on Intelligent Systems and Technology

SN - 2157-6904

IS - 3

ER -