[1907.06902] Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches

Deep learning techniques have become the method of choice for researchers working on algorithmic aspects of recommender systems. With the strongly increased interest in machine learning in general, it has, as a result, become difficult to keep track of what represents the state-of-the-art at the moment, e.g., for top-n recommendation tasks. At the same time, several recent publications point out problems in today's research practice in applied machine learning, e.g., in terms of the reproducibility of the results or the choice of the baselines when proposing new models. In this work, we report the results of a systematic analysis of algorithmic proposals for top-n recommendation tasks. Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., b

41 mentions: @Maurizio_fd@kieranrcampbell@bindureddy@EeHRN@yogthos@lasserith@mgdm@SileyeBaba
Date: 2019/07/22 23:16

Referring Tweets

@nutanc This paper shows enough proof of what I have been feeling, deep neural networks have not provided solutions to any concrete problems. Most of the time, regular approaches are good enough. https://t.co/KTNQ3cIWIb
@EeHRN Trending in critical #AI Out of 18 algorithms presented at top conferences only 7 could be reproduced, of which 6 could be outperformed using simpler methods https://t.co/xzZeHuTigC With so much capital & kudos at stake no wonder companies are faking it #DeepLearning #WizardOfOz https://t.co/xnhYBiwN9N
@yogthos Of 18 algorithms presented at top-level conferences only 7 could be reproduced with reasonable effort. 6 can often be outperformed with comparably simple heuristic. The last did not consistently outperform a well-tuned non-neural linear ranking method. https://t.co/HHUjdxgxmX
@sytelus Ouch! This paper tried replicating 18 papers in the area of recommendations using deep learning but could replicate only 7. And 6 of those 7 could be outperformed with simple heuristic while 7th could be outperformed by well-tuned non-neural method. https://t.co/VvqBXClMIY
@bindureddy Only 7 of the 18 papers in recommendations were reproducible and of which, only one of them beat the state of the art. This encapsulate the core problem with machine learning today - https://t.co/JxYlffTRBT
@lasserith https://t.co/1EGs4JY7zA 'We considered 18 algorithms that were presented at top-level research conferences in the last years... 7 of them could be reproduced with reasonable effort.... turned out that 6 of them can often be outperformed with comparably simple heuristic methods'
@SileyeBaba Interesting read about results reproducibility, a well known reccurent problem https://t.co/VJKm4I575E
@Maurizio_fd Our analysis of 18 neural #recsys shows only 7 are reproducible and 6 can be outperformed by simple baselines "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches" https://t.co/6wPBYCTGpd #sigir19 #TheWebConf #kdd19 #DeepLearning

Related Entries

Read more [1906.00091] Deep Learning Recommendation Model for Personalization and Recommendation Systems
5 users, 2 mentions 2019/09/13 00:50
Read more [1902.07243] Graph Neural Networks for Social Recommendation
0 users, 1 mentions 2019/08/31 17:17
Read more DLRM: An advanced, open source deep learning recommendation model
2 users, 65 mentions 2019/07/03 03:48
Read more GitHub - MaurizioFD/RecSys2019_DeepLearning_Evaluation: This is the repository of our article publis...
0 users, 2 mentions 2019/09/16 15:47