As I had the chance to mention in a previous post, I am currently enrolled in the MSc program in Information Science and Data Analytics at Northumbria University. One day I will probably write a post about the experience, for now let’s say that I am doing quite all right juggling between work and study (less so blogging).

As a mid-term assignment for a module of this semester, we were asked to sketch the outline of a possible research proposal for a Master’s dissertation. Before ditching the idea for reasons that I will explain below, I spent a few weeks reading literature about the question “What makes an online review helpful”, which seems to have received exponential attention in the last few years – as many topics in data science, it basically didn’t exist ten years ago, and now there are tens of papers on the issue.

The business implications behind the question are quite relevant – according to a often-cited statistics, the introduction of a button to evaluate if a review was helpful or not had brought Amazon 2.7 billion $ of extra revenue.

From what I could understand, it is possible to roughly distinguish three main approaches that have been used to answer the question.

The first, explanatory in nature, and more related to traditional marketing research rather than data science, attempts to answer the question by means of surveys, and in some instances with qualitative approaches, directly asking respondents to identify the elements they consider most important in an online review. For the nerds out there, Raffaele Filieri seems to be one of the most prolific authors in this regard.

The second, which I had the impression is the most explored, is predictive in nature, with researches attempting to predict the perceived helpfulness of an online review based on a number of features of the review (or the review writer) itself. We could further divide this group according to a number of factors, like the prediction technique used, or the type of features selected. For the nerds, Hong et al. 2017 and Ocampo Diaz and Ng 2018 offer a comprehensive overview of such predictive studies.

And we could further distinguish a number of studies in a third group, loosely accommodated by the fact that they have adopted a more nuanced perspective, focusing mostly on meta-analysis and on moderating factors to account for different results. As an example, given the apparently contradictory results of some of the predictive studies, Hong et al. 2017 have attempted to account for these differences via a meta-analysis of the studies, accounting for several moderating factors such as the operationalization of the helpfulness measure, the nature of the database, of the products, and so on. Another example is offered by Lu et al. 2018, who have considered the construction of review helpfulness as a dynamic system, analysing how timing affects the weight that various features have in predicting the helpfulness of a review.

This last study appeared to me particularly interesting as it considers the formation of review helpfulness not as a fixed result of an amount of factors, but as the result of a dynamic interplay between reviewer, users and the platform. In fact, most of the predictive studies in the second group have had the tendency to ignore the role of the final user (so to speak, the review evaluator) in the definition of review helpfulness. These studies appear to operate on the assumption that review readers basically share common features, or that their different features don’t play a role in helpfulness evaluation.

I had been able to identify only one study (Tang et al. 2013), which directly hypothesized that the differences in helpfulness scores are not necessarily a consequence of review quality, but of difference of opinion between users, and to advocate for user-specific helpfulness prediction.

As that study seemed a rather isolated example, and its results encouraging, a proposal aiming to find the answer to the question “Does the (perception of) helpfulness change depending on the (features of the) reader?”, seemed a good, if quite ambitious, one.

And that’s basically where I had to dump the idea – and I guess that’s the reason why the topic wasn’t researched much further.

Long story short, there are no data (at least, not freely, or legally, available) tracking information about a review evaluator. Tang et al. 2013 relied on datasets tracking information about the relations review writer-reader, and past purchases the two had in common, within two specific social networks (which are now defunct), but not other personal information of users.

There appear to be no other service making information about “review evaluators” available. I guess companies like Facebook, Google or Apple, with their extensive access to personal information (the issue here is not how legitimate or not) are all too busy doing that but know better than releasing them. Which in insight makes total sense, and on the other hand, in times of GDPR and renewed (totally justified) attention towards personal data, it seems unlikely that the situation will change much – and I guess I feel quite relieved about that.

From the research point of view, the question could still be answered by means of more traditional quantitative approaches like those mentioned above; or it can be tackled by means of some sort of social experiment on a sample of volunteers. Both approaches seem to have strengths and weaknesses, but ultimately seemed way above the scope of a Master’s dissertation and I didn’t feel much inclined towards either of them, so in the end I just opted for a different topic.

This seemed however an interesting example of the limits that research in data science can incur in. I have not researched the topic deeply enough, but I am tempted to say that any predictive technique ignoring the features of the review evaluator, and the dynamic interplay between reviewer, users and the platform, will achieve partially satisfactory results. On the other hand, it seems that exploring the topic in-depth will be impossible for whoever of us is not Facebook.

Main references

Hong, H.,  Xu, D., Wang, A., Fan, W. (2017), “Understanding the determinants of online review helpfulness: A meta-analytic investigation”, Decision Support Systems 107(2017), pp. 1-11, doi: https://doi.org/10.1016/j.dss.2017.06.007

Lu, S., Wu, J., Tseng  S.L. (2018), „How Online Reviews become helpful: a dynamic perspective”, Journal of interactive marketing, 44 (2018), pp. 17-28, doi: https://doi.org/10.1016/j.intmar.2018.05.005

Ocampo Diaz, G. and Ng V. (2018), Modeling and prediction of online product review helpfulness: a Survey, Proceedings of the 56^th Annual Meeting of the Association of Computational Linguistics, Melbourne, Australia, July 15-20 2018, pp. 698-708, available at http://aclweb.org/anthology/P18-1065

Tang, J., Gao, H., Hu, X., Liu, H., (2013), Context-Aware Review Helpfulness Rating Prediction, RecSys ’13. Proceedings of the 7^th ACM conference on Reccomender systems, Hong Kong, China, October 12-16 2013, pp. 1-8, available at http://www.public.asu.edu/~huanliu/papers/recsys13-jtang.pdf