Does Topic Consistency Matter? A Study of Critic and User Reviews in the Movie Industry

People seated in a cinema watching a movie

Do you often read online reviews before deciding to watch a movie? One of the best ways to decide whether to watch a movie is to check the general reviews and ratings. Some popular movie review websites are Rotten Tomatoes, IMDb (Internet Movie Database) and Metacritic, which allow critics and users to leave their impressions of a movie online. Critics and users do not necessarily have the same opinions about a movie’s quality or focus. This division in opinions could be because of differences in critics’ and users’ expertise or knowledge. 

In a recent study in the Journal of Marketing, a group of researchers investigated whether the difference in topics discussed by critics and users affects box office revenues. Alongside Professor Eunsoo Kim from Nanyang Technological University, PhD candidate MengQi (Annie) Ding, Professor Xin (Shane) Wang and Professor Shijie Lu wrote the paper Does Topic Consistency Matter? A Study of Critic and User Reviews in the Movie Industry.

“Inspired by the movie industry’s long-standing engagement with critics, our research was sparked by what critics and users discuss when reviewing the same movie. Hence, we came up with a measure to observe the review content by both groups,” remarked Professor Eunsoo on their intentions behind the research. 

Rotten Tomatoes website with ratings of movies

Screenshot is taken from the Rotten Tomatoes website

Topic consistency 

The paper proposes a measure called ‘topic consistency’, which captures the degree of overlap between critic and user review content. The objective of the research is to study whether topic consistency can affect movie demand and sales. More importantly, the researchers investigate the implications of topic consistency for movie producers and marketing agencies.

In this study, the researchers sampled movie reviews from Rotten Tomatoes for all movies released from 2013 to 2017 in the United States and further filtered movies that have box office revenue and advertising expenses information. In addition to expressing their judgement of a movie in words on Rotten Tomatoes, critics can give an overall label of either ‘rotten’ (bad) or ‘fresh’ (good). At the same time, users can leave a numeric rating on a scale of 0 to 5 (5 being the highest).

To test the effect of topic consistency on movie sales, the team employed a text analysis model and machine learning. The researchers used Bigram Latent Dirichlet Allocation (Bigram-LDA) because it is effective at detecting names of actors, directors and movie titles, which involve sequential words. Findings show that the correlation between the two groups is 0.45, indicating that users’ topics are not always aligned with those mentioned by critics. The relationship between topic consistency and box office revenue is significant and positive. An increase of one standard deviation in topic consistency for a movie is associated with a 4.6% increase in revenue, all else being equal.

“Upon analysing movie reviews from Rotten Tomatoes, we find that the content overlap between critics and users is positively associated with movie demand. Moreover, this association is more prominent with movies with mediocre review ratings than movies with extremely good or bad ratings. It is also stronger when review ratings by critics are similar to those of users,” said Professor Eunsoo.

View of the entrance of a cinema

Why does topic consistency affect movie demand? 

High topic consistency between critic and user reviews suggests greater information recall because the same topics are repeatedly presented. Such repeated information persuades moviegoers to retrieve these movies at the time of movie selection and increases their willingness to watch a movie. Moreover, according to the encoding variability theory, the same movie attribute can become stronger, clearer and accessible to consumers when exposed in different contexts, that is, in reviews of critics and users, as opposed to only one context. 

Implications for the movie industry 

From the study’s findings, the researchers provided a few suggestions for the movie industry. First, movie producers and marketing agencies should take advantage of the topic consistency effect by identifying similarities and differences between critics’ and general consumers’ responses. If there are commonalities between the two groups’ reviews, they should be leveraged and utilised as a part of the movie’s promotion strategy. 

Next, movie producers and advertisers should consider inducing a common topic or theme for critics and users to discuss since high topic consistency can boost movie revenues. One factor to consider when choosing common topics is that users and critics do not need to have the same opinions for topic consistency to affect movie sales positively. Movie producers and advertisers can foreground discussion topics by highlighting narratives in promotional activities such as movie trailers, posters, blogs and commercials. This will naturally lead critics and users to address the topics in their reviews. 

When asked whether the mediation of producers and advertisers is biased, Professor Eunsoo mentioned, “Well, marketers or producers should not boldly say that a specific aspect or a topic of the movie is worthwhile focusing on without having any justification or confidence in the movie. If the movie cannot support the topic that you are promoting, I feel that it will backfire in the long run as moviegoers will eventually leave a bad review online.”. 

A laptop showing a list of movies and shows on Netflix website

Can topic consistency be applied to other settings? 

With the rise in online activities and e-commerce, many websites encourage users to leave their reviews and ratings online. Beyond movies, topic consistency can be extended to other experiential product reviews, for instance, cosmetics or books. Consumers who are uncertain about purchasing a product can then rely on online reviews to help them judge the product’s quality.

“Investigating topic consistency allows the voices of experts and general users regarding a product to be heard. Managers in other experiential industries should be mindful of the interactions between professionals and general users in their marketing and promotional plan,” remarked Professor Eunsoo.

Note: This research paper was published by the Journal of Marketing, on 7 September 2022.

Eunsoo Kim is an Assistant Professor of Marketing at Nanyang Business School. The two main research streams that she is currently interested in are (1) Quantitative modeling of firm- and user-generated contents and (2) Quantitative modeling of social influence in business and consumer settings.

This research paper is a joint work with the following authors.

MengQi (Annie) Ding is a doctoral candidate in Marketing, at Ivey School of Business, Western University Canada.

Xin (Shane) Wang is a Professor of Marketing, at Pamplin College of Business, Virginia Tech, USA.

Shijie Lu is the Howard J. and Geraldine F. Korth Associate Professor of Marketing, at Mendoza College of Business, University of Notre Dame, USA.