Whitepaper: Understanding the drivers of online review ratings

One of the biggest challenges facing marketers in all sectors is how to improve their products’ online review ratings. Numerous studies have shown a strong correlation between high ratings/satisfaction levels and sales. Understanding what drives positive reviews and ratings is therefore paramount.

We share the findings of a client study that used this approach to assess the performance of a competitor’s activity tracker/health watch. The aim of this two-country study conducted in the US using Amazon.com and China using JD.com was to:

• Identify the topics/drivers of both positive and negative consumer appraisals
• Determine the influence of those topics on review ratings
• Establish the differences between Chinese and American consumers’ reviews of both models of fitness watch

Realizing the power of augmented analysis

Until now, marketers have used customer care dashboards to monitor ratings, reviews and social media. Traditionally, automated counts have been used to assess the product ratings and reviews achieved. Such tools are useful for uncovering insights quickly, but these typically lack real depth. Where deeper insights have been needed, human analysis has been used. This method, however, is more costly and time-consuming. In short, both methods have their strengths and weaknesses.

In response to this problem, we’ve developed a robust and scalable approach that combines the best of both methods. Using a combination of groundbreaking AI technology and human analysis, we provide detailed insight into the subject matter of reviews. This augmented intelligence provides a more accurate picture of what drives high ratings and positive reviews for products.

Research methodology

We used this approach for one of our clients to assess the performance of a competitor’s activity tracker/health watch. The research was carried out in three distinct phases:

First, we gathered a total for 4,000 reviews for the selected activity tracker. These consisted of 2,000 reviews per fitness watch in each market. Three-star reviews, which are considered neutral, were excluded from the sample. By doing so, we could focus on the extreme ratings/satisfaction levels. That is, the positive four/five-star rated reviews and the negative one/two-star rated reviews. This approach enabled us to obtain a better understanding of the motivations behind the positive and negative reviews in each market. We then identified and clustered the core subjects of reviews in the US and China using topic modelling.

Next, we used statistical analysis to ascertain a topic’s influence on the rating. This enabled us to identify the drivers of both positive and negative ratings and reviews in both countries.

Finally, human analysis was used to reveal the core topics that had the power to deliver high and low ratings. Sentiment analysis was conducted at the level of individual posts and based on complete reviews. This new approach enabled us to pinpoint the key drivers of high and low ratings for each of the fitness watches in each market.

Uncovering the core topics for (dis-) satisfaction and the impact on the
Amazon/JD star rating

The drivers of positive reviews
Fig. 1 – Identifying the topics of positive reviews in the US

In the US, love for the product and positive experiences relating to feature/software upgrades both emerged as key topic clusters (Fig. 1). Together, they accounted for 66% of the positive reviews about both models. More than one in five (21%) posts were about love for the product. 19% of posts were about positive experiences relating to feature/software upgrades. Specifically, customers liked the improved visuals and designs offered by the upgrades.

In China, the brand and quality of the product was the chief subject of positive reviews. Indeed, we identified that a significant number of Chinese consumers buy the product simply because of the brand. This topic accounted for 28% of positive reviews about both models.

The drivers of negative reviews

In the US, the durability emerged as the chief topic of negative comments. This accounted for 9% of such reviews. Consumers mainly complained that their watch broke after a short time.

In contrast, Chinese consumers were found to be more likely to complain about customer service and pricing. Consumers commented that the prices fluctuated and were too high. However, when the product was discounted, buyers found the reduced price as more acceptable and their reviews were also more positive. We discovered that offering a discount only solved the first barrier of the purchase process. The main obstacle was that customer service employees failed to help consumers activate their discount. This factor was preventing the positive reviews that can increase brand and product preference, and ultimately drive purchase.

A key insight was the difference in reviews between the US and China about the watch wristbands. The quality of the wristband was a key cause of negative reviews by American consumers. Chinese consumers, on the other hand, talked about the wristbands as a fashion item. They were not concerned by its poor quality as they bought a replacement new, fashionable wristband either at the same time, or shortly after, the purchase of their device.

The influence of topics on ratings

Having identified the core topics of reviews in both markets, we used statistical analysis to pinpoint their influence on ratings.

Fig. 2 – Correlation between topics and star ratings in the US

Fig. 3 – Correlation between topics and star ratings in China

Charts Fig. 2 and 3 show the correlation between the subjects discussed in reviews and their star rating. In the US, love for the product and feature experiences can be seen to correlate strongly to four/five–star ratings. In China, it is the brand and quality of the product that achieves the highest ratings. In both countries, however, the strongest correlations appear between the topics of negative reviews and one/ two–star ratings.