Conducting research
in China
Getting to the truth
and making it count
Key take outs
With a population of 1.36 billion and an ever-rising middle class, China represents the biggest potential market in the world. No wonder Western multinationals are keen to understand the Chinese consumer in order to capitalize on the rich opportunity that the Chinese market represents.
But researching the Chinese market is easier said than done. Both academic and market research communities agree that the key challenges to doing this relate to sample quality and social/cultural bias.
Over a period of many years, MetrixLab has conducted numerous projects in China, both multi-country and China-specific studies. Consequently, we have a wealth of experience and data on the Chinese market.
This combined with our status as part of the Macromill Group gives us a unique perspective on the region.
In this white paper, we share the results of our database analysis to provide insight into the key factors at play in this complex market.
Furthermore, we give guidance on how to overcome challenges specific to conducting research in China.
Panel quality in China is variable to say the least. There are two types of panels within China: global companies with a presence in the region and local providers.
Global providers’ panels are typically of a good or moderate quality. However, they tend to be small and centralized around the top tier cities*, which have been ranked as such according to the Chinese central government’s development priorities.
On the other hand, local panels are often of a moderate to low quality, but they are large and have a reach that extends to lower tier small and medium size cities. Once a project has a sample size of 200 or more and an Incidence rate of 5%, it is usually necessary to use a mix of global and local partners for recruitment.
This is particularly important if there is a need for lower tier cities to be included, and to secure a good geographic distribution.
When we compare the results from our standard data cleaning process for large multi-country studies, we can make an assessment about the overall panel quality by country.
With a screen-out rate of 35% (compared to a global average of 9%) and more than four times as many straightliners (22% compared to a global average of just 5%), it is obvious that there are significant challenges around panel quality in China.
As every researcher knows, the quality of research insights can only be as good as the data they are based on.
Strict quality measures are a must. We recommend always including specific control questions. After extensive experience in the region, we have developed a specific set of questions that are designed to screen out low quality respondents in the Chinese market.
This tried and tested approach has become standard practice for us when conducting research in China. And it has had a dramatic impact on panel and consequently data quality.
It’s a fact that the best quality online panels have been recruited in the top tier cities. But if you want to get a more representative view of China as a whole, then you need to recruit panels from the metro cities, rural cities and regions, from every point on the compass.
In practice this means that, whether you like it or not, you do need to use lower quality panels in order to reach the lower tier cities.
Quality measures are therefore paramount in order to improve the reliability of your sample. In some cases, it may be necessary to use a mix of on and offline recruitment in order to ensure your panel is representative.
As in other large regions, you need to ensure that regional differences in China are acknowledged and taken into account in the research design.
One mistake that Western researchers often make is to treat China as a single region. China should no less be treated as one monolithic market than the US should, so plan accordingly.
Two main cultural differences trouble researchers attempting to research the Chinese market: social desirability and claimed ownership.
Left unchecked, each of these has the ability to entirely distort a piece of research. This can render it completely meaningless at best and deeply misleading at worst.
Research has shown that self-administered questionnaires are less prone to social desirability than interviewer-administered questionnaires.
Furthermore, web surveys are more effective at eliciting sensitive information than computer-assisted telephone interviewing (CATI) or interactive voice response (IVR) research.
When it comes to asking consumer questions, we know that full online surveys are best positioned to tackle the social desirability challenge in China.
For this reason, online questionnaires should typically be the first tool in the researcher’s toolkit when considering this market.
Broadly speaking, there are two types of measurement: stated attitudinal measures, whereby respondents are asked what they think, and passive observation measures, which involve evaluating respondents’ behavior.
When considering attitudinal measures, researchers must be aware that Chinese respondents are consistently inclined to agree with given statements and questions.
This phenomenon is culturally specific, and one which researchers ignore at their peril.
Our Copy Test benchmark shows that Chinese respondents tend to answer more positively by a factor of 1.5 on attitudinal measures.
To counteract the impact of this bias, researchers must focus on factual measures and make as little use of self-reported data as possible.
1. Instead, use passive measures such as shelf browsing, T-scope, reel and online eye tracking wherever possible to learn about respondent behavior.
2. Where passive measures are not suitable, choice-based questions such as MaxDiff, image pickers and rankings are preferable.
This is because they prevent respondents from agreeing with all the responses by getting them to choose between answers.
3. If self-reported measures are required, make sure that you use a local benchmark so that you can assess performance in the appropriate context and validate the results.
Be wary of claimed ownership
Just as the Chinese have a strong bias towards social desirability, so they also have a significant tendency to answer purchase-related questions positively.
This Our Copy Test benchmark indicates that Chinese respondents’ claimed purchase intent is three times the global average.
One of the first challenges around purchase related questions is to work out exactly what products Chinese consumers do and don’t actually own.
In order to understand this issue better, we conducted an experiment to test the hypothesis that Chinese respondents over-report brand ownership.
We asked one group of Chinese respondents to upload a picture of the facial skincare products that they own.
At the same time, a second group of respondents was asked to report which brands/products they own.
The findings indicated that brand ownership was over-reported by the self-reporting group. By analyzing the photos uploaded by the first group, we discovered that 10% of respondents owned a Philips product. In the group that self-reported, this figure was almost three times higher at 28%.
Given the Chinese’s tendency to over-report brand ownership, researchers need to apply their knowledge to ensure that results are not incorrectly skewed.
We have devised smart in-survey control measures that correct the inaccurate indication of brand ownership for this reason. Such measures are essential in order to generate valid results.
This is particularly relevant in shopper studies, and usage and attitude (U&A) studies, where it is necessary to filter ‘real’ owners of a specific brand or product from ‘claimed’ owners.
Our experience in researching the Chinese market clearly shows that this is a country where researchers need to work hard to reach the truth.
Cultural differences, geography and social attitudes all conspire to create a situation where consumers’ responses, however well intentioned, must not be taken at face value.
The old adage about insights only being as good as the data they are based on is true.
But, as we have shown, there are a number of systemic challenges that must be overcome in each and every research brief if the resulting insights are to have any true commercial value.
Authors
Roeland Woudstra
Client Director
The Netherlands
Clare Chai
Managing Director
China
Learn more
Discover how we can get you closer to the truth of the Chinese market – and make that intelligence count. Contact us.
About MetrixLab
MetrixLab is a fast-growing global market research and insights company that’s challenging the status quo of insights. By blending evolving technology with passionate experts, MetrixLab helps global and local brands to drive more impact, and forges partnerships to drive sustainably equitable growth. From creative testing to brand tracking, and packaging to e-commerce optimization, MetrixLab’s range of solution suites adapts to fit all types of budgets, timelines and business needs.
Active in more than 90 countries, MetrixLab is a proud partner to more than half of the world’s top 100 brands and part of the Macromill Group.
© Perfectly Possible Designs Ltd
Authors
Roeland Woudstra
Client Director
The Netherlands
Clare Chai
Managing Director
China
Learn more
Discover how we can get you closer to the truth of the Chinese market – and make that intelligence count. Contact us.
.
About MetrixLab
MetrixLab provides consumer insights that drive smarter business decisions. A truly global digital research agency, we pioneer new technologies and integrate multiple data sources to push the boundaries of research. This enables our experts to provide high-quality insights at scale, at speed and for an unparalleled value. Our passion, expertise, and solutions enable our clients to succeed at product innovation, brand engagement, and consumer value.
Active in over 90 countries, MetrixLab is a proud partner of more than half of the world’s top 100 brands and part of Toluna.
© Perfectly Possible Designs Ltd
