Skip to content

Cross Cultural Response Bias

12 January, 2010

I was recently working on a multidisciplinary project that had been written by an academic working in marketing. The team was made up of academics with backgrounds in marketing, economics, sociology and myself with a background in anthropology.  Among the objectives of this project was the development of a new methodological approach to reducing cross-cultural response bias in surveys. With a background in anthropology I felt dubious about the likelihood of realistically reducing cross-cultural response bias.  The basic line of my thinking was that there must be so many factors that would create some sort of bias, such as age, class, gender, experience … that attempting to rule out one factor that may alter an individuals understanding of the question and choice of appropriate response in a survey may be quite futile. Despite my reservations I came up with the following methodological approach:

In order to reduce cross-cultural bias as far a possible four specific steps will be taken, first to identify where cross-cultural bias arises and then to address these differences in creating a larger scale quantitative survey.

  1. Focus groups will be used to identify relevant issues.
  2. Issues raised in the focus groups will be used to construct simple surveys.
  3. The participants will be given these simple surveys during in-depth interviews and asked to fill them out. Interviews will be used to discuss the issues raised in the focus groups and used in the surveys including some questions that exactly match the survey.
  4. Data produced from the surveys and the interviews will then be compared to identify where there are gaps between the response given in the interview, and that given in the survey.

This information will then be used to inform creation of the larger scale surveys to be used during the quantitative phase of the research, both in terms of how the questions are phrased, the issues addressed in the survey questions, and the type of response required.

I have to admit that I felt a bit of an anthropological twinge of ownership of the term “culture” when I first read the objective and initially felt a little offended at seeing the term used to bluntly and unquestioningly. Anthropologists tends to question how “culture” is used (see Adam Kuper’s work for a thorough explanation here), as well as their own supposed authority on the matter. And as a result of this mixture of guilty ownership, odd offence and my general feeling of the futility of attempting to reduce cross-cultural bias, I felt a little convinced that approach outlined above is quite useless. So I thought I would bring it to this excellent blog and ask the audience and participants out there for some feedback. Harsh criticism is welcome and I am interested to hear from any disciplinary background.


8 Comments leave one →
  1. 13 January, 2010 3:26 am

    I can’t help but wonder whether the next step is to move on towards reducing ‘unawareness’ bias? Cultural bias, let alone all the forms of hybrid cultures and sub-cultures, is indeed a significant factor, yet, the question is, to what degree are we (researchers and participants) aware of cultural influences? How easily can we disentangle them?

    Even if we are experts in analysing age, gender, social class and other social markers – the fine lines between culture, religion and political influences may fall apart when we start acknowledging that culture is not a static concept. It means different things to different members of the same cultural group.

    Methodologically, this is a question of emic versus etic approaches and also raises the issue of imposed ethnocentrism. Classic examples of cultural bias are IQ tests and child-rearing practics – usually based on research design that is ultimately grounded in the cultural assumptions of the researchers conducting them. From my viewpoint, resorting to participant-validated research is the most promising approach as it ensures culturally-shaped data is not lost in standardised questionnaires nor in interpretations.

    Thanks for sharing.

  2. 13 January, 2010 8:04 am

    Great points about the difficulty of operationalizing culture when we, as anthropologists, have a much more difficult and skeptical view of what ‘culture’ is.

    I think you’ve done a really good job of moving toward operationalizing it. Because, despite our difficulties with the concept, there IS a there there. Cultural difference exists. We just don’t know how to generalize it.

    At any rate, you don’t need to reinvent the wheel on the issue of factoring in cultural differences to large-scale quantitative surveys. People such as Kathy Ochs, Tom Fricke, and Bill Dressler have been working in this for some time (Fricke’s in demography and anthro; Ochs and Dressler do a lot of public health work cross-culturally). The advantage to these large-scale surveys is that you potentially come up with work that allows cross-cultural comparison. But if you change the survey instruments too much (too accommodate cultural difference), then the results are not comparable to previous surveys’ results.

    In studying health and medical behavior as related to class and worldview in Brazil, Bill Dressler went through a similar process. One difference: he started with free listing (“list every kind of tree you know”) & analyzed that (use Anthropac); did pile-sorting; discussed all of these results in focus groups; and then used that information to construct the surveys. He came with some very powerful and interesting results. Plus, people enjoy free listing and pile sorting.

    I agree with britbohlinger – it’s promising to try to come up with culturally appropriate survey instruments.

    For a start on Bill’s work, see:

    Best, Kate Gillogly

  3. costa permalink
    13 January, 2010 10:46 am

    I think this is an interesting topic. Attempting to source out “bias” in the participants responses and filtering it through the “creativity” of the social researcher’s research design and interpretation of participant’s bias – ambitious at best. Somewhat I see here an attempt to revamp positivism.

    Traditionally “bias” is seen in a negative light often refer to as:

    “…to denote one particular source of systematic error: that deriving from a conscious or unconscious tendency on the part of a researcher to produce data, and/or to interpret them, in a way that inclines towards erroneous conclusions which are in line with his or her commitments” (Hammersley & Gomm, 1997)

    How can we resolve “bias” that comes with social research?

    It has become a common feature in ethnographic work, a section dedicated to standpoint approach to locate the researcher’s bias. To me at times this reads as a sort of “peanut label”: Warning this ethnographic piece may contain traces of the social researcher. In many ways standpoint is a response to the collapse of realism and the idea that a social researcher had a privilege access to social “truth” yielded from their participants.

    Certainly attempting to locate “bias” in participant’s responses is multifaceted. How can we (in a limited time) come to understand the participant’s bias? They are multiple? There is an acknowledgement that the data source from participants is subjective to their interpretation of reality – attempting to reduce this by using the researcher as a medium may not increase validity.

    How about recognising “bias” as a positive feature? It reveals important aspects of a cultural reality that are hidden from other perspectives and it is an inevitable feature of qualitative research.

    I understand social research producing knowledge conveyed under particular descriptions and discourses. I find it hard to see how we can remove cultural bias in social research or minimise it by screening participant’s responses. There are so many assumptions in that model, like the researcher can identify unconscious prejudices and cultural bias.

    Nevertheless I think it serves to flag issues of conscious and unconscious bias both on the side of the social researcher and that of the participants. Recognising the multiplicity of bias and not seeing “bias” as a systematic error.

  4. Palac permalink
    13 January, 2010 6:16 pm

    Fine, discrepancies between the responses given in the survey and the responses given in the interview would flag the existence of a possible bias. I’m guessing that the idea is for the researcher to determine which one is the culturally biased view, the one from the interview, the one from the survey or the one from the focus group. Yet, how is it possible to pinpoint such a thing? After all, is it not the researcher (during the interview), the other respondents (during the focus group) and the respondent’s specific history (during the survey) all sources of cultural influence or “bias” on the answers?

    At the end, there is something odd about this approach, and it’s not that it lies on a positivist epistemology. To say that such an approach is wrong just because it believes in positivism is like saying that religion is wrong because it has no scientific proof. What is really odd about the intention to reduce cross-cultural bias is that it assumes that the real answers and valuable information are somewhere beyond the cultural. Cannot we consider that, even within a positivist framework, the real responses are always shaped by the environment? Wouldn’t it be more productive to think of collaborative research models that frame responses in cultural contexts, instead of reductionist models that desperately try to run away from them?

  5. M.Stockey-Bridge permalink
    15 January, 2010 8:13 pm

    Thank you all for the thoughtful feedback and insights!

    Kate Gillogly thank you for the references, very interesting! I do wonder if we can generalise cultural difference, rather than how we can generalise cultural difference. Perhaps this is one of those points in Anthropology between the philosophical and the practical?

    One of my concerns lies in how ‘culture’ is used. In marketing for example, ‘culture’ is, in this case, a variable that may affect the reliability of the results of a quantitative study. It is a more static and dare I say basic understanding of ‘culture’. As britbohlinger points out culture is far more complex then this representation. So operationalising this simple understanding of culture, from an anthropological perspective, seems contradictory to me. The pretext that all people from one cultural background will respond in the same way, or display a similar bias – this universalising presupposition is one of the key problems in my opinion. As Costa points out it is this positivist approach, the idea that research can be proven by empirical means, that is in itself questionable.

    Palac, it is not so much that the researcher has to work out which is the biased view (that given in the survey or that given in the interview) but more that the interview would be used to “correct” the survey. The interview allows the opportunity for the meaning attached to the survey question to be justified i.e. it can be reworded during the interview if it seems to be unclear to the participant, and at the same time the researcher has the opportunity to probe the participant a little more in asking the same questions asked in the survey in order to find whether the survey questions meanings were understood in the way that the researcher intended. I think the problem is more that once these meanings have all been straightened out they would be used in a larger survey under the assumption that all people of a particular cultural background would understand the meaning in the same way. But overall I agree with you, there is something odd about the approach in itself…

    Wouldn’t it be more productive to think of collaborative research models that frame responses in cultural contexts, instead of reductionist models that desperately try to run away from them?

    This is a great question, but would it be applicable to quantitative scale surveys?

  6. Palac permalink
    16 January, 2010 3:14 pm

    You’re right, quantitative scale surveys are probably not the best place to start from if what we want is to explore culture, not as a source of bias, but as a source of value, not as a world of confusion and misinterpretation, but as one where social meanings are shared across self-contained “cultures” and measurable profiles.

    Thanks again for offering your work for discussion. It’s a brave thing to do, even more in a blog with a dominant qualitative mindset.

  7. Kate permalink
    18 January, 2010 2:19 pm

    Re: culture and what I see as your concern that ‘culture’ is being reified (yes? no?) – of course! Dressler and others who do this work treat shared culture as a point to be proven, not assumed. They do cultural consonance modeling – what’s shared? what are the variations? can they be associated with class, etc.?

  8. 22 January, 2010 5:25 am

    I believe it would be a lot more more productive to leave the reductionist models behind and elaborate on more inclusive models that acknowledge the cultural parameter/s. Instead of relying exclusively on quantitative scale surveys, though, which strikes me as a way to reinforce the reductionist notion, a triangulatory approach could be a way to reconcile the dilemmas.

    Qualitative data, whether collected by help of 1:1 interviews or focus group interviews or some ethnographic research is hard to beat when it comes to understanding more complex issues. Even more so in case such as culture which is difficult to pinpoint for a broad range of participants as common in surveys. Why not backing up the survey findings and giving interviewees a chance to make a contribution by help of some in-depth insights? I could think of reasons against this but they are predominantly shaped by limitations set by financial and other resources.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: