How Do We Measure Cross Cultural Competence?

The Institute for Cross Cultural Management recently presented a paper at the 2^nd annual International Conference on Cross Cultural Decision Making in San Francisco California which discussed this issue.  The research was conducted as part of a funded project for the Defense Equal Opportunity Management Institute (DEOMI).  Cross cultural competence (or 3C) is the ability to adapt and function effectively in new cultural environments.  It is why one executive can travel overseas and seem to fit in quickly, while another might choose to hide in their hotel and eat McDonald’s.  There are many measures of 3C available, and we were asked by DEOMI to evaluate them.  In general, we found these measures were ineffective and discussed the reasons why in our paper.

Measurement Problems

First, most available measures have poor construct validity.  Essentially that means they don’t measure what they say they do.  Some measures define 3C as an ability, others as a personality trait, and others as a set of attitudes.  Research has treated 3C inconsistently and until recently no dominant framework explaining the competencies had emerged.  That has recently changed with a 3C framework authored by Dr. Patrice Reid and colleagues at DLNSEO and DEOMI (Dr. Reid is a graduate of the FIT I/O program).  This framework defines 3C as 6 inter-related skills that are supported by a number of abilities and dispositions that enable effective performance in intercultural settings.  We expect that measures of 3C that are developed in the future will use this, or similar, frameworks.

Predicting the Future

3C has typically been measured using surveys.  However we really aren’t interested in a person’s performance on the survey.  The survey is only useful if it predicts the person’s behavior in the future.  If the measure isn’t related to how a person will actually perform in an intercultural setting, it isn’t very helpful. Unfortunately most of the measures we evaluated did not demonstrate evidence of a connection with future performance.  At best they were related to other measures of adjustment completed by the same person.

I Am Competent, I Swear I Am!

Another problem with the measures of 3C we evaluated was that they relied heavily on self-report measures.  Unlike traditional test items which have a right and wrong answer, self-report items ask the respondent to indicate their level of agreement on a statement that describes effective or ineffective behavior.  First of all, many of the items on existing measures describe situations or thoughts that may be unfamiliar to the person answering the survey.  So they respond with the first thought that pops in their head.  Second, there is no way to determine if the respondent has been honest.  Since these measures may be used to select people for important assignments, as many as 30% of respondents simply say what you want to hear by faking the measure.

Don’t Tell Me, Show Me

Our research suggests that to improve the measurement of 3C we’ll need to use frameworks with good construct validity that are measured in a fashion other than self-report.  Measures that require more active involvement from the respondent are more likely to accurately assess 3C, and ultimately better predict whether a person will be successful in a new cultural setting.  So we are now designing measures of 3C that are more behavioral and require people to respond to more dynamic test stimuli than traditional paper-and-pencil measures. And unlike situational judgment tests, which ask “what would you do if” kinds of questions, our measures assess 3C in a general sense rather than in a cultural specific context.  That flexibility allows the measures to be useful to predict success in a number of cultures rather than just one.

Why is measuring 3C so important? First if we can identify people who are more competent they are better candidates for overseas assignments.  Expatriate failures cost organizations hundreds of millions of dollars every year.  Second 3C can be trained, so we will need a good measure to evaluate the effectiveness of training to determine which trainees benefited the most.

So that is a bit about how we measure cross cultural competence.  How do you tell when someone is flexible enough to succeed in new cultural circumstances?