**4. Measurement of the effect of public diplomacy**

If defining the function and effect evaluation criteria of public diplomacy is to define 'what to measure', then we need also to know what methods are needed to measure the effect of public diplomacy, that is, 'the measuring tools', and how to measure the effect of public diplomacy through these methods, that is, 'how to measure'. There

*Heritage*

**Figure 3.**

Since the function of public diplomacy is to create soft power resources, and soft power resources refer to those assets that are accepted or favoured by the people of the target country, the success of a public diplomacy project is judged according to whether or not it improves or enhances the people of the target country's evaluation of one aspect of the implementing country. In other words, what we actually measure is whether or not public diplomacy expands a country's soft power resources.

**3. Functional boundaries and influencing factors of public diplomacy**

Given that the basic function of public diplomacy is to create soft power resources, could it also play a role in all problem areas by changing the perception or evaluation of the people of target countries? And if such restricted areas do exist, can the function of public diplomacy be fully realized in areas where it could play a role? The answer to all of these questions is clearly no. This is the utility boundary,

First, public diplomacy is likely to be of little use in the 'high politics' of territorial security. Whether the public diplomacy of the US in the Middle East since the turn of the twenty-first century or the publicity of the Chinese and Japanese governments against each other's people in recent years with regard to the Diaoyu Islands issue, the effect is not ideal. Obviously, it is not due to the US's public diplomacy work that people in the Middle East do not recognize the legitimacy of the US's invasion and interventions in Iraq, Afghanistan, or even Libya, and nor do the Chinese and Japanese people object to the territorial claims of their own govern-

This reminds us that communication and mutual trust are ineffective when it comes to territorial security, especially territorial disputes, where one party gains a zero-sum conflict at the expense of the other. It will be difficult to persuade their people to accept 'legitimate' interference in their territory and to give up their claims to territory and rights no matter how convincingly countries tell their own stories. Public diplomacy is of little use, therefore, when it comes to the 'high

Second, in the field where public diplomacy could play a role, its effect will still be affected by the following factors. The first is the inevitable structural contradiction between a country's rising power and that of other countries. The negative impact of this factor on the effect of public diplomacy is typified by China's public diplomacy towards Japan. Global View 2008 surveys taken in Chicago showed that the Japanese had the lowest level of favourable views on China, behind those of South Korea, Indonesia and Vietnam. 89% of Japanese respondents said they were either 'somewhat' or 'very' uncomfortable with the idea of China's 1 day becoming

The key reason lies in the semi-structured interviews with Dinnie and Lio. When asked, 'What are the key challenges facing China in its attempts to build a positive reputation within Japan?', one Japanese journalist interviewee admitted,

and possible influencing factor of public diplomacy.

*Author's relation between public diplomacy and soft power.*

ments due to the publicity work of the other government.

politics' of territorial security.

the leader of Asia [14].

**334**

are two main empirical methods of measuring the effect of public diplomacy: one is the sampling survey method, which is aimed at the general public, namely, the public opinion poll; the other is the unstructured interview with a small group of specific people. The non-structured interview, also known as the non-standardized interview, is either a semi-controlled or uncontrolled interview, which can be divided into four types: intensive interview, in-depth interview, objective statement, and symposium [25].

The public opinion poll method and the interview method are two common widely used empirical methods in modern social science research on which there have been many methodological works with regard to their design procedure and implementation steps in the general sense, but which are not discussed in this chapter. The author's specific concern is: when using these two methods to measure the effect of public diplomacy, which problems need to be resolved to ensure the accuracy of the measured results? In this regard, there are at least three aspects worth discussing.

First, who should be chosen for the investigation? Whether a public opinion poll or an interview, the first question involves the selection of respondents. The author's opinion is: pinpoint the direct audience of public diplomacy and extract from this group (as far as possible) the respondents through which to measure its effect. This is because the fundamental purpose of our survey is to see whether or not the people of the target country have changed their attitude towards a certain aspect of the implementing country due to a specific public diplomacy project on the part of the implementing country, rather than a general change in attitude.

Imagine country A conducting public diplomacy through the medium of international students there from country B. After a period of time, we want to know whether or not country A's public diplomacy activity has been effective, so we conduct questionnaires or interviews with workers in country B. In this case, even if the survey results show that workers in country B have never heard of such public diplomacy as conducted by country A, we should not consider it to be invalid, because country B's international students in country A may be much more familiar with such public diplomacy. Similarly, even if the survey results show that the attitude of workers in country B towards country A has improved, we should not assume that the public diplomacy aimed at international students from country B has been effective, because it is possible that the attitude of country B's international students in country A has deteriorated.

For example, the aforementioned interview study on the effect of China's public diplomacy on Japan is flawed to some extent as regards its design, as discussed here [23]. The two academics asked seven interviewees: 'What effect, if any, do you think the establishment of Confucius Institutes in Japan has had on China's reputation?" Obviously, the best and most convincing way of finding out whether or not establishing a Confucius institute in Japan will improve China's reputation is to ask Japanese students who are studying or who have studied at Confucius institutes in Japan. However, the article clearly identified the seven respondents, none of whom were or had been students at a Confucius institute in Japan. Claiming that China's public diplomacy efforts to promote Confucius institutes in Japan are ineffective, therefore, even though all respondents in the group said that they had not noticed or were only slightly aware of the existence of Confucius institutes, is both futile and unconvincing.

Second, which of the two methods of polling and interviewing should be chosen? As regards public opinion polling, it has the advantage of rapidly affording an understanding of the views of respondents on certain issues and timely reflecting changes in public opinion. At the same time, survey results can infer the general situation as a whole, so achieving high representativeness. The advantage of

**337**

*Public Diplomacy: Functions, Functional Boundaries and Measurement Methods*

the non-structured interview lies in its great flexibility, which gives full play to the enthusiasm of both interviewers and interviewees. The two sides can have in-depth, extensive conversations and discussions on relevant issues, events, and phenomena, from history to current events, from causes to effects, from motivations to behaviours, and from individuals to others and major social environments on given topics, so obtaining rich data that is not forthcoming from structured interviews

In view of the characteristics and advantages of the two methods, whether to choose one or both of them for a specific study should be based on the research question and the desired final results of empirical measurement. If researchers want only to know which public diplomacy activities, or specific aspects of an activity, have had a positive effect and which have not, then polling is the preferred method. If researchers want to know the reasons for the effectiveness or ineffectiveness of a public diplomacy activity, or the actual psychological reaction of the activity target, they should consider the interview method. And if you want to understand both, it

Third, with regard to the interpretation and evaluation of the data, we must focus on the crucial matter of the potential impact of interference variables. One potential dispute is that of whether measuring the effect of something requires excluding the effect of the interfering variable. Those who are extremely optimistic or who place great expectations on the functioning of public diplomacy may put forward the extreme view that public diplomacy must produce the desired change in the attitudes of the people of the target country despite the presence of other disturbing variables, such as those mentioned above, in order for us to consider it effective. Is that a reasonable view? Let us imagine another scenario: how do we determine whether or not the cooling function of an air conditioner is effective? One view is that assessing the cooling effect of air conditioning requires excluding all other factors that affect the temperature. According to this view, it is only when the doors and windows are closed, there is no direct sunlight or heat source in the room, all other possible variables remain unchanged and the indoor temperature drops a certain value (for example, 3°C) after the air conditioner has been on for a certain period of time (such as 10 min) that we can consider this air conditioning

Another view is that assessing the cooling effect of an air conditioner does not require controlling for the effects of other interfering variables. According to this view, whether or not the doors and windows are closed, there is direct sunlight or a heat source in the room, or there are other potential interference variables, and no matter how long the air conditioning has been on, the indoor temperature must drop 3°C before we can consider the air conditioning refrigeration as 'effective'. Obviously, in real life even the pickiest consumer buying an air conditioner would not make such high demands of the manufacturer as listed in the second view above with regard to its refrigeration effect. That being the case, there is no reason to expect public diplomacy to be effective in the presence of interfering variables. In other words, it makes sense to say that a country's public diplomacy towards a country is effective (or ineffective) only when the influence of other interfering variables is excluded. For example, we try to measure the effect of public diplomacy through public opinion poll data, which may be influenced by factors other than public diplomacy. Therefore, the interpretation and evaluation of these data must take into account the potential influence of interference variables. When analysing the trend of the effect of public diplomacy over a period of time, the inflection point of poll data is often an important empirical basis for scholars to analyse any change in its effect. At the same time, however, it is at the inflection point that interference variables are most likely to exist. Therefore, when

*DOI: http://dx.doi.org/10.5772/intechopen.92664*

is best to combine the two methods.

refrigeration as 'effective'.

and opinion polls.

#### *Public Diplomacy: Functions, Functional Boundaries and Measurement Methods DOI: http://dx.doi.org/10.5772/intechopen.92664*

the non-structured interview lies in its great flexibility, which gives full play to the enthusiasm of both interviewers and interviewees. The two sides can have in-depth, extensive conversations and discussions on relevant issues, events, and phenomena, from history to current events, from causes to effects, from motivations to behaviours, and from individuals to others and major social environments on given topics, so obtaining rich data that is not forthcoming from structured interviews and opinion polls.

In view of the characteristics and advantages of the two methods, whether to choose one or both of them for a specific study should be based on the research question and the desired final results of empirical measurement. If researchers want only to know which public diplomacy activities, or specific aspects of an activity, have had a positive effect and which have not, then polling is the preferred method. If researchers want to know the reasons for the effectiveness or ineffectiveness of a public diplomacy activity, or the actual psychological reaction of the activity target, they should consider the interview method. And if you want to understand both, it is best to combine the two methods.

Third, with regard to the interpretation and evaluation of the data, we must focus on the crucial matter of the potential impact of interference variables.

One potential dispute is that of whether measuring the effect of something requires excluding the effect of the interfering variable. Those who are extremely optimistic or who place great expectations on the functioning of public diplomacy may put forward the extreme view that public diplomacy must produce the desired change in the attitudes of the people of the target country despite the presence of other disturbing variables, such as those mentioned above, in order for us to consider it effective. Is that a reasonable view? Let us imagine another scenario: how do we determine whether or not the cooling function of an air conditioner is effective? One view is that assessing the cooling effect of air conditioning requires excluding all other factors that affect the temperature. According to this view, it is only when the doors and windows are closed, there is no direct sunlight or heat source in the room, all other possible variables remain unchanged and the indoor temperature drops a certain value (for example, 3°C) after the air conditioner has been on for a certain period of time (such as 10 min) that we can consider this air conditioning refrigeration as 'effective'.

Another view is that assessing the cooling effect of an air conditioner does not require controlling for the effects of other interfering variables. According to this view, whether or not the doors and windows are closed, there is direct sunlight or a heat source in the room, or there are other potential interference variables, and no matter how long the air conditioning has been on, the indoor temperature must drop 3°C before we can consider the air conditioning refrigeration as 'effective'.

Obviously, in real life even the pickiest consumer buying an air conditioner would not make such high demands of the manufacturer as listed in the second view above with regard to its refrigeration effect. That being the case, there is no reason to expect public diplomacy to be effective in the presence of interfering variables. In other words, it makes sense to say that a country's public diplomacy towards a country is effective (or ineffective) only when the influence of other interfering variables is excluded. For example, we try to measure the effect of public diplomacy through public opinion poll data, which may be influenced by factors other than public diplomacy. Therefore, the interpretation and evaluation of these data must take into account the potential influence of interference variables.

When analysing the trend of the effect of public diplomacy over a period of time, the inflection point of poll data is often an important empirical basis for scholars to analyse any change in its effect. At the same time, however, it is at the inflection point that interference variables are most likely to exist. Therefore, when

*Heritage*

symposium [25].

worth discussing.

are two main empirical methods of measuring the effect of public diplomacy: one is the sampling survey method, which is aimed at the general public, namely, the public opinion poll; the other is the unstructured interview with a small group of specific people. The non-structured interview, also known as the non-standardized interview, is either a semi-controlled or uncontrolled interview, which can be divided into four types: intensive interview, in-depth interview, objective statement, and

The public opinion poll method and the interview method are two common widely used empirical methods in modern social science research on which there have been many methodological works with regard to their design procedure and implementation steps in the general sense, but which are not discussed in this chapter. The author's specific concern is: when using these two methods to measure the effect of public diplomacy, which problems need to be resolved to ensure the accuracy of the measured results? In this regard, there are at least three aspects

First, who should be chosen for the investigation? Whether a public opinion poll or an interview, the first question involves the selection of respondents. The author's opinion is: pinpoint the direct audience of public diplomacy and extract from this group (as far as possible) the respondents through which to measure its effect. This is because the fundamental purpose of our survey is to see whether or not the people of the target country have changed their attitude towards a certain aspect of the implementing country due to a specific public diplomacy project on the part of

the implementing country, rather than a general change in attitude.

tional students in country A has deteriorated.

Imagine country A conducting public diplomacy through the medium of international students there from country B. After a period of time, we want to know whether or not country A's public diplomacy activity has been effective, so we conduct questionnaires or interviews with workers in country B. In this case, even if the survey results show that workers in country B have never heard of such public diplomacy as conducted by country A, we should not consider it to be invalid, because country B's international students in country A may be much more familiar with such public diplomacy. Similarly, even if the survey results show that the attitude of workers in country B towards country A has improved, we should not assume that the public diplomacy aimed at international students from country B has been effective, because it is possible that the attitude of country B's interna-

For example, the aforementioned interview study on the effect of China's public diplomacy on Japan is flawed to some extent as regards its design, as discussed here [23]. The two academics asked seven interviewees: 'What effect, if any, do you think the establishment of Confucius Institutes in Japan has had on China's reputation?" Obviously, the best and most convincing way of finding out whether or not establishing a Confucius institute in Japan will improve China's reputation is to ask Japanese students who are studying or who have studied at Confucius institutes in Japan. However, the article clearly identified the seven respondents, none of whom were or had been students at a Confucius institute in Japan. Claiming that China's public diplomacy efforts to promote Confucius institutes in Japan are ineffective, therefore, even though all respondents in the group said that they had not noticed or were only slightly aware of the existence of Confucius institutes, is both futile

Second, which of the two methods of polling and interviewing should be chosen? As regards public opinion polling, it has the advantage of rapidly affording an understanding of the views of respondents on certain issues and timely reflecting changes in public opinion. At the same time, survey results can infer the general situation as a whole, so achieving high representativeness. The advantage of

**336**

and unconvincing.

analysing the inflection point, we should not judge whether a country's public diplomacy becomes effective or ineffective based solely on the data trend ensuing from it. Instead, we should focus on whether are not there are other interference factors at or before the inflection point that may affect the poll data.
