Artificial Intelligence and Affective Computing

#### **Chapter 3**

## Humanistic Next-Generation Artificial Intelligence Capable of Association

*Seiji Tsuchiya*

#### **Abstract**

The third artificial intelligence (AI) boom focused on the "handling of large amounts of data" and "automated learning." One may think that AI can do anything because it is capable of automated learning, but there are still many problems that AI must tackle. The "necessity of a large amount of data" is and will become an even more significant problem. Obtaining an accurate solution from small amounts of data requires imagination and the detection of trends from a small number of phenomena. One approach is to add artificial data. For example, data can be created by intentionally including noise, and the variation may be expanded by a crossover. Different data can be generated by association or inference. Needless to say, these are artificial data and are not correct cases. "Humanistic AI" must be implemented by devising a scheme to allow accurate learning from small amounts of data. I think that the days when robots are considered enemies are transient and robots will soon be recognized as good partners that support humans instead of being rivals.

**Keywords:** artificial intelligence, humanistic AI, association, inference

#### **1. Introduction**

The third artificial intelligence (AI) boom focused on the "handling of large amounts of data" and "automated learning." Collecting and learning information expressed in various forms in a variety of fields can lead to the automated detection of knowledge and rules. One may think that AI can do anything because it is capable of automated learning, but there are still many problems that AI must tackle. The "necessity of a large amount of data" is and will become an even more significant problem.

Large amounts of data are indispensable to statistic processing. In other words, problems where data cannot be collected cannot be solved. One view is that "problems where the collection of data is difficult are not big problems anyway, and therefore, can be ignored." Is this really acceptable? Ignoring a problem because the amount of data is small separates problems into those that can and those that cannot be processed using AI. Stretching in this direction leads to the formation of gaps, which then leads to discrimination and information gaps. Therefore, although an effective use of information technology should be accessible to the weak, a totally opposite scenario may occur.

#### **2. Problems in Japan**

The population of Japan is approximately 120 million, which is about one tenth of that of all of the Western countries combined or that of China. As a consequence, the amount of data that a large country can collect in 1 month takes around a year to collect in Japan. Therefore, Japan cannot win against large countries, as the performance of AI is currently governed by the amount of collected data.

Therefore, AI developed abroad will be imported and used. This is sufficient to some extent from a macroscopic point of view. However, Japan is a very unique group as compared to large countries, a fact that is often overlooked because Japan is highly ranked, for instance, in terms of economy. Many aspects of the Japanese culture are peculiar; hence, there will be some unreasonableness if AI developed abroad is used as-is. From a microscopic point of view, Japanese people have to put up with some issues in their lives.

For example, a translation service that a large company provides for free mistranslated "Sakaisuji Line," which is a subway line in Osaka, as "Muscular Line," and "Sakaisuji" as "thigh muscle." This is a big problem for people living in Japan, and international sightseeing visitors will be bewildered (**Figure 1**) [1].

This type of mistake can be easily detected and resolved by Japanese people. In contrast, for companies providing services globally, a small mistake in a specific region in a single country might not be an issue. Indeed, this mistranslation must not have been viewed as a problem that needed to be addressed, because no correction was made for a long time after this problem was reported. Thinking from a global macroscopic viewpoint naturally emphasizes covering more countries, regions, and languages. However, minority groups must clearly understand this issue: systems are not always created and provided with sufficient consideration to minority groups.

**Figure 1.** *Example of a translation service.*

#### **3. Limits of statistical processing**

Having more data is not always better. For example, consider someone who loves "tomatoes" very much and eats tomatoes every day. A favorite friend's birthday is approaching, and he wants to give this friend a present. After consultation with various acquaintances, the recommendation was to "give my favorite thing as a present." He therefore decided to give "tomatoes" as the birthday present because "I love 'tomatoes' so much; I eat them every day." Is this a good idea?

#### *Humanistic Next-Generation Artificial Intelligence Capable of Association DOI: http://dx.doi.org/10.5772/intechopen.93559*

Of course not. Giving "tomatoes" as a birthday present clearly contradicts common sense. However, "giving my favorite thing as a present to others" is not a wrong idea. Had the person loved "cakes" instead of "tomatoes," then giving "cakes" as a birthday present to his favorite friend would have been a very successful result. Then, what is the difference between "tomatoes" and "cakes"? These are both food! Distinguishing these just by using statistical processing is very difficult. In contrast, the "tomatoes" lover might be very happy when very rare, top-grade tomatoes are given to him as a birthday present.

As is evident from this example, schemes that do not simply analyze large amounts of data but instead analyze high-quality data or give solutions with good precision will be indispensable.

Many data can be collected if the target is large. One resolution when a large amount of data cannot be collected is to increase the denominator by increasing the target domain. A considerable amount of data can be collected in this case, but a large amount of irrelevant data will also be collected. In contrast, purging unnecessary data that become the noise leads to the narrowing of the target domain. Characteristic data can be collected by focusing on a certain domain, but this results in the problem of how to collect large amounts of data in the limited domain.

The issue of quantity and quality is basically a trade-off relationship. Collecting large amounts of data is necessary to cover various cases and to judge the importance of certain things by investigating the frequency. Therefore, the shortcut to obtaining high-quality data is to secure the variation and sufficient data for determining the relative importance.

#### **4. Mechanism to derive the solution from less data**

Obtaining an accurate solution from small amounts of data requires imagination and the detection of trends from a small number of phenomena. One approach is to add artificial data. For example, data can be created by intentionally including noise, and the variation may be expanded by a crossover. Different data can be generated by association or inference. Needless to say, these are artificial data and are not correct cases. However, if an environment where very delicate simulations are possible can be obtained, artificial data that are not true, but very close, may be created.

We humans can, without actually experiencing everything, imagine and think by reading books or listening to others' experiences. Simulated experience is valuable. I think that computers can also respond to simulated data. However, computers are not as imaginative as humans. Common sense is necessary when imagining things. The timing, circumstances, and situation must be taken into account. Moreover, judgments must consider the position of the counterpart, human relations, the atmosphere, and the underlying background.

However, current computers are not capable of this task, which is understandable because computers live in a world of "0"s and "1"s and things are considered and handled as symbols. AI that can support humans and can be active as partners need a mechanism that has common sense and can understand and share human feelings.

#### **5. Implementation of humanistic AI**

I founded the Artificial Intelligence Engineering Research Center at Doshisha University in 2018, and I am now its Director. Many professors who focus on various AI-related research work at this research center and are conducting research on AI from diverse viewpoints. I am particularly interested in the "implementation of humanistic AI" [2].

"Humanistic" has two aspects: one is "common knowledge," which is "what everyone knows," and the other is "common sense," which is "conscience and sound consideration and judgment." "What everyone knows" can be statistically processed because it can be found explicitly in dictionaries or is what many people agree upon. However, "conscience and sound consideration and judgment" is very tricky. It depends on ethics, morals, manners, virtues, and cultures; thus, the "correct" answer is vague, and judgments can vary from person to person. However, there can be some guidelines such as "this is not good" or "this is impossible" (**Table 1**).

Examples of expressions that the Japanese use casually are "the size of 20 Koshien Stadiums" and "input single-byte alphanumeric characters in this field." These expressions are not strange to the Japanese but are difficult to understand for people outside of Japan. A person unfamiliar with the size of the "Koshien Stadium" cannot relate to what the term "the size of 20 Koshien Stadiums" means. "Single-byte alphanumeric characters" are different from "double-byte" characters, and this expression does not make sense to Westerners who do not use double-byte characters that appear in languages such as Japanese and Chinese. Another example is the following conversation of a married couple: "Help with housework when you are off from work!" "I'm ceaselessly driving a truck, so please let me rest on those rare days off!" The assumption that "the wife is doing the housework, and the truck driver is the husband" is a very outdated common sense in the current world of gender equality.

"Conscience and sound consideration and judgment" therefore changes with, for example, the timing, circumstances, sex, age group, region, position, and/or era. Trying to learn this automatically results in a lack of data. The population of Japan is already small, and classifying data by sex, age group, or region further reduces the amount of data. However, "humanistic AI" must be implemented under these circumstances by devising a scheme to allow accurate learning from small amounts of data (**Figure 2**).


#### **Table 1.**

*"Humanistic" has two aspects.*

**Figure 2.** *Example of "humanistic AI."*

### **6. Estimation of emotions from language**

My specialty is AI with an emphasis on natural language processing. In particular, I have continued to study emotions ever since I started research. With the widespread use of smartphones, speech input is gaining popularity again. New search methods by the use of Siri in iPhones and smart speakers have been proposed, and these are becoming accepted. Further advances in technology and the day when everyone can freely use sophisticated computers are being anticipated. One solution is to use the means of communication between humans as the means to use computers. When robots that can coexist and live together with humans are developed, the ability of robots to understand humans, judge human emotions, and sympathize with humans should be a very important element. Therefore, I focused on reading the emotions of the counterpart from the contents of one's speech and proposed a method to judge the speaker's emotions by analyzing the contents of the speech (**Figure 3**) [3–5].

Emotions were judged on the basis of a knowledge database where emotions were defined for 406 combinations of 203 objective word categories and two verb and normal and nominal adjective word categories, or for 8024 combinations of 34 objective word categories; 59 verb and normal and nominal adjective word categories; active or passive voice (two categories); and positive or negative form (two categories). When categorizing objective words and verb and normal and nominal adjective words, the ambiguity is judged from the relationships between words (**Figure 4**). To effectively use this knowledge database where a small amount of knowledge is registered to judge processing and emotions, I developed a proprietary "concept base" that automatically interprets limited knowledge broadly to respond to diverse expressions [6–9] (**Figure 5**). One result showed that this proposed method is capable of reproducing 74.2% of the emotion judgment ability.

The "concept base" has been developed for over 20 years by my research group. "Word2vec," which has gained considerable popularity recently, is a dictionary built on a similar concept [10]. Word2vec appeared recently; thus, my feeling is that "the times have finally caught up with our ideas." Word2vec is a method that expresses the meaning of a word using multiple numbers on the basis of the hypothesis that "the meaning of a word can be characterized using words that collocate with

**Figure 3.** *Outline of the emotion judgment system.*

#### **Figure 4.**

*Flow of emotion judgment system.*

#### **Figure 5.**

*Technique of automatically interpreting limited knowledge broadly to respond to diverse expressions.*

it." In other words, a word is expressed using a vector in an arbitrary dimension. Expressing the meaning with multiple numbers requires learning from large amounts of data. One method is "Skipgram" that learns words close to a given word (**Figure 6**), and another is "CBoW," which learns words that often appear when a certain number of words exist (**Figure 7**).

*Humanistic Next-Generation Artificial Intelligence Capable of Association DOI: http://dx.doi.org/10.5772/intechopen.93559*

#### **Figure 7.**

*Image of CBoW in Word2vec.*


#### **Figure 8.** *Example of concept base.*


#### **Table 2.**

*Result of accuracy comparison between concept base and Word2vec.*

Word2vec needs to arbitrarily specify a finite number of dimensions. Therefore, the meaning of words must be compressed, and the expressions are slightly forced. In contrast, the concept base expresses the meaning of words with multiple words, which is different from Word2vec. This setup allows the expression of a word with multiple, or, in theory, an infinite number of words. Therefore, all the concepts of a word pictured in a human brain can be expressed (**Figure 8**). Indeed, our concept base captures the meaning of words more precisely than that captured in Word2vec (**Table 2**) [11].

#### **7. Conclusions**

The implementation of "humanistic AI" may lead to humans falling in love with a robot instead of a real human. A robot may know more about you than other humans, will do what you want to be done, will not complain, and can have a good conversation. Therefore, the chances of falling in love with a robot are considerably high.

I think that the days when robots are considered enemies are transient and robots will soon be recognized as good partners that support humans instead of being rivals. When such a time comes, what will humans do and what should humans do? Research on AI forces me to think about humans instead of just on technology. After all, the role model of a robot is none other than us humans. I wish for an evolution to a world where humans can be humans.

*Information Systems - Intelligent Information Processing Systems, Natural Language Processing...*

#### **Author details**

Seiji Tsuchiya Doshisha University, Japan

\*Address all correspondence to: stsuchiy@mail.doshisha.ac.jp

© 2020 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/ by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

*Humanistic Next-Generation Artificial Intelligence Capable of Association DOI: http://dx.doi.org/10.5772/intechopen.93559*

#### **References**

[1] https://www.itmedia.co.jp/news/ articles/1903/20/news091.html

[2] https://ai-eng.doshisha.ac.jp/

[3] Tsuchiya S, Yoshimura E, Watabe H, Kawaoka T. The method of the emotion judgment based on an association mechanism. Journal of Natural Language Processing. 2007;**14**(3):119-238

[4] Tsuchiya S, Imono M, Yoshimura E, Watabe H. Emotion Judgment Method from a Meaning of an Utterance Sentence. Springer LNAI, No. 6881; 2011. pp. 367-376

[5] Tsuchiya S, Suzuki M, Imono M, Yoshimura E, Watabe H. Emotion judgement method based on knowledge base and association mechanism for colloquial expression. Transactions of the Japanese Society for Artificial Intelligence. 2014;**29**(1):11-20

[6] Watabe H, Horiguchi A, Kawaoka T. A sense retrieving method from a noun for the commonsense feeling judgement system. Journal of Artificial Intelligence. 2004;**19**(2):73-82

[7] Hirose T, Watabe H, Kawaoka T. Automatic refinement method of conceptbase considering the rule between concepts and frequency of appearance as an attribute. Technical Report of the Institute of Electronics, Information and Communication Engineers, NLC2001-93. 2002. pp. 109-116

[8] Kojima K, Watabe H, Kawaoka T. A method of a concept-base construction for an association system: Deciding attribute weights based on the degree of attribute reliability. Journal of Natural Language Processing. 2002;**9**(5):93-110

[9] Watabe H, Kawaoka T. Measuring degree of association between concepts for commonsense judgements. Journal of Natural Language Processing. 2001;**8**(2):39-54

[10] Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space. CoRR. 2013;abs/1301.3781:1-12

[11] Sera T, Tsuchiya S, Watabe H. The method of measuring the degree of association between concepts using concept-base and Word2Vec. IEICE Technical Report, AI2017-50. 2018. pp. 43-48

#### **Chapter 4**

## Utterance Emotion Estimation by Using Feature of Syntactic Pattern

*Kazuyuki Matsumoto*

#### **Abstract**

Emotion has been defined as basic emotions by various researchers, however, there are not many studies describing the relation between emotion and language patterns in detail based on statistical information. There are various languages all over the world, and even a language of the same country has different writing styles/expressions depending on which language media is used or who is a writer/ speaker, which is thought to make it difficult to analyze the relation of emotion and language patterns. The author has been engaged in constructing and analyzing emotion corpora in some domains based on different sources. From the analysis results, emotion expressions started to become more understood that they have differences and tendencies according to the attributes of the writers and the speakers. In this chapter, I focused on the differences detected in the attributes of the writer/speaker with respect to language patterns; in usage tendencies or combinations of words, unknown expressions (slangs), sentence patterns, non-verbal expressions (emoji, emoticon, etc.) with relevant emotions, then introduce the outcome of the analytical survey on a large scale corpus obtained from a social networking service.

**Keywords:** emotion estimation, kansei robotics, slang, emoji

#### **1. Introduction**

In the research field of psychology, cognitive linguistics, it has been analyzed and studied about emotion and language [1, 2]. With regard to the relation between basic emotions and language, Fischer [3] performed cluster analysis and created a systematic chart based on emotion categories (emotion word) that can be expressed by language.

In the field of natural language processing, especially, sentiment analysis, a lot of researchers have been engaged in a study on the relationship between language patterns and emotion [4–8]. However, there are various languages all over the world, and language pattern varies depending on language media or writers. For this reason, there are no dictionaries describing language patterns and emotion cyclopaedically.

In the studies by Matsumoto [5] and Tokuhisa [9], they related language pattern dictionaries and occurred emotions. Mera et al. [10] proposed a framework to calculate degrees of positive/negative by using an emotion calculation formula for each case frame pattern. Because most of the methods proposed in these studies were assumed to be applied to "ideal" and "grammatical" sentences, they might not be effective for sentences on Internet.

Matsumoto et al. [11] proposed a method to estimate emotion in utterances including grammatically incorrect expressions such as Internet slangs. In the case of such casual expressions, it is thought to be more effective to take a method by machine learning based on a large scale natural language corpus than to register the knowledge into a dictionary. However, it is difficult to obtain a large scale corpus with labels, and it costs high to make such a corpus. Matsumoto et al. proposed a method to extract features based on word distributed representations as a robust method for unknown expressions. Their method converts words into distributed representation vectors and quantizes them with unsupervised clustering. They demonstrated that the method is robust to unknown expressions compared to existing methods.

After describing emotion estimation methods based on: dictionary, pattern and corpus, we introduce such important elements in corpus-based emotion estimation as gender differences and use of emoji expressions. Then we propose a deep learning-based method that uses a syntactic pattern as a feature combining the corpus-based method and the pattern-based method.

Section 2 introduces the emotion expression dictionary used in our previous research, Section 3 describes the emotion estimation by sentence patterns, and Section 4 explains the corpus-based emotion estimation method. Section 5 analyses emotion estimation with elements of gender and emojis. Section 6 propose a method based on syntax patterns, and Section 7 summarizes this chapter.

#### **2. Emotion expression dictionary**

Dictionaries collecting emotion expressions or evaluation expressions already exist [12–14]. These dictionaries defined emotional kinds that can be expressed with the words or phrases as classification categories and are registered them words or phrases. WordNet-Affect is a database created by extending WordNet thesaurus (conceptual database). A part of the information registered in WordNet-Affect is shown in **Table 1**.

There is a study that converted WordNet-Affect into Japanese language [15]. The evaluation polarity dictionary and the Japanese appraisal evaluation expression dictionary are language resources available for reputation analysis or opinion analysis,


#### **Table 1.** *A-labels and corresponding example synsets.*

#### *Utterance Emotion Estimation by Using Feature of Syntactic Pattern DOI: http://dx.doi.org/10.5772/intechopen.96597*

and they include words with annotation of emotion polarity; positive/negative. To analyze emotion of a sentence written in Japanese, an emotion expression dictionary including Japanese emotion expressions is necessary. It is also necessary to correspond linguistic resources to each language for emotion analysis written in foreign language. Because a framework of linguistic resource might be different according to the kind of language, it is difficult to make a unified dictionary.

In the case of Japanese language, the "Emotion Expression Dictionary" by Nakamura [16] is often referred to and often used in the field of natural language processing. However, many of the expressions included in the emotion expression dictionary are written words appeared in novels, therefore, there are some expressions that are rarely used as colloquial expressions. The Emotion synonym dictionary [17] also includes a few colloquial expressions, listing up the expressions which are thought to be useful for writing novels, scenarios and dramatic dialogs. Currently, as there are no dictionaries that cover practical language expressions such as colloquial expressions, such expressions or patterns are usually extracted from linguistic corpora.

As representative databases with registration of sentence patterns related to emotion expressions, there are EDR electronic dictionary [18], GoiTaikei: A Japanese Lexicon [19], and Kyoto University Case Frame [20]. However, because these linguistic resources are focused on semantic relations, emotion information is not annotated to these databases.

Using dictionaries has an aspect that known knowledge defined by human can be effectively used, however, it is often insufficient when it comes to dealing with things that are greatly related to human sensibilities such as emotions. While some words or expressions always give us unchangeable meanings or impressions, others change their meanings or impressions with the times. For example, the fairness and common sense toward the attributes such as race, religion and gender have changed significantly between decades before and today, so that this issue has been often referred to as one of the problems of artificial intelligence in recent years. Also, as language itself changes, dictionaries need to be updated constantly. In the form of a Wikipedia dictionary, some errors or old information are corrected or updated by being exposed to many people on the Web. However, such descriptions in the Wikipedia dictionary are based on the sensibility of the majority of people, it may not be possible to appropriately estimate the emotions of people with different sensibilities, so there is a limit to emotion estimation with just dictionaries.

#### **3. Relation of sentence patterns and emotion**

This section explains the relation of sentence patterns and emotion from the viewpoint of natural language processing by introducing the studies by Matsumoto [5] and Tokuhisa [9]. Matsumoto et al. [5] focused on the emotion occurrence condition for each sentence pattern to estimate emotion in dialog. They also constructed a dictionary that was registered emotion expressions to consider emotion values of each word. The emotion values mean the strength level of expressing each emotion.

Their study used a sentence pattern database that was extended the emotion calculation formula proposed by Mera et al. [10]. However, because they targeted basic sentence patterns, the method has the same problem with the existing method such as lack of versatility and it is weak to spoken expressions. The "Japanese Lexicon" [19] introduces a sentence pattern of each word. In the example of "Crying, "the sentence patterns are:

• N1 ga N2 wo Warau (N1 laughs at N2)

N1 and N2 are nouns. The emotion expressed by the sentence can differ depending on the noun applicable to N1 and N2. Referring to the example sentence: "Jiro cries over his debt," "debt" generally has a negative image. However, the emotion generated in this sentence can be affected by the speaker's attitude to "Jiro." These patterns were necessary to be annotated rules manually. **Figure 1** shows the case frame pattern of "N1 *ga* N2 *de*/*ni Naku*."

The following table (**Table 2**) shows some examples of sentence patterns and emotion occurrence rules. These information are saved as XML format on account of readability. **Figure 2** shows the emotion occurrence event sentence pattern database with XML format.

Matsumoto et al. [21] also extracted emotion occurrence event sentence patterns from a corpus. The following describes a flow of automatic extraction by Matsumoto et al. showing by example.

Step 1. The inputted sentence is analyzed by dependency parser. "CaboCha [22]"was used as the dependency parser.

First, according to the result of dependency parsing the last segment of the sentence is judged as a predicate. When a segment relates to the predicate and the end of the segment is either case particle or binding particle of "*ga*," "*ha*," "*wo*," "*ni*," "*he*," "*de*," "*to*," "*kara*," "*made*" or "*yori*" is extracted as surface case.

Step 2. The noun included in the obtained surface case element is annotated the semantic attributes based on "A Japanese Lexicon."

If the semantic attributes of the noun cannot be obtained, the basic form of the noun will be set into the surface case slot without annotating semantic attributes. The segment independent from the segment of predicate is not judged as case

#### **Figure 1.**

*Case frame pattern of "N1 ga N2 de/ni Naku".*


#### **Table 2.**

*Example of sentence pattern and emotion occurrence rule.*

*Utterance Emotion Estimation by Using Feature of Syntactic Pattern DOI: http://dx.doi.org/10.5772/intechopen.96597*

```
Figure 2.
```

```
XML format of sentence pattern database.
```
element. Because such sentence segment might be important element for deciding emotion attributes, it is extracted as modifier element. The obtained sentence pattern will be as 'EPT.'

Step 3. The set of emotion attributes 'E' annotated to the inputted sentence is decided as emotion attribute of 'EPT.' The combinations of 'EPT' and 'E' obtained from Step1 to Step3 are registered to the emotion occurrence sentence pattern DB. **Figure 3** shows an example of extraction process when "*Watashi wa odoroki no amari me wo shirokuro saseta*." is inputted.

This study automatically extracted sentence patterns from the emotion labeled corpus, created and evaluated the sentence pattern database. As the result of the cross-validation experiments for eight kinds of emotion estimations from sentences expressing emotions based on the corpus-derived sentence pattern database, approx. 42% emotion estimation accuracy was obtained.

Tokuhisa et al. [23] statistically analyzed the valency pattern of each sentence pattern, and proposed a method for emotion inference. Tokuhisa et al. [24] constructed and evaluated the dialog corpus by annotating emotion tags focusing on facial expressions of characters from manga comics.

Their study mainly target the utterances in dialogs, the target data are utterances not by actual persons but by fictional persons. Although these data are simulated

#### *Information Systems - Intelligent Information Processing Systems, Natural Language Processing…*

#### **Figure 3.**

*Flowchart of creation of sentence pattern (*EPT*).*

real dialog, it is considered that there exist some bias by the authors and generality might be lacking.

It is difficult to register entire colloquial expressions into a dictionary by strictly typifying their sentence patterns, besides, there are few challenging studies that try to annotate emotion that is subjective and sensitive to the sentence patterns. However, I thought that it would not be impossible to extract a relation between emotion and language patterns by studying thoroughly the recent corpus-based methods.

#### **4. Corpus-based emotion analysis method**

This section describes a corpus-based emotion analysis method by referring to the related literatures. The corpus annotated with emotion tags is defined as the emotion corpus. We would like to introduce existing studies that created and evaluated emotion analysis models based on statistical information and machine learning using emotion corpora.

#### **4.1 Japanese-English parallel corpus [Minato et al.]**

Minato et al. [25, 26] annotated emotion tags on word and sentence units included in Japanese and English parallel corpora. The completed corpus included *Utterance Emotion Estimation by Using Feature of Syntactic Pattern DOI: http://dx.doi.org/10.5772/intechopen.96597*

1,190 Japanese-English sentences. Based on the statistic results of the tagged words and sentences, they proposed and evaluated an emotion estimation method. They further considered the relevance between the two languages. Overview of their corpus is shown in **Table 3**.

The annotation to the corpus was made by the author, and evaluation by some examinees were not conducted. Matsumoto et al. [27] conducted an questionnaire on this corpus to several examinees and analyzed precision and recall between the tags annotated by the author and the tags annotated by some examinees. Because all examinees were Japanese people, they evaluated only Japanese sentences (1190 sentences). They calculated reliability of the annotation of the emotion tag by multi annotators. Reliability of tag annotation was calculated based on the match frequency among the three operators (initial tag annotator and two examinees). In their study, they proposed a method to reconstruct an emotion corpus by annotating reliability values. Reliability of tag is calculated with the Eq. (1). P*W Tagx* � � shows the sum of the weight of the tags annotated by the corpus creator and the two examinees

$$\text{Reliability} \left( \text{Tag}\_x \right) = \sum W \left( \text{Tag}\_x \right) \times \left( \frac{\text{Number of Attack Earalutors}}{\text{Total Number of Edward}} \right)^2 \tag{1}$$

They calculated the importance for each emotion category by calculating reliability of tag annotation. For calculation they used the weight of emotion tags according to the reliability as weight for emotion category instead of using simple word frequency. The calculation is based on the TFIDF method. Eq. (2) shows the weight of emotion category.

$$\boldsymbol{W}\_{j}^{i} = \boldsymbol{a}\_{i} \times \sum \boldsymbol{R} \boldsymbol{W}\_{j}^{i} \times \log \frac{\boldsymbol{N}}{\boldsymbol{\sigma}\_{j}} \tag{2}$$

$$a\_i = \frac{1}{\sqrt{\sum\_{m=1}^{l} \left(\frac{\sum\_{R} W\_j^i}{\sigma\_m^i}\right)^2}}\tag{3}$$

P*RW<sup>i</sup> <sup>j</sup>* shows the sum of weight at emotion category '*Ei*' in corpus. The '*cfm*' shows the number of emotion category tagged all sentences which included the word


**Table 3.** *Corpus statistics.* '*wm*'. '*N*' shows the total number of emotion category, and '*l*' shows the total unique frequency of word. '*αi*' is normalization coefficient which is calculated with Eq. (3).

### **4.2 Corpus-based method using N-gram [Mishina et al.]**

Mishina et al. [28] extracted word n-gram features from the emotion corpora, and proposed an emotion estimation method using the similarity score RECARE which was improved from BLEU often used for translation evaluation. The target emotion categories were four kinds; "anger", "joy", "hate", "hope". The problems of the method are; i) necessary to calculate similarity with all sentences in the corpus, and ii) the estimation accuracy affected by the corpus quality because the method is a simple example-based method.

#### **4.3 Corpus creation and analysis [Quan et al.]**

Quan et al. [29] constructed a large size of Chinese weblog emotion corpus "Ren-CECps," and analyzedthe corpus. In Ren-CECps, emotion tags were annotated to sentence, word, paragraph, and article units by some test subjects, and the corpus was analyzed from various viewpoints. The annotation to the corpus required hands, and as the size becomes larger and the corpus includes richer information, the higher annotation costs. There is a demerit that because the target are weblog articles, if there is bias in the writers, that will affect the quality of the corpus.

### **5. Analysis of emotion expressions according to gender**

#### **5.1 The emotion labeled corpus divided according to the users' attributes**

In the study of Matsumoto et al. [30], they targeted the tweet sentences posted on Twitter and targeted each tweet for emotion estimation. Therefore, they needed to annotate emotion tags on each tweet sentence. The emotion estimation model was generated with the following steps:


In Step 4, the feature is extracted. First, the tweet sentence is split into word units by morphological analysis. Then, the words are converted into the distributed representations. They used another corpus to train the distributed representations. For about one year, they continued collecting tweets randomly; then, based on these tweets, they constructed a tweet corpus. They converted the corpus into the word-splitting format and used the text in this format for training the distributed representations.

Then, they annotated the emotion tags on the tweet sentences. Emotion tags annotated to the tweets are as follows:


The total number of the emotion categories is 13. Some examples of the labeled tweets and their user attributes are shown in **Table 4**. The numbers of tweets for each emotion tag are shown in **Table 5**. As shown in **Table 5**, I found that there is bias in the numbers of tweets for each emotion.

In their chapter, they reported that emotion estimation accuracy increase by training the emotion corpus which is prepared for each attributes.

However, one thing to keep in mind when estimating emotion based on the corpus is who to annotate the corpus is. If the annotators' attributes and sensibilities are biased, a biased emotion estimation model would be built by learning the biased corpus. Such model cannot infer appropriate emotions according to the attributes of the authors or the speakers of the object sentence for estimation. To clarify the issue that attributes affect emotion estimation, the next subsection analyses what emotional expressions are used depending on gender based on the corpus.

#### **5.2 Analysis of emotion expressions for each gender**

In this section, I analyze emotion expressions by targeting on an emotion labeled corpus that are divided by gender. By investigating appearance frequency of emotion expressions included in the emotion expression dictionary and the kinds of the emotion labels annotated to the tweets including each emotion expression, I analyze appearance tendency of each expression according to gender by TF-ICF.


#### **Table 4.**

*Labeled tweets and attributes of the users.*


#### **Table 5.** *Annotation frequency of each emotion tag.*

The formula of TF-ICF calculation is shown as Eq.(4) and Eq.(5). TF means Term Frequency, ICF means Inverse Category Frequency. *TF<sup>e</sup> <sup>i</sup>* shows word frequency appeared in emotion category *e*. *ICF<sup>e</sup> <sup>i</sup>* shows the value which is the number of emotion categories divided by the number of emotion categories including word *i*.

$$TF - ICF\_i^\epsilon = TF\_i^\epsilon \times ICF\_i^\epsilon \tag{4}$$

$$ICF\_i^\epsilon = \log \frac{|C|}{|\{C: t\_i \in C\}|} \tag{5}$$

The TF-ICF calculation results for each gender are shown in **Table 6**. In this table, only top 10 expressions and TF-ICF scores are displayed.

From the analysis result, there are not significant difference between male and female. It is cause that only the general expressions are treated in the emotion expression dictionary for expression extraction.


#### **Table 6.**

*TF-ICF calculation results for each gender.*

*Utterance Emotion Estimation by Using Feature of Syntactic Pattern DOI: http://dx.doi.org/10.5772/intechopen.96597*

**Table 7.** *A part of TF-ICF calculation without limitation of emotional expression.*

In addition, the results shown in **Table 7** were obtained by TF-ICF calculation without limitation of emotional expression.

It is found there are expressive differences of gender as seen from this result. For example, in both of gender, the symbols often be used in emotion: "Joy". Above all, female often use emoji. On the other hand, even though, comparatively, female use genial emotional expressions in emotion: "Anger", male often use radical expressions. It is considered that there are specific emotional expressions for each gender, and those express the gender difference of emotional expression.

Difference of emotional expressions by gender might decrease the estimation accuracy of the learned emotion estimation model due to gender bias. In order to avoid this, it would be useful to prepare an emotion estimation model for each

gender or attribute, or to replace the expressions related to attributes with common expressions. In any case, it is clear that some sort of breakthrough is needed to maintain the fairness of machine learning.

#### **5.3 Analysis of emoji**

In this subsection, I analyze the appearance tendency of non-verbal expressions such as emoji according to gender. We analyzed usage trend of emoji from the total 59,009 tweets which were collected separately from the emotion corpus for each gender.

The results are shown in **Figures 4** and **5**. In this figure, the horizontal axes shows Emoji type, the vertical axes shows frequency of use. In the graph of male, emojis with over 20 frequency are shown, and in the graph of female, emojis with over 100 frequency are shown. The types of emoji were set 4 classes; expression, emotion, exclamation and other. **Table 8** shows the result of emoji types and frequencies by counting emojis appeared over 10 times. As seen from this result, females had tendancy to use more emojis than male, and female often used emoji expressing expressions or emotions. As was expected that females would use more rich emotion expressions in their tweets, it was obvious from this usage trend of emoji. On the other hand, males used more exclamation marks than other types of emoji.

This result indicates that not only emotional expressions but also nonverbal expressions such as emojis have sufficient influence on emotion estimation. In addition to emojis, Japanese language has emoticons and ASCII art to convey various emotions. Globally, nonverbal expressions play important roles in

**Figure 4.** *Trend of emoji by male.*

**Figure 5.** *Trend of emoji by female.*

*Utterance Emotion Estimation by Using Feature of Syntactic Pattern DOI: http://dx.doi.org/10.5772/intechopen.96597*


**Table 8.**

*Emoji types and frequencies (over 10 times).*

communication on the Web. From this, it is important to understand nonverbal expressions in order to estimate emotions.

#### **6. Emotion estimation from feature of syntactic pattern by deep learning**

#### **6.1 Creation of emotion estimation by deep neural networks**

We train language patterns that show emotions by using a deep learning method. We use syntactic patterns obtained from the parsing results by the Japanese dependency and case structure analyzer as features for learning. We use KNP [31] as the Japanese dependency and case structure analyzer. KNP is a syntactic, case and reference analyzer developed by Kyoto University. This system uses a noun case frame dictionary constructed by 7 billion web text.

As preprocessing of KNP, it is necessary to annotate morphological features on word unit by using a morphological analyzer. In this study, I make this annotation of morphological features by the morphological analyzer Juman [32]. As seen in **Figure 6**, sentences are analyzed by KNP.

As the result of analysis, the features are annotated on morpheme level and chunk level. The analysis result consists from "Clause layer", "Tag layer", "Morpheme layer". In this study, the features are extracted from the "Tag layer". For training, I use the features that have been annotated on chunk level to associate syntactic patterns with emotions. The examples of features annotated on chunk level are shown in **Table 9**.

The training data are the utterances annotated with emotion tags by manual. These utterances are used in the study by Matsumoto et al. [33], the source sentences are bilingual (Japanese-English). Because these sentences were used as

**Figure 6.** *Analysis results by KNP.*


**Table 9.** *Example of features.*

examples for English composition, it is easy to extract syntactic patterns from sentences. As a preliminary experiment, I confirm emotion estimation accuracy by cross-validation. The breakdown of the five kinds of experimental corpora are shown in **Table 10**.

As the training, I use bi-directional LSTM (bi-LSTM) [34] which is extended LSTM (Long Short-Term Memory) [35]; a kind of recurrent neural networks. LSTM is suited to learning sequences. It enables efficient learning by memorizing and deleting past inputs. **Figure 7** shows the neural network structure using bi-LSTM. I use two LSTM layers.

In this study, I create a feature vector by chunk unit, and input the feature vector from the beginning of a sentence for training. The maximum number of chunks was set as 30 based on the maximum number of the chunks in the corpora.


#### **Table 10.**

*Statistic of emotion tagged corpora.*

**Figure 7.** *Neural networks using bidirectional LSTM.*

*Utterance Emotion Estimation by Using Feature of Syntactic Pattern DOI: http://dx.doi.org/10.5772/intechopen.96597*


**Table 11.**

*F-measures of the preliminary experimental results.*

**Table 11** shows the result of the preliminary experiment. Averaged F-measure was 32–49%. The cause of this was thought to be the bias of emotion tags.

#### **6.2 Experiment**

I apply the emotion estimator trained syntactic features using bi-LSTM to the tweet sentences for each gender and evaluate the estimator by calculating accuracy. The architecture of the neural networks using bi-LSTM is shown in **Figure 8**. The tweet corpus shown in **Table 12** was used for the experiment.

We compare the result of the proposed method and the emotion estimation result based on emoji. The dictionary registered emojis and their expressing emotions is constructed as the Emoji Emotion Dictionary. The emoji emotion vectors of the emojis that are not in the dictionary are estimated. Emoji emotion vector of each emoji is obtained by calculating similarity with the seed emojis included in the emoji emotion dictionary and by acquiring emotion categories and similarities of top 5 similar seed emojis. The cosine similarity between the emoji distributed representations is used as the similarity of emojis. Eq. (6), (7), (8) shows the calculation of an emoji emotion vector.

$$EV\_{e\_i} = \left(ew\_{e\_i}^1 \epsilon w\_{e\_i}^2 \dots \epsilon w\_{e\_i}^j \dots \epsilon w\_{e\_i}^n\right) \tag{6}$$

$$\begin{aligned} EV\_{\text{avg}} &= \frac{1}{|EM\_{\text{top}N}|} \sum\_{e \in \mathcal{EM}\_{\text{top}N}} (sim\_{e\_i} \times EV\_{e\_i}) \\\\ &= \left(ew\_{\text{avg}}^1 \epsilon w\_{\text{avg}}^2 \dots eu\_{\text{avg}}^j \dots ew\_{\text{avg}}^n\right) \\\\ &= \underset{\text{x}}{\text{arg}\,\max} \, ew\_{\text{avg}}^{\text{x}} \end{aligned} \tag{7}$$

Eq.(6) shows emotion vector *EVei* of emoji *emei* . Emoji emotion vector is a weighted mean of the emotion vectors of the top *N* similar seed emojis using similarity *simei* with seed emojis. *ew <sup>j</sup> ei* shows the weight of emotion category j. Eq.(7) is the formula to calculate the mean emoji emotion vector from the top similar *N* emoji set *EMtopN*. The estimated emotion is outputted as the emotion category *x* with the maximum weight value *ew <sup>j</sup> avg* of the mean vector by Eq.(8). The averaged emoji emotion vector is outputted by calculating emojis including in the sentences as the emotion estimation result. In this study, *N* value is set as 5 to estimate emotions.

**Figure 8.** *bi-LSTM neural networks architecture.*

#### *Utterance Emotion Estimation by Using Feature of Syntactic Pattern DOI: http://dx.doi.org/10.5772/intechopen.96597*


#### **Table 12.**

*Number of tweet sentences for each emotion.*


#### **Table 13.**

*Comparison between the accuracies of the proposed method and the emoji-based method.*

#### **6.3 Experimental results**

Because neutral tags were not annotated to the target tweet corpus, the accuracies for 4 emotion categories were calculated: "Joy," "Anger," "Sorrow," "Surprise." The experimental result is shown in **Table 13**. The highest accuracy was found for "Sorrow, "and the second highest was for "Joy. "The lowest accuracy was 24.3% and that was obtained for "Anger".

On the other hand, the overall accuracy was 43.7% by the emoji-based method, which was better than by the bi-LSTM based proposed method. However, the accuracy for "Anger" was low; 4% although the accuracy for "Surprise" was 100%. The primal reason is that the varieties of "Surprise" seed emoji were smaller than other kinds of emotions. It is also because that the number of tweets expressing "Surprise" with emoji was relatively scarce.

This result shows that using the syntax pattern enables effective emotion estimation using deep learning even with a small amount of learning data. It is thought that a more accurate model can be realized by flexibly changing dictionary knowledge depending on the domain or the speaker of the target sentence.

#### **7. Conclusions**

This chapter introduced our study on "emotion analysis on Japanese language" in the research field of the existing natural language processing and linguistic resources. Most of the existing approaches tried to associate emotions and language patterns, however, if language patterns express different emotions depending on the words consisting of the sentences, the rules for millions of combinations must be described.

It will be effective to analyze emotions based on corpora by annotating emotions on the corpora. In this chapter, various features were annotated on sentences by

using a syntactic parser and feature vectors were generated by clause unit. The emotions of the tweet sentences were estimated by training the features using bi-LSTM neural networks.

It was also shown that the capability to development emotions from language patterns by using "emoji" as non-verbal expression. From the experimental results, the emoji-based method was found to be effective to tweet sentences including emoji. Because the amount of the emotion labeled data is limited and the existing dictionary and corpus-based methods cannot cover emotion expressions that are colloquially and depended on users' attributes, improvement of estimation accuracy is limited. Because emojis are non-verbal emotion expressions that can be used for all users, and the emoji expressions are not depended on the kind of languages, it is a hopeful key of emotion analysis in future.

In addition, syntax pattern might not be correctly extracted from the casual sentences that are often seen in dialogs on SNS. In that case, general-purpose neural language models such as BERT [36] and GPT-3 [37] will be useful. Future developments in language models might eliminate the necessity of human-defined linguistic knowledge such as syntactic patterns, however, methods such as fine tuning are still effective to build emotional estimation models satisfying the needs of all the people from large data. In that case, dictionary knowledge and syntax patterns will play effective roles in improving accuracy and presenting the basis for judgment.

#### **Acknowledgements**

This work was partially supported by JSPS KAKENHI Grant Numbers JP20K12027.

#### **Author details**

Kazuyuki Matsumoto Tokushima University, Tokushima, Japan

\*Address all correspondence to: matumoto@tokushima-u.ac.jp

© 2021 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/ by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

*Utterance Emotion Estimation by Using Feature of Syntactic Pattern DOI: http://dx.doi.org/10.5772/intechopen.96597*

### **References**

[1] Kircanski, K, Lieberman, MD, Craske MG. "Feelings into words: contributions of language to exposure therapy" Psychol Sci. 2012 Oct 1;23(10):1086–91. doi: 10.1177/0956797612443830. Epub 2012 Aug 16.

[2] Kuperman, V, Estes, Z, Brysbaert, M, Warriner, AB. "Emotion and language: Valence and arousal affect word recognition", J Exp Psychol Gen. 2014 Jun;143(3):1065–1081. doi: 10.1037/ a0035669. Epub 2014 Feb 3.

[3] Fischer, KW, Shaver, P, Carnochan, P. "A skill approach to emotional development: From basic- to subordinate-category emotions. In W. Damon (Ed.), Child development today and tomorrow, pp.107–136, (1989).

[4] Matsumoto, K, Ren, F, Kuroiwa, S. "Emotion Estimation System Based on Emotion Occurrence Sentence Pattern," International Conference on Intelligent Computing (ICIC 2006): Computational Intelligence. pp 902–911, 2006.

[5] Matsumoto, K, Mishina, K, Ren, F, and Kuroiwa, S. "Emotion Estimation Algorithm based on Emotion Occurrence Sentence Pattern," Journal of Natural Language Processing. 2007; **14**(3):239-271, https://doi.org/10.5715/ jnlp.14.3\_239. (in Japanese)

[6] Žižka, J and Daˇrena, F. "Automatic Sentiment Analysis Using the Textual Pattern Content Similarity in Natural Language," TSD 2010, LNAI 6231, pp. 224–231, 2010.

[7] Han, Y and Kim, KK. "Sentiment analysis on social media using morphological sentence pattern model," 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA), INSPEC Accession Number: 17009997.

[8] Su, S and Lynn, KT. "Extracting Product Features and Opinion Words Using Pattern Knowledge in Customer Reviews," The ScientificWorld Journal Volume 2013, Article ID 394758, 5 pages http://dx.doi.org/10.1155/2013/394758.

[9] Tokuhisa, M, Murakami, J, Ikehara, S. "Construction and Evaluation of Text-Dialog Corpus with Emotion Tags Focusing on Facial Expression in Comics," Conference: Knowledge-Based Intelligent Information and Engineering Systems, 10th International Conference, KES 2006, Bournemouth, UK, October 9–11, 2006, Proceedings, Part III, DOI10.1007/11893011\_91.

[10] Mera, K. and Ichimura, T. "Emotion Analyzing Method Using Physiological State," Proceedings of KES 2004: Knowledge-Based Intelligent Information and Engineering Systems, pp. 195–201.

[11] Matsumoto, K, Yoshida, M, Xiao, Q, Luo, X and Kita, K. "Emotion recognition for sentences with unknown expressions based on semantic similarity by using Bag of Concepts," 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015, pp. 1394–1399, 2015.

[12] WordNet Affect: http://wndomains. fbk.eu/wnaffect.html

[13] Kobayashi, N, Inui, K, Matsumoto, Y, Tateishi, K. "Collecting Evaluative Expressions for Opinion Extraction," Journal of Natural Language Processing 12(3), pp. 203–222, 2005.

[14] Sano, M. (2010) Japanese Dictionary of Appraisal —attitude— (JAppraisal Dictionary ver1.0), published by Gengo Shigen Kyokai.

[15] Torii, Y, Das, D, Bandyopadhyay, S, Okumura, M. "Developing Japanese WordNet Affect for Analyzing

Emotions," Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis, ACL-HLT 2011, pages 80–86, 2011.

[16] Nakamura, A. 1993. Emotional Expression Dictionary (in Japanese). Tokyodo shuppan.

[17] Ackerman, A and Puglisi, B. "The Emotion Thesaurus: A Writer's Guide to Character Expression, " Createspace Independent Pub. 2012.

[18] EDR Electronic Dictionary Version 1 Technical Guide, EDR TR2–003, 1995.

[19] Goitaikei – A Japanese Lexicon (in Japanese), Ikehara, S., Miyazaki, M., Shirai, S., Yokoo, A., Nakaiwa, H., Ogura, K., Oyama, Y., and Hayashi, Y. 1997. Iwanami Shoten

[20] Kawahara, D and Kurohashi, S. "Case Frame Compilation from the Web using High-Performance Computing," In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC2006), 2006.

[21] Matsumoto, K, Minato, J, Tsuchiya, S and Ren, F. "An Algorithm for Extracting Emotion Occurrence Sentence Pattern from Corpus," Proceedings of 2008 AIAC Symposium, 109–123, Tokushima, Mar. 2008.

[22] Kudo, T and Matsumoto, Y. "Fast Methods for Kernel-Based Text Analysis," Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL2003), pp. 24-31, 2003.

[23] Tokuhisa, M, Tanaka, T, Ikehara, S and Murakami, J. "Emotion Reasoning Based on Valency Patterns–a prototype annotation of causal relationships—" Human and Artificial Intelligence Systems — From Control to Autonomy, Proceedings of the 4th International Symposium on Human and Artificial

Intelligence Systems (HART2004), International Series on Natural and Artificial Intelligence. 2004;**1**:534-539.

[24] Tokuhisa, M, Ikehara, S and Murakami, J. "Construction of Textdialog Corpus with Emotion Tags Focusing on Facial Expression in Comics," Journal of Natural Language Processing. 2007;**14**(3):193-217. (in Japanese)

[25] Minato, J, Matsumoto K, Ren, F and Kuroiwa, S. "Corpus-based analysis of Japanese-English of emotional expressions." In Proceedings of IEEE International Conference on Natural Language Processing and Knowledge Enginerring, pp. 413–418, 2007.

[26] Minato, J, Matsumoto K, Ren, F, Tsuchiya, S and Kuroiwa, S. "Evaluation of emotion estimation methods based on statistic features of emotion tagged corpus." International Journal of Innovative Computing, Information and Control, Vol. 4, No. 8, pp. 1931–1941, 2008.

[27] Matsumoto, K, Minato, J, Tsuchiya, S and Ren, F. "An Approach for Evaluating Emotion Tagged Corpus," In Proceedings of IADIS International Conference Interfaces and Human Computer Interaction 2008, pp. 258– 262, Amsterdam, Jul. 2008.

[28] Mishina, K, Tsuchiya, S, Suzuki, M and Ren, F. "An Improvement of Example-based Emotion Estimation Using Similarity between Sentence and each Corpus," Journal of Natural Language Processing. 2010;**17**(4):91-110.

[29] Quan, C and Ren, F. "Sentence Emotion Analysis and Recognition Based on Emotion Words Using Ren-CECps," International Journal of Advanced Intelligence. 2010;**2**(1):105-117.

[30] Matsumoto, K, Yoshida, M, Kita, K, Wu, Y and Iwasa, F. "Effect of Users Attribute on Emotion Estimation from

*Utterance Emotion Estimation by Using Feature of Syntactic Pattern DOI: http://dx.doi.org/10.5772/intechopen.96597*

Twitter," Proceedings of the 2nd IEEE International Conference on Computer and Communications, Vol.3, pp. 1186- 1190, Oct. 2016.

[31] Kawahara, D and Kurohashi, S. "A Fully-Lexicalized Probabilistic Model for Japanese Syntactic and Case Structure Analysis," In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL2006), pp.176–183, 2006.

[32] Japanese Morphological Analyzer: http://nlp.ist.i.kyoto-u.ac.jp/index.php? JUMAN

[33] Matsumoto, K, Ren, F, Yoshida, M and Kita, K. "Refinement by Filtering Translation Candidates and Similarity Based Approach to Expand Emotion Tagged Corpus," Knowledge Discovery, Knowledge Engineering and Knowledge Management. Communications in Computer and Information Science (CCIS). 2017;**631**:260-280.

[34] Schuster, M and Kuldip KP. "Bidirectional recurrent neural networks." IEEE Transactions on Signal Processing. 1997;**45**(11):2673-2681.

[35] Hochreiter, S and Schmidhuber, J., "Long short-term memory," Neural Computation. 1997;**9**(8):1735-1780

[36] Devlin, Y, Chang, MW, Lee, K and Toutanova, K. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019;**1**:4171-4186.

[37] Brown, T, Mann, B, Ryder, N, et al. "Language Models are Few-Shot Learners." In Proceedings of Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.

Section 3
