Abstract

Nowadays, emoji plays a fundamental role in human computer-mediated communications, allowing the latter to convey body language, objects, symbols, or ideas in text messages using Unicode standardized pictographs and logographs. Emoji allows people expressing more "authentically" emotions and their personalities, by increasing the semantic content of visual messages. The relationship between language, emoji, and emotions is now being studied by several disciplines such as linguistics, psychology, natural language processing (NLP), and machine learning (ML). Particularly, the last two are employed for the automatic detection of emotions and personality traits, building emoji sentiment lexicons, as well as for conveying artificial agents with the ability of expressing emotions through emoji. In this chapter, we introduce the concept of emoji and review the main challenges in using these as a proxy of language and emotions, the ML, and NLP techniques used for classification and detection of emotions using emoji, and presenting new trends for the exploitation of discovered emotional patterns for robotic emotional communication.

Keywords: emoji, machine learning, natural language processing, emotional communication, human-robot interaction

## 1. Introduction

Recently, in the episode "Smile" of the popular science fiction television program "Doctor Who," a hypothetical off-earth colony is presented. This colony is maintained and operated by robots, which communicate and express emotions with humans and its pairs, through the usage of emoji. Sure, one may argue that such technology, besides being mere science fiction, is ridiculous since phonetic communication is much simpler and much easier to understand. While this is true for conventional information (e.g., explaining the concept of real numbers), communicating body emotional responses or gesticulation (e.g., to describe confusion) using only phonograms would require many more words to convey the same message than an emoji (e.g., or ). In this sense, emoji serve as a visual simplified form of (affective) communication that broadens the total amount of information (e.g., cues and gestures), which can be shared between humans and virtual/ embodied artificial entities. If we consider that human languages, such as Chinese, Nahuatl [1], or even Sign Language, have evolved from ideographs and pictographs lexicons, can we expect that in the near future, artificial entities (virtual or embodied) would employ emoji in their emotional communication?

The Japanese word emoji (e = picture and moji = word) literally stands for "picture word." Although recently popularized, its older predecessors can be tracked to the nineteenth century, when cartoons were employed for humorous writing. Smileys followed in 1964 and were meant to be used by an insurance company's promotional merchandise to improve the morale of its employee. The first to employ the emoticon :) in an online bulletin forum to denote humorous messages was Carnegie Mellon researchers in 1982, and 10 years later, the emoticons were already widespread in emails and Websites [2]. Finally, in 1998, Shigetaka Kurita devised emoji to improve emoticons pictorially, and became widespread by 2010. From this moment, the use of emoji has gained a lot of momentum, even achieving that the word namely "Face with Tears of Joy" ( ) was chosen by the Oxford Dictionary as the Word of the Year [2–4]. This choice was made under the assumption that the pictograph represented the ideas, beliefs, mood, and concerns of English speakers in 2015.

Since its origin, emoji undoubtedly have become a part of the mainstream communication around the globe allowing people, with different languages and cultural backgrounds, to share and interpret more accurately ideas and emotions. In this vein, it has been hypothesized that emoji shall become a universal language due to its generic communication features and its ever progressing lexicon [2, 5–7]. Although, this idea is controversial [8, 9] since emoji usage during the communication is influenced by factors such as context, users interrelations, users' first language, repetitiveness, socio-demographics, among others [2, 5, 8]. This clearly adds ambiguity on how to employ them and its proper interpretation. Nevertheless, in the same fashion as sentiment analysis mines sentiments, attitudes, and emotions from text [10], we can employ billions (or perhaps more) of written messages within the Internet that contains emoji, to generate affective responses in artificial entities. More precisely, using natural language processing (NLP) along with machine learning (ML), we can extract semantics, express beat gestures, emotional states, and affective cues, add personality layers, among other characteristics from text. All this knowledge can be used to build, for instance, emoji sentiment lexicons [10] that will conform the emoji communication competence [2] that will power the engines of the emotional expression and communication of an artificial entity.

In the rest of this chapter, we first review the elements of the emoji code, and how emoji are used in the emotional expression and communication (Section 2). Afterward, in Section 3, we present a review of the state of the art in the usage of NLP and ML to classify and predict annotation and expression of emotions, gestures, affective cues, and so on, using written messages from multiple types of sources. In Section 4, we present several examples on how emoji are currently employed by artificial entities, both virtual and embodied, for expressing emotions during its interaction with humans. Lastly, Section 5 summarizes the chapter and discusses open questions regarding emoji usage as a source for robotic emotional communication.

#### 2. Competence, lexicon, and ways of usage of emoji

To study how emoji are employed and about its challenges, we cannot simply do it without specifying the emoji competence [2]. Loosely speaking, competence (either linguistic or communicative) stands for the rules (e.g., grammar) and abilities an individual owns to correctly employ a given language to convey a specific idea [11]. Hence, the emoji competence stands for an adequate usage of emoji within

Emoji as a Proxy of Emotional Communication DOI: http://dx.doi.org/10.5772/intechopen.88636

messages, not only in their representation but also in exact position within the message, to address a specific function (e.g., emotional expression, gestures, maintain interest in the communication, etc.) [2]. Nevertheless, even while the emoji competence has not been formally defined yet, and it can only be developed through the usage of emoji themselves [2, 6], here, we elaborate several of its components.

A key element of the emoji competence is the emoji lexicon, which is the standardization of pictograms (i.e., figures that resemble the real-world object), ideograms (i.e., figures that represent an idea) and logograms (i.e., figures that represent a sound or words) into anime-like graphical representations that belong to the ever-growing Unicode computer character standard [2, 6, 12]. These are employed within any message in three different ways: adjunctively, substitutively, or providing mixed textuality. In the first case, emoji appear along text within specific points of the written message (e.g., at the end of it) conveying it with emotional tone or adding visual annotations; it requires an overall low emoji competence. In the second case, emoji replace words, requiring a higher degree of competence to understand, not only the symbols per se but also the layout structure of the message, for instance, if we consider syntagms, which are symbols sequentially grouped that together conform a new idea (e.g., I love coffee = ). The third case intertwines text with emoji in a substitutive form rather than adjunctively. This case is the one that requires the highest emoji competence degree, since its decoding requires sophisticated knowledge about rhetorical structures and the proper usage of signs and symbols.

The emoji lexicon possesses generic features such as representationality, which allows signs and usage rules to be combined in specific forms to convey a message. Similarly, any person who is well versed with code's signs and rules is capable of interpreting any message based on the code (i.e., interpretability). However, messages built using the emoji lexicon are affected by contextualization, allowing that references, interpersonal relationships, and other factors affect the meaning of the message [4, 5]. Besides these, the emoji code is composed by a core and peripheral lexicon [2, 5]. As in the Swadesh list, the core lexicon stands for those emoji whose meaning and usage is, somehow, universally accepted and used, even while the Unicode supports more than 1000 different emoji [10]. Within this stand, all facial emoji also contain those emoji that stand for Ekman's six basic emotions such as surprise ( <sup>1</sup> ) or anger ( ) [2, 13]. On the other hand, the peripheral lexicon is constituted by specialized communication symbols such as the one required for marketing, education [14], promoting national identity, or cultural cues [2], among others. Nevertheless, it is worth mentioning that since emoji may be used as nouns, verbs, or other grammatical structure, even those in the core lexicon can be used as a peripheral element in accordance with users' first language, its position within message, or by concatenating several of them into a syntagm.

#### 2.1 How do we use emoji?

Emoji within any message can have several functions; Figure 1 summarizes these. As shown by the latter, one of the most important functions an emoji has is emotivity, which adds an emotional layer to plain text communication. In this sense, emoji serve as a substitute of face-to-face (F2F) facial expressions, gestures, and body language, to state oneself emotional states, moods, or affective nuances. When used in this manner, emoji take the role of discourse strategies such as intonation or

<sup>1</sup> https://emojipedia.org/face-screaming-in-fear/

Figure 1. Emoji functions within the computer-mediated communications.

phrasing [2, 4, 15]. Emoji emotivity mostly conveys positive emotions, hence it can be employed to emphasize an specific point of view, such as sarcasm, while softening the negative emotions associated with it (e.g., with respect to the one that is being sarcastic), allowing the receiver of the message to focus on the content instead of the negativity elicited [2, 14].

Another important role of emoji is as phatic instrument during communication [2, 16]. In this sense, they are employed as utterances that allow the flow of the conversation to unfold pleasantly and smoothly. In this sense, emoji serve as an opener or ending utterance (i.e., waving hand) to open or close a conversation, respectively, maintaining a positive dialog regardless of the content. Similarly, emoji can be used to fill uncomfortable moments of silence during a conversation avoiding its abrupt interruption. Beat gestures are another function of emoji; the former can be defined as a repetitive rhythmical co-speech gesture that emphasizes the rhythm of the speech [9]. For instance, in the same way that keeping nodding up and down during a conversation emphasizes agreement with the interlocutor, emoji can be repeated to convey the same meaning (e.g., ). Keeping in mind that although emoji, neither as utterance nor as beat gesture, explicitly stands for an emotional reaction, they implicitly convey an emotional (positive) tone to the conversation. Likewise, the other function of emoji, which is also implicitly related to emotion, is personality. The latter stands for basal characteristics that have preestablished effects on thoughts, behaviors, and emotions [17]. Been considered a genetic trait, it suffers less variability over time in contrast to emotions and moods [17]. In this sense, emoji can be used to elucidate the underlying personality traits of individuals, either by data mining or by replacing text-based items by their emoji equivalent in personality tests [18].
