Towards Children-Centred Trustworthy Conversational Agents

*Marina Escobar-Planas, Vicky Charisi, Isabelle Hupont, Carlos-D Martínez-Hinarejos and Emilia Gómez*

### **Abstract**

Conversational agents (CAs) have been increasingly used in various domains, including education, health and entertainment. One of the growing areas of research is the use of CAs with children. However, the development and deployment of CAs for children come with many specific challenges and ethical and social responsibility concerns. This chapter aims to review the related work on CAs and children, point out the most popular topics and identify opportunities and risks. We also present our proposal for ethical guidelines on the development of trustworthy artificial intelligence (AI), which provide a framework for the ethical design and deployment of CAs with children. The chapter highlights, among other principles, the importance of transparency and inclusivity to safeguard user rights in AI technologies. Additionally, we present the adaptation of previous AI ethical guidelines to the specific case of CAs and children, highlighting the importance of data protection and human agency. Finally, the application of ethical guidelines to the design of a conversational agent is presented, serving as an example of how these guidelines can be integrated into the development process of these systems. Ethical principles should guide the research and development of CAs for children to enhance their learning and social development.

**Keywords:** chatbots, children, conversational agents, trustworthy AI, ethics

### **1. Introduction**

Conversational agents (CAs) are computer programs designed to engage in a conversation with humans through voice or text-based interactions [1]. Nowadays, their availability and popularity are dramatically increasing. They start to be embedded in a wide range of devices used on a daily basis and in ubiquitous ways: mobile phones, smart cars [2], home devices and even social robots and toys [3].

Very recent improvements in this technology are driven by ground-breaking artificial intelligence (AI) techniques [4, 5]. They are allowing for a brand new generation of CAs, such as ChatGPT [6] and GPT-4 [7], which have demonstrated unprecedented levels of autonomy and natural language processing capabilities. These systems have the potential to transform the way humans interact with computers and machines,

offering simple and intuitive interfaces for a wide range of applications, from customer service [8] to entertainment [9], personal assistance [10], healthcare [11] or education [12]. However, this rapid progress also raises important ethical concerns, particularly when it comes to vulnerable populations such as children [13]. As such, there is a growing need for research on the ethical challenges to be tackled when it comes to the development and deployment of CAs that are conceived or that can potentially be used by children, to ensure that these systems are designed with the best interests of children in mind.

The goal of this chapter is to explore the unique considerations involved in the development and deployment of conversational agents for children. While there is a growing body of research on the ethical development of artificial intelligence in general, much of this work has yet to be fully applied to the specific challenges and opportunities of CAs for children. Our aim is to highlight the key ethical principles and best practices that should guide the design and implementation of these systems, with a focus on promoting safety, privacy and well-being for young users.

To achieve this goal, in Section 2, we begin by reviewing the related work on CAs and children, highlighting the most popular topics, such as educational and health applications. We also identify the opportunities and risks associated with the use of CAs for children. Next, we present ethical guidelines on the development of trustworthy AI, which provide a foundation for the ethical design and deployment of CAs with children. In Section 3, we present a case study that demonstrates the impact of CAs on children's learning and social development. Finally, in Section 4, we present an adaptation of the AI ethical guidelines to the specific case of CAs and children, highlighting the importance of data protection and human agency. We then demonstrate how these principles can be put into practice, offering a concrete example of how to design and deploy a CA that is both effective and ethical.

By the end of this chapter, readers will have a comprehensive understanding of the unique ethical challenges and opportunities presented by CAs for children, as well as a set of best practices for designing and implementing these systems in a responsible and ethical way.

### **2. Related work**

#### **2.1 Conversational agents**

There are various terms used to describe conversational agents (CAs), such as dialog systems, virtual assistants and chatbots. These are all names for computer programs that allow people to interact with them through conversation [1] by using speech, text, or multimodal input/output. They have become popular in recent years and can be found on many devices such as smart speakers or cars. Traditional CAs typically consist of five main modules (**Figure 1**):


**Figure 1.**

*Modules typically included in a conversational agent: ASR (automatic speech recognition), NLU (natural language understanding), DM (dialog management), NLG (natural language generation) and TTS (text-tospeech). The text close to each module summarizes the specific challenges that need to be tackled for their use with children.*


These modules work together to understand and respond to user input in a natural way. At present, many CAs are using advanced machine-learning methods to enhance their performance in one or more of these modules. For instance, BERT models are used to improve NLU [14], Reinforcement learning is used to improve DM [15] and ChatGPT utilizes deep neural networks to perform functions of NLU, DM and NLG in a unified manner.

CAs could be general, that is, without a specific purpose, such as the nowadays hugely popular ChatGPT [6] system. Alternatively, CAs could be intended to perform a quite specific mission, which is known as task-oriented CAs. This task-oriented nature makes CAs dependent on the corresponding use case they are developed for. For example, a CA that is devoted to home automation control expects interactions very different to a CA that is embedded in a social robot devoted to information tasks and guidance in commercial environments. The first system expects direct commands and it would ask for very few clarifications, whereas the other system will communicate in a more human-like form in order to establish a trusted linkage with human users.

Our focus in this chapter is on task-oriented CAs [16], which include tasks such as clothing selection [17], flight booking [18] and driving assistance [19]. CAs designed to assist users in completing specific tasks have become increasingly popular across various industries, such as e-commerce (e.g., purchasing), customer service (e.g., answering frequently asked questions) and healthcare (e.g., scheduling appointments). Some of these tasks are specifically designed for children, such as science learning [20], but CAs intended for adult-centred tasks are also utilized by children [21]. This may be due to the increasing accessibility and ease of use of these technologies for children.

#### **2.2 Conversational agents and children**

CAs are widely used devices, but it is important to consider their accessibility and popularity among children, as even young ones can interact with them through voice commands. In this section, we first present an overview of the literature in the field of conversational agents and children, and then an in-depth analysis of challenges, risks and opportunities identified in the related work.

#### *2.2.1 Bibliometric study*

We performed a bibliometric analysis to collect insights into how the research community has tackled the study of children and CAs. For that purpose, we used the *bibliometrix* tool [22] that allows extracting, among others, the most frequent keywords, clusters and co-occurrences of terms from a corpus of research papers. We collected the corpus of relevant papers from both Web of Science1 and Scopus2 as a result of the following search query over the papers' title, abstract and author keywords:

*(("child" OR "children") AND ("conversational agent" OR "conversational AI" OR "dialogue system" OR "dialogue systems" OR "chatbot" OR "chatbots" OR "virtual assistant" OR "home assistant" OR "voice assistant"*))

We compiled a total of 440 papers, published from 2000 to 2022. Of the 440 papers, about 54% of them have been written in the period 2020–2022, with a clear increasing trend over the years, which is particularly remarkable since 2015 (from 7 papers per year in 2015 to 83 in 2022). This is probably a consequence of the recent popularity and market availability at a relatively low cost of conversational agents and assistants (e.g. Amazon's Alexa, Apple's Siri) and the emergence of large language models such as chatGPT [6].

Interestingly, when adding the terms *"social robot" OR "robot interaction"* to the search query, the number of papers increases from 440 to 2580, meaning that the research community has generally put a great effort into studying the effects of embodiment and the non-verbal side of interaction (e.g. through gestures, gaze or facial expressions).

For the verbal communication side, many studies on human-robot interaction with children actually rely on *Wizard-of-Oz* experimental settings [23, 24]. However, the full automation of voice interaction is already a reality and will be further enhanced by the aforementioned large language models in the near future. Nevertheless, careful attention has yet to be paid to their—still under-explored—maturity, moral capabilities [25] and trustworthiness when it comes to interacting with children.

<sup>1</sup> Web of Science: https://www.webofscience.com/

<sup>2</sup> Scopus: https://www.scopus.com/

#### **Figure 2.**

*Results from our bibliometric analysis of 440 papers focusing on CAs and children: Most frequent terms, cluster and co-occurrence network. The thicker the link, the more weight the co-occurrence of words has. The size of the nodes indicates the frequency of the keyword (the larger the radius, the greater the use) and their color (red, blue or green) denotes which cluster they belong to.*

**Figure 2** shows the results of the keyword frequency, co-occurrence and clustering analysis carried out over the 440 papers related to children and CAs. The bibliometric algorithm identifies three main clusters of terms (nodes). The most populated cluster and the one with more co-occurrences of keywords, is the one represented with red nodes, having the term "controlled study" at the centre, strongly linked to "humans" and demographic aspects including "male", "adult", "adolescent", "female" and "child". This cluster is therefore likely to be related to a large amount of literature presenting in-lab (controlled) experiments, where "adults" frequently act as guardians. The second cluster, with blue nodes and very close to (almost contained in) the first cluster, presents fewer but highly influential keywords, including "communication" and "interpersonal communication", which translates into studies analyzing the communicative—rather than or in addition to demographic—aspects of child-CA interaction in "controlled studies". The third cluster, with green nodes, appears clearly separated from the other two, which is an interesting finding as it contains more technical words such as "artificial intelligence", "user interfaces" and "(embodied) conversational agents". This decoupling of studies focusing on behavioral analyses from the more technical ones might translate into a gap to be bridged in the research community: fostering multidisciplinary collaborations between social scientists and AI/human-machine-interaction researchers, designers and engineers. This is of particular importance in this new era where large language models will pave the way towards fully automated (and less "controlled") human-CA interaction.

#### *2.2.2 Risks and opportunities for children*

Recent studies show that children actively use and explore CAs in home settings more than adults [21, 26, 27]. Thus, it is essential to consider the potential benefits and risks that these devices present for children when developing and implementing them. In this context, we present and expand upon previous work [13, 28] regarding the risks, challenges and opportunities associated with these devices.

According to research, CAs bring several benefits to children, including:


However, as CAs are designed for a general population, their interactions with children may be affected by the unique characteristics and needs of children [41, 42]. Their language and communication abilities, as well as their particular rights, can pose challenges for the various components of the CA (**Figure 1**). The challenges of childrobot interaction have stimulated extensive research aimed at mitigating potential risks. Researchers have identified the following risks in the literature:


Given those upcoming risks, guidelines are necessary to ensure the development of trustworthy CAs. These guidelines should take advantage of the benefits of these devices while assessing and minimizing potential risks and harm they may cause.
