**4. Towards trustworthy conversational agents for children**

In previous sections, we have demonstrated the potential impact of CAs on children underscoring the need for developing trustworthy CAs that are suitable for them. Previous research on AI and children [28] has acknowledged the need to move from the identification of ethical guidelines to practical implementation. Building upon our experience in the study described in Section 3, in this section, we introduce an adaptation of ethical guidelines for CAs that consider children as potential users, along with their implementation in a practical system design.

#### **4.1 Adaptation of ethical guidelines to CAs and children**

A recent study aimed to adapt ethical guidelines for AI systems to the specific case of CAs and children [13]. A team of four experts in computer science, AI ethics and

children's rights evaluated each item of ALTAI (introduced in Section 2.3) in terms of relevance and particular considerations for CAs and children. A Delphi method approach [58] was followed to perform a risk level analysis [59] as follows. The experts rated each ALTAI item based on the likelihood of application and impact on children and CAs. Later on, the individual ratings were analysed to identify critical points and disagreements. Finally, the identified disagreements were discussed and resolved at an expert meeting in order to reach a consensus.

The quantitative analysis consisted of a risk level assessment, using the formula *Risk = Likelihood Impact* to measure the level of risk of the different items and sections in ALTAI. A thematic analysis was also carried out on the annotated comments provided by the experts. The study's main findings are summarized below.


and self-regulation, transparency can be applied by using a language that is adapted to the user's age.

• **Transparency.** According to the experts, transparency is an important factor that can help address several critical considerations mentioned before. The CA trustworthiness could be improved by providing information about the system's nature, privacy and limitations in a language that is appropriate for the user's age.

The findings of this study are largely consistent with previous work on ethical guidelines for AI [28, 51, 52] but provide a more in-depth perspective of the specific issues related to CAs and children. The risk level assessment revealed that although all the identified points are critical, some present higher risks than others (**Table 3**). Particularly, the concerns of children were rated as having a higher impact, while the likelihood of issues was higher for CAs, resulting in a higher overall risk for CAs. Based on these findings, "Privacy and data governance" and "Human agency and oversight" were identified as the two critical requirements that should be prioritized when developing CAs for children.

For more details on the methodology and results of this study, please refer to ref. [13]. The critical points identified in this study, which have been taken into account in subsequent work [60], were applied throughout an agent's design process as illustrated in the subsequent section.

### **4.2 Proposal of a system**

In this section, we introduce a previous study [60] that applied the presented guidelines (Section 4.1) to the development of a conversational agent intended to create a list of games and toys according to user preferences.

The design of the interaction is limited to one user at a time, and the system can ask questions to provide the user with a list of interesting items to choose from. The system can also ask for data such as hobbies, idols and cost limits to fill the user's profile and determine the restrictions on the products that can be offered. The algorithm for the general CA is presented in Algorithm 1.


*a Risk = Likelihood Impact. <sup>b</sup>*

*Total Risk = Risk on Children Risk on CAs.*

*High risk values are marked with an\* (5-9 for Risks to Children and Risks to CAs, and 18-81 for the Total Risk).*

#### **Table 3.**

*Risk assessment results based on expert evaluations of the likelihood and impact of questions in conversational agents and children.*

The article proposed modifications consider children as possible users, including changes to the design, technology, interaction and post-interaction phases. These are summarized in **Table 4** and described in detail in the following paragraphs.

• Regarding the design phase, the "game recommender system" is redefined as a "games wish list" to improve children's interest. A list of relevant stakeholders is defined to gather ideas on how to adapt the system to a children-friendly version. Stakeholders are involved in defining age ranges to be considered, as well as, in time limits for interaction, identification of positive game properties and evaluation of the system.




**Table 4.**

*Recommendations to the design of a CA that generates a list of preferred toys/games for children.*

Overall, the proposed modifications aim to improve the system's inclusivity and reliability, promote critical thinking and decrease overtrust and data disclosure.

### **5. Conclusions and challenges for future research**

The field of conversational agents with children has made significant strides in recent years. Although extensive research has been conducted on the ethical development of artificial intelligence (AI), in general, there has been a growing emphasis on employing AI in interactions with children. Such work has emphasized the need for systems that enhance opportunities while mitigating risks.

However, the development and deployment of conversational agents for children come with specific challenges and ethical and social responsibility concerns. This chapter is dedicated to exploring the particular considerations conversational agents should have when interacting with children.

The chapter reviewed related research on conversational agents and children, identifying popular topics as well as opportunities and risks. Worldwide ethical guidelines on the development of trustworthy AI were presented as a framework for the ethical design and deployment of conversational agents with children. These guidelines emphasize the importance of the protection of user rights in the development and deployment of AI technologies. Additionally, a case study was presented that demonstrated the significant impact conversational agents can have on children's learning and social development.

An adaptation of previous AI ethical guidelines to the specific case of conversational agents and children was also presented, highlighting the importance of data protection and human agency. The application of ethical guidelines to the design of a conversational agent presented in this chapter served as an example of how these guidelines can be integrated into the development process of these systems.

It is important to note that the variety of conversational agent systems requires personalized study and application of these guidelines for each case. Furthermore, even state-of-the-art technology may not be able to address some proposed considerations, such as ASR modules that cannot understand little children's speech in less popular languages. Therefore, researchers in this area should continue to strive towards achieving new breakthroughs that enable the development of more ethically sound devices for the benefit of future generations.

In summary, the development and deployment of conversational agents with children require a careful balance between innovation and ethical responsibility. Ethical principles should guide research and development, and systems should be designed with the safety, privacy and well-being of children in mind. By doing so, conversational agents have the potential to be a powerful tool for enhancing children's learning and social development.

### **Acknowledgements**

This work was carried out with the support of the Joint Research Centre of the European Commission in the framework of the Collaborative Doctoral Partnership Agreement No.35500.

### **Conflict of interest**

The authors declare no conflict of interest.

*Towards Children-Centred Trustworthy Conversational Agents DOI: http://dx.doi.org/10.5772/intechopen.111484*
