**1. Introduction**

In this chapter I want to show how educationalists might (re)conceptualise digital tools and also reconceptualise teaching and learning as a social practice with digital tools. I do this by focusing on the development of a theoretical and analytical framework for identifying signs-as-agents that was shown to be applicable to other

educational tools and scenarios. In themselves, the proposals may not be very novel, since they include or build on well-established traditions in various fields, though perhaps less so in the field of Education. However, I wish to suggest that a Semiotic Technologies approach, focused on signs-as-agents, applied to educational scenarios, can unite a number of well-established perspectives on digital tool use. Such an approach can also be very relevant for researchers and practitioners working with STEAM and STEM.

Through the presentation of three examples, from various educational, digital scenarios, I will show how the theoretical and analytical framework, involving the identification of initiation and response turns, understood as a multimodal exchange structure, can be used to identify screen-based signs-as-agents. In addition, I show how through the establishment of such multimodal exchange structures, signs can be understood as being agentalised. Throughout this process, I highlight how theoretical and contributions from research in Human-Computer Interaction [1]; [2–4], Social Cognition Theory [5], Critical digital literacy [6–8], Distributed Cognition Theory [9] as well as theoretical and methodological contributions from a Semiotic Technologies perspective [10, 11], can support a more holistic and critical understanding of agency and agents – both 'human' and 'digital' – in educational scenarios. The central contribution of this chapter is a typology of signs-as-agents that has been conceived in order to expand the Critical Digital Literacies teaching and research agenda specific to pedagogy. It is aimed at supporting critiques of how digital tools can shape how teachers and learners *act*, rather than how we *think*. The latter has been the traditional domain of critical digital literacies.

The term Semiotic Technologies refers to "the whole array of technologies people use in order to make meaning as part of specific social practices, such as writing and drawing on a blackboard with chalk in mathematic lessons in schools, photographing beaches and mountains for image bank stock photos, or taking a selfie and uploading it on Instagram for sharing with friends and acquaintances" (p. 596) [12]. A Social Semiotic Technologies perspective is a growing subfield of social semiotics [12]. An analytic model for analysing semiotic technologies has been developed within this perspective [10, 11], that according to Djonov and Van Leeuwen [13] involves an analysis of the semiotic practices and artefacts (e.g., a written text or illustration) that warrant critical attention, as well as an analysis of the technologies that support them. Such a critical multimodal study of software, that includes the evaluation of software design, its use, and its relationship to broader semiotic and cultural practices would conceivably offer a holistic analytical approach [13] to digital technologies and their use in educational scenarios.

While a small number of studies have focused on digital tools in education from a social semiotic perspective (e.g., [14–17]) the model for analytic separation of artefact and practices developed by (Zhao et al. [10] and Zhao and Van Leeuwen [11], has only more recently been applied to other digital tools in education (e.g., [10, 11, 18]). This approach to the study of the relationship between semiotic technology and social practices is highly relevant and applicable for the study of technologies in digital-based teaching and learning practices because teachers may not be the only actors (or agents) participating in what appears on the interface pages that students face [18].

I will argue that this approach is fruitful for understanding agency in digital scenarios because, following the emphasis on analytical separation, it conceivably allows us to look at how agency and agents may present themselves in relation to the digital tool itself, as well as in the social practices by teachers and/or learners with the tool. Another relevant contribution to the study of agents from a Semiotic Technologies perspective is the notion of "exchange structures" that can be realised

#### *Rethinking 'Affordance', 'Agency' and 'User' from a Semiotic Technologies Perspective… DOI: http://dx.doi.org/10.5772/intechopen.99699*

multimodally through "user-user interactions" and "system-user interactions" and must at least contain an initiating move and a response [19]. Such an analytical focus conceivably would give insight into the intentions of human or digital agents as exchanges take place between the teacher or learner and system as well and may also give insight into how these exchanges can shape their semiotic work.

My arguments for a Social Semiotic Technologies approach are based on a critique of three premises that often underly much discourse in studies surrounding digital tool use in educational scenarios. The first premise is that digital tools should be evaluated and researched for their 'affordances' [20] as potential uses for supporting human intentions. While countless studies within the field of Education attest to this premise, it is important to highlight that a digital tool with a screen does not typically have a singular design intention, unlike a chair (which the notion of 'affordance' was initially applied to). But rather, digital tools are increasingly more complex, and are increasingly seen as having unlimited potentials, including new roles like instructing or advising humans (e.g., service bots with Artificial Intelligence) [18]. An interface that learners may face, for example, may have been designed by multiple people including interface designers and teachers and may have also passed through the hands of translation or legal teams for final touches with logos or orthographic corrections. Furthermore, a screen might present the user with many digital tools at once, so that grammar checkers, for example, can be installed 'within' other tools so that the affordances across multiple tools can be customised to each user. Designers may be multiple and the final product or digital tool(s), often incorporating a screen, may be the result of many months or years of design input by each one: each with their own intentions imbued in the presence of their signs.

The second premise is that digital tool use is favourable in supporting individual learner empowerment. Many studies in the field of Education also attest to this. Within a university environment, for example, e-Learning tools can create 'leanercentric' and 'collaborative-learning environments' where learners are empowered to self-control their learning processes [21]. However, this perspective, which can be understood as part of the 'empowerment paradigm' [22] is often typically focused on learners and their agency rather than the potential agency that digital resources on the screen may 'enact' through learners' use of them. This enactment of 'digital' agency, or agency enacted through digital means, can be traced back to studies in Human-Computer Interaction from the 1980's with the creation and the identification of pedagogical agents as 'passive' or 'active' [1, 4]. A passive agent waits until it receives a request to act, while an active agent takes action on its own when it sees an opportunity to do so. These studies have highlighted the technological intentions to communicate and/or shape human action, that originate in the design of human others. More recent studies in Social Semiotics (e.g., [10, 11]) and linguistics (e.g., [23]) highlight the ability of digital technologies to shape and influence human semiotic work. This shaping potential does not refer to the shaping potential of ideas or ideology communicated typically through text and image, but rather its shaping potential is carried out through 'exchange structures'. 'Exchange structures' is a notion that was originally drawn from conversational analysis and functional linguistics. Following this understanding of exchange structures, exchanges must at least contain an initiating move and a response, which can be realised multimodally. Jovanovic and Van Leeuwen [19] also mapped out the ways that digitally-mediated interactions occur across various forms of social media, including "user-user interactions" and "system-user interactions", in what the authors describe as "a multimodal realisation of an exchange structure". This notion has also been reflected in educational research as 'semiotic initiations or responses'

[24] that can be understood as part of 'multimodal turn-taking' [25]. An example of this is a pop-up that 'requests' a user to accept cookies and then the user clicks the screen-based icon as the 'acceptance'. These exchange structures have attributes of design, intention and enactment that can be assigned to constructs of agency. Therefore, the learner as a user may be only one social agent that is being empowered when digital tool use is being enacted: the social interests of each screen-based resource and how they may be empowered through turn-taking with them, should conceivably also be considered when learners carry out their semiotic work. This is especially the case in educational scenarios because, as Djonov and Van Leeuwen [26] propose, there are social interests that permeate the design and use of software.

The third premise is that a screen is solely for learners as 'end' users. However, unlike a page, whatever is communicated by the user through the screen, is connected to a network. The information that is passed through a screen, either entered by humans through typed text or through navigation and touch, is at least a two-way information flow. Therefore, resources on a screen can conceivably be considered more than just signs on an interface in which meaning is communicated to the user. Indeed, it could be argued that there is a relationship between humans and the screen, not only in terms of content (e.g., ideas communicated through text that is read or watched by users) but also in terms of a relationship made up of a human-digital system that is carried out through "exchange structures" [19]. With this altered conceptualisation, the screen is not just for the human users at the 'end' but also for digital and human 'users' at the other 'end' of the exchange which teachers and/or learners do not see. The screen is the point at which, however, potentially part of the exchange structures between the human and digital 'system' can be made visible to teachers and/or learners.

The three premises that I have briefly described above can be critiqued within a Social Semiotic Technologies perspective. I propose that it is therefore a potentially valuable perspective in advancing the development of Critical Digital Literacies [6–8] within educational scenarios which Pangrazio [27] has proposed is in need of expanding. Such an expansion could involve a focus on identifying signs-as-agents. While many definitions of Critical Digital Literacies exist, Luke [28] refers to it as processes of "naming and renaming the world, seeing its patterns, designs and complexities, and developing the capacity to redesign and reshape it" [28] (p. 29).

Such a critique, broadly focusing on 'patterns', 'designs' and 'shaping' [28] signals a number of shifts in how educationalists might (re)conceptualise digital tools and their use through a critical lens.

Firstly, this reconceptualisation is a shift in emphasis from the notion of a digital tool with a singular design intention, to a tool with multiple design intentions and designers and designs with various signs that may change meaning, patterns of use and use for different purposes over time. Secondly, while many studies in the field of Education attest to digital tool-use as being linked with learners' individual and/ or social empowerment, if learners' creations can be shaped in the process of tool use, students' digital semiotic work might be conceivably re-conceptualised more accurately as a *meaning negotiation* process rather than (solely) as an empowering, creative one. That is to say, 'negotiating' what semiotic work from pre-built designs in digital tools by other designers is accepted/rejected, included/excluded and/or suggested/left out by teachers and/or learners. This conceptualisation highlights that digital tool use in (social educational) practices may be a site of struggle between all potential agents in the creation, negotiation, use or re-use of a semiotic piece of work. This in turn, signals the need for a more critical stance towards power relations with digital tools while in use.

#### *Rethinking 'Affordance', 'Agency' and 'User' from a Semiotic Technologies Perspective… DOI: http://dx.doi.org/10.5772/intechopen.99699*

Thirdly, because a Social Semiotic Technologies methodological approach involves analysis of digital tool use and the tool itself, including design intentions of the tool, this approach conceivably also supports an analytical as well as a theoretical shift. This shift may be from tool-use and its affordances towards tool-use for different social practices. I propose that this approach can facilitate a space for identifying and analysing the various 'voices' of the designers and/or creators that may be 'at play' at any point in its use/re-use.

In the next section (Section 2), I will focus on the theoretical and analytical framework for identifying signs-as-agents, how this can be applied to other educational tools and scenarios and finish with some conclusions.

Section 2 focuses on the theoretical framework for identifying signs-as-agents. It highlights the framework's compatibility with theoretical contributions from Human Computer Interaction [1–4], Social Semiotics [29–31], Social Cognition Theory [5], Critical digital literacy [6–8] and Distributed Cognition Theory (see Hutchins [9]).

Section 3, illustrates how signs (or screen-based resources) can be identified and agentalised through use, thus becoming agents. For this, an exploratory study of 'hidden' digital agents in an online language scenario with a digital App called Tandem [18] is summarised in order to highlight key findings and demonstrates how an analytical tool for identifying signs-as-agents emerged.

Sections 4 and 5 focus on how the theoretical and analytical insights from the exploratory study, outlined in Section 3, can be applied to other digital tools and social practices that are commonly used in educational contexts: namely two webpages for Webquests and two 'Google for Education' tools for communication purposes between teachers and students.

Finally, I will conclude with some arguments as to why it is necessary for educationalists involved in educational, digital scenarios such as STEM and STEAM to critically rethink the three premises of 'affordance', 'agency' and 'user' and how this might be carried out.

#### **2. Signs-as-agents**

Sign is a key concept in traditional semiotics. "Signs are elements in which the signified ('meaning') and signifier ('form') have been brought together. Social semiotics holds that the process of sign-making is subject to the interest of signmakers, their availability of semiotic resources and the aptness of those resources to the meanings which they wish to realise" (p.3) [31]. That is to say, the relation between 'form' and 'meaning' is not arbitrary but motivated [29]. While the notion of 'sign' emphasised available resources as part of a system, the notion of 'semiotic resource' that also evolved from social semiotics, focused on how the context of communication and the sign maker shaped signs and meaning. It was focused on people's situated choice of resources rather than a system (p. 3) [31]. This is relevant to our understanding of digital tools and their use because whereas the screen may offer learners signs, that in the designers' intentions have a 'logic' which connects the signs through layout and colour for example, learners can also bring other resources from their own environment. Such resources can include the choice of language that they use while using the tool or their decision about what to include in the background if they are making a video call. Van Leeuwen [30] describes semiotic resource as follows:

Semiotic resources are the actions, materials and artefacts we use for communicative purposes, whether produced physiologically – for example, with our vocal apparatus, the muscles we use to make facial expressions and gestures – or technologically – for example, with pen and ink, or computer hardware and software –

together with the ways in which these resources can be organised [30]. Van Leeuwen [30] also noted that the notion of 'resource' began to replace the notion of 'sign'.

The distinction between 'sign' and 'resource' is important in understanding a digital tool in use because it allows teachers and researchers to choose whose perspective we are looking at a screen-based resource from, at any given time. That is to say, the intentions of the pre-embedded signs on the screen by an interface designer for example, or alternatively, the intentions of the learner while using the signs on the screen, or indeed, the incorporation of their own semiotic resources made available through the screen. A Semiotic Technologies approach allows for both perspectives and analysis of intentions because it attends to a) the materiality of the digital tool and also b) how that tool is used in a social practice. Intentions are an important aspect of the construct of agency because human goals, whether related to expression of identity or carrying out and completing a task, are not achieved without intentionality.

To understand 'signs' as potential agents that can take part in exchanges with humans, we can draw from studies in Human Computer Interaction (henceforth HCI). Knight, Dooly and Barberà [18] highlighted how the screen-based signs, were considered as pedagogical agents in HCI studies in the following way:

Pedagogical (non-human agents have been classified into two different categories: animated pedagogical agents [3] and reactive pedagogical agents [2]. Animated agents simulate human behaviour, such as facial expression, body movement and gesture whereas "reactive agents" respond to events in the environment, for instance displaying messages when certain threshold values have been reached. According to Jondahl and Mørch [4], reactive agents can be further classified as passive or active: a passive agent waits until it receives a request to act; an active agent takes action on its own when it sees an opportunity for doing so.

From the field of psychology, Social Cognition Theory, developed from Social Learning Theory (SLT) by Albert Bandura in the 1960s, also contributed significantly to theory on human agency. There are two notions that are relevant to researching signs-as-agents from Social Cognition Theory. The first is Bandura's notion of "proxy agent" [5] and the second is a systems-based understanding of agency, all be it human systems.

The role of a "proxy-agent" [5] is where a (human) agent can be enacted or represented by others (e.g., a parent completing a task on behalf of a child). To extend this notion, signs on a screen can be understood as proxy agents that act on behalf of their designers in that they are designed to communicate an idea or act on the designer's/designers' behalf.

Bandura's [5] human, systems-based understanding of human agency highlights the motor, sensory and cognitive systems in carrying out of intentional actions. This notion connects with important notions from the field of Social Semiotics, namely 'signs', that can be made up of 'modes', such as image, text and speech. The motor and sensory systems of humans are conceivably co-reliant on such modes in order to be put to use by humans to carry out their intentional actions. If humans initiate or respond to signs on the screen through their sensory or motor systems, the interaction conceivably becomes a shared human-computer one. This interaction can be characterised as "user-user interactions" and "system-user interactions" [19]: the screen facilitates oral/visual interaction between humans and users are faced by a number of signs on the screen to which they can respond to in various ways in what Jovanovic and Van Leeuwen describe as "a multimodal realisation of an exchange structure" [19]. In addition to this, a "user-system" initiated exchange [18] can also be considered a structure, given that users can initiate 'turns' with the digital system, through touch for navigation or voice activation for example. This

*Rethinking 'Affordance', 'Agency' and 'User' from a Semiotic Technologies Perspective… DOI: http://dx.doi.org/10.5772/intechopen.99699*

human computer exchange foregrounds the notion of a systems-based notion of agency: a recognition of human systems to carry out intentional actions and also of digital systems to establish or carry out intentional actions. This understanding of agency recognises digital tools and established human experience as forming "unified ecologies, with agency distributed throughout the system [as] artefacts, context, and humans together create particular morphologies of action" (p. 9) [32]. This notion stems from Distributed Cognition Theory, a theoretical framework that was originally introduced by Edwin Hutchins and his colleagues in the mid-1980s (see Hutchins [9] for the principles of Distributed Cognition).

Following on from the theoretical framework, Section 3 illustrates how signs (or screen-based resources) were identified and agentalised through use, which led to the key concept presented in this section (Section 2) of 'signs-as-agents'. Section 3 revisits the data from an exploratory study, spanning over 5 years, that focused initially on learner agency in an online language learning scenario to develop speaking skills [23, 25, 33] and evolved into a focus on 'hidden' digital agents from a more critical perspective [18].

#### **3. Signs-as-agents in the Tandem tool: The exploratory study**

In order to illustrate how educationalists and researchers might research and identify signs-as-agents, rather than just affordances of digital tools, it is pertinent to revisit the first data set from an exploratory study and its relevant findings. The signs-as-agents in the Tandem tool were initially identified in a study by Knight, Dooly and Barbera [18] but the term 'signs-as-agents' has not been used in relation to the exploratory study until this section.

The following screenshots and analysis relate to a language learning App called Tandem which is an audioconferencing tool that facilitates oral interaction between students in a synchronous mode. Students cannot see each other. Instead, they are faced with various textual instructions, navigation buttons and texts in order to support the student interaction.

As in keeping with a Semiotic Technology approach, the materiality of the tool, in this case the screen-based resources or signs, were analysed separately from their use. However, also in keeping within a Semiotic Technology approach, the tool inuse by students was also analysed in the form of the audio recordings of what students were saying during the process of interacting with the screen as well as each other. This gave insight into intentions and actions taking place during the learning process from the student perspective.

Pop-ups (**Figures 1** and **2**) appeared on the screen at different moments, "inviting" students to "respond" by clicking on the screen 'button' provided, namely 'Start' (**Figure 1**) and 'Close' (**Figure 2**). These were identified and later labelled as 'active agents' according to the classification of Pedagogical agents from studies in HCI, shown in **Figure 3**.

Navigation resources such as 'Next Task' (**Figure 4**) and 'See solution' (**Figure 5**), were identified and later labelled as 'passive agents'. Passive agents were available on the screen and when students wanted to use them to move forward (or back) through the task sequence, they clicked on them.

This identification highlighted the 'semiotic initiation and responses' [24] or 'multimodal turn-taking' [25] that was carried out between student and the screen, also while students were interacting orally. Consequently, these screen-based resources or signs were considered to be "direct discourse agents" [18] because they could carry out turns with humans as 'agentive turn takers' [23]. When initiations, either human or digital, met with a response (e.g., from the screen 'moving on' to
