We are IntechOpen, the world's leading publisher of Open Access books Built by scientists, for scientists

6,700+

Open access books available

180,000+

International authors and editors

195M+ Downloads

156

Countries delivered to

Our authors are among the

Top 1% most cited scientists

12.2%

Contributors from top 500 universities

Selection of our books indexed in the Book Citation Index in Web of Science™ Core Collection (BKCI)

## Interested in publishing with us? Contact book.department@intechopen.com

Numbers displayed above are based on latest data collected. For more information visit www.intechopen.com

## Meet the editor

Asun López-Varela Azcárate is an associate professor at the Universidad Complutense Madrid, Spain. Her research interests are comparative literature, cognitive and intermedial semiotics, and cultural studies. Since 2007, she has been coordinating the research program "Studies on Intermediality and Intercultural Mediation" (SIIM). She has been awarded various grants, including a Fulbright Visiting Scholarship at Harvard

University. A proactive member of the profession, Dr. López-Varela is vice chair of the European Commission Marie Skłodowska-Curie Postdoctoral Fellowships, Social Sciences and Humanities panel. She is also the coordinator and chief editor at the New Directions in the Humanities Network. To strengthen relations between Europe and Asia, Dr. López-Varela coordinates an annual seminar series on cross-cultural dialogue and sustainability funded by the Eurasia Foundation.

## Contents


## Preface

The denomination "visual arts" includes a wide range of artistic forms that have evolved following changes in the use of technical media. The first human settlements already had the knowhow to engage in artistic textile design, cave painting, writing on different materials, creating the first sculptures, and establishing the foundations of architecture. New technological developments, like the invention of the printing press, occurred in relation to aspects such as the new materiality of paper formats discovered in China. Gradually, printing also evolved to include images by means of engraving, etching, and other techniques. While painting on different surfaces has existed since ancient times, the 19th century saw the birth of photography and, soon after, film and later video. With digital media convergence, screens have become part of human lives. The visual arts are no longer only in museums. Exhibitions can be accessed from a wide range of locations, just by searching on our mobile screen devices. The visual arts also have ties with other disciplines like graphic and fashion design, interior decoration, and the performing arts, all of which rely on various aspects of visuality. Contemporary screen-based art can be seen as a scenario for social engagement, activism, and education, as some of the chapters in this volume highlight.

Seeking to explore how the visual arts are reshaped by technology and new materials, Chapter 1, "Literature Review on Intermedial Studies: From Analogue to Digital" by the editor of the volume, Asun López-Varela, looks at "intermediality," a multi-layered concept that involves a cluster of aspects ranging from sensorial configurations to material and technical media, combining also the sociological and cultural aspects that qualify media and cause a particular technical apparatus to be developed, socially accepted, and popularized. Intermedial studies explore art forms as having no borders, perhaps because of the neuroplasticity of the human brain. Thus, the volume interrogates the different forms that the visual arts can take, seeking to study their complexity and plasticity both synchronically, through particular contextualized case studies, and diachronically, contemplating their evolution in relation to changing material media. The opening chapter presents an overview of significant interdisciplinary contributions that have shaped the field of intermedial studies, showing how it encompasses various areas such as media and communication studies, art history and, of course, the visual arts, which include theatre, dance, and performance along with sequential art (comics, graphic novels), photography, radio, film studies, electronic literature, videogames, and more recently artificial intelligence.

Chapter 2, "Cut-Up Transmediality: From Text to Image", by Benjamin J. Heal explores the practice of image-modeling, composition, and the 'cut-up' technique. In this chapter, the focus is on the visual elements of William S. Burroughs and Brion Gysin's "cut-up" project. Instead of categorizing it as intermedial, the chapter examines their use of this collage technique, which is primarily based on chance, as "transmedial." Burroughs' and Gysin's transition from applying the technique to

text and extending it to other forms of media, like sound and film, is considered a remarkably forward-looking exploration that foreshadowed the concept of transmedia storytelling. To provide a theoretical framework for understanding the cut-ups within a broader context of resistance against conventional norms, the chapter draws on various theories, including recent studies on trans- and intermediality. By doing so, the chapter explores how the transmediality of the cut-up project disrupts language and time, ultimately expanding the field and introducing new areas for analysis.

In Chapter 3, "Towards a Semiotics of Visual Music", Dr. Shaleph O'Neill delves into the study of Visual Music, a concept with a rich historical background. Throughout history, numerous distinguished scientists, musicians, and artists have attempted to establish connections between sound and vision. These endeavors have varied, with some grounded in scientific principles, others rooted in genuine synesthetic experiences, and still others stemming from creative aesthetic choices. Given the renewed interest in this field, the argument put forth in this chapter is that it is now a good time to reassess the existing body of knowledge and to clearly identify and expand upon the central concepts it encompasses. The chapter takes a semiotically informed perspective and examines a selection of works by pioneering filmmakers from the 20th century. This analysis aims to reevaluate the historical aspects of Visual Music in conjunction with fresh insights from fields such as psychology and neuroscience. With this objective in mind, the chapter outlines a set of principles or parameters that arguably form the foundation for creative approaches that involve the translation between sound and vision.

Chapter 4, "Audiovisual Music from the Audience's Viewpoint", by Amparo Porta-Navarro explores the intersection of music, the visual, and narrative elements. Screen spaces create a heightened emotional tension through the inclusion of characters, settings, stories, and roles, resulting in an immersive experience. The analysis includes examining the use of leitmotifs, music as a central element and background accompaniment, diegetic and extra-diegetic music, as well as exploring the emotional impact on the audience's perception. To approach these topics, a mixed method approach is adopted, utilizing tools such as literature reviews, content analysis, and mapping. These methods facilitate the exploration of relationships and specific aspects while analyzing the connections between music, images, and narratives.

In Chapter 5, "From the Crippled Devil to the Spectacular House: The Visual Representation of Intimacy in Painting, Comics, Film, and Television", Carolina Sanabria explores the visual manifestations of the interior of homes, expanding upon the literary theme. The primary reference point in the article is the concept of the crippled devil, which originates from 17th-century picaresque literature. This period coincides with the emergence of visual representations of everyday life in Dutch painting. The study focuses on the visual and reflective evolution that stems from these paintings and evolves into panoramic views within the graphic structure of mid-19th-century vignettes. Over time, this evolution continues in the realm of comics, then in cinema and television, mirroring the structure of literary movements. The content of these representations, which involves integrating characters within the same location, reflects a satirical exhibition of social manners. Television, in particular, showcases humorous situations among its characters, contributing to the success of emblematic series such as *Aquí no hay quien viva*, where the viewers

**V**

take on the role of the snooping observer, similar to Hitchcock's *Rear Window*. The protagonist of these visual representations, expressed by the revived crippled devil, extends beyond mere playful snooping. Consequently, the portrayal of the interior of homes conveys a sense of vigilance, especially when considering modern houses specifically designed for observation, such as those seen in the TV show Big Brother,

In Chapter 6, "Transmedia Storytelling of Liangzhu Culture Inscribed Symbols", Jinghua Guo and Weile Weng undertake an exploration of the archaeological ruins of Liangzhu. The site, which was added to the UNESCO World Heritage list in 2019, holds significant materials for studying Liangzhu culture and its inscribed symbols. As media platforms have evolved from traditional paper to digital formats, the integration of technology has introduced virtual reality (VR) as a new narrative medium. This chapter investigates the potential of transmedia storytelling in giving visibility

to Liangzhu's culturally inscribed symbols in recent VR exhibition projects.

connections, and the integration of diverse fields of study.

prevalent in contemporary visual arts.

The volume closes with Chapter 7, "Intermediality in Contemporary Visual Art Education", by Bea Tomšič Amon, which explores the impact of the visual arts in educational environments. The chapter argues that the term "fine arts education," commonly used to describe the subject in schools, no longer adequately captures the goals of artistic expression within an educational context. Contemporary visual education embraces a holistic approach that recognizes the intermedial nature of modern visual art. The author also presents reflections on the pedagogical process through a concrete example of an intermedial performance created by secondary school students, which allows for discussions on multisensory perception, interdisciplinary

This book delves into the realm of intermediality within the visual arts, examining various aspects in each chapter. It covers the evolution of Intermedial Studies in recent decades, the transformative nature of print typography, and the emergence of "cut-ups" as a means of resistance against conventions. It also explores the concept of Visual Music in relation to pioneering filmmaking and analyzes the evolution of visual representations of intimacy across different visual formats like painting, comics, film, and television. Additionally, the book investigates the transmedial potential of cultural symbols in virtual reality, emphasizing the incorporation of multimodal and emotional elements to enhance audience immersion. The book concludes by highlighting the necessity for a comprehensive approach to visual art education and pedagogical methods that foster creativity, underscoring the intermedial aspects

> **Asun López-Varela Azcárate** Complutense University Madrid,

> > Madrid, Spain

intentionally created to be observed.

take on the role of the snooping observer, similar to Hitchcock's *Rear Window*. The protagonist of these visual representations, expressed by the revived crippled devil, extends beyond mere playful snooping. Consequently, the portrayal of the interior of homes conveys a sense of vigilance, especially when considering modern houses specifically designed for observation, such as those seen in the TV show Big Brother, intentionally created to be observed.

In Chapter 6, "Transmedia Storytelling of Liangzhu Culture Inscribed Symbols", Jinghua Guo and Weile Weng undertake an exploration of the archaeological ruins of Liangzhu. The site, which was added to the UNESCO World Heritage list in 2019, holds significant materials for studying Liangzhu culture and its inscribed symbols. As media platforms have evolved from traditional paper to digital formats, the integration of technology has introduced virtual reality (VR) as a new narrative medium. This chapter investigates the potential of transmedia storytelling in giving visibility to Liangzhu's culturally inscribed symbols in recent VR exhibition projects.

The volume closes with Chapter 7, "Intermediality in Contemporary Visual Art Education", by Bea Tomšič Amon, which explores the impact of the visual arts in educational environments. The chapter argues that the term "fine arts education," commonly used to describe the subject in schools, no longer adequately captures the goals of artistic expression within an educational context. Contemporary visual education embraces a holistic approach that recognizes the intermedial nature of modern visual art. The author also presents reflections on the pedagogical process through a concrete example of an intermedial performance created by secondary school students, which allows for discussions on multisensory perception, interdisciplinary connections, and the integration of diverse fields of study.

This book delves into the realm of intermediality within the visual arts, examining various aspects in each chapter. It covers the evolution of Intermedial Studies in recent decades, the transformative nature of print typography, and the emergence of "cut-ups" as a means of resistance against conventions. It also explores the concept of Visual Music in relation to pioneering filmmaking and analyzes the evolution of visual representations of intimacy across different visual formats like painting, comics, film, and television. Additionally, the book investigates the transmedial potential of cultural symbols in virtual reality, emphasizing the incorporation of multimodal and emotional elements to enhance audience immersion. The book concludes by highlighting the necessity for a comprehensive approach to visual art education and pedagogical methods that foster creativity, underscoring the intermedial aspects prevalent in contemporary visual arts.

> **Asun López-Varela Azcárate** Complutense University Madrid, Madrid, Spain

**1**

**Chapter 1**

**Abstract**

Intelligence.

**1. Introduction**

among different medial forms.

*Asun López-Varela Azcárate*

transmediality, visual arts, videogames, AI

Literature Review on Intermedial

Studies: From Analogue to Digital

This chapter traces an overview of the evolution of the research on Intermedial Studies in the last two decades. It expands the research presented in the InTech volume *Comparative Literature. Interdisciplinary Considerations,* in a chapter entitled 'Intermedial Comparative Literature: from the Sister-Arts Debate to the twentieth century Avant-gardes'. The literature review offers a description of the major interdisciplinary contributions that have shaped the field of Intermedial Studies, with areas such as media and communication studies, art history, and the visual arts, including theater, dance and performance, sequential art (comics, graphic novels), photography, radio, film studies, electronic literature, videogames and Artificial

**Keywords:** art, digital convergence, ekphrasis, intermediality, medium, narration,

The word 'medium' has its roots in classical Latin to refer to the middle of something or an intermediate course of action. The Oxford English Dictionary (OED) defines the term in a dual way, as the means or agency through which communication takes place, often synonymous with channel, and as the physical vehicle of expression employed in a representation, ranging from general categories of artistic, technical forms or modes of expression (e.g. photography) to specific materials, tools, and methods. In linguistic discourse, the concept may variously refer to language; speech or writing (i.e. the phonic medium and the visual-iconographic medium), including, for instance, distinctions between handwriting and print. In communication, 'medium' often refers to a specific technical form for interpersonal exchange, for example, via post, telephone, or computer-mediated communication. In its plural form, 'media', it names the instrumental use in mass communication (i.e. newspapers, television, etc.) that appears in everyday practices, hinting also at the interactions

The variety of meanings attached to the notion of 'medium' shows that it stands as a boundary mark that signals the crossroads between its mediating (semiotic) agency and its material format (the physical vehicle or technological artifact used to inscribe any type of representation). In this regard, it is important to consider that relations in the world are simultaneously material (between things) and semiotic (between

#### **Chapter 1**

## Literature Review on Intermedial Studies: From Analogue to Digital

*Asun López-Varela Azcárate*

#### **Abstract**

This chapter traces an overview of the evolution of the research on Intermedial Studies in the last two decades. It expands the research presented in the InTech volume *Comparative Literature. Interdisciplinary Considerations,* in a chapter entitled 'Intermedial Comparative Literature: from the Sister-Arts Debate to the twentieth century Avant-gardes'. The literature review offers a description of the major interdisciplinary contributions that have shaped the field of Intermedial Studies, with areas such as media and communication studies, art history, and the visual arts, including theater, dance and performance, sequential art (comics, graphic novels), photography, radio, film studies, electronic literature, videogames and Artificial Intelligence.

**Keywords:** art, digital convergence, ekphrasis, intermediality, medium, narration, transmediality, visual arts, videogames, AI

#### **1. Introduction**

The word 'medium' has its roots in classical Latin to refer to the middle of something or an intermediate course of action. The Oxford English Dictionary (OED) defines the term in a dual way, as the means or agency through which communication takes place, often synonymous with channel, and as the physical vehicle of expression employed in a representation, ranging from general categories of artistic, technical forms or modes of expression (e.g. photography) to specific materials, tools, and methods. In linguistic discourse, the concept may variously refer to language; speech or writing (i.e. the phonic medium and the visual-iconographic medium), including, for instance, distinctions between handwriting and print. In communication, 'medium' often refers to a specific technical form for interpersonal exchange, for example, via post, telephone, or computer-mediated communication. In its plural form, 'media', it names the instrumental use in mass communication (i.e. newspapers, television, etc.) that appears in everyday practices, hinting also at the interactions among different medial forms.

The variety of meanings attached to the notion of 'medium' shows that it stands as a boundary mark that signals the crossroads between its mediating (semiotic) agency and its material format (the physical vehicle or technological artifact used to inscribe any type of representation). In this regard, it is important to consider that relations in the world are simultaneously material (between things) and semiotic (between

concepts). Thus, a given medium 'mediates' these relations. A medium is more than a channel for the transmission of information; only potentially meaningful. It may refer to a specific material (including nonvisible aspects such as the wave dimensions of the phonic medium that characterizes oral speech) or a technology, more or less complex.

The materiality of media is physically and culturally encoded. The term 'multimodality' denotes the perceptual modes at play in perception and in the use of the medium, whether given in nature or technologically created. It also refers to the affordances enabled by the medium. For instance, while the page of a printed book might afford a visual perceptual experience to most people, those with impaired vision require a page printed in Braille for a tactile experience.

Lars Elleström's opened a debate to explore the differences between concepts like multimodality and intermediality. While the first referred to sensory configurations, intermediality was a complex cluster that involved not only perceptual modes but also technological, sociological and cultural aspects. The sociocultural context 'qualifies' media, causing a particular technical apparatus to be developed, accepted and popularized [1]. Elleström went on to refine his classification of intermedial and intramedial relations that can occur among 'qualified' media as well as among media products from similar or different, also 'qualified' media, noting that these categories are not borders but, rather, plastic contact zones. He also tackled the issue of transmediality [2, 3] (more on this below).

The history of Intermedial Studies is impacted by an entire ecology of communication and productions that can be tackled from a variety of disciplines. These include the material medium, closely related to changing technologies and apparatuses, the affordances of medium specificity, delivery mechanisms and channels, and circulation and reception practices, embedded in particular institutional and sociocultural contexts.

While Dick Higgins's contribution to the concept of 'intermedia' is by now wellknown in academia, not many scholars have noted that the term 'intermediality' was also introduced in 1990 as part of the project 'Edge'90: Art & Life in the Nineties', A Biennale celebrated in the UK and curated by Bob La Frenais and Tracy Warr. Edge'90 included installations, performances, sculpture, a video library and a two-day conference. Inspired by Higgins' and George Maciunas' *Fluxus,* the exhibition aimed to express the collusion between intermedia artworks and the personal spaces associated with everyday life [4]. Thus, since its inception, intermediality has been contemplated as defining the relationships between art forms, technologies, and spaces of meaning. The combination of radio, static images (photography) and moving images (film) to conform cinematography and later broadcast television show that technical media follow a sort of cumulative process of engineering. Each new medium, communicates by transforming multimodal experiences, conforming a novel phase of practices as well as associated creative industries with their own mechanisms of circulation, discussion forums and distribution.

Communication and representation include the mobilization of different medial forms; from the vocal cords used in oral speech, the instruments used in painting and writing, to the more complex technical media employed in cinema, and, more recently, in computer-mediated communication and new media forms such as videogames. These mechanisms communicate by producing a sensuous embodiment of ideas or thoughts. Architecture, painting, printing, photography, film, all the arts exist in the combination of media, perceived through different perceptual modalities. The material medium, whether given in nature or artificially made, as well as the human agents and their perceptual/cognitive abilities, interact in ways that mobilize

#### *Literature Review on Intermedial Studies: From Analogue to Digital DOI: http://dx.doi.org/10.5772/intechopen.112611*

and organize ideas, giving them shape and inscribing them, that is, fixing them in some kind of material substratum. This organization of knowledge is also a social act, situated in a particular spatiotemporal context with particular characteristics that evolve over time.

This chapter offers a panoramic of some of the key concepts present in Intermedial Studies from the perspective of the various disciplines that have shaped the field in the last decades of the twentieth century. In the 1960s, studies were mostly grounded on linguistic aspects, following the tradition inaugurated in Europe by Ferdinand de Saussure. The impact of Harvard professor of pragmatism, logic and semiotics, Charles S. Peirce gave equal prominence to other non-linguistic signs (i.e. images, gestures, etc.), helping extend the debate on the multimodality of human communication. The 1990s saw the emergence of an increasing range of methods for intermedial analysis, including neohistorical analysis, iconography, semiotic analysis, ethnomethodology, poststructuralist, gender or postcolonial approaches. Another fundamental aspect is the relevance of narrative aspects in Intermedial Studies, and the similarities and differences between intermediality and transmediality, a key issue in contemporary gaming theory. The following lines mention some of the most prominent studies coming from several of these areas.

#### **2. The evolution of intermedial studies in the last decades of the twentieth century**

One of the first systematic and comprehensive accounts of the grammar of visual design was introduced by Gunther Kress and Theo Van Leeuwen in their 1996 volume, *Reading Images*. Dress and Van Leeuwen drew on a large amount of examples, from textbook illustrations, photo-journalism to fine art. They built their methodology on M. A. K. Halliday's meta functional theory [5]. Their subsequent research focused on a wide variety of formats, from texts to photographs, magazine pages and film. In Kress and Van Leewen *Multimodal Discourse: The Modes and Media of Contemporary Communication* they included TV programmes, films and the interaction of sound, music, language and images [6]. Meanings they argued were created through complex semiotic interactions which Kress expanded the debate about multimodality and intermediality to also cover a growing number of sign systems such as traffic signs [7].

In 1959, Roman Jakobson had published an article investigating intersemiotic translation and opening the path to the study of translation in intermedial environments. Jakobson differentiated between intralingual (rewording), interlingual (between two natural languages) and intersemiotic translation. This last one encompassed the interpretation of verbal signs by means of signs of nonverbal sign systems ([8], pp. 260–6). This research was expanded by Claus Clüver [9], Heinrich Plett [10], Umberto Eco [11], Henrik Gottlieb [12], Regina Schober [13], or Daniella Aguiar and João Queiroz [14], among many others. Theater and film adaptation, for example, have been considered intersemiotic translations. Other transformations, such as graphic novel into a film, a film into a theatrical performance or opera, opera into ballet, or a computer game into a film, might constitute a change of modality – a change of channel of perception in the recipients' perspective – but not a translation, a term with strong associations to the transfer between different verbal languages. Thus, the notion of media transformation would be more appropriate in these cases.

In the later 1980s and early 1990s, scholars like Ernest B. Gilman [15] questioned the centrality of language in giving voice to images and discussed the imperialism of language as central to inter-art comparisons, emphasizing the rivalry between the arts and disciplines. Grant F. Scott also saw early studies on ekphrasis (see López-Varela for more information [16]) as evidence of appropriation of the 'visual other' and as an attempt to 'transform and master the image by inscribing it' ([17], p. 303). Walter Ong [18] and Walter Mignolo [19] explored the evolution of cultural memory, from indigenous oral forms to written national and colonial histories. Unlike Ong, who claimed that oral cultures did not fade with the spread of writing and later the printed press, Mignolo saw the disappearance of oral cultures in relation to the role of written literacy in nation and empire building. In his studies on the Mesoamerindian oral legacy, Mignolo unveiled the ways in which indigenous oral forms were silenced and destroyed ([19], p. 58, 62).

Jerome McGann studied the changes in the form of mass production during the eighteenth and nineteenth centuries and the sociocultural tensions between oral and written forms [20]. W.J.T. Mitchell began by exploring William Blake's illuminated poetry, and moved on to research the ideological motivations behind the word/ image debate. For Mitchell, the divisions in art and literary genres may have fulfilled the function of segregating the arts in 'an imperialist design for absorption by the more dominant, expansive art' ([21], p. 107) . In his ground-breaking *Picture Theory: Essays on Verbal and Visual Representation,* Mitchell argues that although visual and verbal media are different at the level of sign-types, forms, materials of representation, and institutional traditions, in terms of 'expressing intentions and producing effects in a viewer/listener, there is no essential difference between texts and images' ([22], p. 160). In particular, he turns to *Laokoon* in order to show how Lessing's conception of the differences between painting and poetry is permeated by a rhetoric of gender; with images attached to female narrative and abstract symbols to superior male discourses, 'like the masses, the colonized, the powerless and voiceless everywhere' ([22], p. 157) .

#### **2.1 From symbols to icons**

Sensory modalities determine the way signs are decoded in intermedial configurations. Unlike Saussure, who gave prominence to the linguistic sign, Peirce noted the equal importance of three types of signs: indexes, icons and symbols. In Peirce's classification, the visual modality can distinguish between a gesture (an indexical or iconic sign, or both, depending on the type of gesture), an icon (signs that resemble the real such as maps, diagrams and images) and symbols (abstract and arbitrary signs in a community of users, for example, human discourse). Peirce's classification is based on how signs are related to their objects. An icon signifies its object by virtue of shared qualities, an index by virtue of a causal relation, and a symbol by virtue of an action ruled by a norm or habit, without resemblance or real connection to the denoted object but agreed by convention (for more information see López-Varela 2023) [23].

Indeed, only a historical perspective, as proposed by Jürgen E. Müller [24] can help unveil the changing sociocultural aspects that 'qualify' media. For instance, what Mitchell termed 'the pictorial turn' displaced logocentric criticism and established an iconic logic where the picture might speak to us; a time for the image to strike back, observes Bernd Stiegler in '"Iconic Turn" et réflexion sociétale' [25]. This approach also included a semiotic turn performed by scholars such as Mieke Bal [26] who paved the way for the shift from intersemiotic transposition to the notion of reciprocal

interaction between image and text, stressing the need to move beyond the theoretical deadlock and academic partitions which originated in the binary model of the sister-arts [16].

Following these trends, Birgit Neumann and Gabriele Rippl opened postcolonial studies to the discussion on visuality, the power of images and visual esthetics, examining writers such as Michael Ondaatje, Derek Walcott, Salman Rushdie, David Dabydeen and NoViolet Bulawayo [27]. The verbal-visual relations examined alternative modes of representation that reflect critically on hegemonic visual regimes.

This theoretical turn, which included aspects of cultural studies, visual ethnography as well as narrative theory, triggered the shift from intersemioticity to intermediality and the emergence of cross-boundary concepts, countering the logic of contemplating Intermedial Studies as a closed system. According to Rui Carvalho Homem and Maria de Fatima Lambert, 'the present currency of a relational nexus, as theoretically averse as it is to binary oppositions, entails a reading of the intermedial that underscores notions like contamination and hybridity' ([28], p. 11, 13). From painting to photography, from stage to the screen, modernity and avant-garde experiments have continued to inspire much of the research in the first decade of the twenty-first century. Martin Heusser ([29], pp. 221–236), for instance, has explored visuality in the relationship between text and image, noticing how different historical periods approached this relationship in different ways. Heusser believes that the context that led to the visual development of poetry at the turn of the twentieth century has to do with a turn from anabolic poetry (synthetic, complementary or constructive oriented) to catabolic poetry, where the visual component subverts the linguistic meaning. According to the author, these two tendencies seldom intersect, as they do in the modernist technique of montage. In the *avant-gardes*, catabolic poetry occurred far more frequently and, in its extreme forms (i.e. Lettrism, Dadaism), the resulting text was devoid partially or entirely of semantic content. Heusser connects this experimentation to a desire for a new metaphysical order and language during the inter-war period in Europe.

#### **2.2 Extending ekphrasis**

In 1992, Murray Krieger offered a rereading of the 'ekphrastic principle' in relation to twentieth-century experimentation. The author conceived it as 'a verbal description of something, almost anything, in life or art' ([30], p. 7), a device to 'interrupt the temporality of discourse, to freeze it during its indulgence in spatial exploration' (Ibid.) [30]. Krieger includes an analysis of poems that emulate the pictorial or sculptural arts by achieving a kind of spatiality [30]. Likewise, Wendy Steiner defines it as a description of a 'pregnant moment in painting,' that is, as an attempt to imitate the visual arts by describing a still moment and thereby halting time ([31], p. 41).

Attempts to redefine ekphrasis include James A. W. Heffernan's comprehensive historical approach in *Museum of Words: The Poetics of Ekphrasis from Homer to Ashbery* [32], or Claus Clüver's work, which emphasized that 'contemporary ekphrastic practices have subverted the traditional relation of the representational visual text to its verbal representation, even to the point of discontinuity' ([33], p. 30). With the onset of digitalization, the conceptualization of ekphrasis as a verbal representation of a visual representation has been increasingly perceived as too narrow (for the historical origins of this discussion, see López-Varela 2023 [16]). Margaret Persin expanded the range of ekphrastic studies by discussing 'uncanonical art forms such as television, photography, comics, and cinematography' ([34], p. 19). Other examples

of intermedial crossings between photography and literature appeared in Julia Breitbach's exploration of novels by Don DeLillo, Michael Ondaatje, and Ali Smith [35]. Jørgen Bruhn's *The Intermediality of Narrative Literature: Medialities Matter* (2016) is an excellent study of Anglophone texts ranging from Nabokov, Chandler and Tobias Wolff to Jennifer Egan that demonstrates that literary fiction often includes significant amounts of extra-literary material which expose not just ekphrastic exchanges but also intermediality at large [36].

Regarding ekphrastic encounters in the field of musicology, Siglind Bruhn [37] has explored synaesthesia in the intersections between music, words, and images. Bruhn also explains that very often, composers of instrumental music are inspired by other artworks and try to convey their essence, features and message in their own medium of music. In these cases, a given medium thematizes, evokes and sometimes imitates elements and structures of another medium in order to stretch semiotic levels to their limits, modify perception and conceptual imagery, and increase immersion and esthetic response. An example of these crossings is a novel structured as a fugue, as is the case of William H. Gass' *The Pedersen Kid* (1961) or the section entitled 'A Fugue', from his novel *The Tunnel* (1995). Both show how rhetorical and prosodic devices, such as rhythmical and sonorous patterns, enable the polyphonic techniques that musicalize these works and impact upon emotional aspects of reception [38]. Another example is the influence of John Cage's experimental music in Katherine Norman e-lit piece, *Windows* (2012) [39].

Werner Wolf's *The Musicalization of Fiction*, a title inspired by Aldous Huxley's homologous term from his 1928 novel *Point Counter Point,* defends that intermediality describes a transfer process of structure and esthetic intention from music to fiction. Wolf describes modern and postmodern esthetics in terms of harmony-driven structures, finding that *avant-garde* experimentation not always created meaning. The book contains excellent analyses of modern British experimental fiction modeled on music [40].

#### **2.3 The intertextual grounding of intermediality**

Werner Wolf's contribution went beyond the categorization of musical-textual relationships. He began to distinguish between 'overt' and 'covert' intermediality. In the first type, 'qualities are immediately discernible on its surface' ([40], p. 40). Wolf's categories referred to the distinction between combination and integration of media on the one hand and transfer and transformation of media on the other. In the case of media transformation, the German scholar spoke of 'implicit intermedial reference' (imitation), in which fiction mimics the musical form, usually as repetition or counterpoint, to thematization or 'intermedial reference' (texts that thematize, quote, or describe other media, like a novel devoted to the career of an artist). Music can be thematized through 'associative quotation' ([40], pp. 67–69). 'Intermedial transposition' are adaptations from one medium to another. For Wolf, the underlying connection seems to be the possibility of a narrative structure or 'transmedia', that is, phenomena that can be represented in more than one medium because of their narratological basis.

In an article from 2002, Wolf proposed another intermedial typology with a distinction between 'intracompositional' and 'extracompositional' [41]. According to Wolf, a phenomenon such as a novel transformed into an opera, conceptualized as 'intermedial transposition', should rather be considered as 'extracompositional intermediality', while the musicalization of fiction, labeled 'implicit intermedial reference'

#### *Literature Review on Intermedial Studies: From Analogue to Digital DOI: http://dx.doi.org/10.5772/intechopen.112611*

(imitation), should be considered 'intracompositional intermediality' ([41], pp. 27–29). Yet, both types of intermediality presuppose an awareness of pre-existing media products or media qualities (a certain narrative or musical characteristics) and hence point to something beyond the 'intracompositional', claims Elleström ([2], p. 14). Wolf's research was further expanded in *The Metareferential Turn in Contemporary Arts and Media* [42], with many of his works collected in a celebratory 2017 volume [43].

In some of his studies, Wolf uses the term intermediality to detect implicit and explicit discourses, similar to intertextuality, and this prompted a number of scholarly publications directed to this inquiry. This is the case of Jürgen E. Müller's work [44]. Peter Wagner distinguishes ekphrasis from iconotexts or 'the use of (by way of reference or allusion, in an explicit or implicit way) an image in a text or vice versa.' His definition of intermediality is still tied to pre-digital constrains. He considers that it is 'the "intertextual" use of a medium (painting) in another medium (prose fiction)' ([45], p. 17).

A similar exploration is offered in the collection coordinated by Valerie Robillard and Els Jongeneel *In Pursuit of Ekphrasis: An Intertextual Approach* (1998). Claus Clüver, who participates in this volume, explains that ekphrasis retains a certain degree of Aristotelian *energeia* and defines it as 'the verbalization of real or fictitious texts composed in a non-verbal sign system' ([33], p. 49).

These approaches maintained a certain emphasis on the contents of the information transferred, rather than on its form and material aspects. Some of the ideas that contributed to enriching the discussion on the materiality of intermediality came from film studies but, mostly, with the onset of widespread digitalization in the 1990s, from hypertext and hypermedia theorization, as well as from the field of semiotics. Bernhard F. Scholz also recognized that the concept is a 'complex multidimensional multi-faceted semiotic phenomenon' ([46], p. 75) with ties with intertextual relations ([46], p. 74).

Following Wolf, Irina Rajewsky proposed several categories outlined in her book-length study *Intermedialität* and in subsequent articles. 'Media combination' (Medienkombination), where it is an inherent quality of a certain genre of media, like theater or film, exemplified with opera, comics, etc. 'Medial transposition' (Medienwechsel) or transformation of a given media product, where intermediality manifests itself in how a media product emerges, as in the case of adaptation. 'Intermedial references' (intermediale Bezüge) evoke or imitate entities of a medium in a different medium, e.g. painting or cinema in literature, exemplified with phenomena such as ekphrasis, narrativization of music and references to painting in film ([47], pp. 15–17, [48], pp. 51–53). Thus, Rajewsky's categories run from mere contiguity of two or more material manifestations of different media to genuine integration.

These categories are not closed structures, so film adaptation can be classified as a media combination (of theater and photography) and medial transposition (of a literary text). It is also interesting that she also considers contextual aspects such as the production and specificity of material media patterns, which have changed over time, and she goes on to relate their 'intermedial' qualities to their use and reception. In the case of film adaptation, Rajewsky explains that the viewer receives the original literary text, not as something on which the film production is based, but as another nucleus that produces crossed relations on the horizontal (not vertical/ hierarchical) intermedial level. Rajewsky's approach has the advantage of theoretically distinguishing between intramedial (and thus intertextual) and intermedial references. Within the first, a medium evokes and generates an illusion of another

medium's specific practices, as mentioned above. Intermedial references, however, constitute themselves in various complex combinatory ways in relation to another medium (monomediality) or several media (plurimediality), as in the case of dance theater. In these cases, a given medium thematizes, evokes and sometimes imitates elements and structures of another medium in order to stretch semiotic levels to their limits, modify perception and conceptual imagery, and increase immersion and esthetic response.

#### **2.4 Intermediality, transmediality and the impact of narratological studies**

Elleström's summarizes the types of media transformation in two: 'transmediation' and 'media representation' [2]. They resemble Rajewsky's distinction between 'medial transposition' and 'intermedial references' only at a superficial level. For Elleström, 'transmediation' is 'transfer of media characteristics' ([49], p. 5), 'a central part of intermediality, which is an even broader concept based on the proposition that different media types are interrelated in all kinds of ways' [49]. Transmediation includes previous categories of 'implicit intermedial reference' (imitation) and 'intermedial transposition'. Elleström explains that some perceived media characteristics of the target medium are, in important ways, the same as those of the source medium, 'which is to say that the media characteristics of an initial medium are perceived to be represented again by another kind of medium ([2], pp. 20–27). Thus, 'transmedial narration' 'should be understood to refer to all varieties of transmediality and transmediation where narration is a media characteristic that is significant enough to be observed' ([49], p. 6).

The second category formulated by Elleström, 'media representation', resembles Wolf's 'explicit intermedial reference' (thematization) ([41], pp. 27–29). However, Elleström's notion of 'representation' and 'mediation' are different. He describes 'mediation' as a pre-semiotic phenomenon based on the material properties of objects in general and technical media in particular, perceived through the senses in a particular spatiotemporal configuration. Many analogue processes are sequential, including human language; sequential sign systems can easily turn into narrative forms, whether in oral or written discourse, in melodic music, and even in the case of iconic signs, as in motion pictures. These properties are potentially meaningful when sensory configurations are activated by cognitive import, thus becoming meaningful. This is what Elleström describes as the semiotic phenomenon of 'representation'. Any object or media, for example, the page of a printed book, stimulates the perception of a sensory configuration through the visual mode. In Braille, the activation would take place through the tactile modality. The page can contain a literary piece, a musical score, or a printed image or diagram. The musical score or the poem previously seen on the page can be then transmediated by another modality or sensory configuration, in this case, the voice as the piece is read or sung, as well as mediated by another kind of technical medium ([2], p. 19). Elleström mentions here the differences between the book page emitting photons or sound waves generated by vocal cords, explaining that he does not conceive the 'technical medium' necessarily as a machine or a device for production or storage but rather as a form of distribution that disseminates sensory configurations. In this sense, for Elleström, transmediation is not only remediation in another technical medium (in Bolter and Grusin's 1999 sense) [50] but also includes the possibility of multimodal translation of content from one sign system into another, triggering different sensory configurations. The second category theorized by Elleström, 'media representation', involves the notion of one medium representing

#### *Literature Review on Intermedial Studies: From Analogue to Digital DOI: http://dx.doi.org/10.5772/intechopen.112611*

another medium. This case involves transmediation as well as repeated mediation of equivalent sensory configurations.

While Elleströms' inquiry comes from the field of Communication Studies and Semiotics, Marie-Laure Ryan, and Marina Grishakova have explored intermediality from the perspective of narratology. Drawing on Dan Sperber's notion of 'metarepresentation', in the volume *Intermediality and Storytelling* (2010) coordinated with Ryan, Grishakova expands her differentiation between two forms of intermedial metarepresentations; metaverbal (an attribute of verbal texts that evoke images) and metavisual (an attribute of images that reflect on the incomplete nature of visual representation). Grishakova explains that 'the metaverbal text (e.g. an ekphrastic text, cine-novel, or graphic poetry) reflects on the incomplete nature of the verbal medium by probing the limits of verbal representation and appealing to the visual forms (graphic elements, real or virtual film shots, works of art, dreams, hallucinations, mental imagery, etc.).' On the other hand, 'the metavisual text reflects on the incomplete nature of visual representation by juxtaposing image with verbal message and revealing their discrepancy' ([51], pp. 313–315).

In *Picture Theory,* Mitchell had already noted that he sought 'to experiment with the notion that pictures might be capable of reflection on themselves, capable of providing a second order discourse that tells us—or at least shows us—something about pictures' ([22], p. 38). Marie-Laure Ryan's introductory piece expands this research and by, comparing images and texts, she concludes that images tent to be decoded as less fictive. Some years before, in the volume *Intermediality and Performance,* edited by Freda Chapple and Chiel Kattenbelt [52], Peter M. Boenisch had also tackled a similar issue. For him, mediatized representation always takes the form of a second-order reality, 'whether as idealist representation of a crude reality, in terms of the beautiful and the sublime, as a realist rendering of the actual thing, or as a surreal invention in Science Fiction.' Boenisch affirms that in all cases, the experience is 'present' to audience, and 'whether factual or fictional, would be felt in the same way because it is authentic in terms of its spatiotemporal effects.' ([53], p. 110). Indeed, contributions from semiotics show that iconic signs (images, graphs, and diagrams) acquire primacy because they require lower levels of decoding ([54], pp. 204–5). Discourse typically has a lower sense modality than photography, television or film, judged to be more realistic because of their higher image content.

Jan Baetens' important work on sequential images and word/image combinations has focused on a range of narrative types, from roman-dessiné and comics to film photo-novels [55, 56]. His research has contributed interesting insights into transmedial thematology. Sequential montage of images organizes as a series drives a certain visual movement, a sort of chain of sign patterns that can be interpreted as a narrative framework. Film photo-novels have a hybrid format where pictures with captions and dialog balloons are used to re-create a cinematic story. These lowbrow film novelizations appeared in weekly and monthly magazines and had a significant social impact, particularly in Latin America and the United States in the first half of the twentieth century. The history of graphic narrative and its remediations to various formats has also contributed to the study of transmedial narratologies (see, for instance, [57]; for comics, [58, 59]).

#### **2.5 Intermediality, the performing arts, cinema, television, and beyond**

A number of studies on intermedial exchanges fall under the scope of theater and drama. Before the invention of cinematography, drama and dance performance where possibly the environments with greater intermedial scope. Technical skills in drama production involve a wide range of knowledge, encompassing design and

architecture, drawings, sketchbooks and photography [60]; illustration of stage instructions; text-based instructions such as notes, scripts [61, 62], prompt and cue lists, and so on [61, 62]. Edited by Michael Bull, *The Routledge Companion to Sound Studies* (2018) is an extensive volume presenting the workings of sound in culture and the role that sound media play in the construction of experience [63]. Samples of textiles, furniture and all kind of model artifacts are also used in performance [64]. While puppets were very popular in the past, today, electric shadows as well as video might be projected including digital technologies [65–67]. All these aspects enhance intermedial combinations, explored in a wide range of works like those by Chapple and Kattenbelt [52], Brown, Hauck and Larrue [68], Sager [69], Drumm [70], and Bay-Cheng, Kattenbelt and Lavender [71], among many others.

On the topic of intermedial adaptation from theater to television and cinema, we can mention Mancewicz [72], and Wyver [73]. The presence of cinema upon Intermedial Studies owes much to the collection of books by Edinburgh University Press entitled 'Edinburgh Studies in Film and Intermediality.' In the last 10 years (2012–2022), the collection has published 19 volumes that approach the topic from a wide range of angles; for example, conversations between filmic techniques and literature, photography, theater, painting, architecture or videogames (see for instance, Ingham [74]. Susanne Foellmer, Maria Katharina Schmidt and Cornelia Schmitz [75] deal with dance and performance art as well as opera to explore the transitions among media as well as the dynamics of transfer between the performing and visual arts. Miriam de Paiva Vieira tackled topics spanning narrativity, various modes of literary expressions, intersemiotic translation and multimodal communication [76] (see also [77, 78]).

Research on the intermedial aspects of cinematography is extensive. Jeffrey Geiger's collection, for instance, examines the relations of cinema to literature, photography and other modes of media, from the magic lantern, the zoetrope, or the flick-book, to the iPhone and the computer [79]. Other works on cinema and Intermedial Studies include those by Mariniello [80], Gaudreault, Russell and Véronneau [81], Pisano [82], Nasta and Huvelle [83], Altman [84], Ágnes Pethö [85, 86], De Giusti [87], Gaudreault and Barnard [88], Vieira and Rio Novo [89], Giuliani and Negri [90].

Examples of historical media conjunctions are the combination of radio, static images (photography) and moving images (film) into cinematography and later into broadcast media like television, with the recent rebirth of the serial formats that has accompanied the emergence of subscription streaming services. The 1970s and 1980s were the era of national television. TV content varied considerably from nation to nation, broadcasted in analogue devices with normally one TV screen per household. After the 1950s, magnetic tape video recording became a major contributor to the television industry, via the first commercialized video tape recorders, with two standards, VHS (the Netherlands) and Betamax (Japan), which entered the homes in the 1970s. They joined forces in developing a new digital format for audio recordings, later known as compact disc (CD), which replaced magnetic tapes in the 1990s [91].

The success of TV was not only due to the number of users but in how its content was present in other media, like newspapers, magazines, advertisements, radio etc. ([92], p. 118). John Ellis also demonstrated that television was constructed by means of interrelationships between other audiovisual media, mainly radio, cinema and video recording [93]. Thus, while television was created by previous media, which were adapted to the new forms, it is also important to consider their relative power, for instance, the relationship between radio and television, as well as the significance

#### *Literature Review on Intermedial Studies: From Analogue to Digital DOI: http://dx.doi.org/10.5772/intechopen.112611*

of other media that contributed to circulate information from the TV screen to other media industries, as the tabloid press did. This bond between media industries continues today in Internet platforms that offer background information to the news, for example, or the lives of the performers who appear on television. This cooperation also existed between TV music channels, like MTV. This cable channel was launched in 1981 to promote music videos and interviews with artists. It became a huge marketing machine able to operate in conjunction between the music industry, television and other forms of promotion such as concerts, creating a media event able to distribute its brand into other marketable products (T-shirts, souvenirs, etc.), distributing to consumers via different media at different times. Although different from the transmedia phenomenon, in which a story unfolds across multiple media platforms, with each new text making a distinctive contribution to the whole, 'transmedia storytelling' [94] could even be seen as a result of a brand narrative expressing itself in different media and business areas [94].

The collection edited by Paul N. Reinsch and Laurel Westrup offers a fresh approach to the albums that accompanied the promotion of films, television shows, or video games and the audiovisual relationships that drive the encounters between music and media studies [95]. Louise Harris interrogates the relationship between audiovisual work in diverse media, including for instance audiovisual sculpture [96]. Likewise, for years music videos have helped promote artists and circulate their works widely. With the proliferation of digital technologies, the advent of YouTube in 2005 and social networking sites, music videos visualize the music industry. The collective volume by Lori A. Burns and Stan Hawkins includes a range of essays that explore issues related to the sociocultural impact of music videos as well as their intermedial implications [97].

The 1990s were a turning point for wireless telecommunication and, as mention, artistic trends, like Fluxus video art, took new directions with digitalization. Various technical media saw innovations. For instance, a textual data extension called 'teletext' was added to analogue television, and digital stereo broadcasting systems were also introduced. A full convergence was achieved before the twenty-first century when MIT introduced a computer-compatible digital high-definition television and video (HDTV) system with progressive image scanning that revolutionized the future of television. Digitalization also accelerated changes towards streaming of video via broadband Internet. Nowadays, using the Internet to watch television is common, although young users prefer personalized experiences on Youtube or following the channels of their favorite influencers. The evolution of television captures an ongoing negotiation between emerging technical media, future expectations, past experiences and old content, out of which new things emerge as in flux.

Intermedial Studies with a focus on sound have also been explored from the point of view of the cultural. *Mediations: Body Sound Technology* (2008)*,* edited by Carolyn Birdsall and Anthony Enns, recognizes sound as 'a co-participating agent in cultural practices and performance' ([98], p. 3). Each chapter of the book offers a case study coming from diverse disciplinary backgrounds, such as musicology, film and media studies, art history, comparative literature, philosophy, performance, and Intermedial Studies [98]. Karin Bijsterveld's [99] collection studies the ways in which 'staged sounds' might present the changing identities of cities, like Amsterdam, Berlin and London. Their soundscapes, as recorded in historical documents, radio plays and films, help bring back a rich cultural heritage. Similarly, to ponder the reciprocity between intermedial relations, creativity, different sensory systems and memory [100]. Sarah Durcan's [101] *Memory and Intermediality in Artists' Moving* 

*Image,* contributes to intermedial and memory studies by examining the mid-1990s transitional moment that coincided with the impact of digitalization upon video art and experimental film [49, 99]. These works highlight the ways in which the political and the sensory are linked directly to modalities of representation belonging to a given cultural moment. In this regard, it is important to mention the work developed at CRIalt, heir to the Centre for Research on Intermediality (CRI), established in 1997 in Montreal, Canada. The centre starts from the principle that we cannot study each media in its autonomy because all media interact and define each other in their interactions [101].

As I have pointed out, it is important to contemplate intermedial relations and their semiotic, historical, sociological, cultural and political implications. In this regard, research by Asun López-Varela mainly focuses on semiotic and cultural aspects [102, 103], including the recently published *Palgrave Handbook on Intermediality* (2023) [23].

#### **3. Intermediality and the digital turn**

As seen, the evolution of Intermedial Studies owes much to the peculiarities of the medium, and digitalization has further complicated this interdisciplinary field. In analogue media, the concept of intermediality means that separate elements, taken from different technical media and that have evolved historically (i.e. static imagesphotography turning into moving images-film), are interrelated in ways that their heterogeneous forms of expression come together and produce one integrated effect upon reception. Thus, after the onset of digitalization, intermedial inquiries were initially directed towards the study of similarities and differences between analogic domains, conceptually expressed by means of metaphors and metonymies among sister-arts; that is, in terms of physical phenomenology, like the human analogic perceptual apparatus.

In digital environments, different technical media are integrated within a complex technical system where their distinctions are blurred. Interrelationships are no longer grounded on phenomenological transformation. Unlike their mechanical analogue counterparts, digital machines include layers of mathematical code in combination with human languages. Computing processes simulate, that is, create a physical resemblance on the superficial layers and on the so-called 'interface'. The differences and technical requirements of previous analogue media are dissolved in the inner workings of computation. Experts have spoken of media convergence. This convergence involves not only technical media, but also their associated industries, services, and work practices [23, 94, 104]. This raises the following question: in the digital scenario where the transformations of analogue technical media are dissolved, can we speak of intermediality?

Our answer is a resounding yes. We can still speak of intermediality because the technical medium is just one of the cluster of aspects involved in the multi-layered dimensions of intermediality. In the case of digital media, we may speak of complex forms of 'media representation' ([2], p. 19) , at least on the superficial interface level. In other words, the digital machine simulates exactly those transformation processes that occurred in analogue media, activating equivalent sensorial configurations. Thus, in transmediation, in the sense used by Elleström, we can talk of a simulation of transformative processes which include the repeated mediation of equivalent sensory configurations by another kind of technical medium [49, 105–107].

#### **3.1 From hypertext to cyberspace**

Since the origins of hypertext in the 1940s [108] and the development of Tim Berners-Lee's World Wide Web, digital advance has multiplied. The debate on what a medium is and how it constructs culture has often followed from Marshall McLuhan's pioneer work [107]. In the 1990s, the field of electronic literature began to grow. The first pieces of e-lit included only hypertext and could be read along diverse modular paths, like Borges' short story 'The Garden of Forking Paths' (1941). Gradually, text-based e-lit gave way to digital art and Net. Art, which began to include a growing number of images, sound, video to be manipulated on the screen (see the collections of the Electronic Literature Organization ELO). As in cinema and photomontage, this accumulation blurred, even more, the linearity that conforms to traditional reading patterns, accumulating sign systems into a sort of spatial density. The ordering principles of the logic of print formats and analogue technical media were broken. Everything on the screen could be manipulated, from the words of a text to the images that accompany it. Within years, E-lit and Net. Art pieces included even more links to external 'social' websites that allowed commentary from the readers/users. The term 'social media' refers to technologies, platforms, and services that enable individuals to engage in communication from one-to-one, one-to-many, and many-to-many. This was possible through the development of Web 2.0, which in the first years of the twenty-first century aimed to be user-focused.

Hypertext continued to evolve into a very intricate network. The very essence of the World Wide Web challenges linearity. Undoubtedly, although there is a legacy of the analogue world in the digital, many things have changed. For instance, in analogue electronic media, signals constituted a linear continuity. In digital machines, signals are not continuous but compressed into discrete data units known as digits (1 s and 0 s at the lowest level of machine language, representing voltage bands). These correspond to the two values of 'true' or 'affirmative' and 'false' or 'negative' in binary logic and in the Boolean algebraic domain.

In her description of computer signs as 'flickering signifiers', Katherine Hayles draws a similarity with the film [109]. In analogue projectors, film advanced frame by frame, each frame standing still in front of the lens for a portion of the frame cycle (typically 24 cycles per second) and rapidly accelerating, flickering, advancing, and decelerating again. This intermittent motion is fused by means of a device known as the Geneva drive, first implemented in the world-famous Swiss mechanical watches. It is also known as the 'Maltese cross mechanism' due to its visual resemblance when the rotating wheel has four spokes (a spoke is one of some number of rods radiating from the centre of a wheel). With this mechanism, flickering is no longer perceived by the human eye, and the discontinuous images are seen as fused. The 'flickering signifiers' of digital media also mean that while the analogue figurative is always affirmative and true, resembling the real, the digital allows for a reversible switching mechanism that includes the false and the negative (1 s as well as 0 s; see also [110]). Lev Manovich has stressed that machine languages and the logic of the computer could be expected 'to significantly influence the cultural logic of media' ([111], p. 46).

One of the most important characteristics of digital machines is that they have screens that function as interfaces with the analogic world of human perceptual experiences. In this sense, they resemble the 'cinema effect', as Sean Cubitt put it in his 2004 homonymous volume [112]. Screens are optical media that, like mirrors, allow

a very close similarity with the real. Cinematic representation configured threedimensional space as well as the various elements that constitute sensual phenomenological experience –light, shape, and texture –what Cubitt termed 'spatial effects.' Digital media turns represented space into a simulation. It is a 'heterogeneous space' that features windows not as windows onto the world but as windows that 'open onto other representations or other media' ([113], p. 329) [50]. In doing so, it provides a mechanism to transform analogue information into a schema or map, drawn in binary code at its lowest level; this is almost like the neuroplasticity that the brain performs when filling in the gaps in mental images.

The illusion of transparency, a feature of all screens, including those of digital media, blinds viewers to the part played by the internal aspects of the technical medium in constructing experiential worlds. In digital media, sense modalities are increasingly combined to produce the greatest effect of authenticity (hyper-real as coined by Bolter and Grusin [113], p. 24) [50], beyond the mere similarity of the mirror. Indeed, 'the logic of immediacy leads one to erase or automatize the act of representation,' for it 'acknowledges multiple acts of representation and makes them visible' (1999, p. 329) [50].

#### **3.2 Navigating the digital**

George Landow's first exploration of hypertext noted the modularity and nonhierarchical status of sign units, whether text, image, video, or sound, and explained that it yielded a completely different reading experience [114]. Within texts, signwords acquire meaning in relation to the entire chain in which they occupy a particular position –subject, verb, object, and so on. In digital environments, distributed on the screen space, sign-positions, ruled by mathematical coding, also become signs, with links anchored to texts, images, parts of images, video, or any other webpage. Thus, digital screens do not provide a specific reading direction. They comprise all directions. In conventional narrative structures, which follow temporal paths with a beginning, a middle, and an end, the readers' capacity to envision the end determined meaning. The reading experienced followed a dynamics based on suspenseful prospection, curiosity-driven retrospection and surprise-generated recognition. Any gaps lacking information were filled by inferences. In intermedial multimodal configurations, there are more paths, and the heterogeneity of signs, ruled by the underlying code, problematizes the making of inferences [115].

In trying to explain the reading process in cybertext, a term that in the late 1990s began to substitute hypertext, Espen Aarseth used the word 'ergodic', from ancient Greek 'ergon' (a unit of work or energy) and 'hodos' (threshold as well as path) [116]. Aarseth's concept signalled the spatiotemporal fusion that takes place in the digital technical medium. The prefix 'cyber' used 'cyberspace', came from in 'cybernetics', in ancient Greek 'kubernētēs' meaning pilot or steersman. All these terms came to emphasize the navigational skills required in online omnidirectional environments. The temporal features present in print reading environments are reshaped from unidirectional patterns into multidirectional ones. One of the first examples of multilinear storyworlds [117] is Michael Joyce's *Afternoon, a story* published by Eastgate Systems in 1990. The story includes various paths that the reader/user can navigate by clicking the hypertextual links. The order of the events in the story is not fixed. They appear in the order that the reader activates when clicking on them. In 1992, Stuart Moulthrop created *Victory gardens*, also published by Eastgate Systems. Upon entering the work, the reader is presented with a series of choices as to how to

#### *Literature Review on Intermedial Studies: From Analogue to Digital DOI: http://dx.doi.org/10.5772/intechopen.112611*

navigate the story. The reader may enter the text through a variety of means: the map of the garden, the lists of paths, etc. With no end to the story, the multiple node-links provide a certain sense of closure.

Mark Bernstein provided a summary of the different patterns that links could take, thus influencing the online reading experience: 'cycle, counterpoint, mirrorworld, tangle, sieve, montage, split/join, neighborhood, missing link and feint' [118]. Some of these hypertextual structures, he argued, interweave different narratological voices (in Bakhtin's sense) of equal (or nearly equal) weight within a single exposition (see, for instance, counterpoint). The 'mirror-world' establishes a second voice that separately parallels (or parodies) the main statement. The 'sieve' and the 'split/join' explicitly guide a user's path and may allow users to experience different episodes or points of view. The 'split/join', however, might be misleading in the diversity of viewpoints represented because no matter which path a user takes through the landscape, the exit is always the same. Bernstein suggests that each structure suits different rhetorical purposes, so argumentation, for example, utilizes a 'cycle' by repeating points or modifying them, and a 'tangle' might disorient users in order to make them more receptive to a new argument or an unexpected conclusion.

Leonardo Flores [119] studied electronic poetry (e-poetry) as a model of digital textuality in order to explain how mathematical code languages enabled different forms of engagement on the part of readers/users. For instance, some software features allow the continuation of different statements of text, images, and other types of multimodal content [119]. Recursion or repeated execution of patterns creates loops and subroutines. As systems of layers, each of them programmed in different mathematical languages, digital environments allow great mobility of units. They can be assembled and re-assembled into larger-scale objects and continue to maintain their separate identities ([111], p. 30). Manovich indicates that the major difference between old and new media is the fact that old media involves a manual assembly of visual/verbal elements into a composition or sequence, whereas digital media generate many different versions and variations accomplished by automation and random conditions ([111], p. 36).

Along these lines, Jay David Bolter and Richard Grusin's [113] *Remediation*, brought forth the idea that a medium appropriates the techniques, forms, and social significance of previous media and attempts to rival or refashion them. The authors discussed how new media develop in relation to medium-specific modes of previous media, designed to improve previous shortcomings. Thus, remediation in Bolter and Grusin refers to the historical interplay of older and newer media, paradoxically trying to erase the old and multiply the new media ([50], p. 5). Unlike Bolter and Gusin, Elleström conceives 'remediation' as repeated mediation only concerning the physical realization of entities available in perception. Online digital formats show the dominance of certain sensorial modes. In printed texts, the gaze moves the narrative forward until the reader turns the page. In digital multimodal formats, there is a complex intermediality that synaesthetically engages touch through the movements of the cursor enabled by keyboard or mouse, eye-tracking that follows the multimodal cues provided by links and which access various types of content –whether text, image etc. This complexity was explored in a paper co-authored with e-lit artist Serge Bouchardon [120]. Other synesthetic aspects in multimodal cueing in virtual environments have also been explored to show how changing technological patterns introduce ontological and cultural implications into the discussion on intermediality, a field that continues to expand [121].

#### **3.3 Visual narratology**

As mentioned, one of the fundamental issues raised in Intermedial Studies after digital media convergence is related to narratology. Jens Schröter's research on visual narratology as tertium comparationis between different artistic expressions and transformations from analogue to digital formats was expanded in the 2008 volume he edited with Joaquim Paech [122]. In his 2011 article 'Discourses and Models of Intermediality,' Schröter identifies four models of discourse. Synthetic intermediality is a fusion of different media. Formal (or transmedial) intermediality is a concept based on formal structures not specific to one medium but found in different media. Transformational intermediality is a model centred around the representation of one medium through another medium, leading to the postulate that transformational intermediality is located in the processes of representation. Finally, ontological intermediality is a model suggesting that media always exist in relation to other media. Schröter mentions a fifth model of virtual intermediality that he expands on in his 2014 handbook, published in German. In agreement with his category of ontological intermediality, a medium defines its own ontology by relating itself to another medium; thus, it seems, just as with humans and their others, that it is not possible to define the specificity of a medium in isolation except through a dialogical encounter with another medium [123].

The first decade of the twenty-first century saw the publication of a large number of studies on intermediality. For instance, the edited volume by Herkman, Hujanen, and Oinonen [124] places intermediality as a theory as well as a methodological approach. Some articles provide a critical reflection on media, while others deal with intermediality, the analysis of representative cases, and media institutions and professions [124]. Also, in 2012, research by Bernd Herzogenrath discusses digital photography, comics and graphic novels, performance art, techno, hypertext, and video games. The volume also explores how intermedial interactions shape techniques of representation, strategies of communication, and forms of reception [125].

Among the histories of digital media, Gabriele Balbi and Paolo Magaudda examine the interactions between political, technical and sociocultural elements [126]. Alongside Baetens' work, already mentioned, Brianne Cohen and Alexander Streitberger's [127] provide an exploration of the 'photofilmic' looking at the different ways in which art, cinema, and other forms of visual culture respond to a digitized and networked world. The essays in this collection deal with various practices, ranging from panoramas, drone surveillance, tableau-vivant, press coverage, and computer-based editing [127].

#### **3.4 The contribution of non-Western studies to Intermediality**

The field of Intermedial Studies is in continuous expansion. This paper has not been able to cover research from non-Western sources. However, we can briefly mention Yvonne Spielmann who, during her visits to Japan between 2005 and 2009, discovered the hybridity of Japan's media culture, a blend of digital-analogue connections, Western and non-Western media esthetics. From Japanese fondness for precision and functionality to the poetics of unobtrusiveness and detail, Spielmann examines pioneering mixed media artists like Masaki Fujihata, Seiko Mikami, Sota Ichikawa, Toshio Iwai, or Tatsuo Miyajima [128].

The monograph by Jihoon Kim also considers experimental film and video, essay film, gallery-based installation art, and digital art and explores technical issues such

#### *Literature Review on Intermedial Studies: From Analogue to Digital DOI: http://dx.doi.org/10.5772/intechopen.112611*

as stillness, movement and indexicality as well as forms of storage and archiving under the influence of digitalization. The author explores over 30 artists and filmmakers, including Jim Campbell, Bill Viola, Sam Taylor-Johnson, David Claerbout, Fiona Tan, Takeshi Murata, Jennifer West, Ken Jacobs, Christoph Girardet and Matthias Müller, Hito Steyerl, Lynne Sachs, Harun Farocki, Doug Aitken, Douglas Gordon, Stan Douglas, Candice Breitz, among others [129].

#### **3.5 Postdigital performance and installation art**

Catherine Elwes [130] also focuses on film and televisual installations through architecture, examining galleries and museums, as well as painting, sculpture, performance, expanded cinema, and countercultural film and video from the 1960s, 1970s, and 1980s. Elwes, herself is critic, curator, and practitioner in British feminist art politics, examines political and ideological positions in the works of practitioners such as Anthony McCall, Gillian Wearing, David Hall, Janet Cardiff, Peter Gidal, and Maya Deren [130]. Sally Johnson and Astrid Ensslin's collective volume tackles issues of identity, gender, youth, citizenship, politics and ideology from a sociolinguistic perspective, exploring communication across a range of mediums including television, radio, newspapers, magazines and the Internet [131].

Gabrielle Jennings, Kate Mondloch *Abstract Video*, also explores video experimentation but also Net. Art, installation, new media, expanded cinema, visual music, and experimental film, including celebrated artists such as Jeremy Blake, Mona Hatoum, Pierre Huyghe, Ryoji Ikeda, Takeshi Murata, Diana Thater, and Jennifer West, alongside emerging artists [132]. Claudia Benthien, Jordis Lau and Maraike M. Marxsen [133] explore the literariness of media art starting from the framework of Russian Formalism, where the term 'literariness' was coined. The work of renowned artists such as Chantal Akerman, Mona Hatoum, Gary Hill, Jenny Holzer, William Kentridge, Nalini Malani, Bruce Nauman, Martha Rosler, and Lawrence Weiner is explored in this volume that analyses experimental film, video performance, moving image installations, and other media-based art forms from the perspective of intermediality, remediation, and postdrama [133].

*Digital Theater* (2020) by Nadja Masura examines de interaction of digital technology and performing bodies exploring live theater performances which incorporate video projection, animation, motion capture, robotics and Virtual Reality (VR). The volume includes practitioners like George Coates, the Gertrude Stein Repertory Theater, Troika Ranch, David Saltz, Mark Reaney, The Builder's Association, and ArtGrid [134]. Luis Campos & Fiona Jane Schopf [135] explore the topic of intermedial performance of electronically controlled sounds, embodiment and sound on the stage, and musical speech. Among the works explored, Iannis Xenakis' reworking of ancient Greek in Oresteia; trans-genre adaptation in Strauss's Der Rosenkavalier or Philip Glass's Cocteau trilogy, musical comedy in Ligeti's Le Grand Macabre [135]. *Body Knowledge: Performance, Intermediality, and American Entertainment at the Turn of the Twentieth Century* (2013) by Mary Simonson is a comprehensive monograph on the part played by female performers in North-American onstage and on-screen. The volume explores the ways in which intermedial experiences in early twentiethcentury art are the product of a dialog between technological developments and new modes of perception, involving novel understandings of the body and the self, and shifting conceptions of gender, race, and sexuality [136].

*Activism and Postdigital Performance* by Liam Jarvis and Karen Savage [137] traces how identity can be created, developed, hijacked, manipulated, sabotaged and

explored through performance in postdigital cultures. Considering how technology is reshaping performance, this collection of essays examines the artist-activist as a producer of avatars engaging in performance practices that problematize identity politics. Using a range of examples in theater, film and Internet-based performance practices, the essays examine mediatized digital cultures, expanding the notion of intermediality. Case studies include James Cameron's Avatar, Blast Theory's Karen, Ontroerend Goed's A Game of You, Randy Rainbow's online videos, Sisters Grimm's Calpurnia Descending, Dead Centre's Lippy and Chekhov's First Play, and Jo Scott's practice-as-research [137]. Theater maker and senior lecturer Liam Jarvis had previously explored the field of 'virtual embodiment' in performance, installations, theater/art-making, arts-science and VR, applied to a wide range of contexts where the human body becomes part of computer-generated experiences. His 2019 volume on immersive embodiment included an interesting discussion on the complex experiences in healthcare, for instance [138].

#### **3.6 Transmediality and storyworlds**

Much contemporary e-literature and e-poetry play with the graphic patterns of letters, words, or symbols rather than by the meaning of words in conventional narrative arrangement, like Concrete Poetry did in the twentieth-century avant-gardes.

Digitalization has highlighted the debate between intermediality and transmediality, and its storytelling foundations. Transmedia storytelling has become an influential mechanism in the field of media and entertainment, enabling the transformation of books into films, TV series, videogames, advertising, etc., telling stories across multiple platforms. One of the first scholars to theorize the field of transmedial narratology was Marie-Laure Ryan. In 2014, she published *Storyworlds across Media* together with Jan-Noël Thon [116]. The volume explores how media, old and new, give birth to various types of storyworlds and provide different ways of experiencing them. The volume critically assesses the cross- and transmedial validity of narratological concepts such as storyworld, narrator, representation of subjectivity, and fictionality. Drawing on her previous work, Ryan notes that narratives exist simultaneously as representations or mirrors of another world and as a cognitive mode or structure for understanding out world. As a cognitive mode, it relies on the technical medium that displays it. Thus, print narratives are different from film narratives and from computer worlds, determined not by the text itself but rather by the experience of the text; its manipulation, in the case of digital storyworlds. Ryan claims that 'the convergence of media around a common center that we may call "narrativity"' is 'a center that is itself organized around a storyworld' ([117], p. 3). Thus, the book explores media convergence and transmedial storyworlds by examining forms of storytelling based on multiple media platforms. This book offers a valuable contribution to the literature in the areas of transmedia storytelling, narratology, digital fiction, electronic literature, locative storytelling, performative writing, digital culture studies and human geography.

The collection 'Routledge Advances in Transmedia Studies' includes a number of volumes that explore the concept of transmediality mainly from the perspective of communication studies and journalism. For example, the volume by Renira Rampazzo Gambarato, Geane Carvalho Alzamora, Lorena Tárcia [139] explores transmedia dynamics in various facets of fiction and nonfiction transmedia studies. Moving beyond the presentation/definition of transmediality as a field of study, the

#### *Literature Review on Intermedial Studies: From Analogue to Digital DOI: http://dx.doi.org/10.5772/intechopen.112611*

authors examine novel advancements in the theory, methodological development, and strategic planning of transmedia storytelling [139].

From a sociocultural perspective, Christina Meyer and Monika Pietrzak-Franger [140] examine early transmedia practices in British Victorian media, including books, theater, advertising, newspapers and games, thus tracing a historiographical account of transmedia narratology. The volume also explores nineteenth-century forms of audience participation and the role that social agents –authors, publishing houses, theater producers, games manufacturers, as well as audiences – played in the production, distribution, and consumption of Victorian media. It considers such examples as Sherlock Holmes, Kewpie Dolls, media forms and practices such as cut-outs, theater broadcasting, as well as a number of authors [140].

#### **3.7 Videogames, animation and virtual worlds**

Finally, it is necessary to mention videogames as a growing field within Intermedial Studies. Videogames are played in an increasing range of technical media, including home consoles, mobile devices (smart phones, tablets) or computer games. The can be played individually or online with multiple participants. Markku Eskelinen [141] is one of the first theorists on the situation of videogames within the intermedial panorama together with Susana P. Tosca [142]. Many game scholars have examined relations between the visual arts, musicology, cinema, and videogames. In text-based adventure games, the progress follows traditional narrative lines, although the game might include several levels that depend on the inner programming. Elverdam and Aarseth [143] proposed a model of games according to which the game object is divided into a mechanical layer (the rule-based engine) and a semiotic layer, stressing that these layers exist independently of each other. The semiotic layer is the game interface and that which is perceptible to the player. The mechanical layer contains the rules of game manipulation, which include restrictions and affordances. In adventure games that follow a narrative pattern, the game space is accessed by an avatar or game-character, under the control of the player. Rune Klevjer also characterizes avatars as a sort of prosthetic tele-extension that may take on the player's point of view (there are also games that use third-person panoramic point of view) [144]. Unlike singleplayer games where the player can traverse through a single sequence of events, in multiplayer environments, each player makes diverse choices and follows different paths with various outcomes. In these cases, there are different story lines within the same shared fictional world (see also [145]).

Animation has played a key role in television, cinema and videogames. Christopher Holliday and Alexander Sergeant [146] study the relationship between fantasy cinema and the medium of animation. The term 'machinima', a portmanteau of the words machine and cinema, refers to the use of computer graphic engines to create a cinematic production [146]. This practice of using graphic engines from video games arose from the animated software of the 1980s as an alternative to traditional frame-based animation (see [147]). Holliday and Sergeant's collection provides a range of essays on subjects including Disney, Pixar, and Studio Ghibli, filmmakers such as Ralph Bakshi and James Cameron, and film and television franchises such as Dreamworks' How To Train Your Dragon (2010) and HBO's Game of Thrones (2011).

Other scholars have explored how games also remediate, simulate and integrate other qualified media. Adam Lowenstein *Dreaming of Cinema: Spectatorship, Surrealism, and the Age of Digital Media* is a wonderful example of how surrealist

techniques and ideas can be used as a tool to understand new media [148]. The author focuses on 'enlarged spectatorship' (the artwork enlarged with the spectator's associations), 'interactive spectatorship' (this part on Roger Caillois' *Man, Play and Games* 1961, the exploration of David Cronenberg's *eXistenZ* 1999, and contemporary gaming are particularly interesting). Finally, Lowenstein examines fan videos and a YouTube channel in order to explore 'collaborative spectatorship'.

In their collection, *Metamorphoses of (New) Media* (2016), Julia Genz and Ulrike Küchler examine media uses within social discourse and esthetics covering digital storytelling and videogames [149]. Games employ multimodal sign systems that correspond to different sensorial qualities, so that their content can be explored through the lens of adaptation, transmedial storytelling, and convergence culture.

Michael Fuchs & Jeff Thoss' volume *Intermedia Games* (2019) studies transmedia storyworlds by exploring their remediation in film, television, and literature [150]. As already mentioned, the term 'storyworld' was proposed by Marie-Laure Ryan to differentiate between the spatiotemporal world context and the mode of representation used to mediate it (discursive, iconic, etc.). Like Jan Baetens, Ryan has been exploring the phenomenon of 'transmedial narratology' ([151], p. 35) in an attempt to explain how only certain forms of temporal unfolding of a series of events present in a narrative could be transferred among media [151]. This research can provide an answer to the problem of considering videogames as events that can engage actions while films play on visual images and a sort of internal interpretation.

SusanaTosca and Lisbeth Klastrup have explained that transmedial worlds have three core features: the mythos or backstory of the world, that is, 'the central knowledge one needs to have in order to interact with or interpret events in the world successfully' ([152], p. 4); the topos, which describes the spatiotemporal setting of the world, and the ethos, that describes the codes of behavior and connects the characters to the logic of the world [152]. Although some e-lit, e-poetry and Net. Art tries to break narrative conventions, in general, transmedia is intimately connected to artworks that are built around a story as well as to brands and franchises that spread content across media by corporate conglomerates (e.g., Star Wars, The Matrix, Harry Potter, Disney franchises, etc.) as well as with the entanglement with fanfiction etc. Although transmedia storytelling predated digitalization, media convergence has greatly extended its possibilities.

It is important to note that Elleström argues that 'transmedia storytelling', that is, narratives in different media types working together to form a larger whole, requires that narratives can be largely transmediated. It is in this sense that he formulates his wider notion of 'transmediality' and interrogates its limitations, since media types open possibilities as well as pose constraints that can impact upon narration (for an extensive catalog of studies that explore different media 'affordances', see Elleström [49], pp. 7–10, [49, 105–107].

Worth mentioning is the question of online platforms. The MIT Platform Studies Book Series highlights this neglected aspect that includes research on physical platforms (i.e. arcades, consoles, card decks, and boards), physical interfaces (i.e. mouse and keyboard, controllers, and VR headset), their relations to computational codes and shells (i.e. operational systems and software frameworks), the mechanical layers (set of actions users can perform to change the state of the game) and, finally presentational layers, with objects whose ontological status is context-dependent and may change over time. These platforms are also subject to different levels of regulation such as public or private use, age access, level of audience active participation, and so on.

#### *Literature Review on Intermedial Studies: From Analogue to Digital DOI: http://dx.doi.org/10.5772/intechopen.112611*

In the era of Virtual Reality (VR) and Artificial Intelligence (AI) Intermedial Studies face new challenges. The scopic drive to enter contemporary 3D spaces could be compared to the study of filmic close-ups performed by Laura Mulvey ([153], p. 14–26). The first recorded use of the term 'virtual reality' occurred in a collection of essays entitled *Le Théâtre et son double* (1938) authored by French avant-garde playwright Antonin Artaud who used it to describe the illusory nature of characters and objects in theater. The concept of 'scopic absorption' was coined by media theorist Vivian Sobchack [154] to describe how viewers become fully absorbed to the point of losing themselves in the visual experience with great emotional intensity. New media, like AI, continues to reinforce aspects already pointed out after the emergence of cinematography, for example, in Sobchack's scopic or Mulvey's 'male gaze,' which emphasized the ways in which cinema reinforced traditional gender roles. The benefits of AI might be many, but the logic of scopic absorption is already a danger. The fact of looking is justified and exhausted in itself, without giving a chance to understand the real story or the fake behind it [155]. This goes to show the intricate temporal border crossings in the study of intermediality, defending the need to look at the field in a diachronic perspective, exploring shifts in tech-material considerations as well as the conceptual and sociocultural configurations that qualify media.

#### **4. Conclusions**

As this chapter has shown, Intermedial relations are a 'complex and highly dynamic set' ([156], p. 10) of alliances and relations amid various agents – creators, readers/users, institutional drivers, qualifying aspects, and so on. They stage what Higgins, the father of the term 'intermedia', contemplated as a 'conceptual fusion' ([157], p. 30). In all cases, the conceptual and semiotic level is always conditioned by the material substratum. The tools and media we employ play a significant role in knowledge management, information transmission, and artistic creation. From analogue to digital constraints, these elements can function as enablers as well as barriers in the semiotic processes of communication and knowledge transfer ([158], p. 252).

As the chapter has shown, over the past 50 years, numerous significant studies on media have been conducted by scholars. The literature review presented has shown that intermediality is very much an interdisciplinary pursuit, a cross-breeding between the arts and other cultural practices ([159], p. vii). The main goal has been to illuminate the development of the field of intermediality in the light of adjacent areas of inquiry, such as semiotics, narratology, and visual studies, including, for instance, performance, cinema and television, and more recently, game studies, among others.

The chapter highlights the network of interconnections among media products, practices, and concepts within the broader media culture of the last decades, after the digital computer became a metamedium that concentrated all previous forms of human communication and representation – speech, writing, image-making, sound and radio, cinema, television, and so on – in one single cyberspace, enabling the integration of digital technology into our portable screen devices and, thus, facilitating human engagement with information at any time and from any place. The process of digitalization emphasizes that media not only represent but also provide resources for engaging with and influencing reality, aspects of intermedial agencies that still require further exploration [23].

Within Intermedial Studies, transmedia storytelling poses numerous challenges for narratology, including the need to develop skills in multimedia semiotic analysis. In the context of constantly evolving and expanding stories, a narratological approach must extend beyond studying narrative representations within a single medium, recognizing the rich complexity of natural and artificial languages. Exploring the histories and diverse manifestations of intermedia and transmedia is crucial in order to foster a deeper comprehension of their interconnections.

On the premise that 'all media are mixed media' ([160], p. 260) and that 'everything is intermedial' [161], the main takeaway is that the field of Intermedial Studies can still offer important insights into the nature of meaning-making, being pertinent even after digital media convergence.

### **Author details**

Asun López-Varela Azcárate Complutense University Madrid, Madrid, Spain

\*Address all correspondence to: alopezva@ucm.es

© 2023 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

### **References**

[1] Elleström L, editor. Media Borders: Multimodality and Intermediality. London, UK: Palgrave-Macmillan; 2010

[2] Elleström L. Media Transformation: The Transfer of Media Characteristics among Media. London, UK: Palgrave Macmillan; 2014

[3] Elleström L. Beyond Media Borders. Vol. 1-2. New York, OA: Springer International Publishing; 2021. DOI: 10.1007/978-3-030-49679-1, 10.1007/978-3-030-49683-8 [Accessed: 2023-01-30]

[4] Bennett O, editor. Edge 90: Art and Life in the Nineties. London, Amsterdam: Edge Biennale Trust, Stichting Mediamatic Foundation; 1990

[5] Kress G, Leeuwen VT. Reading Images: The Grammar of Visual Design. London, New York: Routledge; 1996

[6] Kress G, Leeuwen VT. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Hodder Arnold; 2001

[7] Kress G. Multimodality: A Social Semiotic Approach to Contemporary Communication. London, New York: Routledge; 2009

[8] Jakobson R. On linguistics aspects of translation. In: Roman Jakobson's Selected Writings. Vol. II. The Hague: Mouton; 1959/1971. pp. 260-266

[9] Clüver C. On Intersemiotic transposition. Art and Literature. Special issue of Poetics Today. 1989;**10**(1):55-90

[10] Plett HF. Intertextualities. In: Plett HF, editor. Intertextuality. Berlin, NY: Walter de Gruyter; 1991. pp. 3-29

[11] Eco U. Experiences in Translation. University of Toronto Press; 2001

[12] Henrik G. Multidimensional translation: Semantics turned semiotics. In: Nauert S, Gerzymisch-Arbogast H, editors. Challenges of Multidimensional Translation. Proceedings of the Marie Curie Euroconferences MuTra 2005. Vol. 2007. Saarbrücken: Saarland University. pp. 33-61

[13] Schober R. Unexpected Chords: Musico-Poetic Intermediality in Amy Lowell's Poetry and Poetics. Reihe American Studies. Heidelberg: Universitätsverlag Winter; 2011

[14] Aguiar D, Queiroz J. Semiosis and intersemiotic translation. Semiotica. 2013;**196**:283-292. Available from: https://www.degruyter.com/document/ doi/10.1515/sem-2013-masthead196/html [Accessed: 2023-01-30]

[15] Gilman EB. Interart studies and the 'imperialism' of language. Art and Literature. Special issue of Poetics Today. 1989;**10.1**:5-30. DOI: 10.2307/1772553 [Accessed: 2023-01-30]

[16] Azcárate AL-V. Intermedial comparative literature: From the sister-arts debate to the 20th century Avant-gardes. In: Azcárate AL-V, editor. Comparative Literature. Interdisciplinary Considerations. London, UK: InTechOpen;

[17] Scott GF. The rhetoric of dilation: Ekphrasis and ideology. Word & Image. 1991;**7**(4):301-310

[18] Ong W. Orality and Literacy, the Technologizing of the World. London, New York: Routledge; 1982

[19] Mignolo W. Literacy and colonization: The New World experience. In: Jara R, Spadaccini N, editors. In 1492- 1992: Re/Discovering Colonial Writing. Minneapolis: Prisma Institute; 1989. pp. 51-85

[20] McGann J. Black Riders the Visible Language of Modernism. NJ: Princeton University Press; 1993

[21] Mitchell WJ, Thomas. Space and time: Lessing's Laokoon and the politics of genre. In: Iconology: Image, Text, Ideology. Chicago, IL: University of Chicago Press; 1986

[22] Thomas MWJ. Picture theory. In: Essays on Verbal and Visual Representation. Chicago, IL: University of Chicago Press; 1994

[23] Azcárate AL-V, Henry S. Technopoïesis: Transmedia Mythologisation and the Unity of knowledge. Icono. 2017;**14**(15):1-34. Available from: http://www.icono14.net/ ojs/index.php/icono14

[24] Müller J. Media Encounters and Media Theories. Münster: Nodus; 2007

[25] Stiegler B. 'Iconic Turn' et réflexion sociétale. In: Revue Franco-Allemande de Sciences Humaines et Sociales. Paris, France: Trivium; 2008. Available from: https://journals.openedition.org/ trivium/308

[26] Bal M. See signs: The use of semiotics for the understanding of visual art. In: Cheetham MA et al, editors. The Subjects of Art History: Historical Objects in Contemporary Perspective. Cambridge, UK: Cambridge University Press; 1998. pp. 74-93

[27] Neumann B, Rippl G. Verbal-Visual Configurations in Postcolonial Literature. London, New York: Routledge; 2020

[28] Rui CH, de Fatima LM, editors. Writing and Seeing. Essays on Word and Image. Amsterdam, New York: Rodopi; 2006

[29] Heusser, Martin. Radical eye rhymes: Visual strategies in modernist poetry. In: de Carvalho Homem RM, editor. Relational Designs in Literature and the Arts: Page and Stage, Canvas and Screen. Amsterdam, New York: Rodopi; 2012. pp. 221-235

[30] Krieger M. Ekphrasis: The Illusion of the Natural Sign. Baltimore: The Johns Hopkins University Press; 1992

[31] Steiner W. The Colors of Rhetoric. Chicago, IL: University of Chicago Press; 1982

[32] Heffernan JAW. Museum of Words: The Poetics of Ekphrasis from Homer to Ashbery. Chicago, IL: University of Chicago Press; 1993

[33] Clüver C. In: Robillard V, Jongeneel E, editors. Quotation, Enargeia, and the Function of Ekphrasis. Pictures into Words: Theoretical and Descriptive Approaches to Ekphrasis. Amsterdam: VU University Press; 1998. pp. 35-52

[34] Persin M. Getting the Picture: The Ekphrastic Principle in Twentieth-Century Spanish Poetry. Lewisburg: Bucknell University Press; 1997

[35] Breitbach J. Analogue Fictions for the Digital Age: Literary Realism and Photographic Discourses in Novels after 2000. Rochester, NY: Camden House; 2012

[36] Bruhn J. The Intermediality of Narrative Literature: Medialities Matter. Palgrave-Macmillan; 2016

[37] Bruhn S. Musical Ekphrasis: Composers Responding to Poetry and *Literature Review on Intermedial Studies: From Analogue to Digital DOI: http://dx.doi.org/10.5772/intechopen.112611*

Painting. Hillsdale, NY: Pendragon Press; 2000

[38] Stawiarski M. Musical patterns in William H. Gass's "a fugue" and the Pedersen kid. In: López-Varela A, Sukla A, editors. The Ekphrastic Turn: Inter-Art Dialogues. Illinois Research Park: Common Ground Publishing; 2015. pp. 55-77

[39] Azcárate AL-V. Landscapes of time: Windows to the world. In landscapes of loss. Verbeia Journal of English and Spanish Studies. 2020;**4**:53-63. Available from: https://journals.ucjc.edu/VREF/ article/view/4273/3109 [Accessed: 2023-01-30]

[40] Wolf W. The Musicalization of Fiction: A Study in the Theory and History of Intermediality. Amsterdam, New York: Rodopi; 1999

[41] Wolf W. Intermediality revisited reflections on word and music relations in the context of a general typology of Intermediality. In: Lodato SM, Aspden S, Bernhart W, editors. Word and Music Studies. Amsterdam: Brill; 2002. pp. 13-34. DOI: 10.1163/9789004334069\_003 [Accessed: 2023-01-30]

[42] Wolf W. The Metareferential Turn in Contemporary Arts and Media. Amsterdam, New York: Rodopi; 2011

[43] Wolf W, Bernhart W. Selection of Essays on Intermediality by Werner Wolf (1992-2014). Theory and Typology, Literature-Music Relations, Transmedial Narratology, Miscellaneous Transmedial Phenomena. Amsterdam, Netherlands: Brill-Rodopi; 2017

[44] Müller JE, Intermedialität. Formen Moderner Kultureller Kommunikation. Münster: Nodus; 1996

[45] Wagner P, editor. Icons, Texts and Iconotexts. Essays on Ekphrasis and

Intermediality. Berlin: Walter de Gruyter; 1996

[46] Scholz BF. Sub Oculos Subiectio': Quintilian on Ekphrasis and Enargeia. In: Robillard V, Jongeneel E, editors. Pictures into Words: Theoretical and Descriptive Approaches to Ekphrasis. Amsterdam: VU University Press; 1998. pp. 73-99

[47] Rajewsky IO. Intermedialität. Tübingen: Francke; 2002

[48] Rajewsky I. Intermediality, intertextuality, and remediation: A literary perspective on Intermediality. Intermedialités/Intermedialities. 2005;**6**:43-65

[49] Ellestöm L. Transmedial Narration: Narratives and Stories in Different Media. London, UK: Palgrave Pivot; 2019

[50] Bolter JD. Transference and transparency: Digital technology and the remediation of cinema. Intermédialités. 2005;**6**:13-26

[51] Grishakova M, Ryan M-L. Intermediality and Storytelling. Berlin, New York: De Gruyter; 2010

[52] Freda C, Chiel K, editors. Intermediatlity in Theatre and Performance. Amsterdam, New York: Rodopi; 2006

[53] Boenisch Peter M. Mediation unfinished. Choreographing Intermediality in contemporary dance performance. In: Kattenbelt C, Chapple F, editors. Intermediality in Theatre and Performance. Amsterdam, New York: Rodopi; 2006. pp. 151-166

[54] Eco U. A Theory of Semiotics. Bloomington, IN, London: Indiana University Press, Macmillan; 1976

[55] Baetens J, Frey H. The Graphic Novel: An Introduction. Cambridge, UK: Cambridge University Press; 2014

[56] Baetens J. The Film Photonovel: A Cultural History of Forgotten Adaptations. Austin, TX: University of Texas Press; 2019

[57] Stein D, Thon J-N. From Comic Strips to Graphic Novels: Contributions to the Theory and History of Graphic Narrative. Berlin, NY: Mouton de Gruyter; 2013

[58] Beineke CE. Towards a Theory of Comic Book Adaptation. University of Nebraska; 2011. Available from: https:// cursointermidialidade.files.wordpress. com/2014/08/towards-a-theory-ofcomic-book-adaptation.pdf [Accessed: 2023-01-30]

[59] Wiebe R, editor. Polyptych: Adaptation, Television, and Comics. Wilmington, DE: Vernon Press; 2021

[60] Benjamin, Walter. The Work of Art in the Age of Mechanical Reproduction. Trans. J. A. Underwood. London: Penguin; 2008

[61] Ingham R. From Page to Stage: How Theatre Designers Make Connections between Scripts and Images. Portsmouth NH: Heinemann Educational; 1998

[62] Brand N. Dramatic Notes. Foregrounding Music in the Dramatic Experience. London, UK: John Libbey & Company Limited; 1998

[63] Bull M. The Routledge Companion to Sound Studies. London, New York: Routledge; 2018

[64] Crabtree S, Beudert P, editors. Scenic Art for Theatre: History, Tools and Techniques. London, New York: Routledge; 2012

[65] McAdams DA. Caught in the Act. A Visual History of Contemporary Multimedia Performance. New York: Aperture Foundation; 1996

[66] Baugh C. Theatre Performance and Technology. The Development of Scenography in the Twentieth Century. Basingstoke: Palgrave MacMillan; 2005

[67] Lavender A. Multimodal acting and performing. In: Elleström L, editor. Beyond Media Borders (Vol. 1): Intermedial Relations among Multimodal Media. Amsterdam, Netherlands: Palgrave Macmillan; 2021. pp. 113-140

[68] Brown G, Hauck G, Larrue J-M. 'Mettre en scène': une approche intermédiale de la réalité théâtrale actuelle. Intermédialités. 2008. Available from: http://www.erudit. org/revue/im/2008/v/n12/039228ar. html?vue=resume;**12**:9-12 [Accessed: 2023-01-30]

[69] Sager Eidt Laura E. Writing and Filming the Painting. Ekphrasis in Literature and Film. Amsterdam, Netherlands: Brill; 2008

[70] Drumm E. Painting on Stage: Visual Art in Twentieth-Century Spanish Theater. Lewisburg, PA: Bucknell University Press; 2010

[71] Bay-Cheng S, Chiel K, Andy L. Mapping Intermediality in Performance. Amsterdam, Netherlands: Amsterdam University Press; 2010

[72] Mancewicz A. Intermedial Shakespeares on European Stages. Amsterdam, Netherlands: Palgrave Macmillan; 2014

[73] Wyver J. An intimate and Intermedial form: Early television Shakespeare from BBC, 1937-1939. Shakespeare Survey. 2016;**69**:347-360

[74] Ingham M. Stage-Play and Screen-Play. The Intermediality of Theatre and Cinema. London, New York: Routledge; 2016

*Literature Review on Intermedial Studies: From Analogue to Digital DOI: http://dx.doi.org/10.5772/intechopen.112611*

[75] Foellmer S, Schmidt M, Katharina; Schmitz, Cornelia. Performing Arts in Transition: Moving between Media. London, New York: Routledge; 2018

[76] de Paiva VM. Out of space: The complexity of contemplative and performative architectural ekphrasis. In: Pawelec A, Shaw A, Szpila G, editors. Text-Image-Music: Crossing the Borders. Intermedial Coversations on the Poetics of Verbal, Visual and Musical Texts. Lausanne, Switzerland: Peter Lang; 2021. pp. 135-149

[77] Houwen J. Film and Video Intermediality: The Question of Medium Specificity in Contemporary Moving Images. London, UK: Bloomsbury; 2017

[78] Marion S. Intermedial Dialogues: The French New Wave and the Other Arts. Edinburgh Univ. Press, Studies in Film and Intermediality; 2021

[79] Geiger J. Cinematicity in Media History. Edinburgh University Press; 2015

[80] Mariniello S. Cinéma et intermédialité, Cinémas. Revue d'études cinématographiques. 2000. Available from: https://www.erudit.org/fr/revues/ cine/2000-v10-n2-3-cine1881/;**10**(2-3) [Accessed: 2023-01-30]

[81] Gaudreault A, Russell C, Véronneau P. The Cinema, a New Technology for the 20th Century. Lausanne, Switzerland: Payot-Lausanne; 2004

[82] Pisano G. Une archéologie du cinéma sonore. Paris: CNRS; 2004. Available from: https://books.openedition.org/ editionscnrs/2715?lang=es [Accessed: 2023-01-30]

[83] Nasta D, Huvelle D, editors. New Perspectives in Sound Studies. Lausanne, Switzerland: Peter Lang; 2004

[84] Altman R. Silent Film Sound. New York: Columbia University Press; 2004

[85] Pethö Á. Intermediality in Contemporary Eastern European and Russian Cinema. Edinburgh, UK: Edinburgh University Press; 2020

[86] Pethö Á. Cinema and Intermediality: The Passion for the In-between. Cambridge Scholars Publishing; 2011

[87] Giusti D, Luciano. Immagini migranti. Forme intermediali del cinema in era digitale. Venezia: Marsilio; 2008

[88] Gaudreault A, Tim B. From Plato to Lumière: Narration and Monstration in Literature and Cinema. Toronto: University of Toronto Press; 2009

[89] Vieira I, Novo R, Célia., editors. Intermedia. Études en intermédialité. Paris: L'Harmattan; 2010

[90] Giuliani L, Negri S. Missing links: Digital cinema, analogical archives, film historiography. Intermédialités: Histoire et théorie des arts, des lettres et des techniques. 2011;**18**:71-84

[91] Allen Robert C, Annette H, editors. The Television Studies Reader. London, New York: Routledge; 2004

[92] Fiske J. Television Culture. London: Methuen; 1987

[93] Ellis J. Visible Fictions. London: Routledge & Kegan Paul; 1982

[94] Jenkins H. Convergence Culture: Where Old and New Media Collide. New York: New York University Press; 2006

[95] Reinsch PN, Westrup L. The Soundtrack Album: Listening to Media. London: Routledge; 2020

[96] Harris L. Composing Audiovisually: Perspectives on Audiovisual Practices

and Relationships. London, New York: Routledge; 2021

[97] Burns LA, Hawkins S. The Bloomsbury Handbook of Popular Music Video Analysis. London, UK: Bloomsbury; 2019

[98] Birdsall C, Enns A, editors. Mediations: Body Sound Technology. Newcastle upon Tyne, UK: Cambridge Scholars Press; 2008

[99] Carolyn B, Anthony E, editors. Mediations: Body Sound Technology. Cambridge Scholars Publishing; 2008

[100] Bijsterveld K. Soundscapes of the Urban Past: Staged Sound as Mediated Cultural Heritage. Bielefeld, Germany: Transcript Verlag; 2013

[101] Durcan S. Memory and Intermediality in Artists' Moving Image. London, UK: Palgrave Macmillan; 2020

[102] CRIalt. Available from: http:// cri.histart.umontreal.ca/cri/fr/ intermedialites/ [Accessed: 2023-01-30] Bibliographic resource on intermediality. Available from: https://www.zotero. org/groups/159258/intermedialite/ items/V9ESR7EC/item-list [Accessed: 2023-01-30]

[103] Azcárate AL-V, Ananta S. The Ekphrastic Turn: Inter-Art Dialogues. Illinois Research Park: Common Ground Publishing; 2015

[104] Azcárate AL-V. Posthuman Intermedial semiotics and distributed agency for sustainable development. In: Bruhn J, López-Varela A, de Paiva Vieira M, editors. The Palgrave Handbook of Intermediality. 2023. Available from: https://link.springer. com/referencework/10.1007/978-3-030- 91263-5 [Accessed: 2023-01-30]

[105] Castells M. The Rise of the Network Society. Oxford, UK: Blackwell; 1996

[106] Vannebar B. How we may think— The next chapter. Civic intelligence and collective metacognition. Spanda Journal VI. 1945. Reprinted 2015, 7-14 ;**1**. Available from: http://www.spanda.org/ SpandaJounrnal\_VI,1.pdf [Accessed: 2023-01-30]

[107] McLuhan M. Understanding media: The extensions of man. In: Lapham LH, editor. Introduction Cambridge, MA: MIT Press; 1964

[108] Jensen B, Klaus. Media Convergence: The Three Degrees of Network, Mass and Interpersonal Communication. London, New York: Routledge; 2010

[109] Hayles NK. Virtual bodies and flickering signifiers. In: Mirzoeff N, editor. The Visual Culture Reader. London, New York: Routledge; 1998. pp. 152-160

[110] Kittler FA. Gramophone, Film, Typewriter. Stanford University Press; 1999

[111] Manovich L. The Language of New Media. Cambridge, MA: MIT Press; 2001

[112] Cubitt S. The Cinema Effect. MIT Press; 2004

[113] Bolter JD. Grusin R. Remediation: Understanding New Media. Cambridge, MA: The MIT Press, 1999

[114] Landow GP. Hypertext 3.0: Critical Theory and New Media in an era of Globalization. Baltimore: Johns Hopkins University Press; 2006

[115] Ensslin A. Canonizing Hypertext: Explorations and Constructions. London: Continuum; 2007

*Literature Review on Intermedial Studies: From Analogue to Digital DOI: http://dx.doi.org/10.5772/intechopen.112611*

[116] Aarseth E. Cybertext. Baltimore, MD: Johns Hopkins University Press; 1997

[117] Ryan M-L, Thon J-N. Storyworlds across Media: Toward a Media-Conscious Narratology. Lincoln, Nebraska: University of Nebraska Press; 2014

[118] Mark B. Patterns of hypertext. In: Shipman F, Mylonas E, Groenback K, editors. Proceedings of hypertext'98. New York: ACM; 2003. Available from: https://www.eastgate.com/patterns/ Print.html [Accessed: 2023-01-30]

[119] Flores L. Digital Textuality and its Behaviors. In: López-Varela A, Sukla A, editors. The Ekphrastic Turn: Inter-Art Dialogues. Illinois: Common Ground Publishing; 2015. pp. 382-402

[120] Serge B, López-Varela A. Making sense of the digital as embodied experience. In: de Zepetnek ST, Azcaráte AL-V, Saussy H, Mieszkowski J, editors. New Perspectives on Material Culture and Intermedial Practice. Vol. 13. No. 3. CLCWeb: Comparative Literature and Culture; 2011. DOI: 10.7771/1481- 4374.1793 [Accessed: 2023-01-30]

[121] Azcárate AL-V. Intermedial cognitive semiotics: Some examples of multimodal cueing in virtual environments. Review of Cognitive Linguistics. 2013;**11**(2):386-399

[122] Joachim P, Jens S, editors. Intermedialität Analog/Digital. Theorien, Methoden, Analysen. München: Wilhelm Fink; 2008

[123] Schröter J. Discourses and models of Intermediality. de Zepetnek ST, Azcárate AL-V, Saussy H, Mieszkowski J, editors. New Perspectives on Material Culture and Intermedial Practice. Vol. 13. No. 3. CLCWeb: Comparative Literature and Culture; 2011. Available from: http:// docs.lib.purdue.edu/clcweb/vol13/iss3/ [Accessed: 2023-01-30]

[124] Herkman J, Hujanen T, Oinonen P, editors. Intermediality and Media Change. Tampere, Finland: Tampere University Press; 2012

[125] Herzogenrath B. Travels in intermedia[lity]: An introduction. In: Herzogenrath B, editor. Travels in Intermedia[Lity]: Re-Blurring the Boundaries. Chicago, IL: Dartmouth College Press; 2012. pp. 1-14

[126] Balbi G, Magaudda P. A History of Digital Media: An Intermedia and Global Perspective. London: Routledge; 2018

[127] Cohen B, Streitberger A. The Photofilmic: Entangled Images in Contemporary Art and Visual Culture. Leuven, Belgium: Leuven University Press; 2016

[128] Spielmann Y. Hybrid Culture: Japanese Media Arts in Dialogue with the West. MIT Press; 2012

[129] Kim J. Between Film, Video, and the Digital: Hybrid Moving Images in the Post-Media Age. Bloomsbury; 2016

[130] Elwes C. Installation and the Moving Image. New York: Wallflower Press, Columbia University Press, Wallflower Press; 2015

[131] Johnson S, Ensslin A. Language in the Media: Representations, Identities, Ideologies. London, UK: Bloomsbury; 2018

[132] Jennings G, Mondloch K. Abstract Video: The Moving Image in Contemporary Art. Oakland, CA: University of California Press; 2015

[133] Benthien C, Lau J, Marxsen MM. The Literariness of Media Art. London, New York: Routledge; 2018

[134] Masura N. Digital Theatre: The Making and Meaning of Live Mediated. London, UK: Palgrave Macmillan; 2020

[135] Campos L, Schopf FJ. Music on Stage. Newcastle upon Tyne, UK: Cambridge Scholars Publishing; 2020

[136] Simonson M. Body Knowledge: Performance, Intermediality, and American Entertainment at the Turn of the Twentieth Century. New York: Oxford University Press; 2013

[137] Jarvis L, Savage K. Avatars, Activism and Postdigital Performance: Precarious Intermedial Identities. Methuen Drama; 2021

[138] Jarvis L. Immersive Embodiment: Theatres of Mislocalized Sensation. London, UK: Palgrave Macmillan; 2019

[139] Rampazzo-Gambarato R. Carvalho-Alzamora G. Tárcia L. Theory, Development, and Strategy in Transmedia Storytelling. London, New York: Routledge; 2020

[140] Meyer C, Pietrzak-Franger M. Transmedia Practices in the Long Nineteenth Century. London, New York: Routledge; 2022

[141] Eskelinen M. The gaming situation. Game Studies. 2001. Available from: http://www.gamestudies.org/0101/ eskelinen/;**1**(1) [Accessed: 2023-01-30]

[142] Tosca S. Understanding Video Games. London, New York: Routledge; 2008

[143] Elverdam C, Aarseth E. Game classification and game design: Construction through critical analysis. Games and Culture. 2007;**2**(1):3-22

[144] Klevjer R. What Is the Avatar? Fiction and Embodiment in Avatar-Based Singleplayer Computer Games. Revised and Commented Edition. Transcript Verlag; 2022. DOI: 10.14361/9783839445792 [Accessed: 2023-01-30]

[145] Khaled R, Lessard L, Barr P. Documenting trajectories in design space: A methodology for applied game design research. In: Proceedings of the 13th International Conference on the Foundations of Digital Games. ACM Digital Library; 2018. DOI: 10.1145/3235765.3235767 [Accessed: 2023-01-30]

[146] Holliday C, Sergeant A. Fantasy/ Animation: Connections between Media, Mediums and Genre. London, New York: Routledge; 2018

[147] Harwood TG, Grussi B. Pioneers in Machinima: The Grassroots of Virtual Production. Wilmington, DE: Vernon Press; 2021

[148] Lowenstein A. Dreaming of Cinema: Spectatorship, Surrealism, and the Age of Digital Media. New York: Columbia University Press; 2015

[149] Genz J, Küchler U. Metamorphoses of (New) Media. Newcastle upon Tyne, UK: Cambridge Scholars Publishing; 2016

[150] Fuchs M, Thoss J, editors. Intermedia Games—Games Inter Media: Video Games and Intermediality. London, UK: Bloomsbury Academic; 2019

[151] Ryan ML, editor. Narrative across Media: The Languages of Storytelling. Lincoln, Nebraska: University of Nebraska Press; 2004

[152] Tosca S, Klastrup L. Transmedial Worlds in Everyday Life. Networked Reception, Social Media, and Fictional Worlds. London, New York: Routledge; 2019

[153] Mulvey L. Visual and Other Pleasures. Bloomington, IN: Indiana University Press; 1989

*Literature Review on Intermedial Studies: From Analogue to Digital DOI: http://dx.doi.org/10.5772/intechopen.112611*

[154] Sobchack V. Carnal thoughts: Embodiment and moving image culture. Oakland, CA: University of California Press; 2004

[155] Azcárate AL-V. Intermedial semiotics in the age of artificial intelligence. challenges and opportunities for the arts. Semiotica: Journal of the International Association for Semiotic Studies. Forthcoming. 2024. Available from: https://www.degruyter.com/ journal/key/semi/html

[156] Emden CJ, Rippl G. Introduction. Image, text and simulation. In: Emden C, Rippl G, editors. ImageScapes: Studies in Intermediality. London, New York: Peter Lang; 2010. pp. 1-20

[157] Higgins D. Horizons: The Poetics and Theory of the Intermedia. Carbondale, IL: Southern Illinois University Press; 1984

[158] Wolf W. Intermediality. In: Herman D, Jahn M, Ryan M-L, editors. The Routledge Encyclopedia of Narrative Theory. London, New York: Routledge; 2005. pp. 252-256

[159] Eilittä L, Louvel L, Kim S. Introduction: From interdisciplinarity to intermediality. In: Eilittä L, Louvel L, Kim S, editors. Intermedial Arts: Disrupting, Remembering and Transforming Media. viii-xiii. Newcastle upon Tyne, UK: Cambridge Scholars Publishing; 2012

[160] Mitchell WJ, Thomas. There are no visual media. Journal of Visual Culture. 2005;**4**(2):257-266

[161] Elleström L, Fusillo M, Petrocola M. Everything is intermedial: A conversation with Lars Elleström. Between. 2020, 2020;**10**(20). Available from: https:// ojs.unica.it/index.php/between/article/ view/4443

#### **Chapter 2**

## Cut-Up Transmediality: From Text to Image

*Benjamin J. Heal*

#### **Abstract**

This chapter explores the visual aspects of William S. Burroughs and Brion Gysin's "cut-up" project as specifically "transmedial," rather than "intermedial" by critically examining their exploration of this largely aleatory collage technique. Their later shift from applying it to text to other media, including sound and film, can be seen as a fascinatingly prescient sojourn into what would later become transmedia storytelling. By drawing on theories ranging from recent studies on trans- and intermediality to theoretical works by Paul Ricoeur and Roland Barthes, the chapter will offer a theoretical framework for approaching the cut-ups as part of a wider aegis of resistance to conventionalities while considering how the transmediality of the cut-up project has implications for its disruption of language and time. This chapter will also expand the field and open up new areas for analysis.

**Keywords:** cut-ups, collage, avant-Garde, Beat, Burroughs, transmediality, intermediality, liminality

#### **1. Introduction**

Articles by Werner Wolf and Jens Schröter elucidate both the "intermedial turn" and "ontological intermediality," yet they have done so without much recourse to concrete examples. The problems with considering intermediality lie in part with the difficulty of "fixing" a medium, such as a novel, in the first place. Schröter notes that "media are determined only relationally and differentially so […] do not possess any absolutely constant 'being'". While we may consider a particular film, novel, painting, etc. to possess a very clear sense of its medium, that medium is itself merely a construction, holding no ontological essence outside of its relationality to other media [1]. In his defense of literary studies, Wolf discusses the importance of the need to study "(inter)mediality," along with a need for "sound disciplinarity with regard to a well-informed focus on individual media" [2]. This article will attempt to do both, and while moving away from the term "intermedia" will focus on "the cut-up project," a key twentieth-century avant-garde art movement predominantly associated with William Burroughs and Brion Gysin, that I argue led to a thoroughgoing material examination of the spaces between media.

"Intermedia" remains a term most closely associated with Dick Higgins and the Fluxus movement of the 1960s, with his comment that "Much of the best work being produced today seems to fall between media. This is no accident. The concept of the

separation between media arose in the Renaissance" [3]. His point is that as an artistic mode it was nothing new, as in a sense primitive cave paintings, in their ability to be both aesthetic and linguistic, are "intermedia." Higgins is rightly referencing the reductive tendency in Western traditions to categorize "disciplines" and focus on the singular genius of the artist, author or auteur, elements touched upon by French theorists Roland Barthes and Michel Foucault. In her exploration of Barthes' position on interdisciplinarity Diana Knight writes, "In line with the values he was developing elsewhere in his work, he moved too towards something resembling an ethics of interdisciplinarity, whereby a non-competitive intellectual environment would replace disciplinary protectionism. Underpinning Barthes's approach to interdisciplinarity was the conviction that the human sciences needed to question the status of their discourse" [4]. Barthes' critical exploration went far beyond textual analysis to include readings that defied structural boundaries of discipline and media, indeed in many ways this can be seen as one of the central tenets of his theoretical project.

In his response to Barthes' 1966 essay "The Death of the Author" Foucault too considers the problematic connections between the constructed and fluid categorizations that have come to be known as "author" and "work." Marshall McLuhan's theory that the medium/media that carries content holds an additional, easily overlooked message is another way of considering the power of intermedial approaches, as he states, "it is only too typical that the 'content' blinds us to the character of the medium" [5]. In this context he is not referring to "medium" as synonymous with "media," as in the technology that carries content (book, movie reel, tape, etc.), but rather the social and/or environmental *effects* of that technology. Wolf defines medium as "a conventionally and culturally distinct means of communication, specified […] primarily by the use of one or more semiotic systems in the public transmission of contents that include, but are not restricted to, referential 'messages.'" [2]. This allows an inclusivity that can include both "book" and "novel," while remaining necessarily vague. He adds that "media make a difference as to what kind of content can be evoked, how these contents are presented, and how they are experienced," which is the central point that needs exploration, these differential notions of evocation, presentation and most importantly, experience [3].

Although Schröter's attempts to describe the discourse of intermediality are insightful, particularly his notion of "ontological intermediality" which determines media existing relationally, "intermedia" can be seen more simply in current parlance as synonymous with "transmedia" and "multi-media," with only subtle differences. "Multimedia" was first coined *after* Higgins' introduction of intermedia by Bob Goldstein for his 1966 "LightWorks at L'Oursin" art show, and only in the 1990s did the term develop its more simplistic and physical meaning of a multiplicity of text, still photos, animations, videos, audio, and interactive applications. It is occasionally, though perhaps inaccurately, used to describe artists or stars that work in different media, for example famous actors who sing can be described as "multimedia stars" [6]. As noted above intermedia fits works that fall *between* media, an extreme example Higgins uses is the "intermedium between painting and shoes," while also noting Marcel Duchamp's "readymades" as falling between sculpture and "something else" [2]. Such problems persist both with defining a particular media, and "intermedia" itself, for example, film is effectively an intermedia blending photography, sound and potentially many other media, which makes "intermedia," in this context, a particularly vague and confusing term.

"Transmedia" is predominantly associated with the tendency for contemporary franchise entertainment to develop media across a range of formats, such as

#### *Cut-Up Transmediality: From Text to Image DOI: http://dx.doi.org/10.5772/intechopen.111441*

*Star Wars*, which began as a 1977 George Lucas-directed movie but which continues to expand into an ever-growing array of literary, comic-book, graphic-novel, and interactive video-game formats, though these are perhaps better defined as media franchises, or simply mixed-media. The notion of transmedia as storytelling across multiple platforms and formats, developed by Henry Jenkins, is one that allows for a richer development of highly complex worlds with more detailed plot-lines and characterization, filling in any gaps with additional media [7]. Transmedia storytelling, broadly defined as storytelling across media, is problematic and ill-defined as Donna Hancox notes, "literary theory, narratology, semiotics, film theory, media studies, and so on—all contribute important perspectives to the scholarship and practice of transmedia storytelling, but [...] none is sufficient on its own" [8]. It is this notion of "intermediality" that most troubles Wolf, particularly in terms of approaching it as a literary scholar without the expertise in the analysis of other media, reaching into the field of comparative literature. The key problem is that while in franchises like *Star Wars* the use of different media has created a bewildering narrative universe that holds little continuity or canonicity, merely a vague sense of styles and features that has allowed a great deal of artistic freedom. Within this contemporary, ill-defined "intermedial" context, it is helpful to consider Burroughs and Brion Gysin's fundamentally rebellious cut-up project of the 1960s.

In the context of considering the continuity of Burroughs' works as not just a form of transmedia storytelling, but also profoundly "transmedial" in the sense that, corresponding but further than Schröter's relational ontological intermediality, cut-ups have a fluid relationship to mediality; always and importantly located in an unstable position between media highlighting the instability of such constructions. It is useful to note his response in a 1974 interview that, "In a sense, it's all one book. All my books are all one book" [9]. I would argue that Burroughs is unintentionally limiting his use of media here, that his meaning should be that all his works are, in fact, one work. That does not mean that his works are transmedia, that is, between media, but that his ideas, situations, characters, etc. are always in the process of transferring between books and other media in his works, and that the meaning and ideas in those works are communicated much more effectively, as experience, via those different media following McLuhan's dictum of the "medium is the message" [5]. It is in this context that it becomes useful to use "transmedia" to refer to Burroughs' works more broadly with a contextuality and continuity that crosses between media, utilizing the distinct characteristics of those media, and the problematic liminal spaces between, to produce different effects.

#### **2. The cut-up project: transmedial from the start?**

Burroughs' 1959 novel *Naked Lunch* is a ground-breaking collage of text, a phantasmagoria of register and genre blurring the distinctions between memoir, ethnography, fantasy, report, pulp fiction, parody, satire and dialog. It even claims "This book spill of the page" in introducing a section of vivid images and sounds, highlighting Burroughs' intention to transcend the novel medium [10]. While Timothy Murphy's suggestion that "Burroughs' career might be a new paradigm for the writer's active, shaping involvement with other mass media," highlights Burroughs' apparent exceptional status in the transmedial sphere, he does not note Burroughs' natural lineage in Dadaism, Surrealism, Lettrism and Futurism, movements that explored the blending of word and image, sound and performance [11]. Daniel Punday draws the parallel

between the increasing scholarship on Burroughs and his "Multimedia Aesthetic," arguing that "Burroughs wants to emphasize our movement between media as a way to draw our attention back to the materiality of language that makes up his text, and thus that forms the bedrock of our response to his work" [12]. This focus on materiality misses the importance of the liminality of the inter- or transmedia field, while the focus on Burroughs as exceptional in this field is somewhat reductive. Rather it is important to grasp the origins and effects of the essential starting point of Burroughs' interest in moving beyond the written word in terms of experimentation and as part of a project that had both its progenitors and collaborators, and consider more broadly the ramifications of his *trans*media experimentation.

The "cut-ups" were in-part an extension of the collage form of *Naked Lunch*, taken much further and incorporating an aleatory and predominantly visual element, in-part drawn from Tristan Tzara's Dada performance poetics. Gysin, an artist at first associated with the Surrealist movement, is credited with accidentally stumbling upon the cut-up technique of cutting and rearranging text, including works by other writers, at the "Beat Hotel" (9 Rue Gît-le-Cœur in Paris) in 1960. Cutting through newspapers while mounting his paintings Gysin noticed how the texts became rearranged, creating random juxtapositions, surreal breaks and neologisms, and he introduced the method to Burroughs and other artistic tenants in the Hotel, including Harold Norse, Gregory Corso and Sinclair Beiles. What developed was a rigorous, dialectical technique for producing a new kind of text, an offshoot of concrete poetry that would develop throughout the 1960s with an array of collaborative publications, three novels *The Soft Machine* (1961), *The Ticket that Exploded* (1962) and *Nova Express* (1964), tape/sound experiments, live performances and films. Gysin saw the evolution of the cut-ups through his part in the editing of *Naked Lunch* (Gysin, Beiles and others contributed), where visual elements informed the textual material: "Showers of snapshots fell through the air […]. Burroughs was more intent on Scotch-taping his photos together into one great continuum on the wall, where scenes faded and slipped into one another […]" [13]. He notes that they stuck pages of text onto the walls next to the photographs, cutting and pasting the pages together. Again highlighting the intersection of visual art and the literary, and the transmediality of cut-ups, Gysin states that the cut-up technique emerged in part from a discussion with Burroughs about "turning painter's techniques directly into writing" [13].

Defined vaguely as a "project" rather than a "writing experiment" or "transmedial technique," there is an implication that it would persist and be explored broadly within the scope of writing, yet almost immediately it expanded from the mostly poetic experiments of the first "official" cut-up project, the collaborative pamphlet *Minutes to Go* (1960). Referring to the aforementioned cut-up novels specifically, Robin Lydenberg makes the results of the technique seem similarly (and vaguely) prose-poetic, stating that their function is "to make the word an object detached from its context, its author, its signifying function […] to bring […] the collage to writing" [14]. While this misses much of the diversity of the project in terms of the intertextual inclusion of writing from a range of sources and genres, Lydenberg makes an important point regarding the visual concreteness that cut-ups bring to language with the word detached from context foregrounded as visual "object." Burroughs' mission therefore seems to be to push beyond this sense of the (con)textual, in a turn to the visual. The second pamphlet produced by Burroughs and Gysin using the technique, *The Exterminator* (1960), contains 4 pages of calligraphic material, marking out the text for Oliver Harris as "the most important early expression of the two men's creative synergy and of their experiments in verbal-visual form" [15].

#### *Cut-Up Transmediality: From Text to Image DOI: http://dx.doi.org/10.5772/intechopen.111441*

Title *The Exterminator* likely references the surrealist parlor game *le cadavre exquis* [the exquisite corpse], in which participants take turns to collaborate on writing a story or drawing a picture without seeing the other's contribution. In this context, the attempt *The Exterminator* makes to transcend individual authorship is also an attempt to transcend the reader's expectations by including drawings [15]. While the drawings might at a glance be considered peripheral to the main body text, or to use Gerard Genette's term "paratextual," Gysin's asemic writing is directly associated with his preceding Permutation Poems, short sentences comprehensively and repetitively reordered into every possible permutation, presented in block capitals [16]. The most notable is "RUB OUT THE WORDS," describing the attempt with cut-ups to remove, or "exterminate," word associations and the semantic baggage they carry. While the use of block capitals creates an immediate strangeness, not to mention the sense of expressiveness akin to shouting, the following sister poem "RUB OUT THE WORDS" symbolically underlines its title adding further expressiveness, while the poem itself replaces the permutated words with the logogrammatic symbols "#\$%&," highlighting the often arbitrary and purely symbolic meaning associations of words and morphemes. As Harris notes in discussing the recently published cut-up experiment *BATTLE INSTRUCTIONS* (2020), Burroughs uses block capitals in the page headings of *The Exterminator* to act as newspaper headlines, yet with the Permutation Poems the block capitals effect is, in a sense, that the poems should be read out loud, the effect similar to what Harris describes as "their visualisation of a technologically-enhanced, spontaneous *live* quality that models the text of *BATTLE INSTRUCTIONS* on another medium: radio broadcasting" [17]. The block capitals, as with the calligraphic drawings highlights the transmediality of language, oscillating as it does in the space between sound and speech, and drawing and written text.

While there are distinctions between the writing of each collaborator, particularly in *Minutes to Go*, it is only the Permutation Poems and calligraphic drawings that are attributed to Gysin, the other texts, the reader assumes (though cannot be sure), are the blended products of both contributors. Both *Minutes to Go* and *The Exterminator* have a more paratextual sense of separate experimentation that makes them distinct from the intertextual blending of texts and media that would occur later. *The Exterminator* is notable as it began the blurring of visual, textual, and authorship boundaries, where it was philosophically, as Harris states, enacting "a deconstruction of language that goes together with a dissolution of self, a project to exterminate nothing less than "the" space and time of the world known to "thee."' [15]. Time is of central importance to this equation, on a narrative and poetic level, to the effect cut-ups have, regardless of the source of the materials cut, on the reader (thee). Harris writes that material can be, "cut up to produce a collage of phrases and images that approximates the density, rhythmic measure, and uncanny effect of déjà-vu […]" [13]. The fundamental of cut-up text is distorted narrative, with words (morphing into image) repeating as phrases and poetic rhythms, affecting memory, evoking feelings of nostalgia and recreating a kind of dream logic.

The first edition of Burroughs' cut-up novel *The Soft Machine*, which he started writing as *Naked Lunch* was being published in 1959, demonstrates a preoccupation with invoking image using text in its sectional structure of color "Units" (Red, Green, Blue, White). While this would be largely removed from later editions the importance of the visual is underlined by the 1961 publication of the penultimate chapter of the American edition of *The Soft Machine* as a hand-lettered concrete poem arrangement "Wind Hand Caught in the Door" in Hamburg based journal *Rhinozeros* [18]. Similarly, and although not intended as part of *The Soft Machine*, the publication of

"Operation Soft Machine/Cut" in the Fall 1961 edition of *The Outsider* magazine, is prefaced with an illustrated cover and an image of Burroughs with newspaper text as background, with the following pages presented in a three column newspaper format overlaid with images of insects [19]. Such distortions to the traditional flow of textual narrative, and evocation of color as pure image, recalls Paul Ricœur's discussions on metaphor, in terms of poetic language, as an exaggerated or distorted imitation of reality. Such texts can, for Ricœur, be read, understood, and importantly, move people to action; in the sense that the writing of the past enables change in the present [20]. The reality distortion of cut-ups can produce unintended and/or random meaning in the reader, and indeed its poetry and prosody can enact a kind of shock, and can therefore be seen as more likely to produce action, than traditional narrative prose; the reader clearly knows something is up here. Such distortions are in part due to the break-up of genre or media expectation; Burroughs is exploring the effects of transmediality in his work, while training his readership to do the same. He notes this in reference to J.W. Dunne's 1927 book *An Experiment with Time*, a work that theorizes about precognitive dreams, by stating that within such dreams, "You are seeing not the event itself, but a newspaper picture of the event, prerecorded and prephotographed" [21]. The implication here is that cut-ups, by disrupting narrative expectations, can produce the same uncanny precognitive impressions that Dunne believed were presented by precognitive dreams which have pushed those who experienced them to action in attempts to prevent the negative outcomes of those dreams. This "minutes to go" urgency appears to be connected to the future recording and photographing of the event. However, given the orthodoxies of journalism and photojournalism often the representation of new events is linked to those of the past, for example in the careful staging of war photographs [22]. As Sean Bolton notes, in Burroughs' cut-up works there is a lack of fixity where there exists, "a simulacrum of material context, settings that not only simulate recognizable physical and temporal locations but replace them" [23]. Bolton is correct in recognizing Burroughs' experimentation with fixity, but remains rooted in the analysis of narrative, the novels and their readers, "The contexts of Burroughs' novels are determined, then, by readers during reading and according to their own experiences of the 'network of differences'" [23]. What this misses is the wider project of transmediality that further disrupts even this "network of differences".

Burroughs' subsequent 1961 cut-up novel *The Ticket that Exploded* ends with perhaps the most jarring transition in the cut-up trilogy of novels, a definitive blurring of text and image with a final section that morphs into a Gysin calligraphic work that itself is half-handwritten text, half-asemic writing [24]. The depiction of handwriting *becoming* asemic (essentially language without meaning), rather than as separate plates as presented in *The Exterminator*, and making the more mechanical aspects of language become more "bodily" by using handwritten cursive script, it becomes more expressive, a point made by Walter Benjamin, "graphology is concerned with the bodily aspect of the language of handwriting and with the expressive aspect of the body of handwriting" [25]. It also echoes, effectively, the break-down of semantic meaning that results from the cut-up technique and marks where Burroughs is most keenly aware of the importance of language as image, and his interest in developing the cut-up in the spaces between traditionally demarcated disciplines and media. Burroughs (and Gysin) were also fascinated by logographic writing systems, as Burroughs' 1960 letter to Gysin makes clear, "I want to go to Hong Kong to learn Chinese. Imagine repetitive poems and cut ups in Chinese. Shift lingual" [26]. Given that Mandarin contains ideograms, characters that directly represent what they

#### *Cut-Up Transmediality: From Text to Image DOI: http://dx.doi.org/10.5772/intechopen.111441*

mean, there is a sense that they are both words and pictures; this is likely the aspect that appealed to Burroughs as he worked to eliminate what he saw as the confusion of Western thought [27].

Although it can be argued that drawing and writing are fundamentally the same, some, such as Tim Ingold, have attempted to list distinctions, "First, writing is a *notation*; drawing is not. Secondly, drawing is an *art*; writing is not. Thirdly, writing is a *technology;* drawing is not. Fourthly, writing is *linear*; drawing is not" [28]. The problem with such distinctions is that they do not match up with cut-up writing; concrete poetry, for example, can be an art, and the poetic juxtapositions of cut-up writing can be seen in this context. The three-column newspaper arrangements described as a cut-up technique in Burroughs' 1964 essay "The Literary Techniques of Lady Sutton-Smith" certainly challenge the linearity of writing by prompting the reader to read both down and across the columns, "Arrange your texts in three or more columns and read *cross* column"; by challenging linear conventions Burroughs is creating indeterminacy in both reading and writing, producing unstable, oscillating transmedial texts that can be adapted by the reader, opening up the practices of reading and interpretation and pushing it towards the type of aesthetic interpretation favored by art, rather than literary, critics [29].

In "The Mayan Caper" section of *The Soft Machine* the narrator makes the importance of the matrix of time, cut-ups and transmediality particularly apparent. This begins with newspapers, which already blend photographs with words: "when I fold today's paper in with yesterday's paper and arrange the pictures to form a time section montage, I am literally moving back to the time when I read yesterday's paper, that is traveling in time back to yesterday" [19]. The experiments continue with film, "The next step was carried out in a film studio—I learned to talk and think backward on all levels—This was done by running film and sound track backward" [19]. As Harris states, with these new forms of publication Burroughs is proposing, "a different mode of temporality, an irregular temporality" that is designed as part of a broader attack on all forms of traditional conventionality, against any attempt to "standardise time, to narrate it in predictable and commercially stable ways" [15]. Although Dadaists, Surrealists and others had explored collage and aleatory reformulation before him, Burroughs discovered that collage methods removed conventional distinctions between media; what began with cutting up text extended to producing an author as jack-of-all-trades polyglot, utilizing pen, typewriter, and movie camera, although missing from "The Mayan Caper" is the use of the tape recorder [24].

#### **3. Cut-up sound and movies**

*The Ticket that Exploded* foregrounds the element of intertext largely missing from "The Mayan Caper" section of *The Soft Machine*; the sounds invoked by words. It certainly contains a broad range of references to music, as Harris notes: "Whole pages consist of nothing but song titles and sampled lyrics, collages of nursery rhymes and jazz standards, torch songs and blues ballads, cowboy tunes, Negro spirituals and Tin Pan Alley sentimental melodies" [24]. As Ian MacFadyen states in a discussion of Gysin's use of language, "Gysin explored the idea that a word resembles, indeed embodies, in its shape and sound, through alliteration and visual associations, what it describes — that meaning is influenced by the shape and sound of individual letters, and by their combinatory effects" [30]. Such "combinatory effects" mark those same interstices between media forms that can be seen as transmedial. The textual

reference to a song evokes notes and lyrics in the reader not on the page, a way of tapping into memory and meaning that exists entirely *between* the media. A similar consideration seems to have been made regarding the script-like block capitals of *BATTLE INSTRUCTIONS*; as Harris notes this was a part of the cut-up project yet was written as intended to be read out loud, "as a spoken word transmission" or broadcast via radio [17]. This wide matrix of language use and exploitation of liminality appears to have been part of Burroughs' consideration during many of the cut-up experiments.

Filmic experiments, aided by British director Anthony Balch, were also a key component of the cut-up project. "Towers Open Fire" (1962–1963) is a more-or-less linear narrative collage film that includes themes and situations from the books, as well as the grid patterns and glyphs Gysin was exploring that would be included in the later book *The Third Mind* (1976). The soundtrack is a Burroughs narration that includes text from his cut-up novels, alongside stock De Wolfe library music and Moroccan folk-trance music [31]. Also included prominently is Gysin and Ian Sommerville's "Dreamachine." The Dreamachine is a further transmedial artifact closely connected to the cut-up project. The title itself is a contraction of "dream" and "machine," cunningly connecting the two "media." Similar to a zoetrope cinema, the Dreamachine is a stroboscopic piece of light art, first produced in 1960, that functions by producing a flicker effect that if looked at with eyes closed produces a hypnagogic effect [32]. The cross-over of themes and inclusion of artworks is profoundly transmedial, and it is successful as a means of providing some explanation of the often bewildering experience of attempting to read the disjointed poetics of cut-up text. The apparent linearity of "Towers Open Fire" is disrupted further by the later experimental film "The Cut-Ups" (1966), a more precise expression of the cut-up method on film, in which, as John Sargeant notes, "Balch cut all of the original material into four sections, then handed the footage over to a "lady who was employed to take a foot from each roll and join them up"'[…] The editing was purely a mechanical task, most importantly "nobody was exercising any artistic judgment at all. The length of the shots (except for the last) [was] always a foot" [31]. The soundtrack has Burroughs reciting a Scientology auditing script designed to produce a "clear," repeating certain words and phrases intended to elicit an emotional response. Both Balch and Burroughs were interested, following Gysin and Sommerville's invention of the Dreamachine, in the subliminal effects of the moving image, and so the length of the cuts took this into consideration so that they were the shortest length that still allowed an audience to perceive an image. As Balch states, "a foot... is long enough for people to see what's there, but not enough to examine it in detail"' [31]. This consideration of film beyond the conventionalities of narrative cinema challenges the medium itself, and the collaborators took this further with the content of the film itself, which depicts Gysin producing grid-patterned roller painting, and performing asemic calligraphic writing. He is also shown writing phrases before turning the page 90 degrees, and continuing to write over the previous text, which as Sargeant states produces "a visual density of text which transforms into a calligraphic haze" [31]. The film, while still holding true to Burroughs' themes and preoccupations of the time, such as Scientology, the nascent power of language, and the prophetic power of cut-ups, produces some of the same initial dissonance as the books, pamphlets and other cut-up publications, yet the change in media also creates a range of effects, in particular a sense of entrapment as it forces the audience to watch a series of flashing images. Although the images are presented at random, with scenes structured according to lengths of celluloid rather than lengths of action or narrative, there also persists a sense of linearity to the work.

Nevertheless, as Sargeant notes, these film collaborations, "exist beyond the traditionally ascribed borders and limits of cinema [and] cut/edit/splice/jump/fold, beyond the traditionally ascribed limits" [31].

#### **4. Cut-up books further beyond the text**

While the *Minutes to Go* collaboration between Burroughs, Gysin and the poets Beiles and Corso marks the beginning of the cut-up project, it is arguably ended by the publication of Burroughs' didactic call to arms essay collection *Electronic Revolution* in 1970, and also by the belated collection *The Third Mind* (1978), which Burroughs and Gysin originally envisioned as a visual and textual collaboration. In part developing out of Burroughs' obsessive use of collage scrapbooks that developed during the writing of *Naked Lunch*, *The Third Mind* blends newspaper articles, notes, photographs and drawings. First published as œ*uvre croisée* in French (Paris: Flammarion 1976), the French title of *The Third Mind*, which can be translated as one piece of art made from combined and mixed artworks, was added by translator Gérrard George Lemaire, a likely planned distortion from the original intention. Yet this title, with its diagraph merged o and e demonstrates practically in letter units that by cutting and reformulating text and image a third meaning can be uncovered in the spaces between. MacFadyen calls the book "misunderstood," in part because it has still yet to be published as originally intended as the 1970 Grove Press edition was pulled from production even after reaching the proofing stage [30, 33]. This work was started in the 1960s, with an edition planned for publication in 1965, and as Jed Birmingham states, "the visual development of the cut-up that would continue into the 1970s and lead to the collaboration with Malcolm McNeill in the never completed *Ah Puch is Here*" [34]. As documented in McNeill's book *Observed While Falling* (2007), the *Ah Puch is Here* project, eventually more conventionally published as *Ah Pook is Here and Other Texts* (1979), was envisioned as an ambitious full-color graphic experimental novel, using comic-strip framing to further blend media as a "visual narrative where pictures and text interacted in whatever form seemed appropriate" [35]. The basis of the original *The Third Mind* was similarly ambitious, and intended to show the more intensive blended text and art collaborations that Burroughs and Gysin were producing at the time. As Lemaire puts it: "It is not the history of a literary collaboration but rather the complete fusion in a praxis of two subjectivities, two subjectivities that metamorphose into a third; it is from this collusion that a new author emerges, an absent third person, invisible and beyond grasp, decoding the silence" [21]. The dual sense here, of collaboration production, via this liminal space, can similarly be applied to the spaces between media. In a 1966 letter to Gysin, Burroughs considers *The Third Mind* as not only somehow fitting the zeitgeist but also as a defiantly transmedia production including both tape and film experiments: "Actually it is just as well *The Third Mind* publication is delayed since people will be more open to it a year from now. Properly timed it could be a best seller. Its going to need some pruning and of course adding any new tape recorder or film experiments" [26]. This is in keeping with other contemporaneous experimental publications, such as Phyllis Johnson's multimedia magazine *Aspen*, which was published from 1965 to 1971. Each issue came with a range of media, notably nos. 5 + 6 which included the first English publication of Roland Barthes' essay "The Death of the Author" along with a phonograph recording of Burroughs reading "Excerpts from Nova Express (1964)," and even a Super 8 film reel featuring works by Hans Richter and Robert Rauschenberg,

with whom Burroughs would later collaborate [36]. Although Burroughs would arguably retreat to some extent from using the cut-up method in his works of the 1970s and later, the sense of transmedial experimentation continued, and in the 1980s he would begin a transition to visual artist that would incorporate the earlier experimentation into a more visual field.

#### **5. Expanding the field: in time and space**

The transmediality of cut-ups, and their legacy on artists and writers has been profound, yet it is in Burroughs' transition from writer to visual artist that one can begin to see the ramifications of his attitude to media, and one can consider the ways other artists have experimented with and challenged media conventionalities. We have seen some of the key transmedialities that emerged in Burroughs' work and collaborations, yet the works remain for whatever reason predominantly bound to Burroughs as the dominant figure. This is perhaps most notable in relation to the cut-up films, which were directed by Balch but are routinely ascribed to Burroughs. Foucault considers the issue when considering the author function, stating that it, "manifests the appearance of a certain discursive set and indicates the status of this discourse within a society and a culture. It has no legal status, nor is it located in the fiction of the work; rather, it is located in the break that founds a certain discursive construct and its very particular mode of being" [37]. Burroughs' status and apparent particularity has allowed the works he is associated with to become subsumed under the power of his name, much like the categorization of media, and in many ways contrary to the emphasis he so often places on the importance of his collaborators and his resistance to such conventionalities, particularly during the 1960s.

It is important, therefore, to resist this tendency and consider other artists and writers associated with the cut-ups and how their work crossed into transmediality, yet as will become apparent, Burroughs still mysteriously looms large. Poet Harold Norse, whose paratextual cut-up book *Beat Hotel* (1983), composed between 1960 and 1963, and also contained a range of photographs and facsimiles, defies medial categorization. In 1961 Norse exhibited his Cosmograph ink drawings, and *Beat Hotel* contains both a photograph of Norse with his drawings and a Burroughs introduction to the exhibition in which he notes, "What is painting? What is writing? Art? Literature? These words have no meaning now […]. Only the painting and writing that gives us precise maps of some psychic area serves a function at this intersection point of word and image that we call Present Time" [38]. Here Burroughs directly references transmediality as an "intersection point". He explains: "Sit down in a café somewhere drink a coffee read the papers and listen don ́t talk to yourself [...] note what you see and hear as you read what words and look at what picture. These are intersection points. Note these intersection points in the margin of your paper. Listen to what is being said around you and look at what is going on around you" [29]. As Antonio José Bonome García notes, these correspond with the "*umlaut* of semiotic stimuli we may be processing as we try to read a newspaper" [39]. This "*umlaut*" describes, in effect, a transmediality. Burroughs himself makes this clear with reference to another cut-up text, the aptly named *Time*, a 1962 collage collaboration with Gysin that Birmingham calls a "stellar example of Burroughs' attempts to merge the collage technique of art with the cut up technique of literature" [40]. French artist and writer Claude Pelieu and his wife Mary Beach also took to using the cut-up technique, publishing texts such as the former's *With Revolvers Aimed at Finger Bowls* (1967) and the latter's *Electric Banana* (1975),

#### *Cut-Up Transmediality: From Text to Image DOI: http://dx.doi.org/10.5772/intechopen.111441*

which both feature collage covers and introductions by William Burroughs. Angus MacLise, whose 1960s New York happenings were occurring separately from cut-ups, nevertheless involved "the uses of multiple screens, multiple projectors, multiple images, interrelated screen forms and images, film-dance, moving slides, kinetic sculptures, hand-held projectors, balloon screens, video tape and video projections, light and sound experiments", and sound like transmedial phantasmagoria, producing multiple "intersection points" [41]. The "typestracts" of Benedictine monk and artist Dom Sylvester Houédard, who corresponded with Burroughs, present very particular visual poems that utilize the typewriter as an artistic tool, in works where words are, as Guy Brett puts it, "one stream in a convergence of graphic means" [42]. They fail to fit easily into one medium, rather they exist on an intersection point between art and poetry. He was vocal, as Andrew Hunt and Nicola Simpson state, about "the contexts of concrete poetry, opening up its parameters to include mail art, kinetic art, part art and Performance art" [42]. Chinese-born artist Li Yuan-chia moved in the 1960s from painting to kinetic art using magnets to allow viewers to physically interact and move the piece into different forms, to effectively disrupt and "cut" into his work. He was fascinated by what he described as the "cosmic point", stemming in part from Taoism, as a point connecting the "cosmic" and "human" realms, a concept that bears a striking similarity to Burroughs' intersection points [43]. There are further examples, but these texts, writers and artists neatly demonstrate that although a looming figure, Burroughs is not unique in his exploration of transmediality.

Unique to Burroughs, though, is the concrete recurrence of themes and characters across media and works spanning almost 50 years, evident in his visual works and collaborations. His 1989 collaboration with artist Keith Haring titled *The Valley* includes etchings and illustrates sections of Burroughs' 1987 novel *The Western Lands* [44]. His "shotgun" paintings are not all titled, but as Robert Sobieszek notes, "when they are, as in *Escape from Centipede Troughs* and *Are You in Salt?*, their titles come directly from his writings" [45]. Just as many of his characters, notably Clem Snide who appears in *The Soft Machine*, *Exterminator!* (1973) and *Cities of the Red Night* (1981), recur across works, so do numerous phrases and idioms, highlighting how transmediality, even in its definition as storytelling across media, applies so aptly to Burroughs' oeuvre. Sobieszek's book on Burroughs' visual art, *Ports of Entry* (1996), takes a quote from the novel *Port of Saints* (1973), "if you can't see it, you can't say it" to point to Burroughs' wish to be free from what he sees as the straightjacket of language, as "there is no way to effectively portray simultaneous events in writing" [44]. In a 1965 interview with Conrad Knickerbocker Burroughs was asked if he had been successful in "rubbing out the word", i.e. silencing the inner voice, his answer was, "I'm becoming more proficient at it, partly through my work with scrapbooks and translating the connections between words and images" [27]. From the 1980s Burroughs would focus predominantly on visual works, although his transmediality would continue to expand with acting appearances in film and music videos, music collaborations, an opera, and narrating an interactive CD-Rom. Such experimentation and diversity certainly presents what Rosalind Krauss terms the "post-medium condition", yet we are still, it seems, firmly entrenched in media and disciplinary straight-jackets [45]. While Burroughs, and the cut-up project more broadly, through transmediality engaged in far-reaching efforts to transcend reductive and dogmatic conventionalities in artistic production, such efforts were, at least in mainstream transmedial storytelling for example, deemed to have failed. What is clear is that there is much more considered analysis to be done on works and artists that defy simple, media-based, categorization.

*The Intermediality of Contemporary Visual Arts*

### **Author details**

Benjamin J. Heal National Chung Cheng University, Minxiong, Taiwan

\*Address all correspondence to: folbjh@ccu.edu.tw

© 2023 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

### **References**

[1] Wolf W. (Inter)mediality and the study of literature. CLCWeb: Comparative Literature and Culture. 2011;**13**:3. DOI: 10.7771/1481-4374.1789

[2] Schröter J. Discourses and models of Intermediality. CLCWeb: Comparative Literature and Culture. 2011;**13**:3. DOI: 10.7771/1481-4374.1790

[3] Higgins D, Higgins H. Intermedia. Mass: Leonardo. 2001;**34**(1):49-54

[4] Knight D, editor. Interdisciplinary Barthes. Oxford University Press; 2020. p. xviii

[5] McLuhan M. Understanding Media: The Extensions of Man. New York: McGraw-Hill; 1964. p. 9

[6] Leung W. Multimedia Stardom in Hong Kong: Image Performance and Identity. New York: Routledge; 2015

[7] Jenkins H. "Transmedia Storytelling 101" Confessions of an Aca-Fan. 2007. Available from: http:// henryjenkins.org/2007/03/transmedia\_ storytelling\_101.htm

[8] Hancox D. "Transmedia writing: Story worlds and participation at the intersection of practice and theory" Freeman M. Gambarato RR, editors. The Routledge Companion to Transmedia Studies. New York: Routledge; 2019. 165.

[9] Hibbard A. Conversations with William S. Burroughs. Jackson: University of Mississippi Press; 1999. p. 86

[10] Burroughs WS. Naked Lunch: The Restored Text. New York: Harper Perennial; 2010. p. 191

[11] Murphy TS. Wising up the Marks: The Amodern William Burroughs. Berkeley: University of California Press; 1997. p. 204

[12] Punday D. Word dust: William Burroughs's multimedia aesthetic. Mosaic: An Interdisciplinary Critical Journal. 2007;**40**(3):33-49 45

[13] Harris O. "Burroughs Is a Poet Too, Really": The Poetics of Minutes to Go. Edinburgh: The Edinburgh Review; 2005. p. 114

[14] Lydenberg R. Word Cultures: Radical Theory and Practice in William S. Burroughs' Fiction. Urbana: University Of Illinois Press; 1987. p. 44

[15] Burroughs WS, Gysin B. The Exterminator. Anzing: Moloko Print; 2020. p. XVI, XL, XXIV, XXXVI

[16] Genette G. Introduction to the Paratext. New Literary History. 1991;**22**(2):261-272. DOI: 10.2307/469037

[17] Burroughs WS, Gysin B. Battle Instructions. Anzing: Moloko Print; 2020. p. XXXI, XXXI

[18] Birmingham J. Burroughs in Germany and Belgium. Reality Studio. 2006. Available From: https:// realitystudio.org/bibliographic-bunker/ burroughs-in-germany-and-belgium/

[19] Burroughs WS. The Soft Machine: The Restored Text. New York: Grove; 2014. pp. 193-197 80

[20] Ricœur P. The Rule of Metaphor: Multi-Disciplinary Studies of the Creation of Meaning in Language. London: Routledge & Kegan Paul; 1977. p. 43

[21] Burroughs WS, Gysin B. The Third Mind. London: Calder; 1979. p. 27, 45, 46, 14

[22] Bonds R, Youngblood W. Mathew Brady's Civil War. New York: Chartwell Books; 2011. p. 16

[23] Bolton MS. Get off the point: Deconstructing context in the novels of William S. Burroughs. Journal of Narrative Theory. 2010;**40**(1):53-79 53

[24] Burroughs WS. The Ticket That Exploded: The Restored Text. New York: Grove; 2014. p. 230, 31, 11

[25] Benjamin W. "Review of the Mendelssohns' Der Mensch I der Handschrift" Selected Writings. Vol. 2(pt. 1). Cambridge, Mass: Harvard University Press; 1999

[26] Burroughs WS. Rub out the Words: The Letters of William S. Burroughs 1959-1974. London: Penguin Classics; 2013. p. 58

[27] Burroughs WS. Burroughs Live: The Collected Interviews of William S. Burroughs 1960-1997. Vol. 159. Cambridge, Mass: Semiotext(e); 2001. p. 66

[28] Ingold T. Lines: A Brief History. New York: Routledge; 2007. pp. 120-122

[29] Burroughs WS. The literary techniques of Lady Sutton-Smith. Times Literary Supplement. 1964. Available from: https://timesl.it/60118BXqN

[30] MacFadyen I. A trip from here to there. Reality Studio. 2011. Available from: https://realitystudio.org/ scholarship/a-trip-from-here-to-there/

[31] Sargeant J. Naked Lens: Beat Cinema. Vol. 170. London: Creation Books; 1997. p. 171

[32] Hoptman LJ, Gysin B, Geiger J. Brion Gysin: Dream Machine. New York: Merrell; 2010. p. 120

[33] Boon M. The book of methods: Selected writings on the cut up by William S. Burroughs and Brion

Gysin. Marcus Boon. 2020. Available from: https://marcusboon.com/ the-book-of-methods-selectedwritings-on-the-cut-up-by-william-sburroughs-and-brion-gysin/

[34] Birmingham J. The third mind exhibit. Reality Studio. 2008. Available from: https://realitystudio. org/bibliographic-bunker/ the-third-mind-exhibit/

[35] McNeill M. Observed while Falling: Bill Burroughs Ah Pook and me. Seattle WA: Fantagraphics Books; 2007. pp. 19-20

[36] Aspen no 5+6" Unbowed. Available from: https://www.ubu.com/aspen/ aspen5and6/index.html

[37] Foucault M. The Foucault Reader. New York: Pantheon Books; 1984. p. 107

[38] Norse H. Beat Hotel. San Diego: Atticus Press; 1983. p. 54

[39] Bonome García AJ. William S. Burroughs: Pictographic coordinates. Culture of Communication/ Communication of Culture. 2012:1409-1417

[40] Birmingham J. Time. Reality Studio. 2006. Available from: https:// realitystudio.org/bibliographic-bunker/ time/

[41] MacFadyen I. "The Mercury Mirror" Cohen I. into the Mylar Chamber. Lopen: Fulgur Press; 2019. p. 125

[42] Hunt A, Simpson N, Houédard DS. Dom Sylvester Houédard. Vol. 47. London: Richard Saltoun/ Ridinghouse; 2017. p. 9

[43] Yuan-Chia L. Tell me What Is Not Yet Said. London: Institute of International Visual Arts; 2000. pp. 142-143 99

*Cut-Up Transmediality: From Text to Image DOI: http://dx.doi.org/10.5772/intechopen.111441*

[44] Sobieszek RA. Ports of entry: William S. Burroughs and the arts. Los Angeles County Museum of Art. 1996;**145**(102):98

[45] Krauss R. Two moments from the post-medium condition. October. 2006;**116**:55-62

#### **Chapter 3**

## Towards a Semiotics of Visual Music

*Shaleph O'Neill*

#### **Abstract**

'Visual Music' has a long history, much longer than most people realize. Over time, several eminent scientists, musicians, and artists have tried to establish correspondences between sound and vision. Some of these efforts were based on scientific principles, some on genuine synesthetic experiences, while others were more obviously creative aesthetic choices. With the resurgence of interest in this field, the argument presented here is that now is the time to re-evaluate this canon of knowledge, to identify more clearly, and expand the core concepts at its center. A selection of works, by pioneering film makers from the twentieth century, are examined from a semiotically informed perspective, to reevaluate some of the history of Visual Music alongside new ideas from the adjacent fields of science, psychology, and neuroscience. To this end a range of principles/parameters are outlined that arguably constitute the fundamentals of all creative approaches that translate between sound and vision going forward.

**Keywords:** visual music, music visualization, cross-modal processing, synesthesia, sound symbolism, harmony, semiotics

#### **1. Introduction**

'The possibility of communication in terms of symbol, in terms of typography, in terms of language and in terms of music, [and] art, all begin to have potential on this great … medium of the twenty-first century, not today". (John Whitney) [1].

'Visual Music' has a long history, much longer than most people realize. As Maura McDonnell explains [2], the starting point resides well before 1900, as far back perhaps as 570–495 BC, when Pythagoras established the mathematical laws of harmony, which were then applied to both music and color, and so the first colortone correspondence was established. Arguably, the principles of harmony, and their mathematical equivalents, still have much to offer the field today. However, it is widely acknowledged that the correspondence that Pythagoras developed using only four colors, (white, black, red and yellow) leaves much to be desired, both in theoretical and practical terms.

Over time, several eminent scientists, musicians, and artists have tried to establish correspondences between sound and color. Some of these efforts were based on scientific principles, some on genuine synesthetic experiences, while others were more obviously creative aesthetic choices.

For example, Sir Isaac Newton attempted to relate the frequency of light and sound waves in physics [3]; Goethe developed romantic theories of correspondence between color and sound; Castel invented the Occular Harpsichord and Krüger, Guyot, Bishop and Rimington devised other musical/color instruments of a similar nature; Kandinsky's essay "On the Spiritual in Art", sets out his description of synesthesia in relation to making paintings related to music [4]; The artists, Robert Delaunay, Paul Klee and Piet Mondrian which were all concerned at some point with the idea that art and music were deeply linked in some fundamental way [4]; Perhaps most importantly, the twentieth century filmmakers Len Lye, Oskar Fischinger, Mary Ellen Bute, Norman McLaren, Jordan Belson and John and James Whitney, arguably realized 'Visual Music' in its purest form for the first time [5].

There are of course others that have continued, and still continue, to explore the boundaries of visual music (the author being one of them) but to a certain extent it has been forced underground, superseded by developments in technology and culture in our digital age that have occupied the mainstream media. However, it is evident that we are seeing a resurgence in these ideas. Indeed, McDonnell, along with Steve Gibson (in particular), Leon McCarthy, Paul Goodfellow (and others), provide an excellent and thorough going explanation of the history of "Live Visuals" up to the present day in the first part of the recent book of the same name [6]. Unfortunately, there is little space to discuss the depth and breadth of this book here but suffice to say it is exhaustive and timely in describing this old but newly resurgent field as it is right now.

Very simply, we can ostensibly say that there are four phases in the evolution of Visual Music. Phase 1: Early history, which focuses on color tone relations. Phase 2: The evolution of color-tone relations from music into two-dimensional static art. Phase 3: The transition from static to time-based visual art and film. Phase 4: Multimedia experiences, both analogue and digital associate with art-rock happenings, rave culture, and the rise of the VJ. It goes without saying that each phase builds on the experiments and achievements of those that came before. However, it is also clear that fundamental questions remain at the heart of the field in terms of how sound/vision correlations take place. Furthermore, there is still much to be done in terms of establishing some basic principles that enable approaches, developed across continually evolving technical forms of media, to be correlated effectively.

With the resurgence of interest in this field, the argument presented here is that now is the time to re-evaluate this canon of knowledge, to identify more clearly, the core concepts at its center. In short, now is the time to grasp the fundamentals of the correspondence between sound and vision in a way that has previously not been possible. Interdisciplinary approaches to research within twenty-first century academia have the capacity to bring knowledge from a range of other areas to bear on these issues in ways that have not previously been possible. This chapter attempts just that. It brings relevant understanding from, not only art and science as before, but also from psychology, neuroscience, and semiotics to cross reference findings about how we experience sound and vision. The aim is to shift our understanding of Visual Music, away from the arbitrary aesthetics of an experimental art form, towards a kind of language that can be grounded in theory and described more formally.

#### **2. Color/pitch correspondence**

It is almost impossible to discuss Visual Music, without first dealing with the relationship between color and pitch. Indeed, while the main thrust of this chapter

#### *Towards a Semiotics of Visual Music DOI: http://dx.doi.org/10.5772/intechopen.110777*

lies in exploring theoretical input from a range of other different domains beyond this issue, it is necessary to be clear about this point first and foremost, due to the long and complex nature of the history associated with it.

In science, sound and light are often considered as being very different things. Sound is generally understood to be waves that are generated by some external force that acts on a medium, i.e., some initial energy input generates movement in the particles of a gas, a liquid or a solid that behaves like a wave, with a particular frequency (think of a drummer hitting a drum). Light, on the other hand, is generally thought of as particles or photons that sometimes exhibit wave like behavior, sometimes not. Significant debate remains about this dual particle/wave behavior of light [7–9]. Current theory suggests that all particles have this dual behavior. The main difference between the two seems to be one of scale. Sound can be described as energy of a certain frequency making the molecules (larger conglomerations of particles) move at a rate depended on the frequency input that propagates through the medium. Light, on the other hand, is often described as packets of energy at certain frequencies that are embodied as photons (particles in their own right) at a subatomic level moving themselves, even through a vacuum.

What is key is that, in both cases, we have energy, frequency and movement, which can result in wave like behavior. Arguably, this is best described in terms of the electromagnetic spectrum, which positions both sound and light at very different points on an energy scale determined by frequency/wavelength. To the human ear, sound occupies the frequency range of 20 Hz to 20,000 Hz. Whereas light, as visible to the human eye, occupies the frequency range of 4x1014Hz to 7x1014Hz.

Beginning from this set of assumptions allows us to return to the ideas established by Pythagoras in terms of his mathematical principles of harmony. For sound, especially music, the concept of harmony is particularly important. The vibrating string of a guitar or violin, for example, produces sounds of certain frequencies that have been ascribed to certain notes depending on where they are held (i.e., C1, A2, E3 etc.). This is possible because of the mathematical principles of harmonics that divide strings into ratios producing the harmonic series. There are multiple different ways to make these divisions from Pythagorean tuning to 12TET tuning and other more exotic forms. Regardless of their differences, they all stem from the same idea; the division of strings under tension into ratios that result in changes in frequency of sound I.e., changes in pitch.

An excellent example of this is the common place notion of the octave in music, take any fundamental note, say A2, divide its string length in half and you will find the 1st harmonic of that note which happens to double the frequency of the fundamental. This harmonic would be A3 in our case, and thus there is a 1:2 relationship between A2 and A3 in terms of frequency. The peculiar thing about this is that it has led to the introduction of naming conventions that revolve around the repetition of note names between A-G, with the appropriate sharps, flats and cents thrown in depending on the tuning system in play. This naming convention (specifically the western 12TET piano based system) has become engrained in musical nomenclature and notation and tends to obfuscate the significant differences in frequencies between notes. Technically A2 and A3 are completely different pitches but they are both named A. This repeats across a very large frequency range (10 octaves from C0 to C10).

This is not the case in terms of light, where the range of frequencies, although much higher, is much narrower. This would suggest that octaves within the visible spectrum, would seem improbable, at least in terms of equivalence to the range within audible sound. However, recent papers exploring the use of lasers have

shown that frequency doubling (essentially, the creation of octaves) in light is more than possible, even if it does stray into U.V. territory [10]. This point is made here to underline the similarities between sound and light in terms of frequency and to open the way for the development of a primary relationship between sound and light that is based on the fundamental principles of harmony, mathematics, and physics. Essentially, if we agree that all these principles are in play, then it is only a small step to see that the light equivalence of sound is attainable through harmonic principles of transposition. This enables us to establish a relationship between any musical note on any scale (if we know it's frequency) and its equivalent color based on the notion that the corresponding frequency will be 40 octaves above in terms of frequency. No doubt debates about the properties of light and sound make this problematic on one level (i.e., waves versus particles as discussed previously). However, by overlooking these issues in physics (which are by no means resolved) and concentrating on mathematics, frequency and harmonics. We can find a way to translate between the perceptual experiences of two essentially different media.

This has been problematic for a very long time with many scientists, artists and musicians establishing their own (competing) sound/color correspondences, often in very arbitrary ways. Debates about the color of A or the color of C and where the correlation begins have plagued the field for centuries. This frequency based harmonic approach allows us to establish a very simple mechanism of translation from one medium to another. An approach that can be replicated across other parameters as we move beyond those debates about color and pitch and on to consider more interesting debates about what else needs to be established within this re-emergent field.

#### **3. Synesthesia and cross-modal processing**

Early forms of Visual Music are inevitably connected to the concept of Synesthesia. Indeed, issues around color/pitch correspondence most likely derive directly from this concept.

In medical terms, Synesthesia is a neurological condition in which two or more senses are involuntarily and consistently linked together, leading to unexpected sensory experiences. The artist, Wassily Kandinsky, wrote extensively about synesthesia near the start of the twentieth century, which has had an enduring effect of the development of Visual Music as a whole. While it remains debatable as to whether he experienced such a thing himself, his understanding of the subject had a profound effect on his approach to painting. Indeed, the very idea of synesthesia and the correspondence between senses has been a theme for many artists over the years, and in some cases, there are clear examples of people with synesthesia making art about their experiences [11].

For Kandinsky, as for many artists in the first half of the twentieth century, it was the relationship between art and music that inspired him most. His paintings, particularly of his later period, are well known as increasingly abstract improvisational representations of music. The boundary between these two areas of creative endeavor have been crossed many times, both formally and informally, in terms of exploring the territory of correspondence (think simply of how many great rock musicians came out of art schools from the 1960s onwards).

Arguably, creative people seem to a have a propensity to want to explore the boundaries between senses, as to whether this is a feature of their psychological make up is up for debate, but the concept of cross-modal processing is well understood

#### *Towards a Semiotics of Visual Music DOI: http://dx.doi.org/10.5772/intechopen.110777*

within the domain of psychology. Cross-modal processing refers to the way in which the human brain makes connections between different perceptual inputs allowing us to integrate and co-ordinate our experiences of sight, sound, touch and taste. It enables us to perceive the world in a cohesive and meaningful way, rather than as a series of isolated sensory experiences.

There is much debate within the scientific community about the extent to which synesthesia represents a special case of normal cross-modal processing, or whether it is a distinct and separate phenomenon. Some researchers have argued that synesthesia results when the brain fails to properly control cross-modal connections that are commonplace, others suggest that it may be caused by specific differences in brain function or organization.

Some recent studies have shown evidence of increased activation in brain areas associated with cross-modal processing in people with synesthesia, suggesting a 'special case' of normal processing. Other, structural neuroimaging studies, have found evidence of differences in the connections between brain regions in people with synesthesia, suggesting a hard-wired organizational predisposition is necessary for the condition to emerge [12]. Interestingly, Ramachandran and Hubbard [13–15] have proven without doubt that synesthesia is indeed a perceptual effect and not a cognitive or memory driven experience, as was once thought.

Research of this type has used brain-imaging techniques to establish the locations of stimulation when subjects are having synesthetic experiences. Findings have established that synesthesia is likely caused by cross activation in neurons in adjacent areas of the brain, which are responsible for slightly different perceptual or cognitive activities.

Ramachandran et al. have proposed that this might occur because of genetic mutation whereby the dysfunctional pruning of neural linkages during brain development leaves connections that normally would not be there. This in turn is reinforced by learned experience, which cements connections in the brain between normally unconnected areas. This opens the possibility of a resolution to the debate between the 'unity of the senses' thesis and the opposing 'modular' thesis camps [3, 16].

Ramachandran and Hubbard go on to suggest that this mutation could be quite common and spread throughout all brains in a patchy way across both lower perceptual and higher cognitive functions. They also consider how this may play a role in understanding metaphor, creativity, and the development of language. Additionally, they also suggest that this distributed patchy 'cross wiring' of the brain is why we see more examples of synesthesia in artists, poets, and other creative people.

So, why is all this neuroscience relevant to us in terms of developing an approach to understanding Visual Music? Well, fundamentally it lets us move on from many of the debates that have been at the heart of Visual Music for years. Questions about whether synesthesia is real or not, how it occurs and whether it is at the heart of the creative endeavors that explore the boundaries between sound and vision, become obsolete. The science essentially answers those questions for us. We can take it from the experts in those fields that cross-modal processing is a normal aspect of perceptual sense making and that synesthesia, although a special case for individuals that experience it, is derived from those normal cross-modal processing brain mechanisms that we all have.

This is important because for the first time it shows that developing correspondences between these seemingly different domains of knowledge based on two seemingly different perceptual components of existence is a completely natural part of how we make sense of the world around us. The very fact that our brains are built around these cross-modal perceptual mechanisms, allows us to take the next step in terms of understanding what theses mechanisms are. If they are also implicated in terms of understanding language, metaphor and creativity then attempting to also think of Visual Music in these terms makes even more sense (E.g., lexicons, grammar, translations, transpositions). Furthermore, it opens the opportunity for deeper interdisciplinary investigation in terms of developing the parameters of cross-modal translations between mediums. By drawing on additional ideas from related fields.

#### **4. Sound symbolism and its implications for visual music**

Sound symbolism, also known as phonosemantics, is a field of research within psychology that is exploring the way in which certain sounds seem to carry meaning beyond their primary role as simple (sometimes arbitrary) units of sound in a particular language [17]. Specifically, researchers are investigating the ways in which the physical properties of the sounds themselves, such as their duration, pitch, or intensity carry that meaning. As such it is an investigation of the formal properties, or parameters of sounds used in speech rather than the meaning associated with particular words (which Saussure characterized as arbitrary) [18].

The obvious form of sound symbolism that we would all recognize is onomatopoeia, i.e., words that mimic the sounds they represent (such as "hiss" or "buzz"). These types of words are found in all languages and are thought to play a key role in the development of language in children. More interestingly, a significant thread of research on sound symbolism has investigated the way in which certain sounds are consistently associated with particular meanings across different languages and cultures.

Arguably, the most well-known example of this is the Bouba/Kiki effect, first tested in the 1940s [19]. The experiment investigated the associations participants make between these two words and two different shapes, presented to them. A round blob shape and a spikey asymmetrical star type shape. Findings showed that more-oftenthan-not, participants associated the rounded shape with the word Bouba and the spikey shape with the word Kiki. The implication being that the sound of the word and the quality of the shape have some correspondence at a deep psychological level [20].

This experiment has been repeated many times. A recent version that explored the effect over 25 different languages (9 language families and 10 writing systems) found strong evidence of the effect, essentially establishing once and for all that: "The Bouba/Kiki effect is rooted in cross-modal correspondence between aspects of the voice and visual shape … the strongest demonstration to date that the Bouba/Kiki effect is robust across cultures and writing systems" [21]. The importance of these findings cannot be overstated in terms of verifying that our perceptions of both sound and vision are deeply interconnected at some neurological level. Moreover, other related studies have found that additional relationships exist in a similar way.

The words 'Mil' & 'Mal' have been shown to correspond with different sized (small, big) circles and the words 'Wee' & 'Woo', similarly seem to correspond to changes in aspect ratio such as (tall/thin) or (short/fat) ellipses [20]. A recent large study by Johansson et al. [22] looked at sound symbolism across a multitude of lexemes and languages pointing towards the embodied/experiential basis for language formation. Macro-concepts such as hardness, softness, smallness, roundness and flatness were clearly identified as cross-modal, suggesting that certain sound types are correlated to these particular embodied (physical and visual) experiences. Interestingly, Ramachandran and Hubbard have also shown that "associations

#### *Towards a Semiotics of Visual Music DOI: http://dx.doi.org/10.5772/intechopen.110777*

between shapes and sounds are absent in individuals with damage to the angular gyros, suggesting that this is a robust neuropsychological phenomenon" [13–15].

The upshot of all this essentially means that we are, at least to some degree, hardwired to associate certain sounds with specific kinds of visible phenomenon and that our language, is in part at least, built upon that principle, perhaps explaining why we can describe experiences of both light and sound as 'bright' or 'dull'.

Arguably, this is a 'Rosetta Stone' moment in the symbiotic relationship between sound and vision and potentially has far reaching implications for those working in the expanding field of Visual Music. Not only do the outcomes of these studies help make sense of the kind of underlying synesthetic principles that have underpinned Visual Music since Kandinsky's first musings on the subject. They also offer a much more robust and scientifically grounded approach to mapping relationships between sound and vision, which up until now have arguably been quite arbitrary. We will see this when we examine some key examples from the Visual Music film cannon in Section 6.

#### **5. Psychoacoustics and sound symbol oriented semantics**

Psychoacoustics, a branch of psychophysics, is essentially the study of auditory perception, which includes both the sensory and physical aspects of the phenomenon [23]. Its practical implementation has found a home in product and industrial design, as Sound Quality Evaluation (SQE), where psychoacoustic principles are used to measure and evaluate the effects of sounds generated by products and machines on the psychology of human beings, usually in terms of safety. Over the years SQE has established several standardized parameters of sound description that provide a measure of sound quality using semantic differentials and Likert scales [24]. These parameters include:


16 Hz and particularly at a round 4 Hz. As time increases further, fluctuations become more like echoes or delayed repetitions of a sound.

Susini et al. [25], discuss the relationship between psychoacoustic descriptors and auditory attributes, focusing on the way in which sematic differentials across multidimensional bipolar scales can cover a range of overlapping terminology (e.g. sharpdull, pure-rich, cold-warm, colorless-colorful), to capture the often, complex nature of the sounds or soundscapes being investigated.

It is interesting to note that these bipolar scales bear a strong resemblance to the kinds of bipolar relations explored in sound symbolism experiments. While the standardized components of psychoacoustics may not relate directly, the lower-level specifics of the semantic differentials used to make evaluations have striking similarities. For sound symbolism the exploration is in relating the sounds of words to experiential equivalents (e.g. Bouba/Kiki) which tend to differentiate between poles. For SQE its about describing sounds within polarised parameters described by words. What is interesting is that in both cases the relationship between words and parameters is in play and it does not take a great deal of effort to theorize that there may be some significant cross overs, where the parameters being identified are applicable to both sound and vision at the same time.

This is important for the purposes of developing our theoretical understanding of Visual Music because, thinking carefully about the wording of semantic differentials that help frame descriptions of sounds, with reference to what we know about sound symbolism, could help to establish a set of parameters that enable the direct translation to a visual counterpart. In other words, it is entirely possible to start thinking about piecing together a set of fundamental principles of Visual Music that map the parameters of sound to those of vision. This is underlined by the fact that key concepts in psychoacoustics, such as sharpness or roughness are words that operate metaphorically, being that they are 'already borrowed' from the sensory domain of touch and operate within the realm of sound and music and no doubt have a correlate within the visual domain.

The missing piece in all of this is an investigation of the parameters that are in actually in play within the field of Visual Music. It is to this that we now turn, taking these theoretical ideas with us.

#### **6. A brief semiotically informed exploration of visual music**

There is not the time, or the space, here to perform a full-blown semiotic analysis of the entire canon of Visual Music. Indeed, it is only possible to discuss a selection of works by pioneering film makers from the twentieth century, from a semiotically informed perspective. However, while what is presented here is not an exhaustive review of seminal works, it does provide an overview of the early approaches to the making of such works in terms of the theory we have reviewed so far. In doing so the evaluation makes efforts to try and identify where these early experimental film makers were clearly and successfully connecting sound and vision together in terms of trying to establish the structural parameters of this cross-modal medium. In many cases this was an arbitrary approximation or interpretative process driven by the personal aesthetics of the film makers. In others there are much more direct mappings and processes that connect sonic and visual elements together in a more denotative manner where sound and vision truly begin to signify one another.

*Towards a Semiotics of Visual Music DOI: http://dx.doi.org/10.5772/intechopen.110777*

Moreover, the criticisms levelled at the film makers in this section is in no way meant to be derisory or devaluing. On the contrary every single work produced by these pioneers is held in great regard by the community and by the author. However, it is in the nature of applying a critical lens to these works that a separation between 'successful' and 'less successful' films (for want of better expressions) emerges, in terms of how close these early works get to really grappling with a visual language that could claim to be an equivalent to music.

Such an analysis, though cursory, and which belies the much greater need for a systematic semiotic analysis of the whole canon of these films, moves us much closer to a better theoretical understanding of Visual Music, not just in terms of film but for newer interactive digital forms, that are wrestling with the same issues today.

#### **6.1 Len Lye**

A review of the early works by film maker Len Lye, such as, "Kaleidoscope", 1935 [26] and "A Color Box", 1937 [27], reveals a very experimental approach to film making, which is rich, beautiful, and mesmeric. These films consist mainly of montages of shapes, squiggles and blobs of paint and ink either painted or printed directly onto filmstock. The very painterly marks bear no direct relation to the music played alongside them, which tends to shift between rhythmic drumming and jazz.

There is a vague sense of timing that is more coincidental than carefully planned. The visual elements are rough, splodgy and fluid but vibrant in terms of color and the sense of movement, which is lively. From a semiotic point of view we could only really say that there is a vague correspondence between the visual and musical elements in the film. There is no real precision that could denotatively align a particular shape, color or movement to a particular beat, note or instrument. The visual elements are simply loose approximations or interpretations of the accompanying music that connote a sense of rhythm in general. There is a very informal messy quality to them and Lye's style has clearly had a significant impact on rock music culture, through the happenings of the 1960s to prog rock music visualization of the 1970s and on into the punk rock and grunge music videos of the 1980s and 90s. It is the roughness and textural qualities of the works that really stand out and the connect perhaps theoretically with SQE parameters of roughness. It would be interesting to take more time for example to consider both the visual and sonic content of the films from the point of view of establishing where they sit in terms of parameters such as: rough-smooth, lively-calm, rich-impoverished, colorful-colorless.

#### **6.2 Oskar Fischinger**

By contrast Oskar Fischingers early visual music films, such as "An Optical Poem", 1938 [28] were much more structured. Perhaps because he was working with symphonic classical music or perhaps because intentionally at the outset, he was trying to find a stronger visual correlation to the musical element. Indeed, at the start of An Optical Poem there is a statement: *"To most, music suggests definite mental images of form and color. The picture you are about to see is a novel scientific experiment – Its object is to convey these mental images in visual form".* So, from that we can take that there is a very clear intention present in Fischingers work to represent the sensation of music visually.

The film itself is full of simple visual forms (mostly circles, and some other shapes) carefully synchronized to the timing of the music, for example it is clear from the first few bars that there is a strong intentional link between the string section and the

color, size and position of the circles that represent it semiotically. However, in those first few bars there is nothing to represent the horn section (this seems to appear later). Importantly from semiotic point of view, there is evidence of an intentional link between dark-colored circles representing sounds with low pitches and light-colored circles representing higher pitches. This mapping is not precise or consistent in terms of specific pitches, colors or shades but there is a definite general rule at work here.

As already mentioned in relation to the string section there is an attempt to connect shape and movement to the phrases played by that part of the orchestra, but it is not consistent and at times it gets difficult to connect specific shapes with specific instruments. It all feels a bit arbitrary but there are clear attempts at relating shape and color to a particular type of instrument or timbre.

On screen position also suggests a visual relationship with pitch. Visual elements nearer the top of the screen seem to represent higher pitched sounds and elements lower on the screen seem to represent lower pitches. This works to reinforce the connection between light/dark pitch relationships. Additionally, there is a strong sense of a three-dimensional space or field of view within the work where different sizes of shape come towards the camera or recede away from it. This appears to be connected to volume where larger shapes are louder sounds and smaller shapes are sounds fading away, receding into the background and disappearing.

Semiotically, this feels like an intentional mapping of parameters, but it is not consistent and does not always work as expected. What is most significant about Fischinger's works is that they exhibit some of the first real attempts to establish parameters that visual elements might share with sound. Most notably attempts to correlate visual size and audio volume; dark/light shades of color with lower/higher pitched instruments or different types of instruments (e.g., strings vs. trumpets); screen position of visual elements with note pitches.

Unfortunately, from a semiotic point of view, where we are looking for clear denotative correspondences between visual signs of music Fischinger's films fall slightly short, largely because he is working with Symphonic music which is so vast in its complexity that it is almost impossible, given his chosen medium of animated paper cutouts, to achieve the kind of precision necessary to make such precise connections. His films are, however, remarkable in terms of how far they get down that road, paving the way for many who come after him.

#### **6.3 Mary Ellen Bute**

Contemporary to Lye and Fischinger, Mary Ellen Bute was also making films that were experimenting with the relationship between music and a visual equivalent. Her early films such as "Rhythm In Light", 1934 [29], "Synchromy No 2", 1935 [30] and "Parabola", 1937 [31] are shot in black and white, and so immediately there is no color element to relate to sound. However, Bute's take on the relationship between sound and vision seems more akin to interpretive dance than a search for a direct correlation between sonic and visual elements.

These early works are built around abstract montages of overlaid imagery that move in and out of focus across the screen and are vaguely coordinated with the tempo of the music. There are no strong relationships formed between the visual elements and the sonic content. Indeed, from a semiotic viewpoint is seems as if the experimental dreamlike imagery, that plays with chiaroscuro lighting of unusual physical objects, seems to slip in and out of view as an accompaniment to the music, that echoes, the general feel of the music rather than attempts to signify it. At times

#### *Towards a Semiotics of Visual Music DOI: http://dx.doi.org/10.5772/intechopen.110777*

the imagery is beautiful, but it is so abstract and referent to the 'physical' content that it is hard to really think of these as visual music films.

This is in complete contrast to some of her later work, particularly the works she directed with Norman McLaren as the lead animator. Works such as "Spook Sport", 1939 [32] and "Tarantella", 1940 [33] are shot in color and immediately there is more vibrancy and connection to music in some ways. However, it is the sinuous line and, in "Spook Sport" particularly, that the semiotic content begins to connect more strongly.

At its outset, the film clearly identifies the 'characters' within the movie, and these characters (symbolically animated representations of bats and ghosts) go on to dance across the screen space to great effect. Unfortunately, in terms of visual music the outcome is again more like interpretive dance than a visual version of music. The semiotic characterization linking more to traditional film and theatre in terms of developing a narrative, rather than connecting with the sound directly. However, McLaren's input really does show in both films. "Tarantella" is the more abstract of the two, drawing heavily on Russian Modernist forms and colors to drive a quasinarrative forward to some degree, but the drawing in "Spook Sport" connects more directly to McLaren's own work, which takes the semiotic mapping of sound and vision to a much deeper level.

#### **6.4 Norman McLaren**

In Mclaren's work we get a much more direct relationship between sound and vision but arguably it is not as direct as it might have been. McLaren took the ingenious step of using the sound portion of film stock to create his own 'sound effects' by drawing/painting sequences of marks directly on to it in order to use it as a sound generating instrument alongside the images he drew on the visible element of the film. In such a way he was able to precisely control the sounds that were made, when they were made and ensure that the visual and sonic elements were carefully synchronized when the film played. This is a major technological breakthrough in terms of realizing the semiotic potential of the medium to embody a genuinely interconnected visual music for the first time.

Before McLaren, music either came before or after the concerns of film making. McLaren revolutionizes the approach by making sound and vision with the same set of tools, thus bringing the two elements closer than ever before. However, from a semiotic point of view, what is interesting is that while he drew a series of lines or blobs on the sound portion of the film to control pitch, volume and softness [34]. He did not draw the same thing on the visual side of the screen. Indeed, with more screen space to play with he drew more complex visual forms that fold and unfold, explode, and reform on screen in time with his abstract soundtrack. The synchronization is pinpoint precise, but questions remain from a semiotic point of view as to whether the visual elements directly represent the sounds or if they are once again arbitrary representations of things that have sounds, rather than of sounds. If McLaren had painted the same thing on screen as he did on the sound portion of the film there would have been a direct relationship between the two, but this in itself brings significant limitations and is perhaps visually much less interesting than what McLaren actually drew.

"Dots", 1940 [35] is an excellent example of this. It is an incredibly simple little film but there is a very strong relationship between the visual and the sonic elements thanks to McLarens special technique. The size of the dots seem to represent aspects of pitch. Small dots have higher pitched sharper sounds and larger blobby dots having lower softer sounds. The same is true of the relationship between size and volume,

again smaller shapes seem to have lower volume and larger shapes seem to be louder. This approach to parameter mapping is reminiscent of Fischingers work (which McLaren was more than aware of), and his work seems to reinforce these ideas apart from the fact the position on screen does not correlate with changes in pitch as Fischinger's tended to do (as seen in musical notation for instance).

Interestingly, Mclaren's later works seem to move away from the tight correspondence between soundtrack and visual material. Works such as "Lines Horizontal", 1960 [36] for instance, where the animation is a visual interpretation of music by Pete Seeger. The horizontal black lines seem to bear absolutely no direct relationship to the guitar or the flute. They do move up and down the screen, multiplying as they go and it does feel like there is a timing relationship to changes in the music, such as the introduction or removal of instruments in the mix. However, the only thing on screen are black lines and occasional background color changes. This is very different to 'Dot' for example.

Likewise, "Synchromy", 1971 [37] is also quite different. While it does have a carefully orchestrated sense of timing between visuals and soundtrack, it's not as direct a relationship as his earlier work. Visual elements arrive on screen and move in relation to complex rhythm patterns. These change all the time and sometimes there seems to be several different moving strips in different horizontal screen positions that correspond to different instruments that are being played. The multiple rhythms are complex, and it seems that this is where the synchronization is aimed but it's not entirely clear as to which 'track' of rectangles relates to which sound element. It is also clear there are size and shape patterns at work that imply a relationship to the sound but as to whether they relate to beat, volume, or timbre for instance is difficult to establish. This perhaps is the difficulty of working with an external sound source rather than working with the sound portion of the film, which he expertly did in his earlier films.

#### **6.5 John and James Whitney**

The work of John and James Whitney is legendary in the Visual Music community, particularly John for his contribution towards the development of early computer graphics. Again, their contribution to the field began early, in the 1940s. They worked with bespoke film and audio production equipment, customized from military ballistics systems, which enabled them to create complex films. Like McLaren, both the audio and visual elements were carefully synchronized by scoring and animating directly onto film. This resulted in carefully constructed sequences of color and shape compositions based on modernist composition theories using cut out masks that cover and reveal light directly rather than reflecting it (like the cutouts of Fischinger do). "Five Film Exercises", 1943–1944 [38–40], winner of the Brussels Experimental Film competition in 1949, are perhaps the best examples of this way of working. While the timing of the shifts in shape and the strange experimental noises are clearly coordinated, there is little that really connects the shapes or colors directly to the noises being made in a cohesive, nonarbitrary way. The visuals tend to operate alongside the sounds. For instance, size and shape are not consistently correlated with volume or timbre and color. The films are extraordinary in terms of a quality in both sound and vision that seems quite fluid but not linked in any stable way.

As well as working together the brothers produced films separately. John really pushed the boundaries of the medium, moving inexorably towards computers over time. "Permutations", 1968 [41] for example predates his experiments with digital

#### *Towards a Semiotics of Visual Music DOI: http://dx.doi.org/10.5772/intechopen.110777*

computers, showing off the levels of complexity he was able to achieve by pushing his analogue ballistics system in combination with hand drawing series of dots on frames of film. The sound and the visuals are not directly connected in any way, being that the soundtrack is composed of rhythmical drumming and the visuals are visual representations of mathematical figures based around 'rose' equations. The swirling visuals are utterly mesmerizing and the synchronization between them and the sound is strong despite the lack of direct connection. The resultant output is reminiscent of Mary Ellen Bute's work in that it feels like the dots on screen are dancing and shimmering along with the music but not directly tied to it in a strong denotative way.

Likewise, "Arabesque", 1975 [42], operates in a similar vein. This does demonstrate Whitney's early computer programming experiments, and as a result is an order of magnitude more complex than "Permutations". Once again though, the visuals coincide with the music, which is Indian tabla/sitar music, rather than being directly derived from it or used to produce it.

Representing some of the first attempts at the precision manipulation of on-screen shapes using mathematical formula, it is a masterpiece of sequential programming, that clearly demonstrates the power of the computer in coordinating such complex movements. However, and this is controversial, it's not the strongest in terms of demonstrating clear thinking about the relationship between the music and the related visual structure. One clearly informs the other but inevitably the programming seems to take precedent over the music, presumably because it simply took so long to produce frame by frame, making it difficult to connect directly with the soundtrack. All the computer-generated films are like this. One wonders, if it were possible to revisit the mathematical underpinnings of the works, as to whether an alternative soundtrack could be generated by slaving the equations to some form of sound synthesis alongside the visuals using today's equipment.

Criticism aside, the contribution to the field made in these films is a monumental step forward, which perhaps unwittingly via the math, takes us closer to the concepts of harmony outlined right at the start of this chapter. No doubt there are mathematical equations at work that demonstrate complex ratios and relationships (e.g., rose figures, arabesques, polar co-ordinates etc.), it's just that they are not systematically linked to the principles of the harmonic sequence or frequencies of pitch that come directly from the music, but they could be in future.

#### **7. Discussion of analysis**

This brief review of some key seminal works from the Visual Music Canon reveals several important things. Firstly, it shows the way in which some of the early pioneers of film and animation were grappling with the benefits and limitations of the media. It is obvious that the time-based nature of film offered a temporal space for the visual, which was equivalent to music, for the first time. The early films seem to be more like visual interpretations of the music or visual dances in response rather than direct correlations (Bute's early, and Whitney's later, work for example).

In contrast, the works of Fischinger and McLaren (in particular) hint at technical solutions to timing that clearly link the visual and sound elements together in a precise way. This aspect of timing is the first clear parameter that both elements share. Establishing a tight connection between the two seems to be fundamental to being able to translate between sound and vision or to engage with both elements as if they were one.

The second parameter that begins to emerge from this analysis is that of the relationship between visual size and audio volume. To an extent this appears quite strong in Fischinger's work but is again strongest in McLaren's. His technique of drawing the sound on the film alongside the visual elements meant he was able to precisely time the changes in both. So not only do we see a synchronization of duration of noise and duration of shape, we also so see a synchronization between volume of noise and size of shape. So now we have two parameters working together in both the sonic and visual dimensions. This is also consistent with theory from sound symbolism such as 'Mil & Mal' in terms of size differences and 'Wee & Woo', changes to aspect ratio, which play out in animation as squash and stretch.

Additionally, there is a third aspect of correspondence that is built into this in terms of the type of sound and type of shape. The kinds of noises that McLaren was able to make often varied between high frequency sounds and low frequency sounds and the shapes he associated with these two sounds were often simple circles (high) or more blobby type shapes (low), the simple shapes being smaller and the blobby ones being bigger.

It cannot be overlooked that the pitch of the sound McLaren created was determined by the frequency and thickness of the lines that McLaren painted onto the celluloid. Many thin lines created high frequency sounds and fewer rounder lines created softer lower frequency sounds (Wee and Woo again). It is of course logical to follow that principle onscreen. Had McLaren painted the same configuration of lines on the visual part of the film as he had in the sound portion, there would arguably have been a one-to-one relationship between the two.

At this point we are beginning to see multiple correlations overlapping with one another and integrating into compelling Visual Music experiences. Again, the relationship between sound and shape connects with theory from sound symbolism. In this case specifically the Bouba/Kiki effect, where lower frequency sounds connect most strongly with rounder shapes and higher frequency sounds connect with thinner spikier shapes. So, quality of sound or 'timbre' potentially has a correlate within the visual component of shape. That quality of sound might be considered in relation to its frequency signature (its combination of fundamentals and harmonics) in terms of where it sits on the frequency spectrum or musical scale.

The quality of the related shape then seems to operate on a scale from soft and round to sharp and spikey. However, there are other visual correlations that could potentially come into play here. Two spring to mind. Dark vs. light and color mapping. Color mapping has already been dealt at the beginning of this chapter. But it is worth noting that in most of these examples, while color is an important element in most of them, few of the film makers, with the exception of Fischinger, seem to consider developing the relationship between color and pitch in a denotative way. Most of them seem to prefer arbitrary or loose connections, perhaps because strong connections are problematic and complex to establish in the medium of film.

The Dark to light correlation is much more straight forward. It is commonplace to describe different tones and feelings within musical passages as either dark or light and often this difference in darkness or lightness is a feature of the pitch of the notes being played. Again, higher frequency notes tend towards the lighter end of the scale and lower frequency notes tend towards the darker. Some would call this darkness vs. brightness and it would be easy to associate trumpets with bright sounds while cellos would be dark. To a degree this could be related to timbre but in fact it is likely more strongly a feature of the pitch range of those instruments and where they sit in relation to the bass or treble clef in orchestral music.

#### *Towards a Semiotics of Visual Music DOI: http://dx.doi.org/10.5772/intechopen.110777*

Examples of this 'dark to light' correspondence between sound and vision, is to a certain extent evident in Fischinger's work. Perhaps because he worked with orchestrated music but most probably because he was sensitive to those sonic differences and was genuinely attempting to find a visual correlate to it. It is most obvious in the difference between the brighter (whiter and smaller) circles that occupy the upper part of the screen and the darker (bigger) colored circles that occupy the lower part of the screen in his work.

This brings into focus another parameter at play in relating the sonic to the visual that of the screen space itself. The placement of the visual elements in Fischinger's screen space demonstrates a direct correlation to the musical stave, i.e. the positioning of notes on a page in terms of pitch to relative height, the higher the note the higher the position on the stave. It is of course obvious that this is derived directly from musical notation. However, it is interesting to consider that this correlation on the scale of pitch (frequency) to height again is probably derived from an embodied experience related to sound symbolism regardless of its conventional nature. It seems logical then, that vertical positioning on the screen should correspond to pitch to some degree. The thing that is unexplored though is the relationship between the horizontal axis and the sonic. Certainly, in the making of these early films there would have been no such thing as stereo, and the concept of surround sound would have been unimaginable. However, it stands to reason, following the logic developed so far, that a relationship between the stereo field of sound and the stereo field of view is a reasonable one to suppose. While none of the films could exhibit this, projecting forwards, it is easy to imagine a direct mapping of the visual screen space in terms of stereo left and right.

#### **8. Conclusions**

This chapter has attempted to reevaluate some of the history of Visual Music alongside new ideas from the adjacent fields of science, psychology and neuroscience. These new ideas offer the opportunity to build better theoretical foundations for Visual Music and all other creative approaches that translate between sound and vision. To this end a range of principles/parameters are outlined here that need further exploration (see **Table 1**).

#### **8.1 Pitch and color**

The concept of harmony and the harmonic (and subharmonic) series provide fundamental building blocks for both music and visualization from a scientific and mathematical perspective. This has far reaching implications for the visual space on several levels. Applying the principles of harmony to the issue of color mapping results in the concept of sound and color relations operating via the mechanism of octaves. Light is approximately 40 octaves above sound and thus equivalent correspondences can be mapped between sounds/music and color based on frequency.

#### **8.2 Pitch and brightness/darkness value**

One problem of this harmonic approach is that there are approximately 10 octaves within audible sound and only one within the range of visible light, thus C0 and C2, or C10 would be equated to the same color. This is logical but does not consider differences


#### **Table 1.**

*Proposed fundamental visual/music parameters.*

in how we experience low and high musical notes. It is common practice within the sphere of music to describe sounds with low frequencies (bass notes, and the lower octaves) as 'dark' and those with higher frequencies (treble and higher octaves) as 'bright'. Phrases like 'low' and 'high' end are also applied to this phenomenon. Thus, two complementary parameters arise that can be applied to sounds in terms of visual correspondence. The first, dark to light, can be made manifest in color correspondence through the mechanism of the HSV color space, that provides an axis of brightness (value, V) alongside hue (H) and saturation (S). Thus, the octaves of sound can be mapped to variations in brightness or saturation, alongside the corresponding color frequency of hue. This provides a very satisfactory mechanism for taking the experiential aspect of sound into account alongside the harmonic. The second parameter is spatial (see Section 8.5).

#### **8.3 Volume and size**

It is clear in relation to our investigation of Visual Music examples that there is some evidence of both size and shape being used in a way that is consistent with ideas emerging in the study of sound symbolism. It is also clear that the most logical mapping of the size of visual element is related to volume. Changes in the volume of sound from quiet to loud correspond very clearly with our experiences of smaller and larger sized things. As a general rule, small things (including animals, e.g. rodents, insects etc.) make quieter (usually higher pitched) noises. Larger things (including animals such as tigers, bears, elephants etc.) tend to make louder, lower pitched noises. One might also equate the two in terms of closeness. Close noises are louder than the same noises further away and close objects appear larger than the same object further away. So, it follows that our natural mapping of loudness corresponds with the visual equivalent of size. Thus, a triangulated relationship emerges between loudness of sound, size of shape and closeness (or depth of field on screen).

#### **8.4 Timbre and shape**

Shape itself inherits this aspect of size but it also has other features that correspond to a different aspect of sound. The Bouba/Kiki experiment, as already mentioned,

#### *Towards a Semiotics of Visual Music DOI: http://dx.doi.org/10.5772/intechopen.110777*

reveals the relationship between round shapes-softer words and spikey shapes-sharper words. To a degree this must be a feature of frequency (pitch) in that rounder, softer, (darker) sounds have more low-end frequency and higher, sharper (brighter) sounds have more high-end frequency. Arguably, it is no accident that the concepts of 'flat' and 'sharp' in music also correspond with shifts in pitch that change in frequency accordingly. It follows then that shape relates to this element of frequency but arguably it is more complex than this. Timbre or type of noise is a consequence of the variability in the harmonic content of the instrument not just the pitch of the note. Thus, a C2 played on a guitar sounds very different to the same note played on a trumpet. Each of these instruments has a different harmonic signature, which are variations in the levels of the harmonics that develop from the fundamentals of the note being played attributed to the physical nature of the instrument being played. Pure sine waves from a tone generator have the cleanest harmonic structure and it is no coincidence that sine waves are also mathematically equated to the generation of circles. As a baseline relationship then, pure tones seem to correlate directly with circle shapes and changes in volume correspond to changes in size. More complex timbres will have more complex shapes, but a great deal of work needs to be done to establish how the harmonics of sound translates to an equivalent shape in terms of our perception. Potentially the harmonics of geometry has a lot to offer here and it does make sense that the Bouba/Kiki effect will play a part in this as our understanding develops.

#### **8.5 Sonic space and screen space**

While there are surround sound systems and 3D visual experiences that are prevalent today, most of our media experiences are still consumed in terms of 2D screens and stereo sound. Therefore, as a starting point it makes a great deal of sense to establish a baseline relationship between the stereo space of sound and the 2D visual space of the screen. Thus, the pan position of the stereo space and X axis of the screen line up across the horizontal. The Y axis on the other hand relates back to the difference between 'low' and 'high-end' frequencies. Musical notation is spatially oriented both in terms of left to right across that page (a feature of reading and time) and more importantly up and down upon the stave. Thus, it is logical to link vertical positioning on the Y axis with the octave structure of harmonically organized sound (i.e., pitch). So, pitch is double mapped to both brightness and height in relation to experiential descriptions of sound alongside color, which is purely based on octave equivalence.

#### **8.6 Future work**

A great deal more work needs to be done to solidify these foundations. A more rigorous semiotic evaluation of the Visual Music outputs (not just film) would be invaluable. Additionally, basic psychology experiments to verify the relationship between abstract sounds, words and shapes would also be helpful. For instance, it is not beyond the realms of conception, for example, to hypothesize that such experiments would provide evidence that low frequency pure tones correlate with the word Bouba (and its rounded shape) and high frequency pure tones correlate to Kiki (and its spiky equivalent).

This kind of experimental approach to verification has not yet happened, as far as the author is aware, but the wealth of evidence from the range of domains explored alongside the evaluation of Visual Music examples strongly suggests that it follows from what has been discovered so far.

Future work should involve the development of an interdisciplinary approach to research that combines experimental psychology with Visual Music. This would bring many benefits to both psychology, in terms of furthering our understanding of crossmodal processing, while at the same time helping to verify the parameters presented here, that describe Visual Music more fully as it evolves into its next phase, This is particularly relevant for those practitioners that are creating and making artworks and performances at the boundaries of these two mediums (**Figure 1**).

**Figure 1.** *The author performing with his live visual music system, based on the parameters discussed in this chapter.*

*Towards a Semiotics of Visual Music DOI: http://dx.doi.org/10.5772/intechopen.110777*

### **Author details**

Shaleph O'Neill Duncan of Jordanstone College of Art and Design (DJCAD), University of Dundee, Dundee, UK

\*Address all correspondence to: s.j.oneill@dundee.ac.uk

© 2023 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

### **References**

[1] Em D, Em M. An Afternoon with John Whitney, an EM2 Production. Funded by SIGGRAPH Special Projects Committee; 1991

[2] McDonell M. Inventing instruments: Colour-tone correspondence to colourmusic performance. In: Gibson S, Arisona S, Leishman D, Tanaka A, editors. Live Visuals; History, Theory, Practice. London and NewYork: Routledge; 2023

[3] van Campen C. Artistic and psychological experiments with synesthesia. Leonardo. 1999;**32**(1):9-14

[4] Pieperhoff E. The Influence of Synaesthesia on 20th Century Art and beyond: Myths. DVM Verlag: Reality and Artistic Expressions of the Cross-sensory Experience; 2009

[5] Brougher K, Mattis O, Strick J, Ari Wiseman A, Zilczer J, editors. Visual Music: Synaesthesia in Art and Music since 1900. London: Thames and Hudson; 2005

[6] Gibson S, Arisona S, Leishman D, Tanaka A, editors. Live Visuals; History, Theory, Practice. London and New York: Routledge; 2023

[7] Young T. The Bakerian lecture. Experiments and calculation relative to physical optics. Philosophical Transactions of the Royal Society of London. 1804;**94**:1-16. DOI: 10.1098/ rstl.1804.0001

[8] Squires EJ. The Mystery of the Quantum World. 2nd ed. Abingdon: Taylor & Francis Group; 1994. ISBN 9781420050509

[9] Greiner W. Quantum Mechanics: An Introduction. Berlin: Springer; 2001. DOI: 10.1007/978-3-642-56826-8

[10] Shao M, Liang F, Yu H, Zhang H. Angular engineering strategy of an additional periodic phase for widely tunable phase-matched deep-ultraviolet second harmonic generation. Light Science & Applications. 2022;**11**:31. DOI: 10.1038/s41377-022-00715-w

[11] Berman G. Synesthesia and the arts. Leonardo. 1999;**32**(1):15-22. DOI: 10.1162/002409499552957

[12] van Leeuwen TM, Trautman-Lengsfeld SA, Wallace MT, Engel AK, Murray MM. Bridging the gap: Synaesthesia and multisensory processes. Neuropsychologia. 2016;**88**:1-4

[13] Ramachandran VS, Hubbard EM. Psychophysical investigations into the neural basis of synaethesia. In: Proceedings of the Royal Society B: Biological Sciences. Vol. 268. London: The Royal Society; 2001. pp. 979-983

[14] Ramachandran VS, Hubbard EM. Synaethesia—A window into perception, thought and language. Journal of Consciousness Studies. 2001;**8**(12):3-34

[15] Ramachandran VS, Hubbard EM. The phenomenology of Synaesthesia. Journal of Consciousness Studies. 2003;**10**:49-57

[16] Casini S. Synesthesia, transformation and synthesis: Towards a multisensory pedagogy of the image. The Senses and Society. 2017;**12**(1):p1-p17. DOI: 10.1080/17458927.2017.1268811

[17] Hume E, Johnson K. Sound symbolism. In: Hirschfeld L, Gelman S, editors. Handbook of Cognitive Development. Oxford: Blackwell; 2001. pp. 445-469

*Towards a Semiotics of Visual Music DOI: http://dx.doi.org/10.5772/intechopen.110777*

[18] Saussure F d. Course in general linguistics. In: Bally C, Sechehaye A, editors. TransWade Baskin, Subsequently Edited by Perry Meisel & Haun Saussy. NY: Columbia University Press; 2011

[19] Köhler W. Gestalt Psychology. New York: Liveright; 1929

[20] Deroy O, Auvray M. A new Molyneux's problem: Sounds, shapes and arbitrary crossmodal correspondences. In: Proceedings of Workshop, SHAPES 2.0: The Shape of Things. Co-located with UniLog 2013: 4th World Congress and School on Universal Logic, April 3-4, 2013. Vol. 1007. Rio de Janeiro: CEUR-WS Online; 2013. pp. 61-70

[21] Ćwiek A, Fuchs S, Draxler C, Asu EL, Dediu D, Hiovain K, et al. The Bouba/ Kiki effect is robust across cultures and writing systems. Philosophical Transactions of the Royal Society B: Biological Sciences. 2022;**377**:1841. DOI: 10.1098/rstb.2020.0390

[22] Johansson NE, Anikin A, Carling G, Holmer A. The typology of sound symbolism: Defining macroconcepts via their semantic and phonetic features. Linguistic Typology. 2020;**24**(2):253-310

[23] Yost WA. Psychoacoustics: A brief historical overview. Acoustics Today. 2015;**11**:3

[24] Fastl H. Psychoacoustics and sound quality. In: Blaubert J, editor. Communication Acoustics. Berlin: Springer; 2005

[25] Susini P, Lemaitre G, McAdams S. Psychological measurement for sound description and evaluation. In: Berglund B, Rossi GB, Townsend JT, Pendrill LR, editors. Measurement with Persons: Theory, Methods, and Implementation Areas. New York: Psychology Press; 2012. pp. 227-253

[26] Lye L. Kaleidopscope. 1935. Available from: https://www.youtube. com/watch?v=l3ZXUTL3iZs [Accessed: February 28, 2023]

[27] Lye L. A Colour Box. GPO. 1935. Available from: https://www.youtube. com/watch?v=fOEqTwwkB3Y&t=1s [Accessed: February 28, 2023]

[28] Fischingers O. An Optical Poem. Metro Goldwyn Meyer; 1938. Available from: https://www.youtube.com/ watch?v=6Xc4g00FFLk [Accessed: February 28, 2023]

[29] Bute ME, Webber MF, Nemeth TJ. Rhythm In Light. 1934. Available from: https://www.youtube.com/ watch?v=850AmGWl-Ko [Accessed: February 28, 2023]

[30] Bute ME, Nemeth TJ. Synchromy No. 2. 1935. Available from: https://www.youtube.com/ watch?v=ylQRIf3a9IY [Accessed: February 28, 2023]

[31] Bute ME, Nemeth TJ, Rutherford B. Parabola. 1937. Available from: https:// www.youtube.com/watch?v=DUwiVelk-Oc&t=163s [Accessed: February 28, 2023]

[32] Bute ME, Mclaren N. Spook Sport. 1939. Available from: https://www. youtube.com/watch?v=R4uPVzXu90Q [Accessed: February 28, 2023]

[33] Bute M E. Tarantella. 1940. Available from: https://www.youtube.com/ watch?v=czDsy8BYP1M [Accessed: February 28, 2023]

[34] Peters D, Batchelor L. Pen Point Percussion: An Introduction to the Handdrawn Sounds of Norman McLaren. The National Film Board of Canada, 1951. Available from: https://www.youtube.

com/watch?v=Q0vgZv\_ JWfM [Accessed: February 28, 2023]

[35] McLaren N. Dots. The National Film Board of Canada. 1940. Available from: https://www.youtube.com/ watch?v=tKikRLzlrwk [Accessed: February 28, 2023]

[36] McLaren N, Lambert E. Lines Horizontal. The National Film Board of Canada. 1960. Available from: https://www.youtube.com/ watch?v=TDFWuvhqHo4 [Accessed: February 28, 2023]

[37] McLaren N. Synchromy. The National Film Board of Canada. 1971. Available from: https://www.youtube. com/watch?v=\_UA40sL06sU [Accessed: February 28, 2023]

[38] Whitney J, Whitney J. Five Film Exercises, Film 1. 1943. Available from: https://www.youtube.com/ watch?v=kuZbgM8yxtY&t=74s [Accessed: February 28, 2023]

[39] Whitney J, Whitney J. Five Film Exercises, Films 2-3. 1943. Available from: https://www.youtube.com/ watch?v=JdCjwS1OxBU [Accessed: February 28, 2023]

[40] Whitney J, Whitney J. Five Film Exercises. Film 4. 1943. Available from: https://www.youtube.com/ watch?v=nTEABxz1e\_k [Accessed: February 28, 2023]

[41] Whitney J. Permutations. Motion Graphics Inc. 1968. Available from: https://www.youtube.com/ watch?v=kdYxJ85RSds [Accessed: February 28, 2023]

[42] Whitney J. Arabesque. 1975. Available from: https://www.youtube. com/watch?v=sQrq7S0dP54 [Accessed: February 28, 2023]

#### **Chapter 4**

## Audiovisual Music from the Audience's Viewpoint

*Amparo Porta-Navarro*

#### **Abstract**

The music of everyday life crosses areas whose borders are blurred, takes place on screens and forms part of the trinomial that connects it with the visual and narrative realms. This space of high emotional tension is created with characters, settings, stories and roles, and is of a spectacular nature. All this allows it to provide meaning, which confirms the need for this study. This work looks into audiovisual music and its connections with the narrative, image and story. To perform this approach, we will study the leitmotif, music as a figure and background, diegetic/extra-diegetic music and emotions from the audience's viewpoint. Furthermore, to explore intermediality we will use a mixed method, with tools that include bibliographic reviews, content analyses and maps. With them, we will carry out relational and specific studies, while analysing the connections between music, image and narrative. This study explores the audiovisual experience and its perceptions through music.

**Keywords:** music, transmedia/audiovisual, viewer, narrative, soundtrack

#### **1. Introduction**

#### **1.1 Music**

#### *1.1.1 Music and mediums*

Today, references to 'music' encompass a wide range of expressions and descriptions that have always been subjected to the conditioning factors of each time and place. In turn, these factors were closely linked to the possibilities of expression and action offered by their individual context. Thus, both making music and listening to it as an audience, or its performance with voices, instruments and devices, has been a constant factor since mythological tales. The most noteworthy figure is Orpheus, who was able to rule the world with his music: mountains, trees and animals were all swayed by his voice and lyre. He even descended into the underworld to rescue his wife, Eurydice. Even though he was unable to do so, he continued singing [1]. Today, music can be heard on numerous sonic devices based on playing it back, and very frequently connected to screens. The invention of electricity represented a turning point that is only comparable to the appearance of Ars Nova and polyphony [2]. Music has evolved constantly, but especially since the late nineteenth century. It then took off after the second half of the twentieth century with the ability to play music, the

emergence of popular music with rock and roll, the technological development of sound and the industry of information and leisure. In the history of western music, after Graeco-Latin culture (of which there are few remains) come the Early Middle Ages, with the gestural medium of Gregorian chants and hand movements. This was followed by writing neumes above and below a line, until the establishment of the stave, with five lines that made it possible to place the notes at different heights and express their duration. This method finally became established in Europe in the eighteenth century. The perpetuation of sound was another landmark, thanks to the appearance of the press and printed musical scores making music permanent, so it could be performed. Then, in the twentieth century, there was another major leap thanks to mechanical reproduction [3] and large multimodal spaces that combine narration with sound and images that are almost always in motion and often synchronised with the music. Then came the digital transformation, bringing together numerous means of expression, communication and art, as well as recordings, stages, screens or online stages. It encompassed the technique, the audience, live music, recorded music and noise while being a prominent setting for popular and traditional culture. Experience with music is another sign of this era, a time where silence seems to have no place or, as expressed by Benjamin, Aguirre, de Alba [3]. This evolution has made music one of the main (and almost mandatory) constructors of multimedia experiences.

#### *1.1.2 Approach*

Music is a coherent organisation of sounds and silences in its essential parameters: melody, harmony and rhythm. The way of defining its elements varies among cultures, and also among schools of thought. In general, we can say that the melody is a set of sounds and silences—created within a specific sonic environment—that are played in succession, and are perceived as having an identity and a sense of unity; the basic unit of harmony is the chord, and it regulates the concordance between sounds played simultaneously and their connection to neighbouring sounds; the metre refers to the repeated pattern, at regular (and sometimes irregular) intervals, of loud sounds, weak sounds and silences. In a composition, the rhythm is the end result of these elements. It could generally be defined as the ability to create contrasts in music, and is caused by different dynamics, tones, textures and sounds; the tone is the characteristic of sound that is caused by a specific sonic agent. In addition, there are accents, which alter the intensity (dynamic accents) or speed (agogic accent), and the musical structure or form. Lastly, it is worth mentioning that musical elements are directly linked to the characteristics of sound. The rhythm is linked to the duration, the melody and harmony to the pitch and the accents to the intensity and duration. Current artistic trends differentiate between music and sonic art, with the main difference between them being that music uses a language where the musical sound or tone is framed within scales, counterpoints, harmony and rhythm, unlike sonic art, which uses sound as a generating element, releases it and then groups it together again. This is why music must not depend on audio technology. However, Reyes (2006) believes that sonic art emerged from the connection between music, art and technology, where music and sound are located in a new expressive or gestural interface. In this interface, the sonic elements have meaning depending on the other elements and, far from being independent, they act as the sum of their parts, as was proposed by the artistic vanguard. Familiar concepts in mass media linked to sound (e.g. recorded sound, loops, sound simultaneity and manipulation, editing, tempo

changes and phonetic language expressed through items) and others (e.g. stretch, compress, switch, manipulate, see the sound or perceive the visual rhythm) have become part of vanguard art in the twentieth century, and of the general uses made by the youth, as a trend, until today [4].

#### **1.2 Language and communication**

Several authors have studied audiovisual productions as constructors of reality. More specifically, their music has been studied by [5] Morley [6] Porta and Sloboda [5–7]. Furthermore, it was a subject of interest in cinema for authors such as Adorno, Leppert and Chion [8–10].

#### *1.2.1 Narrative*

The story told by music has been studied from several viewpoints. Meir Sternberg [11] talks about the differences in fiction/non-fiction (historical narrative, autobiography, reports, etc.) regarding the action of the individual who remembers, narrates and uses deictic markers that place the actions in specific space/time contexts. From the viewpoint of cinema, the narrative has been studied by Michel Chion, especially in his work of reference in the field, Audiovision [9]. Others have specifically studied the narrative possibilities of music in this field, such as Adorno, [8], among others, and its connection to images. The intervening elements from the viewpoint of major media companies are all the elements of the story, the text, the image, the narrative and the staging, with key roles being held by conflict, catharsis and the figure of the hero. From the viewpoint of semiotics, Bathes [12] studied the significant connections through narratives that express shared cultural values. Dewey [13], taking the viewpoint of the Theory of Art, made a key contribution to this field with Art as Experience. In the field of art, proposed research is based on art, in an attempt to connect what is said with how it is said. Meanwhile, narrative art analysed the form and material mentioned by Vygotsky [14] ('form' in terms of the plot or story, and 'material' in terms of the content or tale). In its connection to music, we can see growing trends towards cognitive aspects, with increased interest in the depiction Gómez-Ariza, Porta [6, 15]. The music and soundtrack are part of the audiovisual production, along with the narration and image. In some scenes, their role can be critical for the cinematographic narrative, as well as a major driver of emotions [6, 16].

#### *1.2.2 Semiotics*

Audiovisual music can be seen as a semiotic system that is structured and configured to create expressive meanings by staging a communicative chain, and whose meaning can be approached from two viewpoints: the person who speaks and the person who listens [6].

Its key points are summarised in the topic at issue. The reader is encouraged to read Porta [16] to flesh out the information. Semiotics is the science that studies the communication systems of human societies. It says that a sign is an item or event that takes the place of another, which is absent, by virtue of a certain code. In these lines, we will go over four of its main proponents. Ferdinand Saussure was the first to talk about semiology, establishing the bases for modern communication by studying everything from fashion to shows, customs, culture, verbal and non-verbal language

and multimodal speeches [16]. Peirce [17] believes the sign, item and interpretation are the three main components, providing feedback to one another, creating a spiral process. Roland Barthes [12] studied popular culture and the significant connections through narratives that express shared cultural values, whereas Umberto Eco, in The Absent Structure [18], talked about the meaning as a cultural unit [4]. Davidson and Peirce summarised the contributions, agreeing on 1) defending a justification, broad and social vision of the meaning, 2) the external reality being the main source of our beliefs and 3) interdependence between thought, language and action. Lastly, we have the contribution of Vygotsky: 'All that is internal in its higher forms must have been external. In other words, it must have been for others what it is now for one's self [ … ]. Any higher psychological function must have gone through an external phase in its development, as it is initially a social function' [19].

#### *1.2.3 Cognitive aspects*

The works reviewed on music show a transition from psychoacoustic to more cognitive viewpoints, with greater interest in aspects such as learning, memory or the performance; Gómez-Ariza, Porta [6, 15]. Specifically, some authors studied the cognitive and significant effects of the media, highlighting the singularity of mental processes, the development of specific skills and its systems of symbols. There is also the research of Tal on comprehension, which took into account the restructuring theories of cognitive psychology. These theories are of a constructivist nature, and use molar units due to their meaningful nature [20]. This contribution is valuable because it shifts the importance towards the context, the habitat and its elements of significance, which, as happens in reality, are never separate [21]. The third contribution we take into account is López-Varela [22], who connected the psychological-cognitive development to the affective relationships created in the family environment [22].

#### **1.3 Emotions**

Psychology commonly defines emotion as a feeling or perception of the elements and relationships of reality or the imagination that are expressed physically, and which include behavioural reactions. Ekman, Levenson and Friesen [23] proposed six basic emotions, with the most complete definition being in Kleinginna and Kleinginna [24], who suggested several categories (e.g. affective and cognitive categories) based on external, physiological, emotional/expressive, disruptive, adaptive, multifactorial, restrictive or motivational stimulation. Furthermore, Vygotsky [25] talked about emotions as primitive feelings that cannot remain indifferent or ineffective on behaviour, and which have signed as mediators. These concepts were used in Porta and Porta, Herrera [4, 21, 26] to create experiences referenced in this study on children's experience when listening to audiovisual productions.

#### *1.3.1 Emotions and experience*

According to Vygotsky [27], emotions are one of the hardest research topics to address. He believed that they should be approached through empirical studies on experience, its indications and connections to thought and internal language. He considered emotions as the internal organiser of our behaviour, which add tension, excite and stimulate us, causing three groups of reactions: reflexive, motor, somatic and secretory; bodily reactions; and a secondary perception of the proprioceptive

field, divided into positive and negative feelings [25]. Emotions are involved in the meanings we create from our life experiences [28].

#### *1.3.2 Music and emotions*

The effects of music on emotions are a field of rising interest, especially in psychology, health sciences and social studies, with a broad bibliography, that we will go over in the bibliographic review. From the experiential side on the effects of music on filmic and audiovisual tales and narratives, Amparo Porta and Porta, Herrera [16, 26] performed audiovisual experiences to study their effects on children from a Vygotskian perspective, finding connections between music, emotions and some narrative elements, the first of which is the hero. The perception of the story, the plot and the personality of the hero cause conflicting emotions and are a key process in the tragedy, which requires specific features [29]. The second is the catharsis. From the Vygotskian viewpoint, emotions, in cinema, create an alternating balancing system of opposite feelings. For the viewer, there is a catharsis when negative emotions are offset or neutralised by positive ones in the filmic process [16].

#### **1.4 Cognition, perception and comprehension**

#### *1.4.1 Perception*

Luria, from the viewpoint of cognitive psychology, says that perception is the first chain in human cognition [30]. In the case of auditive perception, it is generated by the sense of 'hearing', which he differentiates from listening as the cognitive generative element of speech, sound and music.

#### *1.4.2 Trends*

To approach cognition and comprehension, we will begin with the poststructuralist viewpoints that address the construction of conscience [19]. The theory of art focuses on the representation of the world [31]. Music requires a specific approach [15] that helps understand listening as a social creation that has meaning. The musical approach is necessary because music speaks its own language [6]. The literature review revealed that psychology highlights the contributions of cognitive psychology on the processing of information and restructuring theories, with noteworthy works and authors such as Piaget, Vygotsky or Gestalt psychology [17, 19, 32]. One of the significant changes in significance lies in the analysis units, which propose the use of molar units. Of these, we highlight Vygotsky's socio-historical theory, which was subjected to a process of reinterpretation and possible application to audiovisual comprehension, according, and whose principles and some practical implementations have been performed in works by Porta [4]. The Russian author shows how mental processes are explained by the instruments and signs, which act as mediators and have meaning. For several authors, this is the most original and important contribution of Vygotsky, who leans on the study of the communicative nature of the signs [4].

#### *1.4.3 Understanding music and the listening process*

Music cognition has been studied by several authors, such as Meyer or Sloboda [7, 33]. We are interested in the act of listening to the music and the soundtrack, in a significant way and with meaning. Taken as such, audiovisual music represents a polyhedron with numerous sides: musical, narrative, persuasive, patrimonial, temporal, cultural and cognitive, in addition to its connections with identity, communication and society.

#### *1.4.4 Method*

The scientific method frees thought of all doubt, and makes it possible to obtain increasingly reliable and more stable patterns of action, as it sets beliefs based on 'something permanently external' (i.e. a reality detached from our opinions) [34]. In a complementary way in multimodal discourse, modes are resources that allow the simultaneous realisation of discourses and types of (inter) action. This is why the first premise when studying this topic is to locate the different communicative practices and their procedures.

Regarding the positions of the disciplines and their methods, Vygotsky highlights as an issue the connection of emotions with adjacent fields of knowledge, such as the psychology of language, linguistics and the psychology of education. Intermediality suggests dialogue between disciplines interested in expanding fields and the value of experience.

#### **1.5 Intermediality**

According to Irina Rawjesky [35], there are two different approaches to studying intermediality. 1) The approach resulting from literary studies and narrations, and 2) the approach of communication studies.

In this work, we include both approaches, based on audiovisual experiences with music studied as a discourse and content, understanding intermediality as a meeting point between material, semiotic and cultural aspects. We are interested in 'Where are they?' and 'What do they say?' from the viewpoint of music.

#### *1.5.1 Where are they?*

When we talk about audiovisual music in this live and changing environment, music and the media form an inseparable pairing [21]. The audio is the space for the soundtrack and the place where its music lies together with the image, shaping a story (the tale). We approach all of this, taking into account its form and content, as well as its texts and discourse [36]. However, empirical research requires the use of instruments. Music is produced over time, which is why its analysis must have a time-based anchor that must be doubled, in order to answer what is being listened to, and where the material being listened to is located. In this case, the first issue to address is its limitations to being observed and analysed objectively [6, 21].

#### *1.5.2 What do they say?*

Audiovisual productions tell stories with music and moving images. Therefore, they shape a space of interaction that we want to explore as a narrative and also as a discourse. The meaning of audiovisual music can be approached from two viewpoints: from the person who speaks, and the person who listens [6]. From the viewpoint of the person who speaks, linked to poiesis it has been studied by musical language, the theory of music and musicology. From the viewpoint of the person who listens, it has

*Audiovisual Music from the Audience's Viewpoint DOI: http://dx.doi.org/10.5772/intechopen.111567*

been studied by ethnomusicology, sociology and cultural studies, among others, focusing on social discourses. Communication sees audiovisual music as a discourse, an instrument of signs [37], whereas education takes part in both viewpoints due to its scope and history [16]. All this involves the context, the background and the listening skill as a dynamic and unfinished process [16]. From the viewer's perspective, listening and the meaning take place through the summation of languages [6, 21]. This has been studied by different schools, including the School of Frankfurt, with authors such as Adorno, Leppert, W. Benjamin and E. Gombrich [3, 8, 31]. Sometimes, music has a spectacular nature and becomes a ritual. This happens in rock concerts, where mass expressions of popular urban music take place [6, 38]. Considering all the above, music reaches everyone through platforms of mass dissemination, which is received anonymously and produces effects that have hardly been studied [4].

#### **1.6 The importance of experience**

We provide three ideas to shape this section. The first is the concept of habitus. Pierre Bourdieu [39] says that habitus is the generation of actions limited by the social circumstances that withstand them. They are imprinted in the human body and mind and shape subjective social structures.

The second idea is the possibility of describing the world in an objective (consensual) way, which depends on our intersubjective communication skills [22]. The third is to find a place or medium that can make it possible, and, according to López-Varela, the common factor among these platforms is the inclusion of one or several mediums in another form of communication. The methodological result of all this entails not only different physical ways of coding the human experience but an amalgamation of cultural relationships (processes) that include several channels for processing the information obtained from those experiences (visual, aural, etc.) with changes in the communicative contexts [22].

In this study, we want to show music together with the image and narrative: the synchronisation, the creation of worlds and realities from sound, their ability to tell stories, the place and position of the narrator, their evocative power, the task of informing of and describing elements without listening (such as space/time and other external elements), their ability to hold diegetic and extra-diegetic positions, cause reactions that affect a different sense through sound [26] and create dialogues between emotional, cognitive and narrative aspects in an intermedial way.

#### **2. Method**

This study preferentially used a qualitative methodology, analysing content by the importance of the context [40] and its intermediality, as well as quantitative elements, to achieve, through triangulation, the best results possible. All from the global viewpoint of the project that we have been conducting since 2017, with more recent results in Porta [16], and which allows us to continue expanding our position, contents and coverage. We used the grounded theory [41] and the constant comparative method of Glaser and Strauss. This study has a descriptive and interpretive nature, with the noteworthy presence of a qualitative analysis [36, 42] that is systematic and objective.

**Objective**: This study focuses on audiovisual music and its relationship with the story, the narrative and the image. Its objectives are as follows:


To conduct it, we performed the following actions: a) Starting point: Selecting experiences performed by 150 students of bachelor's degrees and postgraduate studies in music-related courses, b) Bibliographic review, c) Content analysis, d) Creation of maps using MxQda and e) With them, we will perform relational and specificity studies, as well as examining the connections between music, image and narrative.

#### **2.1 Design of the investigation, materials and procedure**

This work is part of a financed project on audiovisual listening. This paper addresses one of its objectives. Selecting the methodological system allowed us to approach our objectives, learn the background and implement the necessary tools to conduct it. In all, the design and research were conducted in four phases, with three tools and two content analyses (**Table 1**).

#### **2.2 Starting materials, participants and experiences**

We performed an initial exploration of experiences with audiovisual music, preferably cinema and video games, performed by students of higher studies in musical education [42]. We selected the activities whose topics and approaches had been chosen freely by the students themselves. With the results obtained, we performed an initial content analysis, which determined the topics that were going to be the centrepiece of this chapter.

### **2.3 Participants**

A total of 88 bachelor's degree dissertations (TFG, in Spanish) and master's degree dissertations (TFM, in Spanish) in Music Didactics were selected for reasons of continuity with our current projects.


#### **Table 1.**

*Research design, materials and procedure.*

#### **2.4 Tools used**

We used the Excel program for the quantitative measurements and the bibliometric analysis, and the MxQda program for the qualitative analysis. For the content analysis, we used the procedure that had already been implemented [16, 42].

#### **3. Results**

#### **3.1 Results objective 1. Content analysis 1**

Our objective was to identify the most prominent topics in audiovisual experiences performed by 150 students with bachelor's degrees and postgraduate degrees related to musical education. To do so, we chose the most prominent topics from the dissertations of students' projects from 2020, 2021 and 2022. The audiovisual topic had to have been chosen freely. When searching for them with content analysis 1, using the program MxQda, we found 88 documents. Of these, the most commonly selected topics were: emotion, feeling, diegetic, extra-diegetic, leitmotif and background music. This was the first step to study the presence of this group's interests in academic and research texts. We searched and filtered results in Google, Google Scholar and the Web of Science, thus completing a journey on the topics chosen by the students from the informative sphere to the scientific realm [42].

#### **3.2 Results objective 2. Bibliometric study**

The second objective proposed was to search for the topics on Google, Google Scholar and the Web of Science (from less restrictive to most restrictive), to analyse their presence and communicative interests.

#### *3.2.1 The audiovisual music experience on Google*

We initially searched for the words 'music', 'audiovisual' and 'experience' to limit the scope of the search. Then, the search targeted the topics chosen by the students, obtaining the following results, ordered by the number of results found: Audiovisual music experience: approximately 10,600,000 results; Audiovisual music emotions: approximately 3,740,000 results; Audiovisual music background: approximately 3,330,000 results; Diegetic and/or extra-diegetic audiovisual music: 46,300 results for diegetic music, and 18,900 for extra-diegetic music, for a total of 65,200 results; Audiovisual music leitmotiv: approximately 63,200 results.

#### *3.2.2 Searches in Google scholar*

The results already show a drastic decrease in results. For example, in the case of audiovisual music experience, the number of results went from 10,600,000 to 237. The bibliometric study was performed with the latter amount, using the entries obtained in the report on publications, citations, fields and main concepts. It provided the following results and analysis. The information obtained in publications and citations reveals a slow start until 2007. We obtained the following results, ordered by their appearance (as a percentage): Fields: Behavioural Sciences (74, 35.71%); Psychology (70, 33.17%); Music (66, 31.28% - third place); Neurosciences Neurology

(47, 22.27%); Computer Science (43, 20.379%); Arts Humanities Other Topics (39, 18.483%); General Internal Medicine (29, 13.744%); Communication (28, 13.270%); Film Radio Television (22, 10.427%); Education Educational Research (21, tenth place, last). Regarding the main concepts, the first two spots were held by 'behavioural sciences' and 'psychology', whereas 'music' is in third place, 'method' in sixth and 'educational research' last.

#### *3.2.3 The research response*

We then searched for the students' most prominent topics, in addition to studying the results of their perceptions and research methods used. All in order to learn the research activity present on the Web of Science. We obtained the following results:

#### *3.2.3.1 Featured topics. Music*

In connection to the topics explored in the experiences performed by 150 students, we found 68 results, connected to emotion OR feeling OR diegetic OR extra-diegetic OR leitmotif OR background music. Of these, we analysed the report of citations and publications, fields and main concepts. The analysis yielded the following results: Publications and citations. Publications start being significant in 2007, and citations start rising in 2019. Fields: 1 and 2 psychology, 3 music, 4 neurology and 5 computer science. Main concepts: 1 Coordination, 2 nervous system 3 behaviour, 4 method and 5,6,7 medicine. Country with the most results is the USA.

#### *3.2.3.2 Perceptions*

We conducted a specific study of the perceptions, analysing the relationships between the experience of music and cognitive construction from the standpoint of people's perceptions. A total of 80 results were obtained, of which we analysed the fields of research that had this topic as a focus of interest. The report of citations, fields of research and main concepts were used, with the following results and analyses: Information on publications and citations: Publications begin in 2002, creating constantly rising double and triple peaks. The highest number of citations was registered in 2016, with 240. Regarding fields, the first place was held by 'neural coordination'. Second was 'sensory reception', 'method' was eighth and 'music' and 'education' do not appear. As regards concepts, the most prominent was 'methods and Ttchniques' (8). Most noteworthy country is the USA.

#### *3.2.3.3 Audiovisual music experience. Method*

Lastly, from the results obtained from the audiovisual experience, we explored the research methods. A total of 84 results were obtained, which we analysed through a report of citations and publications, fields and main concepts. Report on publications and citations: Publications were uneven, but follow an upward trend. The highest number of citations was 160. There were five main concepts, which did not include 'method', 'music' or 'education'. Countries: First, the USA.

#### *3.2.4 The elements of analysis in WOS*

Through reports on citations and publications, we obtained the following results:

#### *3.2.4.1 Audiovisual music experience*

Specific results of interest: Experience, perception and method. The highest specificity of music was obtained in 'experience-perception'. The most prominent field of interest was 'experience', with 23 articles and 360 citations, both obtained between 2002 and 2022. It was followed by 'perceptions', which also began in 2002 but started increasing in 2007 (2), reaching its peak in 2016 (8), and registering 250 citations in 2021. Lastly, 'methods' began in 1976 (1) and increased in 2007 (2), growing unevenly until reaching its peak in 2019 (10 articles and 150 citations).

Audiovisual music experience publications: A very slow start in 1976, appearing later, in 1987 and 1988 (1 publication). Since 2007, it had a continued rise, reaching a peak of 21 in 2019. Citations: Significant since 2007, upward trend. Maximum of 360 in 2021.

Perceptions and publications: Started in 2002, maximum of 250 in 2016. Citations: Significant since the maximum of 250 in 2021.

Audiovisual music experience. Method and publications: Beginning in 1976. Highest of 10 (2019). Citations: Significant since the maximum of 160 (2022).

#### *3.2.4.2 Topics and the study of emotions*

Of the prominent topics of interest highlighted in the experiences conducted by students, the Web of Science produced 57 results on emotions and feelings. We conducted a detailed analysis through a report of citations and publications, fields and main concepts.

Report of publications and citations: The publications began around 2008, with uneven peaks and an upward trend. The most prominent years for citations were 2019 and 2022.

Fields: 'Psychology' is first, 'neurology'second and 'music' fifth.

Regarding the main concepts, the first place was taken by 'methods and techniques'.

Countries: First, the USA.

Reports of publications and citations:

The four favourite topics were leitmotiv, background music, diegetic/extradiegetic and emotions. The answer by the Web of Science: The students' four favourite topics start appearing on the WoS around 1976, when there is just one publication, until 2001. The number of publications rose in 2007, and in 2015 it reached its peak, with eight.

Publications: Beginning in 1976 (1 publication), with its peak in 2015 (eight publications). Citations: Significant, with 240 citations in seven publications in 2021.

Only 'emotions' and 'feelings'. Publications: Beginning in 2002, with a peak of 7 in 2019 and 2022. Citations: Significant since 2011, with a maximum of 180.

#### *3.2.4.3 By fields of knowledge*

The presence of music in an analysis by fields: Audiovisual music experience: 'music' in third place, and 'education educational research' tenth; perceptions: 'music' in third place; audiovisual music experience method 'music' in third place and 'education educational' tenth.

#### *3.2.4.4 Concepts*

The presence of music in the main concepts: Audiovisual music experience 'methods and techniques'sixth place; perceptions: 'methods and techniques' eighth place; audiovisual music experience method 'methods and techniques' fifth place.

#### **3.3 Results objective 3. Content analysis 3**

The third objective was to understand the relationships, specificities and connections between music, image and narrative. To do so, we used the program MxQda (for qualitative analyses) to perform a second content analysis. We selected the topics and their presence from the exploration performed in the Web of Science, summarised.

This section has responded to typologies and domains, for which the themes and their records have been used. We present the three tables that comprise it with codes: documents, segments, codes and superior codes (**Table 2**).

#### *3.3.1 Typologies*

Documents with codes.

**On music and sound:** The word 'music' in WoS (72, 85.71%), 'tone' in WoS (28, 33.33%) and characteristics of music (9, 10.71%).

**On Perception:** The perception of emotions (4, 4.76%), the perception of movement (3, 3.57%) and the perception of time (5, 5.95%).

**On the Perception of Health:** Perception of pain (1, 1.19%) and perception of anxiety (1, 1.19%).


**Table 2.**

*Content analysis 2. Documents, topics, codes, segments and records.*

*Audiovisual Music from the Audience's Viewpoint DOI: http://dx.doi.org/10.5772/intechopen.111567*

Specifically, on the perception of music and sound, in WoS, we found 'perception' in ninth place, with 47 results (55.95%). 'Aural and musical perception' had six results (7.14%), and 'the perception of audiovisual music' had two (2.38%).

Segments with codes

Here are the segments obtained and the percentage for three groupings: music and sound, perception and health.

**Music and sound:** The words 'music' (368, 60.63%), 'perception' (103, 16.97%). and 'tone' in WoS (61, 10.05%). Characteristics of music (27, 4.45%), the perception of music (9, 1.48%), the perception of audiovisual music (3, 0.49%) and aural perception (10, 1.65%).

**Perception:** The perception of time (13, 2.14%), the perception of emotions (5, 0.82%) and the perception of movement (4, 0.66%).

**Health:** The perception of pain (3, 0.49%) and the perception of anxiety (1, 0.16%).

#### *3.3.2 Domains*

Total of 88 documents containing music and perceptions on the Web of Science were analysed, showing the results obtained by large categories (**Figure 1**) and by codes (**Figure 2**). It has been made by Matrix of Codes of a selection of documents and marked codes, with indication of the quantities added by adjustment of the figure. The domains observed by large categories correspond to musical characteristics, perception, music, perception in RIS.

The word 'music': (368), 'perception': (151), all music: (88).

In order, the results were as follows: Timbre (61), perception of time (13) and music perception (12) (**Figure 2**).

#### *3.3.3 Relationships between the codes*

Typologies: The nine categories (**Figure 3**).

MxQda 2018. View of the relationships between close codes, with a distance of one paragraph.

Highest number: 'Timbre' (166) and 'musical perception' (123).

Lowest number: 'Anxiety' (11) and 'pain' (6) (health-related topics).

'Timbre' is connected to 'musical perception' (21), 'aural perception' (9), 'emo-

tions' (9), 'the perception of movement' (6), 'audiovisual music' (5) and 'anxiety' (2).

#### **Figure 1.**

*Domains observed in the major thematic categories.*

**Figure 2.** *Domains observed by codes.*


#### **Figure 3.**

*Relationships between the codes.*

'Aural perception' is connected to 'tone' (9), 'the perception of audiovisual music' (3) and 'musical perception' (8).

'The perception of emotions' is connected to 'plantilla' (12) and 'tone' (9).

'The perception of audiovisual music' is connected to 'tone' (5) and 'aural perception' (3).

'Musical perception' is connected to 'tone' (21) and 'aural perception' (8).

The word 'music' in WoS is connected to 'tone' (185), 'aural perception' (48), 'musical perception' (53), 'the perception of audiovisual music' (27), 'the perception of time' (26) and 'emotions' (19).

#### *3.3.4 Method by comparison of documents*

5 MxQda 2018 graph of the comparison of documents (**Figure 4**).

**Figure 4.** *Graph of the comparison of documents. BLUE the characteristics of music. PINK perception. BROWN method.*

**Figure 5.** *Method WoS the model of a code.*

#### *3.3.5 Method WoS the model of a code*

Regarding the method, represented in the map obtained by a code model (**Figure 5**), the results of the investigations (WOS) are shown separated by a line. Above the line, we show the results of the perceptions of music in a broad sense and, below the line, we show the most outstanding non-musical perceptions. In this second group, we highlight the highest results and their relationship with music. They are the perception of time, obtained by the sum of temporal perception and audiovisual time perception (13), followed by the perception of emotions (5).

#### **4. Conclusions**

Music is a form of expression that has accompanied society since its origins. Boulez defined music as sound in time. Its forms have changed, but it has been present in all stages of life, accompanying people, institutions, societies and cultures. This chapter has looked at music as an expressive, communicative and aesthetic element that affects people and takes shape through voices, instruments and devices. One of the most common spaces is screens, where real and fictional spaces are created. It is also through them that stories are told, words are spoken and the viewers understand the message. From the viewpoint of intermediality, Irina Rawjesky suggests two ways of approaching its study and what to focus on: one derived from literary studies and the narrative, and another one derived from communication studies. In this paper we looked at both, understanding intermediality as a meeting point between expressive, material, semiotic and cultural aspects. We strived to answer 'Where are they?' and

'What do they say?' regarding music in audiovisual experiences. On screens, music can be found in the soundtrack, which, together with the image, tells us stories. The second question is no less important: What does music say? Music, together with the narrative and the image, creates a synchronisation and creates worlds. It sometimes takes the position of the narrator and describes aspects that are unnoticeable through sound, such as the temperature, the focal point and others. It settles into invisible angles and achieves a significant symbiosis between emotional, cognitive and narrative aspects in an intermedial way. We have approached the topic using a scientific analysis.

This study focuses on audiovisual music and its relationship with the narrative, the image and the story. Its objectives have been 1. To identify the prominent topics in active audiovisual experiences developed by 150 participants, 2. To analyse their presence on Google, Google Scholar and the Web of Science (WoS) and 3. To understand the relationships, specificities and connections between the music, image and narrative.

For this approach, we used a qualitative methodology by applying a content analysis on the importance of the context and intermediality, and a quantitative methodology to achieve, through triangulation, the best results possible. We used the grounded theory, which has a descriptive and interpretative nature, and whose most noteworthy feature is the qualitative analysis, which is systematic and objective. To do so, we started with a selection of experiences of bachelor's degree and master's degree students, performing the following actions: a bibliographic revision, content analyses, creating maps with MxQda and relational and specific studies.

Selecting the methodological system allowed us to approach our objectives, learn the background and implement the necessary tools to conduct the study. The research was conducted in four phases, with three tools and two content analyses, as well as using expressive and analytical activities. The four phases were the following: P1: Identifying the participants' activities for the initial selection of prominent topics in Excel and MxQda, P2: Bibliographic revision on the Google Scholar and WoS databases, P3: Selection of concepts and topics and analysing them using MxQda and P4: Performing two content analyses, CA1 and CA2, with MxQda.

We performed an initial exploration of experiences with audiovisual music, preferably cinema, cartoons and videogames, performed by students with the Bachelor's Degree in Musical Education. We selected the activities whose topics and approaches had been chosen freely by the students. With the results obtained, we performed an initial content analysis to determine the topics that were going to be the focal point of this chapter.

Participants: A total of 88 bachelor's degree dissertations (TFG, in Spanish) and master's degree dissertations (TFM, in Spanish) in Music Didactics were selected.

Tools used: We used the Excel program for the quantitative measurements and the bibliometric analysis, and the MxQda program for the qualitative analysis. For the content analysis, we used the procedure implemented in Porta [16, 21].

#### **4.1 Results and their analysis**

#### *4.1.1 Objective 1 results*

We selected the most prominent topics from the projects of students from school years 2020 to 2022 with topics related to audiovisual music. Through content analysis 1 with MxQda, we obtained 88 documents. Of these, the most prominent topics were: Emotion, feeling, diegetic, extra-diegetic, leitmotif and background music.

*Audiovisual Music from the Audience's Viewpoint DOI: http://dx.doi.org/10.5772/intechopen.111567*

Then, we explored these initial results in academic and research documents by using progressive filtering on Google, Google Scholar and the Web of Science to make a journey from the communicative or initial environment (of students) to scientific databases, regarding the topics chosen.

The second objective was to perform thematic searches. First on Google, for being the least restrictive, and then on Google Scholar and the WoS to make a journey from the communicative or initial environment (of students) to scientific databases, regarding the topics chosen. The search revealed a drastic decrease in results, with 'audiovisual music experience' going from 10.6 million results to 237 in the different search engines. The exploration of the main publications, citations, fields and concepts provided elements of interest.

#### *4.1.2 Presence of the prominent topics, perceptions and methods in the WoS*

The students' most prominent topics fit into two main categories: 'Music' (diegetic/extra-diegetic, leitmotiv, and music as a figure and background) and 'emotions'. Their publications began in 2007, with 'music' being the third most common keyword, and 'method' the *fourth*. The most prolific country is the USA in all the analyses performed.

#### *4.1.3 The topics of the Web of Science*

The case of emotions. Of the 57 results found, publications began in 2008, with peaks in 2019 and 2022, and an upward trend.

#### *4.1.4 By fields*

The first two spots were held by 'psychology', and 'music' was fifth, with 'methods' and 'techniques' being the most prominent concepts.

#### *4.1.5 Perceptions*

We specifically conducted a study on perceptions, analysing the connections between the experience of music and cognitive construction from the viewer's perception. We obtained 80 results and analysed their fields of research. The publications began in 2002, with different peaks. The highest number of citations (240) was registered in 2016. By fields, the two most common ones were 'neuronal coordination' and 'sensory reception', while 'method' was eighth. Meanwhile, 'music' and 'education' do not make an appearance. By concepts, the most noteworthy is 'methods and techniques'.

#### *4.1.6 Audiovisual music experience. Method*

There is an uneven but upward trend of publications. Regarding citations, the highest number was 160, and there were five prominent concepts, which do not include 'method', 'music' or 'education'.

#### **4.2 From the study of the web of Science and its elements of analysis**

In the report of citations and publications, we obtained the following results: 1) Audiovisual music experience and the most prominent topics of students (leitmotiv, figure/background, diegetic/extra-diegetic, emotions). The results in the WoS of the students' favourite topics start yielding results in 1976 when there was just one publication. There were no more until 2001. The number of publications rose in 2007, reaching its highest peak in 2015. *2) Specific analysis of emotions and feelings:* Publications began in 2002, with peaks in 2019 and 2022. Citations began in 2011, with a peak of 180. 3) *By fields of knowledge:* There is a prominent presence of 'music' (third place) and 'educational research' (tenth place) in the 'audiovisual music experience' category. 'Music' is also in third place in the 'perceptions' and 'audiovisual music experience method' categories. Lastly, 'educational research' is in seventh place in the 'audiovisual music experience method' category.

#### *4.2.1 Objective 3 results: Content analysis 2*

Performed with MxQda to understand the relationships, specificities and connections between music and the other elements and categories, selecting the topics and how often they appeared. Regarding 'typologies', the WoS had 'perception' in 55.95% of the results, 'aural perception' and 'musical perception' in 7.14% and, more specifically, 'the perception of audiovisual music' in 2.38%. Regarding 'domains', we used a matrix of codes to obtain the main categories, which were 'music' and 'perceptions', in 88 documents.

#### *4.2.2 The connections between the chosen codes*

They were obtained using a single-code model. By typologies and number of connections, the highest categories were 'tone' with 166 results and 'musical perception' with 77. The lowest positions were held by 'anxiety', 'pain' and 'health'. Specifically, from highest to lowest: 'tone' is connected to 'musical perception', 'emotions', 'the perception of movement' and 'audiovisual music'; 'aural perception' is connected to 'tone', 'the perception of audiovisual music' and 'musical perception'; 'emotions' is connected to 'musical perception' and 'tone'; 'the perception of audiovisual music' is connected to 'tone' and 'aural perception'.

#### *4.2.3 Method*

The connections were obtained by comparing documents with a single-code model. This allowed us to obtain a map and analyse 'music', 'tone', 'perception' and 'method'.

#### **4.3 In conclusion**

In this way, we have been able to verify the great difference in interests that exist in audiovisual themes related to music, among students, and the interests of scientific society. We have verified that scientific studies are interested in experience, followed by perceptions and finally by research methods. The analysis of the fields of knowledge indicates that the most prominent are occupied by neuronal and psychological aspects. 'Method' ranked 8th (being the most prominent), and, very importantly, 'music' and 'education' do not appear. Regarding concepts, five appear as the main ones, which did not include 'method', 'music' or 'education'. About their domains and relationships. The domains observed by large categories correspond to musical

*Audiovisual Music from the Audience's Viewpoint DOI: http://dx.doi.org/10.5772/intechopen.111567*

**Figure 6.** *Document portrait of one of the experiences used in conducting the study.*

characteristics, perception and music. Regarding the rest of the categories, music maintains relationships in order of importance with emotions, the perception of time and elements such as the perception of movement, time, anxiety and pain. The second content analysis has allowed us to observe through various lenses simultaneously and successively; the presence, domains and relationships of the audiovisual from the music side, covering a territory made up of 88 documents, synchronously analysing its cognitive, emotional, narrative and musical elements from the perspective and sum of each of its components and experiences (**Figure 6**).

#### **Acknowledgements**

Project "Television as a sound habitat. Study of the effects of the soundtrack and its audiovisual narratives in childhood" of the R + D + i call of the Ministry of Economy and Competitiveness of Spain (MINECO) (EDU2012-36404).

### **Conflict of interest**

The authors declare no conflict of interest.

### **Author details**

Amparo Porta-Navarro University Jaume I, Castellón, Spain

\*Address all correspondence to: porta@edu.uji.es

© 2023 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

*Audiovisual Music from the Audience's Viewpoint DOI: http://dx.doi.org/10.5772/intechopen.111567*

#### **References**

[1] Pérez Gutiérrez M. El Universo de la música. Madrid: Sociedad General Española de Librería, SA; 1980. p. 360

[2] Barba AE. Arte y mito: manual de iconografía clásica. Silex. 2008;**651**:456

[3] Benjamin W, Aguirre J, de Alba D. La obra de arte en su época de reproductibilidad técnica in Discursos interrumpidos. Vol. 1. Madrid: Taurus; 1973. pp. 14-57

[4] Porta A. Modos de escucha televisiva en Música y audición en los géneros audiovisuales. In: Gustems J, editor. Música y audición en los géneros audiovisuales. Vol. 13. Barcelona: Edicions Universitat; 2014. pp. 53-67

[5] Morley D. Television, Audiences and Cultural Studies. NY: Columbia University Press, Routledge; 2003. p. 334

[6] Porta A. Músicas públicas, escuchas privadas: hacia una lectura de la música popular contemporánea. Spain: University of València; 2007

[7] Sloboda J. La mente musical: La psicología cognitiva de la música. Vol. 1. Madrid: Antonio Machado

[8] Adorno T, Leppert R. Essays on Music. CA: University of California Press; 2002. p. 743

[9] Chion M. Audio-Vision: Sound on Screen. NY: Columbia University Press; 2019

[10] Hauser A, Tovar A, Varas-Reyes FP. Historia social de la literatura y el arte. Vol. 3. Madrid: Guadarrama; 1969. p. 1909

[11] Sternberg M. The world from the Addressee's viewpoint: Reception as representation, dialogue As monologue. Style. 1986;**20**(3):295-318. Available from: http://www.jstor.org/stable/ 42945610

[12] Barthes R. Semiología y lo Urbano. En La ciudad y el signo. NY: Prensa de la University of Columbia; 1986. pp. 87-98

[13] Dewey J. El arte como experiencia. Barcelona: Fondo de cultura económica; 1949

[14] Vygotsky L. The psychology of art. Journal of Aesthetics and Art Criticism. 1972;**30**:4

[15] Gómez-Ariza C, Bajo C, Puerta C. Determinants of musical representation. Cognitiva. 2000;**12**(1):89-110

[16] la Porta A. música como experiencia audiovisual en la infancia: entre el contar, entender y sentir. Revista Electrónica Complutense de Investigación en Educación Musical. 2022;**19**:107-119. DOI: 10.5209/reciem.74541

[17] Peirce CS. Peirce on Signs: Writings on Semiotic by Charles Sanders Peirce. NC: James Hoopes JSTOR, University of North Carolina Press; 1991. 294. Available from: http://www.jstor.org/ stable/10.5149/9781469616810\_hoopes,

[18] Eco U. Lector in fabula. Barcelona: Lumen; 1982

[19] Vygotsky LS. The Instrumental Method in Psychology. The Concept of Activity in Soviet Psychology. Cambridge, MA: MIT; 1981. pp. 135-143

[20] Perls F, Hefferline G, Goodman P. Gestalt therapy. New York. 1951;**64**(7): 19-313

[21] Porta A. The music that children listen to in movies, series and TV

documentaries. An empirical study on its meaning. International Review of the Aesthetics and Sociology of Music. 2018; **49**(2):311-332. Available from: https:// www.jstor.org/stable/10.2307/26844649

[22] López-Varela A. Génesis semiótica de la intermedialidad: fundamentos cognitivos y socio-constructivistas de la comunicación. CIC. Cuadernos de Información y Comunicación. 2011;**16**: 95-114

[23] Ekman P, Levenson R, Friesen W. Autonomic nervous system activity distinguishes among emotions. Science. 1983;**221**(4616):1208-1210

[24] Kleinginna P, Kleinginna A. A categorized list of emotion definitions, with suggestions for a consensual definition. Motivation and Emotion. 1981;**5**(4):345-379

[25] Vigotsky L. Principios de enseñanza basados en la psicología. Vigotsky: Obras Escogidas. Visor. Madrid; 1926

[26] Porta A, Herrera L. Music and its significance in children favourite Audiovisuals. Comunicar. 2017;**2017**: 83-92. DOI: 10.3916/C52-2017-08

[27] el Vigotsky L. desarrollo de las funciones psíquicas superiores. Barcelona: Crítica; 1979

[28] Albornoz Y. Emoción, música y aprendizaje significativo. Educere. 2009; **13**(44):67-73

[29] Vigotsky LS. La imaginación y el arte en la infancia. Vol. 87. Madrid: Akal; 200

[30] Luria AR. Sensación y percepción. Barcelona: Fontanella; 1981. p. 137

[31] Gombrich EH, Ferrater G. Arte e ilusión: estudio sobre la psicología de la representación pictórica. Debate. 2002; **2002**:415

[32] Piaget J. Intellectual evolution from adolescence to adulthood. Human Development. 2008;**51**(1):40-47. Available from: https://www.jstor.org/ stable/26763966

[33] Meyer L. Emotion and Meaning in Music. Chicago: University of Chicago Press; 2008

[34] Rossi P. Davidson y el pragmatismo clásico. Areté. 2007;**19**(1):119-132

[35] Rawjesky I. Intermedialität. Tübingen: Francke; 2002. p. 240

[36] Cohen L, Morrison Keith M, Lawrence M. Research Methods in Education. 5th ed. New York; 2000

[37] González Requena J. El discurso televisivo: espectáculo de la posmodernidad. Madrid: Cátedra; 1992

[38] Lévi-Strauss C. The bear and the barber. The Journal of the Royal Anthropological Institute of Great Britain and Ireland. 1963;**93**(1):1-11

[39] Bourdieu P. Men and machines. In: Knorr-Cetina K, Cicourel A, editors. Advances in Social Theory and Methodology. Boston, MA: Routledge and Kegan Paul; 1981. pp. 304-317

[40] Krippendorff K. The Content Analysis Reader. CA: Sage; 2009

[41] Strauss A, Corbin J. Grounded Theory in Practice. CA: Sage; 1997

[42] Porta A. Respuestas didácticas y preguntas de investigación en torno a la música transmedia: una combinación necesaria en la educación superior. Revista de Teoría y Práctica de la Educación Superior. 2021;**21**(16):51-60

### **Chapter 5**

## From the Crippled Devil to the Spectacular House: The Visual Representation of Intimacy in Painting, Comics, Film, and Television

*Carolina Sanabria*

#### **Abstract**

This work expands the literary topic of the interior of the home to its corresponding visual manifestations. Its main reference, the crippled devil, dates back to seventeenth century picaresque literature. It is the same century as the first visual representations of common, everyday life in Dutch painting. This study focuses on the visual and reflective evolution that is born with these paintings and takes the form of panoramic views in the graphic structure of the mid-nineteenth century vignette. A century later it evolves in comics, and then in cinema and television, emulating the structure of the literary movement. The content—the integration of characters in the same location—updates a satirical exhibition of the comedy of manners. However, television registers hilarious situations between its characters, which explains the success of the emblematic series, Aquí no hay quien viva, where the snooping of the characters is subrogated to the viewer, as Hitchcock did in Rear Window. Its protagonist, a revived crippled devil, goes beyond playful snooping. Thus, the representation of the interior of the home connotes vigilance if one thinks of the last of the houses built *ad hoc*: those of Big Brother. There, those who see can also take action, but it is a house designed to be looked at.

**Keywords:** intimacy, houses, exposure, films, TV series

#### **1. Introduction**

The visual representation of the main costumbrista topic in literature centers around the inside of the home, as it developed in the movement in Spain and France during the nineteenth century. Its origins date back to the seventeenth century and the popular, satirical figure of El diablo cojuelo, by Luis Vélez de Guevara. Certain authors (Cuvardic, Escobar Arronis) have supported this idea regarding the enigmatic figure that peers into domestic interiority and who penetrates the realm of literature until reaching an increasingly strong presence in modernity. However, the notion that

this chapter articulates is that it is an interdisciplinary motif. In fact, its presence in visual representation has not been studied with the same zeal. In this chapter, we do not neglect its verbal (literary) expression. Based on the above, we intend to examine the visual and narrative possibilities that imply a vision of the other in the intimate through a historical journey. First, in keeping with its literary origins, it is posed as a panoramic vision through supernatural fiction, then as a more or less subtle intrusion, until finally reaching a visual structure that appears in the journalistic movement of the second wave of modernity (Berman). However, it is curious that this form of treatment in art, which, in literature dates back to the seventeenth century with the crippled devil, coincides in graphic expression in a country situated further north in the continent: Dutch painting, whose exponents are the first to depart from traditional and canonical religious and historical topics to deal with issues of their most immediate daily life (Alpers, Todorov). And it is not until the middle of the nineteenth century that a vision is elaborated from a different frame, one which implies a more open blueprint: a diagram built from three floors plus the attic. This outline, clearly simpler than the minutely detailed Dutch pictures, is produced for the purpose of illustrating sketches of manners and soon after becomes autonomous in a structure also intended for a mainstream audience: comics. And yet it is not the culmination of the visual representation because its mass media replacements, cinema and television, take up this structure, as can be seen in Alfred Hitchcock's paradigmatic Rear Window in the mid-twentieth century and in the successful Spanish series Aquí no hay quien viva at the beginning of the twenty-first century. In these visual proposals, the structure is adapted so the characters can interact in a way that updates the satirical tone of its narrative and visual origins until it becomes the sitcom genre. From a narrative perspective, the physical closeness of the inhabitants who share the same block develops in part from the ramifications of the voyeuristic gaze, not only as a component of the show, but also for the purpose of prying. And even so, its scope continues to expand, since it is a fusion of dimensions that is brought even closer to paroxysm—where silent vision is in action—as can be seen in the dynamics of the Big Brother television contest. This program, whose innovative format combines entertainment and information, is still going strong in some countries as they inaugurate the twenty-first century, as well as its derivative TV programs which continue to this day. What is relevant here is that the house acquires an outstanding function, since it has been expressly built to be looked at and, in particular, for its temporary inhabitants, with which the function of entertainment borders on that of surveillance.

#### **2. Initial stages in folk imaginary: the crippled devil**

The motif of the crippled devil should be traced in the folkloric imaginary of the seventeenth century; its development, in turn, begins approximately a century earlier, in the Spanish picaresque novel. At that time, Castilian witches who were poor, needy, and illiterate were prone to repeating prayers to saints and spells to specific demons. The latter came to take the form of a cripple or a minor devil—as opposed to the great or major devils who actually had names: Lucifer, Satan, Barabbas, Belial, and Astaroth—and yet he was the best known and most often identified in literature [1]. In Spanish folklore, so given to jest, the crippled devil was therefore a character of a rather playful nature who explained the hierarchy of the range of these evil figures: dwarfed with respect to the devil, mischievous rather than perverse, he would soon be widely present in literature. He appears in the work

#### *From the Crippled Devil to the Spectacular House: The Visual Representation of Intimacy… DOI: http://dx.doi.org/10.5772/intechopen.110857*

of the playwright and novelist of the Golden Age Luis Vélez de Guevara, who took on the job of compiling proverbs and songs from previous popular literature in the satirical and moral novel *El diablo cojuelo* (1641) [2] (**Figure 1**). Incidentally, one outstanding element is the author's particular use of language; Georges Cirot has called attention to this through the presence of neologisms due to the author's predisposition to speak in popular language and avoid the then current trend toward scholarly and Latinized language ([3], p. 72).

The plot is very simple: it begins with a nobleman—Don Cleofás Leandro Pérez Zambullo—fleeing from justice in the rooftops of Madrid, late at night. Upon entering the attic, he accidentally breaks a vial and frees a crippled devil. In gratitude, the latter rewards him by showing him "the most notable thing happening at this hour in this Spanish Babylon" ([2], p. 80). Applying his magic, at first he takes both of them through the air, from where he raises the roofs of buildings to appreciate what is happening inside the houses, and then later they travel to other areas of the country, further away.

Diverse elements can be identified in Vélez de Guevara's text. On the one hand, the recovery of a character from biblical imaginary with fantastic characteristics whose very nickname calls attention to a physical defect. In the preface, Rodríguez Cepeda argues that this work serves as a metaphor for deformation and the grotesque, as the bad habits and vices that symbolize Spain ([2], p. 17). In this individuation, the physical deficiency is presented in relation to the picaresque tradition, and it will be elaborated in allusion to the figure of the *crippled devil* as one addressed to a type, that of subjects prone to prying into the privacy of others. In this sense, the text acquires importance as the first critical exploration of everyday life, namely, the intimacy of vices and other practices, in a satirical description of urban customs that will be reworked as a stylistic trend in bourgeois society as of the nineteenth-century *costumbrista* movement. Subsequent literary and film narratives have dealt with this figure in innumerable ways, but here it was singled out under the guise of this peculiar and even likeable popular character.

**Figure 1.** *Cover page from El diablo cojuelo (1641).*

Another outstanding aspect of the novel resides in the alternation between depictions: on the one hand, those of a certain dynamism—called *trancos*—of fast action; on the other, descriptive ones, almost static, so that, according to Alfaro, it is possible to identify two tendencies: "a descriptive one, which leads to pictures of customs, and another narrative one that leads to the adventure novel" ([4], p. 2). As an urban description, Vélez de Guevara's novel is loaded with notable irony regarding the customs and vices of the city's inhabitants in their daily practices, without posing a distance—or even a possible condition of superiority—in those it observes; instead the observed are not only defined, but matched to their onlookers.

This is by no means something exclusive to Hispanic culture, since French literature also has its corresponding *Le diable boiteux* (1707) [5] by Alain-René Lesage which was very successful in France. In the roman, the names of the characters and the essence of Velez de Guevara are preserved. Typical of Latino culture, as Rodríguez Cepeda indicates, the theme of the crippled devil will continue until the beginning of the twentieth century ([2], p. 39).

#### **3. Domestic interiors in Dutch painting**

Around that same century, in other geographical coordinates further north specifically in the Netherlands—science was developing technological instruments based on the optics of the lens which allowed for the description of new realities; one example is the microscope, which would facilitate advances in the study of entomology, among other subjects. This interest in detail also extends to the visual arts, of which painting deserves special attention. Dutch artists were the first to venture into the topic of the immediate domestic environment, dedicating themselves to portraying life inside houses, in what seemed to be a preference for representing bourgeois society in contrast to rurality and giving greater relevance to life within the family, not outside the home ([6], p. 38). This was, however, disruptive because it implied a clear distancing from Italian Renaissance painting, which Svetlana Alpers has defined in contrast to the description of the Nordic interior: "Dutch paintings are rich and varied in their observation of reality, dazzling in their ostentation of mastery, domestic and taming in their affairs" ([7], p. 23), as in Pieter de Hooch's painting, for example (**Figure 2**). They were paintings that contained depictions of the domestic and opted for a "description of visible reality, rather than as an imitation of significant human actions" ([7], p. 27). In fact, their acceptance was not easy, because the first considerations pointed to a lack of meaning understood as the narration of a text (biblical, historical, mythological), unlike what happens in Italian art. The Dutch represented scenes of daily life with anonymous subjects: mothers delousing children, tailors working, young people reading letters or playing the clavichord… This practice responds to the formation of a new, more individualistic and private society: If tradition can be said to uphold stories like common legacy, here, the Dutch painters and poets of the seventeenth century were the ones deciding the content ([8], p. 18).

The pictorial representations of the Calvinist Netherlands of the seventeenth century thus carefully portrayed a kind of "human truth" without implying the absence of morality. This made painting a genre of praise (domestic virtues, physical cleanliness as moral purity) and censure (of intemperance and pleasures of the flesh). In this regard, Todorov introduces the difference between the visual image and the written text in representation: Literary quality is incompatible due to an explicit didactic message, such as the one that instead seems to break off from another (later) form of

*From the Crippled Devil to the Spectacular House: The Visual Representation of Intimacy… DOI: http://dx.doi.org/10.5772/intechopen.110857*

**Figure 2.** *A woman nursing an infant with a child and a dog (between 1658 and 1660), Pieter de Hoock.*

description: that of the Criticism of manners in the nineteenth century. The presence of a moral lesson does not necessarily transform the image.

He says—it is superimposed on it, in contrast to the verbal matter that modifies the work from within. In other words, the way to create meaning is not the same in painting as it is in literature. Hence the prerogative of the visual sign over the written one ([8], p. 45). And the implication of the everyday in these works admits the existence of morality—the one that watches over and that the cripple mocks, and that is taken up again in the *sketches of manner*. However, their great love of life stands out, a completely voluntary gesture that transcends morality ([8], pp. 72–73). There is an esthetic sublimation of morality, that is, a hope that the beautiful can also reside in the domestic.

This interest in descriptive art is part of what Marshall Berman calls the recognition of an initial phase of modernity, which arises at the beginning of the sixteenth century and lasts until the end of the eighteenth century, in a kind of first contact, more intuitive than formal, where it is lived as an experimentation of modern life ([9], p. 2). It is an essential moment in history in which the formation of a bourgeois *ethos* as the dominant social class begins and in which existing powers are replaced, not only ecclesiastical but also monarchical. This change that slowly shows its imprint on art and literature begins its journey in specific geographical and sociocultural confines (Dutch painting, popular Mediterranean folk culture) but will expand to the rest of the world in the following phases of modernity.

#### **4. Voyeuristic** *Costumbrismo*

The second stage of modernity, according to Berman, begins with the great revolutionary wave—of political, economic, and cultural scope—that takes place at the end of the eighteenth century. It spreads across the continent and consolidates modernity

as a new vision of the world (*Weltanschaaung*), in a scenario progressively saturated by the demographic concentration implicit in modernization: steam engines, the beginnings of industrial automation, railways, huge cities, means of communication able to report on a massive scale ([9], pp. 4–5), such as newspapers and the telegraph. It is precisely from the journalistic genre associated with the emergence of newspapers that continuity with the activity of the crippled devil is produced: the literary movement known as *costumbrismo* or *sketch of manners*, where the devil is replaced by the journalist or chronicler.

This panorama provokes a change in cultural habits that involves the literary and journalistic traditions in which observation and description of the nascent bourgeoisie are proposed. The discourse that develops around descriptive, everyday life finds its sources in the process of formation of the article of customs which actually began at the start of the eighteenth century with the writings of Richard Steele and Joseph Addison (*essay* or *sketch of manners*), from where it would move to France (*tableau de moeurs*) and from there to Spain (*cuadro de costumbres*). Its appearance in those countries dates from the middle of that same century and it reaches its peak in the nineteenth century ([10], p. 29). The literary movement in question was interested in the representation of civil life as part of its ideological objective of artistically legitimizing bourgeois spheres of action ([11], p. 117) from a descriptive desire of contemporary daily life [12].

The first expressions of the movement date back to the French playwright, journalist, and *sketch of manners* writer Étienne de Jouy who, at the beginning of the nineteenth century, published several volumes of these sketches from which two of the texts that deal directly with the subject in question emerge; "Les six étages d'une maison de la rue Saint-Honoré" (1818) is about a visit in which the chronicler accompanies a friend who intends to buy a house, for which he came to describe himself as "nouvel Asmodé" ([13], p. 195), and another, later writing: "Une maison de la rue des Arcis" (1823) [14]. Both articles, in which the foundations of this motif are formally located, constitute a genre in which basically descriptive elements take precedence over narrative action, which is why they can be linked to what will later develop into journalistic documents. A few years later, between 1845 and 1846, the publisher Pierre-Jules Hetzel published a *sketch of manners* with the eloquent title: *Le Diable à Paris. Paris et les Parisiens. Mœurs et coutumes, caractères et portraits des habitants de Paris, tableau complet de leur vie privée, publique, politique, artistique, littéraire, industrielle*, which consisted of a history and geography of the capital city.

In Spain, the *sketch of manners* writers that correspond to Jouy—whose declared obsession includes imitation [14]—are represented by the writer Mariano José de Larra in one of his newspaper articles entitled: "Las casas nuevas" (1833), which refers to the vicissitudes of buying a new home for a friend. Throughout the narrative, Larra proves himself to be an acute observer of customs and the sociopolitical reality of the moment, with a fine handling of irony. There is also an outstanding unknown journalist who signs his articles with the pseudonym El Observador who collaborated with the editor José María de Carnerero in the *Correo Literario y Mercantil*. Among these texts is one clearly influenced by another French *sketch of manners* writer, Louis Sébastien Mercier, entitled "Costumbres de Madrid" (1828), which approaches various aspects such as the "portrait of the inhabitants, their public and private customs, their classes, their vices and virtues"; the observation of current affairs; the subject of his writings centered on "what is happening [at court] today"; the critical tone and "festive and decorous criticism"; and finally the moral intention, to correct customs ([15], p. 41).

#### *From the Crippled Devil to the Spectacular House: The Visual Representation of Intimacy… DOI: http://dx.doi.org/10.5772/intechopen.110857*

A few years later, under the pseudonym El Curioso Parlante, the writer and journalist from Madrid, Ramón de Mesonero Romanos, signed an article entitled: "Las casas por dentro," published in *Cartas Españolas* in July 1832; it later appeared in the collection *Escenas Matritenses* (1845). This text is narrated in the epistolary style, as the subtitle indicates, "from a curious Provençal to a curious man from Madrid." After the initial justification for the reasons that gave rise to the theme and once the area where the property of the friend who invites him to his home is located, the narrator goes into a lengthy description of an imaginary route: his entrance, the belongings, the layout of the rooms, the lighting. Finally, he goes on to detail the tenants who inhabit the place, basically with references to their trades: "Well, being who he was, he had two shops, and in them lived a hatter and a cabinetmaker; the shoemaker in the vestibule slept in a garret on the stairwell; a skilled fencer, in the mezzanine; a clerk and a merchant, in the main floor; a school teacher and a tailor, in the second floor; a hostess, a dressmaker and a clothes presser in the third floor; a military musician, an engraver, a translator of comedies and two widows occupied the attic [sic]; and there was even a mathematician who had published several observations on the heights of the globe in the small garret next to this" ([16], p. 59).

This tendency of the genre to clarify of what is described—the logistical arrangement of the house, the references to its inhabitants—together with the written and visual expression, becomes an influence that gives rise to a combination of media in the emergence of a topic that was then new.

#### **5. Image and word in domestic descriptions**

The meticulous description in the representation of a reference reality that is common both to painting in the Netherlands and to literary *costumbrismo*—an eminent perspectivist genre according to Escobar Arronis [15]—goes through some adjustments in regard to components such as temporality in narrative language. In homologation with the nature of the word, narration inevitably refers to past events ([17], p. 40), for which certain modifications are required that come to account for the fact that in *costumbrismo* time seems to be suspended, always in the present hence the predomination of the imperfect past tense and the gerund. Reason appears to evoke the procedure of *ekphrasis* with the obligatory variant that what it describes be filtered by a novel component that breaks into the descriptive genre: the humor that is so typical of its antecedent: the picaresque novel. As stated, Dutch painting does not deny virtues and vices, but rather transcends them in face of the existence of the world ([8], p. 73). Both in Dutch painting and in the *sketches of manners*, the main theme is the representation of the everyday, domesticity, but whereas the first sought the transcendence of what was represented, in the second there is, above all, a critique of habits and practices—as will also be present in the caricatures of nineteenth-century France, with Grandville and Daumier. The obvious difference resides in the fact that in Dutch painting everyday gestures serve to illustrate moral principles ([8], p. 75), while in the written expression of literary *costumbrismo*, the tendency is oriented toward exhibition and criticism which takes the form of jest in the texts of the Golden Age and in the moral criticism of the nineteenth century. The situations described in detail can lead to absurd circumstances that, if subjected to irony, demonstrate less seriousness and investiture; since its origins in Classical Greece, comedy was born as a devalued genre. Likewise, the complementarity between written and visual texts is reinforced by the fact that customary discourse is often accompanied by engravings. As early as in seventeenth-century editions, these discourses were complemented by images, such as the one illustrated by the artist Napoléon Thomas in 1707 for *Le diable boiteux*.

It is appropriate now to dwell on a couple of illustrations regarding Lesage's novel. The first is Thomas's image of the intruders who appear to levitate furtively in the same space, not through a window, as if they were glimpsing the libidinous practices of the inhabitants of an urban interior, where they share a privileged position, as in the front row of a show (**Figure 3**). From here, anonymous daily life is presented as a spectacular act, worthy of being viewed. The group's superior and invisible arrangement also indicates the optics through which they contemplate minor miseries. In Thomas's work, this particular distinction entails integrating in the same space the central role of the snooper with the object of contemplation. Although curiosity is proverbially related to the figure of the reader in the narrative (prose) genre, it can also be considered extensible to one of the forms in which visuality derives: the voyeur. In *costumbrista* literature, this function is delegated to the cripple and to Asmodeo. On the other hand, there is a greater abundance of representations of the crippled devil (engravings in biblical or literary editions) than of the act of vision itself, due to the fascination that the fantastic figure generated. For a relative identification or coincidence with the perspective of Asmodeo and Cleofás, the hypothetical corresponding image would have to present a zenithal view of the visual space that is very hard to execute, as evidence that the perspective of the characters must adapt to the two-dimensional characteristics of the representation implemented later.

The other image from the French edition, by an anonymous author, from around 1737, shows an in depth view of the different floors which allows them to be contemplated from a different angle than that which will later become convention: the frontal structure. The aforementioned French edition by Hetzel, from 1845 to 1846, *Le diable* 

**Figure 3.** *Illustration of Le diable boiteux (1707), Napoleón Thomas.*

*From the Crippled Devil to the Spectacular House: The Visual Representation of Intimacy… DOI: http://dx.doi.org/10.5772/intechopen.110857*

**Figure 4.** *Cartoon of Le diable à Paris (1845), Bertall.*

*à Paris* was also illustrated with engravings by various artists, including some by the caricaturist Charles Albert d'Arnoux, better known as Bertall, where the image of the property is based on the frontal arrangement that would become popular in the

following century.1 The structure allows us to verify a series of patterns common to these blocks of flats: the number of floors—between four and five—the presence of an attic and of a stairwell, which is either located on the right or on the left, and, of course, the existence of inhabitants (**Figure 4**).

In the same year, 1845, the edition of Escenas Matritenses which includes an article by Mesonero Romanos titled "The houses inside" was accompanied by an engraving attributed to an artist named Vallejo, which takes up Bertrall's compositional idea. In this regard, Escobar Arronis maintains that the image represents the cross section of a building which allows visualizing the entire location from the ground floor to the attic and all their respective rooms ([14], p. 44). In this way, the graphic text complements the possibility of visualizing scenes that are typical of the human types. Since Vélez de Guevara's novel it has been defined as social types due to their satirical nature: "a phantasmagoria of Quevedesque style in which the author satirizes human defects, especially those of the nascent petty bourgeoisie" ([4], p. 1). In this case, these types are defined by their trades and located in each of the rooms they inhabit that inhabit each of the rooms as defined by their trades: the hatter, the cabinetmaker, the shoemaker, the fencing master, etc., but from a strikingly different angle than that of Vélez de Guevara's character: "Instead of raising the roof, as the Crippled Devil would have done, the artist has to remove the façade of the building so that one can see what is happening inside" ([14], p. 44).

This will become the frame (horizontal, of the whole) that is established for the convention of the representation of the interior of houses in subsequent and contemporary graphic histories. Without intending to revive the phantasmagorical illusion of the *précinema*, Cirot has referred to the speed, variety, and density of vision in *El diablo cojuelo* as "cinematographic." The internal reason for this speed of action and contemplation is concentrated in the expression *there is no time to waste*: "Dans le Tranco II il nous fait jeter un regard rapide su des quantités de tableaux qui se succèdent à vive allure. C'est qu'il *n'ya pas de temps à perdre.* Il faut profiter de la nuit et du sortilège. On voit défiler trente-deux de ces 'cinematographies'" (emphasis added) ([3], p. 69). It is typical of the language of comics, whose graphic simplicity has allowed the display of extravagant or impossible worlds that do not pose technical problems for the cartoonist to solve with his pen ([18], p. 44).

#### **6. The satirical alternative: comic strips in the twentieth century**

The representation of this structure, similar to a fish tank, is present not only in the film genre but also in drawing, and it will evolve in the comic or graphic novel genre. Basic equivalences are maintained: the descriptive function, the simplicity of the structure (in contrast to the detail of Dutch painting), and the possibility—carried out in previous satirical and critical texts—of developing a satirical vein, as in humor. In later, more popular variants will give way to more recent televisión series with choral plots as of the beginning of the twenty-first century in Spain.

The first attempts correspond to a format that begins its evolution in newspaper spreads. In 1902, Joaquim Xauradó published "A house on Christmas Eve" in the magazine *Blanco y negro*. A few decades later, in the late 40s and early 50s, the

<sup>1</sup> As suggested in the third season when one of the characters who is in a situation of financial and personal instability, Andrés Guerra (Santiago Ramos), moves to the uncomfortable attic—just like the aforementioned Vázquez in Ibáñez.

#### *From the Crippled Devil to the Spectacular House: The Visual Representation of Intimacy… DOI: http://dx.doi.org/10.5772/intechopen.110857*

cartoonist Manuel Vázquez Gallego took the same structure as a basis for "A day in villa Pulgarcito" (1959) from his comic *Pulgarcito* published by Brugera. It was a macrostructure matched to the size of the page where the cartoons were distributed corresponding to houses and their respective characters: Mofeta, Jimmy Pintamonas, Gildo…. It is the same model on which Francisco Ibáñez would base his famous comic magazine *Tío Vivo* with the strip *13 Rue del Percebe* (1961–1970), which had notable commercial success. Its title, which refers to the physical address where the property in question is located, pays tribute and gives continuity to the influence of nineteenthcentury French draftsmen. Ibáñez strengthens the idea that timidly emerged with Gallego by combining the frontal plane that Bertall inaugurated but solving its rigidity with a short extension or small expansion to the side that creates the effect of depth, in addition to the addition of other spaces, such as the elevator and the roof next to the attic.2 It should be remembered that the production of these images of urban interiors was not exclusive to the old continent, since around 1949 a renowned artist dedicated to caricatures and architectural drawing based in the United States, Saul Steimberg, published *The Art of Living* with drawings of urban interiors. One of the best known drawings from this collection includes a variant of the established structure: It is divided into three vertical strips, the first describes the building from an external, frontal perspective, and the rest is dedicated to the interiors with a cross section, maintaining the tonic of its European predecessors.

From the three, an evolution can be seen that goes from the simplest, with a more basic frame, to the more complex, with a greater amount of detail, of defined characters, the addition of color, etc. But all of them offer the possibility of being read horizontally or vertically. One particularity is that the shape is adapted to the support, since each page corresponds to a vignette of one floor of a block or community of neighbors, each of which has a window. On the ground floor, there is a grocery store, the concierge and the elevator and on the three upper floors, the neighbors who inhabit the building: a veterinarian, the owner of a pension, an old woman, a scientist, a thief who steals useless things, a woman with her naughty children. Ibáñez pays homage to Vázquez Gallego by dedicating the rather secondary character known as Vázquez, the defaulter who lives on the roof, to him.

However, this abrupt independence is not always present because, although they are cells with autarchic conflicts, what happens in each one is sometimes linked by common events that affect some of their neighbors or the entire block. There is one paradigmatic case in *Playtime* (1967), where the director Jacques Tati raises the clash of technique that is imposed in an increasingly technological world, a futuristic Paris made of glass and steel, in which this scenario is parodied. In it, the astonished protagonist looks at a new construction of flats in the style of comic cells where the circumstances in which some residents live find its respective prolongation in the window of the block next door. In a certain way, the satirical disposition works as an update of the types of characters that distinguish the *costumbrismo* of *El diablo cojuelo* in an expansion of the market given by the medium—as with popular folklore. And yet, despite the commercial success, it seems that its author found *13 Rue del Percebe* too claustrophobic—the condensation of a fixed box for each character on a single

<sup>2</sup> Jeff manages to involve both of them in the scenes he watches—especially his girlfriend who goes so far as to enter Thorwald's apartment—which is a notable difference with the mere panoramic expectation of Don Cleofás and the crippled devil. It could be said that the latter way is more impersonal, documentary, if you will. This leads us to suggest that the trend in the transition from the literary to the audiovisual induces a conflict where there is an interaction in the narrated world.

page—which is why in 1987 he created *Rebolling Street*, where he repeated the formula but in double the pages and with a greater number of characters.

This representation, which has gained sediment from the mental image that originates in the *costumbrista* text, is reinforced as a result of the subsequent proliferation of supports and projection of forms that develop at an iconographic level. Thus, one would be in a later stage of the production of the practice in the genre that takes the big screen to the small screen, that is, to domesticity itself, with a device that over time has been equally domesticated [19]: television.

#### **7. The new cripple**

As expected, the audiovisual invention will also colonize this trend for the visualization of intimacy, in an expansion of reflections on the new medium such as the exercise of the gaze and the spectator. The quintessential film that condenses this situation is *Rear Window* (1954) by Alfred Hitchcock, which adapts Cornell Woolrich's story "It Had to Be Murder" (1954) and questions the integration of intimacy and spectacle. In fact, it begins with the opening of a curtain, in a theatrical style, to show the action that takes place behind the windows of other apartments, inhabited by neighbors who are part of the show. The season of the year, summer, with temperatures of 92°F makes it that much easier, since most of the characters open their windows and even sleep on their balconies; all of this invites the protagonist, Jeff, to peer (**Figure 5**). But he becomes so obsessed by this dangerous three-dimensional screen that he is even willing to risk his life.

The film also works as a representation of everyday life equated to the artistic world: Stam and Pearson argue that this vision constitutes a social microcosm that functions like a *tableaux vivant*: "The architectonic stylization and painterly artifice of the set betray what is transparently a studio product […] Virtually all the members of this cinemato-graphicum mundi are artists or actors, or are engaged in an entertainment-related profession" ([20], p. 200): The workers on the scaffolding are

#### **Figure 5.** *The protagonists with Alfred Hitchcock in the scenario of Rear Window (1954).*

#### *From the Crippled Devil to the Spectacular House: The Visual Representation of Intimacy… DOI: http://dx.doi.org/10.5772/intechopen.110857*

reminiscent of the carpenters who prepare the scenes, of the characters, the bachelor is a pianist, Miss Torso is a dancer, the neighbor who lives on the ground floor is a sculptor, even Thorwald himself is a jewelry salesman, Lisa works for a magazine…. This is the reason why for Román Gubern, it becomes a *theatrum mundi*—where the central theme is love or the lack of it—([21], p. 198) which one attends from the window that offers a view to the show, and its protagonist, like the crippled devil, has a wounded leg and dedicates his time to looking out the window (**Figure 6**). In this case, he does not resort to the fantastic element of flight, but his privileged vision, despite being exercised in the rear wing of the building, happens in a place that lends itself to the contemplation of his neighbors; shortly after, he adds, accessories to expand his vision, such as the camera which allows magnification and a telescope. Hitchcock's contribution to the literary text consists in enriching the protagonist, or rather, the complexity of his conflicted psyche in a personal situation: Jeff sees his hitherto life of risk and adventure in jeopardy for one of stability phases of life and marriage (boredom, in his own words)—which is underlined in the windows that allude to the different phases of life, both for couples and for individuals. Therefore, his activity stems from a source of anxiety that he manages to infect to his interlocutors—his employee, Stella and his girlfriend Lisa—unlike Don Cleofás.2 And it could not be otherwise because the visual drive is an undeniable part of our instinctive nature; in his famous interview with Truffaut, the English director stated that 9 out of 10 subjects who have the option of peeking at what is happening behind a window, do so ([22], p. 79). Without it being possible to suggest an influence between these two productions, so far apart geographically and temporally, this coincidence accounts for a common cultural pattern that is maintained and expressed with the inevitable variants. What seems to be behind it is precisely the animal nature of human beings.

The vision does not emanate, as with Vélez de Guevara, from above, zenithally—except for a quick allusion in one of the initial moments in which, from the aerial perspective of a mischievous helicopter pilot, the camera zooms in to look at some young women sunbathing on the rooftops. *Rear Window* proceeds by identifying with the spectator, leveling the viewer's gaze to that of the protagonist. Without implying a cross section in the structure of the block, as in the illustrations by Bertall and Vallejo, the English director manages to build an equivalent image from a level

**Figure 6.** *James Stewart with his broken leg in Rear Window (1954).*

shot, positioning himself as a character who obsessively looks out the window, and this poses an analogous situation with filmic expectation. The composition of this shot has been repeated by other directors, such as Jacques Tati in *Playtime* or Pedro Almodóvar in *Kika* (1993). In those cases, the action takes place in large western cities: Manhattan, Paris, and Madrid, which are so densely populated that the housing trend consists of vertical constructions.

But in *Rear Window*, which deals with the functioning of expectation, the fishbowl structuring of the dwellings functions as a possibility for spectacle, and there is a consensus among critics that it constitutes a metaphor for cinematic expectation: immobility, darkness, attention embedded in the facing image, that is, the screen of entertainment, namely, the reverse shot whose basic structure will be popularized in graphic novels.

#### **8. From comics to television**

The aforementioned visual structure of the exterior of a building divided into four segments where the action takes place is directly indebted to cartoons or comics. Events happen in a building with a sectioned façade that make the residential interiors visible, in the style of a doll's house (whose origins are not always related to children's games). In the seventeenth century, the latter were display cabinets full miniature rooms on a scale of 1:12, with all the architectural details and minuscule household objects; they were collected by a few matrons residing in Germany, England, and Holland. In some European capitals, these types of constructions of flats that house nuclear family groups are very common.

In the twentieth century, the small screen resumes this graphic tradition with notable products that also draw on comic book sources. One striking example is the successful series *Aquí no hay quien viva* (Antena 3, 2003–2006)—created by Iñaki Ariztimuño and the siblings Alberto and Laura Caballero—which continued (actors, rhythm, idea, only this time in a building located in an area far from downtown) in *La que se avecina* (Tele5, 2007-)—created by the Caballero siblings and Daniel Deorador. Its format can be thought of as an adaptation from comics, both in the configuration of the character types and in the structure of the frontal building.

The presentation of the opening credits is in keeping with the television of the time: It begins with an aerial shot of the Earth, which is crossed by a satellite from which, in a supersonic *tracking shot,* the camera enters the orb and penetrates until it lands on the mentioned block—in a convenient technique developed in nineteenthcentury narration in which a place is chosen at random to begin the narration. In this introduction, a view of the building from the outside is shown and then the main characters appear to the sound of a catchy musical rhythm. This opening fragment has a double purpose: On the one hand, to give credit to the actors who represent the characters and on the other hand, to present them on their respective floors. In this way, there is a general perspective (the block) that in a close-up gives rise to other more individualized viewpoints of cells that look like floors and are divided by strips that represent walls, evoking the graphic tradition of comic strips. In the development of the fiction, this structure is also evident in the elements of the shots from the horizontal or vertical panning, as appropriate. This structure is clearly directly indebted to the aforementioned cartoons by Bertall, Xauradó, Vázquez Gallego, and Ibáñez.

As in the previous comics we have discussed, the script for *La que se avecina* is set in a specific block in a fictitious street in Madrid: Desengaño 21. The building has three

#### *From the Crippled Devil to the Spectacular House: The Visual Representation of Intimacy… DOI: http://dx.doi.org/10.5772/intechopen.110857*

floors above the reception area, the entrance, and a video club. Its façade functions as the framework in which the plot of the sitcom unfolds, where two factors are essential: the physical proximity of the dwellings and its characters (inhabitants and visitors). As expected, the latter are highly divergent and contrast with each other: Some gossipy old ladies, a gay couple, a separated, promiscuous father, a family that is conventional only in appearance, some unruly single women and a young recently married couple. It is, in any case, a structure that is obviously related to its own architectural configuration—which, at a characterological level, has no protagonists, but rather is posed as a choir. This circumstance, in turn, facilitates the hiring and firing of actors for each season, and favors contracting contingencies that allow wide flexibility in the script.

The plot is conceived for a particular configuration of human types, as Balzac would say, who inhabit each dwelling. In a way, these human types refer to what Robertson Wojcik calls *the apartment plot*, which designates a space that arises from the opposition of the house as a private, family, independent unit as the key signifier of urban life after the American post-war period—until its zenith in the mid-1970s. However, *the apartment plot* refers to a block or group of houses that imply a different notion of units, whether they are owned or rental units, with multiple dwellings in a single location of subdivided houses [23]. Once again it is necessary to emphasize the aforementioned physical proximity as decisive in the adventures that lend themselves to humor or satire.

It has already been mentioned that, in diegetic terms, proximity to neighbors has consequences. To begin with, the secrecy that is so typical of intimacy is lost, and in this sense, *Aquí no hay quien viva* and *La que se avecina* can be thought of as a caricature of the panorama proposed by Pierre Nora: "In large building complexes, constructed with concrete and without any frills after World War II, family secrets can hardly be kept" ([24], p. 162). In these properties, the acoustics of the interior patios favor the dissemination of speech more than actual communication: commentaries and gossip among the neighbors in what is known as "Radio Patio". This constitutes another sample of a space for socialization integrated into individual dwellings, similar to the neighborly interaction of other residencies, such as Monsieur Hulot's in *Mon oncle* (1957) by Jacques Tati; in order to access his own home, Monsieur Hulot must travel through an intricate path that forces him to socialize with his neighbors.

Lastly, we still have not mentioned that in the structural composition of the building there is a final level which, in the initial external image, corresponds rather to two upper spaces that timidly appear on its sides: the attic.1 This is not an innocent place: Traditionally it is the space where old objects of little use are destined, as a metaphor of those who are excluded from the dynamic core. It is the same sense that the literary theorists Sandra Gilbert and Susan Gubar propose from a feminist orientation in the title of their well-known book [25]. But it does not always have a negative scope: Finding oneself in a corner (the upper one) becomes a way, according to Bachelard, of filling the void with human life ([26], p. 130), of managing to dominate the entire property with its organic presence. And incidentally, it also contrasts radically with another symbolic sense of that same location: the luxurious *penthouse* of the playboy James Bond, although in this case it is a room linked to the heights that are metaphorically associated with his socioeconomic condition.

#### **9. Television homes: from spectacle to surveillance**

The metafiction of spectacular reception as posed in *Rear Window* and in graphic cartoons expands to another representation of the domestic space in a different

medium—television—as of the emergence of the genres that arose from the macro-genre of the *infoshow.* This category is reminiscent of Umberto Eco's types, which integrate news and fiction ([27], p. 151) in what he called neotelevision. Infoshows contained the reality shows (1999) that would change television in the twenty-first century.

Although there were isolated precursors in the 70s and later, the quintessential space is the Dutch program *Big Brother* (Endemol TV) (**Figure 7**) which had editions all over the world and changed the way we consume television. It was a contest in which, in contrast to what happened in previous television programming *paleotelevision* for Eco—the anonymous contestants who appeared went on to became famous—and television became a medium ruled by "nobodies," according to Prado *et al.* ([28], p. 201). In fact, the process began earlier, in the massive open call which attracted thousands of young people, and during the show's broadcast in the daily domestic life of a house equipped with cameras and microphones that recorded uninterruptedly for a period of 24 hours during approximately three by parallel transmissions, such as weekly galas, daily summaries, and talk shows on the same channel where the show was discussed, which all ended up referring to the essence of *neo-television* as pointed out by Eco: self-referentiality [27].

In the non-visible process of election, the criterion was the coming together of contrasting personalities conducive to clashes—as in the case of comics and sitcoms such as *Aquí no hay quien viva*. Then, different situations were provoked, such as the successive expulsions that resulted from the contestant's own "nominations" and the audience's weekly votes (an audience who was encouraged to identify with their favorite) until the show ended with the last broadcast. In short, we were faced with an exasperated domestic daily life based on a spectacle that, from the beginning, reached the character of a media event.3

The house—or the set built as a house—that temporarily accommodated the contestants was typical of a globalized civilization (with a sybaritic equipment of lubricious pleasures: swimming pool, *jacuzzi*, sauna, the suite). However, not everything was so routine and the experience, like the house, was actually a simulation [29] not so much due to the presence of technical equipment as because of its isolation: the contestants could not leave the house, there was an imposed restriction on any visits from family or friends, as well as on any printed, electronic or digital media, and devices, contrary to what has become inevitable in our current domesticity. Such a panorama contributed to the constitution of a hyper-communicated society of

**Figure 7.** *One of Big Brother's promotional logos.*

<sup>3</sup> The first edition in Spain, and more specifically the final episode, reached a 70.8% audience share, a historical record. The fact that it was on the air for almost two decades also points to the same thing.

#### *From the Crippled Devil to the Spectacular House: The Visual Representation of Intimacy… DOI: http://dx.doi.org/10.5772/intechopen.110857*

the twenty-first century based on the spectacle of human miseries. The intimacy of everyday life typical of a *reality show* has no other way to present itself than through its own simulation ([30], p. 54) because that is what used to be the object of the spectacle. And the space that undoubtedly best represents this spirit was the room paradoxically called *the confessional*, where contestants were called to explain their sentimentality before a faceless voice that questioned them.

From there an infinite variety of possibilities is gestated for television shows that play with the illusion of reality. Location begins to acquire a decisive role as receptacle for goods and services offered as merchandise by an enthusiastic consumer society, where one's dwelling is shaped as a mimetic objectification of its owner. Examples abound: *Extreme Makeover: Home Edition* (2000–2012) about a team of designers who spend a week reconstructing a demolished home; *House Hunters* (1999–) about the process of moving and buying a house; *Brothers* (2011–2015) about two brothers who help families find and repair the house they want; *Love It or List It* (2008–) in which the presenters remodel a house the owners are not comfortable with. Although these programs are made in the United States or Canada, other countries have developed their own, such as *La casa de tu vida* (2004–2007) in Spain or *La casa* (2016) in Chile, which deals with the construction of a home that in the end it will be the prize for one of the participating couples. The house, then, acquires a leading role along with the contestants.

And as can be directly inferred from *foundational reality—*in the present day and not only on television*—*the idea of surveillance is inseparable from transparency. This notion was already present in the block of flats glimpsed by the protagonist of *Rear Window,* where each neighbor dwells in his own cell, like a prisoner, similar to a panoptic structure as proposed by the British philosopher Jeremy Bentham at the end of the eighteenth century [31]. But Jeff is not a jailer and merely contemplates other people's intimacies as a recreational act to avoid the boredom of passivity. Clearly, the initial entertainment given by this visual panorama transforms into an obsession for the character who Hitchcock and his screenwriter, John Michael Hayes, charged with high psychological complexity. The protagonist's gaze holds a nuance of spectacle that will later lead to surveillance.

Byung-Chul Han argues that in modern Western society surveillance is encouraged as an aspiration ([32], p. 67), in contrast to the separation between public and private that still held sway in the eighteenth century. For Juan Martín Prada, surveillance cameras, with their ways of seeing and influencing the behavior of those who are observed, have assumed a palpable role in creative practices for decades, producing a way of working that could be called "surveillance camera aesthetics" ([33], p. 152). The articulation between spectacle and intimate life has been developing since the emergence of generalist television channels in the early nineteenth century. And it can also be appreciated in cinema, as in Mike Figgis's *Timecode* (2000), where the screen appears divided into four, showing continuous shots and simultaneous action converging, as Prada points out, in a visual experience evocative of the monitoring of various characters in a security camera guardhouse ([33], p. 152). As in these places, the viewer finds himself, like Bentham's watchman, in a position of power associated with surveillance, in one case by intervening (voting) and in the other, by changing the channel.

In *Big Brother,* the contestant who manages to stay until the last day is the winner, but like the rest of contenders, he or she must do everything possible to avoid being evicted—in a kind of perverse anticipation of the reality brought about by the global economic crisis (2008–2014). This involved cooperating, cleaning, consoling, crying, lying, manipulating, offending: a circumstance that metaphorizes the human

condition that is extrapolated to one of the basic needs of citizenship: the right to decent housing, although in this case because it is mediated by spectacle, the dignity factor is the first to be lost.

#### **10. Conclusions**

The *topos* of the representation of domestic interiors as it is appreciated in the narrative and visual arts of today has a base that can be traced back to the seventeenth century on two sides: one of them is visual, the painting of the Netherlands. At the time of its appearance, it implied a break with the post-Renaissance tradition by taking distance from the source of creation linked to history, religion, that is, to supranational issues, in order to deal with what concerns the individual, the private, the domestic, in short, the mundane. Almost at the same time but in another geographical context in the same continent, a fantastic character associated with the phenomenon of scrutinizing the daily lives of the inhabitants of urban areas appeared in popular folklore and later in literature. In the nineteenth century, this interest was resumed in the *costumbrismo* literary movement, which put the accent on meticulous, detailed and precise description. It is a form of constant vision, beyond cultural and historical differences, on the traditions that stand at the very foundation of creation.

In the development of representation, this *topos* has gone through phases that incorporate variables that entail the suspension of disbelief with which an attempt is made to identify the viewer in narration: from a devil who is more mischievous than evil and has the ability to fly over and raise roofs, to the obsessive photographer who makes use of his resources, and even including the most recent manifestation in television series: the elimination of the fourth wall—specifically notable in sitcoms like *I Love Lucy (*1951–1057), *The Big Bang Theory* (CBS, 2007–2019) and even in the introduction of *The Simpsons* (20th Television, 1989–), where the characters appear in front of a location (a wall, a television) that represents the camera. This then gives way to the representation of domestic daily life in different media, forms, genres, characters. The present work has tried to demonstrate strategies of visualization of domestic interiors through history until reaching an approach in which the private is transformed by virtue of public entertainment and, ultimately, by the spectacle. However, it does not end there, since in recent times this process entails the recreation of privacy, anonymity, depersonalization, and lately a new manifestation: constant vigilance over the subjects under observation. It is the reason why *Rear Window* was so disturbing: because identification with Jeff was equal to not just looking, but being looked at by the killer.

In short, due to its transdisciplinary nature that includes narrative fiction, architecture, interior design, the stories that are part of this *topos* of voyeurism around a house, tend to favor anonymous, every-day, domestic stories. In a certain sense, some practices, such as the comic strip, insinuated a new return to *costumbrismo* which also allows a rearticulation that goes beyond the descriptive model of the *topos*—subjected to the limitations of picturesqueness—from which it opens to other spaces, such as those that derive in forms like melodrama in contemporary practice, with variants that imply complex relationships of bourgeois life, not always equivalent to boredom.

Translation by Carla Ros.

*From the Crippled Devil to the Spectacular House: The Visual Representation of Intimacy… DOI: http://dx.doi.org/10.5772/intechopen.110857*

### **Author details**

Carolina Sanabria University of Costa Rica, San José, Costa Rica

\*Address all correspondence to: csanabriacr@yahoo.com

© 2023 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

### **References**

[1] Moliné-Bertrand, A. Conjuros, Oraciones y Brujería en Castilla, Siglos XVI y XVII, 2005. Available from: https://www.banrepcultural.org/ biblioteca-virtual/credencial-historia/ numero-103/conjuros-oraciones-ybrujeria-en-castilla-siglos-xvi-y-xvii [Accessed: May 10, 2019]

[2] Vélez de Guevara L. El diablo Cojuelo. 6th ed. Madrid: Cátedra; 2011

[3] Cirot, G. Le procédé dans El diablo cojuelo. Bulletin Hispanique, Tome 45, 1: 69-72; 1949

[4] Alfaro, G. El diablo cojuelo y la picaresca alegorizada, Romanische Forschungen, 83, 1: 1-9; 1971

[5] Lesage A-R. Le Diable Boiteux. Paris: Didot; 1877. (exported from Wikisource February 2023)

[6] Bozal V. La pintura holandesa del siglo XVII y los orígenes del mundo moderno (III). Aula Abierta. Boletín informativo. 2002;**316-317**:33-41

[7] Alpers S. El arte de Describir. El arte holandés en el siglo XVII, Madrid: Hermann Blume; 1987

[8] Todorov T. Elogio de lo Cotidiano. Barcelona: Galaxia Gutenberg; 2013

[9] Berman M. Todo lo sólido se Desvanece en el aire. La Experiencia de la Modernidad. Madrid: Siglo XXI; 2013. Trans. Andrea Morales Vidal

[10] Escobar Arronis J. Costumbres de Madrid: Influencia de Mercier en un programa costumbrista de 1928. Hispanic Review, (Pensilvania). 1977;**45**(1):29-42

[11] Cuvardic D. El tema de las casas por dentro. De El diablo cojuelo a los artículos de costumbres, el cómic

y el cine. Káñina, Revista de Artes y Letras, (San José, Costa Rica). 2006;**XXX**(1):117-131

[12] Bustos Tovar JJ. Diccionario de Literatura Universal. Madrid: Anaya; 1985

[13] Jouy, d'É. L'Ermite de la Chaussée d'Antin, ou Observations sur les Mœurs et les Usages Français au Commencement du XIXe Siècle, Tome Premier. Bruxelles: Aug Wahlen et Comp; 1818

[14] Jouy d'É. Ouvres Complètes. Essai sur les Moeurs. Tome II. Paris: Imprimirie de Jules Didot Ainé; 1823

[15] Escobar Arronis J. Un tema costumbrista: Las casas por dentro en L'Hermite de la Chaussée d'Antin, El Observador y El Curioso Parlante. Revista Canadiense de Estudios Hispánicos (Canadá). 1976;**1**(1):39-47

[16] Mesonero Romanos R. Escenas Matritenses. 5th ed. Madrid: Gaspar i Roig; 1851

[17] Sánchez Noriega JL. De la Literatura al Cine. Teoría y análisis de la adaptación. Barcelona: Paidós; 2000

[18] Gubern R. De los cómics a la Cinematografía. Discurso de Incorporación a la Real Academia de Bellas Artes. San Fernando, Madrid: Editorial Biblioteca Nueva; 2013

[19] Márquez I. Una Genealogía de la Pantalla. Del cine al teléfono móvil. Barcelona: Anagrama; 2015

[20] Stam R, Pearson R. Reflexivity and the critique of voyeurism. In: Deutelbaum M, Poague L, editors. A Hitchcock Reader. Oxford: Blackwell Publishing; 1989. pp. 199-211

*From the Crippled Devil to the Spectacular House: The Visual Representation of Intimacy… DOI: http://dx.doi.org/10.5772/intechopen.110857*

[21] Gubern R. Máscaras de la ficción. Barcelona: Anagrama; 2006

[22] Truffaut F. Hitchcock. Madrid: Akal; 1991

[23] Robertson Wojcik P. The Apartment Plot. Urban Living in American Film and Popular Culture, 1945 to 1975. Durham & London: Duke University Press. p. 2010

[24] Ariès P, Duby G. Historia de la vida privada. In: 5. De la Primera Guerra Mundial hasta nuestros días. Madrid: Taurus; 2001

[25] Gilbert Sandra M, Gubar S. The Madwoman in the Attic The Woman Writer and the Nineteenth-century Literary Imagination. USA: Yale University Press;

[26] Bachelard, G. La poética del espacio. Argentina: FCE; 2000

[27] Eco U. La astrategia de la Ilusión. Barcelona: Lumen; 1985

[28] Prado E et al. El Fenónemo Infoshow: La Realidad Está Ahí fuera. Área 5. Revista de Comunicación Audiovisual y Publicitaria, núm. 6; 1999. pp. 197-210

[29] Sanabria C. Repensar Gran hermano 20 años después: la casa como simulacro. Zer, Revista de Estudios de Comunicación. 2018;**23**(45):185-202. DOI: 10.1387

[30] Ordóñez L. La realidad simulada. Una crítica al reality show, Análisis político. 2005;**54**:49-62

[31] Bentham J. Panopticon: The Inspection House. USA: CreateSpace Independent Publishing Platform; 2017

[32] Han B-C. La sociedad de la transparencia. Barcelona: Herder; 2016. DOI: 10.5209/CIYC.52976

[33] Martín Prada J. El ver y las imágenes en el tiempo de internet. Madrid: Akal; 2018

#### **Chapter 6**

## Transmedia Storytelling of Liangzhu Culture Inscribed Symbols

*Jinghua Guo and Weile Weng*

#### **Abstract**

Liangzhu culture is "the evidence of Chinese civilization for 5000 years" and has great historical and social values. As an essential material for studying Liangzhu culture, the narrative carrier of Liangzhu culture inscribed symbols has experienced continuous expansion from paper media to new digital media. With the advancement and integration of technology, virtual reality technology has also become a unique storytelling medium. The development of new media has continuously optimized the way of cultural communication and information dissemination. This research aims to explore the transmedia storytelling of Liangzhu inscribed symbols, combine the existing VR exhibition project, further promote the dissemination of Liangzhu culture, and ultimately strengthen the construction of Liangzhu inscribed symbols and enhance the influence of Liangzhu culture through new technology.

**Keywords:** Liangzhu inscribed symbols, transmedia storytelling, digitization, VR exhibition, visual transformation

#### **1. Introduction**

At the 43rd session of the World Heritage Committee in Baku, Azerbaijan, in 2019, the archeological ruins of Liangzhu in Hangzhou, Zhejiang Province, were inscribed on the World Heritage List, serving the world's most recognized material evidence of the "5000-year Chinese civilization". At the beginning of Liangzhu's inscription, Liangzhu culture was caught in the dilemma of low brand awareness, poor recognition, and lack of communication power. Liangzhu culture had not received attention commensurate with its value status. However, in recent years, the Hangzhou government has been strengthening the promotion of Liangzhu culture. On February 19, 2023, the government report of Hangzhou first mentioned "building a corridor of Liangzhu culture," which was upgraded from "Yuhang Action" to "Hangzhou Initiative," fully reflecting the importance that the Hangzhou municipal government attaches to Liangzhu culture.

In recent years, the dissemination of Liangzhu inscribed symbols has also become a new path to build the urban image of Hangzhou. The Liangzhu culture inscribed symbols with unique forms, and exceptional cultural value helped create the Liangzhu and Hangzhou symbols. Throughout the propagation of Liangzhu inscribed symbols, it has used various media for narrative, significantly since entering the digital world. Transmedia storytelling greatly expands the influence of Liangzhu culture. American scholar Henry Jenkins defined the concept of transmedia storytelling in his book *Convergence Culture: Where Old and New Media Collide* in 2003. Transmedia storytelling means the complete story text creates a unified, continuous entertainment experience with the help of multiple media platforms. In an ideal environment, each medium plays an irreplaceable role in the overall textual story [1]. At the beginning, it is linked to fiction genres and later to non-fiction genres. Then, key areas where transmedia storytelling is applied are in education and knowledge dissemination, the business world and branding [2]. Taking advantage of different media, the diversification of cross-media narrative channels has undoubtedly promoted the propagation and promotion of Liangzhu culture. The research aims to explore the cross-media narrative effect of Liangzhu culture inscribed symbols hoping to promote the dissemination of Liangzhu culture further.

#### **2. Transmedia storytelling of Liangzhu inscribed symbols**

Media is the material-based tool that is needed to communicate across time and space [3]. The choice of media defines what and how we carry out the dissemination of knowledge. The variety of narrative medium, which integrates the benefits of multiple media, is one of the characteristics of transmedia storytelling of Liangzhu inscribed symbols.

Liangzhu inscribed symbols are a critical element in the study of Liangzhu culture. Found in large quantities on pottery, stone tools, and jade tools of excavated artifacts from the Liangzhu culture site, these symbols, of which there are more than 340 kinds, are distributed on more than 600 artifacts, with a total number of more than 750, which are first-hand information for the study of Liangzhu culture. Liangzhu inscribed symbols appear on pottery, jade wares, and other utensils in carvings with material media as the carrier. However, since the Liangzhu civilization was discovered in 1936, the transmission carrier of Liangzhu culture inscribed symbols had shifted from material form to mass media.

In the start, Liangzhu culture was carried on paper media, and the content was mainly focused on archeological excavation briefs and current political newspapers. For example, *The Excavation Brief of Liangzhu Cemetery in Anti Mountain, Yuhang, Zhejiang* (1988), written by Wang Mingda and *The Excavation Brief of Liangzhu Culture Altar Site in Yao Mountain, Yuhang* (1988), written by Rui Guoyao. In addition, there are also reports from current political newspapers such as the New Agency Newspaper (**Figure 1**). All this information was available in text, supplemented by pictures. However, paper materials specifically recording the symbols of Liangzhu culture inscriptions have yet to appear, mainly in artifact records in other books, such as *Ancient Jade of Liangzhu* (1996), edited by the Zhejiang Provincial Bureau of Cultural Relics. Until 2015, Zhang Binghuo and the Liangzhu Museum jointly published a book dedicated to documenting Liangzhu culture inscribed symbols, *Pictographs of Liangzhu Culture*, which collects many artifacts engraved with symbols and contains more than 3000 pictures of exquisite excavated artifacts, as well as corresponding topographies and facsimiles. At the beginning of the excavation of Liangzhu culture, the dissemination of Liangzhu culture was mainly by the most direct way of information transmission. Until the emergence of auditory and audiovisual recording techniques, writing as a notational system to record the transitory

#### *Transmedia Storytelling of Liangzhu Culture Inscribed Symbols DOI: http://dx.doi.org/10.5772/intechopen.110851*

sound waves of speech was for a long time the main option to store and disseminate information across time and space [4]. On a page, the written word of a text is integrated into other multimodal contexts and interacts with spatial information about images and layouts. As a kind of visual art, the picture reproduces the image symbols with the help of pictures, color, and other expressive elements, bringing intuitive visual impact and showing the image symbols such as "the surface patterns of god, man and beast," which is also named Liangzhu God emblem (**Figure 1**).

After entering the contemporary information society, digital new media have become a tool for storing and presenting Liangzhu inscribed symbols. According to the definition of Professor Xiong Chengyu of Tsinghua University, new media is a media form that emerges and influences based on the foundation of computer information processing technology. Standard digital new media include digital movies, digital TV, digital animation, and digital games. The dissemination of Liangzhu inscribed symbols is mainly based on the Liangzhu God emblem, a cultural totem with the identity of the Liangzhu people. It has become a label for promoting Liangzhu culture.

Reports flocked to the Liangzhu cultural site after it was inscribed on the World Heritage List in 2019. Then, the Liangzhu God emblem appeared widely in streaming media publicity. For example, the creative cultural relic introduction short video *If National Treasures Could Talk,* designed and released by the Liangzhu Museum, the introduction of the divine human-beast-faced motif and jade Cong in variety shows such as *National Treasures* and *China in Heritage*, and the popular science animation *Decoding the Liangzhu God Emblem* released on Bilibili. The transmedia storytelling of the Liangzhu God emblem has also been extended to the field of video games. DIY game platforms such as *My World* and *Animal Samurai* recreate the entire Liangzhu cultural site on the game platform, presenting the transmedia storytelling of the image symbols of the Liangzhu God emblem more innovatively. Henry Jenkins points out, "Ideally each medium makes its own unique contribution to the unfolding of the story." Different forms of media increase the intellectual, engaging, and entertaining nature of the Liangzhu God emblem in different ways. Visual art has more forms of expression in the digital media environment. Visual art can change the thinking process from the referent to the referent of the symbols, and the figurative directly acts on people's vision. Therefore, the publicity method based on videos, comics, and the game brings the viewer a better viewing effect, enhances entertainment and fun, and also becomes one of the main means to promote Liangzhu culture.

In addition, the Liangzhu God emblem also entered the design field, relying on different media to present exquisite ornamentation. FELALA, a fashion jewelry design brand and the Liangzhu Museum, jointly released the *Dreaming of Liangzhu* design jewelry, carrying the image symbols from jade carvings to fashion jewelry,

**Figure 1.** *The new agency newspaper.*

attracting consumers' attention. Transmedia storytelling practices may expand the potential market for a property by creating different points of entry for different audience segments [5]. This cross-field interaction can broaden the scope of Liangzhu culture fans and promote the breadth and depth of Liangzhu culture communication. The vivid digital coding and creative cultural activities of the image symbols of the Liangzhu God emblem give readers great imagination and a rich interpretation experience. The transmedia storytelling of it has also been used on many specific occasions, such as the unique stamps of the *Asian Civilization* and the elements of the 2022 Hangzhou Asian Games mascot, "Cong Cong", (**Figure 2**) which takes its name from the Liangzhu jade and represents the ancient city site of Liangzhu in the shape of a robot. In the Liangzhu culture, jade Cong is a special kind of existence, it is the latest to be unearthed, but the most numerous, and is an essential carrier of the Liangzhu God emblem. The exquisite decoration on the head of "Cong Cong" is taken from the Liangzhu God emblem, which means "to surpass oneself without fear of danger." These kinds of media writing have made the symbols of the Liang God emblem play the role of national cards, uniquely presenting the richness of Chinese civilization.

From the history of human development, the mass media technology of each era, from printing and broadcasting to film and television, has once profoundly influenced the art and culture of that era and thus promoted the progress and development of human society [6]. The new digital media represented by digital TV, the Internet, and cell phone media have significantly impacted contemporary visual arts. Today's era centers on graphical images and visualized culture and media are becoming mainstream. Virtual reality technology, as the latest technology, is also advancing the innovation of visual art.

Virtual reality technology contains virtual reality (VR), augmented reality (AR), and mixed reality (MR), which uses computers to generate a simulated environment with the help of professional equipment, such as sensing helmets and data gloves, allowing users to enter the virtual space and get an immersive and realistic feeling [7]. The new technology has shifted the narrative medium of Liangzhu culture inscribed symbols to virtual reality technology. On October 31, 2019, Liangzhu ancient city site officially released the "Double 5" plan; the combination of 5G and virtual reality technology will gradually become a new way to demonstrate and promote Liangzhu culture.

AR augmented means that visual reality is a particular part of real reality. Liangzhu Museum has launched "AR glasses" and "AR fitting mirror" projects. For the former, visitors can observe critical cultural relics in the collection by wearing AR glasses, such as jade Cong, black pottery carved jars, and ivory scepter. This approach

**Figure 2.** *Cong Cong.*

#### *Transmedia Storytelling of Liangzhu Culture Inscribed Symbols DOI: http://dx.doi.org/10.5772/intechopen.110851*

brings the distance between people and heritage infinitely closer. When visitors look at a jade vessel engraved with "a bird standing on a high platform" (**Figure 3**), they can see the small birds inside the carved symbols come alive and fly out from the jade jar and surround it. AR glasses project will be static inscribed symbols dynamic, the figurative directly on the human vision so that the distance between vision and visual objects is dissolved. This intuitive, profound, and engaging visual experience breaks through the two-dimensional model of visual art to achieve three-dimensional interaction. The latter project increases the visitor's participation by dressing up as a Liangzhu king or queen, wearing a feather crown on his head, holding a scepter in his left hand and a jade scepter in his right hand, which increases the sense of immersion while having zero distance access to jade with engraved symbols.

MR mixed reality technology means virtual reality and real reality overlay each other. The MR holographic artifacts project in the No.1 Interpretation Hall of Liangzhu Ancient City Heritage Park restores the Liangzhu jade engraved with cultural inscriptions. For artifacts inscribed with symbols, visitors can operate in the air by switching and rotating the artifacts on the screen through gestures. It allows visitors to carefully observe the details of the jade Cong in 360° through a set of professional MR display devices jointly developed by Hangzhou Mobile and Liangzhu Museum. MR technology allows visitors to view artifacts that have never been on display, opening up access to rare artifacts and enabling the digital preservation of cultural heritage. Thus, visitors can enjoy a clearer view of the mysterious patterns on the jade Cong and the totems representing the divine power of the time, enhancing interactivity.

In AR virtual reality projects, virtual reality creates a parallel world to real reality, which means that visual art is not just reflecting and communicating the world we live in but also creating it. By realizing the fusion of visual art and auditory and other sensory art, VR technology has full-sensory characteristics. The projects displayed at the Liangzhu Museum include virtual VR online classes and museums. Through virtual reality technology, students separated into two locations are brought together in a virtual VR classroom, allowing them to experience the production life of the Liangzhu ancestors. It also provides a channel to witness the exquisite jade of Liangzhu at close range and observe the Liangzhu God emblem engraved on jade in more detail. VR virtualization of the environment enhances the authenticity of visitors. It promotes the balance of quality educational resources while enhancing the charm of Liangzhu cultural inscription symbols, the specific practice, and the new empowerment of VR technology [8]. The VR synchronized classroom has also become one of the ways of transmedia storytelling of Liangzhu cultural engraving symbols.

**Figure 3.** *AR glasses.*

#### **3. "Online VR exhibition 'bird traces and insect writings: the inscribed symbols of Liangzhu culture"**

The Internet, social networks, and mobile devices are now part of our lives that has increased museums' potential for generating and disseminating content, turning them into machines of mass knowledge transfer [9]. In recent years, VR as a representative of new technology has gradually entered the field of museums. VR exhibition has broken traditional museums' time and space limitations, allowing visitors to visit anytime and anywhere. In 2022, the Museum of the Nanyue King Mausoleum launched the "Discovery of Liangzhu" VR exhibition, which reproduces the production and life scenes of the ancestors of Liangzhu through virtual reality technology and provides a glimpse of the Liangzhu style of the water township 5000 years ago. In the same year, the Museum of Archaeology and Art of Jilin University and the Liangzhu Museum joined hands to create a virtual world of Liangzhu inscription symbols. They jointly launched an online virtual experience exhibition, "'Bird Traces and Insect Writings'—The Inscribed Symbols of Liangzhu Culture." For the first time, Liangzhu inscribed symbols are exhibited in a thematic virtual museum, relying on new technology to realize the cross-media narrative of Liangzhu inscribed symbols.

The "Bird Traces and Insect Writings" virtual exhibition refers to narrative theory regarding overall spatial layout and creative design. It presents the characteristics of inscription symbols of different periods of Liangzhu culture in a sequential narrative manner. The content of the VR exhibition focuses on inscribed symbols appearing on pottery in different periods, single lines with "x" and "flat line" as basic elements, flora and fauna symbols including "Bird standing on a high platform" and studies of writing from an academic perspective. The virtual exhibition hall has the following features.

First, it provides an immersive experience. The online virtual experience exhibition "Bird Tracks and Insects Writing" uses the technology of new images to build a 360° simulation three-dimensional guide system for visitors to create a unique immersive experience and to appreciate the immersive visual experience and impact. Dark tones dominate the entire exhibition hall to render the mysterious and quiet temperament of the Liangzhu culture inscribed symbols. In the choice of lighting, the overall atmosphere of the symbols is matched, taking the symbol of "Bird standing on a high platform" as an example, the yellow light source is used to create a religious ritual atmosphere, containing a sense of mystery, and the lighting and illumination are used to create the atmosphere of the exhibition hall, with the meaning of the cultural engraved symbols to narrate.

Second, it focuses on interactivity. The three-dimensional virtual display of cultural inscriptions provides several different virtual roaming modes for Liangzhu culture inscription enthusiasts and researchers: Visitors can choose the best browsing path and perspective according to their preferences or choose a specific camera to appreciate the cultural inscriptions of interest in the scenic spot according to the preset camera position of the scene; at the same time, according to the intelligent mobile terminal device in hand, visitors can control the direction and adjust the viewpoint of the visit in real time, visit the scenic spot independently, and zoom in and out of the interface by dragging and dropping with the mouse and sliding with the scroll wheel. The design deeply integrates the content and display design capability based on "objects," providing visitors with multiple angles and levels of appreciation of cultural relics and access to the expanded information behind them to obtain the expected communication effect.

And these modes highlight the personalized characteristics of the audience and broaden new channels for promoting Liangzhu culture. Compared with the traditional museum, the esthetic value of the virtual museum is reflected in the virtual reproduction of the real three-dimensional cultural relics, the richness and completeness of the cultural relics information, the flexibility and convenience of the visiting environment, and the infinite extensibility of the display space.

#### **4. Promotion of the VR exhibition "bird traces and insect writings"**

The online virtual experience exhibition "Bird Traces and Insect Writings" gives a general overview of the Liangzhu cultural period. However, the current VR exhibition is insufficient for cultural inscriptions, such as the two unique inscriptions of Liangzhu, "combination symbols" and "Liangzhu God emblem." So the VR exhibition can be further improved.

Firstly, the display of Liangzhu combined symbols in the form of virtual artifact models and individually engraved symbols. Based on the narrative principle of the online virtual experience exhibition, this part can select continuous combination symbols with storytelling, such as inscribed symbols on Liangzhu pottery, not only individually but also in "groups." On a black pottery jar, the Liangzhu people carved 12 consecutive symbols. Paleographer Mr. Li Xueqin identified these symbols as "Yan Jian Shi, Wang Hu Shi Feng." (**Figure 4**) "Zhu" is probably a clan name or a person's name, meaning a red flag; "Jian" means to go; "Shi" is a place name; "Feng" means realm. Taking the symbols as a whole, the meaning is that Zhu went to Shi's land and caught tigers with a net in the stone territory. Introducing the inscribed symbols in the form of a narrative story adds narrative as well as interest. There are many similar combinations of symbols, such as the carved symbols of the Chenghu site, which is considered the "Fang Yue Hui Shi," that is, to conduct a rendezvous in the Yue Kingdom [10]. Simple inscribed symbols may become a record of a military alliance of Liangzhu culture, or even a document of the state's founding, the name of which is the same as that of the later Yue state. Through the combination of symbols, visitors can further experience the charm of Liangzhu inscription symbols in the form of stories.

Secondly, as a cultural totem rich in connotation, highly generalized, and abstract expression, the Liangzhu God emblem is one of the valuable objects of Liangzhu culture inscribed symbols to display. The main elements of the image symbols of

**Figure 4.** *South Lake black pottery jar.* the gods and beasts include the image of the gods and men on the upper part and the image of the beasts on the lower part. The main components of the image of the god-man are a feather crown, a human face, and hands around the animal face, which is inverted trapezoidal in shape, with round eyes, a wide nose, and teeth; the main components of the image of the animal face are a heavy circle with huge eyes, wide nose, wide mouth with fangs, and animal claws. The above components result from the alienation and highly abstract symbolization of several things.

In the Liangzhu culture period, the "divine man" and "animal face" motifs have transitioned from fine carving to simple line outlining. From the naming, the two motifs are related to "divine man" and "animal face". The two parts of the motifs can appear simultaneously or separately on a jade. The elements of the gods and beasts are taken from nature, and the value of the association between man and nature, gods, and totems is highlighted after combination and deformation. The creation of totems is often related to the nature worship of ancient primitive tribes, and natural beasts, plants, and weather phenomena may all become the objects of totem metaphors. Suppose the sacred things in the eyes of the Liangzhu people are abstracted into a half-human, half-beast symbol. In that case, it not only retains the role of human beings but also expresses the meaning of sacredness, linking divinity and humanity as a symbol of the coherence between heaven and earth, involving the early worship of the gods and humans of the Liangzhu ancestors. And the evolution from the animalface pattern to the god-man pattern reflects a transition from nature worship to ancestor worship in the minds of the Liangzhu ancestors. The jade of Liangzhu culture is a kind of religious ritual vessel. As a carrier of the Liangzhu God emblem, the religious beliefs of the Liangzhu people are expressed on it. The Liangzhu God emblem totem is intuitive and symbolic and has an intrinsic dialectical relationship between its referent and its referent, which has rich interpretive value.

Finally, the comparison between Liangzhu cultural inscriptions and ancient Egyptian cultural symbols is presented, which mainly contains two Liangzhu cultural inscriptions, namely the "Bird standing on a high platform" and the "Liangzhu God emblem."

The Liangzhu inscription "Bird standing on a high platform" is often associated with studying other regional civilizations. However, compared with regional civilizations in other parts of the world, the closest image to the Liangzhu "Bird standing on a high platform" is the "Serekh" protective form of the names of certain pharaohs in ancient Egypt. The "Serekh" protective form, meaning the front of the palace, was later regarded as the king's name frame of the ancient Egyptian pharaohs and a symbol of kingship highlighting the king's name. It consists of three main parts, the bird, the "palace," and the symbol inside the palace, meaning the "falcon" power cult, the palace facade, and the ruler's name, respectively, and the combination is highly similar to the trinity of the Liangzhu bird standing on a high platform. According to the testimony of *Discovering the Xia Dynasty* and the proof of the "Egyptian-Xia theory of unity," some of the "Bird-standing high platform" carved symbols on Liangzhu jade can be directly regarded as the names of ancient Egyptian pharaohs, such as Pharaoh Nebre of the Second Dynasty. On the left is a jade bi with carved symbols from Liangzhu, which is in the Freer Museum in the United States. The jade shows a side-standing bird standing directly on a stepped "altar" filled with scrolling clouds. For the interpretation of the scrolling cloud pattern, Wu Hong believes that the circle with four reversed single spiral scrolling cloud patterns is similar to the oracle "sunlight" and the meaning of this symbol [11]. The bottom of the altar is supported by a moon-like figure, forming a combination of "sun and moon," which corresponds to

*Transmedia Storytelling of Liangzhu Culture Inscribed Symbols DOI: http://dx.doi.org/10.5772/intechopen.110851*

#### **Figure 5.**

*Liangzhu bird-standing on a high platform (left) and Nebre names (middle and right).*

#### **Figure 6.**

*Liangzhu god emblem from discovering the Xia dynasty (368).*

a pharaoh of the Second Dynasty of Ancient Egypt whose palace facade contained a combination of sun and moon symbols (**Figure 5**).

The other part compares the Liangzhu God emblem and ancient Egyptian cultural symbols. According to the interpretation of *Discovering the Xia Dynasty*, the Liangzhu God emblem is from the depiction of the ancient Egyptian pharaoh holding a phallic weapon to offer sacrifice to the god. However, this interpretation is not quite the same as the generally accepted division between gods and beast faces. Usually, the division between the divine man and the beast's face is that the upper part of the feathered crown and the upper limbs belong to the divine man. In contrast, the lower part belongs to the beast's face, which is considered to be the kneeling posture of the divine beast. The division in *Discovering the Xia Dynasty* should be the big-eyed part of the pharaoh's phallic weapon, and the part of the forelimbs of the divine beast should be the legs of the divine man, so the Liangzhu God emblem is an image of a divine man holding a phallic weapon (**Figure 6**).

Therefore, the VR exhibition section of the comparison between the Liangzhu culture inscribed symbols and ancient Egyptian cultural symbols will show the comparison between Liangzhu cultural inscriptions and ancient Egyptian cultural symbols. It could be presented in graphic form, using textual forms for better interpretation. The meaning of the bird standing on a high platform and the Liangzhu God emblem must be further explored. Still, the appearance of similar symbols from two ancient civilizations suggests that this coincidence also proves that there were certain

similarities between Chinese and Western cultures, especially in the understanding of thought and religion, at the beginning of the birth of early national societies. The display of this panel also allows visitors to feel the coincidental second clash of ancient ideas and a different perception of Liangzhu culture.

#### **5. Conclusion**

City symbols are iconic things that represent the cultural characteristics of the city, have inheritance value, give people a deep impression, and make them proud [11]. And the promotion and creation of Liangzhu culture inscribed symbols is undoubtedly the best means to shape the city symbols.

In recent years, the Liangzhu inscribed symbols represented by the Liangzhu God emblem have broken through the traditional narrative media and gradually developed into new digital media. Among them, the new media represented by videos and games have increased the knowledgeability and entertainment of Liangzhu inscribed symbols at different levels. Virtual reality technology has the characteristics of immersion and interaction; reproducing virtual scenes, this technology transforms the visual transformation of Liangzhu's inscribed symbols from two-dimensional pictures to threedimensional images. In addition, the Liangzhu VR exhibition further expands the scope of Liangzhu culture dissemination through the flexibility and convenience of the visiting environment and the infinite extension of the display space, enabling visitors to visit the Liangzhu inscribed symbols exhibition online anytime and anywhere. In the digital media environment, Liangzhu inscribed symbols have more forms of expression and promote the dissemination of culture and information. This diversity of narrative media choices is conducive to each media platform giving full play to its own advantages to contribute to the overall narrative and construct a complete story world of Liangzhu ancient culture. As an innovative transformation of Liangzhu culture, the transmedia storytelling of Liangzhu inscribed symbols shows strong vitality and glows with eternal charm, making a unique contribution to the dissemination of the connotation of Liangzhu cultural heritage and contemporary value promotion, as well as city image building.

#### **Acknowledgements**

This research was supported by the Think Tank Construction Project of Hangzhou Normal University (2020), the Key Research Institute of Philosophy and Social Sciences of Hangzhou, and the Literary Criticism Institute of Hangzhou Normal University; the Key Research Base of Hangzhou Philosophy and Social Sciences.

*Transmedia Storytelling of Liangzhu Culture Inscribed Symbols DOI: http://dx.doi.org/10.5772/intechopen.110851*

### **Author details**

Jinghua Guo\* and Weile Weng Hangzhou Normal University, Hangzhou, China

\*Address all correspondence to: guojinghua@hznu.edu.cn

© 2023 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

### **References**

[1] Henry Jenkins, translated by Du Yongming. Convergence Culture: Where Old and New Media Collide <fusion culture:new The conflict zone between media and old media>. Beijing: The Commercial Press; 2012. p. 157

[2] Scolari CA, Bertetti P, Freeman M. Transmedia Archaeology: Storytelling in the Borderlines of Science Fiction, Comics and Pulp Magazines. London: Palgrave Mcmillan; 2014. DOI: 10.1057/9781137434371

[3] Jørgen B, Schirrmacher B. Intermedial Studies: An Introduction to Meaning across Media. Taylor & Francis; 2022. p. 4

[4] Kittler FA. Gramophone, Film, Typewriter. Redwood City, California: Stanford University Press; 1999

[5] Henry J. Transmedia storytelling 101. Confessions of an Aca-fan [Internet] 22.03, 2007. Available from: http:// henryjenkins.org/blog/2007/03/ transmedia\_storytelling\_101.html

[6] Lei D. Innovation of visual art in the context of digital media <digital media context Visual art innovation under> [thesis]. Nanjing Art Institute. 2013

[7] Hao-Lei C, Xiang-Jun Z, Yan C, et al. The latest developments and prospects of virtual reality technology <virtual Evolution, development and prospects of real-world technology>. China Science and Technology Paper Online. 2011;**6**(01):1-5+14

[8] Xiaodong L, Honghui C. Research on seamless flipped classroom based on VR--reinventing the classroom of "English audio-visual journalism" <Relying on Research on VR's Seamless Flipped Classroom – Reshaping" News English audio and video "classroom">. Modern Educational Technology. 2017;**27**(12):69-67

[9] Mateos Rusillo Santos M, Castells AG. Transmedia storytelling and its natural application in museums. The case of the Bosch project at the Museo Nacional del Prado. Curator: The Museum Journal. 2018;**61**(2):301-313

[10] Wu H. Birds images in eastern Yi art <Bird Images in Dongyi Art>. The Fine Arts in Ritual. 2005:11-28

[11] Xinxin L. The use of cultural symbols in city image shaping <city image shaping The use of Chinese cultural symbols>. Contemporary Communication. 2011;**2**

#### **Chapter 7**

## Intermediality in Contemporary Visual Art Education

*Bea Tomšič Amon*

#### **Abstract**

Fine arts education, a term widely used to define the school subject, is no longer appropriate to describe artistic expression goals in the educational context. Contemporary visual education allows for a comprehensive approach that considers the intermediality of contemporary visual art. Pedagogical methods, processes, and goals should be open to different approaches under the teacher's guide. This paper presents reflections on the pedagogical process and a concrete example of a performance developed by secondary school students. It refers to an intermedial project that allows discussing multisensory perception, interdisciplinarity, and the integration of different fields of study, showing an intermedial approach to the pedagogical process. The example shows how creativity develops and grows with such an approach.

**Keywords:** art education, contemporary art, multisensory perception, pedagogical process, intermedia pedagogic strategies, students' competences

#### **1. Introduction**

The concept of intermediality in contemporary art has a broad meaning. It can refer to various aspects of artistic production, from processes and products to creative strategies and material or technical media. It can refer to descriptive, exploratory, or performative approaches to art forms. Many times verbal expressions are influenced by expressions of a similar art but also by other media and their structures [1].

Communicative media convey both commonalities and differences. These are the focus of intermedia studies. An important topic is interaction between media and the process of transferring material from one media type to another. Differences between media types can also be transferred at other levels. All communicative situations in all media types are multimodal and draw on different resources for meaning-making. In such communicative environments, people can negotiate meanings without much effort [2].

As such, it is also related to arts education and raises questions about the relationship between arts education and contemporary art production. Art education, fine arts, and drawing are widely used terms for the school subject that deals with art theory, art history, artistic expression, and the appreciation of art products. The content is often tied to specific areas such as drawing, painting, printmaking, visual

design, sculpture, and architecture. Each area is usually studied without connection to the others.

Art education defined and developed in this way is no longer adequate to describe artistic expression's goals in a contemporary educational context. Contemporary art education allows for an integrative approach that considers the intermediality of contemporary visual art. Approaches, methods, and goals should be revised in all curricula of a subject often referred to simply as drawing. Students need new and different skills to participate in today's cultural life. Only important changes will prepare them to understand that art is moving in a different direction than what many are taught in school. The ideology of closed subjects no longer has anything to do with reality.

In such a broad context, the article proposes to understand and classify the importance of the topic. It presents major changes in the conception, perception, and evaluation of events in the world of art, accompanied by major changes in the school field. The emergence of a new real or virtual spectator, rapid functional changes in his life and self-perception, and the teacher's responsibility as operator of the reproduction machine that is part of the ideological apparatus of culture, of which the school remains a necessary part.

The paper explores and analyzes the world of art education with a brief account of a core document that should represent the body of ideas among contemporary curricula of the subject. This explicitly explains the interwoven intermedial aspects in the definition of the subject. It then reflects on the nature of the pedagogical process, teaching and learning, the role of the teacher, and intermediality as a way of creating knowledge. Finally, the paper presents an example of an intermediate project carried out by secondary school students, showing the process and results of such a sophisticated learning strategy. In light of this, it can be argued that it would be possible to improve the approach that transforms visual culture/education into perceptual culture/education in order to contextualize intermedial artistic production and art education by stimulating sensitive experiences and generating a holistic critical perception of the world.

#### **2. European competencies and the arts**

However, the key competencies listed in the sources published by the European Union [3] show the links between artistic content and effective teaching and learning in the arts, especially at the secondary level. They recommend and encourage an intermedia approach to arts education. Students acquire an important part of their esthetic literacy through direct artistic expression. During the year-long creative and design practice, students acquire new and deepen already developed skills for artistic and esthetic work. Artistic creation is a combination of specific and other key skills. In this way, students materialize their thoughts, ideas, and concepts into esthetic material that is the product of various competencies. They overcome possible language barriers and limitations by using an artistic language that allows them to use additional ways of communication; they learn geometric, chemical, and physical concepts in visual arts, use knowledge of optics and optical color mixing, and learn basic scientific concepts such as relativity and moving perspective. They evaluate how the artist has responded to the state of nature and society and respond critically to the state of nature and society that defines them through their work. Within the esthetic and cultural literacy framework, they recognize the esthetic qualities of artistic

#### *Intermediality in Contemporary Visual Art Education DOI: http://dx.doi.org/10.5772/intechopen.110873*

creation and create an artistic product based on their experiences, understanding, and skills. They engage with the art of different cultures—multiculturalism—and promote independent artistic expression. In the context of esthetic and digital literacy, they recognize various media of visual culture and critically describe their characteristics in terms of communication, content, esthetic meaning, and function. Students use and strengthen their knowledge of materials and digital literacy when searching for working materials. In the context of esthetic literacy, they define the interdependence of function and artistic form. In the context of health literacy, they handle tools and materials appropriately, follow work safety instructions, and protect work and the natural environment. They can express feelings through artistic means and thus regulate their inner balance; they analyze the perceived world. Art competence enables them to read informational art messages. They use information and communication technologies correctly and combine them with creative artistic expression. As intercultural competence, they accept and respect artistic creations from other environments and different eras, as well as the products of students with different national and cultural identities; within the framework of civic competence, they evaluate the importance of domestic and foreign artistic and cultural heritage; they develop tolerance, openness, national consciousness, develop an awareness of interculturality. Students accept differences and similarities when viewing, interpreting, and responding to works of art from other cultures.

In summary, art education is a school subject that provides an understanding of the arts as a fundamental achievement of civilization and encourages creativity in artistic expression and interpretation of works of art. Art history content transparently and comprehensibly introduces the development of the visual arts as one of the most important forms of expression of human creativity and provides a starting point for understanding and experiencing art. In this way, they contribute significantly to a comprehensive understanding of the key role of artistic creation in the civilizational development of humankind. Artistic creation builds on the practical and theoretical knowledge of artistic expression acquired in basic education. In this way, the student develops his or her own creative skills for artistic expression while developing and deepening his or her understanding of the concepts of visual arts theory and becoming familiar with the diversity of visual arts genres and practices in time and space.

The so-called European competencies show that, as Kroflič [4] says in a discussion of theoretical approaches to curriculum design, planning educational activities has a very interdisciplinary character, and so is the nature of intermediality in arts education.

#### **3. Intermediality and the pedagogical process**

The paper refers to a group of competencies that a person can develop through seeing and through collecting and integrating sensory experiences. Effective learning depends on the development of these skills because they enable students to decode and explain actions, natural or man-created symbols, and objects in the environment. The application of these skills fosters communication between people. Analyzing and interpreting visual material is not enough. The ability to create visual signs is also essential. Hybrid, sensitive, and operational experiences promote the use of a specific language that enables them to discern the meaning of images. Digital processing of information has become a cultural communication tool, while technological applications and intermediality contribute to the development of educational practices and

cultural policies. They expand the common heritage and promote cultural diversity and the plurality of identities [5].

The spatial experiences described are important for art classes and other school subjects, since they usually deal with visual representations of all kinds. This statement is important in developing the ability to imagine relationships between objects in space in geography, geometry, physics, biology, chemistry, or sports and visualization in history, literature, or foreign language learning. On the other hand, the school should develop a refined verbal, visual, and auditory perceptive ability—considering all the contents that this term implies and presupposes. They are indispensable in almost all activities. For this reason, students must acquire appropriate practical experience and specific competencies.

Fundamental changes in education are consequences of the development of a particular technology. The educational system in an oral culture differs from that in a written culture. Finally, new media and intermediality affect culture, literacy, and education. New media developments affect the institutions that organize "education" in the broadest sense of the word and inevitably influence what and how we learn [6].

The need to individualize the educational process requires the development of alternative, flexible, and effective teaching and learning strategies. Artistic expression, in all its heterogeneity, allows for deep insight into and contemplation of a variety of content from different angles and promotes multisensory and complementary experiences. In this way, the artistic experience, gathered through different modalities of mediation, becomes a point of connection between different contents and objectives. It supports the design of didactic material and motivates the improvement of teaching and learning in other educational areas. However, behind this is the awareness that new media and digital technologies are essentially about visual images of different kinds and their fusion with other means of expression, as well as auditory, kinesthetic, and verbal experiences, which are essential for explaining and understanding data in different subjects of study.

Teaching perceptual experiences that have their starting point in artistic subjects allows for a complementary relationship between the "world of art" and the "world of science." This is particularly important given the increasing need for individualization of teaching and learning that responds to the different styles of teachers and learners and their prior experiences in the field and individual development in spatial representation, motor skills, etc. This becomes even more important when we consider each student's individuality, cultural background, inclinations, needs, gender, etc. In this context, the role of advising teachers through artistic approaches concerning students' academic achievement and helping teachers develop appropriate teaching and learning strategies through the arts are still questions to which there can be many answers.

#### **3.1 Intermediality as creation of new knowledge**

Creating new knowledge can transform how people see and think by providing new insights into how content gives meaning to ideas, issues, and questions. Images can carry meaning through description, representation, expression, or symbolization. Recent cultural discourse offers a much wider scope for the potential meaning of encountering a work of art. This inevitably raises several very interesting and highly significant questions for the various fields of arts education and education in general, for example: How can perceptual experiences be stable and continuous when other interpretations are possible? There is no longer a division between the

#### *Intermediality in Contemporary Visual Art Education DOI: http://dx.doi.org/10.5772/intechopen.110873*

different disciplines of art. Strong sociocultural changes condition all kinds of artistic expression; discussions about cultural and national identity, minorities, technological changes, and the postmodern philosophy of plurality and fragmentation have changed the premises that also determine the nature of art and education. These issues are changing how we relate to art, learn, and participate in experiences that emanate from art.

Additionally, it is important to approach arts education from a critical perspective addressing the complexity of experiences deeply integrated into everyday life. The esthetic dimension is a unique cognitive process developed by arts education and used by other fields. At this point, we can argue that education should embrace broad, holistic forms and practices critically examined through the interdisciplinary, multidisciplinary, transdisciplinary, and intermedial methodologies associated with the various fields of study, creating new knowledge about the world at every step. The approach to education should be from a critical perspective of the complexity of actual experience.

According to Mueller [7], a change from a media product to an intermedia product takes place with the juxtaposition of multimedia elements and different media, which are transformed into a conceptual juxtaposition of intermedia. Such an intermedial juxtaposition arises in artworks with combinations, transformations, or allusions to other media, Rajewsky explains [8].

It is no longer sufficient to understand media as a mechanical means of transmission that transfers some kind of information from a "producer" to a "receiver." In this new context, the medium is that which mediates between people across historical and spatial distances based on meaningful signs or configurations of signs with the help of efficient transmitters.

#### **3.2 Teaching and learning**

Given the ever-increasing prevalence of electronic and other popular media, the complexity of the relationships among what students see and hear, what they believe, and how they interact with one another underscores now more than ever the need for cross-curricular teaching of critical thinking, critical reading, and critical viewing skills. A consensus is emerging that teaching critical viewing skills strengthens students' abilities in traditional subjects, combats problems of adolescent apathy, and improves student, parent, and teacher attitudes' toward school.

Intermediality challenges the classical teaching of acceptable artistic works that are far removed from students' experiences. Instead, it emphasizes the learning environment over specific content [9].

Intermediality examines, extends, and synthesizes the existing definitions, texts, theories, processes, research, and contexts. It brings into focus the possibilities of working with different media texts. Thus, critical media literacy becomes a competency to interpret and understand how meaning is made and derived from print, photographs, and other graphic visuals [10].

The goals of art education focus on the holistic development of the child, which includes the development of three important categories of behavior based on the taxonomy of educational goals: cognitive, affective, and psychomotor. In receiving information from the environment, it is necessary to learn to respond to different inputs and have the appropriate knowledge that helps their sensory system to complete the activity with a successful product [11].

A form of teaching that incorporates various means of artistic expression is influenced by several factors: the pedagogical, didactic, and methodological competencies of teachers, their personal views on the subject that affect the degree of motivation to include various media in the teaching program, and the physical and organizational conditions for conducting such programs. It is also necessary to emphasize the goals of the subject, adapted to the age of the students, and thus the pedagogical strategies.

In the media-saturated world in which we live, it is perhaps not an excess to speak of intermedial competence as a fundamental concept in communication and education. What do we achieve from teaching intermediality at different levels of education? How can intermediality be incorporated into classroom practice, not only as a pedagogical issue but also even as a teaching method from kindergarten on? Ask Semali and Pailliotet [10].

Young children have a natural need to express themselves [12]. They need freedom because, in this way, they can get to know and understand the world around them. At this stage, intermediality can manifest itself in the combined use of materials and creative strategies, such as motivation by reading a tale, guided by an experienced teacher who appreciates the nature of this mainly physical and cognitive experience for the child. Reading a verbal text that the children have to represent with visual signs is a primary kind of intermediality between languages (**Figure 1**).

Later, at around 12 years (**Figure 2**), the child develops sufficient psychomotor skills and abstract thinking and can think critically and understand issues conceptually. If he or she has sufficient experience in using different materials or technical media, intermediality becomes a rich strategy for the student to access the world of contemporary art without borders [13]. In this case, the verbal input was a discussion about identity, and the task was a self-representation, an auto-portrait. After the conversation, each child created an individual visual view of its meaning*.*

**Figures 1** and **2** show how an artistic expression changes over the years, from early childhood to the beginning of adolescence. In both cases, there was a remediation from verbal to visual language.

The acquisition of knowledge requires a good selection of the most appropriate learning material and must show the connection between different knowledge and

#### **Figure 1.**

*Drawings made by kindergarten children. After listening a tale, each child created an individual view of the meaning of the text using different materials. https://commons.wikimedia.org/wiki/File:Ship\_in\_the\_sea.jpg; https://commons.wikimedia.org/wiki/File:Fishes\_kindergarten.jpg.*

#### **Figure 2.**

*Drawings of elementary school children in the final grades. After participating in a conversation on individual identity, each child created a depiction of its meaning using different materials. https://commons.wikimedia. org/wiki/File:AUTOPORTRAIT\_ELEMENTARY\_SCHOL\_CHILD.jpg; https://commons.wikimedia.org/wiki/ File:Autoportait\_in\_elementary\_school.jpg.*

stimulate the search for the essential. Individual subjects can find meanings in the context of other subjects, complementing and linking objectives, goals, and contents.

Knowledge from different subject areas allows the establishment of the transfer of thinking strategies. On this basis, creative problem-solving can take place in different subjects and promote higher thinking skills [14]. This model is based on the assumption that it is necessary to develop critical thinking skills in order to structure, order, and develop or achieve desired outcomes based on conceptual structures. Content and learning processes are strongly intertwined, with the teacher providing the content link between different subjects. Drake also agrees with this approach because the brain is organized to take in more information at once, and holistically acquired information is more easily and quickly retrieved [15].

The most important reason for students' loss of motivation is often the isolated treatment of the content, removed from its authentic context, which does not meet their interests and needs. In planning the learning process, we understand the group as a community of individuals with different abilities, interests, experiences, and inclinations. The need for increasing individualization of the learning process requires that the teacher always keep the group and the individual in mind when planning the teaching and learning process, which requires the development of flexible, alternative, and highly dynamic instructional strategies.

Rajewsky has made clear how such intermedial coexistence comes into being in works of art via combinations or transformations, or references to other media [8]. This is exactly what usually happens in the pedagogical process of art education.

#### **3.3 The role of the teacher**

One of the most important consequences of new material culture and intermedial practice for education is that it has caused a shift in the relationship between youth and adults and between students and teachers [16].

Intermedial pedagogy calls into question the traditional institutions and fields of knowledge. Contemporary pedagogy must address how to link the question of authority to democratic processes in the classroom that do not promote pedagogical terrorism and yet offer representations, stories, and experiences that allow students to engage critically in constructing their own subjectivity. At the same time, they are in an ongoing process of negotiation between self and others [17].

A motivating, independent teacher modifies or develops the content of related items, independently makes connections, and helps construct meanings. The teacher should become a designer: indeterminacy rather than order should become the guiding principle of a pedagogy in which multiple views, possibilities, and differences are opened up as part of an attempt to read the future contingently rather than from the perspective of a master narrative that assumes rather than problematizes specific notions of work, progress, and agency [18].

An autonomous, motivating teacher modifies or develops the content of analogous items, makes connections independently, and helps construct meanings. The teacher should become a designer: Indeterminacy rather than order should become the guiding principle of a pedagogy in which multiple perspectives, possibilities, and differences are highlighted as part of an attempt to understand the future contingently rather than from the perspective of a master narrative that presupposes rather than problematizes particular notions of work, progress, and agency [18].

He selects the method of teaching (methods, forms of work, teaching aids, strategy for planning learning steps, design of artistic tasks, and approaches to linking the content of different subjects), brings the method of work closer to the students' wishes and needs, artistic abilities and personality traits. He does not impose his own values and views, and allows students' freedom of artistic expression. He moderates the learning process and, promotes motivation, knows how to respond to the individual student.

He encourages student activity in all phases of learning, intense perception, complete experience, internalization of the activity, responsibility for the performance of the activity, acquisition of artistic knowledge, and personal development or lifelong learning; he guides students to independent research.

He guides students to discover their own strategies for solving artistic problems, to be curious and accept differences in artistic expression, and to link content in an interdisciplinary and intermedial way. He motivates students to acquire new knowledge in an experiential way, to use a variety of design and thinking strategies, activities, processes, materials, tools and procedures, and to link problem-oriented assignments to the themes of fine arts and contemporary intermediality in visual culture.

Artistic creativity, a very important component of the personality of the students, develops with artistic activities. Therefore, this allows for the evolution of creativity in general, which today is an irreplaceable factor in the development of the individual and society. The development of artistic creativity includes the promotion of the imagination in the use of materials, the choice of appropriate strategies and working methods, sensitivity in the perception of artistic qualities, entanglement in the transformation of artistic elements and materials, complexity in conceptual aesthetic planning to solve artistic problems, and flexibility.

Undoubtedly, such teaching requires a special kind of organization; it demands great flexibility in planning and evaluating work results. It is also necessary to adjust the teacher's own ideas about what he expects from the learning process. There is a dialectic between the successive experiences, which include the teacher, with his experience, expertise, organizational skills, knowledge, intuition in the field of individual management, etc., and the student, who interprets and builds a picture of the world. The results arise from the way each of them assumes the world and assembles the elements into a new whole that has meaning in a particular context, renewed each time.

#### *Intermediality in Contemporary Visual Art Education DOI: http://dx.doi.org/10.5772/intechopen.110873*

The learning process gradually includes the possibility of introducing an individual strategy for solving art tasks, choosing art techniques and art motifs, and using new concepts in different contexts. This certainly plays an important role in the successful completion of art tasks. In addition, surprise at the unexpected results in connections between contents play an important role in motivating students.

The presented work strategy actively and intermediately involves the participants in the learning process, which increases personal engagement and reveals contradictions between own and others' experiences, between processes and goals. In this way, it helps to change entrenched attitudes, broaden the perspective of viewing certain phenomena, and helps to connect separate aspects, cognitive, emotional as well as action aspects. We could say that it is a realization of Dewey's thought that when we look at a work of art—we would add, create or recreate—emotion and thought are together in their perceptual and sensory connection. Therefore experience is a complex in which the world opens up to us and conveys meanings and values in a nonverbal way [19].

The conditions in which we can ensure the success of such an approach to work are also specific: an atmosphere that motivates the participants and allows the relaxed expression of opinions and feelings. It is also important to ensure adequate preparation of the physical environment in which the activity takes place and of the didactic material, a combination of different forms of work to allow temporary privacy and, simultaneously, to express a diversity of ideas.

Intermediality refers not so much to new kinds of problems per se as (at least potentially) to new ways of solving them, to new ways of representing and thinking about them. It also points to new, or at least different, ways of looking at medial boundary crossings and hybridization; in particular, it points to a heightened awareness of the materiality and mediality of artistic practices and cultural practices in general [8].

#### **4. An example: 'The expected escape'**

The paper presents an example of how strategies of intermedia approach and integration of program orientation content and arts content were the focus of a project at a specialized high school with a music high school program. The content presentation was based on a comparison of terms from the field of music that are also found in art theory, such as rhythm, composition, harmony, chord, high note, middle note, low note, major and minor, etc.

In order to verify the importance of the ability to process visual images and sounds efficiently, an experiment was carried out that resulted in a product, an intermedia performance that allowed to understand what effects these stimuli – sound, and images in their entirety – can have on viewers/listeners.

The first question was about the possibility of finding an analogy between sounds and images. This was the core idea of the project. Although psychologists such as Kubovy argue that the realm of sound sensation in music, strictly speaking, offers an analogy to space, this is manifested in verbal language. "We speak of high and low tones, not of right and left tones, although our musical instruments make the latter designation seem very natural." Von Ehrenfelds, long ago in 1890, was the first to claim that musical chords and visual forms have the property of being substitutable without losing their identity, citing Kubovy [20]. In media studies, an esthetic focus on intermedia relations has been placed in the historical perspective by research on

how a given medium "remediates" other media [21]. One form of intermediality inherent in this example highlights artworks in which there is a reference to another artwork or to another artistic system as a whole. For example, this form of intermediality appears in literary texts describing a painting or a piece of music [1].

Sound enlightens us about causes and events, not about surfaces and material objects: Our language suggests to us that objects are visual in nature; visual objects have considerable control over what we hear. Wightman and Jenison [22] distinguish between concrete auditory objects, which consist of sounds emitted by real objects in the environment, such as an orchestra, and abstract auditory objects, which often do not correspond to real objects in the environment, such as a melody. Listeners distinguish between the auditory subsystem that processes the concrete objects and the auditory subsystem that processes the abstract auditory objects. We need to abandon the vision-focused notion of the object to understand this and provide a more general definition of the perceptual object, whether visual or auditory.

Intermediality as a contemporary concept between different artistic languages or media can be traced throughout art history. Research attempting to demonstrate a connection between the location of a painting or visual sign and the sound value of that location in Paleolithic cave paintings is astonishing. In many cases, it has been demonstrated that the locations with the greatest resonance are the locations for images. An image requires sound, and the extent to which resonant sites were used indicates the importance of sound itself: sound requires an image [23]. Among the most interesting examples from the last century are probably Wagner's concept of the Gesamtkunstwerk, which in this case means the fusion of the arts into an allencompassing Gesamtkunstwerk, or Klee's Polyphonic Painting [24].

Although the expression "hearing the image" may seem paradoxical, it describes a particular response to visual stimuli, a phenomenon called aural-visual synesthesia. A good example of a person with this ability is Kandinsky, who described the psychological effect of colors thus: "A psychological shock produces a corresponding shock by association. The sound of colours is so distinct that it would be difficult to find someone who would express bright yellow with bass tones." The aspect of synesthesia is the ability to perceive objects with particular tastes, sounds, and colors. His formulation of the reactions of the sensitive soul is supported by centuries of mnemonic theory, current cognitive psychology, and many contemporary studies of the visual arts. In other words, visual stimuli can evoke auditory sensations or memories in individuals who are receptive or trained to do so [25].

The metaphor linking sounds produced by the piano to color sequences has been used from the Baroque period well into the twentieth century. Music and painting share a basic vocabulary that includes concepts such as color, tone, harmony, composition, improvisation, modulation, and scale. Hector Berlioz wrote that instrumentation in music is the exact equivalent of color in painting [26].

The project, titled "The Expected Escape," was conceived as a multisensory, intermedial experience. Multisensory and intermedial means that the experience to promote was neither visual nor auditory, but a combination in which neither of these manners was favored. Rather, it was a combination of two different languages, taking into account the idea that music or sound, in this case a Bach fugue, can be transmuted into a visual product. Every fugue has a theme, which should become the main theme of a visual composition. This means that it is possible to select an image that will act as the theme in the context of the sequence of images in the entire visual or pictorial composition. In Bach, the theme appears in different voices as bass or tenor, depending on the fugue chosen. Changes in musical form mean parallel

#### *Intermediality in Contemporary Visual Art Education DOI: http://dx.doi.org/10.5772/intechopen.110873*

changes in pictorial or visual form. The whole composition acquires its meaning only by considering all these elements. Regarding the research experience, it is clear that listening and looking provide descriptive information that helps to find the narrative, and symbolic content of the event. The auditory and visual elements are linked so that the viewer/listener overlooks the fragments in order to discern a unity, a narrative with broader potential content. Both the musician and the visual artist want to evoke specific sensations that the listener/viewer is expected to place within a larger symbolic narrative. The effectiveness of communicative intent is overwhelming when visual and auditory elements form a unity. Each element is different and represents a different sensation and narrative content. When the unit is viewed in its spatiality, it can tell many different stories.

As mentioned earlier, a group of secondary school students developed the project. After selecting the concert program, the students made sketches with drawings, paintings, collages, and photographs that illustrated the different moments of the musical theme. Then they selected about 350 images that were scanned and arranged

#### **Figure 3.**

*Examples of the first sketches for the design of the visual part of the performance. https://commons.wikimedia.org/ wiki/File:The\_expected\_escape\_drawings.jpg; https://commons.wikimedia.org/wiki/File:The\_expected\_escape\_ drawing\_1.jpg; https://commons.wikimedia.org/wiki/File:The\_expected\_escape\_drawing\_2.jpg; https://commons. wikimedia.org/wiki/File:The\_expected\_escape\_drawing\_3.jpg; https://commons.wikimedia.org/wiki/File:The\_ expected\_escape\_drawing\_4.jpg; https://commons.wikimedia.org/wiki/File:The\_expected\_escape\_schema.jpg.*

parallel to the music (**Figure 3**). With the images, they created a presentation that was a visual representation of the sounds. During the performance, the presentation was projected onto a stage set that consisted of a cube of fine white textiles that hung in different directions, reflecting and changing the views of the images depending on the viewer's angle. The piano was placed in the center. The viewers placed around the cube were to follow both media, the visual and the acoustic, simultaneously (**Figure 4**).

The project's title symbolized the expressed content, but it was not intended to function in its classical verbal meaning. To test the eventual hypothesis about the differences that occur when perception is considered holistically, a test was conducted. Forty-five randomly selected assistants to the presentation of the performance agreed to participate. They were divided into three groups: 15 of them could not see the visual part, 15 could not hear the sound, and 15 could perceive the performance as a whole.

Each participant completed a short questionnaire. The first question was, 'Which word would you assign to the presentation? The suggested words were dynamics, harmony, passivity, escape, joy, aggression, and contrast. The most frequent answer was dynamics in the group that could only hear the sound. The most common answer in the group that could only see the images were contrast. The most common response in the group that could consider the performance as a whole was escape. Interestingly, the participants in this group chose a word that theoretically cannot be associated with the world of sound or visual art. Instead, it was a world with a narrative background that gave new meaning to the concept of performance as a whole. The second question was, 'What would you call the performance?' The possibilities were 'Thriller Night', 'The Expected Escape' and 'Luxury, Calmness and Voluptuousness.' The majority of the participants who were able to see and hear the performance as a whole chose the correct title.

#### **Figure 4.**

*Pictures of the final performance. https://commons.wikimedia.org/wiki/File:The\_expected\_escape\_performance. jpg; https://commons.wikimedia.org/wiki/File:The\_expected\_escape\_performance\_1.jpg; https://commons. wikimedia.org/wiki/File:The\_expected\_escape\_performance\_2.jpg; https://commons.wikimedia.org/wiki/ File:The\_expected\_escape\_performance\_3.jpg; https://commons.wikimedia.org/wiki/File:The\_expected\_escape\_ performance\_4.jpg; https://commons.wikimedia.org/wiki/File:The\_expected\_escape\_performance\_5.jpg.*

From the results, it can be concluded that the possibility of "reading" the performance as a whole actually meant a process that fostered the construction of its own meaningful significance.

In this example, intermediality can serve primarily as a generic term for all those phenomena that (as indicated by the prefix inter) take place in some way between media. Intermediality, then, refers to configurations that involve crossing boundaries between media and can thus be distinguished from intramedial and transmedial phenomena (i.e., the occurrence of a particular motif, esthetic, or discourse in across different media) [8].

#### **5. Intermediality as multisensory experience**

This experiment has shown how intermediality is a concept and a methodological tool that allows analyzing the world of contemporary art, which nowadays actually knows no boundaries. Intermediality includes multisensory experiences and is not only bound to the visible space, as in traditional art fields but allows an open definition of what art is. Thus, as media scholar Mitchell has pointed out, all communication involves all our senses. There are no purely visual, textual, or auditory media. All media products are, therefore, mixed and heterogeneous rather than monomedial [27].

Conceptually, spatial perception in a work of art is related to sensory perception, a fleeting action or reaction triggered by a specific context. Generally, it is considered an internal feeling perceived by each individual according to his or her personal interpretive filters, cultural and social dimensions; however, it can also be recognized as a feeling shared by a community with a common denominator. The perception and appropriation of space give it properties that complete its semantic and physical definition. The experience does not emanate directly from the space, but is created in part by the viewer, who changes, defines, and completes it through his active and creative appropriation. He is a producer and creator, more than a mere observer of an 'empty vessel' [28].

Embodied experience means that we comprehend space with all our senses in motion so that qualities that in the Cartesian tradition were considered secondary such as color, texture, the glint of sunlight on the windowpane, and the echo of footsteps, become primary to space. Thus, spatial sensibility can be considered as the unconscious recognition of the fleeting fusion of sensory signals that constitute the sense of place at a given moment [29].

Our perception of space is not continuous but partial and fragmentary. Although all the senses are in action, the image received is a combination of all the senses, with none of them predominating except in moments of concentration on something specific. It is a multisensory experience. Pallasmaa reinforces these ideas by saying that the image is generally considered from a purely visual point of view, but that the qualitative property of the senses is their tendency to integrate; connotative effects involving all the senses always accompany a visual image. The visual image is essentially a fusion of discontinuous fragments and ideas [30]. In art as experience, Dewey says that the qualities of the senses, touch, taste, sight, and hearing, have aesthetic qualities. However, they are not isolated, but connected in a totality in which they interact, rather than as separate entities. These entities are never related to themselves, just as color is related to color or sound is related to other sounds. The author highlights fundamental aspects of perceptual experience

in space, such as individuality, subjectivity, and originality. It is a specific experience of each person [19].

In relation to intermediality, it is also necessary to describe the experience of virtual space as it is peculiar to contemporary visual art. A virtual space is located in an indeterminate place for the subject. On the other hand, the individuality of the gaze confirms the existence of different "viewing techniques" and a particular awareness of attention in perception as a model for the way the subject constructs a coherent idea of the world. The model is not only visual in nature, as perception is not measured only in terms of proximity or actuality. Perception is a function that ensures activity, productivity, adaptability and predictability, and social integration of the subject, Crary said. This means, according to him, that a reconceptualization of perception is very important for the transformation of mass culture and that it may be necessary to acknowledge that digitization has changed the way space is looked at and seen [31]. Regarding the importance of seeing Grau affirms, what is called reality is simply a statement of what can be perceived, seen [32].

Digitization has also changed the way bodies are perceived. It promises the possibility of interaction between one or more subjects, spatially dispersed but able to interact with each other and a shared environment through a computer terminal, acting invasively in increasingly complex situations characterized by increasingly hybrid and subtly designed technology [33]. Much of the excitement generated by virtual reality has to do with the expendability or redundancy of the body. An existence without debt, without obligations, without attachments, the fantasy of the liberal subject creating itself. This fantasy denies the connection between the sexualized body and the sexualized subject. It is a unique and unchanging body that is much more liberal than restrictive, Grosz argues [34].

#### **6. Perceptual literacy instead of visual literacy**

The artist is a central figure in the creation of new knowledge. He has the potential to transform the way people see and think by providing new insights into the way objects carry meaning about themes, ideas, and issues. Images forward meaning in many ways, be it through description, representation, expression, or symbolization. Recent cultural discourse has changed the relationship between the artwork, the artist, and the viewer. It offers a much greater scope for meaning to emerge from an encounter with a work of art.

Researchers in visual culture are interested in the communicative and political role of art and help understand the contexts surrounding art so that it is possible to control the visual information the viewer confronts. The approach is to seek understanding by researching art to determine the many functions and purposes for which art can be used [35].

A genuine understanding of information should develop by conceiving it as a complex consisting of at least visual, auditory, and verbal stimuli combined in specific ways.

In view of this, it would be possible to extend the approach that transforms visual culture/education into perceptual culture/education or media culture/education. With such predisposition, it would be easier to contextualize visual or auditory artistic production that stimulates sensitive experiences and produces a holistic critical perception of the world, insofar as various forms of manipulation are effective when their components address unconnected perceptual capacities. This process should

include learning art concepts, as it is necessary to understand auditory/visual art signs as carriers of meaning and to be able to interpret and use them in creative artistic expression, especially from an art education perspective.

One of the most systematic approaches to fostering a critical attitude would be to develop a wholesale connection of the artwork with daily life conditions. As Dewey wrote, "We fail to see how the works we encounter in museums or their equivalents for other art forms, such as concert halls or classrooms, have actually grown out of the common conditions of life we share with the artists who created them. After that, we can make a second mistake. If we believe that aesthetic experience belongs to a delimited realm, we cannot see how the success of artists in creating expressive, intrinsically fulfilling objects from the raw material of life can be applied to the entire spectrum of human existence… The purpose of aesthetics is to restore continuity between the refined and intensified forms of experience that works of art represent and the everyday events, actions, and sufferings that are generally recognised as experience" [36].

#### **7. Conclusion**

Dynamic sociocultural changes have affected artistic expressions of all kinds; debates about the cultural identity of minority groups, questions of national identity, rapid technological changes, and the emergence of the postmodern philosophy of fragmentation and plurality have reshaped assumptions about the promotion of the arts and education. These changes have affected the approach to the arts. However, also how to learn and teach about them. It is important to approach arts education from a critical perspective that addresses the complexity of experiences strongly integrated into everyday life today.

Due to multimedia technologies, we are often disturbed by visual, auditory, or verbal images and other sources, and we must constantly respond to them and make decisions that require creativity, uniqueness, spatial awareness, motivation, and imagination. However, the abundance of visual cues should not prevent us from having rich experiences with the other senses, especially hearing, closely related to the visual.

For that reason, the multisensory orientation of intermediality corresponds to contemporary trends in art and culture. It enables art educators to encourage the esthetic imagination necessary to attract students and participate in contemporary art and cultural activities. It also allows them to appreciate and understand cultural history holistically, using hybrid methods and environments for teaching and learning in a holistic way [37].

In challenging the current shift in art education toward a visual culture position, these authors argue that art education would be better served if, instead of adopting a visual culture perspective, it adopted broadly holistic forms and practices that can be critically examined through the interdisciplinary, multidisciplinary, and transdisciplinary methods associated with intermedial approaches to study. Vision-centered education, or education that focuses only on the traditional arts disciplines prepares students for art and culture in far too myopic ways.

We support these considerations insofar as we have shown how conditional the importance of vision is, considering it as the only sense that can bring about improvements in the development of spatial representation besides emotional receptivity to the perceived world.

An effective critical attitude toward the world should promote the formation of critical perceivers of the environment who are able to deal constructively with possible dissonance. This should lead the viewer/listener/student to new discoveries, to the representation of their own desires or feelings, and to the spontaneous valorization of their experiences, thus enabling changes in responses to the manipulation of visual and other stimuli in our environment. These ideas become even more important when we consider the individuality of each viewer/consumer/student living in a specific cultural environment. Arts education can be a way to develop understanding of the world create holistic representations of it, and promote creative and critical thinking through intermedia esthetic dimensions.

The esthetic dimension is a unique cognitive process. A global understanding of our past, as well as our present world, requires complex elements and rich unifying experiences, which should be one of the main goals of education at all levels and a key to personal and social growth and emancipation from the various forms of 'cultural slavery' that impose themselves at all levels of our globalized world.

Finally, we engage with media because of their material, sensorial and spatiotemporal qualities. We engage with media products because they mean something. The qualities of media products provide information that we understand to be a representation of something else. Thus, media products employ our ability to make meaning of signs [38]. A key competence in contemporary times.

#### **Acknowledgements**

I could not accomplish this task without the cooperation of colleagues who understood the importance of the challenge. I sincerely thank Samo Amon for his critical ideas and patience during the writing process and Primož Urbanč for accepting the challenge of the project. Finally yet importantly, I would like to thank the students and teachers who participated in the project. We hope that with them, we will be able to promote changes in the way art education is defined and implemented. Our final goal is the development of creativity. Without creativity, we cannot imagine our future life.

#### **Author details**

Bea Tomšič Amon University of Ljubljana, Ljubljana, Slovenia

\*Address all correspondence to: bea.tomsic@pef.uni-lj.si

© 2023 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

### **References**

[1] Eilittä L. Intermedial arts. In: Eilittä L, Louvel L, Kim S, editors. Intermedial Arts: Disrupting, Remembering and Transforming Media. Newcastle upon Tyne: Cambridge Scholars Publishing; 2012

[2] Bruhn J, Schirrmacher B. Intermedial studies. In: Bruhn J, Schirrmacher B, editors. Intermedial Studies - an Introduction to Meaning across Media. London: Routledge; 2021. p. 22. DOI: 10.4324/9781003174288

[3] UNESCO. RoadMap for arts education. The World Conference on Arts Education: Building Creative Capacities for the 21st Century. 2006. Available from: http://www.unesco.org/ new/fileadmin/MULTIMEDIA/HQ/CLT/ CLT/pdf/Arts\_Edu\_RoadMap\_en.pdf. [Accessed: December 19, 2021]

[4] Kroflič R. Teoretski pristopi k načrtovanju kurikuluma [Theoretical Approaches to Planning Curricula]. Ljubljana: Centre for the Development of the University; 1992. p. 10

[5] Rutten K, Soetaert R. Intermediality, Rhetoric, and Pedagogy. CLCWeb: Comparative Literature and Culture 13.3. Indianapolis: Purdue University Press; 2011. DOI: 10.7771/1481-4374.1797

[6] Egan K. The Educated Mind: How Cognitive Tools Shape our Understanding. Chicago: University of Chicago Press; 1997

[7] Müller JE. Intermediality: A Plea and some theses for a new approach in media studies. In: Lagerroth UB, Lund H, Hedling E, editors. Interart Poetics: Essays on the Interrelations of the Arts and Media. Amsterdam: Rodopi; 1997. pp. 295-304

[8] Rajewsky IO. Intermediality, intertextuality, and remediation: A literary perspective on Intermediality. Intermédialités / Intermediality. 2005;**6**:43-64

[9] Semali LM, Watts Pailliotet A. What is intermediality and why study it in U.S. Classrooms? In: Semali LM, Watts Pailliotet A, editors. The Teachers Handbook of Critical Media Literacy. Boulder, Colorado: Westview Press; 1999. p. 5

[10] Semali LM, Watts PA. What is Intermediality and why study it in U.S. Classrooms? In: Semali LM, Watts Pailliotet A, editors. The Teachers Handbook of Critical Media Literacy. Boulder, Colorado: Westview Press; 1999. pp. 1-31

[11] Rand G, Zakia RD. Teaching Photography: Tools for the Imaging Educator. London: Taylor & Francis; 2006

[12] Lowenfeld V, Brittain L. Creative and Mental Growth, Fourth Edition. New York: The Macmillan Company. London: Collier'Macmillan Limited; 1964

[13] Edwards B. Drawing on the Dominant Eye - Decoding the Way we Perceive, Create and Learn. London: Profile Books Ltd; 2020

[14] Erickson HL. Stirring the Head, Heart, and Soul: Redefining Curriculum and Instruction. Thousand Oaks California: Corwin Press; 1995

[15] Drake SM. Creating Integrated Curriculum: Proven Ways to Increase Student Learning. Thousand Oaks California: Corwin Press; 1998

[16] Rutten K, Soetaert R. Intermediality, Rhetoric, and Pedagogy. CLCWeb: Comparative Literature and Culture 13.3. Indianapolis: Purdue University Press; 2011. p. 4. DOI: 10.7771/1481-4374.1797

[17] Giroux HA. Border youth, difference, and postmodern education. In: Castells M, Flecha R, Freire P, Giroux HA, Macedo D, Willis P, editors. Critical Education in the Information Age. Lanham: Rowman & Littlefield; 1999. p. 112

[18] Giroux HA. Border youth, difference, and postmodern education. In: Castells M, Flecha R, Freire P, Giroux HA, Macedo D, Willis P, editors. Critical Education in the Information Age. Lanham: Rowman & Littlefield; 1999. pp. 93-115

[19] Dewey J. Art as Experience. London: Perigee Books; 1980

[20] Kubovy M. Should we resist the seductiveness of the space time: Vision, audition analogy? Journal of Experimental Psychology: Human Perception and Performance, American Psychological Association. 1998;**14**:318-320

[21] Bolter JD, Grusin R. Remediation: Understanding New Media. Cambridge, Masachussets, London: The MIT Press; 1999

[22] Wightman FL, Jenison R. Auditory spatial layout. In: Epstein W, Rogers SJ, editors. Perception of Space and Motion. San Diego California: Academic Press; 1995. pp. 400-465

[23] Reznikoff I. On the sound dimension of prehistoric painted caves and rocks. In: Tarasti E, editor. Musical Signification, Essays in the Semiotic Theory and Analysis of Music. Berlin, New York: Mouton de Gruyter; 1995. pp. 541-558

[24] Düchting H. Paul Klee – Painting Music. Munich, Berlin, London, New York: Prestel Verlag; 2004. pp. 48-65

[25] Valdez Del Álamo E. Hearing the image at Santo Domingo de silos. In: Boynton S, Reilly DJ, editors. Resounding Images: Medieval Intersections of Art, Music, and Sound. Turnhout: Brepols Publishers n.v; 2015. p. 71

[26] Düchting H. Paul Klee – Painting Music. Munich, Berlin, London, New York: Prestel Verlag; 2004. p. 17

[27] Mitchell WJT. There are no visual media. Journal of Visual Culture. 2005;**4**(2):257-266

[28] Coelho C. The Living Experience as a design content: from concept to appropriation. Paris: Direction Générale des Patrimoines — DAPA — MCC, UMR 1563, Ambiances Architectures Urbanités (AAU). 2019. Available from: http:// journals.openedition.org/ambiances/606. [Accessed: September 8, 2021]

[29] Mitrache A. Spatial sensibility in architectural education. Procedia — Social and Behavioral Sciences. 2013;**93**:544-548

[30] Pallasmaa J. The Embodied Image. Imagination and Imaginery in Architecture. London, New York: John Wiley & Sons Ltd.; 2011. p. 50

[31] Crary J. Suspensions of Perception — Attention, Spectacle, and Modern Culture. Cambridge-Massachusetts, London: The MIT Press; 2001. pp. 2-13

[32] Virtual GO, Art. From Illusion to Immersion. Cambridge-Massachusetts. London: The MIT Press; 2003. p. 17

[33] Hoelzl I. Image-transaction, what you see is not what you get. In: Grau O, Hinterwaldner I, editors. Retracing

*Intermediality in Contemporary Visual Art Education DOI: http://dx.doi.org/10.5772/intechopen.110873*

Political Dimensions, Strategies in Contemporary New Media Art. New York: Walter De Gruyter GmbH; 2021. pp. 19-32

[34] Grosz E. Architecture from the Outside, Esays on Virtual and Real Space. Cambridge-Massachusetts, London: The MIT Press; 2001

[35] Sullivan G. Seeing visual culture. Studies in Art Education. 2003;**44**(3):194-197

[36] Hickman L. Reading Dewey - Interpretations for a Postmodern Generation. Bloomington and Indianapolis: Indiana University Press; 1998. p. 6

[37] Blandy D, Bolin PE. Beyond visual culture: Seven statements of support for material culture studies in Art education. Studies in Art Education. 2003;**44**(3):246-263

[38] Bruhn J, Schirrmacher B. Intermedial studies. In: Bruhn J, Schirrmacher B, editors. Intermedial Studies - An Introduction to Meaning across Media. London: Routledge; 2021. DOI: 10.4324/ 9781003174288

### *Edited by Asun López-Varela Azcárate*

*The Intermediality of Contemporary Visual Arts* explores a range of topics within the field. The volume delves into the realm of intermediality within the visual arts. Each chapter explores a different aspect; from the evolution of Intermedial Studies over the past decades to the shifts in print typography and the emergence of "cut-ups" within a context of resistance against conventions, the concept of Visual Music and its relation to pioneering filmmaking, visual representations of intimacy as they evolve from painting to other visual formats like comics, film, and television, and finally the transmedial potential of cultural symbols in virtual reality, all of which involve greater multimodal and emotional elements that enhance audience immersion. The volume closes by highlighting the need for a comprehensive approach to visual art education and pedagogical methods that foster creativity, emphasizing the intermedial aspects present in contemporary visual arts.

Published in London, UK © 2023 IntechOpen © Giorez / iStock

The Intermediality of Contemporary Visual Arts

The Intermediality of

Contemporary Visual Arts

*Edited by Asun López-Varela Azcárate*