**3. The production process**

#### **3.1 The libretto**

Anyone who has worked on a screenplay for a film or a TV episode will attest to this fact—there is nothing "linear" about the process of developing a viable scenario in a group production enterprise. The process took about eight major iterations to arrive at something workable [19], several of these after we had made substantial progress in other areas of the production. Indeed, the scenario and its attendant libretto were still being modified during the final stages of the production, although the changes at that point were limited. The initial problem was to take a 150 page novel and extract from it a scenario that could be presented in a dramatic way within a 20 min span of time. Also, there is very little documentation that exists to guide anyone attempting to develop an opera or write a libretto. The only scholarly text we could find was a small memoire written in 1914 [18] that drew on analysis of existing, mostly nineteenth century operas to make a few suggestions, although there is also information to be gleaned from discussions concerning whether the text or the music should have primacy [20] as well as the process of bringing meraviglia into opera [15].

The original story was focused around the character of Oreph Sodenheim, a young teenager who is the main protagonist of the novel *Pinnacle*: *The First Book of Eng*. Oreph is a central figure for the larger cycle of 15 novels that collectively forms *The Ido Chronicles* [17]. However, in Pinnacle, he passes through a series of events that lead to the destruction of his family. The subject matter, overall, is

**87**

adjustments (iteration #8).

*Designing a Participatory and Interactive Opera DOI: http://dx.doi.org/10.5772/intechopen.82811*

the Humanitat.

therefore depressing. In addition, the main theme of the story is concerned with being bullied, both by conservative factions within the society of the planet called Pinnacle, and also by mysterious groups that belong to the larger civilization called

also involving conflict) and which better reflected the title Pinnacle.

As Edwards was working on the compression of the original story to its essential elements—about eight scenes were identified as key, but this remained too long for the planned production—an opportunity came up to develop a short story based on what was originally a secondary character in Pinnacle, that of Oreph's girlfriend, Relliana Sapuro. The short story that resulted from this effort provided a much better framing for the opera, and iterations #2 and #3 involved recasting the original story to include Relliana (iteration #2), and eventually to focus exclusively on Relliana's story (iteration #3). This led to a much more compact scenario, now limited to four scenes, and a story which was overall much more uplifting (although

In the revised version, Relliana when first encountered is complaining about a set of unusual aches and pains she is experiencing in her body, while also engaging with a friend, Aki Oneya, who is a "gender morph," that is, a person who is bi-gendered. Aki is being bullied by a group of young conformists. Later, Relliana encounters one of the bullies, Mandra Lakso, as it is revealed that her body is undergoing some sort of profound transformation, although we still do not know what this is leading to. During a performance event at the end of the story, Relliana's transformation completes itself and a dragonfly-like winged creature emerges in Relliana's place—the first stage of a new "splinter species" of the human race. By this time, a stage design had been completed (see Section 2.2 below), and the next iteration led to a reorganization of the scenes so that all the different exchanges could take place within the same physical setting (iteration #4). In addition, the discussions concerning interactions (see Section 2.3 below) had singled out the idea of a finale that would be affected by audience participation, and so the climax of the scenario/ libretto was re-written to accommodate a multiplicity of endings (iteration #5).

It should be noted that although in principle the scenario and the libretto are distinct, in practice, given the time constraints of the production schedule, a draft version of the libretto was created as early as iteration #2 of the scenario. The scenario may be viewed as the general description of unfolding events and interactions, whereas the

Interaction elements were also introduced earlier in the scenario, and it was realized that an introductory segment would be useful. As a result, an introduction was written and incorporated into the libretto (iteration #6). During the work on the music, it was realized that the libretto lacked an intimate contact with Relliana and her difficulties—as a result, a new aria was introduced into the libretto (iteration #7). The lead-up to the interactive component in the final scene also needed some

The libretto itself was written in metered verse, but drawing on principles such as harnessing the natural rhythms of the English language rather than slaving the verses to its metric structure. In addition, although rhyming structures were used, they were not consistently maintained at all times, again, to break up the regularity of the resulting text. Because the scenario was science fiction, a certain amount of exposition was required—this made for relatively dense lyrical structures, that is,

The meter and the length of lines were, however, varied across different arias and recitatives as a reflection of the emotional intensities and information being conveyed. Voice range was determined by the available singers—that is, each aria was developed as a function of the vocal range of the singers. This probably follows traditional practice in operatic production—historically, many arias were designed

libretto meant the full verse realization of the text to be sung or declaimed.

arias that contained significant amounts of information.

*Interactive Multimedia - Multimedia Production and Digital Storytelling*

within our original discussions about interactivity, the presence of augmented or virtual reality avatars was implicitly assumed, although exactly how these would be

We also viewed the process as having an addendum—in parallel to, or following, the development of the opera, we would develop an online environment for co-creation, but in our original conception, these tasks were viewed as separate. In the next section, we shall see how the design and production process actually occurred and what the differ-

Anyone who has worked on a screenplay for a film or a TV episode will attest to this fact—there is nothing "linear" about the process of developing a viable scenario in a group production enterprise. The process took about eight major iterations to arrive at something workable [19], several of these after we had made substantial progress in other areas of the production. Indeed, the scenario and its attendant libretto were still being modified during the final stages of the production, although the changes at that point were limited. The initial problem was to take a 150 page novel and extract from it a scenario that could be presented in a dramatic way within a 20 min span of time. Also, there is very little documentation that exists to guide anyone attempting to develop an opera or write a libretto. The only scholarly text we could find was a small memoire written in 1914 [18] that drew on analysis of existing, mostly nineteenth century operas to make a few suggestions, although there is also information to be gleaned from discussions concerning whether the text or the music should have primacy [20] as well as the process of bringing

The original story was focused around the character of Oreph Sodenheim, a young teenager who is the main protagonist of the novel *Pinnacle*: *The First Book of Eng*. Oreph is a central figure for the larger cycle of 15 novels that collectively forms *The Ido Chronicles* [17]. However, in Pinnacle, he passes through a series of events that lead to the destruction of his family. The subject matter, overall, is

integrated was unclear. The diagram incorporates these assumptions.

ences between plan and realization taught us about these processes.

**3. The production process**

*Original concept of the design process.*

meraviglia into opera [15].

**3.1 The libretto**

**Figure 1.**

**86**

therefore depressing. In addition, the main theme of the story is concerned with being bullied, both by conservative factions within the society of the planet called Pinnacle, and also by mysterious groups that belong to the larger civilization called the Humanitat.

As Edwards was working on the compression of the original story to its essential elements—about eight scenes were identified as key, but this remained too long for the planned production—an opportunity came up to develop a short story based on what was originally a secondary character in Pinnacle, that of Oreph's girlfriend, Relliana Sapuro. The short story that resulted from this effort provided a much better framing for the opera, and iterations #2 and #3 involved recasting the original story to include Relliana (iteration #2), and eventually to focus exclusively on Relliana's story (iteration #3). This led to a much more compact scenario, now limited to four scenes, and a story which was overall much more uplifting (although also involving conflict) and which better reflected the title Pinnacle.

In the revised version, Relliana when first encountered is complaining about a set of unusual aches and pains she is experiencing in her body, while also engaging with a friend, Aki Oneya, who is a "gender morph," that is, a person who is bi-gendered. Aki is being bullied by a group of young conformists. Later, Relliana encounters one of the bullies, Mandra Lakso, as it is revealed that her body is undergoing some sort of profound transformation, although we still do not know what this is leading to. During a performance event at the end of the story, Relliana's transformation completes itself and a dragonfly-like winged creature emerges in Relliana's place—the first stage of a new "splinter species" of the human race.

By this time, a stage design had been completed (see Section 2.2 below), and the next iteration led to a reorganization of the scenes so that all the different exchanges could take place within the same physical setting (iteration #4). In addition, the discussions concerning interactions (see Section 2.3 below) had singled out the idea of a finale that would be affected by audience participation, and so the climax of the scenario/ libretto was re-written to accommodate a multiplicity of endings (iteration #5).

It should be noted that although in principle the scenario and the libretto are distinct, in practice, given the time constraints of the production schedule, a draft version of the libretto was created as early as iteration #2 of the scenario. The scenario may be viewed as the general description of unfolding events and interactions, whereas the libretto meant the full verse realization of the text to be sung or declaimed.

Interaction elements were also introduced earlier in the scenario, and it was realized that an introductory segment would be useful. As a result, an introduction was written and incorporated into the libretto (iteration #6). During the work on the music, it was realized that the libretto lacked an intimate contact with Relliana and her difficulties—as a result, a new aria was introduced into the libretto (iteration #7). The lead-up to the interactive component in the final scene also needed some adjustments (iteration #8).

The libretto itself was written in metered verse, but drawing on principles such as harnessing the natural rhythms of the English language rather than slaving the verses to its metric structure. In addition, although rhyming structures were used, they were not consistently maintained at all times, again, to break up the regularity of the resulting text. Because the scenario was science fiction, a certain amount of exposition was required—this made for relatively dense lyrical structures, that is, arias that contained significant amounts of information.

The meter and the length of lines were, however, varied across different arias and recitatives as a reflection of the emotional intensities and information being conveyed. Voice range was determined by the available singers—that is, each aria was developed as a function of the vocal range of the singers. This probably follows traditional practice in operatic production—historically, many arias were designed

with particular singers in mind, although well-established composers could also pick and choose at will, permitting greater flexibility. Additional small changes were made to the lyrics during the musical composition stage (see Section 3.4 below).

**Figures 2**–**4** show some examples of the libretto lyrics that highlight these considerations.

#### **3.2 Stage and 3D design**

As outlined earlier, the original concept was to develop a physical staging of the opera that would incorporate virtual reality or augmented reality elements in support of interactions with users. The design was developed and organized by Morales and his students based on a series of initial discussions about the scenario/ libretto as well as the larger opus that framed the opera. Indeed, Morales and Edwards led several co-design sessions held at the CIRRIS1 . The design was initially organized around the opening scene of the opera, which consisted of a ledge on the side of the airborne city called the Orr Enclave. The Orr Enclave was conceived as being roughly cubical in shape (**Figure 5**), but with numerous extensions and indentations down its sides, one of which formed the stage for the opening scene of the scenario. It became evident that the cost constraints of the production meant that we needed the one stage to support all scenes. For this to work, in addition to the ledge, we needed the main area to have sufficient depth to accommodate bully dancers, we needed a kind of gate-like area that might represent a school playground or patio, and we needed a round "portal" that would provide access to images of diverse types. One of the most important challenges was to decide which parts of the staging should be real and which virtual. We initially assumed that the production would involve both elements within an augmented reality type staging. To do this, the virtual elements needed to be fit perfectly and seamlessly with the real stage objects. We developed and tested virtual components that would operate in this way. In addition, we conceived of the use of VR goggles that could be provided to the public, and which would allow users to select among a range of display options. During the co-design sessions, Morales and Edwards generated many

#### **Figure 2.**

**89**

**Figure 4.**

**Figure 3.**

something simple.

*transformation she was going through.*

drawings to ensure staging ideas were clear for everyone involved. In addition, we were conscious that a complex stage would be costly to build, so we were looking for

*These lyrics are part of the additional aria added for Relliana to sing, to explain her own feelings about the* 

*Example of the lyrics sung by the conformist bullies—the first two verses of the song. Here the meter changes between the snappy chorus and the more substantive main verse. The rhyme scheme for the main verse is abcadeddd, so, again, some regularity and some variation, whereas the chorus is much more structured.*

*Designing a Participatory and Interactive Opera DOI: http://dx.doi.org/10.5772/intechopen.82811*

*Lyrics sung by Aki, the "gender morph" (bi-gendered person). "Moelen" is the term given to the fold of skin that can curl to form either a pseudo-penis or pseudo-vagina as the situation calls for. The text highlights the use of metric verse and near but not perfect rhymes to break up the regularity of the poetry (e.g., "frayed" is a near rhyme to "sage").*

<sup>1</sup> Centre interdisciplinaire de recherche en réadaptation et integration sociale.

*Designing a Participatory and Interactive Opera DOI: http://dx.doi.org/10.5772/intechopen.82811*

**Figure 3.**

*Interactive Multimedia - Multimedia Production and Digital Storytelling*

Edwards led several co-design sessions held at the CIRRIS1

<sup>1</sup> Centre interdisciplinaire de recherche en réadaptation et integration sociale.

*Lyrics sung by Aki, the "gender morph" (bi-gendered person). "Moelen" is the term given to the fold of skin that can curl to form either a pseudo-penis or pseudo-vagina as the situation calls for. The text highlights the use of metric verse and near but not perfect rhymes to break up the regularity of the poetry (e.g., "frayed" is a near* 

considerations.

**3.2 Stage and 3D design**

with particular singers in mind, although well-established composers could also pick and choose at will, permitting greater flexibility. Additional small changes were made to the lyrics during the musical composition stage (see Section 3.4 below). **Figures 2**–**4** show some examples of the libretto lyrics that highlight these

As outlined earlier, the original concept was to develop a physical staging of the opera that would incorporate virtual reality or augmented reality elements in support of interactions with users. The design was developed and organized by Morales and his students based on a series of initial discussions about the scenario/ libretto as well as the larger opus that framed the opera. Indeed, Morales and

organized around the opening scene of the opera, which consisted of a ledge on the side of the airborne city called the Orr Enclave. The Orr Enclave was conceived as being roughly cubical in shape (**Figure 5**), but with numerous extensions and indentations down its sides, one of which formed the stage for the opening scene of the scenario. It became evident that the cost constraints of the production meant that we needed the one stage to support all scenes. For this to work, in addition to the ledge, we needed the main area to have sufficient depth to accommodate bully dancers, we needed a kind of gate-like area that might represent a school playground or patio, and we needed a round "portal" that would provide access to images of diverse types. One of the most important challenges was to decide which parts of the staging should be real and which virtual. We initially assumed that the production would involve both elements within an augmented reality type staging. To do this, the virtual elements needed to be fit perfectly and seamlessly with the real stage objects. We developed and tested virtual components that would operate in this way. In addition, we conceived of the use of VR goggles that could be provided to the public, and which would allow users to select among a range of display options. During the co-design sessions, Morales and Edwards generated many

. The design was initially

**88**

**Figure 2.**

*rhyme to "sage").*

*Example of the lyrics sung by the conformist bullies—the first two verses of the song. Here the meter changes between the snappy chorus and the more substantive main verse. The rhyme scheme for the main verse is abcadeddd, so, again, some regularity and some variation, whereas the chorus is much more structured.*

#### **Figure 4.**

*These lyrics are part of the additional aria added for Relliana to sing, to explain her own feelings about the transformation she was going through.*

drawings to ensure staging ideas were clear for everyone involved. In addition, we were conscious that a complex stage would be costly to build, so we were looking for something simple.

#### **Figure 5.**

*The Orr Enclave, the floating city that serves as the main setting for the events that take place in both Pinnacle: The First Book of Eng (the original novel) and Pinnacle: A One Act Participatory Opera (the opera).*

#### **Figure 6.**

*The basic stage design. The set consisted of a ledge on the side of the city, an oval area onto which images could be projected, a fabric structure (the curved form), and steps from the main stage to the protrubing lower ledge where the nemo plants were to be situated.*

The early designs therefore incorporated these different factors (see **Figure 6**). Since it is supposed that the city itself was composed largely of fabric materials along with some structural, weight bearing elements (that is, because the city floats in the atmosphere, it must be made out of light materials), the final stage design

**91**

**Figure 7.**

*Designing a Participatory and Interactive Opera DOI: http://dx.doi.org/10.5772/intechopen.82811*

we shall show the staging area used in the final production.

incorporated fabric, cord and aluminum struts into an elegant design that could be relatively inexpensive to construct. Different variants of the ledge structure and upper area were proposed—only one of these is shown in **Figure 6**. Further along

The production almost stalled when it was realized that our budget was probably not adequate to cover the stage construction costs. It was at this point in the development that the team decided to go with a virtual staging which was much easier to manage within the context of the grant monies, even though the final costs may have been roughly commensurate to those involved in a physical staging. The difference was that the costs of the virtual production went primarily towards paying students rather than into the construction materials themselves, which would have required storage, which needed to be dismantled and then rebuilt and so robust against such manipulations, and so forth. No doubt a more experienced production team would have known how to circumvent these issues more effectively, although the budget was fixed and rather modest. The advantage of shifting the production towards a virtual staging meant that the two initiatives, which were initially viewed as separate, could be merged together into common development protocols—that is, the construction of the opera and the construction of an online co-creation environment. The latter effort had been focused towards the creation of a 3D version of the "cube city." Edwards had initially developed some rudimentary 3D models of the city (see **Figure 7**) that showed both its internal structure and the locations of key scenes. Drawing on the architectural expertise of students such as Alicia Lamontage, under guidance from Edwards, a rough design for the top of the "cube city" was developed. Within this structure, several key features that were present in the original novel (*Pinnacle*: *The First Book of Eng*) were given a detailed structure—the Agora space in which major events took place, the sunken Concourse area where the final performance was originally planned, the Portal building, where information about other planets and habits could be accessed, and the threader stations where the spindle-like transport vehicles, which moved within dedicated tubes, could be boarded. In addition, for the online co-creation environment, an additional location was eventually intro-

duced, a monument space along the front edge of the city top (**Figure 8**).

The disadvantage with the shift from a physical to a virtual production was that we needed to work harder to find ways to bring "embodiment issues" back into the production of what would be essentially a virtual opera (see next section). We felt

*Early design for the top surface of the cube-shaped city of Orr Enclave; the cleft is the location of the concourse,* 

*the blue dome at the right is the Agora and the arched shape at the left is the monument.*

#### *Designing a Participatory and Interactive Opera DOI: http://dx.doi.org/10.5772/intechopen.82811*

*Interactive Multimedia - Multimedia Production and Digital Storytelling*

The early designs therefore incorporated these different factors (see **Figure 6**). Since it is supposed that the city itself was composed largely of fabric materials along with some structural, weight bearing elements (that is, because the city floats in the atmosphere, it must be made out of light materials), the final stage design

*The basic stage design. The set consisted of a ledge on the side of the city, an oval area onto which images could be projected, a fabric structure (the curved form), and steps from the main stage to the protrubing lower ledge* 

*The Orr Enclave, the floating city that serves as the main setting for the events that take place in both Pinnacle: The First Book of Eng (the original novel) and Pinnacle: A One Act Participatory Opera (the opera).*

**90**

**Figure 6.**

*where the nemo plants were to be situated.*

**Figure 5.**

incorporated fabric, cord and aluminum struts into an elegant design that could be relatively inexpensive to construct. Different variants of the ledge structure and upper area were proposed—only one of these is shown in **Figure 6**. Further along we shall show the staging area used in the final production.

The production almost stalled when it was realized that our budget was probably not adequate to cover the stage construction costs. It was at this point in the development that the team decided to go with a virtual staging which was much easier to manage within the context of the grant monies, even though the final costs may have been roughly commensurate to those involved in a physical staging. The difference was that the costs of the virtual production went primarily towards paying students rather than into the construction materials themselves, which would have required storage, which needed to be dismantled and then rebuilt and so robust against such manipulations, and so forth. No doubt a more experienced production team would have known how to circumvent these issues more effectively, although the budget was fixed and rather modest.

The advantage of shifting the production towards a virtual staging meant that the two initiatives, which were initially viewed as separate, could be merged together into common development protocols—that is, the construction of the opera and the construction of an online co-creation environment. The latter effort had been focused towards the creation of a 3D version of the "cube city." Edwards had initially developed some rudimentary 3D models of the city (see **Figure 7**) that showed both its internal structure and the locations of key scenes. Drawing on the architectural expertise of students such as Alicia Lamontage, under guidance from Edwards, a rough design for the top of the "cube city" was developed. Within this structure, several key features that were present in the original novel (*Pinnacle*: *The First Book of Eng*) were given a detailed structure—the Agora space in which major events took place, the sunken Concourse area where the final performance was originally planned, the Portal building, where information about other planets and habits could be accessed, and the threader stations where the spindle-like transport vehicles, which moved within dedicated tubes, could be boarded. In addition, for the online co-creation environment, an additional location was eventually introduced, a monument space along the front edge of the city top (**Figure 8**).

The disadvantage with the shift from a physical to a virtual production was that we needed to work harder to find ways to bring "embodiment issues" back into the production of what would be essentially a virtual opera (see next section). We felt

#### **Figure 7.**

*Early design for the top surface of the cube-shaped city of Orr Enclave; the cleft is the location of the concourse, the blue dome at the right is the Agora and the arched shape at the left is the monument.*

#### **Figure 8.**

*(Above) The monument site and the threader transport system (tubes), in the early design, and (below) as later implemented within Unity in the co-creation environment, along with the user avatar (labeled) and the dancing avatars.*

from the beginning that the opera needed to be "embodied"—part of both Edwards' and Kiss' larger research programs concern how to enhance embodiment effects within environments that may be partially or wholly virtual. Indeed, our use of live singing/declaiming was also an attempt to enhance the feelings of presence and embodiment ([21], see also Section 2.4 below).

Furthermore, to return to the opera staging, the 3D city design allowed us to stage the opera on the ledge on the side of the city. Rather than redesign this area, the design of the physical stage was simply incorporated into the 3D city design, resulting in the final "stage" for the virtual opera (**Figure 9**). This elegant solution meant that if, at some future occasion, we should want to physically stage the opera, we could do so with no substantive changes to the overall set design. Furthermore, the approach used aligns with cognitive design principles [22, 23].

**93**

2.10 below).

**Figure 9.**

**3.3 Interactive components**

*Aki (in blue) and Relliana in gray and brown next to hir.*

influence the plot itself.

*Designing a Participatory and Interactive Opera DOI: http://dx.doi.org/10.5772/intechopen.82811*

The final virtual rendition of the stage was carried out by a visual arts student, Jonathan Proulx Guimond. As we shall discuss further on, Jonathan also did extensive development, in tandem with our engineering student, Juan Nino Falcon, of the more extensive virtual environment used for the co-creation endeavor (see Section

*Virtual stage for the opera, incorporated into the 3D city design. Shown is the scene with the bully dancers, with* 

As might be expected, designing and incorporating interactive elements into the opera posed enormous challenges to the production team. The problem was to allow the audience to influence the unfolding story in manageable ways that were thematically compelling in the frame of the story itself. Hence, for example, many potential interactions discussed were discarded either because they would have been too costly to do effectively or alternatively, because they allowed game-like

Any attempt to use the audience's interactions to modify the story potentially runs into the problem of combinatorics. If a choice of two outcomes is introduced, then two stories need to be constructed. If a second choice of two outcomes is introduced, at another point of the story, then four versions of the story need to be determined. For a third decision point, we are now at eight versions of the story, and so on. This quickly becomes unmanageable. Therefore a "management strategy" needs to be developed to allow variation without leading to such combinatory effects. For example, audiences could be given the ability to modify aspects of the staging that concern appearance (i.e., esthetics) but which do not substantially

Early on in the design process, even during the stage when we were considering a physical staging with additional virtual elements via augmented reality techniques, we developed an approach where each audience member could have their own "access device" within which they could be given more flexibility for customization than in the common staging, whether the latter be physical or virtual. Possibilities discussed for these devices included introducing virtual flora and fauna that could be influenced by the audience member (**Figure 10**), superimposing a photograph of the audience member's face on one of the actor's (or avatar's) faces so that the

interactions that were, however, poorly integrated with the story line.

#### **Figure 9.**

*Interactive Multimedia - Multimedia Production and Digital Storytelling*

from the beginning that the opera needed to be "embodied"—part of both Edwards' and Kiss' larger research programs concern how to enhance embodiment effects within environments that may be partially or wholly virtual. Indeed, our use of live singing/declaiming was also an attempt to enhance the feelings of presence and

*(Above) The monument site and the threader transport system (tubes), in the early design, and (below) as later implemented within Unity in the co-creation environment, along with the user avatar (labeled) and the* 

Furthermore, to return to the opera staging, the 3D city design allowed us to stage the opera on the ledge on the side of the city. Rather than redesign this area, the design of the physical stage was simply incorporated into the 3D city design, resulting in the final "stage" for the virtual opera (**Figure 9**). This elegant solution meant that if, at some future occasion, we should want to physically stage the opera, we could do so with no substantive changes to the overall set design. Furthermore, the approach used aligns with cognitive design

embodiment ([21], see also Section 2.4 below).

**92**

**Figure 8.**

*dancing avatars.*

principles [22, 23].

*Virtual stage for the opera, incorporated into the 3D city design. Shown is the scene with the bully dancers, with Aki (in blue) and Relliana in gray and brown next to hir.*

The final virtual rendition of the stage was carried out by a visual arts student, Jonathan Proulx Guimond. As we shall discuss further on, Jonathan also did extensive development, in tandem with our engineering student, Juan Nino Falcon, of the more extensive virtual environment used for the co-creation endeavor (see Section 2.10 below).

#### **3.3 Interactive components**

As might be expected, designing and incorporating interactive elements into the opera posed enormous challenges to the production team. The problem was to allow the audience to influence the unfolding story in manageable ways that were thematically compelling in the frame of the story itself. Hence, for example, many potential interactions discussed were discarded either because they would have been too costly to do effectively or alternatively, because they allowed game-like interactions that were, however, poorly integrated with the story line.

Any attempt to use the audience's interactions to modify the story potentially runs into the problem of combinatorics. If a choice of two outcomes is introduced, then two stories need to be constructed. If a second choice of two outcomes is introduced, at another point of the story, then four versions of the story need to be determined. For a third decision point, we are now at eight versions of the story, and so on. This quickly becomes unmanageable. Therefore a "management strategy" needs to be developed to allow variation without leading to such combinatory effects. For example, audiences could be given the ability to modify aspects of the staging that concern appearance (i.e., esthetics) but which do not substantially influence the plot itself.

Early on in the design process, even during the stage when we were considering a physical staging with additional virtual elements via augmented reality techniques, we developed an approach where each audience member could have their own "access device" within which they could be given more flexibility for customization than in the common staging, whether the latter be physical or virtual. Possibilities discussed for these devices included introducing virtual flora and fauna that could be influenced by the audience member (**Figure 10**), superimposing a photograph of the audience member's face on one of the actor's (or avatar's) faces so that the

#### **Figure 10.**

*The nemo plants (blue) emit particles that enhance airborne buoyancy and in addition that respond to audience input (via either voice or breath).*

audience member could experience an enhanced identification with a character of their choice (early experimentation showed this to be possible—see **Figure 11**), and introducing new (virtual) avatars that moved between the actors and which could be influenced by audience members. Furthermore, Kiss believed the possibility existed to incorporate voices from the audience members as well, although this possibility was never fully fleshed out. … In addition, even for a physical staging of the opera, we realized that it would be possible to "add in" the side of the city around the physical stage using augmented reality techniques—again, early experimentation showed this could be achieved sufficiently well to create the impression that the action was taking place on the side of the floating city.

Of course, once we abandoned the idea of doing a physical staging, the integration of virtual elements into the staging was greatly facilitated—we no longer had to worry about anchoring virtual elements into the physical stage (even though we had demonstrated to our own satisfaction that this was feasible).

There were concerns even from the early discussions about the intrusive nature of "viewing technologies," whether these be expensive devices like VR or AR glasses, or cheaper devices such as tablets or smartphones. Among alternatives we discussed (and tested) were the use of simple smart garments—and in the end we did develop and test an interactive belt device.

Another interest was to generate interactive possibilities that allowed audience members to work together to affect the opera, and not merely opportunities for individual customization. Finding an interactive modality that would not corrupt the opera's story line remained a challenge throughout these discussions. Any attempt to provide users with some form of access to or control over avatars on the stage seemed to mean disrupting the way the story evolved. Of course, one option could have been to abandon any effort to control the storyline, and this might have been an interesting endeavor, but having taken the time to select a scenario and construct a libretto, this seemed self-defeating. We therefore sought mechanisms for interaction that preserved the story-line or allowed only for limited variations.

**95**

**Figure 11.**

*object would be invisible or more discrete!).*

*Designing a Participatory and Interactive Opera DOI: http://dx.doi.org/10.5772/intechopen.82811*

Our solution to the problem of allowing audience members to introduce elements that could affect the story in limited ways that nonetheless felt "organic" was to focus on the flora and fauna that form an essential background to the stage setting (in fact, to the world being presented via the staging), and could be allowed to influence the unfolding of the story even if such influences had not been present in the original version (the source novel). The world portrayed in the larger opus (*The Ido Chronicles*) is situated in the far future, at a time when ecosystems are essentially constructed by humans. These constructed ecosystems are designed to provide utilitarian functionality to support a range of human activities as well as to function with each other in ecological ensembles. Among the plants and animals described in the Chronicles are plants called "nemos" and animal hybrids such as "doskies" (combining features of flying squirrels and dogs) or adapted animal species such as "jonahs" (modified sperm whales). Nemos are plants derived from sea anemones that bush outwards via tendril-like leaves that collect water droplets from the air, while "spiners" are adapted from spiders to spin structural elements for buildings. Doskies are a kind of flying or gliding domestic pet. Jonahs have been modified to

*Superimposing a photographed face onto an avatar; above—the software used to capture the image of the face; below—the rendered virtual face using the soda can as the directional reference object (normally, the reference*  *Designing a Participatory and Interactive Opera DOI: http://dx.doi.org/10.5772/intechopen.82811*

*Interactive Multimedia - Multimedia Production and Digital Storytelling*

audience member could experience an enhanced identification with a character of their choice (early experimentation showed this to be possible—see **Figure 11**), and introducing new (virtual) avatars that moved between the actors and which could be influenced by audience members. Furthermore, Kiss believed the possibility existed to incorporate voices from the audience members as well, although this possibility was never fully fleshed out. … In addition, even for a physical staging of the opera, we realized that it would be possible to "add in" the side of the city around the physical stage using augmented reality techniques—again, early experimentation showed this could be achieved sufficiently well to create the impression that the

*The nemo plants (blue) emit particles that enhance airborne buoyancy and in addition that respond to* 

Of course, once we abandoned the idea of doing a physical staging, the integration of virtual elements into the staging was greatly facilitated—we no longer had to worry about anchoring virtual elements into the physical stage (even though we had

There were concerns even from the early discussions about the intrusive nature

Another interest was to generate interactive possibilities that allowed audience members to work together to affect the opera, and not merely opportunities for individual customization. Finding an interactive modality that would not corrupt the opera's story line remained a challenge throughout these discussions. Any attempt to provide users with some form of access to or control over avatars on the stage seemed to mean disrupting the way the story evolved. Of course, one option could have been to abandon any effort to control the storyline, and this might have been an interesting endeavor, but having taken the time to select a scenario and construct a libretto, this seemed self-defeating. We therefore sought mechanisms for interaction that preserved the story-line or allowed only for limited variations.

of "viewing technologies," whether these be expensive devices like VR or AR glasses, or cheaper devices such as tablets or smartphones. Among alternatives we discussed (and tested) were the use of simple smart garments—and in the end we

action was taking place on the side of the floating city.

did develop and test an interactive belt device.

demonstrated to our own satisfaction that this was feasible).

**94**

**Figure 10.**

*audience input (via either voice or breath).*

#### **Figure 11.**

*Superimposing a photographed face onto an avatar; above—the software used to capture the image of the face; below—the rendered virtual face using the soda can as the directional reference object (normally, the reference object would be invisible or more discrete!).*

Our solution to the problem of allowing audience members to introduce elements that could affect the story in limited ways that nonetheless felt "organic" was to focus on the flora and fauna that form an essential background to the stage setting (in fact, to the world being presented via the staging), and could be allowed to influence the unfolding of the story even if such influences had not been present in the original version (the source novel). The world portrayed in the larger opus (*The Ido Chronicles*) is situated in the far future, at a time when ecosystems are essentially constructed by humans. These constructed ecosystems are designed to provide utilitarian functionality to support a range of human activities as well as to function with each other in ecological ensembles. Among the plants and animals described in the Chronicles are plants called "nemos" and animal hybrids such as "doskies" (combining features of flying squirrels and dogs) or adapted animal species such as "jonahs" (modified sperm whales). Nemos are plants derived from sea anemones that bush outwards via tendril-like leaves that collect water droplets from the air, while "spiners" are adapted from spiders to spin structural elements for buildings. Doskies are a kind of flying or gliding domestic pet. Jonahs have been modified to

function in interstellar vacuum and can transport people or freight between planets or habitats inside a solar system.

For the purposes of the opera, we decided that nemos could emit some kind of particle that would provide buoyancy to someone attempting to float or glide in the air. In addition, we assumed they could sense and mimic vocal tones, and that they would emit their buoyant particles following vocal stimulation. By populating the ledge on which the stage is set with nemos, we could introduce virtual elements that could be controlled by humans. We explored several interaction modalities—as suggested in the previous sentences, vocal activation was one of these, and the one we eventually retained. But in an earlier effort, we also developed a belt that activated the nemos via a person's breathing. Both of these interaction modalities were attempts to re-introduce embodied interactions into the opera.

We modified the scenario to accommodate these interactions. In the original scenario, a group of bullies were intimidating the two principal characters, Oreph and Relliana, in the opening scene, and these would eventually be pushed off the city. However, they both wore "windplanes," a kind of glider that allowed them to escape falling to the distant ground. In the early scenario modifications, we determined that if the nemos were activated enough, then the two characters would not fall at all, they would fly up. Later, however, as we simplified the story line to fit it into a sufficiently short time from (less than 30 min), we used the same idea in the final scene of the opera. Now, however, it was the transformed Relliana, with her nascent wings, that was to be supported by the audience interaction. If they successfully supported her, she flew away upwards. If they failed, she eventually fell downwards.

In addition, we posited the existence of a plant species we called "phramae," which could organize into physical forms that mimic, in a ghost-like manner, the movement of real creatures, human or otherwise. These phramae were used in the bully sequence to make a number of additional dancers than the small number of actual bullies, and they were also used in the on-line co-creation environment to provide added control to users to "create" their own dancing avatars (see Section 2.10 below).

#### **3.4 Music**

Developing the music for this opera posed several challenges. For reasons that are too complex to go into here, Lacasse was not as available to do as much composing as had been originally planned. The libretto, which as outlined earlier went through many different versions, incorporated relatively early on a "bully song," to be sung by a main "bully" and his/her troops as they underwent a "bully dance." Lacasse developed quickly a successful musical rendition of the bully song, drawing on both pop culture musical ideas as well as more traditional opera constructions. The result was compelling in interesting ways. The bullies in the scenario are complex individuals in the sense that they are not simply "bad guys." They promote a kind of conformism, but they are in some sense supported by the ambient culture, and their ideas are not unsympathetic. Lacasse captured these ambiguities in his musical rendition, giving them a positive energy offset by the negative sentiments to some extent expressed in the lyrics.

Following this initial effort, however, Lacasse was no longer available to develop the remainder of the songs or the musical introduction. Kiss, who has no claims to be a composer, drew on a process of combining public domain royalty-free samplings, some recordings she had from an earlier collaboration with violincellist Haruko Kido, and a knack for bringing disparate elements from the creative commons together into a musical rendition which while not as "characteristically

**97**

**Figure 12.**

*Designing a Participatory and Interactive Opera DOI: http://dx.doi.org/10.5772/intechopen.82811*

in the production.

**3.5 Choreography**

**3.6 Animation**

individualistic" as Lacasse's work had been, resulted in a musical soundtrack that supported the opera in appropriate and interesting ways. She and Edwards worked in tandem on this, Edwards for overall direction and Kiss for the creative musicality of the ensemble. In addition, we consulted with Marie Louise Bourbeau, another collaborator with extensive experience in lyric singing and musical production, who helped guide us through some of the difficult choices we had to make at that point

Once Serge Lacasse had produced the music for the bully song, we solicited our choreographic collaborator, Coralee McLaren, to develop a musically-responsive dance work. At that time, we were still orienting towards a physical staging, so we were looking for movement that could be performed by a small group of dancers on stage. However, we also imagined using virtual avatars in addition to actual dancers. As such, we organized a movement session with Motion Capture equipment available to us through Edwards' engagement with the Centre interdisciplinaire de recherche en réadapation et intégration sociale (CIRRIS), a research center in rehabilitation. Dr. McLaren flew from Toronto to Quebec City to carry out this work. In preparation, she created a number of movement phrases that complimented Lacasse's rhythms and melodies. Drawing on these phrases, McLaren responded physically to spontaneous feedback called out from the team, and modified/refined her movement in response to fast changing, verbal cues. This resulted in an unparalleled collaborative movement experience and invaluable methodological insight, with each "audience" member providing input that reflected his or her own sensorial expertise or movement preferences (e.g., as singer, stage designer, writer, composer, etc.). The final outcome was a dance that enhanced Lacasse's dynamically rich score, and a subsequent Motion Capture sequence of movement was digitally applied to the avatars during the virtual staging (**Figure 12**). Furthermore, as we shall discuss below, the dance sequences

were included within the co-creation environment in interesting ways.

*Matching the motion capture recorded movements to the avatar movements.*

Once we had opted for a purely virtual staging, the major issues became the process of developing and animating the avatars that would represent the characters in the story. There were a number of issues that affected this process. As mentioned earlier we were experimenting with the idea of applying a user's face to the avatars,

individualistic" as Lacasse's work had been, resulted in a musical soundtrack that supported the opera in appropriate and interesting ways. She and Edwards worked in tandem on this, Edwards for overall direction and Kiss for the creative musicality of the ensemble. In addition, we consulted with Marie Louise Bourbeau, another collaborator with extensive experience in lyric singing and musical production, who helped guide us through some of the difficult choices we had to make at that point in the production.
