**3. GenVirtual musical system**

We developed an augmented reality musical system (GenVirtual) that allows creation, improvisation and music reproduction such as composition and tunes playing. Also the system enables the user to hear sounds and music, and musical memory games.

GenVirtual adds virtual musical instruments to the real world. Colored three-dimensional cubes represent the instruments. Cubes are able to simulate the sounds of wind, string and percussion instruments. Interaction occurs through GenVirtual cards with musical symbols. Cards replace the keyboard for music composition and are low cost since they can be printed on plain paper in various colors and sizes (Figure 4).

Cards to interaction of GenVirtual:


GenVirtual processes the images captured by a webcam to identify the cards on the table. The webcam is connected to a computer and strategically positioned on top of the table (Figure 4). Symbols on the cards are detected and associated with a sound. To execute a sound, the user just hides a card with his/her hand, without necessarily touching them with

Contributions of an Augmented Reality Musical System

Fig. 6. GenVirtual Musical Instruments

Fig. 7. 3D Musical Instruments of GenVirtual

of Winnie the Pooh Disney Enterprises Company.

cubes have the same colors of the colored balls from the score.

learning occurs.

for the Stimulation of Motor Skills in Music Therapy Sessions 281

with symbols corresponding to the percussion instruments, like drums, tambourines,

GenVirtual allows change images of musical instruments (virtual black cubes) for 3D

The main characteristic of this virtual environment is the interaction flexibility. Unlike the keyboard, consisting of fixed keys and sizes, in GenVirtual, the therapist can print cards in different sizes and position them on the table in various ways, according to the motor limitations of each individual. Cards can be arranged on the table, or on the ground, depending on whether the interaction will occur using hands or feet. According to Nascimento (2006), it is important to have a reference of the motor movement so that motor

GenVirtual also has a feature to assist in recreational actives in music. Composite scores were created by melodies that can be read using colored balls. Colors of musical notes of score correspond to the colors of same musical notes GenVirtual. Figure 8 shows one of musical scores available GenVirtual from the book "Song at the piano" featuring the works

Figure 9 shows musical score being projected on the computer screen. Note tha tthe virtual

model of the instrument. Figure 7 shows an image of the violininstrument.

triangles, and symbols that represent sounds of electronic instruments (Figure 6).

his/her fingers (Figure 5). By detecting the occlusion of a card, the system runs the corresponding sound the card represents.

Fig. 4. Musical Cards of GenVirtual

Fig. 5. Interaction forms of GenVirtual

The piano sound is associated to the default sound cards, but there is possibility to change the musical instrument sounds played by the cards. Simply add a new card on the table containing the symbol of the desired musical instrument (Figure 5). Thus, the same virtual elements emit different sounds, tones and may take a wind instrument (flute, trombone, trumpet, etc.). or string instruments (violin, guitar, etc). Another possibility is to use cards

his/her fingers (Figure 5). By detecting the occlusion of a card, the system runs the

The piano sound is associated to the default sound cards, but there is possibility to change the musical instrument sounds played by the cards. Simply add a new card on the table containing the symbol of the desired musical instrument (Figure 5). Thus, the same virtual elements emit different sounds, tones and may take a wind instrument (flute, trombone, trumpet, etc.). or string instruments (violin, guitar, etc). Another possibility is to use cards

corresponding sound the card represents.

Fig. 4. Musical Cards of GenVirtual

Fig. 5. Interaction forms of GenVirtual

with symbols corresponding to the percussion instruments, like drums, tambourines, triangles, and symbols that represent sounds of electronic instruments (Figure 6).

Fig. 6. GenVirtual Musical Instruments

GenVirtual allows change images of musical instruments (virtual black cubes) for 3D model of the instrument. Figure 7 shows an image of the violininstrument.

Fig. 7. 3D Musical Instruments of GenVirtual

The main characteristic of this virtual environment is the interaction flexibility. Unlike the keyboard, consisting of fixed keys and sizes, in GenVirtual, the therapist can print cards in different sizes and position them on the table in various ways, according to the motor limitations of each individual. Cards can be arranged on the table, or on the ground, depending on whether the interaction will occur using hands or feet. According to Nascimento (2006), it is important to have a reference of the motor movement so that motor learning occurs.

GenVirtual also has a feature to assist in recreational actives in music. Composite scores were created by melodies that can be read using colored balls. Colors of musical notes of score correspond to the colors of same musical notes GenVirtual. Figure 8 shows one of musical scores available GenVirtual from the book "Song at the piano" featuring the works of Winnie the Pooh Disney Enterprises Company.

Figure 9 shows musical score being projected on the computer screen. Note tha tthe virtual cubes have the same colors of the colored balls from the score.

Contributions of an Augmented Reality Musical System

needs to cover a card without using adaptations.

shows the music therapist interacting with GenVirtual.

Fig. 10. Evaluation of GenVirtual by music therapist

and motivational tool in the rehabilitation process. As one example:

for the Stimulation of Motor Skills in Music Therapy Sessions 283

individuals are not able to keep their fingers extended on the keyboard, or do not have the enough muscular strength to strike piano keys or strum a guitar (Louro et al, 2005). Typically, in these cases, the patient uses adaptations like tips on the hands to play the piano, tambourine fasteners for bimanual hand use, and the help of a music therapist to

With GenVirtual, the performer can use his/her hands closed into a fist to play a musical note, since the cards can be printed in larger proportions. There is no need to adjust the interaction with the virtual environment. To use GenVirtual to play an instrument, the user

Experiments with GenVirtual were performed in music therapy intervention. Five markers formed by symbols and colors of respective musical notes (C, D, E, F and G). Figure 10

A video monitor was used to visualize the animation of virtual cubes. The webcam was placed on top of the monitor to capture images of cards on the table. As can be seen in Figure 8, the music therapist obstructed cards with her hands without using adapters.

After testing, the music therapist noted that besides the possibility of using GenVirtual in music therapy, it can bring benefits to individuals with disabilities, since it is a facilitator

• Arm extension exercises: the goal is to reach a virtual object. With cards' flexibility, specific challenges can be created for each patient according to their motor limitations.

carry out interventions of music expression, the "music making" (Nascimento, 2006).

**5. Evaluation of GenVirtual in music theraphy rehabilitation** 

Fig. 8. Musical score of GenVirtual

Fig. 9. Musical score of GenVirtual
