**4.1 Volumetric information and pictorial depth cues in the size-weight illusion**

Since volumetric information of objects has long been thought to be critical, numerous studies have focused on the volume of the object as the essential parameter for size in the size-weight illusion (SWI) (Scripture, 1897; H.E. Ross, 1969; Anderson, 1970; Ellis & Lederman, 1993). However, there is no direct evidence as to whether or not volumetric information is encoded in the process of size-weight integration in perceived heaviness. Further, although the dimensions of objects have been examined on reach-to-grasp movements (Westwood et al., 2002; Kwok & Braddick, 2003), no such research has been done on lifting movements or heaviness perception. This study investigated the effects of cues regarding pictorial depth and volumetric size cues on the size-weight integration in perceived heaviness. Weight displays were created using the PHANToM haptic stylus, synchronized and superimposed onto corresponding 2D graphic objects displayed on a 2D monitor, in accordance with manipulation of the stylus (virtual objects) by subjects. It was hypothesized that the degree of the SWI would be weakened, even to the point of disappearing, when the dimensions of pictorial depth cues were reduced from being a 3D cube to a 2D square.

Computer Graphic and PHANToM Haptic Displays:

order across conditions and subjects.

heaviness.

Powerful Tools to Understand How Humans Perceive Heaviness 37

them as a two-dimensional plane. Three different sizes of squares (large: 10.4 x 10.4 cm, medium: 6.5 x 6.5 cm and small: 3.0 x 3.0 cm, respectively) were used. The ratio in size between the large and small squares was about 12 : 1 in planar area. For all graphic stimuli, the medium size was used for control purposes. All stimuli were experienced by all subjects. Subjects performed 4 trials in which the second object was larger in size than the first object (the Larger condition), 4 trials in which the second object was smaller in size than the first object (the Smaller condition), and 4 trials in which the size of the second object was identical to that of the first object (the Identical condition). Thus each subject performed 12 trials for each visual condition, and 36 trials in total. The trials were presented in random

Sixteen right-handed adults participated; none had any visual, muscular, or cutaneous problems. Further, none had previous experience with the experimental tasks or were familiar with the hypothesis being tested. Basically, subjects grasped and lifted virtual objects with programmed haptic and graphic properties, using the PHANToM stylus. They perceived weight of the virtual objects based on the reflected haptic 3D forces, and the graphic object motion viewed during the lift. Instructions were to grasp the stylus with the thumb and index finger of the dominant hand and to confirm that the stylus and graphic object motion were synchronously related to the lifting movement. At the outset, subjects were instructed to maintain the speed of the lift as constant as possible throughout the trials. To facilitate this, there were five to ten practice lifts to become accustomed to the Virtual Haptic and Virtual Vision environment, lifting each medium size stimulus. In addition, subjects were requested to view the object without the eyes being intentionally closed during the lifting process. Next for the test trials, each subject was instructed to grasp the stylus when the first of two graphic objects appeared on the monitor, to lift it vertically only once with a single smooth movement to a height of approximately 5 cm, and then to replace it after attempting to memorize their perception of its heaviness. Immediately after replacing the first graphic object, it disappeared. Following that, the second graphical object immediately appeared, to be maneuvered in exactly the same manner as the first. After completing the two lifts, each subject was requested to state whether the second object was perceived to be Heavier, Lighter, or Similar in comparison to the first object. Because cues related to visual size were very weak on perceived heaviness (Ellis & Lederman, 1993) and any cognitive bias is liable to affect judgment of heaviness in the final decision phase (Mon-Williams and Murray, 2000) or in the initial lifting phase in the form of expectation (H.E. Ross, 1969), the subject was requested to report perception of heaviness exactly and without hesitation, once the two lifts were completed in a given trial. These instructions, requests, and requirements were deemed essential to minimize such cognitive biases on perceived

Similar to previous studies (Kawai, 2002a, 2002b), the percentage of subject responses were calculated as a function of the visual size of the second object and response categories (Heavier, Similar, and Lighter) for each individual subject. For example, if a subject's response was Heavier in two trials, Lighter in one trial, and Similar in one trial from among four trials for a particular visual condition, the percentage of responses for each category in this condition is 50 % for Heavier, 25 % for Lighter, and 25 % for Similar. The individual percentages were then averaged across subjects and visual conditions. To assess the effect of pictorial cues (3 levels), size (2 levels), and directions of size variation (2 levels) on perceived heaviness, a three-way within-subject ANOVA was performed on the individual means of

Fig. 6. A. Virtual Vision + Virtual Haptics, B. Visual stimuli with different sizes and pictorial depth cues.

## **4.2 Augmented environment created by computer graphics and computer haptics**

Figure 6A is a schematic of the programmed augmented environment, (see Kuang, et al., 2004 for details). Software on a Silicon Graphics Inc. computer was developed to display on the monitor (a), a graphic hook (b), suspending a graphic object (c) by a graphic wire. A PHANToM haptic device (e) by SensAble Technologies Inc., integrated into the Virtual Hand Laboratory system, was programmed with forces (at 1000 Hz) to reflect the subject's manipulation of the PHANToM stylus (d), as the object was lifted with the haptic stylus/graphic hook. Either 0.5 or 0.75 N was programmed for virtual object weight. This weight was constant for the visual conditions in Fig. 6B. Thus, subjects grasped and lifted virtual objects with programmed haptic and graphic properties. They perceived the heaviness of the virtual objects based on the lift of the virtual object with the stylus/graphic hook. 3D position and acceleration of lifting were processed by the system at 200 Hz. Temporal lag for the system to sample human-generated forces acting on the system, calculate reflected force directions for the haptic device, and display the motion of the graphic object was 2-3 frames at 60 Hz, i.e., 50 ms max. Subjects did not notice any delay between their initial movement with the PHANToM stylus and initiation of movement of the graphic object.

Figure 6B shows three graphic stimuli with different sizes and pictorial depth cues (solid, wire-frame and square). Solid cubes (Fig. 6B-a) had 3D pictorial cues, with different colors on each of the three visible sides so that participants readily perceived them as solid or three-dimensional. Three different sizes of solid graphic cubes were used (large: 6.9 x 6.9 x 6.9 cm, medium: 5.0 x 5.0 x 5.0 cm and small: 3.0 x 3.0 x 3.0 cm, respectively). The large and small cubes were used to assess the frequency of the SWI, while the medium cubes were used for controls. The size ratios between the large and small solid cubes were approximately 12 : 1 in volume and 5.4 : 1 in planar area. Wire-frame graphic cubes (Fig. 6Bb) provided weaker 3D cues compared to solid cubes since colors were not added to any sides. As a result, the perception of depth or dimension was unstable like the Necker's cube; sometimes the graphic object was perceived with a cube shape and sometimes a plane hexagon. The ratios between the large and small frame cubes were identical to those for solid cubes. Graphic squares (Fig. 6B-c) had no 3D pictorial cues so that subjects perceived

Fig. 6. A. Virtual Vision + Virtual Haptics, B. Visual stimuli with different sizes and pictorial

Figure 6A is a schematic of the programmed augmented environment, (see Kuang, et al., 2004 for details). Software on a Silicon Graphics Inc. computer was developed to display on the monitor (a), a graphic hook (b), suspending a graphic object (c) by a graphic wire. A PHANToM haptic device (e) by SensAble Technologies Inc., integrated into the Virtual Hand Laboratory system, was programmed with forces (at 1000 Hz) to reflect the subject's manipulation of the PHANToM stylus (d), as the object was lifted with the haptic stylus/graphic hook. Either 0.5 or 0.75 N was programmed for virtual object weight. This weight was constant for the visual conditions in Fig. 6B. Thus, subjects grasped and lifted virtual objects with programmed haptic and graphic properties. They perceived the heaviness of the virtual objects based on the lift of the virtual object with the stylus/graphic hook. 3D position and acceleration of lifting were processed by the system at 200 Hz. Temporal lag for the system to sample human-generated forces acting on the system, calculate reflected force directions for the haptic device, and display the motion of the graphic object was 2-3 frames at 60 Hz, i.e., 50 ms max. Subjects did not notice any delay between their initial movement with the PHANToM stylus and initiation of movement of

Figure 6B shows three graphic stimuli with different sizes and pictorial depth cues (solid, wire-frame and square). Solid cubes (Fig. 6B-a) had 3D pictorial cues, with different colors on each of the three visible sides so that participants readily perceived them as solid or three-dimensional. Three different sizes of solid graphic cubes were used (large: 6.9 x 6.9 x 6.9 cm, medium: 5.0 x 5.0 x 5.0 cm and small: 3.0 x 3.0 x 3.0 cm, respectively). The large and small cubes were used to assess the frequency of the SWI, while the medium cubes were used for controls. The size ratios between the large and small solid cubes were approximately 12 : 1 in volume and 5.4 : 1 in planar area. Wire-frame graphic cubes (Fig. 6Bb) provided weaker 3D cues compared to solid cubes since colors were not added to any sides. As a result, the perception of depth or dimension was unstable like the Necker's cube; sometimes the graphic object was perceived with a cube shape and sometimes a plane hexagon. The ratios between the large and small frame cubes were identical to those for solid cubes. Graphic squares (Fig. 6B-c) had no 3D pictorial cues so that subjects perceived

**4.2 Augmented environment created by computer graphics and computer haptics** 

depth cues.

the graphic object.

them as a two-dimensional plane. Three different sizes of squares (large: 10.4 x 10.4 cm, medium: 6.5 x 6.5 cm and small: 3.0 x 3.0 cm, respectively) were used. The ratio in size between the large and small squares was about 12 : 1 in planar area. For all graphic stimuli, the medium size was used for control purposes. All stimuli were experienced by all subjects. Subjects performed 4 trials in which the second object was larger in size than the first object (the Larger condition), 4 trials in which the second object was smaller in size than the first object (the Smaller condition), and 4 trials in which the size of the second object was identical to that of the first object (the Identical condition). Thus each subject performed 12 trials for each visual condition, and 36 trials in total. The trials were presented in random order across conditions and subjects.

Sixteen right-handed adults participated; none had any visual, muscular, or cutaneous problems. Further, none had previous experience with the experimental tasks or were familiar with the hypothesis being tested. Basically, subjects grasped and lifted virtual objects with programmed haptic and graphic properties, using the PHANToM stylus. They perceived weight of the virtual objects based on the reflected haptic 3D forces, and the graphic object motion viewed during the lift. Instructions were to grasp the stylus with the thumb and index finger of the dominant hand and to confirm that the stylus and graphic object motion were synchronously related to the lifting movement. At the outset, subjects were instructed to maintain the speed of the lift as constant as possible throughout the trials. To facilitate this, there were five to ten practice lifts to become accustomed to the Virtual Haptic and Virtual Vision environment, lifting each medium size stimulus. In addition, subjects were requested to view the object without the eyes being intentionally closed during the lifting process. Next for the test trials, each subject was instructed to grasp the stylus when the first of two graphic objects appeared on the monitor, to lift it vertically only once with a single smooth movement to a height of approximately 5 cm, and then to replace it after attempting to memorize their perception of its heaviness. Immediately after replacing the first graphic object, it disappeared. Following that, the second graphical object immediately appeared, to be maneuvered in exactly the same manner as the first. After completing the two lifts, each subject was requested to state whether the second object was perceived to be Heavier, Lighter, or Similar in comparison to the first object. Because cues related to visual size were very weak on perceived heaviness (Ellis & Lederman, 1993) and any cognitive bias is liable to affect judgment of heaviness in the final decision phase (Mon-Williams and Murray, 2000) or in the initial lifting phase in the form of expectation (H.E. Ross, 1969), the subject was requested to report perception of heaviness exactly and without hesitation, once the two lifts were completed in a given trial. These instructions, requests, and requirements were deemed essential to minimize such cognitive biases on perceived heaviness.

Similar to previous studies (Kawai, 2002a, 2002b), the percentage of subject responses were calculated as a function of the visual size of the second object and response categories (Heavier, Similar, and Lighter) for each individual subject. For example, if a subject's response was Heavier in two trials, Lighter in one trial, and Similar in one trial from among four trials for a particular visual condition, the percentage of responses for each category in this condition is 50 % for Heavier, 25 % for Lighter, and 25 % for Similar. The individual percentages were then averaged across subjects and visual conditions. To assess the effect of pictorial cues (3 levels), size (2 levels), and directions of size variation (2 levels) on perceived heaviness, a three-way within-subject ANOVA was performed on the individual means of

Computer Graphic and PHANToM Haptic Displays:

**4.4 Discussion** 

previously, according to Weber's Law (Kawai, 2002a).

VIRTUAL HAPTIC environment of this experiment.

**5. Chapter conclusion 5.1 What is heaviness?** 

Powerful Tools to Understand How Humans Perceive Heaviness 39

the reports of test subjects regarding the small object being Heavier was similar to that of the large object being Lighter. This phenomenon has been described in detail and discussed

All subjects experienced the SWI for all three visual stimuli in Fig. 6B without any significant differences, suggesting that, contrary to our hypothesis, the SWI occurs to the same degree with any pictorial depth cues! Volumetric information of an object has long been thought to be critical in bringing about the SWI (Jones, 1986). As a result, numerous studies have investigated the phenomenon of the SWI and discussed based on the volume of an object as the essential parameter for size (Scripture, 1897; H.E. Ross, 1969; Ellis & Lederman, 1993). Furthermore, in studies related to motor programming for the lifting of an object, information of volume has also been thought to be a critical factor in estimating weight of an object as part of the process for producing the required lift forces (H.E. Ross, 1969; Gordon et al., 1991). Thus, Westwood and his colleagues (2002) proposed the necessity of a volumetric object description in the process of specifying the grip force and the load force necessary for lifting and manipulation because such forces must be scaled to the anticipated mass of the object either by estimating its volume and density or by accessing stored knowledge related to the mass of the other object. It was, therefore, hypothesized at the beginning of this study that the SWI would decrease in frequency or even disappear when dimensional cues were reduced from 3D structure such as a cube to a 2D structure such as a square. The results, however, indicated that all the subjects equally experienced the SWI for all three visual stimuli with any cues of a 2D graphical object. This suggests that whatever the pictorial depth cues are, these factors are not critical for the occurrence of the SWI. The present results, therefore, conclude, contrary to commonly accepted theory, that pictorial depth cues responsible for 3D perception are not crucially necessary in the process of producing the SWI. In short, the perceptual system of heaviness may not be responsive to the dimension provided by pictorial depth cues, at least under the VIRTUAL VISION +

Various physical attributes are involved in the perception of heaviness of objects, leading to an understanding that psychological heaviness is not equal to physical weight. Heaviness perception is not simply a function for sensing object weight. Rather it functions as a form of recognition of the object. That is, all information obtained synchronously when humans grasp and lift an object may be gathered across modalities, integrated with or subtracted from each other, interpreted by object knowledge, and then formed as perceived heaviness. Among physical properties influencing heaviness, rotational inertia can be considered vital for future investigation, given evidence rotational inertia firmly affects heaviness (Turvey & Carello, 1995; Amazeen, 1999; Streit et al., 2007; Cf. Zhu & Bingham, 2011). However, such evidence has not explained neurologically how it is obtained from lifting, holding and manipulating an object and how it contributes to forming its heaviness. Gaining an understanding of how humans process the information about the rotational inertia of objects

frequency of subject responses. We summarize below the effects on the frequency of occurrence of the SWI.

#### **4.3 Results: Effects of depth cues and size on perceived heaviness, and the sizeweight illusion**

Figure 7 indicates the mean values and standard errors of the percentages for the occurrence of the SWI obtained from each pictorial depth cue, i.e., solid cube (Cube), wire-frame cube (Frame) and square (Square). The white bars (A) indicate the Smaller condition in which the second object was smaller in size than the first. The smaller object was reported as heavier than the larger one. The gray bars indicate the Identical Condition in which sizes were identical between the first and second object. Although the graphical sizes remained constant in the Identical condition, for some reason, subjects accidentally reported a difference in heaviness between the first and the second lifts (See Sec. 2-1). The black bars (B) are those obtained from the Larger condition in which the second object was larger in size than the first. For these conditions, the larger object was reported as lighter than the smaller one. Again, the grey bars are those obtained from the Identical condition in which they reported the second object to be perceived as lighter.

Fig. 7. Mean percentages (%) of the size-weight illusion (SWI) for each pictorial depth cue for each condition through which subjects compared the heaviness of the small size with that of the large (white bars), the large with the small (black bars), and the median with the equivalent median (gray bars).

There was a significant effect of change in size (F (1,15) =28.15, p< .001), indicating that the SWI occurred significantly for each pictorial depth cue when the size was changed compared to the conditions when the size was not changed. There were, however, no effects of pictorial depth cues (F(2, 30) =0.26) nor for the direction of size change (F(1, 15 = 0.46), suggesting pictorial depth cues do not affect the frequency or strength of the SWI. A significant interaction was observed only between pictorial depth cue and direction of size change (F(2, 30) = 31.53, p < .001). This is due to the fact that the SWI occurred more frequently in the Smaller condition (57.8% in Cube, 67.2% in Frame and 59.4 % in Square) than in the Larger condition (51.6% in Cube, 34.4 % in Frame and 45.3% in Square). There were no significant differences in probability in the Identical condition: the probability when the reports of test subjects regarding the small object being Heavier was similar to that of the large object being Lighter. This phenomenon has been described in detail and discussed previously, according to Weber's Law (Kawai, 2002a).
