**6. General discussion**

There exists ample evidence that vision and touch activate common neurological sites (Amedi *et al*., 2001; Ernst & Banks, 2002) and that objects experienced visually or haptically can, with fair success, be recognized in the alternate modality (Klatzky, Lederman, & Metzger, 1985; Pensky *et al*., 2008). However, almost nothing is known about the transfer of *categorical* information between these modalities. That is, can it be demonstrated that abstract categories, learned in one modality, maintain their categorical identity in an alternate modality? The answer, at least for the forms used here and considering only the visual and haptic modalities, is clearly yes.

We purposely selected fairly complex three dimensional objects that were comprised of continuous distortions from three prototypes that, informally at least, appeared to preclude simple naming of objects or even features. The major results of the three experiments that explored the learning, transfer, and retention of concepts acquired visually, haptically, or combined can be summarized: (a) Visual learning of categories, as expected, was more rapid than haptic learning, but haptic learning reached the same errorless criterion after only four study blocks; (b) When categories were learned in one modality, the classification of novel forms on a transfer test was virtually perfect, even when presented in the alternate modality; (c) The interposition of a week's delay had a statistically significant but minimal effect on classification accuracy.

Haptic Concepts 21

for each category. The last outcome seems least likely, since the learning of multiple categories should produce a slowing of category learning, an outcome not obtained. Regardless, additional research with categories composed of more distinctive features, e.g., texture differences, might permit separation of these competing explanations. Finally, the category prototype and midpoint objects were falsely recognized less often than other new objects. This occurred even though the midpoint objects were flanked by two similar training objects as were the new objects; the category prototypes similarly had two training objects that were similar as well. What seems likely is that exemplar similarity (e.g., Nosofsky, 1988) alone was not the sole determinate of recognition. Rather, categorical influences likely mitigated false recognition, since, for the midpoint objects, the two flanking training objects belonged to different prototypes. Why the category prototypes were not falsely recognized more often (or at least as often as the new objects) is less clear. However, the location of the prototypes, as an object at the vertex of two divergent paths, may have insulated the category prototype from false recognition because of extra-experimental knowledge, e.g., the subject might sense that the prototype is a generative pattern, not an old one. Regardless, there exists prior evidence that the category prototype may be treated as a novel ideal point rather than a familiar one based on object similarity alone (Homa *et al*.,

Future research into multi-modal concepts, including situations where less than full stimulus information is available, is critical to a comprehensive theory of concepts. Creative paradigms that involve modalities other than visual and haptic processing is obviously needed, as are the criteria needed to address what is perhaps the most fundamental question of all in this domain – what evidence would suggest that our concepts become

Amedi, A., Malach, R., Hendler, T., Peled, S., & Zohary, E. (2001). Visuo-haptic objectrelated activation in the ventral visual pathway. *Nature Neuroscience, 4,* 324-330. Bergmann-Tiest, W. M., & Kappers, A. M. L. (2006). Analysis of haptic perception of

Busemeyer, J. R., & Pleskac, T. (2009). Theoretical tools for understanding and aiding dynamic decision making. *Journal of Mathematical Psychology, 53,* 126-138. Castiello, U., Zucco, G. M., Parma, V., Ansuini, C., & Tirindelli, R. (2006). Cross-modal

Catherwood, D. (1993). The haptic processing of texture and shape by 7- to 9-month-old

Cooke, T., Jakel, F., Wallraven, C., & Bulthoff, H. H. (2007). Multimodal similarity and categorization of novel, three-dimensional objects. *Neuropsychologia, 45*, 484-495. Ernst, M. O. (2007). Learning to integrate arbitrary signals from vision and touch. *Journal of* 

Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a

infants. *British Journal of Developmental Psychology, 11*, 299-306.

and compressibility. *Acta Psychologica, 121*, 1-20. Biederman, I. (1972). Perceiving real-world scenes. *Science, 177,* 77-80.

statistically optimal fashion. *Nature, 415*, 429-433.

materials by multidimensional scaling and physical measurements of roughness

interactions between olfaction and vision when grasping. *Chemical Senses, 31*, 665-

1993; Homa *et al*., 2001).

**7. References** 

671.

*Vision, 7,* 1-14.

modality-free or modality-preserving?

The results for recognition were, however, less impressive: (a) Recognition accuracy was less accurate than classification, especially when learning occurred haptically and recognition occurred in the visual modality; (b) Transfer between the modalities was more accurate when the learning was visual rather than by touch; (c) Within-category, cross-modal conflict had no impact on learning and even appeared to enhance later recognition; and finally, (f) The psychological space for concepts acquired visually or haptically was virtually the same. We also found that presentation of the objects in a systematic, rather than random, order speeded learning and slightly improved overall transfer performance, and that the haptic space was somewhat better structured into the three categories than was the visual space.

Recognition following categorical learning was superior when the categories were formed visually and tested haptically rather than the reverse. This outcome could be explained most readily by assuming that two, distinct processes are involved in categorical recognition, an initial encoding of features relevant to the category, and a transfer of categorical information from one modality to another. A safe assumption is that the visual modality encodes more information than does the haptic modality. If the transfer from one modality to the other is not perfect, e.g., 50% of the information is transferred accurately and 50% is not, then the obtained ordering on the recognition test can be explained. That is, VV > HH = VH > HV. The multidimensional scaling of the category space, following either no learning or criterion learning, supports this interpretation, albeit indirectly. To see this, consider each object to be encoded with N-categorical features + K idiosyncratic features. Since classification transfer was accurate, with relatively few errors, we could assume that the two modalities encoded the categorical features to a similar degree. However, if the idiosyncratic features were more numerous following visual inspection, and if the idiosyncratic features are critical to later discrimination, then two outcomes would occur – recognition would be more accurate following visual training (more idiosyncratic features) and the similarity judgments, used to map the categorical spaces, would be more distinctive when objects were compared visually. Phillips et al. (2009) found that increasing object complexity influenced haptic judgments more than visual judgments, an outcome that would be consistent with the view suggested here. An alternative test would require that features more amenable to haptic than visual processing, such as texture and weight differences, be incorporated into a categorical paradigm. Under these circumstances, haptic recognition might improve overall and produce an MDS space that represented within-category objects as slightly less similar to each other.

Four other results are notable. First, systematic training had a small but consistently positive effect both in learning and later recognition, a result that replicates Zaki and Homa's (1999) study using two dimensional categorical stimuli. Second, the placement of the category prototypes in the multidimensionally-scaled space failed to preserve the prototype as an endpoint object of its category. Rather, the category prototype, especially following a learning phase, was found to gravitate more toward the center of its psychological category. Third, cross-modal conflict had a negligible effect in either learning or later transfer. In fact, this conflict seemed to enhance later recognition. Our impression is that most subjects failed to notice a conflict when the object explored visually and haptically were different, presumably because the objects were not namable, lacked dramatically different features, and belonged to the same category. It is less clear whether the subject integrated the slightly disparate sensations from the two different stimuli on each trial, formed a composite memory trace that included both visual and haptic features, or formed bi-modal concepts

The results for recognition were, however, less impressive: (a) Recognition accuracy was less accurate than classification, especially when learning occurred haptically and recognition occurred in the visual modality; (b) Transfer between the modalities was more accurate when the learning was visual rather than by touch; (c) Within-category, cross-modal conflict had no impact on learning and even appeared to enhance later recognition; and finally, (f) The psychological space for concepts acquired visually or haptically was virtually the same. We also found that presentation of the objects in a systematic, rather than random, order speeded learning and slightly improved overall transfer performance, and that the haptic space was somewhat better structured into the three categories than was the visual space. Recognition following categorical learning was superior when the categories were formed visually and tested haptically rather than the reverse. This outcome could be explained most readily by assuming that two, distinct processes are involved in categorical recognition, an initial encoding of features relevant to the category, and a transfer of categorical information from one modality to another. A safe assumption is that the visual modality encodes more information than does the haptic modality. If the transfer from one modality to the other is not perfect, e.g., 50% of the information is transferred accurately and 50% is not, then the obtained ordering on the recognition test can be explained. That is, VV > HH = VH > HV. The multidimensional scaling of the category space, following either no learning or criterion learning, supports this interpretation, albeit indirectly. To see this, consider each object to be encoded with N-categorical features + K idiosyncratic features. Since classification transfer was accurate, with relatively few errors, we could assume that the two modalities encoded the categorical features to a similar degree. However, if the idiosyncratic features were more numerous following visual inspection, and if the idiosyncratic features are critical to later discrimination, then two outcomes would occur – recognition would be more accurate following visual training (more idiosyncratic features) and the similarity judgments, used to map the categorical spaces, would be more distinctive when objects were compared visually. Phillips et al. (2009) found that increasing object complexity influenced haptic judgments more than visual judgments, an outcome that would be consistent with the view suggested here. An alternative test would require that features more amenable to haptic than visual processing, such as texture and weight differences, be incorporated into a categorical paradigm. Under these circumstances, haptic recognition might improve overall and produce an MDS space that represented within-category objects as slightly less similar

Four other results are notable. First, systematic training had a small but consistently positive effect both in learning and later recognition, a result that replicates Zaki and Homa's (1999) study using two dimensional categorical stimuli. Second, the placement of the category prototypes in the multidimensionally-scaled space failed to preserve the prototype as an endpoint object of its category. Rather, the category prototype, especially following a learning phase, was found to gravitate more toward the center of its psychological category. Third, cross-modal conflict had a negligible effect in either learning or later transfer. In fact, this conflict seemed to enhance later recognition. Our impression is that most subjects failed to notice a conflict when the object explored visually and haptically were different, presumably because the objects were not namable, lacked dramatically different features, and belonged to the same category. It is less clear whether the subject integrated the slightly disparate sensations from the two different stimuli on each trial, formed a composite memory trace that included both visual and haptic features, or formed bi-modal concepts

to each other.

for each category. The last outcome seems least likely, since the learning of multiple categories should produce a slowing of category learning, an outcome not obtained. Regardless, additional research with categories composed of more distinctive features, e.g., texture differences, might permit separation of these competing explanations. Finally, the category prototype and midpoint objects were falsely recognized less often than other new objects. This occurred even though the midpoint objects were flanked by two similar training objects as were the new objects; the category prototypes similarly had two training objects that were similar as well. What seems likely is that exemplar similarity (e.g., Nosofsky, 1988) alone was not the sole determinate of recognition. Rather, categorical influences likely mitigated false recognition, since, for the midpoint objects, the two flanking training objects belonged to different prototypes. Why the category prototypes were not falsely recognized more often (or at least as often as the new objects) is less clear. However, the location of the prototypes, as an object at the vertex of two divergent paths, may have insulated the category prototype from false recognition because of extra-experimental knowledge, e.g., the subject might sense that the prototype is a generative pattern, not an old one. Regardless, there exists prior evidence that the category prototype may be treated as a novel ideal point rather than a familiar one based on object similarity alone (Homa *et al*., 1993; Homa *et al*., 2001).

Future research into multi-modal concepts, including situations where less than full stimulus information is available, is critical to a comprehensive theory of concepts. Creative paradigms that involve modalities other than visual and haptic processing is obviously needed, as are the criteria needed to address what is perhaps the most fundamental question of all in this domain – what evidence would suggest that our concepts become modality-free or modality-preserving?
