**3. Does color information improve object recognition?**

Traditionally, theories of object recognition suggest that objects are recognized based only on shape information, largely ignoring the potential role of color information (Biederman, 1987; Marr & Nishihara, 1978). For instance, in the recognition-by-components (RBC) model, proposed by Biederman (1987), objects are described as spatial arrangements of a restricted set of roughly 30 basic component shapes, such has wedges and cylinders, called geons. This idea suggests an analogy with words, which are constructed from a restricted set of phonemes. Biederman (1987) suggested that the first stage of object recognition involves the segmentation of the contour in regions of sharp concavity. This segmentation divides the contour into a number of parts that then are matched against the set of geons. Biederman (1987) used view-invariant representations. According with the RBC model, geons are defined by properties that are invariant over different views. Object representations are simply assemblies of geons constructed by inferring the qualitative spatial relations between them. Because geons and the relationships between them are viewpoint-invariant, the recognition process is likewise viewpoint-invariant. One strong point of this theory is the fact that geons are not only view-invariant, but also to other surface properties, such as size, color or texture.

The Contribution of Color to Object Recognition 77

parts, musical instruments, tools). Objects belonging to structurally similar categories activate a larger set of structural representations, leading to a higher competition within the visual system, and thus color can help resolve this competition (Price & Humphreys, 1989). Other studies have proposed that color can provide useful information when objects are high color diagnostic objects, that is, when objects are strongly associated with a color (Nagai & Yokosawa, 2003; Tanaka & Presnell, 1999). For example, a color diagnostic object, such as a strawberry, is strongly associated with the color red. A comb, however, which is a

In a recent meta-analysis we systematically review the scientific literature on the effect of color information on object recognition (Bramão, Reis, Petersson, & Faísca, 2011). Thirty-five independent experiments, comprising 1535 participants, were included in this metaanalysis. Overall, we found a moderate significant effect of color on object recognition (*d* = 0.28, *p* < 0.001), establishing in that way that color information plays a role in object recognition and should be considered in the visual object recognition models (**Figure 4**).

Fig. 4. Mean effect size (*d*) and 95% confidence intervals. The moderator variables tested in specific meta-analytic comparison are labeled on the left side. Labels to the right side of the figure indicate the number of independent effect sizes (experiments) which contributed to each meta-analysis (*NE*), and the number of subjects these effect sizes were based upon (*Ns*).

Additionally, we tested the specific moderator effect of a series of potential moderator variables on the role of the color information during object recognition (e.g., *stimuli type*, *object recognition task, etc…*). Here, we just present the results concerning the *object's semantic category* and *color diagnosticity* (for the complete analysis, see Bramão, Reis, Petersson, & Faísca, 2011). The impact of color in the recognition of objects from different semantic categories was first addressed by Price and Humphreys (1989). The authors found that object naming was facilitated by color when objects were from natural categories. Because objects from natural categories tend to be more structurally similar than artifacts, the competition within the object recognition system is greater for natural objects, and color information appears to be an important cue in resolving this competition. Wurm and colleagues (Wurm, Legge, Isenberg, & Luebker, 1993) showed that prototypical images exhibit a smaller color advantage compared to non-prototypical images. These observations led to the idea that color plays an important role in object recognition when shape is not

Adapted from Bramão, Reis, Petersson and Faísca (2011).

non-color diagnostic object, is not strongly associated with any particular color.

However, more recently, a large body of behavioral, neuroimaging and neurophysiological studies suggest that color might contribute to object recognition. Tanaka and colleagues (Tanaka, Weiskopf, & Williams, 2001) proposed the "Shape + Surface" model of object recognition that takes into consideration the recent evidence for the role of color information in object recognition (**Figure 3**). The model recognizes that object recognition is primarily a shape-driven system (e.g., blue strawberries are still recognized as strawberries); however, color and possibly other surface properties, such as texture, are perceptual inputs for the object representation system. The Shape + Surface model draws a distinction between surface color at the input level and stored color knowledge and considers object recognition to be jointly determined by the bottom-up influence of surface color and the top-down influence of color knowledge. According to this model, visual color knowledge can be triggered either by the perceptual object during object recognition or by its lexical label during mental imagery. Finally, the model maintains a separation between linguistic and visual representations of object color. For example, it is possible to know that strawberries are red without having to consult a visual representation.

By examining whether there is an advantage to recognizing the typical colored version of an object (e.g., a red strawberry) over its black and white or atypical color version (e.g., a purple strawberry), it is possible to verify whether color information contributes to object recognition. However, this relatively straightforward test has yielded mixed results. Some studies have shown that recognition times are essentially unaffected by color information (Biederman & Ju, 1988; Davidoff & Ostergaard, 1988; Ostergaard & Davidoff, 1985). However, other studies have found that objects presented in their typical color version are recognized faster than when individuals are presented with their black and white or atypical color versions (e.g., Humphreys, Goodale, Jakobson, & Servos, 1994; Price & Humphreys, 1989; Therriault, Yaxley, & Zwaan, 2009; Wurm, Legge, Isenberg, & Luebker, 1993).

Fig. 3. The Shape + Surface model of object recognition. Adapted from Tanaka, Weiskopf and Williams (2001).

Different explanations have been proposed for these apparently contradictory results. For instance, color information may facilitate the recognition of objects within structurally similar categories (e.g., animals, fruits) but not structurally dissimilar categories (e.g., body 76 Advances in Object Recognition Systems

However, more recently, a large body of behavioral, neuroimaging and neurophysiological studies suggest that color might contribute to object recognition. Tanaka and colleagues (Tanaka, Weiskopf, & Williams, 2001) proposed the "Shape + Surface" model of object recognition that takes into consideration the recent evidence for the role of color information in object recognition (**Figure 3**). The model recognizes that object recognition is primarily a shape-driven system (e.g., blue strawberries are still recognized as strawberries); however, color and possibly other surface properties, such as texture, are perceptual inputs for the object representation system. The Shape + Surface model draws a distinction between surface color at the input level and stored color knowledge and considers object recognition to be jointly determined by the bottom-up influence of surface color and the top-down influence of color knowledge. According to this model, visual color knowledge can be triggered either by the perceptual object during object recognition or by its lexical label during mental imagery. Finally, the model maintains a separation between linguistic and visual representations of object color. For example, it is possible to know that strawberries

By examining whether there is an advantage to recognizing the typical colored version of an object (e.g., a red strawberry) over its black and white or atypical color version (e.g., a purple strawberry), it is possible to verify whether color information contributes to object recognition. However, this relatively straightforward test has yielded mixed results. Some studies have shown that recognition times are essentially unaffected by color information (Biederman & Ju, 1988; Davidoff & Ostergaard, 1988; Ostergaard & Davidoff, 1985). However, other studies have found that objects presented in their typical color version are recognized faster than when individuals are presented with their black and white or atypical color versions (e.g., Humphreys, Goodale, Jakobson, & Servos, 1994; Price & Humphreys, 1989; Therriault, Yaxley, & Zwaan, 2009; Wurm, Legge, Isenberg, & Luebker,

Fig. 3. The Shape + Surface model of object recognition. Adapted from Tanaka, Weiskopf

Different explanations have been proposed for these apparently contradictory results. For instance, color information may facilitate the recognition of objects within structurally similar categories (e.g., animals, fruits) but not structurally dissimilar categories (e.g., body

are red without having to consult a visual representation.

1993).

and Williams (2001).

parts, musical instruments, tools). Objects belonging to structurally similar categories activate a larger set of structural representations, leading to a higher competition within the visual system, and thus color can help resolve this competition (Price & Humphreys, 1989). Other studies have proposed that color can provide useful information when objects are high color diagnostic objects, that is, when objects are strongly associated with a color (Nagai & Yokosawa, 2003; Tanaka & Presnell, 1999). For example, a color diagnostic object, such as a strawberry, is strongly associated with the color red. A comb, however, which is a non-color diagnostic object, is not strongly associated with any particular color.

In a recent meta-analysis we systematically review the scientific literature on the effect of color information on object recognition (Bramão, Reis, Petersson, & Faísca, 2011). Thirty-five independent experiments, comprising 1535 participants, were included in this metaanalysis. Overall, we found a moderate significant effect of color on object recognition (*d* = 0.28, *p* < 0.001), establishing in that way that color information plays a role in object recognition and should be considered in the visual object recognition models (**Figure 4**).

Fig. 4. Mean effect size (*d*) and 95% confidence intervals. The moderator variables tested in specific meta-analytic comparison are labeled on the left side. Labels to the right side of the figure indicate the number of independent effect sizes (experiments) which contributed to each meta-analysis (*NE*), and the number of subjects these effect sizes were based upon (*Ns*). Adapted from Bramão, Reis, Petersson and Faísca (2011).

Additionally, we tested the specific moderator effect of a series of potential moderator variables on the role of the color information during object recognition (e.g., *stimuli type*, *object recognition task, etc…*). Here, we just present the results concerning the *object's semantic category* and *color diagnosticity* (for the complete analysis, see Bramão, Reis, Petersson, & Faísca, 2011). The impact of color in the recognition of objects from different semantic categories was first addressed by Price and Humphreys (1989). The authors found that object naming was facilitated by color when objects were from natural categories. Because objects from natural categories tend to be more structurally similar than artifacts, the competition within the object recognition system is greater for natural objects, and color information appears to be an important cue in resolving this competition. Wurm and colleagues (Wurm, Legge, Isenberg, & Luebker, 1993) showed that prototypical images exhibit a smaller color advantage compared to non-prototypical images. These observations led to the idea that color plays an important role in object recognition when shape is not

The Contribution of Color to Object Recognition 79

processing. To recognize an object, different processing stages must be resolved (Humphreys, Price, & Riddoch, 1999). First, the perceptual input must be encoded and matched against a template form stored in the long-term memory. Next, the semantic object representations are accessed, and, finally, the object name is activated. Color information might be useful for recognition of both color and non-color diagnostic objects in the early stages of the visual processing. Specifically, this information could be used to match the perceptual input with a known shape representation or, at an even earlier visual processing stage, segregate and organize the visual input. However, in the later stages of the recognition process, color information might play different roles depending upon the color diagnosticity status of the specific objects. Although color information might be important for semantic representation of a color diagnostic object, color information is probably not as important for semantic representation of a non-color diagnostic object. When we think about the properties of a strawberry, the property red is one of the first that comes to mind; however, if we think about the features of a comb, its color is not one of the first properties one might think of. Thus, we proposed that color information might participate in the recognition of color and non-color diagnostic objects at different levels of visual processing. More specifically, we hypothesize that color information participates in the recognition of both types of objects in the early visual perceptual stages, helping both segmentation and organization of the perceptual input. Studies have indicated that color information is an important cue in the early visual processing stages (Gegenfurtner & Rieger, 2000; Wurm, Legge, Isenberg, & Luebker, 1993); however, these studies did not control for or manipulate the color diagnosticity level of the presented objects. Color information is expected to play an additional role during the recognition of color diagnostic objects at the semantic levels of visual processing. Color is an intrinsic property of these objects. For example, Naor-Raz and Tarr (2003), using a variation of the Stroop paradigm, asked participants to name the displayed color of objects and words. They found that color is an intrinsic property of color diagnostic objects at multiple levels. Thus, the presence of color information in an image of a color diagnostic object might be important for the activation of semantic object

**4. The influence of color information on the recognition of color diagnostic** 

the recognition of color diagnostic objects at higher levels of the visual processing.

In this section we present the results from two studies that aimed to clarify the conflicting results found in the literature and to test the hypothesis that the color effects in object recognition depend on the color diagnosticity status of the specific objects. More specifically, we hypothesize that color information influences the recognition of both color and non-color diagnostic objects at the low-level of vision (e.g., improving the segmentation and the organization of perceptual input). However, color is expected to play an additional role in

In a first study, participants performed three object recognition tasks with different cognitive demands at the perceptual, semantic and phonological levels: an object verification task, a category verification task, and a name verification task. Humphreys and colleagues argued that performance of these tasks poses different challenges for the cognitive system (Humphreys, Price, & Riddoch, 1999; Humphreys & Riddoch, 2006; Humphreys, Riddoch, & Quinlan, 1988; Riddoch & Humphreys, 1987). In the name

representation and recognition of the object.

**and non-color diagnostic objects** 

diagnostic or typical. Moreover, the observed color advantage for natural objects might be related to the fact that they are typically strongly associated with a specific color and therefore, their color tends to be more diagnostic compared to artifacts. This interaction between category and color diagnosticity was addressed by Nagai and Yokosawa (2003), who reported a color advantage for high color diagnostic objects regardless of their category. Corroborating this idea, other studies have reported a similar color advantage for natural objects and artifacts (Bramão, Faísca, Forkstam, Reis, & Petersson, 2010; Rossion & Pourtois, 2004; Uttl, Graf, & Santacruz, 2006). Our meta-analysis also supports the idea that color is important for the recognition of objects from both categories: we observed that color facilitates the ability to recognize both natural (*d* = 0.45, *p* < 0.001) and artifact objects (*d* = 0.36, *p* < 0.001) (**Figure 4**; Bramão, Reis, Petersson, & Faísca, 2011).

Color diagnosticity, however, showed a great moderator effect on the influence of color on object recognition: studies using color diagnostic objects showed a significant color effect (*d* = 0.43, *p* < 0.001), whereas a marginal color effect was found in studies that used non-color diagnostic objects (*d* = 0.18, *p* = 0.06) (**Figure 4**; Bramão, Reis, Petersson, & Faísca, 2011). Color diagnosticity is probably the most investigated property in studies exploring the role of color information in object recognition. According to the color diagnosticity hypothesis, color diagnostic objects are the most likely candidates to show an advantage due to color information in object recognition tasks (Nagai & Yokosawa, 2003; Tanaka & Presnell, 1999). For example, Tanaka and Presnell (1999) showed that the presence of color information has a significant impact on the recognition of high color diagnostic objects and no effect on the recognition of objects with low color diagnosticity. In a control condition, when high and low color diagnostic objects were matched for structural complexity, reliable color effects were still found, suggesting that color made a unique contribution to recognition in a manner that is independent of shape. Similar results were found in the recognition of everyday scenes (Oliva & Schyns, 2000). Scenes that are rich in color diagnostic content (e.g., coast, forest) are best recognized in their typical color versions when compared to black and white or atypical color versions. On the other hand, non-color diagnostic scenes (e.g., city, shopping area) showed no difference in recognition performance across the typical, blackand-white and atypical color versions (Oliva & Schyns, 2000). Thus, the concept of color diagnosticity generalizes to the recognition of both objects and scenes.

However, recent studies have failed to replicate this finding and have documented that color information, independent of the color diagnosticity status of the object, improves its recognition (Rossion & Pourtois, 2004; Uttl, Graf, & Santacruz, 2006). For example, Rossion and Pourtois (2004) colored the 260 line-drawings from the Snodgrass and Vanderwart (1980) set with texture and shadow details. Norms for the color diagnosticity level of the objects were collected and correlated with the advantage provided by color alone in the naming responses. The authors did not report a significant correlation between these two measures (r = 0.05), showing that color information improves object recognition independently of its color diagnosticity level.

The interactions between color diagnosticity and the observed advantage due to color information are not well understood, and the reasons for the apparently contradictory results reported in the literature are not obvious. One possibility is that color information helps the recognition of color and non-color diagnostic objects at different levels of visual 78 Advances in Object Recognition Systems

diagnostic or typical. Moreover, the observed color advantage for natural objects might be related to the fact that they are typically strongly associated with a specific color and therefore, their color tends to be more diagnostic compared to artifacts. This interaction between category and color diagnosticity was addressed by Nagai and Yokosawa (2003), who reported a color advantage for high color diagnostic objects regardless of their category. Corroborating this idea, other studies have reported a similar color advantage for natural objects and artifacts (Bramão, Faísca, Forkstam, Reis, & Petersson, 2010; Rossion & Pourtois, 2004; Uttl, Graf, & Santacruz, 2006). Our meta-analysis also supports the idea that color is important for the recognition of objects from both categories: we observed that color facilitates the ability to recognize both natural (*d* = 0.45, *p* < 0.001) and artifact objects (*d* =

Color diagnosticity, however, showed a great moderator effect on the influence of color on object recognition: studies using color diagnostic objects showed a significant color effect (*d* = 0.43, *p* < 0.001), whereas a marginal color effect was found in studies that used non-color diagnostic objects (*d* = 0.18, *p* = 0.06) (**Figure 4**; Bramão, Reis, Petersson, & Faísca, 2011). Color diagnosticity is probably the most investigated property in studies exploring the role of color information in object recognition. According to the color diagnosticity hypothesis, color diagnostic objects are the most likely candidates to show an advantage due to color information in object recognition tasks (Nagai & Yokosawa, 2003; Tanaka & Presnell, 1999). For example, Tanaka and Presnell (1999) showed that the presence of color information has a significant impact on the recognition of high color diagnostic objects and no effect on the recognition of objects with low color diagnosticity. In a control condition, when high and low color diagnostic objects were matched for structural complexity, reliable color effects were still found, suggesting that color made a unique contribution to recognition in a manner that is independent of shape. Similar results were found in the recognition of everyday scenes (Oliva & Schyns, 2000). Scenes that are rich in color diagnostic content (e.g., coast, forest) are best recognized in their typical color versions when compared to black and white or atypical color versions. On the other hand, non-color diagnostic scenes (e.g., city, shopping area) showed no difference in recognition performance across the typical, blackand-white and atypical color versions (Oliva & Schyns, 2000). Thus, the concept of color

0.36, *p* < 0.001) (**Figure 4**; Bramão, Reis, Petersson, & Faísca, 2011).

diagnosticity generalizes to the recognition of both objects and scenes.

independently of its color diagnosticity level.

However, recent studies have failed to replicate this finding and have documented that color information, independent of the color diagnosticity status of the object, improves its recognition (Rossion & Pourtois, 2004; Uttl, Graf, & Santacruz, 2006). For example, Rossion and Pourtois (2004) colored the 260 line-drawings from the Snodgrass and Vanderwart (1980) set with texture and shadow details. Norms for the color diagnosticity level of the objects were collected and correlated with the advantage provided by color alone in the naming responses. The authors did not report a significant correlation between these two measures (r = 0.05), showing that color information improves object recognition

The interactions between color diagnosticity and the observed advantage due to color information are not well understood, and the reasons for the apparently contradictory results reported in the literature are not obvious. One possibility is that color information helps the recognition of color and non-color diagnostic objects at different levels of visual processing. To recognize an object, different processing stages must be resolved (Humphreys, Price, & Riddoch, 1999). First, the perceptual input must be encoded and matched against a template form stored in the long-term memory. Next, the semantic object representations are accessed, and, finally, the object name is activated. Color information might be useful for recognition of both color and non-color diagnostic objects in the early stages of the visual processing. Specifically, this information could be used to match the perceptual input with a known shape representation or, at an even earlier visual processing stage, segregate and organize the visual input. However, in the later stages of the recognition process, color information might play different roles depending upon the color diagnosticity status of the specific objects. Although color information might be important for semantic representation of a color diagnostic object, color information is probably not as important for semantic representation of a non-color diagnostic object. When we think about the properties of a strawberry, the property red is one of the first that comes to mind; however, if we think about the features of a comb, its color is not one of the first properties one might think of. Thus, we proposed that color information might participate in the recognition of color and non-color diagnostic objects at different levels of visual processing. More specifically, we hypothesize that color information participates in the recognition of both types of objects in the early visual perceptual stages, helping both segmentation and organization of the perceptual input. Studies have indicated that color information is an important cue in the early visual processing stages (Gegenfurtner & Rieger, 2000; Wurm, Legge, Isenberg, & Luebker, 1993); however, these studies did not control for or manipulate the color diagnosticity level of the presented objects. Color information is expected to play an additional role during the recognition of color diagnostic objects at the semantic levels of visual processing. Color is an intrinsic property of these objects. For example, Naor-Raz and Tarr (2003), using a variation of the Stroop paradigm, asked participants to name the displayed color of objects and words. They found that color is an intrinsic property of color diagnostic objects at multiple levels. Thus, the presence of color information in an image of a color diagnostic object might be important for the activation of semantic object representation and recognition of the object.
