**3. The semiotic approach and semiosis: no more binaries**

The semiotic perspective requires a re-evaluation of research techniques and applications. Such a re-evaluation brings into question the very object of study, as well as the instrumentation used to evaluate the object. Once it is recognized that the focal point of Peircean semiotic theory is not the sign, but *semiosis (the sign in action*, *the prerequisite process of the exchange of information [signification])*, the next step is to take a closer look at the interface between functioning signs as they are used [6, 10].

Semiotic theory that focuses on *signs in action* provides an epistemological framework where any linguistic speech act includes the *users* of the language, and *codes and messages are embodied in the users* themselves. This idea can be extended to neuroscience and can provide the basis for proposing that the sensory-motor neural image of a given linguistic phenomenon, be it grammatical or lexical, is multisensory and involves cross-modal effects—a notion that is central to all of the current neuroscience research on *embodied cognition* (including [15, 16])*.* Another outcome of a theory of dynamic signs requires treating *referential* meaning as a *type* of meaning—not the *only kind* of meaning and not even the most important kind in all instances. In other words, the outcome of this framework is that *linguistic meaning is never solely determined or determinable by reference alone*. 3 This outcome has a profound impact on the modeling of *multimodality* as an organizing principle of human cognition.

## **3.1 Defining icon/index/symbol: redefining non-arbitrariness**

Following Peirce (Peirce 8.368, [13] pp. 4–23;[21]), the *icon* is defined as a qualitative likeness between a sign and its object. The quality upon which the similarity is based belongs to the sign, whether or not its object exists. The *index* is a real connection between the sign and its object, and the relationship is given via the dynamic object of the sign. The *symbol* is a general rule between the sign and object, and the object is related to the sign via the *interpretant.* [Jakobson's rendering of these terms is reanalyzed back into the dyadic terms *signifier/signans* and *signified/signatum, similarity/*contiguity (i.e., paradigmatic/syntagmatic axes), and the notions of *factual* (effectual) and *habitual* [22, 23]] Each of these 3 sign-object relationships are embedded in each other such that none of these categories may

<sup>3</sup> For a full discussion of the Jakobsonian speech act model, which is applicable to explicating the generation of linguistic meanings, as well as language usage both in healthy subjects and in pathology, see ([17], pp. 66-71; [5], pp. 51-58; [18–20]).

**Figure 1.** *Peirce's sign-object triadic relation.*

ever occur alone, the icon is embedded in the index and symbol, and the index is embedded in the symbol (Peirce 2.306, 4.447 [21]). Note the following diagram with Jakobson's terminology: **Figure 1**

The principle of **iconicity**, as based on Peirce's definitions, is critical to the reimagining of the non-arbitrariness of the linguistic sign as a relative phenomenon. Bolinger ([1], pp. 18–23), in his work on morphemes, presents interesting examples from English of phonological iconicity at the syllabic and lexical levels, including "hard of hearing" (alliterative/iconic phonological properties, [h\_r/h\_r], and "short circuit," [rt/r\_t]), not "hard of seeing/smelling" and "long circuit." Bolinger also exposes important iconic relationships in prosody in later works and fine-tunes his approach to *relative non-arbitrariness* ([24], pp. 92–93).4

## **3.2 The importance of the interpretant: how to talk about meaning**

Peirce posits three basic types of *interpretants* in the sign complex—*immediate, dynamic*, and *final* (Peirce 8. 315, 8.372, [13]). The *interpretant* is the level of the triadic sign where the *meaning* of the sign is realized:

*"…the meaning of a sign is the sign it has to be translated into;" (4.132)*

*"creates in the mind…an equivalent sign, or perhaps a move developed sign;" (2.228)*

*"the proper significate outcome of the sign" (4.127)*

It is interesting that while linguistic applications of Peirce are rich in the use of the icon/index/symbol triad, this is not the case with interpretants.<sup>5</sup> However, the one fundamental point that has persisted is the **certainty of translation** and the recognition that **all translations change meanings.** Lotman ([28], p. 15) does not ever use Peirce's term but makes an important argument about the inevitability of *tensions* that arise in any speech/communication act. In so doing, Lotman articulates a new perspective—the most valuable information derived from speech acts is a result of the *translation of that which is seemingly untranslatable*.

**17**

Bowring et al. [44].

challenge.6

*Semiotic Principles in Cognitive Neuroscience DOI: http://dx.doi.org/10.5772/intechopen.89791*

**4. Cognitive neuroscience studies of languages**

*with neuronal activity" ([21], pp. 127-8).*

**generation of meaning(s) and activations**

*"How does fMRI create images of neuronal activity? The short answer is that it does not. Instead, fMRI creates images of physiological activity that are correlated* 

Imaging techniques that involve the hemodynamic response (PET, fMRI) only measure correlates of neuronal activity and processes and not actual neuronal firings. Electrophysiological techniques (including EEG, MEG) can measure electrical activity in ensembles of neurons and changes in magnetic fields related to neuronal activity. Invasive direct single-neuron mappings, as found in cortical stimulation mappings (CSM), are only conducted during surgery. George Ojemann, who developed the technique, conducted over 1100 surgeries that included CSM mappings to identify areas important to production and comprehension of language(s). This technique was developed in order to identify areas related to important functions of production (motor) and comprehension (sensory) in language(s) and has proven successful in preserving these functions in surgeries that require removal of tissue. Dr. Ojemann has published over 170 papers of CSM data with important conclu-

Understanding activations found in BOLD fMRI studies presents a significant

much broader range of activations across both hemispheres in areas not represented in the traditional Broca/Wernicke targets from the "classical model"; these results are in keeping with research findings in fMRI language studies and represent a consensus among the neuroscience community (cf. [4, 5, 45–51]). There are challenges

*"…impossible to distinguish inhibitory from excitatory cellular activity on the basis of changes in either blood flow or metabolism. Thus, on this view a local increase in inhibitory activity would be as likely to increase blood flow and the fMRI BOLD* 

Other challenges in using fMRI for cognitive studies include the timing delay in the hemodynamic response that lags behind neuronal activity. It is essential to remember that there can be no one-to-one relationship between neural activations

Interpreting the activations recorded from ROIs in fMRI language studies is strengthened when significance can be found with behavioral data (e.g., empirical proficiency testing data) using multivariate analysis of covariance (MANCOVA). One such example is a longitudinal study of second language acquisition that also includes Common European Framework proficiency testing data [5, 45]. The use of empirical **language proficiency** data is particularly important for bi- and

<sup>6</sup> Eklund et al. ([42], pp. 7900-7905) is one of a series of papers that discusses problems with fMRI analysis and parametric statistical methods, including "false positives" that resulted in a series of studies using the following fMRI analysis software: AFNI, FSL, and SPM. Eklund et al. note that results are more reliable for voxel-wise inference and invalid for cluster-based inference ([42], p. 7903). Additional sources that examine inter-method discrepancies in brain imaging include Katuwal et al. [43] and

When considering the results of fMRI studies of language(s), one finds a

**4.1 Understanding activations: defining the relationship between the** 

sions about neural mappings of language and languages [29–41].

in analyzing neural activations, and Raichle reminds us that it is:

acquired during imaging and the subject's knowledge or ability.

*signal as would a local increase in excitatory activity" (*[48]*, p. 12).*

<sup>4</sup> Bolinger's work [1, 24–26] provides robust explanations for deconstructing the nineteenth-century view of form/meaning relations in language as "arbitrary" (as posited by Saussure) by exploiting iconic principles manifested in definitions that are relativized, not absolutes.

<sup>5</sup> For examples of analyses using Peircean interpretants, see Savan [13], Andrews [12, 27]. For an example of the use of interpretants to explain principles of language acquisition, see Andrews [12].

*Cognitive and Intermedial Semiotics*

**Figure 1.**

with Jakobson's terminology: **Figure 1**

*Peirce's sign-object triadic relation.*

approach to *relative non-arbitrariness* ([24], pp. 92–93).4

triadic sign where the *meaning* of the sign is realized:

*"the proper significate outcome of the sign" (4.127)*

principles manifested in definitions that are relativized, not absolutes.

*(2.228)*

*untranslatable*.

**3.2 The importance of the interpretant: how to talk about meaning**

*"…the meaning of a sign is the sign it has to be translated into;" (4.132)*

of the icon/index/symbol triad, this is not the case with interpretants.<sup>5</sup>

the one fundamental point that has persisted is the **certainty of translation** and the recognition that **all translations change meanings.** Lotman ([28], p. 15) does not ever use Peirce's term but makes an important argument about the inevitability of *tensions* that arise in any speech/communication act. In so doing, Lotman articulates a new perspective—the most valuable information derived from speech acts is a result of the *translation of that which is seemingly* 

<sup>4</sup> Bolinger's work [1, 24–26] provides robust explanations for deconstructing the nineteenth-century view of form/meaning relations in language as "arbitrary" (as posited by Saussure) by exploiting iconic

<sup>5</sup> For examples of analyses using Peircean interpretants, see Savan [13], Andrews [12, 27]. For an example of the use of interpretants to explain principles of language acquisition, see Andrews [12].

*"creates in the mind…an equivalent sign, or perhaps a move developed sign;"* 

It is interesting that while linguistic applications of Peirce are rich in the use

However,

ever occur alone, the icon is embedded in the index and symbol, and the index is embedded in the symbol (Peirce 2.306, 4.447 [21]). Note the following diagram

The principle of **iconicity**, as based on Peirce's definitions, is critical to the reimagining of the non-arbitrariness of the linguistic sign as a relative phenomenon. Bolinger ([1], pp. 18–23), in his work on morphemes, presents interesting examples from English of phonological iconicity at the syllabic and lexical levels, including "hard of hearing" (alliterative/iconic phonological properties, [h\_r/h\_r], and "short circuit," [rt/r\_t]), not "hard of seeing/smelling" and "long circuit." Bolinger also exposes important iconic relationships in prosody in later works and fine-tunes his

Peirce posits three basic types of *interpretants* in the sign complex—*immediate, dynamic*, and *final* (Peirce 8. 315, 8.372, [13]). The *interpretant* is the level of the

**16**
