Audio-Aero Tactile Integration in Speech Perception Using an Open Choice Paradigm

Chapter I - Background

Humans are equipped with a board category of senses with the classic five being vision, hearing, smell, taste and touch which all together makes up the sensory system. Experiences whether enjoyable or miserable, can be experienced because of these sensory modalities. Our sense organs provide the interface for the brain, our coordinating center for sensation and intellect, enabling us to interpret and understand our surrounding environment. There are obvious benefits associated with having multiple senses. Each sense is of optimal use in various circumstances and collectively they increase the likelihood of identifying and understanding of events and objects in our everyday life. For example, integration of auditory and tactile stimuli would be beneficial in understanding our environment when we are a dark room. Risberg and Lubker (1978) found that integrating information from different sensory channels had a supra- additive effect on comprehension of speech as well. This interaction among the senses and the fusion of their information is described by the phrase ‘multisensory or multimodal integration’. So, multisensory integration refers to the influence of one sensory modality over another in the form of enhancement or suppression relative to the strongest “unimodal” response (Stein and Meredith, 1993).

Stimulus identification and localization has been found to be enhanced significantly due multisensory integration (Stein et al., 1988; 1989). Behavioural outcome of multisensory integration in speech perception has been studied exclusively in numerous perception studies. (for reviews, see Calvert et al., 1998; Gick & Donald,2009).

When an auditory stimulus is presented simultaneously, but a different location from a visual stimulus, localisation of sound is perceived to be in the location of the visual stimulus. This perceptual effect or illusion that arise in stimulus localisation as a result of multisensory integration is generally termed as “ventriloquism effect” (Howard & Templeton, 1966). A similar type of illusion has been replicated in speech perception studies as well. McGurk effect is the capability of modifying perceptual outcome in speech perception which is another typical illustration of multisensory integration in humans. (McGurk & MacDonald, 1976). This study will be discussed in detail later in 2.2.1 of chapter II.

The ventriloquist's illusion and the McGurk effect both arise due to interaction between auditory and visual stimulus and thereby highlights the impact and influence that interaction among senses have on the perception of an individual.

In addition to behavioural studies, many electrophysiological studies using neuroimaging techniques have made it possible to investigate multisensory processes in humans. Macaluso and his colleagues (2000) conducted an fMRI study to prove that visual cortex is enhanced by the tactile stimulation of the hand on the same side as the visual stimulus. A similar finding by Sadato et al. (1996) using positron emission tomography study (PET) indicated the activation of primary and secondary visual cortical areas induced by Braille reading in early blind subjects. Likewise, an event-related potential (ERP) study also suggests that activity in the visual cortical areas is modulated by sound (Shams et al.,2001).

The array of behavioural and electrophysiological studies advocates for the influence that interaction among senses can bring in. Even if auditory and visual modalities dominate over others, other modalities like tactile also has been effective in contributing to better perception. This thesis concerns how multisensory integration can enhance communication, more specifically how tactile information can help us to perceive speech better.

Initially, multisensory studies in speech perception focused primarily on the integration of audio and visual information (for reviews, see McGurk & MacDonald, 1976; Green & Kuhl, 1989, 1991). Research along this line has shown that visual information does not only enhance our perception of speech but can also alter it. Recently, researchers started to focus on the effect of tactile sensation beside the auditory signal, starting to reveal the impact of tactile information on speech perception (for reviews see Reed et al.,1989; Gick & Donald,2009). Moreover, different modes of response elicitation like open choice, forced choice has been used to study these multisensory interactions (for reviews see Colin et al.,2008; Sekiyama & Tohkura 1991; Van Wassenhove, V., Grant, K. W., & Poeppel, D. ,2005). In the following sections the associated findings will be elaborated, quoting evidences from relevant behavioural studies, thereby substantiating the fact of our ability to perceive speech as a multimodal sensation.

Outline of thesis

In this thesis, I employ the open choice as the response elicitation paradigm and investigate if tactile stimuli when presented with auditory signal improve perception of monosyllables, in order to explore how the human perceptual system synthesizes speech.

In Chapter II, 2.1 discusses production of speech in relation to speech characteristics that distinguishes perception of phonemes. 2.2 reviews speech perception as a multisensory event with evidences from behavioural studies and 2.3, methodological aspects like response type and stimulus type that can influence speech perception is elaborated with supporting literature.

Chapter III and IV, describes the statement of the problem and methodology respectively.

In Chapter V, results provided by the study will be discussed.

Finally, in the General Discussion (Chapter VI), I summarize the empirical findings and suggest further directions of research.